Characteristics of Effective Teaching in Canadian UniversitiesAn Analysis based on the Testimony of a Thousand Graduates EDWARD F. SHEFFIELD* Abstract A thousand graduates of nineteen Canadian universities, in a variety of disciplines, responded to an invitation to name the professors they remembered as excellent teachers and to say why. Analyses of their comments reveal what they considered t o be characteristics of effective university teaching. Comparisons of various sub-groups suggest that there are some differences in the patterns of good teaching between anglophone and francophone faculties and between fields of study—humanities, social sciences, biological sciences and physical sciences—but the patterns are basically similar in most university settings. As a contribution to the improvement of teaching in the universities of Canada, a recent study sought to identify professors** who were effective teachers, and to discover from them and from their former students what was characteristic of them and their teaching. Deans and alumni directors provided the names and addresses of roughly seven thousand graduates of twenty-four faculties and schools in nineteen universities—members of the classes of 1968 (the previous year), 1963 (five years earlier) and 1958 (ten years earlier). Among them they represented the fourteen top faculties in terms of numbers of first degrees awarded in 1966—67, the latest year for which data were published at the time. There were four of arts, three of science, two each of education, engineering, commerce/administration, law and medicine, and one each of nursing, physical education, agriculture, h o m e economics, pharmacy, dentistry and library science. Some were in large universities, some in small. Five of the twenty-four were French-language faculties, nineteen were English-language, and the list included at least one university in every one of the * Edward F. Sheffield is Professor of Higher Education, University of Toronto * * As is customary in Canada and the United States, the term "professor" is used here to include university teachers of all ranks. 8 STO A ten provinces, from Memorial University of Newfoundland to the University of British Columbia. Letters were sent to these seven thousand graduates inviting them to name professors they remembered as excellent teachers, and to say what there was about those professors and their teaching that made them effective. It was explained that of the teachers so identified a score or more would be invited to write essays on the teaching of undergraduate students, reflecting their own beliefs and practices, and that these would be published. (Copies of the letter and attachments form Appendix A.) Replies were received from just over a thousand. On the average each named two professors; some named as many as six. The number of different professors identified as excellent teachers was 1,002. Forty-one graduates took the trouble to send back the response form saying that, unfortunately,they had had no professor whom they would classify as an excellent teacher. After applying a simple weighting scale to take account of the differences in class size and the order in which professors' names appeared on the returned forms, in each faculty there was identified at least one, in most cases several, of those mentioned with sufficient frequency to suggest that they were excellent teachers. Because only one was sought in each faculty, the final choice was to some extent arbitrary. Twenty-four professors so chosen were invited to contribute essays on their teaching. Twenty-three did so and their essays have been compiled as a book entitled Teaching in the Universities: No One Way (Sheffield, 1974). The chief purpose of this paper is to present the results of the analyses of comments on effective teaching made by the thousand responding graduates. Earlier it was reported that seven thousand graduates were invited to participate in the study. To be precise, they numbered 6,928, and 1,014 (14.6%) responded to the invitation. The percentage of response was highest (15.9%) for the class of '68, lower (14.2%) for the class of '63, and still lower (11.4%) for the class of '58. The range of participation rates varied greatly from one faculty to another—from less than ten per cent in five cases to more than twenty per cent in seven. (A note on the representativeness of the sample is to be found in Appendix B.) Some indication of the interest of those who responded is seen in the fact that 88% of them signed their names and gave their addresses in order that they might receive a report on the results of the project. When the request to graduates was being planned, it was decided to invite free comment on the characteristics, qualities, m e t h o d s and procedures which, in their view, identified each of the professors they named as an excellent teacher. They could have been asked, instead, to check a list of possible characteristics phrased in advance by the investigator. The replies received indicated that the decision taken yielded a wealth of factors and forms of expression which would have been obscured by a checklist. But there was posed the problem of classifying the responses. These were grouped, on the basis of personal judgment and therefore somewhat imprecisely, into four main categories and 53 sub-categories (see Table 1). The data provided by the responding graduates were subjected to a number of analyses and comparisons. Those which are reported here are: STO A 9 analysis of the overall pattern for all 24 faculties and schools; comparison of faculties of arts and science with professional schools; comparison of anglophone and francophone faculties; comparison by size of university; comparison of humanities, social sciences, biological sciences and physical sciences; comparison of pure and applied studies; comparison by course level; comparison by class size; comparison by year of graduation; comparison of undergraduates and graduates in arts and science. The essentials of these analyses follow. Table 1 shows the overall pattern of the characteristics identified. It will be described in detail later. Perhaps of most immediate interest is the list of characteristics most frequently mentioned. Here are the top ten, in order of frequency of mention: 1 2 3 4 5 6 7 8 9 10 Master of his subject, competent Lectures well prepared, orderly Subject related to life, practical Students' questions and opinions encouraged Enthusiastic about his subject Approachable, friendly, available Concerned for students' progress, etc. Had a sense of humour, amusing Warm, kind, sympathetic Teaching aids used effectively The Overall Pattern As has been mentioned, consideration was given at the beginning to the possibility of making a list of perhaps a score or more characteristics of effective university teaching which might be checked by the graduates to indicate which applied to the professors they remembered as excellent teachers. This was discarded, however, in favour of an open-ended kind of question which would leave the graduates free to say what came to their minds about the teachers they remembered. What resulted was a fantastic range of words descriptive of teachers and their ways of teaching. The responses were grouped first into four main categories: A B C D Personal qualities or attributes Subject mastery, scholarship, devotion to teaching Attitudes toward and relations with students Methods and procedures course content and organization presentation of material other N 1,758 1,603 1,295 4,390 % 19 18 14 48 Each of these was then divided into a number of sub-categories, as discrete as possible. 10 STO A Inevitably there was a good deal of overlapping, not only between sub-categories within main categories but also between categories. It could have been avoided, or at any rate it could seem to have been avoided, if a check-list had been provided in the beginning, but then who would know what would have been said by the graduates if they had not been so guided? Table 1 lists the four categories and 53 sub-categories, with additional items provided for "other c o m m e n t " (by graduates who named excellent teachers but did not say why) and " n o c o m m e n t " (for the graduates who responded only to say that they had met no good teacher while they were undergraduates). The number of comments by graduates in each sub-category was tallied and expressed also as a percentage of the total number of comments recorded. It is of significance to note that if a particular point about a professor was made more than once by a graduate, that point was tallied only once. To illustratewhere a graduate described a former teacher as being kind and, later, as having a warm personality, one tally only was recorded for sub-category 5. Table 2 presents these characteristics of effective university teaching in order of the frequency with which they were mentioned by responding graduates. Attention has been drawn already to the first ten, each of which accounts for at least 3.0% of the total number of comments. Including these, 28 items (SC 13, consisting of "other comments," is omitted from this count) were mentioned at least 100 times by respondents. It is perhaps needful to point out again, however, that the comments presented are descriptive rather than prescriptive. If graduates had been asked what they thought the characteristics of effective university teaching should be the pattern might have been somewhat different. It certainly would have omitted the critical comments which were in fact made. Another way of presenting the data is to tally the number and calculate the percentage of professors for whom each characteristic was mentioned by at least one responding graduate. The results of this analysis do not differ significantly from those shown in Charts 1 and 2 so they are not spelled out here. Characteristics of Effective Teaching Identified in Other Studies In a study based on the opinions of undergraduates at the State University of New York at Albany in 1966 (Musella and Rusch, 1968), students were asked "what teacher behaviors promoted their thinking." Classification of 395 responses to this open-ended question indicated that the top ten were: 1 2 3 4 5 6 7 8 9 10 Attitudes toward subject (enthusiastic, interested) Attitudes toward students (sympathetic, helpful, actively concerned) Questioning (effective use of questions) Speaking ability (skilled in presenting material) Knowledge of subject (exceedingly well informed) Organization of subject matter (systematic and thoroughly organized) Discussion (extensive and effective) Ability to explain (explanations clear and to the point) Tolerance to [sic] disagreement (encourages and values reasonable disagreement) Instructor as " h u m a n being" (attractive personality) S T O A 13 Table 1 Characteristics of Effective University Teaching Graduates' Comments - All 24 Faculties and Schools (arranged in order of code numbers) Code Categories Suh-eategories 1 Personal D y n a m i c , vivacious 2 qualities or Dramatic 3 attributes 4 A Intelligent, brilliant M a t u r e , wise Comments No. % 260 2.8 4 0.0 110 1.2 8 0.1 5 Warm, kind, sympathetic 278 3.0 6 Had a sense of h u m o u r , a m u s i n g 321 3.5 7 Humble, modest 60 0.7 8 Flexible, open-minded 53 0.6 9 Honest, sincere, genuine 10 Fair, impartial 1 ! Conscientious, thorough 12 O r g a n i z e d , logical, e f f i c i e n t 13 101 1.1 43 0.5 166 1.8 90 1.0 O t h e r (including critical) c o m m e n t s 264 2.9 14 Subject M a s t e r of h i s s u b j e c t , c o m p e t e n t 717 7.8 15 mastery, Kver l e a r n i n g , u p t o d a t e , i n q u i r i n g 132 1.4 16 scholarship. E x p e r i e n c e d in h i s f i e l d 133 1.5 17 d e v o t i o n to E n t h u s i a s t i c a b o u t his s u b j e c t 385 4.2 18 teaching Broad k n o w l e d g e and interests 59 0.6 19 B 20 21 22 L o v e d t e a c h i n g , g a v e it h i g h p r i o r i t y Other comments 163 1.8 14 0.2 Attitudes R e s p e c t f o r s t u d e n t s as p e r s o n s 142 1.6 toward and C o n c e r n e d for students' progress, etc. 325 3.6 23 relations D e m a n d i n g of hard w o r k , exccllence 155 1.7 24 with Sensitive to s t u d e n t s ' feelings 214 2.3 25 students A p p r o a c h a b l e , friendly, available 372 4.1 S o u g h t social c o n t a c t s w i t h s t u d e n t s 27 0.3 27 I n t e r e s t e d in s t u d e n t a c t i v i t i e s 15 0.2 28 Other (including critical) c o m m e n t s 45 0.5 26 C 29 Methods and Students expected to do much on their o w n 30 procedures A t t i t u d e s , principles stressed over facts 145 1.6 69 0.8 31 D 555 6.1 32 -course A p p r o a c h to subject critical, scientific 66 0.7 33 c o n t e n t and P r o f e s s o r e x p r e s s e d o w n p o i n t of view 55 0.6 34 organization C o u r s e p l a n n e d at s t u d e n t s ' level 138 1.5 35 36 C o u r s e goals and e x p e c t a t i o n s clear Course organized, systematic 42 251 0.5 2.7 37 S u m m a r i e s and reviews provided 28 0.3 38 Other comment on course organization 38 0.4 39 L e c t u r e s well p r e p a r e d , o r d e r l y 40 41 Voice audible, well-modulated, etc. 93 1.0 Use of language, v o c a b u l a r y , n o t a b l e 83 0.9 -presenta- Lectures oratorical, dramatic 55 0.6 43 tion of Professor rarely used n o t e s ; i n f o r m a l 175 1.9 44 material Main p o i n t s stressed; o u t l i n e p r o v i d e d 187 2.0 42 Subject related to life, practical 712 7.8 45 Professor q u e s t i o n e d , used Socratic m e t h o d 92 1.0 46 Students' questions, opinions encouraged 481 5.3 47 T e a c h i n g aids used effectively 287 3.0 48 Professor experimented with m e t h o d s 45 0.5 49 Other (including critical) c o m m e n t s 87 1.0 50 S m a l l - g r o u p a r r a n g e m e n t s used well 257 2.8 51 Assignments reasonable, helpful 195 2.1 52 Grading fair, innovative 253 2.8 53 Other c o m m e n t s o n m e t h o d s and procedures 10 0.1 8 0.1 54 Other comment 55 No c o m m e n t 89 1.0 9,143 100.1 STO A 12 Table 2 Characteristics of Effective University Teaching Graduates' Comments - All 24 Faculties and schools (arranged in order of frequency) Code Sub-category Category * No. % 717 7.8 14 B Master of his s u b j e c t , c o m p e t e n t 39 D L e c t u r e s well p r e p a r e d , o r d e r l y 712 7.8 31 D Subject related to life, practical 555 6.1 46 D Students' questions, opinions encouraged 481 5.3 17 B Enthusiastic a b o u t his subject 385 4.2 25 C A p p r o a c h a b l e , friendly, available 372 4.1 22 c Concerned for students' progress, etc. 325 3.6 6 A Had a sense of h u m o u r , a m u s i n g 321 3.5 5 A Warm, kind, sympathetic 278 3.0 47 D Teaching aids used effectively 278 3.0 13 A Other c o m m e n t s o n personal qualities or attributes 264 2.9 1 A D y n a m i c , vivacious 260 2.8 50 D Small-group a r r a n g e m e n t s used well 257 2.8 52 D Grading fair, innovative 253 2.8 36 D Course organized, systematic 251 2.7 24 C Sensitive to s t u d e n t s ' feelings 214 2.3 51 D Assignments reasonable, helpful 195 2.1 44 D Main p o i n t s stressed; outline provided 187 2.0 43 D Professor rarely used notes; informal 175 1.9 11 A Conscientious, thorough 166 1.8 19 B L o v e d t e a c h i n g , g a v e it h i g h p r i o r i t y 163 1.8 23 C D e m a n d i n g of h a r d w o r k , e x c e l l e n c e 155 1.7 29 D Students expected to d o m u c h on their o w n 145 1.6 21 C R e s p e c t f o r s t u d e n t s as p e r s o n s 142 1.6 34 D C o u r s e p l a n n e d at s t u d e n t s ' level 138 1.5 16 B E x p e r i e n c e d in h i s f i e l d 133 1.5 15 B Ever learning, u p to d a t e , i n q u i r i n g 132 1.4 3 A Intelligent, brilliant 110 1.2 9 A Honest, sincere, genuine 101 1.1 40 45 D D Voice audible, well-modulated, etc. Professor questioned, used Socratic m e t h o d 93 92 1.0 1.0 Organized, logical, e f f i c i e n t 90 1.0 No comment 89 1.0 12 A 55 49 41 D Other c o m m e n t s o n presentation of material 87 1.0 D Use of language, v o c a b u l a r y , n o t a b l e 83 0.9 30 D Attitudes, principles stressed over facts 69 0.8 32 D A p p r o a c h to subject critical, scientific 66 0.7 7 A Humble, modest 60 0.7 18 B Broad knowledge and interests 59 0.6 33 D Professor expressed o w n p o i n t of view 55 0.6 42 D Lectures oratorical, dramatic 55 0.6 8 A Flexible, open-minded 53 0.6 28 C Other c o m m e n t s on relations with students 45 0.5 48 D Professor experimented with methods 45 0.5 10 A Fair, impartial 43 0.5 35 D Course goals, e x p e c t a t i o n s clear 42 0.5 38 D Other c o m m e n t s on course organization 38 0.4 37 D Summaries and reveiws provided 28 0.3 26 C S o u g h t social c o n t a c t s w i t h s t u d e n t s 27 0.3 27 C I n t e r e s t e d in s t u d e n t a c t i v i t i e s 15 0.2 20 B Other c o m m e n t s on subject mastery, etc. 14 0.2 53 D Other comments on methods and procedures 10 0.1 4 A Mature, wise 8 0.1 54 - Other comment 8 0.1 Dramatic 4 0.0 2 A •Categories: (1-13) Personal q u a l i t i e s or a t t r i b u t e s (14-20) Subject mastery, scholarship, devotion to teaching C (21-28) Attitudes toward and relations with students D Source: A B (29-53) Methods and procedures Table 1 STO A 13 "Although the students were asked to indicate characteristics promoting thinking," explain Musella and Rusch, "it is not inconceivable that the responses of the students were really responses to what characterizes 'most effective' teaching." With specific reference to qualities important for teaching in general, the students were asked to rank ten pre-formulated items. The five thought to be most important by the students are listed below in order of importance: 1 Expert knowledge of subject 2 Systematic organization of subject matter 3 Ability to explain clearly 4 Enthusiastic attitude toward subject 5 Ability to encourage thought A study of Criteria of Effective Teaching in an Institution of Higher Education by the Office of Institutional Research of the University of Toledo (1967), produced sixty criteria of effective teaching arranged in rank order on the basis of ratings by students, faculty and alumni of that University. The first dozen were as follows: 1 Being well prepared for class 2 Establishing sincere interest in the subject being taught 3 Demonstrating comprehensive knowledge of his subject 4 Using teaching methods which enable students to achieve objectives of the course 5 Constructing tests which search for understanding on the part of the students rather than rote memory ability 6 Being fair and reasonable to students in evaluation procedures 7 Communicating effectively at levels appropriate to the preparedness of students 8 Encouraging intelligent independent thought by students 9 Organizing the course in logical fashion 10 Motivating students to do their best 11 Treating student with respect 12 Acknowledging all questions to the best of his ability Two years later the same Office (Toledo, 1969) tested its findings in three other universities: New Mexico State University, Northern Illinois University and Western Kentucky University. The first eleven criteria of the original study were the first eleven in the second, although in a slightly different order. Based in part on the Toledo studies, the University of British Columbia (1970) developed a list of criteria for use in making its Master Teacher Award: 1 Having a comprehensive knowledge of the subject 2 Being habitually well prepared for class 3 Having enthusiasm for the subject 4 Having the capacity to arouse interest in it among the students 5 Establishing good rapport with the students both in and out of classes 6 Encouraging student participation in class 7 Setting a high standard and successfully motivating students to try to attain such a standard 14 STO A 8 Communicating effectively at levels appropriate to the preparedness of students 9 Utilizing methods of evaluation of student performance which search for understanding of the subject rather than just ability to memorize. Hildebrand, Wilson and Dienst (1971) identified characteristics which distinguished best f r o m worst teachers and summarized the characteristics which were peculiar to the best: 1 Has command of the subject, presents material in an analytic way, contrasts various points of view, discusses current developments, and relates topics to other areas of knowledge. 2 Makes himself clear, states objectives, summarizes major points, presents material in an organized manner, and provides emphasis. 3 Is sensitive to the response of the class, encourages student participation, and welcomes questions and discussion. 4 Is available to, and friendly towards students, is interested in students as individuals, is himself respected as a person, and is valued for advice not directly related to the course. 5 Enjoys teaching, is enthusiastic about his subject, makes the course exciting, and has self-confidence. As a result of a survey of students at the University of Bradford, Smithers (1969) reported that "students expect a lecturer to be able to lecture. In particular, they look to him for expertise and lucidity. They also want lecturers to thoroughly prepare their lectures, to give them an obvious structure and to respond to questions." In his review of research on the student-teacher relationship in the college classroom, Wagner Thielens, Jr. (1971) reports: The student portrait of the ideal professor is relatively consistent from campus to campus. Thorough subject knowledge is the trait most widely sought . . . . Other highly regarded qualities are the teacher's capacity to convey information and ideas clearly, his ability to organize his presentation well, and his enthusiasm for his topic. All of which suggests that the characteristics of effective teaching revealed in this study are similar to those usually identified. Comparisons In the comparisons which follow, emphasis is placed on those items or characteristics which, like the ten most frequently mentioned in the overall pattern (page 10 above), are at or above the three per cent level.* This kind of selection is intended to make it easier to comprehend the data by simplifying the presentation. Complete data are available on request from the author. Comparison of Faculties of Arts and Science with Professional Schools In this comparison, thirteen items are at or above the three per cent level either for * An item at or above the three per cent level is one for which at least three per cent of the total number of comments were recorded, i.e. at least 274 out of the total of 9,143. STO A 15 faculties of arts and science (N = 3,362) or for professional faculties and schools (N = 5,781). In no case is one of these items mentioned twice as often by one group as by the other; in no case even fifty per cent more often. It would appear, therefore, that between these two groups there is no substantial difference in the patterns of identified characteristics of effective teachers and teaching. Comparison of Francophone and Anglophone Faculties There are five francophone faculties in the total sample (N = 1,143): sciences de l'éducation and sciences de l'administration at l'Université Laval, droit and chirurgie dentaire at l'Université de Montréal, and arts at l'Université d'Ottawa. These are compared with the five most similar anglophone faculties (N = 2,024) in the project: Education at the University of Alberta, Commerce at Sir George Williams University, Law at the University of British Columbia, Medicine at Dalhousie University, and Arts at St. Dunstan's University. In this case fourteen items are at or above the three per cent level. In two instances there are more than twice as many comments by one group as by the other, and in another seven instances (six, when SC 13 consisting of "other c o m m e n t s " is excluded) one set of comments is at least fifty per cent more numerous than the other. Thus for eight of the fourteen items selected for scrutiny the difference is worthy of examination. Those mentioned markedly more often by graduates of the anglophone faculties were: Sub -category Ratio 6 Had a sense of humour, amusing 25 Approachable, friendly, available 50 Small-group arrangements used well 17 Enthusiastic about his subject 22 Concerned for students' progress, etc. 4 7 Teaching aids used effectively (3.1 to 1) (almost 2.0 to 1) (1.8 to 1) (1.7 to 1) (1.6 to 1) (1.6 to 1) Those mentioned much more often by graduates of the francophone faculties were: 16 31 Experienced in his field Subject related to life, practical (2.8 to 1) (1.7 to 1) There are real differences between these two groups. Can they be explained? Would it be appropriate to suggest that in the case of the francophone graduates there seems to have been more experience of distance between the lecturing professor and his students than in the case of the anglophone graduates? If so, was this really a difference in experience or only a difference in those parts of the experience which were thought worth mentioning? Are the differences culture-linked? Comparison by Size of University The nineteen universities in the study were divided into three groups by the number of full-time students enrolled in 1967-68: L a r g e - m o r e than 10,000 (Alberta, British Columbia, McGill, Montreal and Toronto), medium-sized—5,000 to 10,000 (Carleton, Laval, Manitoba, McMaster, Ottawa, Queen's, Saskatchewan, Sir George Williams and Western Ontario) and small—less than 5,000 (Dalhousie, Memorial, New Brunswick, Nova Scotia 16 STO A Technical and St. Dunstan's). (The numbers of comments recorded by group are 4,167, 3,915 and 1,061.) When responses from graduates of these three groups are compared, there are sixteen sub-categories at or above the three per cent level. If the highest in each group of three is related to the lowest, there is one item (SC 44) for which the ratio is greater than 2 to 1, and one other for which it is at least 1.7 to 1 :* Sub-category Ratio 44 Main points stressed, outline provided (2.4 to 1) 1.0 2.4 Had a sense of humour, amusing (1.7 to 1) 1.7 1.0 6 Large Medium Small Sub-category 44 (main points stressed; outline provided) is most characteristic of the small universities, least of the large. The reverse is true of SC 6 (had a sense of humour, amusing). It must be concluded that few differences in characteristics of effective teaching seem to be related directly to university size. Comparison of Humanities, Social Sciences, Biological Sciences and Physical Sciences When graduates named professors whom they remembered as good teachers, they indicated what courses they had taken from them, e.g. History 15, Pharmacology 4 4 3 . The data by subject were grouped by field—humanities (N = 2,033), social sciences (N = 3,796), biological sciences (N = 1,533), physical sciences (N = 1,781)—in accordance with the classification used by the Dominion Bureau of Statistics (now Statistics Canada) in its reports on Salaries and Qualifications of Teachers in Universities and Colleges (see, for example, the report for 1968-69, Tables 8-11). For eighteen sub-categories there were comments at or above the three per cent level. When one relates the highest and lowest percentage of comments in each group of four, it is seen that there are three for which the ratio is at least 2 to 1, and another six (making half of those at the three per cent level) for which it is at least 1.7 to 1. Sub-category 44 15 34 Main points stressed, outline provided Ever learning, up to date, inquiring Course planned at students'level Ratio Humanities (6.5 to 1) 1.0 (3.8 to 1) 1.0 (2.8 to 1) 1.0 Social sciences Biological sciences Physical sciences 6.5 3.8 1.0 3.4 The next six items in order of difference in frequency of mention among the four fields are: *When comparing pairs of series, ratios of at least 1.5 to 1 are drawn to the attention of the reader as indicating differences worthy of mention. Such a ratio is more likely when highs and lows of more than two series are being compared. For this reason, in such cases only ratios of at least 1.7 to 1 are pointed out. STO A 6 Had a sense of humour, amusing 17 Enthusiastic about his subject 47 Teaching aids used effectively 36 Course organized, systematic 39 Lectures well prepared, orderly 46 Students' questions, opinions encouraged 17 (almost 2.0 to 1) 2.0 (1.9 to 1) 1.9 (1.8 to 1) 1.0 (1.8 to 1) 1.0 1.8 (1.7 to 1) 1.0 1.7 (1.7 to 1) 1.7 1.0 1.0 1.0 1.0 1.8 It was in the physical sciences that item 44 (main points stressed; outline provided) was mentioned with relatively great frequency, and in the humanities in which it was seldom mentioned. Indeed, on this point there is a gradation down from physical to biological to social sciences to humanities. On item 34 (course planned at students' level) the physical sciences are high again, with the other three fields about equal, and much lower. With respect to sub-category 15 (ever learning, up to date, inquiring) mention is noticeably more frequent in the biological sciences than in the other three fields. There seem to be substantial differences in the patterns of characteristics of effective teaching among these fields. It is not obvious, however, that the patterns can be described and differentiated in general terms. Comparison of Pure and Applied Studies The DBS classification of fields of study used for the data on which the previous comparison is based was used also to classify responses into two other dimensions: pure and applied studies (N = 5,226, 3,917). The comments made about a chosen professor who taught French, for example, were classified as " p u r e " while those about a professor who taught soil mechanics were classified as "applied." This comparison for the items at or above the three per cent level shows that in no case is the percentage of comments as much as twice as great for one series as the other. In only one case, with respect to sub-category 6 (had a sense of humour, amusing), was the ratio as high as 1.5 to 1, and indeed it was no higher than that—the advantage being to the pure studies series. The conclusion to which this comparison leads is that the patterns observed in pure and applied studies are for all practical purposes the same. Comparison by Course Level The information provided by responding graduates made it possible to know whether a mentioned professor had been met at the level of one, two, three, four, five, six, seven or eight years beyond the level of junior matriculation. For example, a course in the first year of Arts at St. Dunstan's (where the level of entry was that of junior matriculation) would be classified as level 1, while a course in the third year of Education at Alberta (where the level of entry was that of senior matriculation) would be at level 4. The numbers at levels 1, 6, 7 and 8 were relatively small, so level 1 is grouped with level 2, and STO A 18 levels 6, 7 and 8 are combined. (Total numbers of comments for the five resulting categories, from 1-2 up, are 1,838, 2,000, 2,162, 1,697 and 1,397.) Here we find no item for which the ratio of responses is greater than 1.7 for the highest to 1 for the lowest. At 1.7 to 1 there are two: Sub-category 17 6 Enthusiastic about his subject Had a sense of humour, amusing Ratio Years 1-2 Year 3 Year 4 (1.7 to 1) 7.7 1.0 (1.7 to 1) 1.7 Year 5 Years 6-7-8 1.0 For sub-category 17, levels 1-2 and 3 are high, and levels 4 to 8 low. Much the same is true of sub-category 6, although in this case level 6-7-8 too is low. These differences are minor, suggesting that patterns of effective teaching do not vary in accordance with course level. Comparison by Class Size The data were originally grouped in accordance with seven ranges of class size: Up to 10, 11-30, 31-60, 61-90, 91-120, 121-200, and over 200. In order to create groupings large enough to allow valid comparison, the first two classes were joined, as were the last two. (Numbers of comments in the four ranges of class size, from small to large, are 3,563, 2,998, 1,282 and 1,256.) There are three items for which differences in the frequency of comments are as great as 2 to 1 and they are worthy of note: Sub-category 36 6 52 Course organized, systematic Had a sense of humour, amusing Grading fair, innovative Ratio Up to 30 (2.7 to 1) 1.0 (2.0 to 1) (2.0 to 1) 2.0 31-60 61-120 Over 120 2.7 1.0 2.0 1.0 Items 36 and 6 were both mentioned with noticeably more frequency by graduates commenting on professors met in the largest classes (over 120). Were these characteristics actually more common in such large classes, or is it just that they were especially appreciated when found there? On the other hand, item 52 was more common as a comment on professors met in the smallest classes, with a fairly regular gradation from small classes to large. Is fairness in grading thought by students to be a function of the ratio of students to professors, within single classes? There are no other sub-categories for which the differences are as great as 1.7 to 1. It would seem, then, that although the differences in patterns of characteristics of effective teaching by class size are not general, there are a few items on which the largest classes are noticeably different from the smallest. STO A 19 Comparison by Year of Graduation The possibility that graduates' views of their teachers might change over the years following departure from the university, or that there might be changing styles in the appreciation of good teaching, accounts for the choice of three graduating classes at intervals of five years as participants in the study. As has been mentioned elsewhere, the latest class for which names and addresses were available when the project was begun was that of 1968 (N = 5,330). To it, then, were added the classes of 1963 (N = 2,452) and 1958 (N = 1,350). This meant that some members of the class of '58 were commenting on professors with whom they had studied as many as fifteen years earlier. The outstanding trait of the class of '58 is that they remembered their good teachers as persons who were demanding of hard work, excellence (SC 23); that is the one item for which the percentage of comments is more than twice as great by members of one graduating class than another, in this case a ratio of 2.6 for the class of '58 to 1 for the class of '68. There are only two other items for which ratios are as great as 1.7 to 1.: Sub-category Ratio 1958 23 Demanding of hard work, excellence 5 Warm, kind, sympathetic 52 Grading fair, innovative (2.6 to 1) (1.7 to 1) (1.7 to 1) 2.6 1.7 1.0 1963 1968 1.0 1.0 1.7 There is a higher proportion of comments about the warmth, kindness and sympathy of their teachers (SC 5) by members of the class of '58 than by members of the two later classes, whereas comments on fair or innovative grading (SC 52) are most common in the case of the class o f ' 6 8 , less common for the class o f ' 6 3 and least common for the class of '58. On the whole, therefore, it appears that there are some but not many differences associated with year of graduation, or years since graduation. Were the teachers of members of the class of '58 really warmer, kinder and more demanding, or had a legend time to grow? It is easier to observe the relation between fair or innovative grading and years since graduation because there is no doubt that more emphasis has been placed recently on this feature of university teaching. Comparing Undergraduates and Graduates in Arts and Science In the fall of 1968, when plans for this study were being shaped, a draft of the questionnaire intended for graduates was distributed in an appropriately modified form to a class of 139 students in the third year of arts and science at the University of Toronto. They were told the purpose of the questionnaire and invited to complete it and comment on it in order to test and improve it. The essential difference between the form as they received it and the form which went later to graduates across the country was that the undergraduates were invited to name the one professor whose teaching they thought was best among those under whom they had studied in their first two years at the University. (It was at their suggestion that the final form of the questionnaire invited graduates to name one or more professors whose teaching was remembered as being especially effective.) The comments of the arts and science undergraduates (N = 808) were compared with those of the graduates of the seven faculties of arts and science (N = 3,362) included in the 20 STO A project. There are only four items for which the difference in percentage of comments is greater than 1.4 to 1, but in these cases the difference is marked, ranging from 2.9:1 as follows: Sub-category Voice audible, well-modulated 47 Teaching aids used effectively 14 Master of his subject, competent 44 Main points stressed, outline provided Ratio Graduates Undergraduates 40 (2.9 to 1) 1.0 2.9 (2.4 to 1) 1.0 2.4 (2.4 to 1) 2.4 1.0 (1.9 to 1) 1.0 1.9 It was the graduates who stressed subject mastery (SC 14) much more than the undergraduates, but in the other three cases highlighted here the undergraduates were those with the higher percentages of comment. One can speculate on the possible reasons for these differences. The data provided by the undergraduates revealed, for one thing, that few of them had had experience of classes of fewer than 100 students in their first two years, so almost all of the teachers they mentioned had been met in large classes. Another factor which may be meaningful is that the majority of the students, and the professors they mentioned with greatest frequency, were in science courses, for which audio-visual aids were not only appropriate but available. It is possible, therefore, that the comparison attempted here is not wholly valid, that it is more accurately described as one between classes of first and second year in predominantly science courses and classes throughout the range of course levels from 1 to 5 in courses divided between arts and science. Summary and Confirmation Of the nine comparisons made above there are two which show marked and probably significant differences in pattern—between francophone and anglophone faculties and among the four main fields of study: humanities, social sciences, biological sciences and physical sciences. In another two cases there are some differences worthy of mention— among class sizes and according to graduating year. Another comparison—between undergraduates and graduates in arts and science—may belong in this group but there is some doubt whether the data available are sufficiently relevant to make the comparison valid. Finally, there are four comparisons which yield little or no evidence of pattern differences—between faculties of arts and science and professional schools, among universities of different size, between pure and applied studies, and by course level. These observations are confirmed by the calculation of rank order correlations between several of the series of comments on characteristics of effective teaching. Rank correlation was calculated for the two comparisons mentioned in the previous paragraph as showing " m a r k e d " differences in pattern, one of the two showing " s o m e " differences, and three of the four said to "yield little or no evidence of pattern differences." The results are shown below. STO A Pairs of series Comparisons compared showing 21 Rank "marked" differences correlation* in pattern Francophone faculties Anglophone faculties .59 Humanities Humanities Humanities Social sciences Social sciences Biological sciences Social sciences Biological sciences Physical sciences Biological sciences Physical sciences Physical science .78 .67 .73 .63 .71 .72 Comparisons Class of '58 showing "some" differences in pattern Class of '63 Class o f ' 5 8 Class of '63 Comparisons Class o f ' 6 8 Class of '68 showing "little"difference .82 .76 .81 in pattern Faculties of arts and science Professional faculties and schools .79 Large universities Large universities Medium-sized universities Medium-sized universities Small universities Small universities .83 .80 .82 Pure studies Applied studies .79 Note that, with the exception of the comparisons between francophone and anglophone faculties and all but one of those between fields of study, rank correlations are in the neighborhood of .80, indicating much agreement or, to put it in the terms used in this analysis, little difference in patterns of comments. Even the comparison with the least correlation (.59) indicates considerable agreement. Of the ten characteristics drawing three per cent or more of graduates' comments in the overall pattern (see Table 2), nine appeared at or above the three per cent level in all of the sub-divisions made for comparisons, and the tenth appeared in all but one. On none of these was there agreement in all comparisons. Those which came nearest to commending complete agreement, no matter what sub-divisions of comments were made, were these six: Sub-category Agreement 14 Master of his subject, competent 5 Warm, kind, sympathetic 25 Approachable, friendly, available 31 Subject related to life, practical 39 Lectures well prepared, orderly 46 Students' questions, opinions encouraged 8 7 7 7 7 7 out out out out out out of of of of of of 9 8 9 9 9 9 in cases cases cases cases cases cases The one item among the top ten in the overall pattern on which there was marked disagreement when data were sub-divided for the making of comparisons was sub-category Corrected for ties. Standard error at the .60 level is .15; at the .70 level, .14; at the .80 level, .11. STO A 22 6 ( h a d a sense o f h u m o u r , a m u s i n g ) . W i t h r e s p e c t t o t h i s c h a r a c t e r i s t i c t h e r e w a s a d i f f e r e n c e o f at l e a s t 1.5 t o 1 in six cases o u t o f n i n e . In s u m , it c a n b e said o f t h e c o m p a r i s o n s t h a t , a l t h o u g h t h e r e are a f e w e x c e p t i o n s , t h e c h a r a c t e r i s t i c s of e f f e c t i v e u n d e r g r a d u a t e t e a c h i n g are b a s i c a l l y similar in m o s t u n i v e r s i t y settings. Conclusion This s t u d y has n o t revealed n e w i n f o r m a t i o n a b o u t t h e general characteristics of effective u n i v e r s i t y t e a c h i n g . R a t h e r , it h a s u n d e r l i n e d w h a t w a s a l r e a d y k n o w n a b o u t t e a c h i n g in some countries, n o t a b l y t h e U . S . A . , B r i t a i n a n d A u s t r a l i a . It is t h o u g h t , h o w e v e r , t o b e t h e first s u c h n a t i o n - w i d e s t u d y in C a n a d a a n d , t h e r e f o r e , t h e first t i m e it h a s b e e n possible t o d o c u m e n t t h e f a c t t h a t t h e overall p a t t e r n of g o o d t e a c h i n g in C a n a d i a n u n i v e r s i t i e s is l i k e t h a t o f g o o d t e a c h i n g e l s e w h e r e . T h e c o m p a r i s o n s of p a t t e r n s o f c o m m e n t s m a d e b y v a r i o u s s u b - g r o u p s o f g r a d u a t e s are m o r e n o v e l t h a n t h e d a t a o n t h e overall p a t t e r n . T h e y t o o stress t h e u n i v e r s a l i t y o f t h e basic c h a r a c t e r i s t i c s o f e f f e c t i v e u n d e r g r a d u a t e t e a c h i n g . It m u s t b e r e m e m b e r e d , h o w e v e r , t h a t t h e p a t t e r n s r e p o r t e d h e r e are d e s c r i p t i v e o f g o o d u n i v e r s i t y t e a c h e r s collectively r a t h e r t h a n i n d i v i d u a l l y . In d e v e l o p i n g h i s o w n s t y l e , a n y o n e p r o f e s s o r m a y d e p a r t n o t i c e a b l y f r o m t h e overall p a t t e r n a n d still b e a n e f f e c t i v e t e a c h e r . References British Columbia, University of, Office of Academic Planning (1970). "Alternative Criteria to Evaluating Teaching Effectiveness." Vancouver, B.C.: University of British Columbia (unpublished memorandum). Eble, Kenneth E. (1970). The Recognition and Evaluation of Teaching. Salt Lake City, Utah: Project to Improve College Teaching. Gagné, Françoys et Chabot, Marie (1970). " P E R P E , une conception neuve du perfectionnement pédagogique," Perspectives, 6: 160-181. Hildebrand, Milton, Wilson, Robert C. and Dienst, Evelyn R. (197 1). Evaluating University Teaching. Berkeley, Calif.: Center for Research and Development in Higher Education. Michigan, University of, Center for Learning and Teaching (1965). "Evaluation of Instruction by Students," Memo to the Faculty. Ann Arbor, Michigan: University of Michigan. Musella, Donald and Rusch, Reuben (1968). "Student Opinion on College Teaching," Improving College and University Teaching, 16: J 37-140. Sheffield, Edward F., ed. (1974). Teaching in the Universities: No One Way. Montreal: McGill-Queen's University Press. (In press) Smithers, A. (1969). "Some Factors in Lecuring," Educational Review, 22: 141-150. Thielens, Wagner, Jr. (1971). "Teacher-Student Interaction, Higher Education: Student Viewpoint," in Deighton, Lee C., ed., Encyclopedia of Education, 9: 54-63. New York: The Macmillan Company and the Free Press. Toledo, University of, Office of Institutional Research (Richard R. Perry, Director) (1967). Criteria of Effective Teaching in an Institution of Higher Education. Toledo, Ohio: University of Toledo. Toledo, University of, Office of Institutional Research (1969). "Criteria of Effective Teaching in Four Public Universities." Toledo, Ohio: University of Toledo (unpublished report). STO A Appendices 23 STO A Appendix A Letter Sent to Graduates, Inviting their Assistance in the Project UNIVERSITY OF TORONTO. ONTARIO. TORONTO CANADA Innis College, May 1969. Dear University Graduate: This request for assistance in a research project is being addressed to a representative sample of men and women who graduated from Canadian universities in 1958, 1963 and 1968. As the first step in an investigation of the characteristics of effective teaching in university, I am asking recent graduates to help identify professors (of whatever rank, including lecturers and instructors) who, in their view, are excellent teachers. Of the teachers so identified, a score or more will be invited to write essays on the teaching of undergraduate students, reflecting their own beliefs and practices. These essays and a digest of the views of the graduates who have responded to this request will be published. It would be greatly appreciated if you would take the 10 or 15 minutes required to complete the attached simple questionnaire, and then send it to me in the enclosed envelope. If you care to add your name and address, I shall be glad to keep you informed of the results. Yours sincerely, Edward F. Sheffield, Professor of Higher Education. STO A 25 List of Universities and Faculties whose Graduates are being Invited to Assist in the Study University Faculty or School University of Alberta University of British Columbia Carleton University Dalhousie University Université Laval Education Home Economics; Law Science Medicine Sciences de l'administration (Commerce); Sciences de l'éducation (Pédagogie, Orientation) Agriculture Arts Physical Education Science Chirurgie dentaire; Droit Science Engineering Arts (programmes destinés aux francophones) Engineering Arts Pharmacy Commerce Library science; Medicine Arts (honours only); Nursing University of Manitoba McGill University McMaster University Memorial University of Newfoundland Université de Montréal University of New Brunswick Nova Scotia Technical College Université d'Ottawa Queen's University at Kingston St. Dunstan's University University of Saskatchewan Sir George Williams University University of Toronto University of Western Ontario 26 STO A Questionnaire on the Characteristics of Effective University Teaching Please complete and return to: Edward F. Sheffield, Professor of Higher Education, University of Toronto, Toronto 181, Ontario. 1 Your university (i.e. the one from which you took a bachelor's or first professional degree in 1958, 1963 or 1968): 2 Degree earned (e.g. B.A., M.D.) 3 Please name one or more university teachers from whom you took courses leading to the degree named in item 2 above whose teaching you considered to be especially effective. In each case, give his/her name and indicate the course(s) you took from him/her. e.g. Name R.D. Lawrence Field History 3 Course (approximation of title) Greek and Roman Year (level) III Year: 19 Approximate class size 20 (1) (2) (3) 5 In 200 words, more or less, describe the characteristics, qualities, methods, procedures, etc. which, in your view, identify each of these as an excellent teacher. Please describe each separately, numbering your descriptions to match the numbers in item 4 above. (Use the back of this sheet.) 6 Your name and address (optional): STO A 27 Appendix B Two Notes on the Representativeness of Samples (1) The Responding Graduates When response in a survey is less than 100% one must ask whether those who responded are representative of those who did not. In this case the response rate varied from 9.3% of the Laval graduates in administrative science to 29.2% of the honours arts graduates of Western Ontario, and was 14.6% overall. There are four kinds of evidence which may be relevant. The first is that although the response rate for the class o f ' 5 8 was 11.4% as compared with 15.9% for the class of '68, the patterns of comments by these two groups are similar in most respects. To sharpen this kind of comparison, the patterns of the five faculties with the highest response rates (arts at Western Ontario, 29.2%; education at Laval, 24.7%; agriculture at Manitoba, 21.9%;science at Memorial, 21.1%; and nursing at Western Ontario, 20.9%) (N = 1,815) were compared with those of the five faculties with the lowest response rates (arts at St. Dunstan's, 9.8%; arts at Ottawa, 9.8%; engineering at Nova Scotia Tech, 9.8%; commerce at Sir George Williams, 9.7%; and administrative science at Laval, 9.3%) (N = 1,170). Examining only comments which accounted for at least three percent of the total number there are two characteristics mentioned more than twice as often, proportionately, by one group as the other, and an additional four mentioned at least fifty per cent more often by one group than the other: High response 16 29 1.2% 3.8% (3.2 to 1) 2.0% 4.1% (2.0 to 1) 2.5% 4.4% (1.8 to 1) 22 6 36 17 rate Group Low response Sub-category rate Ratio Experienced in his field Students expected to do much on their own Concerned for students' progress, etc. Had a sense of humour, amusing Course organized, systematic 2.3% 4.2% (1.8 to 1) 3.2% 1.8% (1.8 to 1) Enthusiastic about his subject 4.5% 1.5% (1.5 to 1) With respect to these items the respondents may not have been representative of the total population. The second kind of evidence results from the fact that there were so few responses from Memorial science and Saskatchewan pharmacy graduates that they were sent a second letter urging them to participate. In both cases the second request elicited new responses numbering twice as many as the first. The numbers were still too small for statistical treatment, but careful reading of the comments made by the two groups revealed no noticeable differences in the patterns. Why, then, did the second group not respond to 28 STO A the first request? They did not explain. Lack of interest, maybe? Oversight? Procrastination? For such mundane reasons, it would seem. As the project was being planned, the questionnaire was tried out in a class of 139 third-year students in arts and science at the University of Toronto. Their response rate was 100% because the exercise was undertaken wholly within a class period. This pilot project is the third source of evidence. The pattern of their responses was compared with that of the graduates in arts and science whose data were used in this study. As has been pointed out, there are differences between these two groups but one cannot tell to what extent they are related to rate of response. Other explanations seem more satisfactory. Besides, the items on which there are observed differences are not the same as those noted in the comparison of faculties with higher and lower response rates, above. Even when comparisons on this basis are made down to comments constituting no more than one per cent of the total (excluding the "other comments" sub-categories) the items identified in the two series are all different. If the differences in patterns of comments were attributable to the influence of the response rate one would expect sameness here rather than difference. The final type of evidence offered is much less direct. It is, as pointed out above, that other approaches to the identification of the characteristics of effective teaching (e.g. the SUNY Albany and the University of Toledo studies) have led to much the same list, although the manners of arriving at it have differed significantly. Short of surveying the non-respondents themselves, it is unlikely that there can be complete assurance that the responses received are representative. What evidence there is, though, seems to point in that direction. A related question which might well be asked is whether, in the evaluation of teaching, there is a difference between the patterns of responses of students who get high grades in their courses and those who get low grades or fail. The data gathered for this study do not help to answer that question, but it should be noted that all of the comments received were from students who had graduated—whose grades, therefore, had been at least satisfactory. Other studies have found no significant difference between good and poor students on this point. In a Memo to the Faculty from the Center for Research on Learning and Teaching, University of Michigan (1965), the evidence is summarized in one sentence: "Rather surprisingly, neither the difficulty of the course nor the grade received is related to the pattern o f student evaluation." This is reiterated by Kenneth E. Eble (1970): "The effect of grades upon responses to a course or teacher evaluation has been studied as much as any single aspect of the subject. Little evidence emerges to support a belief that grades determine response." The question was explored also with CEGEP* students in Quebec by Gagné and Chabot (1970). Their finding was the same: Il s'avère que la satisfaction exprimée par un étudiant à l'égard de son professeur n'a aucun lien avec son succès dans la matière enseignée par ce professeur, non plus qu'avec sa réussite moyenne dans l'ensemble des cours. * CEGEP: Collège d'enseignement général et professionnel (offering the 12th and 13th, and for some the 14th, years of schooling). STO A 29 (2) The Chosen Professors It is interesting to note, too, how typical are the comments which were made about the 24 chosen professors. Comparison of these (N = 1,062) with the basic profile of all comments about all professors mentioned in all 24 faculties and schools (N = 9,143) reveals no noteworthy differences between the two. This encourages the conclusion that, different as the chosen professors are from each other, as a group they are representative of the excellent teachers identified in the study.