The Canadian Journal of Higher Education La revue canadienne d'enseignement supérieur Volume XXVII, Nos. 2,3,1997 pages 157-156 Good Teaching and Satisfied University Graduates HARVEY KRAHN & JEFFREY W. BOWLBY University of Alberta Abstract This paper examines the extent to which students' evaluations of university teaching and classroom dynamics contribute to overall satisfaction with their university experience. Data were collected from 1453 graduates of the University of Alberta who completed questionnaires following the 1993 spring C o n v o c a t i o n . A multi-item index m e a s u r i n g students' evaluations of university teaching and classroom experiences was employed as the central predictor in a multiple regression analysis of overall satisfaction with the university experience. Positive perceptions of teaching had a strong impact on satisfaction, controlling on gender, age, faculty of enrollment, GPA, prior postsecondary experience, assessments of skill development, satisfaction with university learning-related resources, and several other control variables. The findings highlight the continued importance of efforts to encourage good teaching in universities. An earlier version of this paper was presented at the joint C S A A - C S S E sessions on the Sociology of Education at the 1995 Learned Associations Conference, Montreal, June 4, 1995. T h e u s e f u l s u g g e s t i o n s f r o m two a n o n y m o u s r e v i e w e r s for this j o u r n a l w e r e appreciated and incorporated into a revised version of this paper. 158 H. Krahn & J. Bowlby Résumé Cette étude examine dans quelle mesure les évaluations faites par les étudiants de l'enseignement universitaire et la dynamique de la salle de c l a s s e c o n t r i b u e n t à la s a t i s f a c t i o n g é n é r a l e de leur e x p é r i e n c e universitaire. On a recueilli des données de 1453 étudiants diplômés de l ' u n i v e r s i t é de 1'Alberta ayant rempli des questionnaires après la Convocation du printemps de 1993. Une grille cretérielle mesurant les évaluations par les étudiants de l'enseignement et de leurs expériences à l'université fut utilisée comme indicateur prévisionnel, sur la base d'une analyse de régression multiple de la satisfaction globale de l'expérience universitaire. Les perceptions positives de l'enseignement ont un grand effet sur la satisfaction. Cela est le cas en tenant compte du sexe, de l ' â g e , de l ' i n s c r i p t i o n en f a c u l t é , de la m o y e n n e des n o t e s , des e x p é r i e n c e s p r é c é d a n t le p o s t - s e c o n d a i r e , des a p p r é c i a t i o n s de d é v e l o p p e m e n t d ' h a b i l e t é , de la s a t i s f a c t i o n a v e c r e s s o u r c e s universitaires liées à l'apprentissage, et de plusieurs autres variables de contrôle. Les résultats mettent en lumière l'importance continue des efforts pour encourager un bon enseignement dans les universités. Introduction Universities have a dual mission of teaching and conducting research but from the perspective of students, their primary clients, teaching is the primary goal. No doubt universities have always tried to emphasize the importance of good teaching but, in today's rapidly-changing political and economic climate, this goal has become increasingly important. As Common (1987) observed a decade ago, "it is in the best interest of the university to have quality teaching as a goal, as central to its mission, and as an accomplishment" (p. 59). Today, we might wish to underline "best interest." Publicly funded universities in Canada have been experiencing budget cuts while the pressures of accountability have been increasing. Students, the government, the public, and the media (e.g., the Maclean's annual ranking of Canadian universities) have been asking more critical questions about the educational "product" being provided (Emberley, 1996). In particular, criticisms of the quality of teaching within universities, and of the perceived imbalance between teaching and research, are frequently heard. The 1991 Commission on Canadian University Education made the point The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3,1997 Good Teaching and Satisfied University Graduates 159 forcefully when it asked: "how can we develop measures to evaluate the quality of learning that will encourage universities to improve their educational programs and motivate professors to improve their teaching?" (Smith Commission, 1991, p. 129). Five years later, the same theme was echoed in a meeting of university administrators (organized by the Association of Universities and Colleges of Canada) who were told that: . . . universities need to explore such reforms as higher rewards for good teaching, changes in the relationship between professors and students, improvements in content and delivery of curriculum, and an increased commitment to find out students' needs on campus and whether they are satisfied after graduation. (NBEC News & Views, 1996, p. 2) Graduates' satisfaction with their overall university experience may be a function of many factors, including their initial expectations on entering university, their assessments of their own performance and of the skills they acquired, the quality of their extracurricular experiences, and their evaluations of the university's facilities and services. But since students' main reason for attending university is to learn, their evaluations of university teaching would be expected to be an important determinant of their overall satisfaction with their university experience (Neumann & Neumann, 1981). In turn, graduates' satisfaction with their university experience is of critical importance to university administrators seeking to convince the public and the government to continue support for the institution. In this paper we focus directly on this hypothesized link between graduates' assessments of teaching and overall satisfaction with the university. We define the former broadly to include graduates' evaluations of instructors' attitudes toward students, the extent to which two-way communication within the classroom was encouraged, the accessibility of instructors, and the apparent pride they took in their teaching. Thus, this is a study of the impact of graduates' assessment of the university teaching environment on their satisfaction with their overall university experience. Our basic analysis strategy is to statistically control on as many other sources of satisfaction and dissatisfaction as possible, allowing us to determine the net impact of the teaching environment on satisfaction. 'Ihe Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3,1997 160 H. Krahn & J. Bowlby Previous Research A growing body of research has addressed various aspects of teaching in postsecondary institutions. In our review of this literature, we have identified at least three general strands of research, the first focusing on pedagogical practices to improve postsecondary teaching, the second on classroom-based teaching evaluations and what they measure, and the third on good teaching as one of the determinants of general satisfaction with the overall university experience. Since we demonstrate that good teaching is associated with more satisfied graduates, our study clearly fits within the last type. However, we briefly review studies within the other two categories since they offer some useful insights for our analysis. Over the past several decades a number of innovative pedagogical practices intended to improve the quality of university teaching and enhance the learning of students have been promoted. In both Canada and the United States, various universities have developed teaching centres, promoted teaching seminars and workshops, established teaching awards, and encouraged improved teaching practices in a variety of other ways (Elrick, 1990; Shore, 1974). However, criticisms of university teaching practices, some justified, some perhaps not, continue to be heard (Watkins, 1989). For example, in his 1991 report on university education in Canada, Stuart Smith commented on the perceived undervaluing of teaching in universities and recommended, among other things, the need to recognize "technological or other innovations in university pedagogy," to train Ph.D. candidates in "modern teaching methods," and to redefine scholarship to "include much more than the publication of research articles" (Smith Commission, 1991, pp. 134-135). Donald (1986) has suggested that, despite the availability of various teaching resources such as instructional development centres, greater understanding of the learning tasks for students in individual programs is still required in order to promote more effective teaching. Research on teaching evaluations has typically examined students' assessments of specific courses and instructors (Broder & Dorfman, 1994; Cohen, 1981; Feldman, 1993; Marsh, 1987; Neumann & Neumann, 1981, 1983; Shapiro, 1990; Stumpf, 1979). Some of these studies have questioned whether students' satisfaction with specific courses and instructors is an indicator that real learning has occurred (Brown, 1976; Howard & M a x w e l l , 1980, 1982; K e n n e d y , 1975; N i m m e r & S t o n e , 1991; The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3,1997 Good Teaching and Satisfied University Graduates 161 Palmer, Carliner, & Romer, 1978), or whether other factors are the main correlates of student satisfaction. Neumann and Neumann (1981), for example, identified four primary determinants of students' satisfaction with a course: satisfaction with presentations and lectures, with tests and assignments, with the human relations skills of professors, and with teaching techniques. Broder and Dorfman (1994) report that enjoyment of the learning experience translates into higher teacher ratings. Shapiro (1990) found that higher grades, classes taught over a shorter time-period, and smaller class size were associated with more positive course evaluations. However, in a recent literature review, Gilbert (1995) concludes that "[w]hat matters is not the size of the class but what goes on in the class." In short, while there has been useful research in this area, at the level of the individual course and instructor the satisfaction-equals-learning relationship has not been conclusively established. Teaching quality and effectiveness may, however, be associated with student retention (Beal & Noel, 1980; Hossler, Bean, & Associates, 1990; Louis, Colten, & Demeke, 1984; Saluri, 1985). While many other factors including poor study skills, a lack of integration into campus life, limited resources, ill health, and family responsibilities may lead to attrition from university programs (Johnson, 1994), dissatisfaction with instructors and/or course content may also have an impact. A third line of research employing data from surveys of continuing students or from "exit surveys" of graduating students has focused on satisfaction with specific programs, university services, or the overall university experience. While such "satisfaction" studies may be seen by some as " s o f t " approaches to program evaluation, measures of student/graduate satisfaction can, along with other carefully chosen and monitored "performance indicators," be very useful for identifying program strengths and weaknesses, and for encouraging institutions to change (AUCC, 1995). As Astin (1977) maintains: Given the considerable investment that most students make in attending college, the student's perception of value should be given substantial weight. Indeed, it is difficult to argue that student satisfaction can be legitimately subordinated to any other educational outcome (p. 164). Over the past few years, as more postsecondary institutions have chosen (or been pushed) to develop systematic sets of performance indicators The Canadian Journal of Higher Education Volume XXVII, Nos. 2,3,1997 162 H. Krahn & J. Bowlby (Evers & O'Hara, 1996; Gilbert, 1994), satisfaction surveys have become more common. However, many of these institutional studies have been basically descriptive, that is, reporting little more than levels of satisfaction by program. A few studies have been more analytic, discussing the range of determinants of overall student satisfaction, including program and college/university characteristics, student characteristics and experiences, student/faculty ratios, faculty workloads, and grades (Bare, 1980; Gruber, 1980; Knox, Lindsay, & Kolb, 1992; Pike, 1991, 1993). The specific association between students' evaluation of teaching and overall satisfaction with the university experience has received only limited research attention (Morstain, 1977; Winteler, 1981), perhaps because it has been taken for granted. Hearn (1985) found that stimulating course work and good teaching were more important for student satisfaction than were opportunities for faculty-student interaction or perceptions of instructors' knowledge of their subject matter. But the f e w studies that have examined the relationship between students' assessments of teaching and their overall satisfaction with the university experience have generally not controlled on the wide range of other possible determinants of student satisfaction. Consequently, in this paper, we introduce such statistical controls to highlight more clearly the strong and important relationship between graduates' assessments of university teaching and their satisfaction with their university experience. Sampling and Data Collection T h e data f o r this study were collected in the 1993 University of Alberta Graduand Survey, the second in an annual series of "exit surv e y s " d e s i g n e d to assist the University in its on-going e f f o r t s to improve the quality of education received by students. These surveys of graduating students were begun to develop an over-time database that will allow a systematic monitoring of the University's efforts to achieve its teaching goals. 1 The questionnaires ask graduates about their: a) socio-demographic characteristics and educational history; b) motives for attending university and the University of Alberta in particular; c) evaluations of their educational experiences at the University; and d) assessments of the skills they have acquired while attending university. Thus, each year the survey provides a current evaluation of the University, from the perspective of its most recent graduates, that is of considerable value to faculties, The Canadian Journal ofHigher Education Voiume XXVII, Nos. 2,3,1997 Good Teaching and Satisfied University Graduates 163 departments, and administrative units attempting to improve the educational experience for students.2 The 1993 survey was conducted in June in conjunction with spring Convocation. All students graduating from undergraduate or professional programs were surveyed. Students who attended Convocation (about 60 percent of those receiving a degree) were given a package containing the questionnaire, a cover letter from the Vice-President (Academic), and a stamped, return-addressed envelope, and were asked to return the completed questionnaire to one of several designated locations on campus or by mail. Those who did not attend were sent the same package along with their diploma and asked to return the questionnaire by mail. The questionnaires were not pre-coded in any way, and graduates were instructed not to sign their name, in order to maintain anonymity. A total of 4100 undergraduate degrees were awarded at the June Convocation, and 1453 graduates returned a completed questionnaire.3 This translates into a response rate of 35 percent. Respondents appeared to be typical of the population of graduates in terms of age and faculty of enrollment, although female graduates were somewhat more likely to respond to the survey.4 Survey respondents were typically young, single, and from Edmonton or elsewhere in Alberta. However, the slowly changing nature of the University's undergraduate population is mirrored in the 14 percent who identified themselves as belonging to a designated visible minority group, and the 15 percent who were 30 years of age and older at the time of graduation. Measurement and Analysis The dependent variable in this study, an index of overall student satisfaction, was constructed from four separate questions about general satisfaction with the University (See Table 1). For each of these four indicators, roughly three-quarters of the sample expressed satisfaction with the University. As response categories varied for each question, scores were standardized prior to constructing the overall satisfaction index, which exhibited a high degree of inter-item reliability (Alpha = 0.83). In an attempt to identify the various specific dimensions on which graduates evaluate their university experience and that might be predictors of overall satisfaction, an exploratory factor analysis was conducted using 33 questionnaire items addressing teaching, learning, and selfreported skill development. A total of nine factors were identified in this The Canadian Journal ofHigher Education Volume XXVII,Nos.2,3,1997 164 H. Krahn & J. Bowlby Table 1 Components of Indexed Student Satisfaction Variable and Distribution of Responses (%) Question Percent If you could choose again, would you still choose the University of Alberta? Yes Maybe No 70 24 6 If a close friend was interested in attending university, would you recommend the University of Alberta? Yes Maybe No 71 24 5 The University of Alberta is a university of which its graduates can be proud. * Agree Neutral Disagree 78 18 4 Overall, how satisfied are you with the education you received at the University of Alberta? ** Satisfied 75 Neutral 20 Dissatisfied 5 N = 1453 *** * Respondents answered on a scale of "strongly disagree" (1) to "strongly agree" (5); scores of 1 and 2 are combined into "disagree," while scores of 4 and 5 form the "agree" category. ** Respondents answered on a scale of "very dissatisfied" (1) to "very satisfied" (5); scores of 1 and 2 are combined into "dissatisfied," while scores of 4 and 5 form the "satisfied" category. *** Sub-sample sizes are slightly smaller for each question, because of small amounts of non-response. preliminary analysis. These included assessments of the teaching environment and evaluations of the learning experience, as well as seven skill-development factors that we have labeled social, citizenship, independent thinking, career-related, communication, mathematics-science, and arts-appreciation skills. The specific variables that make up the nine factors, and their factor loadings, are reported in Appendix 1. The teaching environment factor that became our critical independent variable (Alpha = 0.92) is composed of nine questions asking about The Canadian Journal ofHigher Education VoiumeXXVII,Nos. 2,3,1997 Good Teaching and Satisfied University Graduates 165 instructors' teaching approach, accessibility, and attitude towards students. As shown in Table 2, more students provided positive than negative responses for all nine items. The most favourable ratings were in response to statements that "instructors were reasonably accessible outside of class" (67 percent agreeing), " . . . displayed a positive attitude towards students" (63 percent agreeing), and " . . . treated students with respect" (62 percent agreeing). However, it is also apparent from Table 2 that a substantial minority of graduates provided negative responses to some of these questions. Specifically, about one in four survey respondents disagreed with statements that instructors provided useful feedback, encouraged class participation, and tried to ensure that learning was taking place. Our basic analysis strategy was to determine, with multiple regression techniques, the net effect of graduates' assessments of the teaching environment on overall satisfaction with the University, taking into account the possible effects of the many control variables that might be correlated with both the dependent variable (satisfaction) and the key independent variable (assessment of the teaching environment).5 In other words, our goal was to isolate the net effects of assessments of the teaching environment on overall satisfaction with the university, to remove from the causal argument as many alternative explanations of the "good teaching — satisfaction" relationship as possible. Our control variables included the skill-development factors discussed earlier, since more positive self-assessments of skills acquired while attending university might be associated with higher levels of overall satisfaction. 6 In addition, age, prior postsecondary experience, gender, academic performance, faculty of enrollment, reasons for attending university, and satisfaction with university learning-related resources were included in the multiple regression analysis as control variables. Age (measured in years) was controlled because older students, with more academic and/or labour market experience, might be more discriminating in their evaluations of the University. Whether or not the respondent had previously attended another postsecondary institution was measured as a binary variable. Again, previous experience with another institution might offer a potential comparison for assessments of the University of Alberta. Gender was controlled, not because of any prior expectation about gender differences in satisfaction, but because of the over-representation of women in the sample (see footnote 4). The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3,1997 166 H. Krahn & J. Bowlby Table 2 Components of the "Teaching Environment" Index and Distribution of Responses (%) Question * Disagree Instructors were reasonably accessible outside of class. Neutral Agree 9 24 67 Instructors treated students with respect. My instructors took pride in their teaching. 8 11 12 29 27 32 63 62 56 Class participation was actively encouraged. 16 28 My instructors took an active interest in my learning. 16 30 56 54 Instructors made an effort to verify that students were learning the material taught. 22 30 48 Instructors provided helpful feedback throughout courses. 19 37 44 Instructors encouraged feedback from the class regarding their teaching. 23 33 44 Instructors displayed a positive attitude toward students. N= 1453 ** * Respondents were asked to answer on a scale of "strongly disagree" (1) to "strongly agree" (5); scores of 1 and 2 are combined into "disagree," while scores of 4 and 5 form the "agree" category. ** Sub-sample sizes are slightly smaller for each question, because of small amounts of non-response. Graduates with a higher grade-point average (self-reported for their final year) might be more positive in their assessments of the University as a whole, due to their relatively greater academic success. Faculty of enrollment (14 separate binary variables) was controlled to reflect the fact that graduates of different faculties differed in their level of overall satisfaction with the University. The faculty of Law was omitted as the reference category because its graduates appeared to be least satisfied. Two additional control variables addressed reasons for attending university (for personal development/fulfillment reasons or because of The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3, 1997 Good Teaching and Satisfied University Graduates 167 the influence of parents/friends).7 These variables were included as controls because it is possible that initial reasons for attending university might influence overall satisfaction with the institution. Finally, two variables reflecting satisfaction with learning-related academic resources (classroom facilities and library resources/study space)8 were included as controls since they might also affect overall assessments of satisfaction. Results Results from the two-step multiple regression analysis are presented in Table 3. The first column contains the standardized regression coefficients (betas) for each independent variable in an equation from which the teaching environment index was omitted. Column two displays standardized coefficients for a second equation including the teaching environment index. The third column presents zero-order correlations between each of the independent variables and the overall satisfaction index (Table 4 contains the zero-order correlations among all the variables in our analysis).9 Thus, comparisons across columns allow us to see how zero-order effects of individual variables change when the full range of predictors are controlled. Column 3 in Table 3 clearly reveals that the teaching environment index has the strongest zero-order impact (r = .54) on overall satisfaction. When the full range of control variables is taken into account, this effect is reduced (beta = .37; column 2). Thus, some of the bivariate relationship between graduates' assessments of the teaching environment and their reports of overall satisfaction is due to variance shared between the teaching environment index and other independent variables (see Table 4). Nevertheless, the effect of the teaching environment index is still much stronger than that of any of the control variables included in the analysis (column 2). By itself, the large set of control variables accounts for one-third of the variation in the dependent variable (column 1). When the teaching environment index is added to the equation, R2 increases significantly to 42 percent (column 2). In addition to the strong effect of the teaching environment index, eight of the control variables had weak statistically significant net impacts on the dependent variable. Graduates who reported a greater degree of improvement in their career-related skills were more likely to report satisfaction with their overall university experience (beta = .13), controlling on the other independent variables. In addition, self-reported The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3, 1997 168 H. Krahn & J. Bowlby Table 3 Regression Analysis Results: Effects on Overall Student Satisfaction a n d Zero-Order Correlations with Overall Satisfaction Independent Variable Teaching Environment Index Age (years) Gender (male=l) Average GPA Prior Postsecondary Experience (yes = 1 ) Skill Development Indices Social Skills Citizenship Skills Independent Thinking Skills Career-Related Skills Communication Skills Math/Science Skills Arts Appreciation Skills Satisfaction with Resources Library/Study Space Classroom Facilities Beta: (excluding Teaching Environment Index) Beta: (including Teaching Environment Index) _ r .06* .01 .04 .05 .01 -.01 .54 .06 .03 .10 .04 .02 .00 .10** .09** [ ] ** .05 .07* io*** 13*** .04 .02 .04 .35 .24 .33 .35 .26 .15 .18 09*** 12*** .27 .29 jg*** .07* .03 .06 12*** 19*** Reasons for Attending University Influence of Parents/Friends -.02 07** Personal Develop./Fulfilment -.01 .05* -.01 .20 Faculty Agriculture/Forestry Arts Business Dentistry Education .01 -.02 -.10* .01 .00 .02 -.02 -.10 .02 .00 .06* .06 -.06 .05 .06 The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3, 1997 Good Teaching and Satisfied University Graduates 169 Table 3 (cont.) Beta: (excluding Teaching Environment Index Faculty Engineering Home Economics Medicine Nursing Pharmacy Physical Education Rehabilitation Medicine Faculté Saint-Jean Science R2: F: Beta: (including Teaching Environment Index) .04 -.05 .09** .07* .06* .05 .03 .02 .10* Independent Variable .00 _ 07** .04 .02 .01 -.02 -.03 -.03 .05 .33 26.86*** .42 19.06*** r -.01 -.06 .03 .08 .07 .02 .06 .04 .01 * p< .05 ** p< .01 *** p< .001 improvement in independent-thinking skills (beta = .10) and in citizenship skills (beta = .07) were positively associated with the dependent variable index. Weak but significant net effects were also observed for satisfaction with academic resources on campus; graduates reporting more satisfaction with library/study space (beta = .09) and with classroom facilities (beta = .12) were more likely to express satisfaction with the overall university experience. Graduates who identified "personal development/fulfillment" reasons as important in their decision to attend university were more likely to be satisfied with the university experience (beta = .05), although the relationship was very weak. Finally, graduates from the faculties of Business (beta = -.10) and Home Economics (beta = -.07) were somewhat less positive (in comparison to Law graduates who constituted The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3,1997 170 H. Krahn & J. Bowlby O O 00 — O f CN O f OS o o < N OO OO —| CN < N < N t in so so o O H co rt rH F H H O rS OS in to o o (N OS O CN in co co CO CN in co so o o CO O CN '—' CN O CN O SO — R < No R o < N —1 SO O R CO ' CO so CN CN O Tf in CO CN O in o M C N oo O co co — O co CN O OS O in — i co O — CN O —' o f f co in co in co O O l' f rf rt co oo O O O O R SO — co in VI O O Oif Tt CO O " TJCN ON 8 oo co —I os C N o in o C N in — o so o CN CN —' »-< < N o r^O0s0000Ti-s0-5t O - h O O — -HOO CN O I I I I CO in o >n in co o — < -H o o CO oo n N in CN co O rs| — i o o CO < NO O r- rt r» o i o —' o — so < N O O CN co CO so CN — in CO O OS O R o R (N o O OS r r i> >1 o o •«t in c o c »1 JS u -s •a es > es o c (Z) H W 1) O S 2 S e S >> B w "O U C H 00 O > < O ^ < co --s- in •si 8 I <o H" « Y s -S Y ï T3 O O W vi rso The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3, 1997 W ) C cfl ^ S .S J3 00 A H -o _ u C L uC _ e S •a 13 •C T3 „ S c2 c u T i U Q, U N U U es •a T3 U û >5 oo os o O O -o c u ,-i W C c U o ° a J3 I1 f i• 00 « 2 55 2 £ eu oi c e u u S 3 S s u O C L S u C L 'S O •g S Cl, O U < £ U > U es b! c/o « < 00 < Q M 00 C N co Tj" in so Good Teaching and Satisfied University Graduates 171 the reference category in this regression equation) in their overall assessments of the university experience. One might speculate about possible explanations for these eight statistically significant coefficients. However, as already noted, all were weak net effects and, furthermore, our interest really lies in assessments of the teaching environment.10 Taking into account the effects of all the other possibly relevant factors that we could measure, the impact of assessments of the teaching environment (beta = .37) was almost three times as large as the next largest effect, that of assessments of improvement in career-relevant skills (beta = .13). Thus, the relationship between graduates' assessments of university teaching and their overall satisfaction with the university experience is not a function of other factors such as age, gender, GPA, skill development, faculty of enrollment, satisfaction with university resources, prior postsecondary experience, and reasons for attending university. Repeating our main finding, controlling on all possible predictors that might influence overall satisfaction with the university experience, we find a strong, positive effect for the teaching environment index. Graduates who positively evaluated the teaching environment they had experienced while attending university were more likely to express satisfaction with their overall university experience. Discussion As noted in our review of related research, student satisfaction has typically been one of the measures included in exit surveys that many postsecondary institutions have begun to conduct. Most often, these institutional studies have simply reported descriptive results (e.g., "75 percent of graduates were satisfied"), although a few more analytic studies (e.g., Hearn, 1985) have gone on to examine some of the sources of graduate satisfaction. However, our study takes into account many more aspects of the broader university experience and controls on a wider range of possibly relevant variables. Its explicit goal is to eliminate from the causal argument as many alternative explanations of the "good teaching - satisfaction" relationship as possible. Thus, our study demonstrates much more conclusively that the experience of good teaching translates into greater satisfaction with the overall university experience. The analytic approach taken in multiple regression studies such as these emphasizes, first, the relative explanatory strength of theoretically-important The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3, 1997 172 H. Krahn & J. Bowlby predictor variables, and, second, the total amount of variance one can explain with a full set of predictor variables. It is quite clear from the results of our analysis that, compared to the wide range of other predictor variables included in the equation, the teaching environment index has a much more powerful net impact on overall satisfaction. As for the combined effect of all the independent variables, the full equation accounted for 42 percent of the variation in the dependent variable. Compared to many other studies of the predictors of attitudes, beliefs, and other subjective states, this is a respectable finding but, even so, one might argue that 58 percent of the variation in graduate satisfaction remains unexplained. There are, no doubt, important factors missing from our study which, if present, might increase the amount of explained variance. As Feldman and Newcomb (1994) .convincingly argue, students' backgrounds and their own personalities condition the impacts of college life on students, in a variety of complex ways. In addition, every student's university life consists of countless individualized experiences (e.g., a chance encounter with an influential person, a powerful book, a stubborn room-mate, a helpful coach, bad meals, an unpleasant relationship, a super apartment), any of which might come to influence their own overall feeling of satisfaction with the university experience. Thus, recognizing the incredible diversity of potential influences, and the important conditional effects of background and personality, we are satisfied with our ability to account for 42 percent of the variation in the dependent variable with a limited number of predictor variables. Having addressed the substantive significance of our findings, we should also comment on their generalizeability. One possible concern might be the relatively low response rate (35 percent). Perhaps the minority of graduates who responded to the 1993 exit survey over-represented the very satisfied or, alternatively, the least satisfied. However, our interest is not in the overall level of satisfaction, an estimate that might be biased upward or downward by a somewhat skewed sample. Instead, we are interested in the relationship between assessments of the teaching environment and overall feelings of satisfaction after graduation. One should still be able to observe such a relationship, if it exists in the population, in a sample that over-represents satisfied or dissatisfied graduates. Further analysis using data collected in a different exit survey conducted at the University of Alberta in 1996 (see footnotes 1 and 2) which had a response rate of 49 percent (since it was conducted by telephone) support our contention. Although we were unable to completely replicate The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3,1997 Good Teaching and Satisfied University Graduates 173 our analysis, given the absence of some of the variables available in the 1993 data, we nevertheless found a strong positive relationship (r = .381) between an index of overall satisfaction with the university experience and a teaching environment index. After controlling for as many variables as possible, the net effect of the teaching environment on satisfaction remained strong (beta = .282) and over twice as large as the next strongest predictor." Thus, using a separate data set with a higher response rate and, presumably, a more representative sample, we were able to replicate our core finding showing that the university teaching environment is an extremely important predictor of overall satisfaction with the university experience among recent graduates. 12 This finding, along with previous research showing that good teaching is associated with higher levels of student retention, reinforces calls for efforts to improve university teaching (e.g., NBEC News & Views, 1996; Smith Commission, 1991). And in today's political and economic climate where universities are increasingly having to justify their programs to the public, the media, and politicians (Emberley, 1996), good teaching is critically important. Collectively, students represent the most important resource universities have for validating and promoting their educational role and academic programs. Hence, as Astin (1977) maintains, students' evaluations of their university experience should not be ignored or underestimated. More satisfied graduates allow universities to better justify their existence (and the continued expenditure of tax-dollars on postsecondary education) and, as this study shows, graduates who evaluate the university teaching environment more positively are, ultimately, more satisfied with their university experience.^ References Association of Universities and Colleges of Canada (AUCC). (1995). A primer on performance indicators. Ottawa, ON: AUCC, Research File, June. Astin, A. (1977). Four critical years. San Francisco, CA: Jossey-Bass. Bare, A.C. (1980). The study of academic department performance. Research in Higher Education, 72(1), 3-22. Beal, P., & Noel, L. (1980). What works in student retention. Iowa City, IA, and Boulder, CO: American College Testing Program and the National Center for Higher Education Management Systems. The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3, 1997 174 H. Krahn & J. Bowlby Broder, J.M., & Dorfman, J.H. (1994). Determinants of teaching quality: What's important to students? Research in Higher Education, 55(2), 235-249. Brown, D.L. (1976). Faculty ratings and student grades: A university-wide multiple regression analysis. Journal of Educational Psychology, 68, 573-578. Cohen, P.A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies. Review of Educational Research, 51, 281-309. Common, D.L. (1987). Teaching in universities: The power to touch. The Canadian Journal of Higher Education, 17(3), 59-66. Donald, J.G. (1986). Teaching and learning in higher education in Canada: Changes over the last decade. The Canadian Journal of Higher Education, 16(3), 77-84. Elrick, M. (1990). Improving instruction in universities: A case study of the Ontario Universities Program for Instructional Development (OUPID). The Canadian Journal of Higher Education, 20(2), 61-79. Emberley, P.C. (1996). Zero tolerance: Hot button politics in Canadian universities. Toronto, ON: Penguin. Evers, F.T., & O ' H a r a , S.A. (1996). Educational outcome measures of knowledge, skills and values. Educational Quarterly Review, 5(1), 43-56. Feldman, K.A. (1993). College students' view of male and female college teachers: Part II — Evidence from students' evaluations of their classroom teachers. Research in Higher Education, 34(2), 151-191. Feldman, K.A., & Newcomb, T.M. (1994). The impact of college on students. New Brunswick: Transaction Publishing, [originally published by JosseyBass, 1969], Gilbert, S. (1994). The search for educational indicators. Education Quarterly Review, 7(4), 44-53. Gilbert, S. (1995). Quality education: Does class size matter? Canadian Society for the Study of Higher Education, Professional File, 14, 1-6. Gruber, J.E. (1980). Sources of satisfaction among students in postsecondary education. American Journal of Education, 88(3), 320-344. Hearn, J.C. (1985). Determinants of college students' overall evaluations of their academic programs. Research in Higher Education, 25(4), 413—437. Hossler, D., Bean, J.P., & Associates. (1990). The strategic management of college enrollments. San Francisco, CA: Jossey-Bass. Howard, G.S., & Maxwell, S.E. (1980). Correlation between student satisfaction and grades: A case of mistaken causation? Journal of Educational Psychology, 72, 810-820. Howard, G.S., & Maxwell, S.E. (1982). Do grades contaminate student evaluations of instruction? Research in Higher Education, 16, 175-188. The Canadian fournal ofHigher Education Volume XXVII, Nos. 2,3, 1997 Good Teaching and Satisfied University Graduates 175 Johnson, G.M. (1994). Undergraduate student attrition: A comparison of the characteristics of students who withdraw and students who persist. The Alberta Journal of Educational Research, 40(3), 337-353. K e n n e d y , W . R . (1975). Grades expected and grades received - their relationship to students' evaluations of faculty performance. Journal of Educational Psychology, 67, 109-115. Knox, W.E., Lindsay, P., & Kolb, M.N. (1992). Higher education, college characteristics, and student experiences. Journal of Higher Education, 63(3), 303-328. Krahn, H., & Silzer, B. (1995). The University of Alberta graduand survey. College and University, 77(1), 12-23. Louis, K.S., Colten, M.E., & Demeke, G. (1984). Freshmen experiences of the University of Massachusetts at Boston. Boston, MA: University of Massachusetts Press. Lowe, G.S., Krahn, H., & Bowlby, J. (1996). Undergraduate experiences at the University of Alberta: Results from the 1996 University of Alberta student satisfaction survey. Edmonton, AB: Population Research Laboratory, University of Alberta. Marsh, H.W. (1987). Students' evaluations of university teaching: Research findings, Methodological Issues, and Directions for Future Research. International Journal of Educational Research, 11, 253-388. Morstain, B.R. (1977). An analysis of students' satisfaction with their academic program. Journal of Higher Education, 48( 1), 1-16. NBEC News & Views. (1996). Universities need to explore a number of reform m e a s u r e s . The National Business and Education Centre (NBEC) Newsletter, November. Ottawa, ON: Conference Board of Canada. Neumann, Y., & Neumann, L. (1981). Determinants of students' satisfaction with course work: An international comparison between two universities. Research in Higher Education, 14(A), 321-333. Neumann, Y., & Neumann, L. (1983). Characteristics of academic areas and students' evaluation of instruction. Research in Higher Education, 19(3), 323-344. Nimmer, J.G., & Stone, E.F. (1991). Effects of grading practices and time of rating on student ratings of faculty performance and student learning. Research in Higher Education 32(2), 195-215. Palmer, J., Carliner, G., & Romer, T. (1978). Leniency, learning, and evaluations. Journal of Educational Psychology, 70, 855-863. Pike, G.R. (1991). The effects of background, coursework, and involvement on students' grades and satisfaction. Research in Higher Education, 32(1), 15-30. The Canadian journal of Higher Education Volume XXVU, Nos. 2,3,1997 176 H. Krahn & J. Bowlby Pike, G.R. (1993). The relationship between perceived learning and satisfaction with college: An alternative view. Research in Higher Education, 34( 1), Saluri, D. (1985). Case studies and successful programs. In L. Noel, R. Levitz, D. Saluri, & Associates (Eds.), Increasing student retention: Effective programs and practice for reducing the dropout rate (402-447). San Francisco, CA: Jossey-Bass. Shapiro, E.G. (1990). Effect of instructor and class characteristics on students' class evaluations. Research in Higher Education, 31(2), 135-148. Shore, B.M. (1974). Instructional development in Canadian higher education. The Canadian Journal of Higher Education, 4(2), 45-53. Smith Commission. (1991). Report: Commission of inquiry on Canadian university education. Association of Universities and Colleges of Canada. S t u m p f , S.A. (1979). A s s e s s i n g academic p r o g r a m and d e p a r t m e n t effectiveness using student evaluation data. Research in Higher Education, 11(A), 353-363. Watkins, B.T. (1989). Colleges urged to avow that teaching is central to their mission and to take steps to upgrade it. The Chronicle of Higher Education, (April 26), p. A13. Winteler, A. (1981). The academic department as environment for teaching and learning. Higher Education, 10, 25-35. Appendix 1 Exploratory Factor Analysis (Factor Loadings in Paranthesis) 1. 2. assessments of the " TEA CH1NG ENVIRONMENT" a) instructors made an effort to verify that students were learning the material taught b) instructors displayed a positive attitude toward students c) instructors treated students with respect d) class participation was actively encouraged e) my instructors took an active interest in my learning f) instructors provided helpful feedback throughout courses g) instructors encouraged feedback from the class regarding their teaching h) my instructors took pride in their teaching i) instructors were reasonably accessible outside of class evaluations of the "LEARNING EXPERIENCE" a) overall, my learning experiences were intellectually stimulating b) my program offered an enriching learning experience c) my learning experience has been enjoyable d) instructors stimulated me to want to learn more The Canadian Journal of Higher Education Volume XXVII, Nos. 2,3,1997 (-776) (-740) (.736) (-710) (.708) (.708) (.695) (.694) (-691) (.832) (-781) (-714) (.638) Good Teaching and Satisfied University Graduates 3. 4. 5. 6. 7. 8. 9. development of "SOCIAL SKILLS " a) improvement in social skills b) improvement in ability to work well with others c) improvement in leadership skills/abilities development of "CITIZENSHIP SKILLS " a) improvement in awareness of the rights and responsibilities of citizenship b) improvement in political awareness c) improvement in awareness of ethical issues d) improvement in appreciation of other races, cultures and religions 177 (.790) (.702) (.686) (.837) (-781) (.711) (.658) development of "INDEPENDENT THINKING SKILLS ' a) improvement in ability to learn on your own b) improvement in problem-solving skills c) improvement in ability to work independently (.759) (.606) (.602) development of "CAREER-RELATED SKILLS" a) improvement in career-related prospects b) development of skills employers want c) new ideas about career possibilities (.829) (.815) (-745) development of "COMMUNICATION a) improvement in speaking skills b) improvement in writing skills (-731) (.674) SKILLS" development of "MATH/SCIENCE SKILLS" a) improvement in mathematical skills b) improvement in computer skills c) improvement in understanding of science (.801) (.730) (.632) development of "ARTS APPRECIATION SKILLS" a) improvement in appreciation of literature b) improvement in appreciation of the fine arts (.786) (.763) Notes 1 The fourth survey was conducted in June, 1995. In 1996, a fifth exit survey with somewhat similar questions was conducted by telephone, rather than via the self-administered method used in the previous four surveys. Krahn and Silzer (1995) discuss the history, administration, and functions of these annual University of Alberta exit surveys that collect information from graduates of The Canadian foumal ofHigher Education VolumeXXVII, Nos. 2,3, 1997 178 H. Krahn & J. Bowlby undergraduate (e.g., Arts, Science, Education, Engineering) and professional (e.g., Business, Medicine, Dentistry, Law) faculties. * In 1996, the provincial government began to request student satisfaction results from exit surveys as part of its Key Performance Indicators initiative. Measures of student satisfaction form one of 18 sets of performance indicators that all universities are required to report to the provincial government on a regular (annual or biannual) basis (see Lowe, et al., 1996 for additional details). ^ Since all graduating students receive a questionnaire, this study is essentially a census, not a random sample survey. Hence, significance tests are not really appropriate. Nevertheless, we employ significance tests, not to estimate confidence intervals for survey estimates, but as a general guide for identifying relationships in the data that are strong enough to be noteworthy. ^ Since women are over-represented in the sample, we included gender as a control variable in the multivariate analysis in order to, at least, statistically account for this sampling bias. ^ The learning experience factor was excluded from the multiple regression analysis on grounds of uncertain temporality/causality. Specifically, its component measures (see Appendix 1) asking respondents to evaluate their learning experiences were difficult to distinguish conceptually from the measures of overall satisfaction (our dependent variable), unlike the teaching environment measures that clearly referenced specific aspects of the classroom experience. 6 See Evers and O'Hara (1996) for a useful overview of the types of questions about skills, knowledge, and values asked in student and graduate surveys conducted by colleges and universities in Canada. 1 A single item asked about personal development/fulfilment reasons, while responses to two items ("parents wanted me to go"; "friends were going") were averaged to create the second variable. Responses were measured on a scale ranging from "very unimportant" (1) to "very important" (5). ® Survey respondents were asked to evaluate facilities and services on campus using a five-point "poor" to "excellent" scale. A single question asked about "classroom facilities" while responses to three items (library holdings, library hours, and study space) were averaged to create the second measure. ^ Although some of the correlations among independent variables were of moderate strength, no evidence of multicolinearity problems was observed in our analysis. The fact that the positive zero-order correlation between self-reported grades (GPA) and overall satisfaction (r = .10) virtually disappears is interesting. Other studies that have found higher grades to be associated with higher The Canadian Journal of Higher Education Volume XXVII, Nos. 2,3,1997 Good Teaching and Satisfied University Graduates 179 levels of overall satisfaction may not have controlled on evaluations of the teaching environment. ' ' Results from this additional analysis are available from the authors on request. Only two of the four satisfaction items used in our 1993 dependent variable (the first and the last in Table 1) were available in the 1996 data. The 1996 teaching environment index (Alpha = .89) was constructed from 15 separate items, four almost identical in wording to equivalent 1993 items, and the rest quite similar. Along with gender and faculty, the 1996 data also provided a range of single-item measures of skill acquisition, as well as single-item measures of satisfaction with classrooms and study space, and personal development and social ("friends were attending") reasons for attending university. it is interesting to observe that the 1996 telephone survey tended to show higher levels of satisfaction and more positive evaluations of various aspects of the university than did the 1995 exit survey conducted in the same manner as the 1993 survey analyzed in this paper. Unfortunately, differences in question wording, as well as in the timing of the survey (before final exams rather than at Convocation), along with the different data collection method make it impossible to conclude that the higher levels of satisfaction are due to a higher response rate (Lowe, et al., 1996). The Canadian Journal of Higher Education Volume XXVII, Nos. 2,3,1997