Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 78 CSSHE SCÉES Canadian Journal of Higher Education Revue canadienne d’enseignement supérieur Volume 40, No. 2, 2010, pages 78-94 The Impact of Enquiry-Based Learning on Academic Performance and Student Engagement Alastair Summerlee and Jacqueline Murray University of Guelph ABSTRACT Previously, we reported qualitative findings showing that students who experienced a problem- or enquiry-based course (EBL) in a first-year seminar program had greater confidence in their academic abilities, were more engaged, and were better prepared for upper-year courses. In the current paper, we provide quantitative data to substantiate the students’ qualitative conclusions. We present results to show that these students do perform at a significantly higher level compared with members of the control group who did not experience an EBL course. Using survey data, we show that the EBL students shift the way they access information compared with peers: they preferentially use more sophisticated resources for research. At the same time, students report greater engagement in the community, and student engagement is known to contribute to increased academic performance. RÉSUMÉ Dans un article précédent, nous avons rapporté les résultats de nos analyses qualitatives, qui démontraient que les étudiants inscrits dans un programme de première année utilisant une méthode d’apprentissage par explication ou par l’action (enquiry-based learning ou EBL) avaient une plus grande confiance en leurs capacités scolaires, étaient plus engagés socialement et se sentaient mieux préparés pour les cours de niveau supérieur. Dans l’article qui suit, nous présentons des données quantitatives pour justifier les conclusions subjectives des étudiants. Nos données démontrent une performance nettement plus élevée de ces étudiants que ceux du groupe témoin, qui n’employaient pas la méthode d’apprentissage par l’action. Grâce aux données recueillies à même le sondage, nous démontrons que les étudiants du groupe dont l’apprentissage CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 79 était basé sur l’action modifient la façon dont ils acquièrent l’information, contrairement aux étudiants du groupe contrôle. En effet, les étudiants du premier groupe préfèrent recourir à des ressources plus sophistiquées pour leurs recherches. Ils rapportent aussi une plus grande participation dans la collectivité, une activité qui contribue à améliorer la performance académique. Enfin, les auteurs suggèrent que l’expression « apprentissage par l’action » est plus appropriée pour décrire cette méthode d’apprentissage . INTRODUCTION There are strong imperatives to change the undergraduate experience in universities. The advent of the Internet, which has made information ubiquitously available, ought to change our approaches to teaching and learning. There is a body of evidence that demonstrates that students’ active engagement in their postsecondary institution is one of the key drivers for academic success (Kuh, 2003). At the same time, the learning styles and attitudes of so-called millennial students are different from those of previous generations. To remain effective, universities need to accelerate their pace of evolution to accommodate to these changes. In 2007, we reported that introducing problem-based learning in a first-year seminar course at the University of Guelph significantly affected the learning behaviours of students, excited in them greater motivation to succeed, and led to enhanced reasoning and processing skills that were both transferable and persistent throughout the students’ undergraduate experience (Murray & Summerlee, 2007). The changes we documented in that paper were self-reported observations by students and based on qualitative observation. The purpose of the present paper is to provide quantitative evidence for the persistent effects of problem-based learning, when offered as an elective in the first year of university. We followed the progress of students who participated in the problem-based seminar courses through the remainder of their academic program to determine if their own beliefs about the increase in their performance could be substantiated by measurable changes in their academic performance as revealed by grades in other courses. In addition, we undertook a systematic study of the students’ approach to the learning resources that they used during their seminars to determine whether changes in their academic performance were related to, or could be attributed to, differences in their skills at scholarly research, in particular by accessing more appropriate resources. Research also points to a reciprocal relationship between engagement in the classroom and effective learning (Kuh, 2003), and between active learning and engagement in the university or wider community (Kuh, 2005a). There are also data to suggest that students in small classes show a greater level of community engagement (Ahlfeldt, Mehta, & Sellnow, 2005). Based on these findings, we also explored whether students who became engaged in learning through a problem-based course might also show a greater level of community engagement. Definition of the Problem-Based Experience We developed a suite of seminars that use a specific approach to teaching and learning known as closed-loop reiterative problem-based learning (Barrows, 1986; CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 80 Schmidt, 1983; Murray & Summerlee, 2007: see definition below). Although there has been analysis of the effectiveness of problem-based learning (Albanese, 1993; Vernon & Blake, 1993; Ahlfeldt et al., 2005), comparisons are confounded by the use of different approaches to problem-based learning. In some cases, the term is loosely applied to diverse classroom approaches where problems are used to illustrate principles or ideas: in other situations it is used to refer to situations where problems are used to stimulate learning. Despite a number of specific publications focused on this approach, there remains a remarkably low level of acceptance of the fact that closed-loop reiterative problem-based learning is quite a different pedagogy from the simple use of problems in large classes to demonstrate or reinforce the principles conveyed in a traditional lecture format. Part of the challenge may come from the confusion created by the use of the word “problem” in the phrase “problem-based learning,” because it seems to emphasize the centrality of the “problem.” It implies that there is a problem for which students must find the answer when the situation is quite the contrary. Closed-loop reiterative problem-based learning uses a “problem” or a “situation” to incite students to question context, to find information that supports understanding the principles that lie behind the problem, and to reflect upon the wider implications. The problem, better termed scenario, is used to pique students’ interest and motivate them to enquire about the underlying issues. The authors, therefore, strongly recommend that the discourse adopt the language of enquiry-based learning (EBL), a term that better reflects the emphasis of the pedagogy. This will shift the pedagogical focus to enquiry and away from problem solving. Even this change of language may not provide sufficient clarity. It is, therefore, important that we are precise about the nature of the educational experience in the research that we are reporting in this paper. In closed-loop reiterative EBL, students are presented with a scenario in their first session. They are required to identify issues, research the principles underlying the issues, and learn material in context. (For full details of the approach, see Murray & Summerlee, 2007). It is not necessary for the students to solve the problems; in fact, many of the scenarios are deliberately designed so they do not have a ready solution. The scenarios are intended to motivate the students to engage in learning and understand the issues that underlie the scenario. Students research their learning issues and return to the second class to share information and integrate new knowledge into the scenario; this process may be repeated. The reiterative nature of the process encourages students to practise effective communication and to learn how to criticize and how to behave in academic situations, in addition to learning and assimilating content. The students are encouraged to find appropriate resources that lead to understanding, and they share information and resources with each other. At the end of every class meeting, time is devoted to group processing. This is an absolutely crucial part of the learning process and is often avoided by other pedagogical approaches that have adopted the label “problem-based learning.” In group processing, each member of the group provides concise and precise feedback to every other member of the group, as well as engaging in self-assessment of their own performance. This provides the opportunity for the students and the facilitators to enhance communication, critical thinking, and research skills. One third of every class period is spent in group processing. CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 81 Study Groups We have explored the application and longer-term implications of EBL in a number of different areas over the past five years. This has involved students taking one of three different first-year seminar courses taught in an EBL mode: Sex, Gender and Sexuality; Politics, Science and the Environment; and Confronting Cultural Dilemmas. All three courses are interdisciplinary and we have completed more than ten seminar groups comprising close to 100 students in total. The students crossed the full spectrum of undergraduate programs and disciplines at the university. Upon completion of the EBL course, students continued their studies in traditional courses and programs, and the data in the present paper are based on their performance, during the remainder of their studies, compared with that of students in control groups. METHODOLOGY There were four objectives to the research: (1) to follow the academic progress of students who had completed an EBL first-year seminar (determined by analysis of their grades) compared with students in the control groups; (2) to explore whether the observed differences in grade performance could be associated with the average grades of students upon entering university; (3) to investigate the impact of the EBL seminar on the types of resources that students used in their research; and (4) to study whether or not there was a correlation between the level of volunteerism and community engagement of the EBL students. Objective 1: To Follow Academic Performance of Students Who Completed an EBL Seminar Course To follow the academic progress of students who completed an EBL seminar course on their subsequent performance in more traditional courses, the grades of a group of seventeen students who had taken the EBL course called Sex, Gender and Sexuality were tracked throughout their undergraduate program. Every semester following the EBL seminar, the grade averages of this group of students was compared with two control groups: one, a group of students who had participated in a different first-year seminar (i.e., they had elected to have a small group experience but were not taught in an EBL mode), and the second, a group of students chosen from a course delivered in traditional large-lecture format. The students in the two control groups were matched by gender and grade average upon entrance to university with students in the experimental group. Statistical differences in performance were determined using ANOVA and compared by students’ t test scores. This review of performance was repeated with a second group of students who had taken an EBL seminar called Politics, Science, and the Environment. There were nine students in this group. The goals were to determine whether an EBL course would have repeatable impact on subsequent performance with a different set of students and to determine whether the students’ performance was affected by the subject matter of the course, or, more likely, was attributable to the pedagogical approach. CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 82 Objective 2: To Determine Whether Entering Grades Were Correlated to Subsequent Performance To determine whether the entering grades of students correlated with the overall impact of the EBL course on subsequent performance, students who participated in enquiry-based learning courses were divided into four groups according to their academic record upon entrance to university (admission average): • Stratum 1 – students with entering grades of 71–75 per cent (n = 16) • Stratum 2 – students with entering grades of 76–80 per cent (n = 29) • Stratum 3 - students with entering grades of 81–85 per cent (n = 55) • Stratum 4 – students with entering grades of >85 per cent (n = 24). The performance of these four strata was compared at the end of their sixth semester of study (3 years of undergraduate courses) in two ways: (1) the average grade in all courses completed, and (2) the average grade improvement by the end of semester six, compared with the entering average. Objective 3: To Investigate Whether the EBL Course Changed the Students’ Approaches to Resources/Research Materials To study the immediate impact of the enquiry-based seminar, students (n = 18) completed a survey on their use and access to resources before and immediately after completion of the EBL course in their first year. The questions in the survey focused on three areas: (1) access to electronic resources; (2) access to scholarly journals, encyclopedias and specialized websites; and (3) access to human resources (faculty, librarians, staff, and fellow students). The questions are shown in Table 1. Students ranked the frequency with which they accessed different types of resources on a five-point scale from never to always, and the results from before and after the course were compared to highlight any shifts in patterns of behaviour. A second survey was designed to identify persistence in the use and access to resources. The survey was administered again to the same group of students (n = 18), upon completion of their second year of study (4 semesters). The instrument was also completed by a control group of students (n = 18) who had taken a non-EBL first-year seminar. The results of the surveys were collated and compared. Statistical differences in the reported use of resources were determined using ANOVA and compared by students’ t test scores. Objective 4: To Study Whether Participation in an EBL Course Influenced Engagement There is a wealth of literature on engagement (Ahlfeldt et al., 2005; Kuh, 2003; Kuh & Hayek, 2004). The measure of engagement in the learning process in many circumstances is judged by asking students how many times they participate in class discussion, work in small groups, and challenge each other’s assertions and observations. In an EBL class, students would expect to score highly on these types of surveys so we sought an additional measure of engagement. We argue that time spent volunteering and participating in an international experience during the summer could serve as a measure of student engagement. The responses of the same two groups of students (n = 18) surveyed for Objective 3 were compared using χ2 analysis. Students were asked CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 83 to complete two surveys: one at the start of semester two (before the first-year seminar) and one after completion of the second year (four semesters) of study. The survey asked students to identify how much time per week they spent volunteering inside and outside the institution, and to report if they had participated in an international experience during the summer between their first and second years of study. RESULTS Objective 1: To Determine the Impact of an EBL Course on Subsequent Performance With their permission, a cohort of 13 students was followed from semester two until graduation. Of those 13 students, two were in a science program, two in commerce, two in arts, two in applied science, two in environmental science, and three in the combined arts and science program. 100 Grades 90 80 70 60 Entry 1 2 3 4 5 6 Semester level (standard 4 year - 8 semester program) 7 8 Figure 1. Average grade in all courses completed. Notes: The impact of one first-year course using enquiry-based learning (EBL – black diamonds, n = 13) on the average grades during the remainder of the course of study compared with a control group of students in a self-selected seminar taught in a more traditional mode (Non-EBL – black squares, n = 13) and a control group of students (grey triangles – n = 13) taught in standard didactic lectures. The mean and standard error of the mean is shown for each group. There is a trend for the students in the EBL group to have higher average grades after their experience in the EBL class. The performance of the 13 students in the EBL group was compared with the two control groups (students who were matched for gender, program of study, and entering grade); the results are shown in Figure 1. It is notable that the students in both first-year Sex, Gender and Sexuality seminars performed better during their second semester, i.e., when they were working in small groups, compared with students in traditional classes, but, more significantly, the EBL group showed statistically superior performance on average in subsequent semesters compared with members of the control groups. By graduation, the difference between these groups had increased to an average of 8.9 per cent grade points (range 2.5 – 11.2 per cent). A second group of nine students, who were in the Politics, Science and Environment seminar, were followed to determine whether the subject matter of the course influenced the outcomes. Of this group, three students were in science, two in enviCJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 84 ronmental science, one in business, one in arts, and two in the combined arts and science program. In semester six, the students demonstrated statistically improved performance compared with the control groups, similar to the finding for the first cohort studied. Objective 2: To Determine Whether Entering Grades Affect Subsequent Performance The academic performance of all students who have taken an EBL course (n = 106 students) was evaluated at the end of their third year of university (after completing semester 6). The students were divided into four groups, according to their grades upon admission to university. Their overall academic average for all courses completed by the end of semester 6 is shown in Figure 2. 7 80 6 75 5 70 Stratum 1 Stratum 2 Stratum 3 Stratum 4 Grade Improvement (%) 8 85 Grade 90 4 Figure 2. Average grade at end of year 3 Note: The impact of one first-year course using enquiry-based learning (EBL) on the average grades during courses taken in the second year of undergraduate study compared with a control group of students in a small-group seminar course not taught in an enquiry-based learning mode. Students were stratified into four levels according to their grade average on admission to university. Performance is shown in two ways: the average grade performance for each stratum after completing three years of study shown in black diamonds and the actual grade improvement (comparison of end year three grades and entering grades) shown in grey squares. Note the greater improvement by students with the lowest entry grades but all groups show an overall improvement. The average grades for students in all four strata improved, as would be anticipated based on the data reported in Figure 1. To explore this effect further, we analyzed the change in grade performance between grades upon entry to university and grades at the end of the third year of study (upon completion of six semesters). The results appear in Figure 2. The data reveal that the improvement in performance is greatest for those students with the lowest grades upon entry to university. Objective 3: To Investigate Whether the EBL Seminar Changed the Students’ Approaches to Resources/Research Materials Eighteen students completed a survey on their use of research resources before they started and immediately upon completion of an EBL seminar. Students ranked the frequency of use of particular types of resources on a five-point scale from never to frequently in three areas: (1) access to electronic resources; (2) access to scholarly journals, encyclopedias, and specialized websites; and (3) access to human resources CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 85 (faculty, librarians, staff, and fellow students). There were significant shifts in the pattern of access to resources. The results are shown in Table 1. Table 1. Changing Patterns in Accessing Research Resources. The patterns of access to resource materials by students (n = 18) were assessed by surveys immediately before and after completion of an enquiry-based course. Significantly greater or less use of specific resources is shown. Greater Use Access to electronic search agents and resources Internet search engines (Google, Yahoo etc.) Wikipedia Online data bases p Access to scholarly journals, encyclopedias and specialized websites Original scholarly articles p Specialized websites of research institutions p General encyclopedias p Special encyclopedias p Access to personnel as resources Reference librarians p Professors Teaching assistants Family and friends Fellow students Less Use No change < 0.05 p < 0.05 - < 0.01 < 0.05 < 0.05 < 0.05 - < 0.01 - p < 0.05 p < 0.05 p < 0.01 No change The EBL students shifted away from a relatively naïve mode of research that relied upon Wikipedia and talking to professors, teachers, and family as important resources. They developed a more sophisticated research repertoire including consulting scholarly articles, specialized websites of research institutions, general and specific encyclopedias, and online data bases. The specific shift to consulting scholarly articles is striking. A comparison of sources consulted before and after the course appears in Figure 3. Students appear to have developed more independent research capability and confidence because they relied less on family, friends, professors, and teachers. Moreover, they reported a significant increase in the frequency with which they consulted reference librarians to help to identify appropriate sources. Extending this study further, we explored the persistence of different patterns of accessing resources as students progressed through their programs of study. We compared the patterns of access to information between the same group of students (EBL course) surveyed above, with a control group (students who had not completed an EBL course in their first year). The control group was matched for entrance grades and gender with the experimental group. The results of the comparison of their responses are shown in Table 2. The EBL group used Wikipedia, professors, and teachers less frequently as a resource compared with the control group. They continued preferentially to use scholarly articles, more sophisticated data-bases and encyclopedias, and reference librarians as resources. CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 86 12 10 8 6 4 2 0 Never Sometimes Always Figure 3. The impact of an enquiry-based course on access to original scholarly articles. Note: The responses of students (n = 23) before the course indicating how often they accessed journal articles in preparing to write an essay or make a presentation is shown in black. The response at the end of the course is shown in grey. There was a definite and clear shift in the behaviour of students. Table 2. Change in Patterns Comparison. This table compares the change in patterns of access to resource materials between students who experienced an enquiry-based course (EBL) with a control group of students who did not take an enquiry-based course (Non-EBL) (n = 18 in each group). Significantly greater or less use of specific resources is shown for the EBL group compared with the controls. Greater Less Comparison of access to resources use by EBL group use by EBL group Access to electronic search agents and resources Internet search agents (Google, Yahoo etc.) No difference Wikipedia p < 0.01 Online data bases p < 0.05 Access to scholarly journals, encyclopedias and specialized websites Original scholarly articles p < 0.001 Specialized websites of research institutions p < 0.05 General encyclopedias No difference Special encyclopedias p < 0.05 Access to personnel as resources Reference librarians p < 0.001 Professors p < 0.01 Teaching assistants p < 0.05 Family and friends No difference Fellow students No difference CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 87 Objective 4: To Study Whether Participation in an EBL Course Influenced Student Engagement Students reported on their level of engagement through volunteerism by completing a survey before starting their small group experience in the second semester and again at the end of their second year of study. The results appear in Table 3. When subjected to χ2 analysis, students who participated in the EBL seminar course reported a significant increase in the time they spent volunteering in the community (outside the university), and a significant increase in the time they spent volunteering inside the university (in clubs, societies, and in governance processes) compared with members of the control group. In addition, a higher proportion of EBL students were engaged in international experiences during the summer vacation between their first and second years of study. These results are summarized in Table 3. Table 3. Student Engagement Measure. A measure of engagement of students. Engagement of students in an enquiry-based seminar (EBL) is compared with those who did not participate (non-EBL). EBL Group (n = 56) Start semester 1 End semester 4 Total time volunteering (h/wk) 3.4 + 0.7 5.7 + 0.3* Within the university (h) 1.6 + 0.3 2.8 + 0.1* Outside the university (h) 1.8 + 0.2 2.9 + 0.3* International experience summer 92% Note: * indicates significant difference. Non-EBL group (n = 56) Start semester 1 End semester 4 3.3 + 0.6 4.6 + 0.6 1.7 + 0.2 2.0 + 0.3 1.6 + 0.4 2.6 + 0.3 66% DISCUSSION We set out to examine quantitative outcomes from using EBL as part of the firstyear experience at the University of Guelph. We have already reported on qualitative outcomes (self-reflection by students) of the impact of such a course and disseminated those results. In that research, students reported that they not only learned content, they also learned processing and reasoning skills that enhanced their learning experiences in other courses and gave them the confidence to believe that they were more effective and responsible learners (Murray & Summerlee, 2007). In the present paper, we report that those students showed a superior level of performance in subsequent courses, had different patterns for accessing research resources compared to the control group, and had a higher level of engagement in their university and community. Moreover, the effects of EBL on academic performance appear to be inversely correlated with academic achievement prior to entering university: students with lower entering grades showed the greatest percentage increase in grades-based performance compared with their peers in the control groups and compared to their EBL peers with higher entering grades. Impact on Performance Students in two different EBL classes showed superior performance after first year when compared to control groups (Figure 1). At one level, it might be argued that CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 88 the higher rate of academic achievement in the small group seminars is explained by the fact these seminars serve as learning communities. Alexander Meiklejohn (Smith, 2001) first introduced this concept in his short-lived experimental college in Wisconsin in the 1920s. He argued that students who engaged in learning together – effectively a learning community – showed superior academic performance when compared to students learning alone. The concept re-emerged in the late 1980s, supported by the growing recognition that student engagement in educationally purposeful activities, inside and outside the classroom, is a precursor to higher levels of achievement (Kuh, 1996; 2003; Zhao & Kuh, 2004). Lenning and Ebbers (1999) recognized four generic learning communities: curricular communities (students co-enrolled in two or more courses linked by a common theme), classroom learning communities (group process learning in a classroom setting), residential learning communities (students in residence clusters taking two or more classes together), and student-type learning communities (targeted groups of similar academic achievement and/or underrepresented groups). Students in the EBL seminars certainly fit within the concept of classroom as learning community. However, while the shared academic experience in the class may be part of the basis for the immediate success experienced by the EBL students, learning together in a small group is not an adequate explanation for their persistently enhanced success. In subsequent semesters, students from the EBL group consistently performed better than their peers in the control groups, including a control group of students who selected different small group seminars but were not taught in an enquiry-based mode. Moreover, the EBL group showed higher performance for the remainder of their undergraduate experience, compared to the didactically taught group (Figure 1), something students in the non-EBL small group did not do. Theoretically, learning communities are intentionally structured to help students make two types of links. The first is to encourage them to connect ideas from different disciplines; the second is to forge enduring social connections within the group that will allow them to develop their own identity and voice within the group. These tenets are fundamental underpinnings of closed-loop reiterative EBL and are practised and reinforced throughout the EBL courses. This focus alone, however, does not seem sufficient explanation for the persistence of the superior academic performance by the EBL groups. We have already reported that students in EBL seminars show a superior level of motivation and have greater confidence in their ability and responsibility to carry out independent learning (Murray & Summerlee, 2007). Data presented in the present paper demonstrate that, working in an enquiry-based mode, students refine their approach to research (Table 1): they seek more sophisticated resources, rely less on family and friends for information, rely less on Wikipedia, and more frequently consult scholarly papers and articles. There was also a notable increase in the use of reference librarians as resources to help students find materials. This supports the contention of Kuh and Gonyea (2003) about the increasingly important role that libraries and librarians must play in student engagement and support for learning. Moreover, these shifts in patterns of information seeking differentiate the EBL students from their colleagues (Table 2). A comparison of the ways students sought information in their third year, highlights significant differences between EBL students and their peers. It is possible that the higher academic performance in the EBL groups is related to a combination of factors: being part of a tight-knit learning community, being more motivated to CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 89 find out and learn, and having developed the research skills and the understanding of the importance of research, to be able to find information to support their learning. Finally, students reported increased confidence in their own abilities (Murray & Summerlee, 2007), recognized the limits of their knowledge, and had the ability to find information efficiently compared to non-EBL students. These may also be important factors in driving more effective learning. There has been a considerable amount of research, especially in the United States, about the effect of academic selectivity on the undergraduate student body at socalled quality institutions (Bowen & Bok, 1998; Kuh & Pascarella, 2004; Pascarella & Terenzini, 1991; Pascarella et al., 2006; Rumberger & Thomas, 1993; Thomas, 2003). Although there are variations in the definition of selectivity in admissions, the generally accepted surrogate for quality is the notion that the average or median score of entering or enrolled students, on standardized tests such as the ACT or SAT, reflects academic ability and drives a higher level of quality in a given institution (Bowen & Bok, 1998; Flowers, Osterlind, Pascarella, & Pearson, 2001; Pascarella et al., 2006). The logic underlying this belief, at first glance, appears reasonable: the interaction among students of comparable abilities constitutes a dimension of institutional impact (Kuh, Shuh, & Whitt, 1991; however, it is difficult to find supporting evidence. In fact, such beliefs belie the significance of the diversity of backgrounds, experiences, and abilities that underpin much of society’s critical drive for greater inclusiveness (Bridges, Cambridge, Kuh, & Leegwater, 2005; Harper, Carini, Bridges, & Hayek, 2004; Kuh & Umback, 2005). In other words, entering grades are used as a convenient proxy for quality, and an institution with a higher grade-average cut-off is deemed to be of higher quality, although there is little evidence to support this conclusion (Ehrenberg, 2003). There have been three major studies on the impact of admission grades on a variety of alumni outcomes (Bowen, 1977; Pascarella & Terenzini, 1991; Pascarella et al., 2006) that have come to two fundamentally similar conclusions. First, there is a positive link between entering grades and career success (in particular, earnings). So if quality is measured in economic terms, there may be an argument to support the contention that the higher the entry grades, the better students will perform and, concomitantly, the higher the quality of the institution. The second finding is that there is very little evidence that students’ cognitive, developmental, or psychosocial outcomes are correlated with entering grades: the correlation at most institutions is very weak and there are huge variations between educational establishments. In other words, these studies imply that educational experience can influence student performance and is by no means necessarily related to the starting point (that is, grades on entry). Data in the current paper support this latter conclusion. The overall level of superior performance was not positively correlated with entering grades; in fact, it appears to be the reverse. The number of students studied here is not sufficient to allow for statistically significant correlations, however, the trend is very clear: all students in the EBL seminar groups performed better, on average, in their subsequent courses, compared to their peers in the control groups, and the effect on students with lower entering grades was greatest. Perhaps this finding is not surprising; academic high flyers do not have as much leeway for grade improvement. Nevertheless, the self-reflections suggest that all students experience a greater level of satisfaction with their educational experience and increased confidence as they move forward (Murray & CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 90 Summerlee, 2007). Moreover, students come to appreciate that all their colleagues, of whatever academic, social, ethnic, or economic background, enhance their learning experience. In a meta-analysis of engagement by students, where varying levels of problembased learning (PBL) methodology were used, Ahlfeldt and colleagues (2005) demonstrated that engagement increases in three circumstances: (1) as course level increases; (2) as class size decreases; and (3) as the component of PBL increases. Their analysis was based on responses to questions in the National Survey of Student Engagement (NSSE) by students from a number of institutions practising various forms of PBL. As we mentioned above, there are numerous different forms of pedagogical approach encompassed in the term PBL. Nevertheless, Ahlfeldt and colleagues (2005) were able to demonstrate statistically significant improvement in engagement among students involved in small-group active learning. The inference is that greater engagement will lead to higher academic performance and greater satisfaction with the educational experience. The data presented by Ahlfeldt and colleagues also suggest that PBL classes are more effective at upper levels of undergraduate study. The implication is that upper year students are more likely to have the skills to be able to take advantage of PBL. The data presented in the present paper, however, suggests otherwise. In the circumstances that we examined, students with an EBL experience in their first year performed better than their peers (Figure 1), were more engaged in the community (Table 3), and demonstrated research skills that set them apart from their peers (Tables 1 and 2). It is tempting to suggest that the rigorous approach to EBL that we implement in the classroom, and report on in this paper, accounts for these differences. It is possible that there is a correlation between participation in the EBL seminar and subsequent performance, but the data sets are not adequate to substantiate this observation. Both Ahlfeldt et al. (2005) and Murray and Summerlee (2007) have analyzed the relative success of PBL based on self-reporting by students (completing surveys). The quantitative data provided in this paper augments and confirms the conclusions of student self-reflection. Additionally, and perhaps more persuasively for faculty who are concerned about the apparent lack of sequence and structure in enquiry- or problem-based learning, the data demonstrate that students also develop conventional intellectual rigor. Impact on Engagement The literature around engagement and academic performance is compelling (Ahlfeldt et al., 2005; Kuh, 2003;Kuh & Hayek, 2004), and in the present study, the students in the EBL seminars report a higher level of engagement. The literature on engagement has tended to rely on information gathered by the National Survey of Student Engagement (NSSE), which asks students to respond to questions such as how often they have asked questions in class, worked with other students, worked with classmates outside class, or tutored other students? (Ahlfeldt et al., 2005). These sorts of questions address the fundamentals of EBL, so it would be reasonable to expect that the meta-analysis of performance of EBL/PBL students would reveal greater levels of engagement. At the same time, the students are asked questions about analyzing, synthesizing, and organizing data and applying information. Again, the latter are absolutely critical elements CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 91 of PBL/EBL (Barrows, 1986; Murray & Summerlee, 2007) so we would anticipate that under these criteria for student engagement, students in PBL/EBL classes would score highly. Consequently, to avoid this potential predictability, we chose to use a different definition of engagement: participation in volunteer activity in the community and participation in an international experience in the summer break between first and second year of university. Students at the University of Guelph already have a very high level of volunteer activity compared with university students across Canada: the participation rate is 70 per cent while the Canadian average for volunteer activity is 40 per cent. Nevertheless, as Table 5 indicates, students in the EBL seminar groups showed an increase in overall level of volunteer activity after completing the seminar class, and a greater proportion chose an international summer experience. In the survey, students had the opportunity to provide commentary on their level of participation. Some of the comments reflect a clear understanding of the value of engagement to overall self-awareness: “I learned so much from volunteering that I decided to expand my horizons and visit another country,” and “I realized that I was discovering so much about myself as I volunteered – I was motivated to do more and I found I learned so much more.” One potential drawback in the way we sought to establish participation could be that volunteering and participating in an international experience may not be available to all students: students who lacked financial means may not have had time to volunteer owing to work commitments, or they may have been unable to afford an international experience. None of the students reported this as an impediment to participation in either volunteer activities or an international experience, and the same concern would have applied to both the EBL and the non-EBL group. It is also possible that students who chose the EBL seminars were already more confident and this might affect the engagement results but this could not be tested. SUMMARY In conclusion, the observations reported in this paper support the qualitative data previously published: students show superior academic performance after completing a first-year experience that truly engages them in learning and in learning about how to learn. Such an experience builds their confidence, but more importantly, it changes their approach to research and to the use of resources. The experience of EBL provides a context for students to develop the skills necessary to cope more effectively with subsequent material presented in traditional formats. These results raise intriguing questions about how to redesign the first-year experience for university students to create a more engaged cohort of learners. CONTACT INFORMATION Alastair Summerlee Presidential Offices University Centre University of Guelph Guelph, ON N1G 2W1 [email protected] Jacqueline Murray Department of History MacKinnon Extension University of Guelph Guelph, ON N1G 2W1 [email protected] CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 92 Alastair Summerlee is the 7th President of the University of Guelph. He has continued teaching while holding administrative positions and was awarded a prestigious 3M Teaching Fellowship for outstanding leadership in teaching, education and academic program development. His research in biomedical sciences is acclaimed internationally and he has attracted significant funding to support his research, published extensively and been an invited lecturer at universities and colleges around the world. For the past five years Dr. Summerlee has served on the board of directors of World University Service of Canada (WUSC), one of the country’s leading international development agencies. Jacqueline Murray is Professor of History at the University of Guelph, where she served as Dean of the College of Arts from 2001-2006. She is a member of the Graduate Faculty at the Centre for Medieval Studies at the University of Toronto. Prior to joining Guelph she was Professor of History and Director of the Humanities Research Group at the University of Windsor. She has been involved in numerous projects to redesign curricula and implement new or alternative pedagogies. Her internationally-recognized research on gender and sexuality in medieval society has been funded by the Social Sciences and Humanities Research Council of Canada. ACKNOWLEDGEMENTS The authors would like to thank the students for their energy, enthusiasm, and constant drive to make us do more and do better and the various and several people who have provided feedback on the manuscript as it was being prepared. REFERENCES Ahlfeldt, S., Mehta, S., & Sellnow, T. (2005). Measurement and analysis of student engagement in university classes where varying levels of PBL methods of instruction are in use. Higher Education Research and Development, 24(1), 5-20. Albanese, M. A. (1993). Problem-based learning: A review of literature on its outcomes and implementational issues. Academic Medicine, 68(1),52-81. Barrows, H. (1986). A taxonomy of problem-based learning methods. Medical Education, 20, 481-486. Bridges, B. K., Cambridge, B., Kuh, G. D., & Leegwater, L. H. (2005). Student engagement at minority serving institutions: Emerging lessons from the BEAMS project. In G. H. Gaither (Ed.), What works: Achieving success in minority retention, (pp. 2543). New Directions for Institutional Research. San Francisco: Jossey-Bass. Bowen, H. (1977). Investment in Learning. San Francisco: Jossey-Bass. Bowen, W. & Bok, D. (1998). The shape of the river: Long-term consequences of considering race in college and university admissions. Princeton, NJ: Princeton University Press. Ehrenberg, R. (2003). Reaching the brass ring: The US News and World Report rankings and competition. Review of Higher Education, 26,145-162. CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 93 Flowers, L., Osterlind, S., Pascarella, E., & Pierson, C. (2001). How much do students learn in college? Cross-sectional estimates using the College Basic Academic Subjects Examination. Journal of Higher Education, 72, 565-583. Harper, S. R., Carini, R. M., Bridges, B. K., & Hayek, J. (2004). Gender differences in student engagement among African American undergraduates at historically Black colleges and universities. Journal of College Student Development, 45, 271-284. Kuh, G. D. (1996). Guiding principles for creating seamless learning environments for undergraduates. Journal of College Student Development, 37, 135-148. Kuh, G. D. (2003). What we’re learning about student engagement from NSSE. Change 35, 24-32. Kuh, G. D. (2005). Putting student engagement results to use: Lessons from the field. Assessment Update, 17, 12-13. Kuh, G. D., & Gonyea, R. M. (2003). The role of the academic library in promoting student engagement in learning. College and Research Libraries, 64, 256-282. Kuh, G. D., & Hayek, J. C. (2004). Quality time: An in-depth look at the National Survey of Student Engagement. Currents, 30, 11-12. Kuh, G. D., & Pascarella, E. T. (2004). What does institutional selectivity tell us about educational quality? Change, 36, 52-58. Kuh, G. D., Shuh, J., & Whitt, E. (1991). Involving colleges. San Francisco: JosseyBass. Kuh, G. D., & Umbach, P. D. (2005). Experiencing diversity: What can we learn from liberal arts colleges? Liberal Education, 91, 14-21. Lenning, O., & Ebbers, L. (1999). The power of potential of learning communities: Improving education for the future. ASHE-ERIC higher education report 26 (Number 6). Washington DC: George Washington University, Graduate School of Education and Human Development. Murray, J., & Summerlee, A. J. S. (2007). The impact of problem-based learning in an interdisciplinary first-year program on student learning behaviour. Canadian Journal of Higher Education 37(3), 87-107. Pascarella, E. T., & Terenzini, P. (1991). How college affects students. San Francisco: Jossey-Bass. Pascarella, E. T., Cruce, T., Wolniak, G. C., Kuh, G. D., Umbach, P. D., Hayek, J. C., Carini, R. M., Gonyea, R. M., & Zhao, C. M. (2006). Institutional selectivity and good practices in undergraduate education: How strong is the link? The Journal of Higher Education, 77, 251-285. Rumberger, R., & Thomas, S. (1993). The economic returns to college major, quality and performance: A multilevel analysis of recent graduates. Economics of Education Review, 12(1), 1-19. Schmidt, H. (1983). Problem-based learning: Rationale and description. Medical Education, 62, 305-315. CJHE / RCES Volume 40, No. 2, 2010 Engagement, Performance and Enquiry-based Learning / A. Summerlee & J. Murray 94 Sharp, B., & Dianda, J. (2005). Changes in students learning behaviours. Papers of Academic Colleges, Council of Ontario Universities – on-line publications http://www. cou.on.ca/_bin/publications/onlinePublications.cfm Smith, B. L. (2001). The challenge of learning communities as a growing national movement. AAC&U, 4(1), 1-5. Summerlee, A. J. S., & Christensen Hughes, J. (2010). The future of university education: Obsolescence or survival? In J. Mighty, & J. Christensen Hughes (Eds.), Taking stock: Research on teaching and learning in higher education (pp. 243-250). Kingston, ON: McGill-Queen’s University Press. Thomas, S. (2003). Long-term economic effects of college selectivity and control. Research in Higher Education, 44, 263-299. Vernon, D. A., & Blake, R. L. (1993). Does problem-based learning work? A metaanalysis of evaluative research. Academic Medicine, 68, 550-563. Zhao, C., & Kuh, G. D. (2004). Adding value: Learning communities and student engagement. Research in Higher Education, 45, 115-138. CJHE / RCES Volume 40, No. 2, 2010