J. Murray & A. Summerlee / Positive impacts of problem-based learning 87 CSSHE SCÉES Canadian Journal of Higher Education Revue canadienne d’enseignement supérieur Volume 37, No. 3, 2007, pages 87-107 www.ingentaconnect.com/content/csshe/cjhe The Impact of Problem-Based Learning in an Interdisciplinary First-Year Program on Student Learning Behaviour Jacqueline Murray and Alastair Summerlee University of Guelph ABSTRACT Canadian universities are struggling to address seemingly contradictory challenges pertaining to undergraduate education: high demand and underfunding. A number of instruments, including the National Survey of Student Engagement (National Survey of Student Engagement, n.d.), have led to greater priority being placed on the undergraduate experience. Yet, strategies to ensure student satisfaction with their education, through initiatives such as small classes and personal contact with faculty, seem at odds with the large classes necessitated by fiscal imperatives. We carried out a systematic investigation of the impact of one problem-based learning course on first year students’ experiences. We also investigated the persistence of skills and attitudes learned in this single exposure to problem-based learning. The results of our investigation show that this course had very positive effects on the immediate and persistent behaviours of students. Our research provides empirical evidence of the effectiveness of problem-based learning and leads us to suggest how a problem-based approach might help universities enhance the quality of education and the undergraduate experience. RÉSUMÉ Les universités canadiennes sont aux prises avec deux défis apparemment contradictoires en matière d’enseignement au premier cycle : une demande élevée et un financement insuffisant. Plusieurs instruments, dont un sondage sur l’engagement étudiant (National Survey of Student Engagement), ont accru l’importance accordée à l’expérience étudiante au premier cycle. Néanmoins, les stratégies 88 CJHE / RCES Volume 37, No. 3, 2007 mises en œuvre pour accroître la satisfaction, comme les classes de petite taille et les relations individualisées avec les enseignants, vont à l’encontre des mesures imposées par les contraintes financières. Nous avons réalisé une enquête sur l’impact sur l’expérience étudiante de première année d’un cours basé sur l’apprentissage par problèmes. Nous avons aussi étudié la persistance des compétences et attitudes acquises suite à cette unique expérience de l’apprentissage par problèmes. Les résultats de notre enquête démontrent que ce cours a eu des effets très positifs sur les pratiques immédiates et à long terme des étudiantes et étudiants. Notre recherche offre des preuves empiriques de l’efficacité de l’apprentissage par problèmes. Selon nous, cette approche pourrait aider les universités à améliorer la qualité de leur enseignement et l’expérience de la population étudiante.. INTRODUCTION Universities are currently facing the dual challenges of enhancing the quality of teaching and learning, particularly at the undergraduate level, while at the same time they struggle to mitigate the negative effects of financial constraints, enrolment increases, and larger classes. Against this context of competing and seemingly irreconcilable priorities, it is difficult to engage in a meaningful discussion of curricular and pedagogical innovation without fear that the very foundations of postsecondary education are being eroded. Our research invites both university administrators and faculty members to step outside their respective comfort zones to consider one pedagogical method that may address both the imperatives of fiscal restraint and enhanced learning. We argue that the widespread implementation of problem-based learning across the undergraduate curriculum would address both demands, although it would require a profound cultural shift in approaches to higher education. For departments and faculty members, our research challenges conventional views of teaching and learning and discipline-based education. Fundamental notions of coverage, demonstration of content-based expertise, and familiar modes of course delivery are all questioned. For university administrators, our research suggests the need to reassess such structures as course weightings, timetabling, and the types of infrastructure provided for teaching and learning. Yet, it is our view that the outcomes of problem-based learning, as supported by our research findings, if applied widely across the curriculum, would not only overcome resistance, but would also address many areas of lingering dissatisfaction in the current organization and culture of universities. The pedagogical literature is replete with subjective accounts and assertions about the benefits or otherwise of active learning strategies and, in particular, problem-based learning (Barrows, 1996). There is however a dearth of empirical evidence, particularly outside the context of medically-based curricula, about J. Murray & A. Summerlee / Positive impacts of problem-based learning 89 the impacts of problem-based learning. The present paper provides empirical data to demonstrate the advantages of problem-based learning within a single undergraduate course, compared with other small group experiences, and with traditional methods of first-year course delivery. It also provides some insight into the persistence of learning techniques and the confidence gained in a problem-based course, beyond the initial encounter with problem-based learning, throughout the remainder of an undergraduate degree. In a major conference on the impacts of problem-based learning in 2000, Samford University (Birmingham, Alabama) reported that this approach enriches undergraduate learning, can be conducted in conventionally funded and organized undergraduate programs, is better applied across a whole curriculum than in single courses, requires faculty training, and leaves students more enthusiastic about learning. It also suggested that problem-based learning is more difficult to apply to undergraduate students outside medical curricula (Chapman, Keller & Fournier, 2002). Many of these claims, however, were based on subjective observation (Blumberg & Eckenfels, 1988). Our study offers a systematic analysis of students’ perceptions of the immediate and longer-term effects of learning in a single, interdisciplinary problem-based course offered in first-year university. All of the students were enrolled in their second semester and, in addition to the problem-based course, were enrolled in an array of other traditionally taught courses and were carrying a full course load. The specific objectives of the research were to explore the students’ perceptions of the value of the problem-based course in terms of developing processing and reasoning skills and in meeting their expectations. We also explored the transferability and persistence of learning skills developed in the problem-based course for the remainder of the students’ various programs of study. The opportunity to conduct this research was presented when the University of Guelph implemented a first-year seminar program. Students who were registered in the first year of various baccalaureate degree programs in the arts, sciences, combined arts and science, and commerce were afforded the opportunity to register for small group classes which were designed by senior faculty with the express goal to foster learning across disciplines. We designed a course that bridged our areas of expertise – Murray is a medieval historian and specialist in the history of sexuality and Summerlee is a biomedical scientist with experience developing and delivering curricula using problem-based learning methods. In this course we sought to explore the cultural and scientific approaches to understanding sex, gender, and sexuality from antiquity to the present. Each problem presented to the students was designed to address a coordinated series of process and reasoning skills, and to empower the students to learn behavioural traits important for continuous learning, as well as to accumulate a body of knowledge and develop an understanding of the content. Enrolment in the course was limited to 16 students who were split into two groups of eight students. All classes were held with the eight students and two facilitators (Murray and Summerlee) 90 CJHE / RCES Volume 37, No. 3, 2007 present. How students perceived the learning process was determined through pre- and post-semester surveys, which were compared with the responses of two other sets of students: those who had also actively selected a small group experience in a different first-year seminar course, and those first-year students who were registered in a traditionally taught course. We have chosen to report the research in a manner that reflects both the way a scientist might report experimental research and the way a humanist might analyze and contextualize research. Consequently, we present both quantitative and qualitative data and conclusions. PROBLEM-BASED LEARNING Our course was designed according to a specific mode of problem-based learning: closed-loop reiterative problem-based learning (Barrows, 1986; Schmidt, 1983). This is quite distinct from the variations of problem-based learning upon which other studies, in particular those reporting negative or mixed results, have been based. Closed-loop reiterative problem-based learning demands that both faculty and students invest completely in the pedagogy (as described in detail below). As a result, both students and the course facilitators must trust the methodology and avoid the temptation to pedagogical hybridization; that is, they must adhere completely to a problem-based approach. Our adherence to a 100% closed-loop reiterative problem-based learning format sets this course apart from others discussed in the literature. For example, at Samford University, despite a campus-wide commitment and significant external funding to support the conversion of courses and indeed whole programs to problem-based learning, of the 132 courses involved, no course was completely based on problem-based learning; all were hybrids to a greater or lesser degree (Chapman, Keller & Fournier, 2002). Faculty continued to use lectures to set the context, to supply information and background material that students were assumed to lack, or to synthesize the “answers” to problems with conventional content. Many courses at Stamford appear to have implemented what might better be termed “problem-assisted learning”; they used problems as a means to provide small group and/or team experience, or to ensure active student participation (Chapman, Keller & Fournier, 2002). In other studies, authors report the use of lectures to introduce foundational theories or materials, with problems providing case studies for the application of concepts (Furmanski, Kane, Gupta & Pruitt, 2006). These hybrid courses also report mixed results. Students resisted the aspects of the course that required self-directed learning, and feared they were missing key content that was delivered by lecturers in conventional courses (Chapman, Keller & Fournier, 2002). In a so-called problem-based engineering course at the University of California, Berkeley, traditional testing of standard discipline content was used with unhappy results. Although students had developed intellectual self-confidence, they did not perform as highly on examination instruments as students from conventionally-taught course sec- J. Murray & A. Summerlee / Positive impacts of problem-based learning 91 tions (Furmanski, Kane, Gupta & Pruitt, 2006). Problem-based learning does not lend itself to conventional forms of assessment – nor does hybridization necessarily enhance the pedagogy or student learning. Rather, the inclusion of some aspects of conventional instruction and evaluation undermines the real potential of problem-based learning to develop students’ higher order research, analytical and critical skills, and their capacity for integrating knowledge. In closed-loop reiterative problem-based learning, students are presented with a problem which requires they explore issues that arise, and they research and learn in the context of that problem. The ultimate goal is not to solve the problem, but rather to use the problem as a starting point to reach a level of understanding that integrates research and encourages students to assimilate and integrate knowledge. The design of the problem creates additional conditions for students to learn how to communicate, criticize, and behave in academic situations (Barrows, 1985). The behavioural aspects of this learning process are equally as important as the content. Over the length of a course, students receive a series of cases or problems that have been constructed to focus the students on specific learning issues. Each case has been developed around a particular theme, concept or idea that will drive the students to develop a series of learning issues around the themes. These learning issues create a web of information and define the learning outcomes for the whole course. Students, then, are presented with a problem which they discuss, and they identify issues they do not understand. These are the learning issues that they will subsequently research. The students assume the responsibility to research issues that are not understood, to find the required information, to present it to the group to discuss, and to use their lack of background as the foundation for their learning. The value of this method of problem-based learning is that it slows down the problem-solving process and articulates its various stages so that all participants understand the process and their role in the process, and focuses understanding of the students at the sophisticated levels identified in Bloom’s taxonomy of educational objectives (Bloom, 1956). When students are presented with a problem, they are asked to identify three things: (1) what they know; (2) what they do not know; (3) what the best way is to find the information they need. Thus, the problem is presented in stages. Students receive a page of information and work through the issues that are generated on this page before they proceed to the next page. The problem comprises a series of pages and the facilitators serve as gatekeepers, managing the speed of the process, and not permitting the group to move to a new page until the current page is fully analyzed. As the group reads and discusses the information, they will accumulate a long list of learning issues, and may become bewildered by the information and daunted by the prospect of researching so many learning issues. However, the list of issues can usually be simplified and focussed on the basic issues that underlie the problem at hand. Students then divide the learning issues amongst themselves and agree to present their research at the next meeting, ideally two to three days hence. 92 CJHE / RCES Volume 37, No. 3, 2007 In the second meeting, each student presents a short discussion of his/her research. This includes presenting interesting books and articles, pictures, artefacts, or hand-outs that have been prepared for the group. Over time, the quality of the presentations, and the degree of critical thinking applied to the resources, will develop so that the students become more rigorous in their appraisal of the information provided. The information is then integrated into the problem, and the process of reading through additional pages resumes. If necessary, the group will again pause, distribute additional learning issues, and pursue and present further research. Every meeting of the group ends with a period of group processing. This is an absolutely critical component of the learning process, yet it is also one of the aspects that is most frequently absent from various experiments in applying problem-based learning techniques. This method of learning requires constant feedback and evaluation. At the end of every session, each member of the group is asked to provide oral feedback to every other member of the group. Each member is also asked to discuss his or her individual performance and contributions to the group. The feedback must be specific, explicit, and focussed on behaviours and actions that helped or hindered the group while working on the problem and the learning issues that emerged from it. The role of the facilitator is to ensure that this aspect of the process is completed at every session and to model honest, respectful, and focussed evaluation. The intent of group processing is to create a safe space in which students can genuinely appreciate how they are seen by others and how their behaviours contribute to the group’s learning process. Initially, there will be discomfort and even resistance to group processing; however, it is an essential component of problem-based learning. It is also critical that the facilitator(s) provide and receive feedback from members of the group in the same way as the students. Another fundamental principle of closed-loop reiterative problem-based learning is ensuring that academic assessment and grading are carried out in the same way as the pedagogical approach; for example, it is not effective to teach process but then examine content, or to teach using a problem-based approach but then examine students using multiple choice tests. In our course students were evaluated in two ways. The first evaluation was derived from the students’ written self-reflections and assessments of their peers and of the facilitators both halfway through the semester and again at the end of the course. The midpoint evaluation was used to provide feedback about the process and to help set the relative grading expectations. For the final assessment, the grades given by the student, his or her peers, and the facilitators were all averaged. This comprised 70% of the final grade. This part of the evaluation consisted of questions related primarily to process in identifying and researching learning issues, and sharing those issues with the group. In short, the first part of the evaluation of student learning focused on process rather than content. The second part of the evaluation was based on individual written reports. Students were provided with a series of articles from books, newspapers or the J. Murray & A. Summerlee / Positive impacts of problem-based learning 93 Internet and asked to analyze the issues that arose and their wider implications in light of the course themes. In this assessment, the facilitators evaluated process, content, and writing ability. Students were encouraged to submit drafts or outlines of their arguments in advance to either or both facilitators, who provided written feedback and suggestions. The written report comprised 30% of the final grade. METHODOLOGY There were three objectives of the research: (1) to explore the value of the problem-based experience compared with the experience of other pedagogical formats; (2) to investigate the degree to which problem-based learning provides, or does not provide, an advantage in improving students’ processing and reasoning skills; and (3) to determine the transferability and persistence of learned skills. Two experimental approaches were taken to analyze the impact of the course in problem-based learning. The first approach was designed to explore the immediate impact of problem-based learning on the skills and experiences of first-year students and to understand whether their experiences met their expectations. Students were asked to complete a survey of their perceptions before they started the course and immediately upon completing the course. The questions in the survey focussed on three categories: (1) process skills and experiences; (2) knowledge and reasoning skills; and (3) expectations of the university experience. The questions appear in Tables 1 to 3. Students were asked to rank their experiences on a scale of 1 to 10 where 5 was considered average. The responses of the three groups of students were compared: (a) students who completed our problem-based first-year seminar course; (b) students who completed a firstyear seminar course at Guelph (also small-group based) that was not taught in a problem-based manner; and (c) first-year students who were in regular courses in the Baccalaureate of Arts and Science program at Guelph. The results of the surveys were collated and compared. Significant differences in perception were determined using ANOVA and comparing students’ ‘t’ test scores. In the second approach, the students’ perception of the persistence of learning habits developed during the first-year seminar program was explored. Students in their third year completed a survey to explore how they perceived their experiences in the first-year problem-based class as having affected their subsequent approach to, and success at, university. The responses of students who had completed the problem-based course were compared with those of third-year students who had completed a first-year seminar that was not taught in a problem-based manner. The questions posed are shown in Table 4. In this survey, students were asked to rate their perceptions of the extent to which their experiences in the first-year problem-based course had affected their subsequent learning abilities and experiences. They were asked to rate according to a five-point scale: not at all; some; moderate; significantly; and very significantly. In addition to the survey responses, the students were invited to provide 94 CJHE / RCES Volume 37, No. 3, 2007 qualitative feedback about their first-year seminar and its impact on their subsequent educational experiences. RESULTS The goal of the first approach – the pre- and post-survey of first-year students – was to learn about their expectations of university teaching and learning and to determine the immediate impact of small group learning on students’ perceptions of processing and reasoning skills. The responses of the three groups are shown in Table 1 (perceptions of the impact on processing skills), Table 2 (perceptions of the impact on reasoning skills), and Table 3 (perceptions of the extent to which the experiences at university met expectations). There were 15 student respondents in each group (100% response rate). Overall, the responses of the students in the problem-based seminar group show a marked and significant improvement in scores compared with the other firstyear seminar group and the group in conventional courses. These changes between pre- and post-survey responses were present in almost every aspect of Table 1: Comparison of perception of processing skills before and after semester PBL group Seminar group Lecture group How much experience do you have with group work * 0 0 How much experience do you have with problemsolving * * 0 How much experience do you have with giving feedback to people * 0 0 How much experience do you have with independent research * 0 0 How comfortable are you using the library * 0 0 How comfortable are you using the internet for research 0 0 0 How comfortable are you assessing the quality of research data * 0 0 How much contact did you have with faculty last semester ** * 0 How much experience do you have making class presentations 0 0 0 n = 15 in each group * Significant (p <0.05) improvement compared with pre-test ** Significant (p <0.01) improvement compared with pre-test 0 No significant change observed J. Murray & A. Summerlee / Positive impacts of problem-based learning 95 Table 2: Comparison of perception of reasoning skills before and after semester PBL group Seminar group Lecture group Your ability to generate ideas * 0 0 Your ability to identify problems effectively * 0 0 Your ability to engage in constructive debate (*) 0 0 Your ability to participate in discussions * 0 0 Your ability to participate in group work (*) 0 0 Your experience of group work * 0 0 Your abilities to resolve conflict (*) 0 0 Your ability to write coherently * 0 # Your ability to deliver constructive criticism Your ability to receive and respond to constructive criticism (*) 0 0 * # # n = 15 in each group Significant (p <0.05) improvement compared with pre-test (*) Trend to improvement compared with pre-test 0 No significant change observed # Significant (p <0.05) reduction in performance compared with pre-test Table 3: Comparisons of the expectations of experiences before and after semester PBL group Seminar group Lecture group In class experience * * 0 Experiences in residence 0 0 0 Individual/personal contact with faculty Amount of active learning rather than passive presentation Receipt of prompt feedback from faculty on assignments/evaluations Respect for diversity of talent Respect for diversity of ways of learning ** ** * * 0 # ** 0 0 * * 0 0 # ## Sense that faculty have high expectations of your performance ** * 0 Evaluation procedures and fairness of the examinations * 0 0 n = 15 in each group * Significant (p <0.05) improvement compared with pre-test ** Significant (p <0.01) improvement compared with pre-test 0 No significant change observed # Significant (p <0.05) reduction in performance compared with pre-test ## Significant (p <0.01) reduction in performance compared with pre-test 96 CJHE / RCES Volume 37, No. 3, 2007 students’ perceptions of process and reasoning skills, and in their expectations of their university experiences. In particular, the problem-based learning group indicated a significant (p < 0.01) change in their direct personal contact with faculty, the amount of active versus passive learning, their receipt of direct feedback, and the sense that there were high expectations placed on them by faculty. In contrast, there were relatively few areas in which students in the non-problem-based learning first-year seminar course reported significantly improved perceptions. Students who did not participate in the first-year seminars scored themselves significantly worse in their ability to write coherently and to receive and respond to constructive criticism. Their surveys also showed that over the course of the semester they perceived a significant decrease in their ability to respect diversity of learning. Their responses also indicated that there was significantly less active learning in their program compared with their expectations. For the second part of the investigation, the transferability and persistence of skills developed during first-year seminars were explored during the third year of the students’ undergraduate programs at the University of Guelph. Two groups of 15 students were surveyed. An overview of the questions is shown in Table 4, and a summary of the results of the survey appear in Figure 1. In general, when compared with the group who participated in a small group seminar in the first-year but did not participate in the problem-based course, students who participated in the problem-based course reported a greater sense of awareness that process and reasoning skills learned in the course were transferable to learning experiences in other courses, and that these skills persisted throughout their university careers. The problem-based students noted that process skills such as the ability to work in groups, to be effective problem-solvers, and to be comfortable interacting with faculty were transferable and persistent attributes. They also noted that their abilities to engage constructively in debate, to Table 4: Survey questions posed to senior students Process Skills and experiences Knowledge and reasoning skills Group work Ability to identify problems and issues Problem solving Ability to engage constructively in debate Giving feedback to peers Ability to participate effective in group work Comfort with independent research Ability to deal with and resolve conflict Using the library Ability to critique information Using the internet for research Ability to write coherently Comfort with contact with faculty Ability to deliver constructive criticism to peers Making class presentations Ability to receive and respond to criticism For each question, students were asked to determine the extent to which the process, knowledge and reasoning skills, and experiences had been affected by the problem-based or non-problem-based experiences in their first-year seminar. They were asked to respond on a five-point scale: not at all, some, moderately, significantly, and very significantly. J. Murray & A. Summerlee / Positive impacts of problem-based learning 97 70 60 50 40 30 20 10 0 Not Some Moderate Significant Very significant Non problem-based group Problem-based group Students were asked to rank the extent to which their experiences in more senior courses had been affected by their experiences in their first-year seminars in the following categories: group work; problem-solving abilities; providing feedback to peers; comfort with independent research; using the library; using the internet for research; comfort with contacting faculty; and making class presentations. In general, students in the non-problem-based group (dotted histogram) indicated that effects of the first-year experience were moderate to significant on later performance. In contrast, students in the problem-based group (solid black) reported that their subsequent experiences and skills were significantly or very significantly affected by their first-year experiences. Process Skills and Experiences 70 60 50 40 30 20 10 0 Not Some Moderate Non problem-based group Problem-based group Significant Very significant Students were asked to rank the extent to which their abilities to accumulate knowledge and demonstrate reasoning skills in more senior courses had been affected by experiences in the first-year seminar classes. They were asked to grade the impact in the following areas: ability to identify problems, engage constructively in debate, participate in group work, deal with and resolve conflict, critique information, identify appropriate resources that provide understanding of issues, write coherently, and deliver, receive, and respond to constructive criticism. In general, students in the non-problem-based group (dotted histogram) reported a moderate impact of their first-year experiences on subsequent learning experiences. In particular, they observed a complete absence of influence of the first-year seminar on their ability to deliver, receive, and respond to constructive criticism. In contrast, students from the problem-based seminar group (solid black histogram) reported significant to very significant impact on their subsequent abilities to accumulate knowledge and demonstrate effective reasoning skills. Knowledge and Reasoning Skills Figure 1: Comparisons of the Transferability and Persistence of Processing Skills 98 CJHE / RCES Volume 37, No. 3, 2007 critique information, and to receive and respond to criticism were profoundly influenced by the problem-based course. Students in the non-problem-based group reported that their small-group course in their first year of study had a moderate to positive impact (using the five-point scale in the survey) on their abilities and skills later in their degree work (Figure 1), with the exception of their ability to deliver, receive, and respond to constructive criticism, which they reported was not at all affected by the first-year experience. When students were given the opportunity to express opinions freely about the ways in which the small group experience had influenced the rest of their university career and learning experiences, there was a significant difference in the responses of the two groups. Twelve of the 15 students in the problem-based group provided comments that indicated the experience was profound and significantly influential: “I developed a passion for learning” and “I developed life-long skills about how to learn” were simple reflections on the experience. Many of their musings were more reflective. For example, one student wrote, “The most important skill that I learned was to believe that no problem, no issue was beyond my comprehension.” This same student continued, “I arrived at University to do business because I did not believe I could learn science and was not interested in history and the arts. This course made me realize that learning is a passion and every subject, every aspect of learning about anything is not only exciting but fun.” The students who took the problem-based course also noted that the course provided them with transferable skills. One student wrote, “The skills that I took . . . . have been at the heart of my success at University.” Another stated, “This course has helped me in my overall studies . . . I learned . . . how to organize my thoughts and arguments into coherent sections.” Students identified specific skills that they learned and carried into other courses. For example, one person remarked, “I learned how to think for myself, how to be independent, how to manage my time, and to have confidence in my abilities.” Another commented, “I am more comfortable asking questions; I have a greater ability to take in information that is being presented and then researching issues afterwards in an effective way.” More than one student identified that he/she had learned to use the library effectively. Finally, one commented that the experience had provided her with “a subtle awareness of the difference between information and knowledge.” In contrast, only three students from the non-problem-based seminar group provided commentary about the transferability of skills that they had learned in the seminar class. Their common theme was that they looked forward to more “small group classes” which they found more demanding and more interesting than large, traditional lecture courses. “The seminar course was the course in the semester that provided the most interest [for me]. I wish there was more opportunity to engage in small group learning and to have more time to study material in greater depth.” The students in the traditional course provided comments that underscored their disappointment with large class size and lack of J. Murray & A. Summerlee / Positive impacts of problem-based learning 99 challenging material to study. For example, one student wrote, “All my first year courses were a disappointment. The first part of the course was spent making sure that we were all at the same level. For me, this was repetitious and boring. After that we started work on new material but never spent time on anything in depth. There was an obsession with covering a large amount of material and no time to think.” DISCUSSION The results of our research demonstrate that providing students with an opportunity to be immersed fully in a closed-loop reiterative problem-based learning experience profoundly affects their perceptions of their ability to research and analyze information, and meets or exceeds their expectations for learning in a university setting. Moreover, the skills learned in this single course are considered by the students to be transferable and persistent for the remainder of their university experience. Also, students believe the problem-based course to have a significant effect on their motivation, approach to learning, and organizational skills. These students perception of their abilities is significantly different compared with that of the students who participated in other first-year seminar classes that were not taught in a problem-based manner, and with students who were taught in more conventional, didactic first-year classes. There are some disturbing changes noted for students in the more traditional classes that are consistent with findings in other surveys of the first-year university experience overall. These students identified challenges in writing coherently and in appreciating the importance of diversity and tolerance of diversity in the classroom (see Tables 2 & 3). Our findings are significant for universities currently facing a series of interrelated challenges: in particular, the imperatives from government and the public alike that universities be accountable for delivering demonstrable valueadded education that prepares students to succeed in a globalized economy, despite a generation of underfunding that places Canadian universities among the poorest funded in North America (Rae, 2005). The implementation of Key Performance Indicators and benchmarking comes at a time when a university education is more highly prized than ever and enrolments have reached historic levels. In particular, the various public reports that rank universities, such as Macleans’ and the Globe and Mail’s Report Card, add considerable urgency to internal discussions about the quality of education and the most effective use of resources to achieve quality education. Increasingly, Canadian universities are participating in the National Survey of Student Engagement (NSSE) in order to assess how students are responding to their educational experience. The Ontario government has made participation in NSSE mandatory as part of the Reaching Higher plan for postsecondary education. How individual universities fare within their comparison group will affect future investment and program development. There are five benchmarks of Effective Educational Practice that underlie the NSSE survey. These are (1) 100 CJHE / RCES Volume 37, No. 3, 2007 level of academic challenge; (2) active and collaborative learning; (3) studentfaculty interaction; (4) enriching educational experience; and (5) supportive campus environment. For the NSSE survey, first-year and fourth-year students are surveyed. This provides a sense of the quality of student experience over the course of an undergraduate education. When the results of our research on students’ perception of their educational experience are compared with the NSSE benchmarks, it is evident that the problem-based learning format addresses all five of these areas. Moreover, the students in the problem-based learning first-year seminar scored significantly higher than either those who were in a small seminar or a traditional course, which implies that problem-based learning might address many of the issues raised in the NSSE survey. The problem-based learning format provides educational opportunities that exceed the simple desire for smaller classes, which is often proffered as a panacea for the ills that currently beset universities. With problem-based learning techniques, students develop transferable skills that persist throughout the course of their program of studies, and they develop qualities of self-directed learning that endure despite the return to large lecture format and content-focussed courses. Moreover, the results reported by first-year students in traditional courses are cause for concern, given they report lower expectations, negative experiences, and an overall decrease in their learning abilities over the course of the first year of university studies. Recently, at the request of the Council of Ministers of Education of Canada, the Ontario Council of Academic Vice Presidents developed a set of Guidelines for University Undergraduate Degree Level Expectations, aimed at providing a standard against which new and existing undergraduate programs could be measured (Ontario Council of Academic Vice Presidents, 2005). The various expectations for a baccalaureate degree, regardless of discipline, were set out under a series of rubrics: (1) depth and breadth of knowledge; (2) knowledge of methodologies; (3) application of knowledge; (4) communication skills; (5) awareness of the limits of knowledge; and (6) autonomy and professional capacity. This framework is largely based on Bloom’s taxonomy (Bloom, 1956). Again, students who participated in the problem-based learning seminar at the University of Guelph indicated increased competence in a number of key areas – for example, the ability to gather, review, evaluate, and interpret information; the ability to take initiative and to undertake a critical evaluation of arguments; and, the ability to propose solutions and frame appropriate questions. The students demonstrated a superior knowledge of methodologies, application of that knowledge, communication skills, an awareness of the limits of knowledge, and an understanding of autonomy and responsibility for learning. They were able to use the library effectively and to critique sources of information in ways that are normally expected of more senior students. We contend that problem-based learning techniques enhance students’ abilities to succeed in their degree programs and to meet the expectations of an undergraduate degree better than traditional lecture-based courses, and significantly better than a small class experience alone. J. Murray & A. Summerlee / Positive impacts of problem-based learning 101 There are a number of misconceptions around problem-based learning that are used as evidence that this mode of course delivery cannot be implemented outside well-resourced areas such as medicine, which has low faculty/student ratios and focuses on so-called “real world” applications of learning (Aronowitsch & Craaford, 1995; Bradbeer, 1996; Mennin & Martinez-Burrola, 1986; Smith, 1985). In particular, with the tremendous growth in university enrolment, coupled with chronic underfunding, there is a widespread perception that low enrolment courses are a luxury the contemporary multiversity cannot afford, especially at the lower levels of instruction. If one accepts the premise that one single problem-based learning course can provide in-depth education in which the students are more effectively employed in carrying out their own research rather than sitting in class – as evidence would suggest from the research in this paper – it might be possible to develop a model for education that is sustainable and financially possible even with the current level of funding. It might be necessary to use graduate or senior undergraduate student facilitators to achieve this but such models are worth investigating (Duch, Allen & White, n.d.). Canadian universities have tended to use a pyramid model to establish class size. In first year, classes are very large and then become progressively smaller as students proceed through a program of study. By the fourth year, the desire to provide a capstone experience for majors leads most disciplines to provide small-group learning opportunities. It has been suggested that problem-based learning is only successful at the fourth year, and that it is appropriate for material requiring higher order thinking as outlined in Bloom’s Taxonomy (Furmanski, Kane, Gupta & Pruitt, 2006). Our research, and the experience of our students, on the contrary, suggests that the fourth year may be too late and that problem-based learning should be available at the beginning of students’ university education. The greatest educational value, based on the transferability and persistence of skills and self-directed learning strategies, will accrue from problem-based learning in first year. Students can hone their skills in a small problem-based learning context and then implement them to enhance their learning at senior levels, including in lecture-based courses. Again, this may be a way to conceive of introducing and using problem-based learning to greatest effect and could be financially achievable. Problem-based learning is a dynamic process. The same problem will not generate identical responses from different groups. Some groups may move quickly through one case and need to move more slowly through another. Consequently, because of the unique and dynamic character of each group, the time devoted to a specific problem cannot be set arbitrarily. This necessary process has been criticized as dragging on and inhibiting the instructor’s ability to cover a set amount of material (Chapman, Keller & Fournier, 2002). However, arguably the most important learning outcomes, especially at the first-year level, are process and behavioural rather than content and the accumulation of fact-based information. Each group needs to have the flexibility to dictate the number of sessions needed, according to the background, experience and personalities of the members. 102 CJHE / RCES Volume 37, No. 3, 2007 Furmanski, Kane, Gupta and Pruitt (2006) suggested that students are not aware of their own abilities and limitations in self and peer assessment processes. This is not the experience in our courses and does not accord with the work of Barrows (1996) and others. Others have suggested that problem-based learning leads to grade inflation (Chapman, Keller & Fournier, 2002). Our experience again indicates that this is not the case, providing there is a careful discussion of expectations and if, by consistently engaging in group processing, students learn how to assess themselves and others. Each time we have collated our grades with those that students have assigned to their peers and themselves, there has been considerable consistency. Students develop a sense of rigour and standards of performance against which they measure themselves and their peers in a fair and responsible fashion. Grade inflation is not a temptation in a context which values intellectual honesty, hard work, and the quality of thought. One of the limitations of our study is that we did not attempt to follow the grade performance of students in the three groups reported in the current study. It would have been useful to compare the entering and exiting grades of students in the different groups. This work is currently underway. Universal small-group experience in the first year has been considered too expensive for under-resourced institutions. This view is based on traditional notions of course weighting which count a seminar as but one of the 10 semesterlong courses that comprise the average first-year student’s course load. Based on our experience and feedback from our students, this adherence to traditional notions of what counts as one course significantly undervalues problem-based learning seminars. Students report devoting more time and energy to this one course than to their other four courses combined. Given the learning outcomes and the transferability and persistence of skills that result from a problem-based learning course, arguably one course such as this can provide a learning experience equivalent to multiple large courses based on traditional modes of delivery and passive, rote learning; this comment was made frequently by students in their overall assessments of the course. By reducing the number of courses which students take, resources would be freed to permit universal first-year, problem-based learning seminars. The implementation of problem-based learning in the first year of university does not so much require increased resources, as a dynamic and radical rethinking of the purpose and goals of the first year. Given the well-demonstrated limitations of lectures, including the short-term retention of fact-based information, it is more a question of policy and pedagogy rather than of finances. Academic disciplines and departments need to recognize the inefficiency and limitations of conventional information transfer in lectures and instead begin to appreciate how the development of process and behavioural skills will lead to long-term student success. There are a number of key factors that are absolutely necessary for problem-based learning to succeed. As our research results have shown, not every small group experience will deliver the same learning outcomes as problem- J. Murray & A. Summerlee / Positive impacts of problem-based learning 103 based learning that is not partial or hybridized. Moreover, many of the mixed or negative results reported in the literature are directly attributable to modifications to the methodology or the complete absence of a critical component, particularly the group processing. To be successful, problem-based learning must be implemented in all its aspects. One critical factor in the success of problem-based learning is the size of the group. In our experience the optimum size of a group is eight students plus facilitator(s), although we have also had success with groups of seven or nine students. Some initiatives report groups as small as four to six (Chapman, Keller & Fournier, 2002; Dion, 1996; Rhem, 1998), while others do not discuss the size of the groups, suggesting a lack of attention to this aspect of the course structure. It is important that groups be large enough that there is a critical mass sufficient to take on learning issues, but groups that are either too large or too small impede the development of the necessary group dynamic. Some studies have suggested it is difficult for students to find a time to meet outside of the scheduled class. Our students were asked specifically not to discuss their research and learning issues outside of class but to wait until we were all present. This is essential for the integration of each individual’s research contribution to the whole group’s collective learning. Moreover, it is important for each student to have the experience of presenting her/his research to the group and to receive feedback in order to develop presentation skills. Students are also then afforded the opportunity to learn how to listen and to debate effectively and respectfully in the context of a group. When presenting and assessing the presentations of others is an iterative process, students have the opportunity to develop different critical and rhetorical strategies. Group feedback helps students to learn to recognize how to present core information and key points and relate these to the problem at hand. Working in the formal group means that the students benefit from group processing, which includes critiques of their presentation skills and the quality of their research. Although there does not appear to be published data to support this, our experience indicates that containing discussions within the group meeting, followed by group processing, creates a higher functioning group than otherwise. Nevertheless, flexible scheduling is necessary to ensure that every member of the group (including facilitators) is present at every class meeting. We were prepared to reschedule classes to accommodate travel, family emergencies, or illness. Students became so committed to the principle that every member be present that they were willing to meet at 7:00 am or in the evening, if that was the only way to accommodate everyone’s schedule. A third factor key to the success of problem-based learning is the role of the facilitator(s). It is critical that facilitators do not convey content, provide necessary background, or lapse into mini-lectures. If facilitators assume the role of expert, students will come to count on them to provide the context, the necessary data, or even “the answers” to the problem. Perhaps even more importantly, facilitators need to be mindful always to stimulate the discussion through asking open-ended questions (Rhem, 1998). Questions that have 104 CJHE / RCES Volume 37, No. 3, 2007 a “right” answer or, indeed, questions that reveal the fact that the facilitator is a content expert will elicit silence. These types of questions may also lead to the expectation that the facilitator will become the teacher and the leader and will ultimately take over the learning process. This will have a directly negative impact on the students’ experience and their learning. Students will then not only resist taking responsibility for the learning process, but they will begin to rebel against it and resent the facilitator’s expectation that they do research independently when ultimately they know they will be given “the answers.” Problem-based learning requires a considerable investment of time and imagination from the facilitators as they develop the problems prior to the start of the class. The focus of problem-based learning is not to solve the problem and find the correct answer. The best problems are ones that cannot be solved, but rather need to be understood from multiple perspectives offered by different disciplines, approaches, and experiences. This is central to one of the fundamental goals of problem-based learning and, indeed, of a university education in general: the appreciation of the diversity of truths and the multiplicity of perspectives in all areas of knowledge. The complexity of problems and the web of behavioural learning issues that they generate should not lead to the conclusion that there can be no purpose or defined learning outcomes for a problem-based learning seminar. Indeed, the facilitators worked assiduously to ensure the learning outcomes of our course were met. These were not stated goals, however, and the students in our course never saw our rather ambitious list of learning outcomes (see Appendix 1). Group processing is perhaps the most important factor to the success of problem-based learning. It is central to the pedagogy and yet it is the aspect most frequently dropped from hybrid and partial applications of problem-based learning. Group processing functions to build trust and a sense of community and helps students develop the skills to work as members of a team. It is the mechanism by which students learn to evaluate realistically their strengths and weaknesses. Members of the group identify when someone exceeds expectations or cuts corners. However, it is vital that the facilitators model and encourage feedback that is focused and accurate. It is not adequate for members of the group simply to heap praise on one another. They have to appreciate their own skills and weaknesses, and use the feedback from others to find ways to improve their performance and participation. With consistent practice at group processing, students develop a sense of intellectual responsibility to themselves and each other, and come to appreciate the reciprocal responsibility they have for their learning. Facilitators must ensure that there is sufficient time available for group processing at the end of each meeting, but otherwise they should function as regular members of the group. Facilitators should model how to give and receive honest and constructive feedback. A high point in our experience was the meeting when, rather than being told we did not participate sufficiently, a member suggested that we were too involved in the discussion of learning issues. This marked a point when the students had truly taken ownership of their own learning. J. Murray & A. Summerlee / Positive impacts of problem-based learning 105 CONCLUSION Although universities have adopted enthusiastically the language of learner-centred education and the goal of developing life-long independent learners, there are few empirical studies that demonstrate success in meeting these objectives. This study demonstrates that students in a problem-based first-year seminar report significantly increased levels of skill development and enhanced educational experiences compared to their peers in small seminars or in conventional lecture courses. Moreover, the skills and abilities developed in the problem-based seminar proved to be transferable to other learning environments and persisted throughout subsequent years. Problem-based learning addresses explicitly a number of the learning objectives and the educational goals of Canadian universities. This pedagogical method enhances student experience and empowers students to take control of their learning – the goals inherent in the language of learner-centredness. Although benchmarking and Key Performance Indicators may be anathema to how most university faculty evaluate the goals and objectives of postsecondary education, there are few who would dissent from the expectations set out for undergraduate degrees. The problem-based learning format addresses all the indicators established for assessing university education, whether it be through students’ perception of their education, or through the application of objective qualitative criteria. This research also demonstrates that small class size does not in and of itself significantly enhance student experience or the acquisition of knowledge and the development of critical and analytical skills. It is only through a profound reshaping of our learning paradigms that there will be a fundamental change in learning and teaching. These research results demonstrate that there is as yet unrealized potential to enhance student education and experience through the implementation of problem-based learning techniques. Moreover, if one course has the ability to transform all subsequent learning, imagine how much more could be done with widespread application of problem-based learning across the curriculum at all levels. REFERENCES Aronowitsch, E. & Crafoord, C. (1995). Problem-based learning in psychotherapeutic training. Psychoanalytic Psychotherapy, 9.1, 31-40. Barrows, H. (1985). How to design a problem-based curriculum for preclinical years. New York: Springer Publishing Company. Barrows, H. (1986). A taxonomy of problem-based learning methods. Medical Education, 20.6, 481-486. Barrows, H. (1996). Problem-based learning in medicine and beyond: a brief overview. In L.Wilkerson and W. Gijselaers (Eds.), New directions for teaching and learning 68 (pp. 3-12). San Francisco: Jossy-Boss. 106 CJHE / RCES Volume 37, No. 3, 2007 Bloom, B. (1956). Taxonomy of educational objectives: The classification of educational goals: Handbook I. Cognitive Domain. New York: Longmans, Green. Blumberg, P. & Eckenfels, E. (1988). A comparison of student satisfaction with their preclinical environment in a traditional and a problem based curriculum. Research in Medical Education. Proceedings of the Annual Conference on Medical Education, 27, 60-65. Bradbeer, J. (1996). Problem-based learning and fieldwork: A better method of preparation. Journal of Geography in Higher Education, 20(1). 11-18. Chapman, D., Keller, G. & Fournier, E. (2002). Implementing problem-based learning in the arts and sciences. Birmingham AL, Samford University Press. Dion, L. (1996). But I teach a large class… about teaching. A newsletter of the Center for Teaching Effectiveness, 50. University of Delaware. Retrieved October 31, 2007 from http://www.udel.edu/pbl/cte/spr 96-bisc2.html. Duch, B., Allen, D. & White, H. (n.d.). Problem-based learning: Preparing students to succeed in the 21st century. In Essays on teaching excellence: toward the best in the academy. Retrieved from http://teaching.uchicago.edu/pod/duch. html/ Furmanski, J., Kane, S., Gupta, S. & Pruitt, L. (2006). Work in progress: Problem-based learning and assessment of competence in an engineering biomaterials course. 36th ASEE/IEEE Frontiers in Education Conference. Retrieved October 31, 2007 from http://fie.engrng.pitt.edu/fie2006/papers/1710.pdf Mennin, S. & Martinez-Burrola, N. (1986). The cost of problem-based vs. traditional medical education. Medical Education, 20.3, 187-194. Ontario Council of Academic Vice Presidents. (2005). Guidelines for university undergraduate degree level expectations. Retrieved October 31, 2007 from http://www.uwo.ca/univsec/handbook/general/OCAV_Guidelines_2005.pdf Ontario Government. (2005). Reaching higher: The McGuinty government plan for postsecondary education. Retrieved from http://www.fin.gov.on.ca/ english/budget/bud05/pdf/bke1.pdf National Survey of Student Engagement. (n.d.). Benchmarks of effective educational practice. Retrieved October 31, 2007 from http://nsse.iub.edu/pdf/ nsse_benchmarks.pdf Rae, Bob. (2005). Ontario: a leader in learning. Report and recommendations. Toronto: Queen’s Printer for Ontario. Rhem, J. (1998). Problem-based learning: An introduction. The National Teaching & Learning Forum, 8,1. Schmidt, H. (1983). Problem-based learning: Rationale and description, Medical Education, 62, 305-315. J. Murray & A. Summerlee / Positive impacts of problem-based learning 107 Smith, B. (1985). Problem-based learning: The social work experience. In D. Boud (Ed.), Problem-based learning in education for the professions (pp. 109-120). Sydney: Higher Education Research and Development Society of Australia. ACKNOWLEDGEMENT The authors would like to thank Leslie Howsam, Alan Shepard, and Leonard Adams for their invaluable advice and assistance in the preparation of this article. They would also like to acknowledge the continuing inspiration provided by the students in their seminars. CONTACT INFORMATION Jacqueline Murray Department of History MacKinnon Extension University of Guelph Guelph, ON N1G 2W1 Tel: 519 824 4120 x58521 Fax: 519 837 8634 [email protected] Jacqueline Murray is Professor of History at the University of Guelph, where she served as Dean of the College of Arts from 2001-2006. She previously taught in the Department of History at the University of Windsor, where she was also Director of the Humanities Research Group. At both institutions she has been involved in numerous projects to redesign curricula and implement new or alternative pedagogies. Her research into gender and sexuality in medieval society crosses the boundaries of discipline and has received funding from the Social Sciences and Humanities Research Council of Canada. Alastair Summerlee is the 7th President of the University of Guelph. He has continued teaching while holding administrative positions and was awarded a prestigious 3M Teaching Fellowship for outstanding leadership in teaching, education and academic program development. His research in biomedical sciences is acclaimed internationally and he has attracted significant money to support his research, published extensively and been an invited lecturer at universities and colleges around the world. In 2006, Dr. Summerlee was named as the chair of the board of directors of World University Service of Canada (WUSC), one of the country’s leading international development agencies.