Understanding the Student Experience / A. C. K. Wong 60 CSSHE SCÉES Canadian Journal of Higher Education Revue canadienne d’enseignement supérieur Volume 45, No. 2, 2015, pages 60 - 80 Understanding Students’ Experiences in Their Own Words: Moving Beyond a Basic Analysis of Student Engagement Arch Chee Keen Wong Ambrose University Abstract This study examines the lived experiences of students as expressed in their reflections on their experiences of learning at Ambrose University in Calgary. It uses quantitative outcomes-related data from the National Survey of Student Engagement and the Theological School Survey of Student Engagement to illuminate qualitative data obtained through student focus groups. The analysis of the qualitative data was conducted using the constant comparative method developed by Glaser and Strauss. The study concludes with recommendations for improving student engagement. Résumé Cette étude examine les expériences vécues par des élèves telles qu’exprimées dans leurs réflexions sur leurs expériences d’apprentissage à l’Université Ambrose, à Calgary. Afin d’éclairer les données qualitatives obtenues par le truchement de groupes de discussion d’étudiants, l’étude utilise les données liées aux résultats quantitatifs de l’Enquête nationale sur la participation étudiante (NSSE) et de l’Enquête de l’école de théologie sur l’engagement des étudiants (TSSSE). L’analyse des données qualitatives a été réalisée selon la méthode comparative constante développée par Glaser et Strauss. L’étude se conclut par des recommandations afin d’améliorer l’engagement des élèves. CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 61 Many institutions of higher learning are under pressure from both internal and external stakeholders to report and clarify student learning outcomes. Faculty members, for example, want to know whether the curriculum is doing what it is supposed to be doing: that is, whether their students are actually learning (Astin & Lee, 2003), and whether they are teaching effectively (Chickering & Gamson, 1999; Scott & Scott, 2011). University administrators and student development personnel see a close relationship between student retention and financial benefit: that is, when students perceive that they have received a good education, the university fulfills its mission and also acquires tuition income (Seidman, 2005). At the same time, the standards of outside stakeholders, such as regional and national accrediting bodies, require institutions of higher learning to use both direct and indirect measures to demonstrate learning outcomes (Gordon, Ludlum, & Hoey, 2008). To provide the evidence that they are meeting their educational goals and fulfilling their mission, many institutions are beginning to analyze student engagement. One of the tools that colleges and universities use to measure student engagement is the National Survey of Student Engagement (NSSE), which has become the industry standard. Indeed, NSSE is a reliable means of both providing evidence of student learning and benchmarking it against that of peer institutions (Axelson & Flick, 2010; Gordon, Ludlum, & Hoey, 2008; Kahu, 2013). It calls attention to institutional strengths and deficiencies by rating students’ expectations in a certain area—say, the accessibility of faculty—along with their assessment of whether the institution has met their needs in this area. By taking steps to respond to concerns identified by NSSE results, institutions can improve the level of engagement of their students. According to Kuh (2003): The engagement premise is deceptively simple, even self-evident: The more students study a subject, the more they learn about it. Likewise, the more students practice and get feedback on their writing, analyzing, or problem solving, the more adept they become. The very act of being engaged also adds to the foundation of skills and dispositions that is essential to live a productive, satisfying life after college. (p. 25) NSSE depends on self-reporting, but its designers have attempted to address the problems of validity associated with this kind of instrument by paying attention to five factors that are known to increase reliability: the familiarity of the question, clarity, currency of information, merit, and maintaining the privacy of the respondents (Pascarella, 2001). Although NSSE draws attention to an institution’s weaknesses in meeting its students’ expectations, it does not analyze them adequately from the perspective of the lived experiences of students, and so it has limited usefulness as a means of defining solutions. To compensate for this deficiency, the concerned institution must also take a qualitative approach to data collection. Qualitative researchers “emphasize the value-laden nature of inquiry. They seek answers to questions that stress how social experience is created and given meaning. In contrast, quantitative studies emphasize the measurement and analysis of causal relationships between variables, not processes” (Denzin & Lincoln, 2000, p. 8). In spite of NSSE’s heavy emphasis on quantitative data, both the researcher and the administration of Ambrose University agree that it is still, according to the literature, the best measure of student engagement for Ambrose. Both parties also realize, however, that NSSE results need to be complemented by qualitative data. CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 62 The purpose of this study is to compare the 2011 NSSE results from Ambrose undergraduate theology students and the 2011 Theological School Survey of Student Engagement1 (TSSSE) results from Ambrose graduate students with the qualitative data gathered from two focus groups, one composed of undergraduate theology students and one composed of graduate theological students. The researcher has used both quantitative and qualitative approaches in the hope that the findings will suggest a course for future action on teaching, learning, and retention; but the researcher has chosen to give greater attention to the data supplied by the focus groups because they are based on the students’ own words and therefore provide a richer description of their experiences. The researcher hopes that the results of this study will make a modest contribution toward filling a gap in the literature about the engagement and learning of undergraduate and graduate students of theology. Context Ambrose University is a small liberal arts university and graduate theological school (fewer than 1,000 students) in Calgary, Alberta. Ambrose began to use NSSE in 2010 for undergraduate students and TSSSE in 2011 for graduate theological students. Early in each fall semester, the Student Development Department and the Office of Institutional Planning and Assessment report NSSE and TSSSE results to the faculty so that they may consider the implications for their teaching and for their students’ learning. As faculty members talked to the dean about the results of NSSE 2010, they were also concerned about student retention and about finding practical ways to improve their classroom practice. As a result of these concerns, the researcher began a conversation with the vice-president for student development to discuss further the results of the NSSE 2010 survey and how it might be used to improve learning and retention. The researcher and the vice-president for student development decided that, if they expected to supply the faculty with understandable and meaningful information, they would need to do a deeper analysis of students’ experiences than either the NSSE or the TSSSE statistics could provide. They therefore decided to follow up on the NSSE and TSSSE surveys the following spring with student focus groups in the fall of 2011. This initiative would also supply the vital information for the reaccreditation reviews of Ambrose’s undergraduate and graduate theological programs in the fall of 2012. Method and Participants This study used a mixed methodological approach known as a sequential study, in which “the researcher first conducts a qualitative phase of a study and then a quantitative phase, or vice versa. The two phases are separate” (Tashakkori & Teddlie, 1998, p. 18). As mentioned, this study began with the quantitative phase in the spring of 2011 and concluded with the qualitative phase in the fall of 2011. Again, the focus of this article is on the qualitative phase. NSSE and TSSSE Participants In all, 129 randomly selected Ambrose undergraduate students responded to an invitation to fill out the NSSE survey in the spring of 2011. The response rate was 45%, which CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 63 was higher than the Canadian institutional average (32%) and the Alberta average (33%). Of Ambrose respondents, 81% were full-time students (nine or more credits) and 19% part-time (fewer than nine credits). NSSE only reports the means scores for first-year students and senior students. However, institution-reported class levels are used. Table 1 compares selected characteristics of Ambrose respondents with those of other NSSE respondents in Canada, Alberta, and and the top 10% of all NSSE institutions (NSSE 2011). Table 1 Characteristics of NSSE Respondents, 2011: Gender, Age, Enrolment Status, and Residence Ambrose (%) Gender Female Male Age 24 or older Less than 24 Enrolment status Full-time (9 or more credits) Part-time Place of residence On campus Off campus Canada (%) Alberta (%) NSSE 2011 (%) 64 36 64 36 65 35 64 36 20 80 18 82 20 80 21 79 81 19 89 11 96 4 89 11 35 65 20 80 14 86 32 68 As Table 1 indicates, Ambrose respondents to the 2011 NSSE survey were similar in age and gender to NSSE respondents in Alberta, Canada, and the top 10% of all NSSE institutions. However, Ambrose had fewer full-time respondents and more respondents living on campus than the provincial, national, and the top 10% of all NSSE institution averages. For the 2011 TSSSE survey, 50 randomly selected graduate students from Ambrose responded, for a response rate of 40%. Data for comparison with other institutions were not available, but Table 2 provides internal comparisons of respondents’ characteristics. Focus Group Participants According to Krueger and Casey (2009), focus groups are most helpful when “the purpose is to uncover factors that influence opinion, behavior or motivation. Focus groups can provide insight into complicated topics when opinions or attitudes are conditional or when the area of concern relates to multifaceted behavior or motivation” (p. 19). The semi structured focus groups in this study met over several weeks in October and November 2011. They consisted of undergraduate and graduate theological students who had responded to an email invitation circulated by the Student Development Department. The undergraduate theology focus group consisted of three women and three men. Five were full-time students (nine or more credits), and one was part-time. Five were under 24 CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 64 years of age, and one was older than 24. Three students lived in residence and three lived off campus. The graduate theology focus group had seven students, four men and three women. Three were full-time students, and four were part-time. Five were 31 years old or older, and two were under 31. Only one of the seven lived on campus. Table 2 Characteristics of Ambrose Graduate Students Responding to TSSSE, 2011: Gender, Age, Enrolment Status, and Residence Year of Study First year (%) 26 Gender Female 39 Male 61 Age 31 or older 62 Less than 31 38 No response 0 Enrolment status Full-time (9 or 46 more credits) Part-time 54 Place of residence On campus 0 Off campus 100 Second year (%) 30 Third year (%) 24 Fourth year (%) 4 Other (%) Total (%) 27 73 33 67 0 100 62 38 36 64 87 0 13 42 42 16 100 0 0 100 0 0 72 20 8 67 42 0 12 44 33 58 100 88 56 0 100 8 92 0 100 0 100 2 98 16 The characteristics of the undergraduate theology focus group were very similar to those of the Ambrose students who had completed the NSSE survey, except in gender and place of residence. In the focus group, women were 50% of the participants; among the Ambrose NSSE participants, 64% were women. The researcher ensured that 50% of the undergraduate students in the focus group lived on campus, because the Ambrose NSSE comparison group included a relatively high proportion of on-campus students (35% vs. the Canadian average of 20% and the Alberta average of 14%). The characteristics of the graduate theology focus group resembled those of the Ambrose TSSSE respondents, except that 14% percent of the participants lived in residence whereas only 2% of the Ambrose TSSSE respondents did. The percentage of female participants was also higher in the focus group (43% vs. 36%). The researcher used only two focus groups because the students’ characteristics were very similar to those of the Ambrose NSSE and TSSSE respondents. According to Patton (2002), “The validity, meaningfulness, and insights generated from qualitative inquiry have more to do with the information-richness of the cases selected and the observational/analytical capabilities of the researcher than with the sample size (p. 245).” CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 65 The researcher met individually on campus with each focus group for about two hours. At the beginning of each meeting, the researcher had the participants sign a consent form that indicated the purpose of the study and how the data would be used. The researcher also received oral and written consent from the participants in both focus groups to videorecord the sessions and transcribe the discussion. The interview guides for the focus groups contained 11 questions each. The questions were mainly based on those areas of engagement in which Ambrose respondents to the NSSE and TSSSE surveys scored lower than their peers. The researcher decided to use these deficiencies as a point of departure because he wanted to increase his understanding of the factors that promote student engagement. Along with the questions, the researcher used probes to stimulate the participants to be more introspective and to share more deeply (Glesne & Peshkin, 1992). The researcher emailed the final transcripts to each student for member checking, so that they could personally verify the viewpoints they had expressed and the statements they had made and thereby capture more fully their unique experiences (Gall, Gall, & Borg, 2003). Some students responded with simple affirmation, others responded with corrections to the final transcript, and still others did not respond at all. For the analysis of the focus group data, the researcher used the constant comparative method (CCM) developed by Glaser and Strauss (Glaser & Strauss, 1967; Strauss, 1987; Glaser, 1992), along with a six-step process for using the CCM that was developed by Savin-Baden and Howell-Major (2013): • Identify categories in events and behaviour. • Name indicators in passage and code them (open coding). • Continually compare codes and passages with those already coded to find consistencies and differences. • Examine consistencies or patterns between codes to reveal categories. • Continue the process until the category “saturates,” and no new codes related to it are identified. • Determine which categories are the central focus (axial categories) and, thus, form the category. (p. 437) Charmaz (2005) states that the CCM of grounded theory involves • comparing different people; • comparing data from the same individuals with themselves at different points in time; • comparing incident with incident; • comparing data with category; and • comparing one category with another. (p. 515) The researcher modified the steps used by Savin-Baden and Howell-Major (especially bullet points two and six ) because both NSSE and TSSSE already provided categories, called benchmarks, of empirically tested sets of practices that cause students to experience satisfaction in the experiences of learning (Kuh, 2009; Lerer & Talley, 2010). In other words, the coding of student experiences from the focus group used the already predetermined categories of NSSE and TSSSE. The researcher made comparisons between the NSSE and TSSSE results and the data collected from the focus groups. For both the undergraduate and the graduate theology student experiences, the researcher compared CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 66 data with category and with individual responses, and also compared one category with another. Moreover, colleague checking was used for the codes for reliability and triangulation purposes. Reliability means consistency in the use of the data between colleagues (Richards, 2011). The method used was to compare two codings of the same data. Findings The researcher organized findings under the five NSSE/TSSSE benchmarks2 of effective educational practice: student-faculty interaction, supportive campus environment, enriching educational experiences, level of academic challenge, and active and collaborative learning. The NSSE and TSSSE scores in the Tables have been given by the mean and the demographic characteristics have been given in percentages. The mean is the weighted arithmetical average of the student-level benchmark scores. Effect size indicates the practical significance of the mean difference. It is calculated by dividing the mean difference by the pooled standard deviation. In practice, an effect size of .2 is often considered small, .5 moderate, and .8 large. A positive sign value indicates that the institution’s mean was greater than that of the comparison group, thus showing an affirmative result for the institution. A negative sign indicates the institution lags behind the comparison group, suggesting that the student behaviour or institutional practice represented by the item may warrant attention. The researcher will examine below in detail the scores that have negative signs. Undergraduate Theology Students After examining the NSSE benchmark comparisons, the researcher decided to examine closely two groups of survey findings with comparatively low scores: results for the enriching educational experiences and active and collaborative learning benchmarks. Ambrose lagged behind its peer institutions in these two benchmarks. This was then followed by consideration of the student-faculty interaction and supportive campus environment benchmarks. Since the undergraduate theology focus group respondents said little about the level of academic challenge, this category was not given extensive treatment. Enriching educational experiences (EEE). NSSE defined this benchmark as complementary learning opportunities that enhance academic programs: diverse experiences that teach students valuable things about themselves and others. Technology facilitates these experiences by enhancing collaboration between peers and instructors. Internships, community service, and senior capstone courses provide opportunities to integrate and apply knowledge. The survey questions in this category measured indicators such as these: • hours spent participating in school-sponsored activities (e.g., student life events, chapel services, spiritual formation groups, and mission endeavours); • practicum or field experience; • community service or volunteer work; • cross-cultural experience; • foreign-language coursework; • independent or directed study; • culminating senior experience (e.g., capstone courses, senior projects or theses, and comprehensive exams); CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 67 • serious conversations with students of a different race or ethnicity; • serious conversations with students who are very different from the respondent in terms of theological convictions, political opinions, or personal values; and • whether the campus environment encourages contact among students from different economic, social, religious, and racial or ethnic backgrounds. • using electronic medium (e.g., listserv, chat group, Internet, instant messaging, etc.) to discuss or complete an assignment. Table 3 Mean Comparisons for Enriching Educational Experiences Benchmark, NSSE 2011 First-year students Senior students Ambrose 20.8 36.7 Canada 24.1 34.3 Effect size –.27 .15 Alberta 24.1 Effect size NSSE 2011 Effect size –.27 27.0 –.46 35.4 .08 39.5 –.15 The NSSE 2011 comparison results are shown in Table 3. They suggest that Ambrose needed to focus on this particular benchmark because the effect size of -.46 for first-year students indicates that Ambrose’s overall EEE lagged behind that of its peer institutions. However, the undergraduate theology students in the focus group mildly disagreed with this finding because they had benefited from field experiences in their courses and from practicums and internships. One student put it this way: “The practical application in class where we discussed a topic and then went on a field trip really helped my learning.” In addition, many students in the focus group had taken travel-study courses. One such student found it “cool” to have received academic credit for a course that had enhanced learning. In summary, the students in the focus group believed that learning happens when they feel engaged in the learning process and that they had had this sense of engagement in many of their classes. The members of the focus group also indicated that Ambrose provided many opportunities for co curricular activities that enriched their educational experience. Many of the co curricular activities provided opportunities for conversations about life issues and personal values. However, the members of the group also mentioned that they had to take the initiative to participate in these activities if they expected to reap the benefits of involvement. As mentioned above, Ambrose had a higher percentage of students living in residence than the average Alberta, Canadian, or the top 10% of all NSSE institutions, so the researcher concluded that it was important to understand the perspective of off-campus students with respect to on-campus participation. The three residence and three off-campus students in the focus group all acknowledged that it was harder (but not impossible) for off-campus students to be involved in on-campus co curricular activities. One off-campus student summarized the sentiments of the three off-campus students as follows: “I need to work, and although I want to come I don’t have time. . . . Besides that, it’s an extra trip.” A number of students in the focus group, as stated earlier, indicated that they had no lack of co curricular opportunities on campus. However, they found it hard to balance their co curricular involvements with the demands of their academic program. Their un- CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 68 ease was made more intense by the implicit pressure they felt to be involved in as many co curricular activities as possible. The group suggested that the administration find ways of helping students to balance these competing expectations. Ambrose scored low on the EEE section that dealt with the faculty’s use of electronic media. Such media help students collaborate with one another and with the instructor. The focus group concurred with the Ambrose NSSE finding (mean score of 2.40), noting that some professors used Moodle (a learning management system) effectively, while others did not use it at all. With respect to the survey question on “culminating senior experience,” the two seniors in the focus group regarded it as being inapplicable to their situation, because their degrees were more professionally oriented and did not require any kind of senior project. Active and collaborative learning (ACL). This benchmark assumed that students learn more when they are intensely involved in their education and when they are asked to think about what they are learning in different settings. Collaborating with others in solving problems or mastering difficult material contributes to this sense of involvement and prepares students for the messy, unscripted problems they will encounter daily during and after their studies. The survey questions for the ACL benchmark included these components: • asking questions in class or contributing to class discussions; • making a class presentation; • working with other students on projects during class; • working with classmates outside of class to prepare class assignments; • participating in a community service project or ministry endeavour as part of a course or for academic credit; • discussing ideas from your readings or classes with others outside of class (students, family members, co-workers, etc.); and • hours spent working in a ministry-related position. The undergraduate theology students in the focus group said that a lot of their classes were discussion-based. One student put it this way: “I learn best in discussion, it hits it home for me a bit more. . . . It helps me make sense of my learning—it’s not just knowledge for exams.” Table 4 Mean Comparisons for Questioning and Discussion Component of Active and Collaborative Learning Benchmark, NSSE 2011 Asking questions in class or contributing to class discussions First-year students Senior students Ambrose Canada Effect size 2.45 2.32 .14 3.12 2.74 .41 Alberta 2.30 Effect size .17 NSSE 2011 2.72 Effect size –.31 2.72 .43 3.04 .09 CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 69 Ambrose had above-average scores in one ACL question: as shown in Table 4, Ambrose was doing better than other schools in Canada and Alberta with respect to asking questions in class and classroom discussion. Senior students did even better, exceeding even the the top 10% of all NSSE institution scores for this item. One senior student in the focus group had this to say about classroom discussions: “I actually find that I build friends from discussion with my classmates. I get to know people through what they say in class, and I find that becomes very real because we’re allowed to express how the topic relates to us.” First-year students tend not to speak up or to ask questions as much as senior students. All the mean scores indicate this. Part of the reason is that first-year classes tend to be bigger than upper-level classes. The first-year students in the focus group said that they did not speak up in class or ask questions because they were unfamiliar with the subject matter and felt it was best to listen. One senior student, reflecting on his first year, observed, “Sometimes professors rein you in because you’re giving an uneducated response. . . . You do need the foundations to get to the real stuff. . . . Although foundations are important they can be boring.” Table 5 Mean Comparisons for Group Work Component of Active and Collaborative Learning Benchmark, NSSE 2011s Ambrose Canada Working with other students on projects during class First-year students Senior students 1.9 2.13 Effect size –.22 1.7 2.23 –.53 Alberta 2.19 Effect size –.29 NSSE 2011 2.38 Effect size –.51 2.26 –.58 2.51 –.84 Ambrose students did not score as well on the question about working with classmates on projects during class. The researcher posed to the focus group the NSSE questions whose results are summarized in Table 5: “To what extent do you feel working with other students on group assignments inside and outside of the classroom would enhance your learning? What disadvantages do you see in doing group assignments together and how might they be addressed?” The students responded that they had concerns about group work, and these concerns (if they were shared by Ambrose respondents as a whole) may have accounted for Ambrose’s low NSSE scores. Many of the students in the focus group thought that their instructors did not take into account such factors as differences in learning styles or differences in personality when they assigned group projects. In any event, the group observed that a few people in the work group did the majority of the work. As one student put it, “Some in the group didn’t carry their weight for the class assignment.” The participants suggested that group work would become more meaningful if instructors were to provide several options for group projects, and if they were to take into account personality and learning styles in assigning groups. Student-faculty interaction (SFI). This benchmark had to do with how students learn first-hand by interacting with professors inside and outside the classroom and by observing how faculty-as-experts think about and solve practical problems. In the pro- CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 70 cess, professors may become role models, mentors, and guides for continuous, lifelong learning. The SFI items included these indicators: • discussing grades or assignments with an instructor; • talking about career plans with a faculty member or advisor; • discussing ideas from your readings or classes with faculty members outside of class; • working with faculty members on activities other than coursework (committees, orientation, student-life activities, etc.); • receiving prompt written or oral feedback from faculty on your academic performance; and • working on a research project with a faculty member outside of course or program requirements. On the question that dealt with student-faculty interaction outside of class time on ideas from readings or classes, Ambrose first-year students did better than their peers in schools in Canada and Alberta. Senior students also exceeded not only the mean Alberta and Canada scores but also the top 10% of all NSSE institution scores. The focus group agreed with this finding and attributed Ambrose’s good showing to small class size and the willingness of faculty to discuss further or clarify ideas presented in class or in the readings. All participants thought that their instructors went the extra mile in helping their students understand the subject matter. Some also mentioned that they had talked with faculty about vocational matters. Others confessed that they had left this resource untapped. For example, many of the first-year students in the focus group seemed to be unaware that their academic advisors could help them navigate course sequencing or gain a comprehensive understanding of the curriculum. These findings suggest communication between students, especially first-year students, and faculty needed to improve. Supportive campus environment (SCE). This benchmark assumed that students perform better and are more satisfied at universities that are committed to their success and cultivate positive working and social relations among different groups on campus. The SCE-related survey questions covered the following indicators: • campus environment that provides the support needed to help succeed academically; • campus environment that helps students cope with their non-academic responsibilities (work, family, etc.); • campus environment that provides the support needed to thrive socially; • quality of relationships with other students; • quality of relationships with faculty members; and • quality of relationships with administrative personnel and offices. All students in the focus group had attended the orientation at the beginning of their first fall semester. They described it as a welcoming and enjoyable experience because, thanks to the involvement of upper-year orientation leaders, it gave them the opportunity to meet other students. The orientation period also provided them with helpful suggestions for navigating university life. The students in the focus group also appreciated the variety of co curricular activities available to those willing to initiate the necessary contact. All spoke positively about residence assistants and floor meetings, both of which helped them meet social and spiritual needs. They also valued the high quality of their re- CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 71 lationship with faculty. Some said that they went to faculty not just for academic advising but also to seek advice on personal and spiritual matters. The members of the focus group also availed themselves of the counselling services offered by the Student Development Department. They also mentioned that they found it easy to get spiritual and personal help on campus. However, group members expressed a concern that commuter students needed to be “super-intentional” in developing relationships. Some mentioned that off-campus students found it easier to befriend other off-campus students than to develop relationships with residence students. According to one residence student, “We are with each other 24/7 and sometimes we forget to include others not in residence.” Graduate Theology Students TSSSE, the survey instrument that measures the engagement levels of graduate theology students, used the same indicators, definitions, and terms as NSSE. Four out of the five NSSE benchmarks have been defined above; the fifth, level of academic challenge, will be elaborated on below. Further, not only are the mean scores reported for first-year students and senior graduate theology students but also institution-reported class levels are used and reported because master degrees are typically two or three years in duration. Thus, all randomly selected or census-administered students are included in the analysis. Enriching educational experiences (EEE). The mean score for EEE in the TSSSE 2011 results for Ambrose first-year graduate theology students was 21.1; for peers at other institutions, it was 28.7. For senior students, the mean scores were 37.7 and 41.0 respectively. The TSSSE results indicate that some graduate theology students were missing the benefits of full participation in the life of the campus by choosing not to attend events or engage in co curricular opportunities. The focus group helped identify the primary motivation of those who participated, the barriers for those who did not, and the perceptions of the value of participation in activities outside of the classroom (the discussion in the next section below will elaborate further on this). Most of the graduate theology students in the focus group valued the spiritual formation component of their educational experience and believed that aspects of spiritual formation enriched their educational experience. Their primary motivation for involving themselves in spiritual formation was to meet others and to find and build community. A number of them mentioned that they or their fellow students have other communities for personal support, and that these other communities also enhanced their educational experience. One student put it this way: “My primary community that supports me is my church, but it’s nice that there are other students who can more specifically understand the journey of graduate education at Ambrose.” Supportive campus environment (SCE) and student-faculty interaction (SFI). The SCE mean scores for first-year students were 50.1 (for Ambrose) and 57.3 (for TSSSE 2011 respondents overall); for senior students they were 62.8 (Ambrose) and 51.7 (TSSSE). These scores indicate that, contrary to the experience of their TSSSE peers, Ambrose graduate theology students’ perception that their campus environment was supportive increased over the years. When asked “To what extent do you feel that you and your success are a high priority for the staff and faculty who work at Ambrose?” most CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 72 students in the focus group answered, “The faculty places a high priority on our success.” One student offered an explanation: “If my professor didn’t mention the student services in class, I wouldn’t have used it.” The service that the students in the focus group used the most was Learning and International Services. They believed that the director of this department was determined to help them succeed academically. They also mentioned that the Finance Office and the Registrar’s Office served them efficiently and professionally. Although graduate theology students are older than undergraduate students, they still need both community and physical space to build deep relationships with their fellow students. Most students in the group believed that Ambrose offered student services only to undergraduates. Some thought that they would like to use these services but were not sure whether they could legitimately do so. A number of them echoed the sentiment of the student who said, “I feel guilt in using student service because I might be wasting the person’s time or I’m taking time that an undergrad student could use.” Another student added, “I wanted to use the career services but I wasn’t sure it was for me to use.” Ambrose clearly needed to address this misperception. When asked about the physical layout of the campus, the members of the focus group responded that they liked the existing campus but that they initially found it hard to navigate. The maps they were provided with did not help because the terminology the maps used was incomprehensible. The 2011 SFI mean scores (Ambrose first-year students, 22.2; TSSSE first-year students, 21.7; Ambrose senior students, 32.0; TSSSE senior students, 27.1) showed that Ambrose was keeping up with or slightly exceeding its comparison group in this area. The focus group interviews indicated that graduate theology students sought support from two sources: other students and their faculty advisors. All the students in the focus group appreciated the presence of faculty at community events, their desire to build community, and their helpfulness in providing academic and personal guidance. A number of students in the focus group mentioned that they had discussed vocational issues with faculty. Level of academic challenge (LAC). Challenging intellectual and creative work is central to student learning and collegiate quality. Universities promote high levels of student achievement by emphasizing the importance of academic effort and setting high expectations for student performance. LAC-related survey questions covered such areas as these: • hours spent preparing for class (studying, reading, writing, doing homework or lab work, etc., related to academic program); • number of assigned textbooks, books, or book-length packs of course readings; • number of written papers or reports of 20 pages or more, between five and 19 pages, and fewer than five pages; • coursework that emphasizes analysis of the basic elements of an idea, experience or theory; • coursework that emphasizes synthesis and organizing of ideas, information, or experiences into new, more complex interpretations and relationships; • coursework that emphasizes making of judgments about the value of information, arguments, or methods; • coursework that emphasizes applying theories or concepts to practical problems or in new situations; CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 73 • working harder than you thought you could to meet an instructor’s standards or expectations; and • campus environment that emphasizes spending significant amount of time studying and on academic work. Table 6 Mean Comparisons for Level of Academic Challenge Benchmark, TSSSE 2011 Ambrose TSSSE 2011 First-year students 73.0 55.2 Second-year students 79.8 55.4 Senior students 75.4 54.2 As Table 6 indicates, the mean score of Ambrose first-year graduate theology students on this part of the survey was 73.0; the mean score for TSSSE respondents as a whole was 55.2. The first-year students in the focus group indicated that they did spend considerable time preparing for class and doing homework and assignments. Full-time students especially seemed to spend a significant amount of time studying; as mentioned above, this diligence might have hindered their ability to participate in social events and other community building activities. Table 7 Mean Comparisons for Active and Collaborative Learning Benchmark, TSSSE 2011 Ambrose TSSSE 2011 First-year students 32.5 33.2 Second-year students 37.9 38.4 Senior students 40.5 39.0 Active and collaborative learning (ACL). Table 7 shows the overall mean comparison with respect to ACL: Ambrose’s mean scores are similar to those of TSSSE respondents as a whole. Students in the focus group did not discuss to any significant degree this aspect of their student experience. However, they did mention collaborative learning activities that took place outside of the classroom in such places as the graduate lounge and the group study rooms in the library. As mentioned above, they did not believe that the graduate meeting area provided enough privacy for small-group conversation; and even though they enjoyed conversing in the group study rooms, they thought they were monopolizing the space. Both the overall TSSSE results and the focus group interviews indicated that graduate theology students believed that working with other students inside and outside of the classroom enhances learning. Two items from the TSSSE ACL-related questions made the point: CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 74 • working with other students on project during class; and • working with classmates outside of class to prepare class assignments. The answer options were sometimes, often, and very often. According to the raw data, Ambrose first-year graduate students responded that they worked with others during class 61.6% of the time; senior students said they did so 75% of the time. Ambrose first-year graduate students said they worked with others outside of class 53.9% of the time; senior students said they did this 66.7% of the time. Most members of the focus group, when asked “Where have some of your positive educational engagement experiences happened inside and outside of classes?” responded that working with others helped in their learning. One senior student said: “One or two of the professors have group assignments and that helps me, especially when I don’t understand something.” Discussion Undergraduate Theology Students Ambrose’s mean overall comparison score for EEE falls below the provincial and national benchmarks for first-year students and slightly above the provincial and national scores for senior students. The student focus groups provided an explanation for this finding. Indeed, they slightly disagreed with one aspect of it. They indicated that professors in their classroom teaching provided a lot of EEE and that students had ample opportunity to connect with one another by way of the co curricular activities on campus. However, they also identified barriers to EEE: namely, travel and lack of time. Both residence and off-campus students mentioned lack of time, because of either part-time work or academic studies, as the main deterrent to participation in co curricular activities. Many respondents voiced a concern for off-campus students who had to travel back to campus for events or who had to stay longer on campus to attend them. The matter of commuter students’ lack of connection with other students also came up in the focus groups’ discussion of SCE. This sense of disconnection is commonly felt by commuter students at all institutions, and it has received considerable attention in the literature (Alfano & Eduljee, 2013; Astin, 1984; Bradley, 2006; Kuh, Gonyea, & Palmer, 2001; Newbold, Mehta, & Forbus, 2011). The ACL narratives from the student focus group confirmed the NSSE finding that Ambrose was on par with or lagging a bit behind its provincial and national counterparts in this area. Ambrose’s chief weakness in ACL had to do with group work. The student focus group feedback indicated at least one reason for this: students did not like group work because there is no way to measure the relative contribution of each group member, and because professors did not know how to use group work effectively. As both the NSSE results and the focus group discussions indicated, Ambrose actually did well in other aspects of ACL, such as asking questions and classroom discussion. These data aligned well with the findings of teaching and learning and student engagement literature, which indicate that instructors can foster engagement in a variety of ways, among them discussion in classes, knowing students’ names, responsiveness to students’ questions, urging students to seek help, and assigning effective aids to learning (Ambrose et.al, 2010; Barkley, 2010; Miller, 2008). CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 75 Students in the focus group spoke very little about the level of academic challenge, even though some of the focus group questions targeted this category. The researcher found this rather disconcerting, as there was little conversation about the teaching and learning moments in the classroom, course objectives, or the nature of assignments. The researcher discovered a partial explanation for this deficiency on reviewing the overall NSSE 2010 results for LAC (Table 8). The scores for Ambrose met or exceeded those of institutions in Canada in 2010, but in 2011 the Ambrose first-year scores fell slightly below the national and provincial averages. The mean comparison scores for 2011 (Tables 9 and 10) indicate more precisely the single items in which Ambrose needed to increase the level of academic challenge. With the exception of the results for campus environment, Ambrose’s single-item scores followed the provincial and national trends in that they increased from the first year to the senior year. Ambrose’s statistically significant areas of weakness vis-à-vis national and provincial results were the following items: • for first-year students: analyzing the basic elements of an idea, experience, or theory, such as examining a particular case or situation in depth and considering its components; • for both first-year and senior students: number of written papers or reports of 20 pages or more; and • for senior students: number of written papers or reports of between five and 19 pages. Table 8 Mean Comparisons for NSSE Level of Academic Challenge (LAC), 2010-2011 First-Year 51.1 50.9 .01 57.2 -.47 53.9 -.20 48.2 51.2 -.23 50.6 -.19 Senior 56.3 56.6 -.03 60.4 -.30 57.5 -.09 55.7 55.5 .02 54.3 .10 Effect Size NSEE 2011 Effect Size Alberta Effect Size Canada Ambrose Effect Size NSSE 2010 2011 Effect Size Alberta Effect Size Canada Ambrose 2010 53.2 -.38 57.1 -.10 Table 9 Mean Comparisons for Analyzing an Idea, Experience, or Theory Component of Level of Academic Challenge Benchmark, NSSE 2011 Ambrose Effect Alberta Effect NSSE Effect size size 2011 size Analyzing the basic elements of an idea, experience, or theory, such as examining a particular case or situation in depth and considering its components First-year students 2.87 3.11 –.31 3.14 –.35 3.16 –.38 Senior students Canada 3.20 3.25 –.07 3.25 CJHE / RCES Volume 45, No. 2, 2015 –.08 3.30 –.14 Understanding the Student Experience / A. C. K. Wong 76 Table 10 Mean Comparisons for Written Assignments Components of Level of Academic Challenge Benchmark, NSSE 2011 Ambrose Canada Effect Alberta size Number of written papers or reports of 20 pages or more Effect size NSSE 2011 Effect size First-year students 1.11 1.41 –.40 1.29 –.28 1.30 –.27 Senior students 1.36 1.81 –.53 1.76 –.48 1.65 –.36 Number of written papers or reports between 5 and 19 pages First-year students 2.34 2.26 .10 2.15 .25 2.21 –.16 Senior students 3.15 2.70 .48 2.58 .63 2.56 –.62 From 2010 to 2011, Ambrose’s mean single-item scores for these three components fell slightly. For example, the first-year mean score for analyzing the basic elements of an idea, experience, or theory slipped from 2.98 to 2.87 (Table 11). Over the same period, senior students’ scores for this item improved slightly, from 3.19 to 3.20. The faculty of Ambrose needed to devise a long-term strategy for increasing the level of academic challenge. Table 11 Mean Comparisons for Analyzing an Idea, Experience, or Theory Component of Level of Academic Challenge Benchmark, NSSE 2010 and NSSE 2011 2010 2011 Analyzing the basic elements of an idea, experience, or theory, such as examining a particular case or situation in depth and considering its components First-year students 2.98 2.87 Senior students 3.19 3.20 Graduate Theology Students Ambrose’s TSSSE 2011 results seemed to be consistent with the feedback from the graduate student focus groups. However, a few inconsistencies occurred with respect to EEE. Enriching educational experiences really depends on the degree to which students engage with one another. Engagement, in turn, has a direct correlation with how often students are on campus. Obviously, the full-time students felt more engaged. Like their undergraduate counterparts, graduate theology students attributed their lack of time spent on campus to the demands of coursework. “Work, lots of hard studying!” was how one student summed up the Ambrose experience. However, graduate theology students also mentioned other time-limiting factors, such as full-time employment and family. This is understandable because most graduate theology students were studying part-time CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 77 and taking fewer than nine credits. The 2011 enrolment figures from the Registrar’s Office confirmed this: 88 (69%) were studying part-time, and 40 (31%) were full-time. Indeed, most graduate theology students in North America are studying part-time (Aleshire, 2008; Association of Theological Schools, 2011–2012). Another inconsistency with respect to EEE centred on space and communication. All the students in the focus group mentioned two primary hindrances to building community, the first being the lack of common space for in-depth conversation. Graduate theology students had only a couple of tables in a hallway as their meeting space, which led one student to complain that “the space we have now feels like an add-on, it’s not private enough.” Another half-jokingly remarked, “There are two places you can talk theology, at the bar and here at Ambrose. You can’t really talk theology in the church. . . . Here on campus we’re allowed to ask questions and ponder and don’t feel like we have to have all the answers. Having a good designated space would enhance and welcome conversation from a diverse group of people in age, culture, and denominational backgrounds.” The communication of social events is another aspect of enhancing social life that was being done poorly. One student had this to say: “You have to look for things, it’s not communicated; or if it is, it needs to be communicated multiple times!” Another student added, “I found out about the event only after the fact.” Most members of the focus group agreed that the publicizing of community-building events needed to be improved. Recommendations for Moving Forward The following recommendations are based on the narratives from the two focus groups: • That a collegium for undergraduate commuter students be established to facilitate the following educational outcomes: (1) enhanced community within the student body and between students and faculty; (2) increased opportunities for individual and group study; and (3) social integration via the sharing of meals. • That the director of commuter life oversee the development of a commuter life program (to facilitate further the development of community). • That a dedicated space for graduate theology students be found and developed appropriately to enhance (1) community life within the student body and between students and faculty; (2) opportunities for individual and group study and theological conversations; and (3) social integration through the sharing of meals. • That both the Student Development Department and the faculty find ways to assist students in balancing academic and cocurricular expectations. • That the faculty, through the Teaching and Learning Committee, consider ways to enhance active learning in the classroom and group learning. • That the faculty devise strategies to increase the level of academic challenge. Limitations The findings and recommendations of this preliminary study are limited to a religiously based university and graduate theology school in western Canada. The findings and recommendations would likely be different at other institutions of higher learning in other locations and in settings of greater diversity. If other aspects of engagement were measured by the NSSE and TSSSE surveys, and if other aspects of the focus group in- CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 78 terviews had been emphasized, the results might have differed or changed. The small number of focus groups was also a limiting factor; a greater number of groups might have affected the findings. Moreover, because this study is qualitative in nature and not quantitative, it would require considerable refinement to make its findings generalizable. Finally, this study is exploratory in nature, and so it should not be used on its own as a means of explaining the intricacies and complexities of student engagement. Conclusion Ambrose has used the NSSE and TSSSE surveys as means of improving student learning inside and outside the classroom. Kuh’s (2003) comment about the use of NSSE accords well with this intention: “Although NSSE does not directly assess learning outcomes, the results from the survey point to areas where colleges are performing well in enhancing learning, as well as to aspects of the undergraduate experience that could be improved” (p. 26). Kuh’s view applies equally to the graduate experience as well. For Ambrose, the NSSE benchmark of effective educational practices provided a point of departure for conversations about learning, student engagement, and institutional improvement. As Morehouse (2012) aptly puts it, “Surveys provide mathematical descriptions of trends, attitudes, and opinions of a group of participants” (p. 78). The present study has demonstrated the need for an additional analysis of student engagement that would complement NSSE and TSSSE by providing qualitative data to supplement the quantitative data provided by the two survey instruments. Whereas previous studies have used qualitative data simply to triangulate the quantitative data to affirm or disprove a particular NSSE benchmark (Cohen, Manion, & Morrison, 2000), the qualitative data from the focus groups in the present study provided—in the students’ own words—insight into the “why” of their lived educational experiences. Notes 1 2 The Theological School Survey of Student Engagement is an adaptation, for graduate students of theology, of the National Survey of Student Engagement and the Law School Survey of Student Engagement. For validity and reliability issues see Fuller (2009). NSSE has recently replaced the term “benchmarks” with “engagement indicators” and revised the categories. References Aleshire, D. (2008). Earthen vessels: Hopeful reflections on the work and future of theological schools. Grand Rapids, MI: Eerdmans. Alfano, H., & Eduljee, N. (2013). Differences in work, levels of involvement, and academic performance between residential and commuter students. College Student Journal, 47 (2), 334–343. Ambrose, S., Bridges, M., DiPietro, M., Lovett, M., & Norman, M. (2010). How learning works: Seven research-based principles for smart teaching. San Francisco, CA: Jossey-Bass. CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 79 Association of Theological Schools. (2011–2012). Annual Data Table. Retrieved from http://www.docs.ats.edu/uploads/resources/institutional-data/annual-data-tables/20112012-annual-data-tables.pdf Astin, A. (1984). Student involvement: A developmental theory of higher education. Journal of College Student Development, 40 (5), 518–529. Astin, A., & Lee, J. (2003). How risky are one-shot cross-sectional assessments of undergraduate students? Research in Higher Education, 44 (6), 657–672. Axelson, R., & Flick, A. (2010). Defining student engagement. Change: The Magazine of Higher Education, 43(1), 38–43. Barkley, E. (2010). Student engagement techniques: A handbook for college faculty. San Francisco, CA: Jossey-Bass. Bradley, G. (2006). Work participation and academic performance: A test of alternative propositions. Journal of Education and Work, 19 (5), 481–501. Charmaz, K. (2005). Grounded theory: Method for the 21st century. London, UK: Sage. Chickering, A., & Gamson, Z. (1999). Development and adaptation of the seven principles for good practice in undergraduate education. New Direction for Teaching and Learning, 80 (Winter), 75–81. Cohen, L., Manion, L., & Morrison, K. (2000). Research methods in education (5th ed.). NewYork, NY: Routledge/Falmer. Denzin, N., & Lincoln, Y. (Eds.). (2000). Handbook of qualitative research (2nd ed.). Thousand Oaks, CA: Sage Publications. Fuller, T. (2009). Student engagement in graduate theological education (Unpublished doctoral dissertation). University of Alabama, Tuscaloosa, AL. Gall, M., Gall, J., & Borg, W. (2003). Educational research: An introduction (7th ed.). White Plains, NY: Longman. Glaser, B. (1992). Emergence vs. forcing: Basics of grounded theory analysis. Mill Valley, CA: Sociology Press. Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago, IL: Aldine. Glesne, C., & Peshkin, A. (1992). Becoming qualitative researchers: An introduction. White Plains, NY: Longman. Gordon, J., Ludlum, J., & Hoey, J. (2008). Validating NSSE against student outcome: Are they related? Research in Higher Education, 49 (1), 19–39. Kahu, E. (2013). Framing student engagement in higher education. Studies in Higher Education, 38, 758–773. Krueger, R., & Casey, M. (2009). Focus groups: A practical guide for applied research (4th ed.). Los Angeles, CA: Sage Publications. Kuh, G. (2003). What we’re learning about student engagement from NSSE: Benchmark for effective educational practice. Change: The Magazine of Higher Education, 35 (2), 24–32. CJHE / RCES Volume 45, No. 2, 2015 Understanding the Student Experience / A. C. K. Wong 80 Kuh, G. (2009). The National Survey of Student Engagement: Conceptual and empirical foundations. New Directions for Institutional Research, 141 (Spring), 5–20. Kuh, G., Gonyea, R., & Palmer, M. (2001). The disengaged commuter student: Fact or fiction? Commuter Perspectives, 27 (1), 2–5. Lerer, N., & Talley, K. (2010). National Survey of Student Engagement’s (NSSE) benchmarks—one size fits all? On the Horizon, 18 (4), 355–363. Miller, R. (2008). The role of instructional factors in student engagement. Retrieved from http://www.nsse.iub.edu/institute/workshops/2008/suny_b/docs/Miller_Butler_ Additional_Materials.pdf Morehouse, R. (2012). Beginning interpretative inquiry: A step-by-step approach to research and evaluation. New York, NY: Routledge. Newbold, J., Mehta, S., & Forbus, P. (2011). Commuter students: Involvement and identification with an institution of higher education. Academy of Educational Leadership Journal, 15 (2), 141–153. Pascarella, E. (2001). Using student self-reported gains to estimate college impact: A cautionary tale. Journal of College Student Development, 42 (5), 488–492. Patton, M. (2002). Qualitative evaluation and research methods (3rd ed.). Thousand Oaks, CA: Sage. Richards, L. (2011). Handling qualitative data: A practical guide (2nd ed.). Thousand Oaks, CA: Sage. Savin-Baden, M., & Howell-Major, C. (2013). Qualitative research: The essential guide to theory and practice. New York, NY: Routledge. Scott, D., & Scott, S. (2011). Effective university teaching and learning. Retrieved from http://www.ucalgary.ca/provost/files/provost/Scott&Scott EffectiveUniversityTeachingandLearning.pdf Seidman, A. (2005). College student retention: Formula for student success. Westport, CT: ACE/Praeger. Strauss, A. (1987). Qualitative analysis for social scientists. Cambridge, UK: Cambridge University Press. Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and approaches. Thousand Oaks, CA: Sage. Contact Information Arch Chee Keen Wong Faculty of Theology Ambrose University [email protected] Arch Chee Keen Wong is professor of practical theology and former dean, Faculty of Theology, at Ambrose University. His research interests include reflective practice, the relationship between theory and practice, qualitative research methodologies and epistemologies for practical theology, the scholarship of teaching and learning, and curriculum. CJHE / RCES Volume 45, No. 2, 2015