Don’t Tell It Like It Is / I. Iqbal 108 CSSHE SCÉES Canadian Journal of Higher Education Revue canadienne d’enseignement supérieur Volume 44, No. 1, 2014, pages 108 - 124 Don’t tell it like it is: Preserving collegiality in the summative peer review of teaching Isabeau Iqbal The University of British Columbia Abstract While much literature has considered feedback and professional growth in formative peer reviews of teaching, there has been little empirical research conducted on these issues in the context of summative peer reviews. This article explores faculty members’ perceptions of feedback practices in the summative peer review of teaching and reports on their understandings of why constructive feedback is typically non-existent or unspecific in summative reviews. Drawing from interview data with 30 tenure-track professors in a research-intensive Canadian university, the findings indicated that reviewers rarely gave feedback to the candidates, and when they did, comments were typically vague and/or focused on the positive. Feedback, therefore, did not contribute to professional growth in teaching. Faculty members suggested that feedback was limited because of the following: the high-stakes nature of tenure, the demands for research productivity, lack of pedagogical expertise among academics, non-existent criteria for evaluating teaching, and the artificiality of peer reviews. In this article I argue that when it comes to summative reviews, elements of academic culture, especially the value placed on collegiality, shape feedback practices in important ways. Résumé Si de nombreux écrits ont été consacrés aux évaluations et au perfectionnement professionnel des professeurs dans les rapports formatifs des pairs, peu de recherches empiriques portent sur ces questions dans leurs évaluations sommatives. Cet article explore les perceptions des professeurs d’université quant aux pratiques d’évaluation adoptées dans les évaluations sommatives CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 109 de l’enseignement par les pairs, et expose leur compréhension des raisons expliquant l’absence d’évaluations constructives, ou leur imprécision, dans ces mêmes évaluations sommatives. À partir de 30 entrevues avec des professeurs occupant un poste menant à la permanence dans une université canadienne axée sur la recherche, les résultats ont indiqué que les examinateurs donnaient rarement de commentaires aux candidats et que, quand ils le faisaient, ces commentaires étaient généralement vagues ou ne soulignaient que les aspects positifs. En conséquence, ces rapports n’ont pas contribué au progrès professionnel du travail de l’enseignant. Des professeurs ont avancé les raisons suivantes pour expliquer les limites de la rétroaction : l’importance de l’enjeu de la permanence et des publications, le manque de compétence pédagogique chez les universitaires, l’absence de critères d’évaluation de l’enseignement et le caractère artificiel des évaluations par les pairs. Dans le présent article, je soutiens que, lorsqu’il s’agit d’évaluations sommatives, des traits propres à la culture universitaire, en particulier la valeur accordée à la collégialité, influencent grandement les pratiques d’évaluation par les pairs. Introduction Within Canadian universities, summative peer reviews of teaching are increasingly being used to inform personnel decisions such as tenure, promotion, and reappointment (Gravestock, 2011; Ackerman, Gross, & Vigneron, 2009). The peer review process often consists of a tenured professor (or more than one) who conducts at least one classroom observation of teaching of a departmental peer. Summative reviews may also include an assessment of a candidate’s teaching philosophy statement, course materials, or teaching portfolio. Based on what they observe and read, the reviewers write a report that they submit to the chair of the peer review committee and/or the department head. Tenured departmental colleagues with input into decisions about tenure, promotion, and reappointment have access to information about the peer review outcomes and use it to rank or compare the candidate. In this way, information from summative reviews is open to public inspection (Chism, 2007a). The increased use of peer reviews has, in large part, come about in response to criticisms that student evaluations of teaching are not a reliable or valid means of measuring an instructor’s effectiveness at facilitating student learning (Ackerman et al., 2009; Chism, 2007a). Since it is almost universally accepted that multiple sources of data strengthen faculty evaluation (Arreola, 2007; Benton & Cashin, 2012), considering data from peer reviews together with information from student evaluations is deemed preferable to using data from a single source. Summative peer reviews can also enhance the evaluation process by virtue of the fact that, as compared to students, peers are more qualified to evaluate certain aspects of teaching, such as course goals and content, instructional materials and methods, appropriateness of assessment practices, and professional/ethical behaviour (Chism, 2007a; Courneya, Pratt, & Collins, 2008). Though summative peer reviews may strengthen the validity of the evaluation process, they do not typically contribute to an instructor’s professional growth in teaching (Byrne, Brown, & Challen, 2010; Kell & Annetts, 2009; Peel, 2005). In this regard, they have been unfavourably compared to formative peer reviews, a process that is praised for CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 110 enhancing teaching and for promoting collegiality (Atkinson & Bolt, 2010; Bell & Cooper, 2013). In a typical formative peer review, an instructor voluntarily initiates a review with a colleague of his choice and also determines the objectives of that review (e.g., the instructor may want ideas on how to augment classroom participation or on how to strengthen his assessment techniques). The reviewer shares her observations and reflections with the person who initiated the request, and when done well, formative peer review fosters self-reflection and discussion of teaching, and generates critical insights into teaching for both parties (Byrne et al., 2010; Kell & Annetts, 2009; Shortland, 2010). In a formative review, information is kept confidential between the reviewer and instructor (unless the instructor chooses to share it). Scholars of peer review have put forth at least four reasons why summative peer reviews offer few benefits in terms of professional growth in teaching. First, they point out that meaningful learning and reflective practice occur most often when academics engage in pedagogical practice for its own sake, and when it is not based on external demand (Atkinson & Bolt, 2010; Byrne et al., 2010). Summative reviews, which are high-stakes evaluations imposed upon faculty members, fall into this latter category and have been linked to the accountability movement in higher education, whereby faculty are increasingly required to measure and quantify their activities in the guise of improving quality and efficiency (Shanahan, 2009). Second, previous research also suggests that in researchintensive universities where faculty members are rewarded for their research productivity above their teaching (Chalmers, 2011), many professors think summative peer review is a “time sink” (Chism, 2007b, p. 6) or merely another task to cross off their to-do list. As such they may be disinclined to spend additional time on dialoguing with their peers about the review. A third reason why peer reviews may not foster professional growth in teaching is because giving quality feedback on teaching is a demanding skill for which few academics have training (Yiend, Weller, & Kinchin, 2012). Finally, it has been suggested that academics are uncomfortable with giving feedback to peers on their teaching, especially when it is of a critical nature (Cosh, 1998; Hammersley-Fletcher & Orsmond, 2004; Kemp & Gosling, 2000). If the act of providing feedback in a constructive manner is perceived as being too risky to collegiality, faculty members may opt to provide only positive comments. When this is the case, the process becomes “a form of mutual back patting, meaningless for genuine staff development” (Cosh, 1998, p. 172). Thus, in spite of having the potential to foster professional growth in teaching, summative peer reviews rarely meet this goal (Chism, 2007a; Iqbal, 2012). This study explores faculty members’ perceptions of feedback practices in summative peer reviews of teaching and reports on the professors’ understandings of why constructive feedback was typically non-existent or unspecific in their experiences of summative reviews. As compared to the voluminous research done on feedback in formative reviews, this investigation probes an underexplored area of research and aims to examine the role of academic values and norms in shaping feedback practices when reviews of teaching are conducted for career advancement purposes. In this paper I argue that the value placed upon collegiality by faculty members limits the extent to which professors share constructive feedback with one another; as a result the developmental potential of summative peer reviews is constrained. This information will be of use to those who seek to strengthen and advance the use of summative peer review as a way of fostering professional growth in teaching. CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 111 Approach to the Study I approached this study as an educational developer who, by virtue of working at a teaching and learning centre, has been involved in various formative peer review of teaching initiatives. I had no personal or professional experience with summative peer review as I was not a faculty member and had never had my teaching reviewed for summative purposes. Curious about faculty members’ perceptions of the link between summative peer review and professional growth in teaching, and hopeful that the process might offer some of the benefits of formative peer review, I undertook this research project. I took a qualitative approach to this embedded case study (Yin, 2009). As an interpretive researcher, I focused on how participants made sense of their experiences and combined these interpretations with my own to better comprehend the issues (Schram, 2003). The lens I used was that of academic culture. An academic-culture perspective considers the role of values, beliefs, and assumptions in higher education; it examines how these are displayed in attitudes, behavioural norms, rituals, and other symbolic activities (Alvesson & Sveningsson, 2008; Bunch, 2007; Trowler, 2008) and how they are maintained and passed along (O’Meara, 2011). After I obtained approval from the university’s Institutional Review Board for human subject research, I recruited participants via email from departments within the Faculty of Science (3 departments, 15 participants) and Faculty of Arts (3 departments, 15 participants). Participants were 30 tenure-track professors working at the University of Western Canada (UWC [pseudonym]), a large, research-intensive Canadian university. All the participants, with one exception, had been peer reviewed and/or had peer reviewed a colleague for summative purposes at UWC (specifically, to inform tenure decisions). Eight had been reviewers and candidates under review at different times; 11 had been peer reviewed but had not yet conducted a summative peer review; and 10 had been in the role of reviewer but had never had their teaching evaluated through the summative peer review. All the reviewers were senior in rank to the candidates. Participants were purposefully selected to get a range in gender and rank; they were men (17) and women (13) with the titles of assistant, associate, and full professor (8, 9, 13 respectively). I conducted one semistructured interview with each faculty member over a single semester, based on Mayan’s (2009) opinion that the use of semistructured interviews is appropriate when the researcher has a sufficiently good sense of the phenomenon to develop interview questions but not a strong enough idea to predict the responses. The interviews, which lasted approximately one hour, included questions about the participants’ experiences of summative peer review and how they described the purposes of summative peer review. Since my main objective was to hear how faculty members made sense of their summative peer review experiences, I chose interviews as my sole source of data. Furthermore, at the time of the study, only one of the six departments had a departmental policy on peer review, and only two had a standard form that faculty members consistently used when conducting a classroom observation of teaching—thus, there was little documentation to study and I had no access to the summative peer review of teaching reports. Twenty-eight of the 30 interviews were audio-recorded and then transcribed wordfor-word; during the two interviews that were not recorded, I took detailed notes. With the help of Atlas.ti, I reviewed each interview line by line and conducted a thematic analy- CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 112 sis of the data. I simultaneously coded and categorized the data, exploring how codes and categories related to one another, while remaining attuned to patterns in the data and negative cases (Creswell, 2012). Findings My analysis identified that, among the thirty participants I interviewed, only one spoke about receiving extensive and valuable feedback from his peers in a post-review conversation. This faculty member described how he and his reviewer had engaged in a lengthy and “really good conversation about teaching” and he said the experience was “great” in terms of his own instructional growth (interview 20, full professor). Aside from this individual, nobody else talked about receiving or giving rich verbal or written feedback from or to their peers. Thus, though the findings below may appear biased (i.e., they report on negative aspects of feedback in summative peer reviews), they correctly represent the dominant themes from the data. Feedback Is Not Shared Reviewers and candidates reported that in the summative peer review of teaching, feedback was hardly ever shared. To the regret of many candidates, they and their reviewers had never debriefed together after a classroom observation of teaching. I was disappointed not to have gotten more feedback because it could have been a mentoring opportunity as well. The two people who are doing my peer evaluation are both good teachers and it would have been interesting to have gotten more feedback from them. (interview 11, assistant professor) I’m always looking for ways to improve my teaching in ways that are not superintensive so I followed up with the faculty member who did the review. There was a bit of back and forth email where he said, “Yeah, sure, at some point we’ll do this [discuss the peer review of teaching].” I followed up again but I felt like it was a bit of an imposition because he never got back to me again. So I didn’t push it. (interview 19, associate professor) Not only was verbal feedback rare, but in most cases, the candidates did not see the peer review report. “In our department, the reviewers produce a report, which the candidate does not see” confirmed an associate professor (interview 29). Furthermore, in annual meetings with the department head, the head infrequently or scarcely mentioned the outcomes of the peer review process. I don’t have anything that ever came out of this [peer review]. In the annual meeting with my head, I asked him [about it] and he said “Oh yeah it was fine.” … He didn’t have it in front of him. It was not part of that conversation. And I did not receive a copy of it afterwards. (interview 11, assistant professor) They [the reviewers] gave a formal written report to the head of department and then he read that report to me [at our annual meeting] … I was only allowed to look at it and didn’t actually get to keep it, which is interesting. (interview 12, associate professor) CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 113 The majority of participants said the annual meetings focused principally on their research progress. Teaching was rarely discussed with their head and faculty members interpreted that to mean that teaching, as compared to research, was of less relevance to discussions about career progress. Those who had been reviewed stated that because reviewers and candidates rarely discussed the peer review outcomes, the process did not contribute to their professional growth in teaching. “Largely useless” Feedback Occasionally, those who had been peer reviewed did receive feedback from their colleagues. Most of them described it as being of little value to their growth as instructors because the feedback was unspecific and not constructive in nature: “So in fact I’ve not gotten any feedback yet on mine other than any other sort of three-sentence conversation, ‘Oh that was a really interesting class; I learned something’” (interview 15, assistant professor). Another participant, reflecting on the vague feedback experience recounted: “When I met with my reviewer, I was told, ‘There are no concerns; things went fine; you managed to make a very interesting topic really engaging.’” (interview 14, assistant professor). She, and others with similar experiences, found that vague feedback that focused on the positive and offered no constructive suggestions for improvement was unsatisfying. With respect to the written reports, the few people I interviewed who had been privy to reading them said that similar to the verbal comments, written comments were also typically positive. I mean if anything there might be a tendency to ignore problems or just focus on the good things … I mean I’m sure there’s lots more they could have said by way of criticism that they didn’t. (interview 1, assistant professor) However, even though most participants derided vague feedback or feedback that was not sufficiently constructive, a few noted that these phrases at least assured them that the reviewer positively perceived what happened. Participants were also critical of feedback that focused only on the “mechanics of teaching” (interview 28, associate professor), rather than issues of more substance, such as student learning. Both reviewees and reviewers said that feedback was often confined to issues that included whether the students were paying attention (e.g., were they checking emails? falling asleep?), whether the instructor was speaking loudly enough, or if she or he had distracting habits such as fiddling with keys. When I met with my reviewer, he suggested I write more neatly and face the students more often. His comments just focused on the mechanics and were not very useful … In my opinion, all these conversations are skirting teaching and are not getting at the important issue of students and their learning. (interview 28, associate professor) The participants in this study were keen to learn about their teaching. They considered feedback that was unspecific, not sufficiently concerned with student learning, or primarily positive, as “largely useless” (interview 16, assistant professor). They believed CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 114 that constructive feedback that highlights what they are doing well, makes suggestions for improvement, and is provided by a knowledgeable and experienced educator could contribute to their growth. Reviewers’ observations, they said, had the potential to prompt a teacher to examine what had happened during the class and consider how to apply the feedback to other classes and teaching situations: “I look on the peer review as being an opportunity for an outsider to suggest things that I might change to make my lectures better” (interview 17, associate professor). Feedback could also help faculty members detect issues they may not be cognizant of and help identify areas for improvement: “Peer evaluation could be potentially helpful because they [reviewers] see things that are in your blind spots” (interview 14, assistant professor). Thus, most participants, recognizing that feedback encourages reflection on teaching and their growth as instructors, expressed disappointment with the existing feedback practices because they felt they were missing out on an opportunity to learn about their teaching. Participants’ Perspectives on Why Feedback Is Limited or Non-Existent Participants suggested several reasons why the quality of feedback from their reviewers, including the department head, was limited or non-existent. As it concerned the written reports, the majority thought that the inclusion of constructive feedback might impede a colleague’s progress to tenure and therefore chose to leave it out. Faculty members who considered summative peer review an essential part of career maintenance and advancement frequently spoke about the need to carefully choose their words and language when crafting their reports. They acknowledged being more comfortable documenting constructive suggestions for teaching improvement at reappointment, when the stakes are less high, than at tenure and promotion. Most participants said that constructive suggestions were not included in the peer evaluation reports for tenure because these might have an “unwarranted negative effect on the person’s circumstance” (interview 8, full professor). Given that these reports were tied to important career decisions, faculty members did not want to “pollute” (interview 7, associate professor) their colleagues’ files with comments that might be misconstrued. “It wouldn’t be appropriate to put those things in the summative evaluation because they’re just going to be viewed as negatives when they’re really meant to be helpful, constructive suggestions” (interview 2, full professor). Hence, when they believed tenure or promotion was deserved, they refrained from documenting suggestions for improvement. With the exception of three faculty members who claimed that the report on teaching presents a “balanced view,” the majority of participants agreed that “everything” that goes into a file “is supposed to be good” (interview 4, associate professor). Consequently most written reports contained only positive statements. When reviewers opted to share their constructive feedback, they did so verbally only. In addition, faculty members alluded to the demanding nature of faculty careers, the high expectations for research productivity, and the limited time they had as a way to explain the existing feedback practices. Several of the professors I interviewed also expressed doubt about whether the summative peer reviews actually contributed to decisions about career advancement. Given the above, participants said that they and others were disinclined to invest large amounts of time in the review process. Some suspected that faculty members might think they “have done their thing” (interview 9, associate CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 115 professor), that is, fulfilled their obligations once they had observed the class and handed in the report. Since there were no departmental guidelines specifying that candidates and reviewers share the results, faculty members might have conjectured that the pre- and post-review discussions were simply an extra, optional activity. Furthermore, some participants thought that academics could provide only limited insights into their colleagues’ teaching because many of them lack pedagogical knowledge or expertise in the evaluation of teaching: Rarely do you see anything of substance [in the reviewer’s feedback]. And why is that? Because people who know nothing about teaching assessment are asked to assess someone’s teaching, you know … People all have expertise in some field of their own, [but] zero expertise in teaching. In fact, some of our worst teachers assess other people’s teaching for tenure. So it’s just sort of like the blind leading the blind. (interview 27, full professor) This quote is representative of two recurrent themes. The first theme is that most academics lack formal knowledge of teaching. Because of this, some participants said that evaluating a colleague’s teaching was awkward: “And I think there’s a reluctance to be overly critical if you see something that you yourself wouldn’t do [in teaching]. Because you wonder, ‘Does it work? Does it not work?’ It’s like, ‘I don’t know’” (interview 20, full professor). An associate professor added (interview 28), “We don’t know about pedagogy and don’t have training as teachers. So, there is a reticence about telling people how to do their job.” The second theme is that faculty members are conducting peer reviews “blindly”—that is, without any “objective” (i.e., research-informed) criteria or standards upon which to base their evaluations. For many faculty members—and especially those in science—the lack of objectivity inherent in peer reviews was a concern, and they bemoaned the subjective nature of these reviews. Certain participants proposed that peer reviews could be more objective if one were to adopt the following practices to evaluate teaching: select expert educators who can base their judgments on the literature and sound research, use a faculty-wide team of trained reviewers to avoid bias, measure student learning to determine instructor effectiveness, and use pre-established criteria and standards that are known to both the reviewer and person reviewed. A final reason that faculty members hesitated to provide constructive feedback to one another is because they perceived classroom observations as “artificial” teaching situations, where what happened was not necessarily indicative of the usual dynamics in the classroom. “There’s a high artificiality that the process involves because the students know what’s going on [that the instructor is being evaluated] and certainly the instructor has prepared beyond conception” (interview 13, full professor). Several professors acknowledged that the presence of someone who is not usually in the classroom had an effect: “Having a more senior faculty member sit in a classroom observing how someone teaches invariably does something to the actual classroom situation” (interview 6, full professor). In addition, some faculty members’ expressed concern about judging a peer’s overall teaching ability based on only one or two classroom observations of teaching. When faculty members were uneasy with the limited data for evaluating teaching and/ or were concerned that what they had observed was not representative of an instructor’s overall teaching, they were disinclined to engage in the peer review process, including the feedback-giving component. CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 116 Discussion Those who study academic culture agree that higher education is value laden and have identified collegiality as a dominant value and essential element of academic work (Gappa, Austin, & Trice, 2007; Tapper & Palfreyman, 2010). For the purpose of this study, I take collegiality to be a value that when put in practice, provides “opportunities for faculty members to feel that they belong to a mutually respectful community of colleagues who value their unique contributions to their institutions and who are concerned about their overall well-being” (Gappa et al., 2007, p. 142). A collegial colleague, therefore, offers professional and personal support to others. She does so in a number of ways that include, but are not limited to, collaborating with her peers, performing tasks that benefit the unit as a whole, communicating respectfully, and “relating to others in ways that are constructive, supportive, and professional” (Cipriano & Buller, 2012, p. 47). Studies have found that among North American faculty members, collegiality contributes significantly to job satisfaction (Cipriano & Buller, 2012; O’Meara, Terosky, & Neumann, 2008). Consequently, it has direct implications for retaining tenure-track faculty members (Johnston, Schimmel, & O’Hara, 2012; Russell, 2010). The findings reported in the previous section expose the extent to which academic values and norms shape feedback practices. In this discussion I argue that within the context of summative peer review of teaching, faculty members refrain from sharing constructive feedback and limit their overall feedback as a way of preserving collegial relationships. Constructive Feedback: A Barrier to Tenure The study findings confirm previous research that has established tenure as highly valued in North American research-intensive universities because it confers job security, power, and prestige (Chait, 2002; McClain Da Costa, 2012). This structural entity is an important cultural force in faculty careers (O’Meara, 2011; Tierney, 1997). In this study, the majority of participants believed that the inclusion of constructive feedback in the summative peer review of teaching report might present a barrier to achieving tenure (and/or a promotion). Consequently, when professors opined that their colleague deserved tenure or other career promotion, they avoided documenting their constructive written feedback in summative peer review reports. In adopting this practice, they supported their colleagues (i.e., behaved in a collegial manner). Since academic careers are demanding and because faculty members know that research productivity is rewarded above teaching in the research-intensive university (Chalmers, 2011), academics make choices about where to allocate their time (Link, Swan, & Bozeman, 2008). Participants in this study repeatedly spoke about the challenges involved with developing as teachers in a context where research matters more in decisions about career advancement. Though only a few participants said that limited feedback in peer review might be due to time constraints, many expressed doubt about whether peer reviews actually played a meaningful role in informing tenure and promotion decisions. Thus, the findings suggest that the uncertain significance of peer reviews, combined with the established value of research productivity, might have a bearing on how much time and effort academics are willing to take conducting and debriefing peer reviews of teaching. From an academic-culture perspective, the academic reward system is a major source CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 117 of extrinsic motivation that works to socialize and shape faculty members’ behaviours (O’Meara, 2011). As such, it has bearing on feedback practices between colleagues, including those between the department chair and faculty members. Giving Feedback: Divisive, Difficult, and a Violation of Norms Massy, Wilger, and Colbeck (1994) along with Becher and Trowler (2001) and Roxå and Mårtensson (2009) have noted that academics are often unwilling to pursue issues that may be divisive or spark debate. Instead, they are “inclined to play safe—to minimize the risk of making professional enemies by not opposing or being critical of colleagues’ views” (Becher & Trowler 2001, p. 127). This lack of exchange maintains a surface calm but results in a lack of discussion about issues of curricular structure, pedagogical alternatives, and student assessment (Massy et al., 1994). Candidates who receive constructive feedback from their peers may perceive the judgments as threatening and can become defensive; the exchange may then cause friction among colleagues (Blackmore, 2005; Carter, 2008; Chism, 2007a; Shortland, 2010). If faculty members perceive that the act of providing feedback may disturb collegial relationships, they may prefer to avoid conversations that address the ambiguities and contradictions that surfaced in peer review. Though participants in this study did not explicitly state that they were worried their feedback might be poorly received by colleagues, this may have been one of their underlying concerns. That many of the professors described giving feedback to their peers as “awkward” also underscores the link between feedback and maintaining collegiality. In this study, the awkwardness of the feedback process came about principally because reviewers, (1) did not think they had sufficient pedagogical expertise to adequately critique their peers, (2) had to conduct peer reviews without any clear pre-established departmental criteria for good teaching and therefore believed the process was too subjective and unreliable, and (3) were of the opinion that they could not provide an adequate evaluation of someone’s overall teaching based on reviewing only a small portion of a colleague’s teaching activities. These sentiments reflect the knowledge that giving constructive feedback is a demanding skill because it is a complex communication process involving considerations about the content of the message, the modes of feedback, and timing (Hattie & Timperley, 2007; Shortland, 2010). The challenges inherent in feedback are augmented by the fact that few academics are trained on giving (and receiving) feedback in the context of a peer review of teaching (Cosh, 1998; MacKinnon, 2001). Recognizing that they lack training and the pedagogical knowledge to give informed feedback on teaching, some may prefer not to engage in this practice. Therefore, even though individualized feedback has immense potential to affect an individual’s teaching and learning, it does not do so; this is true of feedback in many contexts that could foster professional growth in teaching (Hattie & Timperley, 2007). The awkwardness and discomfort that numerous participants spoke about in terms of minimal pedagogical knowledge, few criteria, and little data on which to base a judgment can also be examined by considering academic norms, which, like values, occupy a central place in academic culture. Braxton (2010,) defines norms as “shared beliefs of a particular social or professional group about expected or desired behaviors in a given situation or circumstance” (p.243). Formal and informal norms develop for behaviours that the majority of the group considers important. Academics internalize these norms, and over CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 118 time these shape academic codes of conduct (O’Meara, 2011). In universities and colleges, where faculty members have a lot of autonomy, norms provide guidelines for appropriate (and inappropriate) behaviours (Braxton, 2010). One can probe the link between feedback and collegiality by considering two, interrelated Mertonian norms of science: organized skepticism and universalism. These norms, based on the ideas of American sociologist Robert Merton, are relevant to scientists who work in natural or physical sciences or in social sciences in research universities (O’Meara, 2011). Organized skepticism pertains to critically scrutinizing scientific claims before accepting them. Within this norm, an individual suspends judgment about a claim until he has been able to examine the methods and findings according to accepted standards and criteria (Anderson, Ronning, De Vries, & Martinson, 2010). As previously mentioned, many of the participants in this study criticized the peer reviews for not being representative of someone’s overall teaching, said that reviewers did not normally have the pedagogical expertise required to give quality feedback, and lamented the lack of criteria and guidelines in the peer review. These three criticisms indicate that the norm of organized scrutiny was being violated because faculty members did not have accepted standards and well-established criteria by which to evaluate their peers’ teaching. The norm of universalism, which is related to that of organized skepticism, involves evaluating merit based on impersonal and universal criteria rather than on criteria such as gender, personality traits, and nationality (Mitroff, 1974; O’Meara, 2011). It presumes that a judgment will be made objectively, “without reference to the scientists’ nationality, race, religion, professional affiliations and other irrelevant characteristics” (Anderson et al., 2010, p. 368). Thus, when faculty members are asked to review teaching but have only limited pedagogical expertise and vague criteria to go by, or have only a small amount of information, they may also feel they are behaving against the norm of universalism. Since adherence to academic norms reflects an academic’s commitment to group values (Braxton, 2010) and providing feedback to peers under the conditions listed above counters the norms of organized skepticism and universalism, faculty members may prefer to avoid engaging in quality feedback practices when doing could be perceived as inappropriate by their peers. Recommendations for Enhancing the Developmental Potential of Summative Peer Reviews The results of this study have several implications for enhancing the developmental potential of peer reviews. I have summarized these in Table 1. There are some who will argue that summative peer review should not attempt to promote instructional growth. I, however, do not subscribe to this belief. I see summative peer reviews as an opportunity to foster “learning conversations” among peers (Byrne et al., 2010). These conversations are essential since they help faculty members expand their understanding of teaching and student learning (Knight, Tait, & Yorke, 2006; O’Meara et al., 2008; Roxå & Mårtensson, 2009) and also, more broadly, help the academic profession and their institution (Gappa et al., 2007). CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 119 Table 1 Recommendations for Enhancing the Developmental Potential of Summative Peer Reviews of Teaching Faculty Members’ Comments (Major study finding) Recommendation Reviewers rarely provide verbal feedback to candidates on the outcomes of their peer reviews of teaching. Implement a policy whereby a post-observation discussion is part of the summative peer review of teaching process. Ideally, the candidate hears feedback from the reviewer(s) shortly after the review. However, this discussion could take place after a decision about career advancement is made. Department heads rarely speak about the peer review of teaching results in their annual meetings with candidates. Discuss the summative peer review results (or report) in the annual meeting between a department head and candidate, when the latter’s career progress is examined. The peer review report is not shared with the candidate. Share the reviewers’ written comments and recommendations with the candidate; this report could be shared as is or findings could be summarized/interpreted by an educational developer working at the institution’s teaching and learning centre. The reviewers’ findings are not Formulate a plan that addresses the reviewers’ feedback, as tied to a professional growth plan. needed. This could be done in consultation with an educational developer. Provide training to reviewers on conducting peer reviews of Reviewers have limited pedateaching. gogical knowledge to draw upon when conducting summative peer Enlist the assistance of an educational developer from the reviews. teaching and learning centre since these individuals have pedagogical expertise and strong skills in providing constructive feedback on teaching. Provide an itemized list of research-based criteria (or rubric) that faculty members can refer to as they evaluate their colleague’s teaching. Faculty members should have the opportunity to discuss and agree upon this list of criteria, which ought represent a broad range of teaching practices in the department. Peer reviews are not representative of a faculty members’ overall teaching. While being mindful of the time needed to engage in peer review, consider a process that includes a review of multiple class visits (e.g., to an undergraduate class and graduate class), an individual’s teaching portfolio, course materials (e.g., syllabi, assessment methods, assignments), student supervision, and other teaching sites (e.g., laboratories, small-group tutorials). CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 120 Limitations of the Research and Possibilities for Future Directions It is important to note some of the limitations of this study. First, the study relied on individual faculty members’ perceptions of feedback practices; I did not observe candidates and reviewers giving feedback to one another, nor did I have access to reading any of the peer review reports. Since verbal feedback was rarely given and post-observation discussions were not customary, capturing such an exchange might not have been possible. However, it might be feasible for a researcher to read a candidate’s summative peer report if she could acquire this once the candidate’s tenure and promotion decision had been made. Since the candidate would need to provide a copy of this report to the researcher, I suspect that only successful candidates would participate. Second, participants self-selected to participate in this study, and in our conversations, it became apparent to me that these individuals were not only highly committed to teaching but also to their own professional growth as teachers. As I indicated in the findings, many of them were disappointed that summative peer reviews did not generate better feedback that could contribute more to their growth. Had participants been randomly selected to participate in the study, perhaps fewer would have been troubled by the limited feedback in peer review. Third, all study participants were recruited from within one institution that at the time had no university-wide policy on summative peer review. Given that each institution of higher education has its own unique culture (O’Meara et al., 2008), and that the number of participants in this study was small, the results of a qualitative study conducted elsewhere may produce different results. Were I to design a subsequent study, I would design a survey and distribute it randomly to a greater number of people, so as to capture a broader sample of faculty members with a wider range of experiences. Finally, I stated in the introduction that one of the principal reasons peer reviews of teaching have become more widespread is to improve the process of faculty evaluation. Though this study did not set out to examine this question directly, the findings indicate that even if the criteria for an improved system include promoting collegial dialogue, the benefits of summative reviews are questionable at best. Future research might examine whether the combined use of peer reviews and student evaluations of teaching actually strengthen the faculty evaluation system. Conclusion Attention to values in a study about peer review teaching is relevant because these guide academics who, as Schuster and Finklestein (2006) explain, “are among the most likely—given the relative autonomy they enjoy—to ‘act out’ their values in fulfilling their professional responsibilities” (p. 87). This study profiled the role of collegiality, as an academic value, in feedback practices in summative peer reviews of teaching. It underscored that existing practices do not contribute to academics’ professional growth in teaching and suggested ways that these practices might be modified. As the use of summative peer reviews of teaching becomes more common in Canada, a better understanding of faculty members’ experiences is required to enhance the process of faculty evaluation and contribute further to career satisfaction. CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 121 References Ackerman, D., Gross, B. L., & Vigneron, F. (2009). Peer observation reports and student evaluations of teaching: Who are the experts? Alberta Journal of Educational Research, 55(1), 18–39. Retrieved from http://ajer.synergiesprairies.ca/ajer/index.php/ ajer/index Alvesson, M., & Sveningsson, S. (2008). Changing organizational culture: Cultural change work in progress. New York, NY: Routledge. Anderson. M.S., Ronning, E.A., De Vries R., Martinson, B.C. (2010). Extending the Mertonian norms: Scientists’ subscription to norms of research. The Journal of Higher Education, 81(3), 366–93. doi: 10.1353/jhe.0.0095 Atkinson, D. & Bolt, S. (2010). Using teaching observations to reflect upon and improve teaching practice in higher education. Journal of the Scholarship of Teaching and Learning, 10(3), 1–19. Retrieved from http://josotl.indiana.edu/article/ viewFile/1798/1795 Arreola, R. (2007). Developing a comprehensive faculty evaluation system (3rd ed.). Bolton, MA: Anker Publishing Company. Becher, T., & Trowler, P. R. (2001). Academic tribes and territories: Intellectual enquiry and the culture of disciplines (2nd ed.). Philadelphia, PA: Society for Research into Higher Education & Open University Press. Bell, M. & Cooper, P. (2013): Peer observation of teaching in university departments: A framework for implementation, International Journal for Academic Development, 18(1), 60–73. doi: http://dx.doi.org/10.1080/1360144X.2011.633753 Benton, S. & Cashin, W. (2012). Student ratings of teaching: A summary of research and literature (IDEA Paper 50). Manhattan, KS: The IDEA Center. Retrieved from http:// www.theideacenter.org/research-and-papers/idea-papers Blackmore, J. (2005). A critical evaluation of peer review via teaching observation within higher education. International Journal of Educational Management, 19(3), 218–232. doi: 10.1108/09513540510591002 Braxton, J. M. (2010). Norms and the work of colleges and universities: Introduction to the special issue—Norms in academia. The Journal of Higher Education, 81(3), 243– 250. doi:10.1353/jhe.0.0097 Bunch, K. (2007). Training failure as a consequence of organizational culture. Human Resource Development Review, 6(2), 142–163. doi:10.1177/1534484307299273 Byrne, J., Brown, H., & Challen, D. (2010). Peer development as an alternative to peer observation: A tool to enhance professional development. International Journal for Academic Development, 15(3), 215–228. doi:10.1080/1360144X.2010.497685 Carter, V. (2008). Five steps to becoming a better peer reviewer. College Teaching, 56(2), 85–88. doi:10.3200/CTCH.56.2.85-88 Chait, R. (2002). Does faculty governance differ at colleges with tenure and colleges without tenure? In R. Chait (Ed.), The questions of tenure (pp. 69–100). Cambridge, MA: Harvard University Press. CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 122 Chalmers, D. (2011). Progress and challenges to the recognition and reward of the Scholarship of Teaching in higher education. Higher Education Research & Development, 30(1), 25–38. doi:10.1080/07294360.2011.536970 Chism, N. (2007a). Peer review of teaching: A sourcebook (2nd ed.). Bolton, MA: Anker Publishing Company. Chism, N. (2007b). Why introducing or sustaining peer review of teaching is so hard, and what you can do about it. Department Chair: A Resource for Academic Administrators, 18(2), 6–8. doi:10.1002/dch.20017 Cipriano, R. E., & Buller, J. L. (2012). Rating faculty collegiality. Change: The Magazine of Higher Learning, 44(2), 45–48. doi: http://dx.doi.org/10.1080/0009138 3.2012.655219 Cosh, J. (1998). Peer observation in higher education—A reflective approach. Innovations in Education and Teaching International, 35(2), 171–176. doi:10.1080/ 1355800980350211 Courneya, C., Pratt, D. D., & Collins, J. (2008). Through what perspective do we judge the teaching of peers? Teaching and Teacher Education: An International Journal of Research and Studies, 24(1), 69–79. doi:http://dx.doi.org/10.1016/j.tate.2007.01.009 Creswell, J. W. (2012). Educational research: planning, conducting, and evaluating quantitative and qualitative research. (4th ed.). Upper Saddle River, NJ: Pearson Education. Gappa, J. M., Austin, A. E., & Trice, A. G. (2007). Rethinking faculty work: Higher education’s strategic imperative. San Francisco, CA: Jossey-Bass. Gravestock, P. (2011). Does teaching matter: The role of teaching evaluation in tenure policies at selected Canadian universities (unpublished doctoral dissertation). University of Toronto, Canada. Retrieved from http://tspace.library.utoronto.ca/ bitstream/1807/31764/6/Gravestock_Pamela_S_201111_PhD_thesis.pdf Hammersley-Fletcher, L., & Orsmond, P. (2004). Evaluating our peers: Is peer observation a meaningful process? Studies in Higher Education, 29(4), 213–224. doi:10.1080/0307507042000236380 Hattie, J. & Timperly, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. doi:10.3102/003465430298487 Iqbal, I. (2012). Faculty members’ professional growth in teaching through the summative peer review of teaching and other departmental practices. Unpublished doctoral dissertation, University of British Columbia, Vancouver, BC. Johnston, P. C., Schimmel, T., & O’Hara, H. (2012). Revisiting the AAUP recommendation: The viability of collegiality as a fourth criterion for university faculty evaluation. College Quarterly, 15(1). Retrieved from http://www.eric.ed.gov/PDFS/ EJ976454.pdf Kell, C., & Annetts, S. (2009). Peer review of teaching embedded practice or policyholding complacency? Innovations in Education and Teaching International, 46(1), 61– 70. doi:10.1080/14703290802646156 CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 123 Kemp, R., & Gosling, D. (2000). Peer observation of teaching. The Higher Education Academy Subject Centre for Education ESCalate. Retrieved from http://escalate.ac.uk/ resources/peerobservation2/ Knight, P., Tait, J., & Yorke, M. (2006). The professional learning of teachers in higher education. Studies in Higher Education, 31(3), 319–339. doi:10.1080/ 03075070600680786 Link, A. N., Swan, C. A., & Bozeman, B. (2008). A time allocation study of university faculty. Economics of Education Review, 27, 363–374. doi:10.1016/j.econedurev. 2007.04.002 MacKinnon, M. (2001). Using observational feedback to promote academic development. The International Journal for Academic Development 6(1), 21–28. doi: 10.1080/13601440110033689 Massy, W. F., Wilger, A. K., & Colbeck, C. (1994). Departmental cultures and teaching quality: Overcoming ‘hollowed’ collegiality. Change, 26(4), 10–20. doi:10.1080/000913 83.1994.9938496 Mayan, M. J. (2009). Essentials of qualitative inquiry. Walnut Creek, CA: Left Coast Press. McClain Da Costa, K. (2012). The tenure system, disciplinaryboundaries and reflexivity. Ethnic and Racial Studies, 35(4), 626–632. doi:http://dx.doi.org/10.1080/01419870.20 12.644802 Mitroff, I. I. (1974). Norms and counter-norms in a select group of the Apollo moon scientists: A case study of the ambivalence of scientists. American Sociological Review, 39(4), 579–595. Retrieved from http://www.jstor.org/stable/2094423 O’Meara, K. (2011). Inside the panopticon: Studying academic reward systems. In J. C. Smart & M. B. Paulsen (Eds.). Higher education: Handbook of theory and research (pp. 161–220). New York, NY: Agathon Press. doi:10.1002/aehe.3403 O’Meara, K., Terosky, A., & Neumann, A. (2008). Faculty careers and work lives: A professional growth perspective. ASHE Higher Education Report, 34(3). doi:10.1002/ aehe.3403 Peel, D. (2005). Peer observation as a transformatory tool? Teaching in Higher Education, 10(4), 489–504. doi:10.1080/13562510500239125 Russell, B. C. (2010). Stress in senior faculty careers. New Directions for Higher Education, (151), 61–70. doi:10.1002/he.401 Roxå, T., & Mårtensson, K. (2009). Significant conversations and significant networks: Exploring the backstage of the teaching arena. Studies in Higher Education, 34(5), 547559. doi:10.1080/03075070802597200 Schram, T. H. (2003). Conceptualizing qualitative inquiry. Columbus, OH: MerrillPrentice Hall. Schuster, J., & Finkelstein, M. (2006). The American faculty: The restructuring of academic work and careers. Baltimore, MD: Johns Hopkins University Press. CJHE / RCES Volume 44, No. 1, 2014 Don’t Tell It Like It Is / I. Iqbal 124 Shanahan, T. (2009). Accountability initiatives in higher education: An overview of the impetus to accountability, its expressions and implications. Retrieved from http:// www.yorku.ca/univsec/senate/committees/exec/reports/2008-2009/Shanahan%20 Paper.pdf Shortland, S. (2010). Feedback within peer observation: Continuing professional development and unexpected consequences. Innovations in Education and Teaching International, 47(3), 295–304. doi:10.1080/14703297.2010.498181 Tapper, T. and Palfreyman, D. (2010). The collegial tradition in the age of mass higher education. Dordrecht, Ger.: Springer. Tierney, W. G. (1997). Organizational socialization in higher education. Journal of Higher Education, 68(1), 1–16. Retrieved from http://www.jstor.org/stable/2959934 Trowler, P. R. (2008). Cultures and change in higher education: Theories and practices. London, Eng.: Palgrave Macmillan. Yiend, J., Weller S., & Kinchin, I. (2012). Peer observation of teaching: The interaction between peer review and developmental models of practice. Journal of Further and Higher Education, ifirst Article, 1–20, doi:10.1080/0309877X.2012.726967 Yin, R. K. (2009). Case study research: Design and methods. Los Angeles, CA: Sage. Acknowledgments I would like to acknowledge the support I received from the Social Sciences and Humanities Research Council (Joseph-Armand Bombardier Canada Graduate Scholarship) for conducting my research. I also wish to thank the anonymous reviewers of this article for their helpful comments and suggestions, and the thirty faculty members who participated in this study. Contact Information Isabeau A. Iqbal University of British Columbia Faculty of Pharmaceutical Sciences (Office of Educational Support and Development) [email protected] Isabeau Iqbal is an educational developer at the Office of Educational Support and Development in the Faculty of Pharmaceutical Sciences (University of British Columbia, Canada). She is a qualitative researcher whose interests include professional growth in teaching, faculty careers, and departmental cultures of teaching; more recently, Isabeau has become involved in health education research. Isabeau Iqbal holds a doctorate in education from the University of British Columbia. CJHE / RCES Volume 44, No. 1, 2014