The Canadian Journal of Higher Education La revue canadienne d'enseignement supérieur Volume XXXIII, No. 1,2003 pages 2 5 - 6 0 Selecting and Using Course Readings: A Study of Instructors' and Students' Practices* KATHERINE BISCHOPING York University ABSTRACT U n i v e r s i t y s t u d e n t s ' p r a c t i c e s of r e a d i n g required c o u r s e m a t e r i a l s h a v e rarely b e e n studied s y s t e m a t i c a l l y o u t s i d e the laboratory and are given short shrift in c o u r s e evaluation q u e s t i o n n a i r e s . T h i s study e x a m ines, first, t h e r e a s o n s w h y a s a m p l e o f instructors at a large university c r e a t e s e l e c t i o n s o f c o u r s e r e a d i n g s and t h e f a c t o r s , both p e d a g o g i c a l and o t h e r w i s e , that i n f l u e n c e their selections. S e c o n d , s t u d e n t s ' patterns o f r e a d i n g in t h e s e i n s t r u c t o r s ' c o u r s e s a r e s t u d i e d to d e t e r m i n e h o w t h e y vary with the quantity o f assigned readings, t i m e of year, and the provision o f s y s t e m a t i c student f e e d b a c k to instructors. T h i r d , the m o s t p r e v a l e n t t h e m e s in student f e e d b a c k a b o u t r e a d i n g s are e x p l o r e d , in tand e m with i n s t r u c t o r s ' p r o p o s a l s a b o u t h o w to revise their r e a d i n g selections or t e a c h i n g strategies. * This study was supported by a Release-Time Teaching Fellowship and a Teaching & Learning Development Grant from the York University Faculty Association. My thanks to Marcel Martel, who kindly translated the abstract, and to the research participants. 26 RÉSUMÉ L ' u t i l i s a t i o n de r e c u e i l s de t e x t e s , r e q u i s d a n s les cours universitaires, a rarement fait l'objet d'analyse sauf dans les laboratoires dirigés par les psychologes. De plus, elle reçoit très peu d'attention dans les q u e s t i o n n a i r e s utilisés p o u r l ' é v a l u a t i o n d e s c o u r s . C e t t e é t u d e s'intéresse d'abord aux causes incitant certains universitaires, oeuvrant d a n s de g r a n d e s institutions d ' e n s e i g n e m e n t , à créer des recueils de textes ainsi q u ' a u x facteurs, pédagogiques et autres, qui influencent la sélection des textes. Ensuite, les habitudes de lecture des étudiantes et des étudiants dans les cours sélectionnés par cette étude sont analysées afin de déterminer si elles varient en fonction du nombre d'articles à lire, du m o m e n t de l ' a n n é e scolaire et des c o m m e n t a i r e s f o u r n i s par les é t u d i a n t e s et é t u d i a n t s a u x e n s e i g n a n t s . E n f i n , l ' a r t i c l e e x p l o r e les t h è m e s les p l u s i m p o r t a n t s c o n t e n u s d a n s les c o m m e n t a i r e s d e s étudiantes et étudiants concernant les articles choisis par les enseignants ainsi que ceux des enseignants sur la manière de revoir leur sélection de textes et leurs stratégies pédagogiques. C a n a d i a n university instructors c o m m o n l y c o m p i l e selections of j o u r n a l articles and book chapters to s u p p l e m e n t , or replace, course t e x t b o o k s . For e x a m p l e , at my h o m e institution of York University, which has 38,500 students and is C a n a d a ' s third largest university, the three leading suppliers estimated that they were asked to produce close to 1,250 different selections of readings for the 2 0 0 0 - 2 0 0 1 school year. Despite the popularity of reading kits, remarkably little is known about how students use them or h o w instructors can c h o o s e their contents most effectively. One source that a concerned instructor might consult is the substantial body of psychological research on university-level reading, in which experimental methods are used to understand the cognitive processes of a student who is engaged in a comprehension task in a laboratory setting (e.g., Alexander, Kulikowich, & Schulze, 1994; Beishuizen, Stoutjesdijk, & Van Putten, 1994; Marton & Sâljô, 1976; Schnotz, Picard, & Hron, 1993). For example, Schnotz et al. (1993) found that, when using a map The Canadian Journal of Higher Education Volume XXXIII, No. I, 2003 Selecting and Using Course Readings 27 alongside text, successful learners distinguished t h e m s e l v e s f r o m their unsuccessful peers by f o c u s i n g better on the information pertinent for constructing mental models. By tracking h o w students m a d e their way through a hyptertext chapter, Beishuizen et al. (1994) determined that in contrast with their " d e e p processing" peers, students with "surface processing" learning styles m o v e d chaotically a m o n g text cards and benefited from instructions about h o w information was organized. In a study of h o w students read physics materials, Alexander et al. (1994) propose t h a t p e r s o n a l l y - i n v o l v i n g d e t a i l s in a t e x t ( s u c h a s h o w S t e p h e n H a w k i n g ' s research work m a y have slowed the progress of his disease) can seduce students' attention away from key information. Such studies d o p r o v i d e insight into p r e c i s e l y h o w s t u d e n t r e a d e r s with d i f f e r e n t learning styles formulate mental models, benefit from instructions, remain focused, or b e c o m e distracted. But all of these studies — not to m e n t i o n the n u m e r o u s o b s e r v a t i o n s of r e a d e r s ' e y e m o v e m e n t s (see Starr & Rayner 2 0 0 1 ) — apply only to students w h o have their readings in hand. They d o not address the thorny question of w h y university students decide whether or not to pick up a reading in the first place. A small but p r o m i s i n g body of research a d d r e s s e s this issue. C h a m b e r s (1992), for example, reasons that students are deterred by university courses with high workloads. She r e c o m m e n d s that instructors use a mathematical model f o r calculating the hours that various course c o m p o n e n t s , including reading, require. Williams ( 1 9 9 7 ) details h o w instructional strategies, such as pop quizzes, failed to encourage student reading in her tutorials. Based on a survey of her students, Williams reco m m e n d s strategies including interdependent learning, that could better foster reading. M a n n ( 2 0 0 0 ) a d v o c a t e s understanding m o r e fully h o w students find their usually private and pleasant experiences of reading to be unsettled when they must read in order to be evaluated publicly in a c a d e m e , while Francis and Hallam (2000) also discuss the emotional costs of students of efforts to understand a c a d e m i c text. These studies extend the boundaries of reading research by considering the naturalistic contexts in which students learn, yet for the most part their r e c o m m e n d a tions have not yet been tested systematically. The Canadian Journal of Higher Education Volume XXXIII, No. I, 2003 28 K. Bischoping C o u r s e evaluation questionnaires, a potential source of systematic i n f o r m a t i o n f o r instructors a b o u t student reading, also fall short. For e x a m p l e , the s t a n d a r d c o u r s e e v a l u a t i o n q u e s t i o n n a i r e used at York University, C a n a d a , m e n t i o n s r e a d i n g only in the item: " T a k i n g into account all aspects of the c o u r s e (instruction, readings, a s s i g n m e n t s ) , h o w would you rate the course overall [on a five-point scale]?" Sample q u e s t i o n n a i r e s provided in the c o u r s e e v a l u a t i o n literature include at most two items on readings (Liow, Betts, & Lit, 1993; Marsh & Bailey, 1993; Wilson, Lizzio, & Ramsden, 1997). Such items hardly provide the " s p e c i f i c , d e t a i l e d , and i d i o s y n c r a t i c f e e d b a c k " t h a t M u r r a y ( 1 9 8 4 , p. 123) deemed essential for formative purposes. In this investigation, I a d d r e s s limitations on existing research or evaluation q u e s t i o n n a i r e s pertaining to student r e a d i n g by e x a m i n i n g empirically the contexts in which students read, including the instructors' p e d a g o g i c a l a i m s , n o n - p e d a g o g i c a l i n f l u e n c e s on instructors, c o u r s e workload, and timing in the academic year. I also provide a student questionnaire that elicits systematic and detailed feedback on course readings, and assess the efficacy of providing instructors with this feedback. T h e first research question addressed is w h y instructors decide to assign reading kits and h o w they choose their contents. The accounts of six instructors help us to understand how pedagogical aims influence kit development. Surprisingly, their accounts show that factors unrelated to pedagogical aims often influence h o w readings are chosen. Next, 1 explore how students' reading patterns vary by other aspects of course context. Prompted by arguments that a high workload deters learning ( C h a m b e r s , 1992; Franz, Ferreira, Loh, Pendergast, Service, Stormont, Taylor, T h a m b i r a t n a m e , & Williamson, 1996; Gibbs, 1992, cited in Hartley, 1998), I use student questionnaires to e x a m i n e whether students' claims to recall readings are affected by the quantity of readings an instructor assigns and whether reported recollection varies by the p o i n t in the a c a d e m i c y e a r at w h i c h a r e a d i n g w a s a s s i g n e d . I a l s o explore whether receiving systematic feedback from students leads instructors to change their courses in w a y s that increase students' recall of readings. T h e c o m p l e x methodological issues that render these explorations tentative are taken up in some detail. The Canadian Journal of Higher Education Volume XXX1H, No. /, 2003 Selecting and Using Course Readings 29 Finally, I determine students' most prevalent concerns about reading material and show that these vary little amongst the disciplines studied. Students typically recommend that there be less required reading, and that it be easier, better written, and more relevant. The instructors did not take these recommendations at face value but, rather, refrained them in w a y s consistent with their p e d a g o g i c a l a i m s . The n e w instructional strategies that the instructors formulated are discussed with reference to studies on reading and learning processes (e.g., Alexander et al., 1994; Baxter Magolda, 1992; Bean, 1996; Schnotz et al., 1993) and on instruct o r s ' and s t u d e n t s ' d i f f e r i n g c o n c e p t i o n s of learning (e.g., Child & Williams, 1996; Franz et al., 1996; L i o w e t a l . , 1993). METHODS First Year: 1998-1999 In 1998-1999, a sample of six instructors from a large urban university where reading kits are frequently used, agreed to participate in this study. Although the sample was opportunistic, the instructors and the eight courses they were teaching with kits, covered a wide range of possibilities. The instructors came from disciplines in the general areas of business, humanities, and the social sciences. (For ethical reasons, the names of these general areas, capitalized and set in quotation marks, are used as pseudonyms for departments. Pseudonyms are also used for the names of authors of course readings.) Half of the instructors were offering their course at this institution for the first time. One of the courses was at the 2nd year level, five were third year courses, and two were fourth year courses. In two courses, the reading kit w a s the only required reading material. In five others, textbooks were used; in one, several novels were assigned. First interview with instructors. Each instructor participated in a one to t w o hour long structured qualitative interview asking them to describe their course(s), their reasons for using a reading kit, and their method of selecting readings. Probes included whether materials from an earlier kit, colleague or canon were chosen; whether (and how) student The Canadian Journal of Higher Education Volume XXXIII, No. I, 2003 30 K. Bischoping f e e d b a c k w a s sought; and w h e t h e r criteria other than the content of a reading, such as its publication date or its a u t h o r ' s sex, were considered. First student survey. Questionnaires were administered to 183 students in the eight courses between March 24 and April 9, 1999 (i.e., the s e c o n d - or third-last w e e k o f term). T h e q u e s t i o n n a i r e first listed the a u t h o r and title of each kit r e a d i n g . S t u d e n t s w e r e asked to indicate w h e t h e r it s h o u l d be k e p t or d r o p p e d t h e n e x t t i m e the c o u r s e w a s o f f e r e d , w h e t h e r t h e y had n o o p i n i o n o f it, or w h e t h e r t h e y did not r e m e m b e r it. Q u a l i t a t i v e c o m m e n t s a b o u t e a c h r e a d i n g w e r e a l s o invited. The questionnaire concluded with a pair of open questions, asking what distinguished kit materials that should be kept, and what other c h a n g e s to course materials w e r e r e c o m m e n d e d . T h e s e questionnaires required 10 to 25 minutes to administer. Confidentiality w a s protected by having instructors leave the room while questionnaires w e r e administered, by asking students not to identify themselves, and by providing typed feedback to instructors only after final grades had been submitted. The proportion of registered students w h o participated in the study v a r i e d g r e a t l y f r o m c o u r s e to c o u r s e , f r o m a p p r o x i m a t e l y 2 5 % t o approximately 68%.' These modest rates can best be explained by low attendance late in the winter since, in all but one class, m o s t students present appeared to be participating. Q u e s t i o n n a i r e results w e r e p r o v i d e d to instructors in reports that included: a table s h o w i n g for each reading, the n u m b e r of students w h o said they would keep it or d r o p it, had no opinion of it, or could not r e m e m b e r it; a list of the quartiles of readings (or as close to quartiles as possible) that had received the highest and lowest ratings; 2 a list of the readings that at least 2 5 % of students reported not remembering; a verbatim transcript of responses to the open-ended questionnaire items; and a s u m m a r y of t h e m e s in those responses. Follow-up interview with instructors. These reports were disc u s s e d in a s e c o n d i n t e r v i e w with each instructor. P a r t i c i p a n t s w e r e asked to think aloud about their reactions to the report, identify patterns in students' ratings, and indicate h o w they might use the report. The Canadian Journal of Higher Education Volume XXXIH, No. 1, 2003 Selecting and Using Course Readings 31 Second Year: 1999-2000 Follow-up student survey. In 1999-2000, five of the eight courses s t u d i e d in 1 9 9 8 - 1 9 9 9 w e r e a g a i n o f f e r e d by f o u r i n s t r u c t o r s f r o m " H u m a n i t i e s " and "Social Science" disciplines. Survey questionnaires based on the updated kit materials were completed by 151 students in these courses between March 20 and March 23, 2000 (i.e., the second- or third-last week of term). Procedures were unchanged, except that in one course where time was short, students were asked to return completed questionnaires at the subsequent lecture. Because only two of these 25 students complied, this course is omitted from analysis. In the remaining courses, between approximately 2 7 % and 8 0 % of registered students completed questionnaires. The results were compiled following the same procedures as in 1 9 9 8 - 9 9 , with the addition of tables c o m p a r i n g the 1 9 9 8 - 9 9 and 1 9 9 9 - 2 0 0 0 distributions of ratings and p e r c e n t a g e s of remembered materials. Final interview with instructors. These interviews addressed: the changes in students' evaluations of readings between year I and year II; instructor's predictions about which readings would obtain the highest and lowest scores; patterns in the readings that actually had obtained the highest and lowest scores from students; predictions about the times in the academic year for which students do the most reading; a chart showing how reported recall changes over the year; and the utility of the systematic feedback Sampling Issues While the samples of students and readings studied here are large — 334 and 394 respectively — the research involves only six instructors and 12 courses, chosen using non-random methods. Some parts of the analysis are limited to the four courses in which students completed questionn a i r e s in both 1 9 9 8 - 9 9 and 1 9 9 9 - 2 0 0 0 . T h e r e f o r e , the q u a l i t a t i v e analysis of instructors' statements must be seen as a tentative sketch that outlines tendencies of the sample rather than making claims about a population. 1 signal in the text that the results hold for "the sample of instructors" (rather than "instructors"). The quotations selected illustrate themes mentioned by more than one instructor, unless I have noted otherwise. The Canadian Journal of Higher Education Volume XXXIII, No. I, 2003 32 K. Bischoping In the quantitative analyses (in which courses are the units of analy- sis), I have used simple graphs and tables, rather than the formal statistical m e t h o d s that r e q u i r e r a n d o m s a m p l e s . Further, as t h e y arise, I identify interpretation issues that result from the small sample or f o r other methodological reasons. RESULTS Choosing Readings Instructors' reasons for using kits. Despite their varied disciplines, the sample of instructors had considerable common ground in the rationale for using reading kits. Most mentioned that kits could be used more effectively than textbooks to challenge students, echoing one of the criticisms of textbooks in Dorn (1989). A "Business" instructor said: " A f t e r the first two levels of courses in this area, students have 80 to 90 credits, and they can go beyond the textbook by starting to criticize topics and the discipline." A "Social Science" instructor remarked that "[textbook] authors think that the students are cretins, that they know nothing. They have figures, they choose an important author, say, Alvarez, and have two pages on him." "I want to show them a killer," was a "Humanities" instructor's comment on his most difficult reading. Another frequently-mentioned reason for using kits w a s that they expose students to diverse sources. One " H u m a n i t i e s " instructor explained, "I wanted to have some examples of art, so I wanted some poetry and short stories. I had a number of primary texts in philosophy — brief ones, it was a kind of tokenism. I wanted to have examples of contemporary academic articles." A "Social Scientist" said, "I have a kit because there are no good Canadian textbooks and therefore I have to s u p p l e m e n t the A m e r i c a n text with C a n a d i a n material. T h i s reason m i g h t not h a v e been as p r e v a l e n t had I i n c l u d e d N a t u r a l S c i e n c e s instructors in the study, because their disciplines employ less diverse p a r a d i g m s and have less contested c a n o n s (Schachter, C h r i s t e n f e l d , Ravina, & Biloux, 1991; Smeby, 1996). Other reasons for using kits, which the sample of instructors mentioned less often, were that no suitable text existed, that kits could cover a better range of publication dates, The Canadian Journal of Higher Education Volume XXXIII, No. I, 2003 Selecting and Using Course Readings 33 that kits best reflected instructors' politics or learning traditions, and that kits w e r e less expensive f o r students to purchase. Although the instructors concurred that students should be exposed to diverse sources, they disagreed about whether readings by authors of diverse identities should be sought. For e x a m p l e , to half of this small sample, an a u t h o r ' s sex mattered. " I ' m aware that 5 0 % of the students are w o m e n , " noted a " H u m a n i t i e s " instructor, while one "Social S c i e n c e " instructor indicated that she had discussed with students the s h o r t a g e of f e m a l e a u t h o r s in her field. T h e i r a p p r o a c h e s to r e a d i n g selection reflect feminist and other p e d a g o g u e s ' arguments for the recognition of diversity in the curriculum (e.g., Boud, 1993; T h o m a s , 1990), t h o u g h they d o not necessarily p r o b l e m a t i z e c o n c e p t s such as "integration" or "providing role m o d e l s " (Andersen, 1988; Fisher, 1988). In contrast, o t h e r instructors c l a i m e d that an a u t h o r ' s sex w a s of n o import. Interestingly, these instructors overestimated the percentage of female authors in their kits. For example, a " B u s i n e s s " instructor, w h o guessed that 3 0 % to 4 0 % of his readings had f e m a l e authors because "there are m a n y w o m e n accountants," found only 2 2 % actually did. A "Social Science" instructor, w h o believed that over half her readings had f e m a l e first authors, found the correct percentage to be 4 1 % . A private undertaking. At Canadian institutions, instructors typically have complete autonomy in their selections of reading kit materials. Four o f the six instructors interviewed depicted the choice of their kit materials as a remarkably solitary task. Apparently in near silence about their purposes, they selected n e w materials by w o r k i n g through the library stacks, eyeing others' syllabi, assessing n e w textbooks and j o u r n a l s , a n d a t t e n d i n g p r o f e s s i o n a l m e e t i n g s or c o u r s e s r e l a t e d t o course topics. T h e y explained their isolation variously by referring to confidence about reading choices, lack of relevant colleagues in a small f i e l d , a n d h a b i t s f o r m e d w h i l e w o r k i n g as an o s t r a c i z e d s e s s i o n a l instructor or as an untenured faculty m e m b e r fearful of scrutiny. S o m e w e r e surprised to realise j u s t h o w isolated their w o r k had been, as in t h e s e e x a m p l e s : " N o [I h a v e n ' t s p o k e n with a n y o n e ] . N o t b e c a u s e I d o n ' t want to. Nobody. I never had a conversation with colleagues about readings. It is seldom that [instructors in m y discipline] raise questions The Canadian Journal of Higher Education Volume XXXIII, No. I, 2003 34 K. Bischoping like this," and "I never talked to a living soul about my kits! M y colleagues n o w w o n ' t pull m e d o w n , the students appreciate m y work and my colleagues know it. So w h y w o n ' t I speak about kits?!" The silence about kits may not be golden. N o n e of the instructors consulted the professional librarians w h o select this university's library collections and w h o are well informed about n e w research and resources in their fields. Such c o n s u l t a t i o n with librarians or c o l l e a g u e s m i g h t have averted certain errors instructors felt they had m a d e when choosing materials. For example, one instructor w h o had taught her course several times confessed, " S o m e t i m e s when I ' m pressed for time and choose an article based on the abstract and I h a v e n ' t read it, then w h e n I get to it I think, 'holy shit!' " An instructor preparing a n e w course explained, "I had less than a month to m a k e the selection. I based [it] on what I knew, or w h e n 1 t h o u g h t , ' t h i s is a n i c e l o o k i n g a r t i c l e , ' a f t e r r e a d i n g the abstract and the first t w o or three pages. I try to put a brave f a c e on when they say, 'this is a horrible article.' " S t u d e n t f e e d b a c k q u a n d a r i e s . T h e teaching evaluation questionnaires students routinely complete provide little information about stud e n t s ' v i e w s o f c o u r s e r e a d i n g s . T h u s , the i n s t r u c t o r s I i n t e r v i e w e d tended to seek students' impressions of the readings by other means, e.g., "1 get ad hoc feedback; I might say, 'I found it blah blah blah' and they'll tell me they agree. But it's not at all systematic" or "I get reactions in class and in their j o u r n a l s . I s h o u l d e v a l u a t e , but here w e d o n ' t and there's no point in sticking out too much. [In another course] if t h e y ' v e not read an article well, t h e y ' r e quite quiet about it." T h e following quotation describes one of the most thorough attempts to obtain feedback: In October I asked students for feedback, and I wanted it written, because oral feedback might intimidate students. If they say 'sorry sir, it's b a d ' , under those conditions, it is a courageous student. I asked their opinion of the lectures, the topics, the workload, the tutorials, the readings, and any r e c o m m e n dations f o r improving the course. Students said the readings were okay. The instructors criticized these methods because they lacked rigor or were untimely, and because of the difficulty of preserving student confidentiality. The Canadian Journal of Higher Education Volume XXX///, No. I, 2003 Selecting and Using Course Readings 35 Further, one instructor was particularly concerned about the conflict between her pedagogical values and the values implicit in students' feedback. Faced with poor formal evaluations in a course she believed had gone well, she commented: For me, [students] have to be really independent and most of them are dependent. It's true that they are independent in that they work outside, but they are really consumers. If they are unhappy with the course, they say, 'but I paid for it,' and not, 'I w a n t to w o r k in the area o f [X] and this c o u r s e is not related to my objectives.' I wanted a course where students w o u l d feel able to speak. I t h o u g h t I had d o n e that but it s e e m s it is not the case. But I am stubborn. Even if I am untenured, I will do what I think best to do and that's it. This instructor's perception that feedback can be embattling, particularly when it is employed in tenure decisions, illustrates Murray's (1984) point that evaluation practices may lower standards or reduce morale, d e p e n d i n g on h o w the evaluation o u t c o m e s are used. W h i l e M u r r a y (1997) later concluded that student evaluations actually have positive effects on several aspects of teaching and that "there is no clear evidence that student evaluation of teaching has led to negative side-effects commonly attributed to it" (p.21), the faculty survey data he reviewed show substantial variation a m o n g institutions in instructors' perceptions about negative effects. Using Readings: Course Context and Student Reading Patterns We now turn from the pedagogical and non-pedagogical factors that influence instructors' decisions about kits, to the questions of how much and when students read. This section explores, first, whether students' claims to remember readings are associated with the quantity of assigned readings; second, whether students change their reading patterns in predictable w a y s during the academic year; and third, whether students' r e a d i n g patterns c h a n g e d b e t w e e n 1 9 9 8 - 9 9 and 1 9 9 9 - 2 0 0 0 ( o w i n g either to the research intervention of providing systematic student feedback or to other intervening factors). The Canadian Journal of Higher Education Volume XXXI11, No. /, 2003 36 K. Bischoping T h e w o r k l o a d c o n t e x t . In their q u a l i t a t i v e r e c o m m e n d a t i o n s , at least 10% of students in all but one course requested that instructors thin kit m a t e r i a l , by u s i n g f e w e r or s h o r t e r r e a d i n g s . T y p i c a l c o m m e n t s include: " S h o r t e r r e a d i n g s are better j u s t b e c a u s e it is e a s i e r to stay attentive until the e n d ; " "Too much material to take in all at one time sometimes. Too m u c h material, needed to spread out so can be better understood," and "Overall, I think the readings were good, but too long — I think a s u m m a r i z e d version w o u l d be easier f o r the s t u d e n t s to u n d e r s t a n d . " D o students read m o r e in courses w h e r e a smaller n u m b e r of readings are assigned? T h e percentage of readings students reported r e m e m b e r i n g in each course is plotted against the n u m b e r o f assigned readings in Figure 1. N o relation appears: regardless of the quantity of readings required, students in each course said they remembered some 7 0 % to 9 0 % of readings. Accordingly, the higher the expectations these instructors set, the greater the absolute n u m b e r of readings their students say they r e m e m b e r (see Figure 2). This finding, at first glance, appears to overturn hypotheses that a high course workload deters from learning. However, this conclusion m a y be too hasty. T h e m e a s u r e used here, t h o u g h s u i t e d t o a q u i c k o n e - t i m e - o n l y i n - c l a s s s u r v e y , is c r u d e . Reports of w h e t h e r an article is r e m e m b e r e d m a y be incorrect, r e m e m bering an article is not necessarily the s a m e as having read it, and even r e a d i n g an article d o e s not g u a r a n t e e learning deeply, as o p p o s e d to " s k i m j m i n g ] along the ' s u r f a c e ' of t h i n g s " ( C h a m b e r s , 1992, p. 145). Further, the m e a s u r e accounts for neither the n u m b e r of pages in a typical reading in a discipline, nor the additional texts or novels set in most courses. Additional research that a c c o u n t s f o r these factors is needed f o r the relations between workload, reading, and learning to be e x a m ined more effectively. T h e timing o f readings. The six instructors' decisions about reading requirements are also informed by their beliefs about h o w students' reading ebbs and f l o w s with the a c a d e m i c year. For example, one instructor f r o m "Social S c i e n c e " c o u r s e believed that students w o u l d read most late in the fall term, while a " H u m a n i t i e s " instructor expected students to read least in N o v e m b e r and March. The Canadian Journal of Higher Education VolumeXXXIII, No. I, 2003 Figure 1 Percentage of Readings Recalled, by Number Assigned 100 95' o L U _J < 90' o LU CC co S «i O » a a. a' 3 Si S a b s O Q S- o t>1 I" C s a K > * < = > 5' u. s CD 85" 0 •Z. Q < UJ 80" cn UJ o < o CC I' 1 Si. 75' s I' 70 H I 65 H i? 6010 15 20 NUMBER OF READINGS PER TERM 25 30 Ï 38 K. Bischoping -a o <u •O >> -D •o "3 u 4 » 02 Ml •5 a u a a n v o a a S9Niav3y j o aaaiAinN œ (N J O jQ = E an s il Z The Canadian Journal of Higher Education Volume XXXIII, No. I, 2003 Selecting and Using Course Readings 39 I divided the readings f o r each course into six g r o u p s of r o u g h l y equal size, according to where the reading fell on the list of kit materials f o r the year. 3 T h e first sixth c o r r e s p o n d s r o u g h l y to r e a d i n g s set f o r S e p t e m b e r ; the next sixth, to O c t o b e r , and so forth. T h e g r o u p i n g is rough because instructors might drop readings during the year, change their order, or assign heavier reading loads at times when they believe students read more. T h e m e a n p e r c e n t a g e s o f s t u d e n t s in each c o u r s e w h o said t h e y remembered readings from each ' m o n t h ' are shown in Table 1. Variation a m o n g the courses appears much m o r e marked than any time-linked tendency. T h e exception is that students in full-year courses appear to report r e m e m b e r i n g f e w e r readings toward the end of the year. This may be a m e t h o d o l o g i c a l artifact, as s t u d e n t q u e s t i o n n a i r e s w e r e a d m i n i s t e r e d before the March readings had been covered completely. Even so, the "Social Science A " instructor argued that there appeared to be a genuine drop in recall toward the end of the year: T h e y had t w o assignments in February and March, based on readings, so it's odd to see they say t h e y ' r e not reading them, even though they had to use all relevant course materials for the assignments, which were given before you c a m e to do the survey in March. And most used them, too. W h e n t h e s a m p l e o f i n s t r u c t o r s learned w h a t s t u d e n t s r e p o r t e d remembering of their own course readings, they began to question their usual methods of assigning readings. For example, t w o instructors w h o s e students' relatively poor recall patterns are shown in Figure 3 said: Probably I should pile it on at the beginning. It slipped out of my consciousness, the crowding [of readings assigned at the end of the year]. It suggests I should average out the readings, look at the n u m b e r of pages per week, and figure out what I can do. I will work on being more scientific about the readings and w e i g h t t h e m t o w a r d the b e g i n n i n g o f the c o u r s e . ("Humanities B") and M a y b e I ' m t o o k e e n or t o o s o f t . W h a t d o t h e o t h e r s d o ? Assign less? O r w h a t d o they do to m a k e students read the The Canadian Journal of Higher Education Volume XXXI11, No. 1, 2003 Table 1 Mean Percentages of Students who can Recall Reading, by Approximate Time of Year that Reading was Assigned, Year and Course September November January February March "students "readings 73.8 73.8 67.4 72.9 91.2 69.8 75.3 79.5 86.5 89.5 86.2 84.8 66.3 59.2 88.4 77.0 96.5 81.5 90.3 61.3 76.2 86.3 83.5 84.2 82.7 89.2 58.3 60.7 82.6 69.2 78.9 27 15 16 56 23 19 19 20 26 32 36 39 44 39 79.0 91.4 85.1 94.3 74.4 84.4 81.5 91.7 79.0 86.7 71.8 87.1 72.8 82.6 90.3 86.7 72.5 70.0 89.2 92.1 62.2 50.0 72.4 94.2 20 72 39 20 57 27 28 39 83.6 79.0 80.6 81.0 80.7 72.8 326 387 1998-1999 "Business A" "Business B" "Humanities A" "Humanities B" "Social Science A" "Social Science B" "Social Science C" 71.3 82.1 83.2 84.2 82.0 1999-2000 "Humanities A" "Humanities B" "Social Science A" "Social Science B" Unweighted average of all courses October Note: "Business A" and "Business B" were winter term courses, not full-year courses. Figure 3 Highlights of Recall Trends 100 •o _ < d "câ o a> cn co CD £ S O a s a a. a' s ^ a a c: T3 ro o> DC. o < D cn ro C oD q3 CL 90 AVERAGE for ALL COURSES 80 0 I' § 70 S I' 60 ^ fc 3: 50 cn I" K) o a S' s 1 'Humanities B' 99-00 September October November January Approximate Month T February March fO S 42 K. Bischoping stuff? Your graph is very interesting because it forces us to ask other questions, for example, should we give assignments on the readings? In [my discipline], when w e give assignments they are supposed to go to the library. ("Humanities A " ) In contrast, the "Social Science B " instructor, w h o s e 1 9 9 9 - 2 0 0 0 students reported r e m e m b e r i n g the highest proportion of readings (see Figure 3), thought a reason might be: T h e organization of the course. We discuss the readings, students have to do presentations on the readings, it's a c o m p u l sory thing. T h e y have to read three readings at least [for the presentations], it's not m u c h , but...And also I think the fact that I ask q u e s t i o n s a b o u t w h a t t h e y u n d e r s t a n d and w h a t they d o n ' t , and then I a n s w e r them in the next class, q u o t i n g the authors. 1 9 9 8 - 9 9 v s 1 9 9 9 - 2 0 0 0 . Student recall of the readings w a s greater in 1 9 9 9 - 2 0 0 0 than in 1 9 9 8 - 9 9 f o r each of the f o u r classes surveyed (Table 2). The most dramatic improvements in reports of r e m e m b e r i n g readings occurred in "Social Science B : " 6 4 % of readings were said to be r e m e m b e r e d by at least t h r e e q u a r t e r s o f s t u d e n t r e s p o n d e n t s in 1 9 9 8 - 9 9 , w h i l e in 1 9 9 9 - 2 0 0 0 , t h i s h e l d f o r 9 5 % o f r e a d i n g s . T h e improvements in other courses ranged f r o m 12 to 19%. These changes were accompanied by substantial increases in students' ratings of readings they recalled. Students w h o said they remembered a given r e a d i n g had indicated w h e t h e r it s h o u l d be kept ( c o d e d 2), or dropped (coded 0), or whether they had no opinion on the matter (coded 1). Using the 1 9 9 8 - 9 9 student data, I grouped readings in each course a c c o r d i n g to w h e t h e r they fell in the top quartile, bottom quartile, or middle of the c o u r s e ' s ratings. 4 In Table 3, w e see that when the 1 9 9 8 - 9 9 c u t - o f f p o i n t s f o r t h e q u a r t i l e s in e a c h c o u r s e w e r e a p p l i e d in 1999-2000, ratings had improved. In " H u m a n i t i e s A , " "Social Science A,'' and "Social Science B," the percentages of readings with high scores rose by 24, 10, and 29, respectively. In " H u m a n i t i e s B," the improvement occurred in the category for moderate scores. F a c t o r a r t i f a c t ? A l t h o u g h t h e c h a n g e s b e t w e e n 1 9 9 8 - 9 9 and 1 9 9 9 - 2 0 0 0 are impressive, they should not be attributed too quickly to The Canadian Journal of Higher Education Volume XXXIII, No. I, 2003 Selecting and Using Course Readings O ? 02 os < u Os o Os c O o O N o o U-l Os 43 "3- < L > 1 1£ 0s H O O O vo OO OS 3 Os o ? < UO S ox Os r- O S ô o Os 1 on O O Os 0s vo < o Os c Os .a — '5 © o I m g: u o O S ( U — Os Os I S O O S O S x o C) so r l 2 oo (N o o oo (N O N m _ _ © < N T T T O O Csl 00 Csll o o vo m r-m Tt- 0s- O S ^ < C 3 / £ O S U — Os Os I O O Os Os £ s? § fe f 1 Ë is § -S S § 0s (N (N O Tl O S C 3 < L > s 0* • o o o O O (NI s 0 © so tU -a c D o o < N m O J ) c '•B ca 4 > The Canadian Journal of Higher Education Volume XXXIII, No. I, 2003 5= o 5" | 0 5 \ ^ ^ S s- Table: P) ?! S § Distributions of Students' Assessments of Readings They Recalled, Before and After the Research Intervention, by Course 5 Mean Score "Humanities A" 1998-99 1999-00 "Humanities B" 1998-99 1999-00 "Social Science A" 1998-99 1999-00 "Social Science B" 1998-99 1999-00 S3 C s aI' High S-s fn Ia 5' 3 25% 49% 33% 30% 26% 36% 25% 54% Medium 44 42 25 44 51 46 61 33 Low 31 _9 42 26 23 18 14 13 100 100 100 100 100 101 100 100 (32) (57) (36) (27) (39) (28) (44) (39) "readings Note: The cutoff points for high and medium scores, respectively, were: 1.8 and 1.4 in "Humanities A;" 1.6 and 1.4 in "Humanities B;" 1.8 and 1.3 in "Social Science A;" and 1.6 and 1.2 in "Social Science B." Selecting and Using Course Readings 45 my intervention of providing instructors with systematic student feedback. In this section I address briefly two methods artifacts that could also explain the improved ratings. First, the students participating in this study were self-selected, rather than selected randomly. This means that c h a n g e s in h o w s t u d e n t s s e l e c t e d c o u r s e s b e t w e e n 1 9 9 8 - 9 9 and 1999-2000 could contribute to the improved recall and assessment of readings in 1999-2000. For example, "Humanities A," "Humanities B," and "Social Science B" were all taught for the first time by their instructors in 1998-99. The 1999-2000 students might have had more information about the courses and their instructors, and therefore could have matched their course selection to their interests and aptitudes better than the 1998-99 students. Nonetheless, ratings also improved in "Social Science A," which had been offered repeatedly by the same instructor. Second, we must consider whether the improved ratings could result from refinements the instructors ordinarily might have made to their courses, independently of the influence of the systematic feedback. The "Humanities A," "Humanities B," and "Social Science A " instructors thought that they had already finished revising their kits before receiving the systematic feedback. The "Humanities B" instructor made a few more changes after seeing the systematic feedback, but the other two instructors made no further changes and still had higher ratings. This suggests that instructors' ordinary methods of revising course materials, absent of systematic feedback, can sometimes be very successful. However, as we shall see later, at other times, instructors can make spectacularly poor predictions of students' preferences. Finally, we should consider that the two instructors whose ordinary revision plans had concurred with students' recommendations said that participating in the initial interview about their readings had been constructive nonetheless: Instructor: I had a better selection of readings because of this experience with you, because 1 tried to have 50% by men and 50%) by women. Researcher: Why are you taking gender so seriously — it was only a question 1 asked you last time [i.e., not a directive]? Instructor: Your background influences the way you look at some issues. I found it was an interesting challenge. The Canadian Journal of Higher Education Volume XXXIII, No. I, 2003 46 K. Bischoping and Instructor: I think our focused discussion of the course kit at the end of the year really made me rethink a lot. In any case, I believe in the value of more systematic evaluation. I did it intuitively and ad hoc. Researcher: Why do you believe in it, if the results are the same as what you did yourself? Instructor: One reason to do a systematic assessment is so they get a sense their f e e d b a c k matters, b e c a u s e I d o n ' t always tell them that I'm considering their feedback. Since collégial discourse about reading kits is rare among the instructors I studied, participating in the first research interview gave some instructors an unusual opportunity to reflect on their pedagogy. Accordingly, the research intervention should be considered to consist of both the initial interview, which invites reflection, and the provision of systematic feedback. This suggestion is consistent with statements that feedback has heightened value when it is accompanied by self-assessment (Brinko, 1993) or consultation (Murray, 1984). Student Feedback and New Instructional Strategies Surprises. Brinko (1993) states that feedback is most effective when it creates cognitive dissonance. Certainly most instructors in the sample seemed to find the surprising student feedback to be the most informative, as the following quotations indicate: First interview: I say to the students, you have to fly casual [with the reading by Hecht], we know each other. Colleagues would say 'the kids can't handle it.' I would say, they're not 'kids' and yes, they can, look at the course evaluations. Follow-up interview, after reading student comments such as 'Hecht was pretty impossible to understand. I didn't get anything out of the reading ': For Hecht, there are massive comments. I say, ok, that could make me drop it. and The Canadian Journal of Higher Education Volume XXXI11, No. 1, 2003 Selecting and Using Course Readings 47 [I expect high ratings for] Anshaw, because students could relate to the experience of these women. I had two mature students w h o said this w a s the e x p e r i e n c e of their parents. Divakaruni and Gilman — again, 1 remember one student who said 'this is the experience my mother had.' [Instructor learns that none of these three are among the most positivelyrated readings and that only a quarter of his predictions about positively-rated readings are correct.] I'm surprised. I don't know what to say. To determine how attuned the instructors were to student preferences, I a s k e d the f o u r i n s t r u c t o r s w h o s e s t u d e n t s had been s u r v e y e d in 1999-2000 to predict which readings students would rate among the best and w o r s t . D e s p i t e his a c c u r a t e p r e d i c t i o n s a b o u t 1 9 9 8 - 9 9 , the "Humanities A" instructor was correct only 11 out of 28 times. Another instructor, who confidently placed half of her students' favourite readings in the worst-rated category, had 11 correct predictions out of 23; a third instructor had just four correct predictions out of 12. The instructor who fared best, with 14 correct predictions out of 20, was dubious about her predictions: "I'm just doing it to get it done, I really don't know." Therefore, although students rated kit materials more favourably in the study's second year, this sample of instructors either had not become conversant with student preferences, or had become fairly conversant without realizing it. From surprises to guidelines. Many of the instructors' conclusions about the systematic student feedback were discipline-specific. For instance, "Business" students' enthusiasm for a reading issued by the O f f i c e of the A u d i t o r G e n e r a l m i g h t not carry o v e r to other programmes. However, in other regards, students made strikingly consistent recommendations. This conclusion is based on the qualitative feedback students gave when asked how they identified readings that should be retained and what other changes to the kit they would recommend. Averaging across courses, 78.0% of students provided such recommendations. Table 4 shows the distribution across courses of major themes (mentioned by at least 30% of students in a course), moderate themes (mentioned by The Canadian Journal of Higher Education Volume XXXI11, No. 1, 2003 48 K. Bischoping Table 4 Distribution Among Courses of Top Themes in Qualitative Student Comments Number of courses where theme is: Theme Major Moderate Absent/minor Less 1 10 1 Easier 6 6 0 More challenging 0 3 9 Better written 5 7 0 Relevance 7 5 0 Discipline-specific 2 7 3 Other 0 2 10 10-29% of students), and minor themes (mentioned by 0-9% of students). As discussed earlier, several students say that they would prefer to have less reading material. However, they were even more concerned that readings be easier, better-written, and 'relevant.' In several courses, students also made discipline-specific recommendations. In responding to this feedback and the lists of most positively- and negatively-rated readings, the instructors first assessed whether they had received any information during the year that would counter-indicate the feedback. For example, one instructor questioned students' poor ratings of a reading that many had used effectively in their exams. Another instructor disagreed with students' high ratings of materials that had, in her view, left them 'completely lost for over four weeks.' The instructors then tried to identify which of their teaching strategies had led to the feedback and to think of ways to change these strategies in response to the feedback. In the following sections on students' main themes, I will show how the instructors strove for consistency with their pedagogical goals of challenging students, reinforcing central course objectives, and exposing The Canadian Journal of Higher Education Volume XXXIII, No. 1, 2003 Selecting and Using Course Readings 49 students to diverse sources. At times, this process made salient differences between students' and instructors' orientations to learning — the former, characterized variously in the literature as mercenary, consumerist, literal, or practical, and the latter, depicted as passionate, learning-oriented, holistic, or analytical (Cruse, 1987; Child & Williams, 1996; Franz et al., 1996; L i o w e t a l . 1993). 'Shorter, easier, and better written.' Although students' concerns about the amount of reading have been addressed, they are worth revisiting to see h o w length, difficulty, and poor writing c o m p o u n d one another. Students' remarks included: "For me, 1 found the reading very hard to understand and with the length of the articles I was not able to finish the majority of them;" "Too lengthy becomes boring and tedious, because most of the time, people are just repeating one idea over and over;" and "The other course material is just too monotonous and difficult to understand." These statements are corroborated by reading research: Hartley and Trueman (1992) find that many psychology journal articles are better suited to graduate readers than undergraduates, when their reading ease is systematically estimated. Bean (1996) remarks that students are 'baffled by the strangeness and complexity of primary sources and by their unfamiliarity with academic discourse' (p. 133). The instructors I studied formulated several approaches to these interrelated concerns. As a last resort, they suggested dropping a reading — "Fielding is brilliant but the students have been complaining about it for five years. They don't get it. It's finally out. Seth is poorly written and I decided just to can it." Several instructors who endorsed this emphasized that they were not 'catering' to students. A less drastic approach was to make the rationale for assigning a reading more explicit: In a second year course, students should be reading articles, identifying the thesis, looking at the author's approach, but maybe the course still does not give them enough on how to do that. For example, there's a reading on human sacrifice in which the author is trying to explain how religious world v i e w s lead to sacrifice, and it is only at the end that he describes sacrifices. I pointed out that this was a rhetorical The Canadian Journal of Higher Education Volume XXXIII, No. /, 2003 50 K. Bischoping decision. They should be challenged with difficult readings but something that comes through is that, if it is difficult because they're left on their own to judge it or if it is analyzed but insufficiently, that's bad. Unknowingly, these instructors were recommending strategies consistent with reading research findings. For example, Schnotz et al. (1993) found that successful learners in a laboratory exercise were distinguished, not by the quantity of information they could command, but by their ability to determine which information could best be used to construct mental models. In practice, Bean (1996) suggests that students be taught how experts read — that is, how they vary their reading speed for different parts of an article, how they adjust to primary and secondary sources, how they identify the rhetorical function of text sections, and so forth. 'Readings should be relevant.' Students invoked the concept of 'relevance' in two main ways. Often, students regarded a reading as relevant if it was taken up in class or if its concepts were reinforced in course requirements, as these quotations suggest: "Some were irrelevant articles. If the readings are not mandatory do not include them in the course kit and expect us to purchase it. Last semester's kit cost $65 and only some of these readings were compulsory;" "It became obvious in tutorials which readings seemed valuable to the overall course. Noticing the amount of discussion on each of the readings showed that some were more important to the course;" and "I based my opinions on the assignments that were given out. I think that a lot of the readings were just other information that was not useful for assignments and exams." The students' emphasis on this type of relevance in their qualitative comments mirrors Liow et al. (1993) conclusion that students regard exam preparation as a more important course goal than do instructors, as well as Williams' (1997) finding that many students feel they would prepare better for tutorials if their preparatory r e a d i n g work w e r e graded in some way. The students' comments are also confirmed by their quantitative ratings of course materials. Instructors discovered that some surprisingly positive or negative ratings could derive from how well they had explicated the readings in lectures or required their use in assignments or exams: 5 The Canadian Journal of Higher Education Volume XXXlll, No. 1, 2003 Selecting and Using Course Readings 51 Maybe that's why they recall the last three, they're all used in an assignment. That's true of all the other most positively rated ones except Choy, Roberts, and Woodrow. Woodrow is a really interesting discussion of meat-packing, and 1 really get into it in class, these carcasses coming along the assembly line. and [I did not expect students to rate the case studies so positively] because the cases are much more demanding, and there's more pressure in class to discuss them. But I give them the 12 in class and I tell them that one of them is going to be on the final so usually they study them quite well. Relevance: Take Two. When requesting 'relevant' readings, students also voiced strong preferences for the familiar, the contemporary, and the personally meaningful, such as: "Any current events might be included during the time they are happening, especially if there is a relation to the [disciplinary] context;" "Some are very outdated. It is hard to discuss in 1999 about ideas as old as 1992;" and "Materials added I would like to see would be articles that are less philosophical, and that pertain more to society, controversies, issues that are going on in our lives." These instructors applauded students' interest in having academic work speak to their experiences — cases in point are that a "Humanities" instructor was pleased to see how readings about working-class women resonated with some students, and that a "Business" instructor aspired to design courses that students could apply at work. Nonetheless, several instructors insisted that students should develop the analytic skills to deal with both the personal and the general, both the familiar and the strange: What they like is related to, except for some surprises, the exams or things that are easy, or experiential work, which is good. At the same time, they have to go beyond their experiences, to think outside them in a certain sense. The theory is not something we can forget if we are studying at the university level and it is a framework that helps to organize the way we see the world. and The Canadian Journal of Higher Education Volume XXXIII, No. I, 2003 52 K. Bischoping [On a negatively-rated reading about McDonald's restaurants] Some students working at McDonald's had a happy experience there and they disagreed with [the author, w h o criticizes McDonald's], People in the class said, 'but my experience is x' and 1 tried to make students aware that to reject a reading because of its method of using one person's experience, and yet to base your analysis on your own experience, is a bad analysis. In response, one instructor speculated that students' interest in contemporary substantive topics could be an avenue for introducing difficult concepts or methods: "I'm sobered by the universal voice that the material is abstract. If I were clever, I would use the desire for the contemporary and feed it back into the course. For example, [by analyzing] ads that use the Adam and Eve myth." These instructors' ambivalence about students' desire for 'relevance' h a s its c o u n t e r p a r t in t h e p e d a g o g i c a l l i t e r a t u r e . O n e g r o u p of researchers locates students' personal experiences at the heart of learning. Boud (1993) positions experience as the basis of knowledge; Franz et al. (1996) proposes that learning is the process of finding personal meaning in material; and Baxter Magolda (1992) illustrates how maturing undergraduates increasingly see themselves (rather than authorities) as the source of knowledge. In the classroom, Hollander (2000) and Widerberg (1998) are among the many who use personal journals or stories as teaching tools. Other researchers decry students' penchant for the personal. Alexander et al. (1994) warn that 'seductive details' and personally involving content can detract from understanding material that should fascinate students on its own merits. Bean (1996), who refers to 'the 'cognitive egocentrism' of new college students who have trouble w a l k i n g in the shoes of persons with u n f a m i l i a r v i e w s and v a l u e s ' (p. 135), considers this part and parcel of a normal developmental stage that can be fostered by effective pedagogical strategies. Finally, students' comments about personal relevance and their discipline-specific recommendations at times mentioned identity politics issues. Examples include requests that the Bible not be used because it does not reflect all students' faiths, that a person of colour delete material on race, that a Québécois instructor place less emphasis on information The Canadian Journal of Higher Education Volume XXXIII, No. 1, 2003 Selecting and Using Course Readings 53 about Québec, and that a feminist change her course readings because 'a lot of the femininity material isn't as important'. Cumulatively, these comments indicate that students can regard readings as statements about the bodies or identities in the classroom, with the potential to feel or to create a chilly climate. Some of the comments also illustrate Magnusson's (1999) point that students can use teaching evaluations to resist curricula that address systemic inequities, with implications for tenure decisions. CONCLUSION When choosing reading materials, the instructors in this sample from five disciplines were informed by pedagogical objectives, such as challenging students, exposing students to diverse sources, selecting material at a suitable level of reading difficulty, and reinforcing central course themes. Less obviously, their choices were informed by disparate nonpedagogical factors, including the desire to reduce students' expenses, isolation from other instructors with different statuses, and dissatisfaction with the format or implicit meaning of student evaluation methods. Another factor — the use of sex (or other author identities) — calls into question the very distinction between pedagogical and non-pedagogical factors, as this sample of instructors came to no consensus as to whether authors' identities influenced learning. Little is known about university student reading patterns outside the laboratory context. In this study in naturalistic classroom contexts, I examined how recall of readings is affected by the quantity of readings required for a course and by timing in the academic year. Students' recall of readings was found to be unaffected by workload: regardless of whether a course required 7 or 57 readings, or quantities in between, students remembered roughly the same percentage of readings. This finding appears to overturn C h a m b e r s ' (1992) and others' proposals that an excessive workload can detract from learning, though further study is indicated because of the cude measures that were employed. Further, patterns of student recall over time vary greatly from course to course, but show a possible decline for readings assigned late in the academic year. The Canadian Journal of Higher Education Volume XXXI11, No. 1, 2003 54 K. Bischoping Students' reported remembering and ratings of course materials improved markedly from 1998-1999 to 1999-2000. Some of the improvement might be attributed to a research intervention in which I provided instructors with systematic feedback f r o m students. Alternately, the improvement might be attributed to the revisions instructors ordinarily make to their kits in the absence of systematic feedback. Yet, the instructors tended to indicate that their discussions of the feedback had been helpful and to find that the feedback included many revelations about students' stated preferences. Therefore, instructors may well benefit from reflective discussions about their reading selections, systematic student feedback about readings, or both. Because course evaluation questionnaires typically neglect reading (e.g., Marsh & Bailey, 1993; Wilson et al., 1997), even when they purport to focus on an instructor's course design rather than lecture performance (e.g., Liow et al., 1994), this conclusion points to a valuable new avenue for formative evaluation research. Across disciplines, the systematic feedback students provided was remarkably consistent. Students asked for shorter, easier, better-written, and more 'relevant' materials. The instructors I studied did not take these requests at face value, but instead, interpreted them as indicators of underlying issues of instructor-student communication, students' motivations, and students' cognitive abilities. Accordingly, they devised several responses to students' requests, including giving students more information about how to read, avoiding readings that are both long and complicated, ensuring that course readings are taken up in lectures or connected to graded work, and using the draw of familiar, personal, and contemporary topics to introduce complex analytic methods. An instructor perusing this manuscript for its practical recommendations might find many helpful, or at least benign, pointers: discuss your reading selection with someone, test your predictions about how the sex (and other identities) of authors are distributed, be aware that students may recall late readings least, and so forth. However, of greater sociological interest is a more abstract recommendation: that instructors recognize that their choices of readings, students' use of and responses to readings, and instructors' revisions to reading materials and pedagogical strategies all entail certain a s s u m p t i o n s about how to c o n c e i v e of The Canadian Journal of Higher Education Volume XXXIII, No.1,2003 Selecting and Using Course Readings 55 instructors' and students' roles. For example, if a student is thought of primarily as a "consumer" and an instructor as a "provider of a product," in a consumer-driven market, course readings would be revised simply to match students' tastes. As Child and Williams (1996) argue, such a model discourages critical thinking. The academy is without consensus on this point. Throughout this manuscript alone, the research participants and cited literature cast students and instructors in complex and sometimes contradictory roles. Either party may be seen as a potential victim or promulgator of a chilly climate. A student may be depicted as, at once, mercenary, egocentric, overwhelmed, independently knowledgeable, and wary of providing feedback. An instructor may be, at once, harried, eager to challenge, isolated by colleagues, embattled by consumerism, or surprised by the difficulty of predicting students' preferences. This recommendation differs from the others in a second way. It is not directed to an instructor taking individual initiative to "talk to someone about my readings," so much as to instructors collectively w h o could engage in a larger conversation that identifies how political and institutional factors — such as government funding decisions, enrolment rates, tenure procedures, or methods of allocating merit pay — influence how the instructor-student relationship is conceptualized and the resulting pedagogical practices. As Melrose (1998) has argued: if education is to be conceived of as emancipatory or transformative, instructors must be empowered by their organisation and by t h o s e e x t e r n a l to it to act to c o n t i n u a l l y change and improve teaching and learning processes f o r their students....If the employer or the industry as customer or c o n s u m e r is the only determinant of curriculum goals then teachers themselves are in danger of being disempowered to n e g o t i a t e a b o u t the c u r r i c u l u m as p r o f e s s i o n a l educators, (p. 42) ^ The Canadian Journal of Higher Education Volume XXXI11, No. 1, 2003 56 K. Bischoping Notes ' Response rates are approximate because 1 did not record the number of registered students in each course, but instead used the maximum possible enrolment in the denominator of the estimate. 2 These scores were calculated by assigning 2 points if a student recommended keeping a reading, 1 point if a student had no opinion, and 0 points if a students recommended dropping a reading, and then taking the average of those points. Students who did not remember a reading were omitted from this calculation. J One "Humanities" course is omitted here because the sample of only seven readings is unsuited to so detailed an analysis ^ Because of ties among ratings, the groups of readings are not precisely of the desired sizes. ^ Although my sample of instructors found that readings taken up in class were well-rated, Bean (1996) recommends making students responsible for material not covered in class to break "the vicious reading cycle...teachers explain readings in class because students are poor readers; students read poorly because teachers explain the readings in class" (p. 139). References Alexander, P., Kulikowich, J.M., & Schulze, S.K. (1994). How subjectmatter knowledge affects recall and interest. American Educational Research Journal, 31(2), 313-337. Andersen, M. (1988). Changing the curriculum in higher education. In E. Minnich, J. O'Barr, & R.Rosenfeld (Eds.), Reconstructing the academy, (pp. 36-68). Chicago, IL: University of Chicago Press. Baxter Magolda, M.B. (1992). Students' epistemologies and academic experiences: Implications for pedagogy. The Review of Higher Education, 15(3), 265-287. Bean, J. (1996). Engaging ideas: The professors guide to integrating writing, critical thinking, and active learning in the classroom. San Francisco, CA: Jossey-Bass. The Canadian Journal of Higher Education Volume XXXIII, No. 1, 2003 Selecting and Using Course Readings 57 Beishuizen, J., Stoutjesdijk, E., & Van Putten, K. (1994). Studying textbooks: Effects of learning styles, study task, and instruction. Learning and Instruction, 4, 151 - 1 7 4 . Boud, D. (1993). Experience as the base for learning. Higher Education Research and Development, 12( 1), 33-44. Brinko, K.T. (1993). The practice of giving feedback to improve teaching: What is effective? The Journal of Higher Education, 64(5), 574-593. Chambers, E. (1992). Work-load and the quality of student learning. Studies in Higher Education, 17(2), 141-153. Child, M., & Williams, D.D. (1996). College learning and teaching: Struggling with/in the tensions. Studies in Higher Education, 21(1), 31 —42. Cruse, D.B. (1987). Student evaluations and the university professor: Caveat professor. Higher Education, 16(6), 723-737. Dorn, D. (Ed.) (1989). Textbooks. The Teaching Newsletter, 2(3), 1-3. Fisher, B. (1988). Wandering in the wilderness: The search for women role models. In E. Minnich, J.O'Barr, & R. Rosenfeld (Eds.) Reconstructing the academy, (pp. 234-256). Chicago, IL: University of Chicago Press. Francis, H., & Hallam, S. (2000). Genre effects on higher education students' text reading for understanding. Higher Education, 39, 279-296. Franz, J., Ferreira, L., Loh, H., Pendergast, D., Service, M., Stormont, D., Taylor, L., Thambiratnam, D., & Williamson, B. (1996). Students' and lecturers' conceptions of learning in context: An interdisciplinary study. Teaching in Higher Education, 1(3), 325-339. Gibbs, G. (Ed.) (1992). Improving the quality of student learning. Bristol, England: Technical and Educational Services Ltd. Hartley, J. (1998). Learning and studying: A research perspective. New York, NY: Routledge. Hartley, J., & Trueman, M. (1992). Some observations on using journal articles in the teaching of psychology. Psychology Teaching Review, /(I ), 46-51. Hollander, J. (2000). Fear journals: A strategy for teaching about the social consequences of gendered violence. Teaching Sociology, 28(3), 192-205. Liow, S.R., Betts, M., & Lit, J.K.L. (1993). Course design in higher education: A study of teaching methods and educational objectives. Studies in Higher Education, 18( 1), 65-79. Magnusson, J.-L. (1999). The evaluation of university teaching: Exploring the question of resistance. In K. Armatage (Ed.), Equity and how to get it: Rescuing graduate studies, (pp. 194-212). Toronto, ON: Innana Publications & Education, Inc. The Canadian Journal of Higher Education Volume XXXIII, No. I, 2003 58 K. Bischoping Mann, S.J. (2000). The student's experience of learning. Higher Education, 39, 297-317. Marsh, H.W., & Bailey, M. (1993). Multidimensional students' evaluations of teaching effectiveness: A profile analysis. The Journal of Higher Education, 64( 1), 1-18. Marton, F., & Saljô, R. (1976) On qualitative differences in learning: 1 — outcome and process. British Journal of Educational Psychology, 46, 4-11. Melrose, M. (1998). Exploring paradigms of curriculum evaluation and concepts of quality. Quality in Higher Education, 4( 1), 37-43. Murray, H.G. (1984). The impact of formative and summative evaluation of teaching in North American universities. Assessment & Evaluation in Higher Education, 9(2), 117-132. Murray, H.G. (1997). Does evaluation of teaching lead to improvement of teaching?. The International Journal for Academic Development, 2(1), 8-23. Schachter, S., Christenfeld, N., Ravina, B., & Biloux, F. (1991). Speech disfluency and the structure of knowledge. Journal of Personality and Social Psychology, 60(3), 362-367. Schnotz, W., Picard, E., & Hron, A. (1993). How do successful and unsuccessful learners use texts and graphics? Learning and Instruction, 3(3), 181-199. Smeby, J.-C. (1996). Disciplinary differences in university teaching. Studies in Higher Education, 2/(1), 69—79. Starr, M.S., & Rayner, K. (2001). Eye movements during reading: Some current controversies. Trends in Cognitive Sciences, 5(4), 156-163. Thomas, K. (1990). Gender and subject in higher education. Bristol, PA: Society for Research into Higher Education and Open University Press. Widerberg, K. (1998). Teaching gender through writing 'experience' stories. Women's Studies International Forum, 21(2), 193-198. Williams, A.M. (1997). Making the most of assigned readings: Some alternative strategies. Journal of Geography in Higher Education, 21(3), 363-371. Wilson, K.L., Lizzio, A., & Ramsden, P. (1997). The development, validation and application of the course experience questionnaire. Studies in Higher Education, 22(1), 33-53. The Canadian Journal of Higher Education Volume XXXIII, No.1,2003