Evaluative Devices as Learning Opportunities DENNIS L. FOTH* Abstract Thirty-three third- and fourth-year university students reported their preferences for eight evaluative devices which they had suggested might be used in a course they were taking. Students were instructed to base their preferences on the learning opportunities afforded by the different devices. The four most preferred evaluative devices comprised a pool of options from which students were required to select and complete two in addition to writing a final examination. The results of both the initial preferences stage and the actual choices stage suggested that the majority of students viewed the preparation for in-class or take-home examinations as better learning opportunities than the preparations for writing a paper, a research proposal or an annotated bibliography, for presenting a seminar, and for bi-weekly quizzes or an oral examination. Most students and instructors assume that an impending examination, seminar presentation, or paper deadline aids in motivating a student to indulge in activities which result in the acquisition of some subject matter. On the basis of a student's performance on such tasks, the instructor in some way "measures" what and how much the student has learned. Subsequently, the instructor usually informs the student as to how he has performed on a given task relative to some internal standard the instructor has, and often in relation to the performance of other students taking the same course. After a number of such tasks, the instructor is called upon to make a judgement as to the student's competence and to report this judgement to a principal, a dean, a registrar, or even a potential employer of the student. Consequently, the performance tasks instructors request students to do may be considered as serving two primary functions: motivational — to encourage behavior which results in learning, and evaluative — to provide a basis upon which competence judgements can be made. This is not to say that this type of system is necessary for learning or that examinations, seminar presentations, and papers are the best means to evaluate student competence. However, because this type of system, or a portion of it, ' T h e author would like to thank Professor Stephen Foster of the Faculty of Education, University of British Columbia, for his c o m m e n t s on an earlier draft of this paper. Appreciation is also due to Mr. Brian Sutherland, the teaching assistant for the course, w h o had to spend many more hours than expected consulting with students on paper and research proposal topics. *Assistant Professor, Department of Psychology, The University of British Columbia. 66 Dennis L. Foth is used extensively in most colleges and universities as well as in most primary educational institutions, it does warrant consideration per se. The intent of this paper is to examine, in an exploratory fashion, which of a number of evaluative devices (tasks to measure competence) students view as providing the best opportunity for learning. That is, which evaluative devices do students prefer in terms of the potential of such devices to encourage and result in the greatest acquisition of knowledge on the part of the student. It is assumed that there exists a strong positive relationship between the amount of knowledge acquired and grades obtained by students on most, if not all, evaluative devices and that students are aware of, and believe in, this relationship. It is also assumed that the majority of students are motivated to attain the highest grades possible. A second question examined is as follows: if students have options with respect to a number of evaluative devices which they prefer in terms of the learning opportunities afforded (such devices are hereafter referred to as learning-evaluative devices), which ones do they actually choose to do? In other words, what is the correspondence between students' stated preferences and actual behaviors with respect to selection of learningevaluative devices. Information pertinent to these questions was recently obtained from undergraduate students in their third or fourth years of university study. The selection of such a group was both purposeful and convenient: most senior students have, by virtue of their years at college and university, been exposed to a variety of evaluative devices currently employed at this level (for example, in-class and take-home written examinations, papers, seminar presentations, annotated bibliographies), and the author was teaching a course to such a group of students. The procedure, which is described in more detail below, was essentially very simple. Students were asked to consider that the preparations involved in getting ready to take an examination, to write a paper, to present a seminar, et cetera, were really learning opportunities. They were also to consider that such preparations might differ as to how much could be learned through them. With these considerations in mind, each student then submitted four choices for learning-evaluative devices, rank ordered as to preference. The four most preferred choices, combined across all students, were then considered as options from which the student had to select two in addition to writing a final examination. The data of interest were the student preferences and the actual choices from the list of preferred options. The course the students were taking was the second semester of a two-semester course on the psychology of learning. The first semester (12 weeks duration) had dealt with historical and contemporary theoretical approaches to learning. The second semester (13 weeks duration), taught by the author, dealt with various facets of human learning. Lectures were given for three fifty-minute sessions per week. The assigned readings for the second semester of the course consisted of about 600 pages of textbook material and selected journal articles. There was no laboratory requirement associated with the course. Thirty-eight students started the second semester and 35 completed it. Of these 35 students, two failed to submit preferences for learning-evaluative devices. Hence, the data presented below consist of preferences and actual choices of 33 students. All of the students were registered in the Faculty of Arts and all but one were majoring in psychology. Thirty of the students were registered as third-year students and three were in their fourth (graduating) year. The two sexes were almost equally represented (17 67 Evaluative Devices men and 16 women). The age range was 19.6 to 35.7 years with the mean and median age being 22.6 and 21.4 years, respectively. During the first lecture session of the second semester of the course the instructor previewed the course content, indicated what he hoped to accomplish, and what his expectations of the students were. A great deal of emphasis was placed on the students being able to integrate material which would be presented during the course. The second lecture session was devoted to a consideration of the devices that the instructor might use to assess student competence. A final examination was mandatory according to regulations. However, the weight assigned to the final examination was at the discretion of the instructor with usual values falling between 30 and 70 percent. The instructor indicated that he was willing to extend this range by 40 percent: that is, from 20 to 100 percent, depending on the wishes of the students. Students were asked to consider, on the basis of their past experience, the opportunities for learning that examinations, papers, seminars, et cetera afforded, before deciding on the weighting for the final examination. The students were also informed that if they desired it, some choice among different kinds of learning-evaluative devices would be allowed. After much discussion among the students, the consensus was that the final examination should be worth 50 percent of the semester grade and that two other types of learning-evaluative devices (each worth 25 percent) also be used. In addition, the students wanted some options from which to make their choices. In order to facilitate the gathering of the requisite information, the instructor proposed that each student list four types of learning-evaluative devices that he would like to prepare for, bearing in mind the learning opportunities afforded by the different devices. Students were asked to rank order their four choices ("one" being most preferred) and to submit this information to the instructor at the next lecture session. The instructor also proposed that the four learning-evaluative devices having the lowest mean rankings (that is, most preferred) would form the pool of options from which each student would select two. Both proposals met with student approval with the modification that the four choices they would select and rank order should come from some longer agreed upon list of possibilities that they would derive. The possibilities ultimately agreed to by the students in the course were the eight which are listed in Table 1. It is worth noting in passing that the learning-evaluative device list agreed to by the students contains only tasks that would allow the instructor to use some objective basis for assigning a substantial portion of a grade. Table 1 shows the distribution of first, second, third, and fourth preferences for the eight proposed learning-evaluative devices as submitted by the 33 students. They are arranged from most to least preferred on the basis of mean ranks. The distribution of ranked preferences does differ significantly from that which would be expected if students simply randomly selected four evaluative devices from the eight possibilities and randomly rank ordered them ( x 2 = 78.82, d.f. = 31, p < .001, corrected for small expected frequencies, expected frequencies based on random selection and random ranking of selections by 33 hypothetical students). An examination of Table 1 indicates that the four most preferred learning-evaluative devices, both in terms of the number of students expressing a preference for and in terms of the ranks assigned, were the take-home examination, the mid-term examination, a research proposal, and a term paper. The take-home and mid-term examinations received 68 Dennis L. Foth Table 1 Student Preferences for Learning-Evaluative Devices Type of Evaluative Device Mean Rank Number of Students Ranking Evaluative Device First Second Third Fourth Total Take-home Examination 1.86 12 10 4 2 28 Mid-term Examination 2.28 9 8 7 5 29 Research Proposal 2.38 4 4 6 2 16 Term Paper 2.76 5 5 11 8 29 Seminar Presentation 2.83 3 1 3 5 12 Oral Examination 3.20 0 2 0 3 5 Bi-weekly Quizzes 3.28 0 2 1 4 7 Annotated Bibliography 3.50 0 1 1 4 6 the highest number of first and second ranks of all types of devices. The term paper was preferred by more students than the research proposal as a third or fourth choice. Following the procedures agreed to by the students and the instructor, students were required to complete any two of the four most preferred learning-evaluative devices. The mid-term examination was held about half-way through the second semester and the takehome examination was circulated two weeks before the end of the semester. Five days were allowed for the completion of the take-home examination. Papers and research proposals could be submitted at any time up to and including the day the take-home examination was due. Penalties were applied to late submissions. Students did not have to select the two devices they had initially ranked as most preferred nor did they have to indicate in advance which devices they were going to do. Students were not permitted to do three or four of the four options and take the two best grades. Paper and research proposal topics were not set by the instructor although they were to be within the domain of the 69 Evaluative Devices Table 2 Student Preferences for and Selection of Learning-Evaluative Devices Initial 1st and 2nd Preference Combinations of Evaluative Devices Take-home and Mid-term Examinations Selection 1st and 2nd Preference Change 1st or 2nd Preference Selection after Change Total 15 10 Take-home Examination and Term Paper Mid-term Examination and Research Proposal Research Proposal and Term Paper Take-home Examination and Research Proposal Mid-term Examination and Term Paper Other (22 possible combinations) Total 33 20 13 13 33 content of the course. To ensure that the topics were relevant to the course and that the amount of work would be approximately equivalent to that which the instructor thought would be experienced by students preparing for examinations, all paper and research proposal topics had to be discussed with, and approved by, the teaching assistant. Students had been informed of both reasons for having to consult with the teaching assistant over paper and research proposal topics when these devices had first been proposed by the students during the second lecture session. A summary of the actual selections made by the students and the differences between their selections and their initial preferences is presented in Table 2. The six combinations of the four agreed upon learning-evaluative devices are listed together with the number of students who initially preferred such a combination (that is, ranked the two devices first and second), the number who actually followed through on their initial first and second preferences, the number who changed their choices within the set of six combinations (that is, selected one or both devices from their third or fourth preferences) or who were 70 Dennis L. Foth forced to choose a combination because one or both of their initial preferences were not within the possible combinations, the number selecting a particular combination after changing, and the total number of students who did a particular combination (sum of the second and fourth columns). In terms of final selections, 45.5 percent (15/33) of the students did only examinations and 91 percent (30/33) did at least one examination. Seventy-six percent (25/33) of the students did the take-home examination. Only 9 percent (3/33) of the students opted not to do any examinations at all (other than the required final). Of the students whose initial first and second preferences were incorporated in the final list of four options, 83 percent (20/24) completed their preferences. The four students who switched changed only their second preference. Of the students who were forced to change because either one or both of their preferences were not incorporated in the list of options, all chose a combination that consisted of at least one examination and 6 of the 9 students chose the two examination combination. Eight of the 9 students who were forced to change had initially indicated a first or second preference for one of the two types of examinations among their four choices. The most interesting finding from this investigation is the high proportion of students who prefer and actually do one or more written examinations when they are not absolutely required to do so. On the basis of this finding it might be inferred that students view the preparation for examinations as the best method, among the alternatives considered here, for learning a subject matter. There are, of course, qualifications and alternate explanations that should be considered before accepting such an inference. Firstly, the data presented here are based on a relatively small number of students not randomly selected from the student population. Hence, generalizations to students who are not third or fourth year psychology majors must be made with caution. However, many of the students in the class described here may be generally considered to be liberal arts students rather than unique students of psychology. At this university it is not until the end of the second year that students within the Faculty of Arts are required to make a decision as to the discipline they wish to major in. In addition, at least two of the courses taken by the students in their third and fourth years must be from a discipline other than the one the student is majoring in. Further, since psychology at this university is considered by most, if not all, of the instructors in the discipline as an empirical science, the students have had exposure to methods and modes of thought isomorphic with those espoused by most instructors in a Faculty of Science. Consequently, it is worth considering that the students in this investigation are perhaps more representative of the major part of the student population than might be first supposed. Secondly, students were required to take a final examination in the course and it is possible that many of the students who chose the two-examination option did so in order to obtain familiarity and practice with a particular instructor's questions and expectations for answers. While this is undoubtedly true to some extent, it would seem that one examination would be sufficient to learn about an instructor's examination style and expectations. In addition, it does not seem logical that students who initially had the option to attach minimum weight to a final examination (20 percent) would request a 50 percent final examination without sincerely believing that preparing for examinations would result in learning per se, and which would ultimately be reflected in good grades. Thirdly, the possibility exists that time factors encouraged students to select examina- 71 Evaluative Devices tions rather than, in this case, papers or research proposals. This could certainly be true for those students who elected to do the mid-term examination as it would account for one-quarter of the course grade for the second semester of the course at the mid-point of the semester. However, slightly less than one-half of the students did the mid-term examination whereas three-quarters of the students did the take-home examination. It will be recalled that the take-home examination was handed out near the end of the semester. This time period, presumably, is one in which demands from other courses are also heavy. The student who is concerned about time pressures might indeed select something other than a take-home examination. Students did have three months to work on papers or research proposals and could submit them at any time during this time period. Fourthly, certain courses may be perceived by students as being more amenable to the written examination format than to papers and research proposals. While in this course a certain amount of definitional material had to be learned, the emphasis in the course was on the integration of research findings and concepts. Integration should be more easily demonstrated to the instructor by the student in papers and research proposals than in written examinations. In addition to the emphasis on integration, it was expected that more students would opt for the paper and research proposal than actually did for other reasons. The students in this course were in the Faculty of Arts and thus, many of them had considerable experience in the writing of papers. Further, most of the students were concurrently taking a course in research methods that included, as a course requirement, the writing of research proposals. Of the 66 papers and research proposals that could have been submitted (one of each type for each of 33 students), less than one-third were. Finally, many students may have selected written examinations because such examinations have been "a way of life" to them for the past 15 or more years. While it is probably true that students have had a disproportionate amount of experience with written forms of examinations relative to the other learning-evaluative devices considered here, I think it is also true that students sincerely believe that they can effectively learn material through preparation for examinations rather than because they have been conditioned to expect examinations per se. Relevant to this point are the verbal reports from informal interviews with 22 of the 25 students who did the take-home examination (three students inadvertently were not interviewed). Seventeen of the 22 students reported that they opted to do the take-home examination because it would "force" them to study the course material intensively prior to and during the time period allowed for the take-home examination. As a result, they felt they would be able to obtain good grades on the take-home examination. In addition, these students reported that doing the take-home examination served another purpose: namely, preparing for the final examination. Of the five other interviewed students who did the take-home examination, three reported they did it because they were unable to complete a paper or research proposal on time and two reported they did it because they thought it would be easier than the mid-term examination or take less time than that assumed to be involved in writing a paper or research proposal. The oral examination and bi-weekly quizzes were not generally preferred by students. It will be recalled that no student ranked either device as most preferred and only four students indicated a second-choice preference for them. The fact that few students preferred an oral examination, which ostensibly would present the same learning opportunity as would preparation for a written examination, may be attributed to the relative unfamil- 72 Dennis L. Foth iarity of many undergraduate students with this type of evaluative device and the apprehension among many students about exposing some assumed (or real) deficiency in knowledge of a particular subject matter in face-to-face contact with their "learned" instructor. On the basis of discussions with some of the students in the course, bi-weekly quizzes are not popular because they frequently turn out to be speed tests, the questions are usually superficial and do not allow a student to demonstrate an in-depth knowledge of certain matters, they consume valuable lecture time, and it is difficult to organize worktime efficiently to meet demands from other courses when one or more courses have weekly or bi-weekly quizzes. While most students selected at least one type of written examination, three decided not to do any examinations at all (other than the required final). Discussions with these students suggested that they, were really "examination anxious" and that they believed that they would do much better on a paper and a research proposal than on examinations. Their grades on these tasks relative to their scores on the final examination were in general consonant with these expectations. In summary, most students in this class perceived preparation for written examinations as valuable learning experiences and actually followed their convictions when allowed a choice. Being able to choose among different kinds of learning-evaluative devices, however, may also be important, both for students who find examinations useful devices but who want to do something different as well as for those students who are anxious about examinations to the extent that their performance on them is adversely affected. The matter of choice of learning-evaluative devices and subsequent achievement of students given their choices relative to the achievement of students not allowed a choice is worthy of further consideration. On the basis of the apparent belief on the part of most students in this investigation about the learning that occurs as a result of preparations for examinations and because of their actual selections of examination type learning-evaluative devices, perhaps it would be appropriate for instructors to begin considering what kinds of examinations really do encourage students to learn material. The fact that 76 percent of the students in this investigation chose to do the take-home examination suggests that this type of examination format should be seriously considered if the instructor's purpose is not only to evaluate students but to encourage efficient learning.