Modelling the realities of research experience: collaboration against common and merciless foes JOHN J. FUREDY* and CHRIS FUREDY** ABSTRACT This paper describes a fourth-year undergraduate thesis-research course, the form of which has been changed in recent years to maximise the simulation aspect of undergraduate learning. It is suggested that both the "grudging slave"and "unconstrainedscientific-genius" are ineffective models of genuine research activity, and that an "adversary" model is more appropriate. The adversary model system uses a journalsubmission analogue in which external judgement (the first "foe") is provided by having the thesis marked not by the supervisor but by other faculty in a complex, not perfect, but apparently fair, grading system. The other "foe" of the student is the deadline for submission of the thesis, a deadline which is rigidly enforced by means of very severe grade penalties for lateness. Responsibility for the thesis is thereby shifted completely to the student who, while contending against these "foes", is also engaged in a number of academically significant collaborative relationships. These relationships include that with the supervisor, with fellow students (in research seminars), and with the thesis co-ordinator, who has negligible influence on the final grade assigned, and can therefore function as a non-authoritarian, disinterested critic. Experience with this adversary model system has shown that it is necessary to provide "symbols of significance" for each component (e.g., the research seminar) of the course, in the form of (small) grades. The function of these grades is to ensure that each component is perceived by students as academically significant. The system inevitably produces sources of tension between the student, supervisor, other students, and co-ordinator. However, it is suggested that this feature is potentially beneficial inasmuch as it produces better simulation of real-life research experience, as well as increasing the student's sense of intellectual responsibility for his own work. Finally, the general applicability of the Toronto experience is discussed. It appears that such factors as departmental size, or research eminence of the faculty, are not critical for successful implementation. What is necessary is a general agreement on what constitutes good and bad research, and a commitment to disinterested enquiry and academic values by a sizeable majority of the faculty of the department in question. * D e p a r t m e n t of Psychology, University of T o r o n t o ** Division of Social Science, Y o r k University 80 Furedy and Furedy RÉSUMÉ La présente étude décrit un programme de quatrième année universitaire des méthodes de recherches en vue du mémoire. La structure de ce cours s'est modifiée au cours des dernières années, pour tirer le maximum de l'aspect de simulation de l'enseignement des premières années universitaires. En tant qu 'activité authentique de recherches, n 'est-il pas vrai que "l'esclave-travaillant-à-contre-coeur" et "le génie-scientifique-libre " s'avèrent tous deux des modèles inefficaces. Un modèle dit "d'adversaire", s'apparentant aux pratiques pour évaluer une soumission à une revue, convient davantage. Le système du modèle d'adversaire admet une évaluation venant de l'extérieur (le premier "adversaire"). Il s'agit de l'évaluation du mémoire effectuée, non pas par le patron de celui-ci, mais plutôt par d'autres membres de la faculté qui accordent à la soumission de l'étudiant un résultat en vertu d'une échelle de cotes complexes, imparfaites, mais apparemment justes. L'autre "adversaire " de l'étudiant est la date limite pour la soumission du mémoire, date rigidement appliquée au moyen des baisses importantes du résultat de ceux qui remettent leur travail après cette date. L'étudiant se trouve alors seul responsable de son mémoire. Tout en faisant face à ces adversaires, il s'engage aussi à plusieurs relations académiquement significatives et qui visent la collaboration. Il s'agit des rapports avec le conseiller de ses études, avec les autres étudiants (dans les séances d'étude sur les recherches), ainsi qu 'avec le coordonnateur du mémoire qui a peu d'influence sur la note finale accordée à l'étudiant et qui peut donc agir en tant que critique désintéressé et informel. La mise en pratique du système du modèle d'adversaire a révélé qu 'il faut accorder à chaque composant du cours (ex. les séances d'étude sur les recherches) "des symboles de signification" - ces derniers en forme de notes. Ces notes ont été conçues pour assurer que l'étudiant perçoive la signification académique de chaque composant. Inhérents au système sont les tensions qui se manifestent entre l'étudiant, le conseiller, les autres étudiants et le coordonnateur. N'est-il pas toutefois possible que cet élément s'avère utile dans la mesure où il aide pour la création d'une meilleure simulation de la vie réelle dans les recherches et qu 'il active chez l'étudiant le sens de responsabilité intellectuelle envers son propre travail. En dernier lieu, il est question d'une évaluation de l'applicabilité de l'expérience effectuée à Toronto. Il paraît que des facteurs tels que l'importance du département, la distinction des recherches effectuées par un corps enseignant, ne s'avèrent pas primordiaux pour exécuter le modèle avec succès. Ce qu 'il faut c'est que tout le monde s'entend sur la nature de bonnes et de mauvaises recherches, et qu 'une majorité confortable du corps enseignant du département concerné s'engage d'une façon désintéressée à poursuivre leurs recherches et à s'adhérer aux valeurs académiques. 81 Realities of Research Experience Can undergraduates be taught how to do research by means of a thesis research course? Surely the answer is yes, but one must be clear about what the students are to learn. If emphasis is placed on the content of the course or the ultimate or applied significance of the research project results, the student easily becomes a cog in the supervisor's research machine. Rather what has to be transmitted is a set of attitudes towards investigation, and a set of skills which the student may apply not only to the research problem he is studying for his thesis but also to the problems of different topics in the future. In effect the student should be given the opportunity to experience some of the important and general aspects of doing research. Such an approach is based on the assumption that the undergraduate thesis research course will be successful to the extent that it manages to simulate real-life research. It draws upon the growing number of courses which make simulation, whether informal or highly systematic, an integral part of undergraduate learning. The mere fact that students carry out research under the guidance of a supervisor in a thesis research course is not sufficient to guarantee that they will participate in effective modelling of research activity. One may readily caricature two ineffective models, reflections of which are frequently seen in reality. On the one hand, there is the "grudging slave," who, to the extent that he or she functions at all, merely follows orders from the thesis supervisor. Equally undesirable, from an academic point of view, is the "unconstrained scientific genius" type of student who is allowed such complete freedom of research activity that "anything goes." The unstated assumption seems to be that the mere fact that the project is the student's "own idea" puts it beyond any criticism or constraint. This model of research concentrates on the fostering of positive affect and on a positive attitude towards research activity, which is not exposed to any searching criticism. A little reflection is enough to remind us that real-life research is not modeled by either the "grudging-slave" or the "unconstrained scientific genius" paradigms. Aside from the rare cases of persons working in total isolation (e.g., Mendel in his genetic research), research can be characterized as a collaboration of two or more people (perceived as "us") in an enterprise within an adversary system. An essential component of the experience is the conflict between the researchers and their "adversaries." The principal adversarial factors are: (a) the judgments of other experts in the field (i.e., "peer evaluation") and (b) temporal constraints (e.g., deadlines). These strains, especially in those moments of negative affect which strike all researchers, are perceived as emanating from "them." We should stress that to assert that this "us vs. them" characteristic exists in real-life research is not to deny that the successful research experience ultimately produces a contribution to knowledge that transcends adversarial conflicts and animosities. Indeed researchers do not always experience explicit conflict in every research project. We suggest, however, that the fires of implicit or explicit conflict between collaborating researchers and their adversaries forge understanding of both the nature of research in general, and the complexities of the subject of any one project. These assumptions underlay the revision of the undergraduate thesis course by the first author in the University of Toronto's Department of Psychology (St. George Campus). Work on developing the "adversary model system" began in 1971 when the first author took over the role of course coordinator. The main changes (detailed below in the "Symbols of Significance" section) occurred 82 Furedy and Furedy during the first two "trial" years, and, with minor adaptations, the course has been running according to the "adversary model" since that time. Our two aims in this paper are to describe the system for setting up such an "adversary model" for undergraduate thesis research, and to make some comments on problems which arise in implementing the system in an undergraduate thesis course. The Adversary Model System Journal-submission analogue. The aim is to simulate, with appropriate modifications, the processes entailed in the submission of a paper to a high-quality journal. It is assumed that the research involved is of high quality and that the two "adversaries" that have to be faced are the judgments of external referees, and the time schedules involved in planning, running the experiment and writing up the results for publication. The external judgment component is provided by having the final version of the thesis marked not by the supervisor, but by other faculty who have nothing to do with the thesis. The course coordinator matches these "referees" to the thesis author's area of interest as closely as possible. They are given an anonymous copy, the maximum length of which is 20 double-spaced pages, a little more space allotment than given by most current journals. The only instruction for evaluation given to a referee is that he use standard high-quality journal criteria, except that neither the intrinsic interest or "timeliness" of the topic chosen, nor whether the results turned out to be "significant," should affect the evaluation. In addition to assigning a mark, the referee is asked to make comments either on the thesis or separately. These comments, but not the referee's identity, are transmitted to the student directly without any editing by the thesis coordinator. The grading system is complex: it could be described as a grader-checker-arbitrator system. Each thesis is first assigned to an assessor who is asked to write extensive comments, and to award a percentage grade. The thesis is then passed on to a checker who is asked to read both the thesis and the comments, to add any comments of his own, and, most importantly, to state whether he agrees with the grade awarded within a 10% range. The discrepancy percent value is deliberately set high in recognition of the fact that this type of grading, like the refereeing of journal articles, is an essentially subjective and relatively crude decision. With such a discrepancy criterion, "graderchecker disagreements" occur approximately 8% of the time, with the grader wanting to award a higher grade approximately half of the time. In such cases, the thesis, complete with grader's comments and grade, and checker's comments and suggested revised grade, is passed onto an "arbitrator." The arbitrator is asked to make further comments and may award a grade only in the range represented by the grader's and the checker's judgments. (In fact all arbitrator marks have fallen between the two grades.) All faculty "referees" (from 12 to 15 volunteers from the department) serve both as graders and as checkers, and all potentially serve as arbitrators. "Referee identity," however, is known only to the coordinator who in this respect models an editor. Aside from this both students and referees are provided with full information about the comments, grades, and reasons for assigning those grades. 83 Realities of Research Experience As in the case of real-life journal submission, it sometimes happens that the judgments made do not satisfy the student and/or the supervisor, and on these occasions the student concerned is invited to write to the referee regarding his or her source of dissatisfaction. In addition, if the referee is willing and the student desires it, a meeting is arranged. It bears emphasis, however, that these steps do not involve appeals for grade changes. The grade is immutable even if it may be an "unjust" grade. In the same way, of course, journal editors judge a small portion of papers unjustly: occasional injustice is part of the research experience. The second of the students' adversaries, the schedule component, can be described more briefly. In contrast to previous systems followed in the department in which supervisors marked the thesis of their own students, and in which negotiations for extensions with the course coordinator could occur, the present holds the student fully responsible for submitting work on time; no extensions are given. This is made clear to the students at the beginning of the year. The complex and time-consuming nature of the graderchecker-arbitrator evaluation scheme allows this refusal to negotiate and grant extensions to be represented as non-arbitrary. The penalty for late submission is 20% off for every day overdue. Thus a thesis which is 5 days late is of merely academic interest in terms of marks. It has never been necessary to apply this penalty, which at least some of the first author's colleagues regard as "barbaric." In the last five years, all theses have been in on time. In comparison, approximately 40% of theses under the previous system were handed in late, with a range varying from 1 to 14 days, and with many hours of negotiation involving the coordinator, supervisor and tardy students. It should be emphasized that the incidence of reported research "horror stories" (absence of human subject, deaths of animal subjects, apparatus breakdowns, nervous breakdowns, etc.) has remained unchanged. The difference is that students are given a clear account of the contingencies in the situation, and they seem to be able to adapt to the stringently-imposed temporal constraints. It is also important to note that while the ultimate constraints are rigid, the intermediate steps are the student's responsibility. For example, while the dates for final thesis submission are rigidly enforced, dates for the completion of data gathering, and for determining the student-supervisor relationships are mere suggestions which the students are free to ignore. Collaborative relationships Set against the adversaries of external evaluation and rigid time schedules are a number of relationships conducive to that collaborative behavior which is intrinsic to modern scientific research. The most important collaborative academic relationship is that between student and supervisor. However, there are bound to be disagreements over how things are to be done in a research project. Under this system, the supervisor is able to concentrate his/her energies fully on being a helpful, but disinterested critic, since he is not involved in judging the thesis. Thus disagreements which are exaggerated when the supervisor is an examiner as well as a colleague are usually kept under control. In any dispute, there is a nice balance between the supervisor's greater experience in the field and the student's knowledge that he alone is ultimately responsible for the final version of the thesis. The student may be constrained by the supervisor's criticisms (unlikely in the unconstrained- 84 Furedy and Furedy scientific-genius paradigm); elimination of the supervisor from the grading process of the thesis allows the student independence while assigning responsibility to him (in contrast to the grudging slave paradigm). The next important set of relationships in the total collaborative effort is that formed between the student and his fellow students. These relationships are given academic substance by research seminars. Since these seminars start three weeks after the commencement of the academic year and are concentrated in approximately an eight-week period, the projects are presented in a very early stage of formulation. The presentations consequently lack the sort of polish which many undergraduate reports of finished work possess. But polish is not the goal. The work is presented in a relatively incomplete form so that it may be more open to change as a result of criticisms offered by other students. The analogue of this component in real-life research is found in situations in which researchers receive criticism of ongoing research projects from colleagues. Very often the critical input received from "outsiders" concerning a project is wildly off the mark, whether those outsiders be students or professional researchers. Nevertheless, there are rare but valuable instances where an outsider's comment will lay bare a problem which has been overlooked during the student-supervisor interaction. In the course seminars, even when all the criticisms provided by other students turn out to be incorrect, they do at least serve the purpose of preventing the development of the "grudging slave" paradigm, since the student has to justify decisions made about the project with reasons other than simply "my supervisor told me to do it that way." The student-coordinator relationship is the last in order of importance. Like the student-students relationship this is also developed during the research seminars, but individual meetings between the student and the coordinator further strengthen it. There are many instances when the coordinator is less familiar with the area of research than many other students in the research seminar to say nothing of the student and the supervisor. This naivete with respect to content is useful at times. In particular, the student is compelled to define terms which he and others familiar with the area take for granted. On the other hand, of course, the coordinator is more familiar with research in general than the student critics, and thus can sometimes provide constructive help especially with respect to methodology and design. However, since the coordinator does not grade the final version of the thesis, his input is not authoritarian; it rests only on the rational arguments he presents in defense of his views. Symbols of Significance In the first year of applying the course model the coordinator announced that the theses would be bound and placed in the departmental library to provide a year-by-year record of undergraduate research activity. He explained the method of evaluation by nonsupervisor referees. It was assumed that this would suffice as a symbol of significance for the importance of thesis-related activities such as the research seminar. By the end of the year, however, it was painfully clear that the importance of the research seminar in the whole scheme had not been accepted. Indeed the seminar component that year might be described as a disastrous failure. Attendance was low; those present were often there more in body than in spirit. Even the seminar presenters themselves were unpredictable in their attendance: the mean probability of their appearance was only about .5! 85 Realities of Research Experience We have all experienced mutually wasted time and low morale in seminars. But this was no group of unmotivated, dull first-year undergraduates herded into a room for an exercise in student discussion. These final-year students had elected to do the course, having obtained a B+ average to gain entrance. The topics of the seminar were not "irrelevant" topics dreamt up by remote instructors but the students' own research which, at the end of the year, was to be subjected to rigorous external examination. Nevertheless it seemed that further symbols had to be devised before the seminars would be accepted as intrinsically valuable for the course goals. The symbol utilized was that instrument which is anathema to many modern educational theorists: grades. For seminar participation, which included not only the presentation of the project but also the quality of discussion and criticism offered in response to other students' presentations, a bonus mark of up to 5% could be earned; for seminar nonparticipation, defined as not turning up to present one's own project, a penalty mark of 20% would be lost. The smallness of the former grade gave it a largely symbolic significance in the final mark for the course; the largeness of the latter grade served to completely eliminate nonparticipation, while attendance at seminars was close to 100%. The initial reactions of the students to this modification were, predictably, negative. They protested at being treated like "children," and at the implication that the seminar could not stand on its own academic feet but had to be propped up by psychologically demeaning bonuses and penalties. Fortunately this source of conflict between students and coordinator was short-lived because the seminars very quickly developed "functional autonomy": the focus became the thesis problem not the (actually insignificant) bonus mark. This development was undoubtedly aided by two factors. First, the students were eminently competent to function in a highly sophisticated seminar context if they were so disposed. Secondly, the bonus grading scheme, in contrast to the thesis grading scheme, was presented as a highly unreliable and relatively unimportant method of evaluation. The same procedure of devising a symbol of significance had an effect upon another important component of the thesis course: an annual undergraduate thesis convention held at the end of the academic year among half a dozen nearby universities. Despite the fact that contributing to such a convention undoubtedly helped the preparation of a final thesis report, participation was in fact minimal until a small grade symbol-(5%) was attached to participation. As in the case of the seminars, the collaborative behavior between supervisors and students in preparing for the conference very quickly became functionally independent of the insignificantly small grade bonus; it was simply that the conference was now perceived as something academically significant. Invevitable Sources of Tension This course can hardly be termed a completely liberal and easy-going way of transmitting the experience of research to undergraduates. It contains inevitable tensions which may become actual conflicts among the main parties to the scheme. First, there is the potential for conflict between the thesis evaluators and the thesis supervisors. When a supervisor has spent considerable time in helping a student design and write a study, it is unlikely that criticisms of this work by his colleagues will be received with unqualified enthusiasm. A second source of conflict is tension between the supervisor and students other than his own. This tension is displayed in the research seminars where, as a result 86 Furedy and Furedy of the discussion of project plans, supervisors are open to indirect criticisms from people who are not only outside their own areas of expertise but are also "mere undergraduates" as well. Thirdly, tensions are inevitable between the coordinator and students throughout the course of the whole year, especially with respect to deadlines and the final evaluation. It is possible, however, to recognize the positive function of these tensions. If the aim is to model the realities of professional research, then they serve to make the point that research is full of similar stresses and strains. The source of some of these strains is, to be sure, intellectual. Research in psychology is a logically complex enterprise, if only because so many of our terms are badly defined, and much of our data have poor reliability. But there are also emotional sources of professional research conflicts arising from the humanity of the researchers involved. Those who pretend these sources do not exist delude themselves with an image of scientific controversy in which conflict is only between ideas and not also between people. General applicability of the Toronto experience Given an interest in training for research and a faculty attuned to the characteristics of good research most problems of applying this model should be readily overcome. Departments vary enormously in size and composition, but these variations do not, in themselves, impose limits on the applicability of the model. The Toronto department has 35 full-time equivalent faculty members, and it is oriented towards basic, experimental psychology, with an emphasis on verbal learning and memory in particular. The annual number of thesis students accommodated in this course has ranged from 20 to 40, with up to 30 thesis supervisors participating. There should be no problem in transferring this model to another department of another size as song as: (a) the faculty/student ratio is maintained, and (b) there is a sufficient number of students (say, 12) to run the research seminar component successfully. It is not necessary for students' interests to neatly interlock with faculty interests. As long as the faculty member has experience in research of a similar form, though not necessarily of content, the student-supervisor relationship is generally as fruitful as in the case of a shared content area. Similarly, for thesis evaluations, although an attempt is made by the coordinator to match the grader with the thesis, poor matches have not brought dire consequences (as judged by the frequency with which arbitration has to be resorted to between the two graders). What is essential for the successful adoption of such a model is an agreement on what constitutes good and bad research. For example, in a department which is so divided between the so-called "hard" and "soft" areas, that respective faculty are convinced that no good research is conceivable in the "other" area, it would hardly be possible to institute this model. There would be too many difficulties in the research seminars and the judgment of the theses. Again, a department which is markedly heterogeneous with respect to the intellectual quality of its faculty would probably not provide suitable conditions for the system. However, it is not at all important to have homogeneity of research output among faculty participants. Some of the best supervisors and thesis evaluators at Toronto are relatively inactive in research as measured by conventional publications criteria. These faculty, however, all agree on what constitutes good research and their intellectual competence is recognized by their peers. 87 Realities of Research Experience Clearly, a general willingness of a considerable proportion of faculty members to cooperate in the course's implementation is vital. Each year at least half as many faculty as students must be prepared to serve as thesis evaluators. They must be willing to undertake critical marking of the sort which is clearly not part of their formal "teaching load," for which rigid deadlines have to be met, and which, in the end, may well earn them more enemies than friends among the more touchy of their colleagues. Thesis supervisors, in turn, have to be willing to have their advice to their own thesis students subjected to criticism from students and faculty "outsiders," criticism which, for the most part, is based on ignorance of the supervisor's field of expertise. That the model has been and apparently continues to be successfully implemented is a tribute to the commitment to disinterested enquiry and academic values which most members of the department exhibit by their cooperation with the system. Conclusion The adversary model system presented here has been described rather than analyzed. Systematic data are difficult if not impossible to collect, and, in any case, before any adequate analysis of different systems could be undertaken, those alternatives would have to be described. In our opinion perhaps the best argument in favor of the system was implied in the remarks of an undergraduate from another university who heard an abbreviated version of this paper. His reaction was that if real research entails adversary relationships rather than the one-big family concept, he and many other undergraduates would prefer not to be thrown into the "seething cauldron" so early. Some students may still view the university as an ivory tower! We hold that if there is a cauldron out there, we owe it to our undergraduates to show them "the way it is" as quickly and as realistically as possible. Carefully designed simulation provides a means of doing this. Through coping with inevitable adversarial factors, the collaborative aspect of research may be more quickly and effectively learned by the researchers of the future in the formative period of their intellectual lives. Preparation of this paper was aided by grants from the National Research Council of Canada and from the Ontario Universities Programme for Instructional Development. Critical comments of an anonymous reviewer of an earlier version of this paper were helpful in sharpening a number of issues, as were the comments of participants in a symposium on "The Undergraduate Research Experience" held at Ithaca College, April, 1975 where some of the ideas contained in this paper were presented. However, the most important debt is owed to G. Thornton, R.S. Lockhart, and other members of the Department of Psychology at the University of Toronto, without whose collaboration in the thesis course described in this paper there would have been nothing for us to write about.