The Canadian Journal of Higher Education, Vol. IX-1, 1979 Course Evaluation for Academic Management — A Case Study in Biology L'évaluation des cours et la gestion académique: Un dossier en biologie W.H. DOWDESWELL* H .M. GOOD** In September of 1975 a new course was introduced to the Biology Department of Queen's University at Kingston, Ontario. This course was entirely traditional in content, being an introductory survey of the plant and animal kingdoms. It was, however, not traditional in the range of its aims and instructional methods. In brief, the course placed more emphasis than usual on student independence, on the use of information rather than its acquisition, on ability to generalize, on communication among students, and on direct interaction with staff. The lectures as information-giving sessions were abandoned and twenty very generalized lectures replaced the usual forty-eight dealing with specific detail. There were reservations, in some cases approaching opposition, among some of the staff. Indeed, there were problems in the first presentation of the course and significant changes were made for the 1976-77 year. It then became important to evaluate the course so that objective information was available on what aspects were now successful and what further modifications were needed. The evaluation was carried out by the authors. W.H.D. acted as outside consultant, and H.M.G. represented the team presenting the course. The role of an outside consultant as independent critic and synthesizer was considered important from the outset of the evaluation. Experience with this evaluation has if anything strengthened our views of the importance of having such an outside consultant. This presentation completes the first stage of a collaborative project on evaluation and academic management which will be carried forward at Bath University in 1977-78. The Queen's course was chosen for the initial study for three reasons: 1. Its experimental nature (involving several reallocations of resources and changes of instructional m e t h o d ) made it somewhat easier to see the need for evaluation. 2. The two principal designers of the course were committed to the principle of iterative course review and re-development. 3. The sabbatical leave of one evaluator (HMG) in 1977-78 required that the project in Bath University be carried out in that year. * School of Education, The University of Bath, England. * * D e p a r t m e n t of Biology, Queen's University 24 W.H. Dowdeswell and H.M. G o o d The choice of an experimental course for the initial part of the project should not be taken to indicate that the authors feel such courses need evaluation more than do those which are traditional and long-established. If anything those which have not been recently re-thought will need critical appraisal more. However, the evaluation of experimental courses is often a more politic way to initiate systematic course assessment in a department. The collaborative basis of the work — with one collaborator an entirely external evaluator — is reflected in the way in which the work is divided. The general approach and the questionnaires were developed jointly, the interviews and interpretations were done by W.H.D., the statistical analyses were shared and this presentation is collaborative. The evaluators were concerned not only with this course, but with the development of a fairly generalized model. They have, therefore, considered this study as an example on , which certain general recommendations might be based. This presentation then, deals first with the general background to course evaluation, proposes a model, develops and tests the model in the context of Biology 200 at Queen's, and then returns to the issue of the generalized process and the transferability of the model to other courses and to other disciplines. Consistent with our broad view of the nature of evaluation and its role in the management of academic programs we stress the basic relationships involved rather than detailed methodology. Our belief is that the problems of achieving worthwhile evaluation lie in the way in which the evaluation process is woven into the fabric of university governance and management. In this paper we deal specifically with the departmental level but the approach can be generalized to higher levels of administration. The case for course evaluations is implicit in the universities' quest for excellence, in the need to adapt to change, and in the concept of accountability and will not be argued further here. The issues are to choose appropriately between formal and informal evaluation and to develop what may be called 'due process' when instituting procedures which involve an implicit evaluation of those teaching the course and have, therefore, important implications for fairness in promotions, salaries and other rewards. The expression 'due process' in evaluation is based on our concept of evaluation as having two dimensions — i.e. who shall be asked and about what? This view of evaluation leads to seeing its organization in the form of a table in which the several populations whose opinions are relevant are listed on one axis and some organizations of the questions to be put to them appears on the other. Each cell of the table must be considered if 'due process' is to be achieved. However, there may be some cells which are considered essentially empty. The important issue for 'due process' is that each cell has been examined. In this report the populations to be surveyed are described briefly. The nature of the questions which should be put to them is illustrated in greater detail in the questionnaires. An incomplete evaluation, i.e. one which ignores the 'due process' model is at best likely to be useless and at worst to be grossly misleading. Some evidence for this view is presented below and its implications are discussed in the final section. WHA T IS TO BE JUDGED? Studies of curricula and their effectiveness are sometimes clouded by uncertainty regarding the terminology commonly used in such enquiries. Thus, in some quarters, it is supposed 25 Course Evaluation for Academic M a n a g e m e n t : A Case Study in Biology that a curriculum amounts to little more than a statement of course content. This is a syllabus. From the student's viewpoint, a curriculum can be defined as the totality of planned educational experiences which he received within a given subject area at a particular level. For the curriculum designer this total coverage includes objectives, content (syllabus), modes of presentation, assessment and evaluation. Confusion also tends to arise between the terms assessment and evaluation. Assessment is the means by which student performance is determined. Evaluation is the process of gauging the overall achievements of a curriculum in terms of its stated objectives. Within this framework, assessment of student performance is an important part of the curriculum undergoing evaluation. In the course herein described assessment plays a larger role than in many other courses and some preliminary explanation is required. Students were assessed mainly on performance in traditional written exams responding to problem-solving essay type questions. The unusual feature, at least among biology courses, was that the student papers were marked during an interview with the student. This permitted feedback and an opportunity to explore areas of doubt on the part of either teacher or student. The time required for this system of marking was slightly, but only slightly, longer than for conventional marking and both staff and students regarded it as a better and fairer system. To avoid very long answers students were subject to space constraints on their essay answers but were given a good deal of time to organize succinct statements. The purpose of these preliminary remarks is to emphasize the point that a curriculum is essentially a complex system of interacting parts. Just as objectives determine content, so modes of assessment may influence methods of presentation. The system must possess a number of sensitive pathways for internal and external feedback. A further point which is not always sufficiently appreciated is that a curriculum represents people, not just materials. For a course to be effective, those involved in teaching it must be basically of like mind, although they will no doubt differ in detail and degree regarding approached and emphasis. It follows that if a curriculum is to change substantially the persons responsible must voluntarily modify attitudes and preferences for what they believe to be compelling reasons. On the other hand, if they are not persuaded to change their strongly held attitudes can provide a powerful barrier against change. WHO IS TO JUDGE? Having indicated that it is the curriculum as a total system which is to be judged, who is t o do the judging? For a single course, i.e. a relatively small section of one year's study, there are three groups whose opinions must be sought. They are: 1. The department staff 2. The students 3. The staff of other departments whose students make the course a part of their degree offerings. In addition to these there are two groups which are more important in relation to programs rather than courses. These are the wider university community with which a 26 W.H. Dowdeswell and H.M. G o o d department must share resources and the society which provides the resources for the activities of the university and its departments. Departmental Staff For a department to function adequately it is essential not only that there be a considerable common denominator of purpose among its members but also that the administrative head have a clear understanding of the nature and extent of this common denominator and its obverse the elements of fundamental disagreement among members. Only thus will he be able to fit staff into appropriate assignments (and in appropriate combinations) and bring about functional compromise through rational debate. The evaluation process should be designed to give him a clearer basis for judgements in this area. The views of a departmental staff on a particular curriculum will represent three levels of involvement: (i) senior management, usually the Head of the Department, (ii) those involved in teaching the course, (iii) the remainder of the staff who will have views and expectations derived from a wide variety of information and interests. Senior management will be concerned with such matters as the status of the course in relation to others provided by the department and the extent to which a particular curriculum conforms to departmental policy, for instance in developing desirable student attitudes. Of particular importance will be the provision of resources such as time, space, manpower, and other facilities which must be assessed in the light of overall commitments. Those engaged in teaching the course are likely to have strong vested interests and to be preoccupied with more parochial matters such as the achievements of the course as evidenced by student assessment, student attitudes and managerial problems of a relatively short-term kind. Other members of staff not directly involved in implementation are likely to display attitudes varying greatly in kind and intensity. Some will have positive expectations in that they may have based their own courses on an assumed quota of background knowledge, attitudes and skills. At the other extreme, some will have no such expectations and may even view a particular course with some hostility as an absorber of resources which might otherwise have been deployed elsewhere. Between these two poles there is likely to be a more or less continuous spectrum of opinion, some reflecting broadly based rationales and some rather narrow considerations. Students The usefulness of such course feedback as can be supplied by students is often misjudged and underexploited. One of the reasons for this is that guidance is seldom given regarding the most useful form for registering criticisms. In consequence these are usually so broad and generalized as to amount to little more than opinion or popularity polls. As such, they are unlikely to contribute significantly to change in a system as complex as a course curriculum. To be effective, particularly in the short term, student feedback needs to be as specific as possible. Other Departments The contribution that staff of other departments can make to course evaluation is a matter 27 Course Evaluation for Academic M a n a g e m e n t : A Case Study in Biology of subtle debate. It will often happen that students from other departments who attend a particular course do so as a compulsory part of their instruction. In other cases there is free choice. Whatever the relationship, one department clearly has the right to voice its expectations for its students enrolled in courses of a cognate department and even to express views on course content and conduct although it should not expect its views to override the basic concerns of the department offering the course. There is a nice balance here involving political and academic considerations; just where the balance should be struck must depend upon each individual set of circumstances. However, the issue must be faced and formal evaluation has much to offer as a basis for acceptable decisions in which the interest of students are given first consideration. HOW DO WE JUDGE? As we emphasized earlier, a course curriculum is a complex and integrated system. The problem in evaluation is to select an appropriate set of criteria. No curriculum worthy of the name can be altogether lacking in some statement by its originators of what it was designed to achieve. A prospective evaluator needs information not only on the broad intentions (aims) of the course as a whole, but also on the more specific outcomes (objectives) expected from particular parts. As curriculum studies have frequently shown in the past, the writing of objectives at the outset of a course is not easy; indeed, the cynics would have us believe that the best time to write objectives is when a course is finished. However, a preliminary, if imperfect, attempt at specifying objectives is essential if we are to design a meaningful system of student assessment or a realistic model for overall evaluation. We owe this not only to ourselves as teachers and evaluators, but also to the students we are trying to assess. When attempting an evaluation in the context outlined above, we must also remember that we are dealing with an iterative process, in that one evaluation is merely the prelude to the next. In gauging the effectiveness of a course due credit must, therefore, be given to any attempt to incorporate concurrent monitoring (formative evaluation) into the system. A related set of criteria f o r j u d g i n g the effectiveness of curricula makes use of the concept of resource-effectiveness. Here the approach combines the academic considerations with the managerial in a way which has not been much utilized in universities. The Biology course at Queen's provides a number of examples of such academic management. One is the interview marking described earlier. It was felt that a slight increase in marking time was amply justified by a more effective contact with the student. Another example was the decrease of lecturing — partly to give time for tutorials and interview marking and partly to throw more responsibility on the student. A third example was the case of student-led tutorials to provide more contact and discussion time without commensurate load on the staff. A single "master tutorial" with the student leaders looked after 5 tutorial groups for the week in question. These examples illustrate what we mean by academic management. They involve re-allocation of resources, which to be fully justified must be tested for their contribution. If a course is resource-intensive in terms of such commodities as time, space, manpower and equipment we might reasonably expect a greater return from it than from one which was less demanding of resources. The quantifying of such managerial criteria is useful in that it serves to reveal the all too prevalent and fallacious assumption that effective design of curriculum can be achieved using 28 W.H. Dowdeswell and H.M. G o o d academic considerations (objectives) alone. At its crudest, it is no good designing on academic grounds a course lasting 100 hours if resources are only sufficient for 75, or constructing a new laboratory programme when the facilities required are only partially available. Evaluation as a collaborative enterprise In this account we have attempted to highlight some of the elements of course evaluation and to draw a distinction between the potential contributions of staff and students. Both are important contributors; the problem is to bring them together in a complementary context rather than in an antagonistic one. In general students appear to have been somewhat more responsible than have the staff in the matter of evaluation. They have spent a great deal of time, effort and money in evaluation efforts with remarkably little visible effect. The reason for this has been twofold — 1) many university staff members have gone little beyond lip service in supporting the concept of evaluation; 2) students have received little critical and constructive help in relating their efforts to a total process of evaluation because the latter is usually lacking in any formal sense. An evaluation cannot be better than the use to which it is p u t . If student evaluations are to succeed, departmental and faculty collaboration both in designing useful evaluative tools and in the essential follow up and use of data are required in equal measure. In most university departments and faculties the evolution of an effective partnership between students, staff, and departments has rather a long way to go. Evaluation in university management has indeed acquired a bad name precisely because a careful collaborative model has not evolved and results have often been less than useful. This was neatly illustrated in a sequel to the evaluation herein reported in which a staff member used a student designed questionnaire in parallel with the one described below. His experience bears on two issues. 1. The ready adaptability of an instrument designed for one course to another in the same department (and perhaps beyond). 2. The danger of using incomplete evaluations. He summarizes the experience as follows: "I used the course evaluation questionnaire developed for Biology 200 to aid in the evaluation of Comparative Animal Physiology (338). I found that with minor modification (e.g. omitting questions relevant only to Biology 200) that it was quite suitable. When I showed the questionnaire to the Department Student Council representative for 388 she became quite enthusiastic and we decided to use your evaluation scheme as well as the standard DSC evaluation form. After comparing the information that one gets from the two questionnaires there is no doubt in my mind that your form is far superior to the DSC form. In fact there are instances when the DSC evaluation can be misleading and lead to revisions in a course that are detrimental." "As a case in point let me give you the comparison for the laboratory component of 338. The DSC questionnaire informed me that the student reaction to the laboratory was mixed. They did not enjoy the laboratories nor did they feel that they were worthwhile. However, upon closer examination of this problem, using the series of questions outlined 29 Course Evaluation for Academic Management: A Case Study in Biology in your evaluation it became apparent that the problem was not in the laboratory itself but in the way the laboratory was assessed. The students did enjoy doing the laboratory exercises and felt that the information and the techniques learned were very relevant and valuable. However, they felt that the time spent in writing laboratory reports, in the detail I had requested, was not reflected in the value assigned to the laboratory section." "On the basis of the DSC evaluation I would have attempted to revise the laboratory exercises. On the basis of the more critical evaluation possible with your questionnaire it is apparent that what needs revision are not the experiments but the marking system used for the laboratories. Having more precise information has saved an enormous amount of work which even after being completed would not have improved the course and in fact might have proved detrimental." "In my opinion the development of a questionnaire such as yours for general use would be extremely valuable in the continuing process of evaluation and revision so necessary in maintaining high quality in the educational process." The issue of transferability of instruments has indeed attracted a good deal of attention and it is clear that adaptation of carefully developed models and instruments is to be preferred to individual development for each application. THE MODEL - TYPES AND TIMING OF EVALUATIONS To some extent every teacher and every student evaluates each course each time it is given. This informal, and often unstructured, evaluation is important. It may, however, be based on a substantial element of self delusion. Moreover, it is rarely complete enough to provide a precise indication of what changes should be made. Even an apparently complete student evaluation can by itself produce a distorted expression of changes required as we have noted. On the other hand, evaluations done comprehensively with what we have called 'due process' are expensive in terms of time and effort and cannot be done in every course in every year. It would, therefore, seem practical to propose that course evaluations be done in three stages. With a new course which is likely to have starting up problems, we picture these stages as being in three successive years as follows: 1. Year 1 —i.e. the first year a new course is offered: no formal evaluation. Teachers will easily recognize enough trouble spots to undertake significant revisions. 2. Year 2 — a formal evaluation but dealing with general issues of success or failure in meeting aims, and in reaction to different sections of the course. This would not cover performance of personnel. 3. Year 3 — a critical analysis of the success of specific units. By implication this must bear on the success of the teacher and evaluation of personnel is not only permissable but inevitable at this stage. There might then be a period of 2-3 years before any further formal evaluation is attempted. 30 W.H. Dowdeswell and H.M. G o o d Follow up We have argued above in some detail that an evaluation which is not linked to an adequate follow up is a farce. Evaluation must be seen as a part of the iterative cycle-design evaluate, improve. Such a cycle for courses within a department should, of course, be related to ongoing cycles at the level of the faculty and the university as a whole. THE QUESTIONNAIRES These are presented in the following order: Student Staff teaching the course Staff not teaching the course Department Head Other Departments The student questionnaires give a fairly detailed picture of the structure of the course and this allows the other questionnaires to be presented with a minimum of additional explanation. Student responses have been tabulated in the questionnaire itself. The presentation of questionnaires and results are followed in each case directly by summary of written comments by respondents to the questionnaire and the evaluators interpretation of the overall results. The student questionnaire is illustrated here by sections on the investigative laboratories, tutorials and assessments. Similar sections covering lectures and open laboratories have been omitted since the questions are for the most part the same. Anyone interested in further details can obtain them from the Department of Biology, Queen's University. Student Questionnaire Introduction: An evaluation of Biology Course 200 is being carried out both by staff and students. The purpose of this questionnaire is to find out how the course is going and to identify any parts of it which may need alteration. In the questionnaire that follows you are asked to do two things, i) to put a tick (V) to complete each statement appropriately. ii) to add short comments to qualify your answers when necessary. If you do not wish to add any comments there is no need to do so. Your answers will be valuable nonetheless. Personal (Details of year, program, previous courses, etc.) Lectures (See laboratory section for general approach) 31 Course Evaluation for Academic M a n a g e m e n t : A Case Study in Biology Investigative laboratories I. I n v e s t i g a t i v e l a b o r a t o r i e s succeeded i n : (a) (b) (c) (d) s t i m u l a t i n g your t h i n k i n g in b i o l o g y providing f a c t u a l information improving your problem s o l v i n g a b i l i t y g i v i n g you an understanding of the r e l a t i o n of an organism to i t s organ systems d e v e l o p i n g your a b i l i t y to argue s c i e n t i f i c a l l y developing your a b i l i t y to use b i o l o g i c a l knowledge developing your a b i l i t y to use g e n e r a l i z e d or p r i o r knowledge in the new s i t u a t i o n s provided by t h i s course encouraging you to do independent study developing your a b i l i t y to g e n e r a l i z e providing you with a u s e f u l t e c h n i c a l vocabulary i n c r e a s i n g your i n t e r e s t in b i o l o g y providing enjoyment g i v i n g you an adequate knowledge of information s o u r c e s . improving your v e r b a l communication s k i l l s developing your t e c h n i c a l and manipulative s k i l l s h e l p i n g you to r e l a t e the information from these labs to that in the open l a b s h e l p i n g you to r e l a t e the information from these l a b s to that in the l e c t u r e s (e) (f) (g) (h) (i) (j) (k) (1) (m) (n) (o) (p) (q) II. The l e v e l of d i f f i c u l t y of I n v e s t i g a t i v e l a b o r a t o r i e s was: easy [] about r i g h t difficult III. 1)91 [] Other comments on the I n v e s t i g a t i v e i labs. Tutorials I. (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) T u t o r i a l s have succeeded i n : s t i m u l a t i n g your t h i n k i n g i n b i o l o g y providing f a c t u a l information encouraging you to do independent study i n c r e a s i n g your i n t e r e s t i n b i o l o g y providing enjoyment improving your a b i l i t y to communicate v e r b a l l y improving your a b i l i t y to communicate in w r i t i n g i n t e g r a t i n g the d i f f e r e n t p a r t s of the course demonstrating t u t o r s ' i n t e r e s t s and enthusiasm in b i o l o g y . . . . s o r t i n g out problems from other p a r t s of the course d e v e l o p i n g your a b i l i t y to use g e n e r a l i z e d or prior knowledge in new s i t u a t i o n s provided by t h i s course '. (1) i n c r e a s i n g your s e l f - c o n f i d e n c e ft W.H. Dowdeswell and H.M. G o o d Other comments on T u t o r i a l s : I. The f o l l o w i n g a s p e c t s of the assessment procedure are acceptable and do not need a l t e r a t i o n : (a) (b) (c) (d) (e) the proportion of the t o t a l time devoted to assessment the methods of a s s e s s i n g laboratory s k i l l s the methods of a s s e s s i n g f a c t u a l knowledge the method of using essay q u e s t i o n s the r e l a t i o n s h i p between examination q u e s t i o n s and the nature of the course Assessment (f) (g) (h) (i) II. agree Assessment disagree II. not sure 32 7/ 21 /4 n 1% u a n */ it 3? v t 30 3<> (continued) 21 feedback of assessment r e s u l t s to students the d i f f i c u l t y of the q u e s t i o n s s e t the c l a r i t y of q u e s t i o n s s e t the essay as a bonus mark assignment Isr HP. Ï 1 a \ii 71 f : It Other comments ( e . g . on the value of i n t e r v i e w marking). Recommendation I. Would you recommend the course to future s t u d e n t s . Yes A 1 II. No AWtSU^I H Give b r i e f l y the reasons for your answer. Following are interpretive c o m m e n t s on t h e student questionnaire (completed by a sample of 116 students o u t of t h e total of 2 8 0 taking Biology 2 0 0 ) . The i n f o r m a t i o n has been derived partly f r o m t h e quantitative summary of t h e responses t o t h e questions and partly f r o m t h e c o m m e n t s invited as a supplement to each section o f t h e questionnaire. It will be n o t e d that responses t o particular questions do n o t always total 116. This is because some students failed t o answer all t h e questions. In t h e summary that follows it will be seen t h a t certain aims were n o t achieved b y particular parts of t h e course. Reference t o t h e instructor questionnaire in t h e following section m a y show that some of these were accorded low priority by t h e instructors and a high level of success was hardly t o be e x p e c t e d . Lectures They seem t o have succeeded in stimulating thinking, a m a j o r aim of t h e course. This is borne o u t by t h e frequent student c o m m e n t s t h a t t h e lectures were stimulating and that they showed there was more t o biology t h a n just t h e learning of facts. 33 Course Evaluation for Academic M a n a g e m e n t : A Case Study in Biology Evidently the students did not associate the issues raised in the lectures with problem solving. This is also reflected in their comments where problem solving is hardly mentioned. The ideas of systems theory came through strongly. These seem to have appealed to some students but not to others. Several asked whether the approach would not be more suited t o the 4th year. However, in spite of the variety of comments about complexity no less than 68% of the sample agreed that the level of difficulty was about right. The ability to generalize seems to have been considerably increased. The lectures do not seem to have provided over much enjoyment to the students and, not surprisingly, there are numerous comments on their acceptability both for and against. While some found the lectures easy and understandáble, others thought them too abstract and generalized. The strong reactions of students in different directions suggests that the lectures had to their credit at least a provocative effect. The three most universal complaints about them were, (i) that while the handout material was admirable, it frequently reached the students at the wrong time. There was a widespread demand that it should be distributed well in advance of each lecture. (ii) the TV productions were monotonous and the way they are produced detracts from the value of the subject matter. (iii) the lectures were disjointed in that they did not relate one to another. If the intention is to deal with " k e y " biological topics and not to provide a sequential "course" in the traditional sense, perhaps there is a case for making this aim more explicit. Having viewed one of the TV programmes (The Parasitic Way of Life) and read nearly all of the handout material, perhaps I (WHD) as external evaluator may add a few personal comments. In my opinion, much of the material is admirable and presented in an original and challenging manner at about the right level of sophistication. On the other hand, I agree with the student criticism that considerable portions of the televised lectures tend to be monotonous (and therefore become boring). This is a problem of TV production which I believe could be fairly easily overcome by comparatively simple and inexpensive means, some of which I have discussed with the professor who gave all the lectures. Investigative Laboratories They were considered to have succeeded in achieving a number of aims notably, stimulating biological thinking, providing factual information, relating the organism to its organ systems developing the ability to use biological knowledge and to apply it in new stiuations, providing a technical vocabulary, increasing interest, and developing manipulative skills. On the other hand, the laboratories do not appear to have been successful in improving problem solving, developing the ability to argue scientifically, encouraging independent study and relating the information in the lectures. Clearly, the students considered the level of work to be about right. Student comments were overwhelmingly favourable, numerous compliments being paid to the two supervisors of the laboratories for their efficiency in conducting the practical work and the readiness with which they were prepared t o give help to students. Such criticisms as were forthcoming were of a trivial kind, such as the time of allotment of various practicals, the disjointedness of some topics and too much detail in others. These are matters which can easily be rectified, may well have been resolved already, 34 W.H. Dowdeswell and H.M. G o o d through the process of formative evaluation, or are ones in which student views of what is appropraite should perhaps not be given more weight. Open Laboratories These are thought to have succeeded in stimulating thinking, providing factual information , relating the organism to its organ systems, developing the ability to use knowledge in new situations, providing a technical vocabulary, increasing interest and providing a knowledge of information sources. They do not appear to have been successful in improving problem solving and developing manipulative skills. As in the Investigative Laboratories, there is strong evidence that the level of treatment is about right. Student comments were generally favourable, for instance that the laboratories helped to "bring the course to life". Criticisms were trivial and of the same general kind as for the Investigative Laboratory. Tutorials Judged by the student responses, these have undoubtedly been the most successful part of the course. Indeed, they appear to have achieved almost all their aims except, encouraging independent learning, improving written communication, (only two brief essays were required) and increasing self-confidence, a question which seems to have been widely misunderstood by the students. As would be expected, the general comments were highly favourable. Tutorials were considered to provide opportunities for debate, to tie up loose ends, to encourage a more personal approach and so forth. Opinions differed regarding tutor and student-led tutorials. Preference for tutors predominated but there was a strong following for students. So it looks as if the present mixture is about right. It is noteworthy that in the student responses, tutorials are the only parts of the course that have undoubtedly provided enjoyment. Assessment of student performance The following aspects of assessment procedure seemed to be acceptable to students: the proportion of time devoted to assessment, the methods of assessing laboratory skills, the methods of using essay questtions, feedback of assessment results to students, and the essay as a bonus mark assignment. On the other hand, there was a good deal of uncertainty regarding most of the other aspects of assessment and a strong feeling that clarity had not been achieved in the questions. Other adverse comments referred to the time taken by tutors in correcting papers and weighting of marking schemes. Interview marking was regarding as a blessing by some (the majority) and a waste of time by others. The general comments of students were rather equivocal and tended to bear out the distribution of their responses to the questionnaire. 35 Course Evaluation for Academic Management: A Case Study in Biology Questionnaire to Instructors in the Course The following questionnaire was completed by 6 members of staff, 3 of whom are at present involved in teaching Course 200 and 3 who will be involved next year. This total change of staff is unusual but provides more viewpoints than one would normally obtain. Aims and Objectives Q . l . Is the philosophy of the course (with an emphasis on student independence and the use, as well as the acquisition, of knowledge) perpetuated in years 3 and 4? A. Answers differed but all expressed varying degrees of doubt. Some felt that it was a feature of year 4 but not of year 3. Others thought it occurred to varying extents in both. Some considered that it could be induced in some students but not in others. Q.2. Does this philosophy conform to Departmental policy? A. This questions seems to have occasioned some surprise! There was a general feeling (either stated or implied) that there was no Departmental Policy on this subject. Whether there should/could be was another matter. Q.3. Rate the following aims of the course in terms of priority (high/medium/low). A. H s t i m u l a t i n g t h i n k i n g in Biology providing factual information improving problem solving a b i l i t y g i v i n g a n u n d e r s t a n d i n g of t h e r e l a t i o n o f a n organism t o i t s organ systems d e v e l o p i n g an a b i l i t y t o argue s c i e n t i f i c a l l y d e v e l o p i n g an a b i l i t y t o u s e b i o l o g i c a l knowledge d e v e l o p i n g an a b i l i t y t o use g e n e r a l i z e d or p r i o r k n o w l e d g e i n t h e new s i t u a t i o n s p r o v i d e d b y t h i s course encouraging independent study developing a b i l i t y to generalize providing a useful technical vocabulary increasing i n t e r e s t in biology providing enjoyment g i v i n g an a d e q u a t e knowledge of i n f o r m a t i o n s o u r c e s improving v e r b a l communication s k i l l s improving w r i t t e n communication s k i l l s d e v e l o p i n g t e c h n i c a l and m a n i p u l a t i v e s k i l l s i n t r o d u c i n g s y s t e m s t h e o r y a s a means of a n a l y z i n g problem s i t u a t i o n s increasing student s e l f - c o n f i d e n c e in dealing with biological topics t r a n s m i t t i n g t u t o r s ' i n t e r e s t s and e n t h u s i a s m s i n biology M 6 3 4 - - 2 2 1 2 1 1 3 5 6 - 4 6 5 - — _L - - 2 - o 1 2 2 2 1 1 h 3 1 U 3 3 3 2 1 4 1 1 1 2 3 3 - 3 2 1 2 1 3 - - Although there was considerable variation of opinion regarding priorities, the items on which high priority was almost completely unanimous were, stimution of thinking, the ability to argue scientifically, the ability to use biological language, independent study and developing the ability to generalize. In this respect, the staff could justifiably be said to be in agreement regarding the main components of course philosophy. Q.4 In drawing up a list of the kind above, what weighting (high/medium/low) would 36 W.H. Dowdeswell and H.M. G o o d you assign to the requirements of (a) general education, (b) parallel or later Biology courses (c) the needs of other Departments supplying students? H-5 M~ L~ A. (a) General education H-5 M~ L(b) Parallel or later biology courses L-3 (c) Needs of other Departments supplying students. H~ M-2 One member of staff felt doubt about the significance of this question and did not reply. Q.5 Regarding question 4 above, to what extent are these requirements compatible/ incompatible? A. There was a strong feeling that the requirements of (a) and (b) are, or should be compatible, but there was some doubt as to whether this compatibility is always expressed. The requirements of (c) are more difficult to equate with (a) and/or (b) and there will be circumstances when they are incompatible. We have argued in the introduction that formal evaluation offers the best chance of resolving such incompatibility when it occurs. Course Design Q.6 In designing course Biology 200, what were your main expectations of (a) the lectures (b) the open laboratory (c) the investigative laboratory (d) the tutorials? A. a. Lectures (i) Exposition of conceptual systems with only such factual material as was needed to illustrate the functioning of such systems. (ii) Stimulation of interest (iii) S t i m u l a t i o n o f i n d e p e n d e n t t h i n k i n g . (iv) To provide aframework of thought and knowledge for other activities in the course. (v) To introduce generalization (vi) To provide examples for problem solving. b. (i) (ii) (iii) (iv) Open Laboratory Stimulation of interest. Provision of factual material To encourage the transfer of information from one situation to another. To develop independent thinking and the capacity to integrate concepts and information. c. (i) (ii) (iii) (iv) (v) (vi) Investigative Laboratory Stimulation of interest. The provision of factual material. Transfer and use of information Developing of skills Conduct of experiments and handling of data derived from them. To provide exercises in problem solving. d. Tutorials (i) To ensure understanding of the lectures and related material. (ii) To exercise verbal communication skills 37 Course Evaluation for Academic Management: A Case Study in Biology (iii) To build confidence (iv) To provide opportunities for independing thinking. (v) Provision of situations for thinking out problems. (vi) Development of students' ability to integrate and synthesise. (vii) Stimulation of interest through discussion. (viii) To facilitate personal interaction between students and staff. Judging by the responses to the student questionnaire it would seem that for the lectures, expectations (i), (iii), (iv) and (v) were achieved, while (ii) and (vi) were not. For the open laboratory, expectations (ii), (iii) and (iv) were realized but (i) was less certain. For the investigative laboratories, expectations (ii), (iii), (iv) and (v) were realized, (i) was doubtful and (iv) was unsuccessful, For the tutorials expectations (i), (ii), (iv), (vi), (vii) and (viii) were all realized, (iii) was uncertain, and no information is available on (v). Q.7 When designing the course, what was the rationale that led to the existing balance between numbers of lectures, the two laboratories and tutorials? A. (a) To achieve a balance between input of information, guided thought and independent synthesis. (b) To optimize teacher input (c) The belief that students tend to be overtaught (hence the reduction in number of lectures). (d) The balance chosen provided the best means of meeting the other objectives. Q.8 What assumptions do you make in course 200 by way of background of biological knowledge? A. A great variation of opinion here from virtually nothing to Grade 13 High School biology or Courses in cellular biology and genetics. Assessment and Evaluation Q.9 Can we assess all kinds of objectives e.g. student independence? A. There was some agreement that these could be assessed but often with difficulty. Moreover, the usual objective methods of assessment were often inappropriate, so more subjective means are necessary. Q.10 If not, should such objectives be omitted? A. Yes 1; No 2; Unsure 3 Q.l 1 Do you consider that an evaluation of the kind now being carried out on Biology 200 is of high/little/no value? A. High 4; Little -; No - ; Unsure 2 Resources Q . l 2 Do you consider that an advanced course requires more staff resources per student than an elementary course? A. Yes 1; No 5 A proviso was that if advanced courses are specialized and small they will need proportionately more staff. 38 W.H. Dowdeswell and H.M. G o o d Q.13 Do you consider that the staff resources deployed in Biology 200 on appropriate? A. Yes 4; No-; Uncertain 2 Q.14. What considerations justify the extra expenditure of staff time on interview marking? A. (a) Use of assesment procedure for student learning. (b) Not much more staff input but a high education payoff. (c) Facilitates good feedback to and from students. (d) Provides an opportunity of coaching students in good habits of study. One member of staff commented that notwithstanding its advantages, interview marking was on balance, too costly in staff resources. Questionnaire to Staff not Teaching the Course The following is a summary of interviews with 11 members of staff not teaching the course. Where appropriate numbers have been included in brackets to indicate the proportion of staff holding a particular view. Views on the Course Q.l What is your general attitude to Biology Course 200? A. Favourable 4; Neutral 5; Unfavourable 2 Two staff whose views were neutral indicated that as a result of what they had heard of this year's course their opinion was becoming more favourable. Q.2 What do you consider to be the best features of the course? A. The best features of the course were considered to be, stimulation of critical independent thought (7); the systems approach (3); emphasis on laboratory experience (2); use of tutorials (2); problem solving (1); diversity of experience (1) and built-in concurrent evaluation (1). Q.3 What do you consider to be the worst features of the course? A. The worst features of the course were considered to be, lack of smooth operation (i.e. administrative confusion) (4); lectures tended to be too complex and generalized (3); lack of factual content (3); lack of integration of materials (1); naive view of systems approach (1); lack of dedication by staff (1). Q.4 Do you teach a course following directly on or in parallel with Course 200? A. Yes (7); No (4) Q.5 Do you teach a course which depends upon the content of Course 200? A. Yes 4 ; No 7 Q.6 On a 3-point scale (high, medium, low) what are your expectations of Course 200 in terms of information, attitudes and skills? A. Information H (6) M (2) L (3) Attitudes H (6) M (-) L (5) Skills H (5) M (4) L (1) In spite of a certain amount of adverse comment, it is clear nonetheless that expectations of the course were high in all three categories. 39 Course Evaluation for A c a d e m i c M a n a g e m e n t : A Case S t u d y in Biology Q.7 On a 3-point scale (satisfactory, neutral, unsatisfactory) what was your experience of student achievement on last year's course (i.e. in the first year given) in terms of information, attitudes and skills? A. Information S (1) N (1) U (2) Attitudes S (2) N (2) U (1) Skills S (2) N (-) U (1) The low response was due to one member of staff being away and 6 having insufficient knowledge of the course to give an opinion. The uniform distribution of the replies seems to indicate a rather limited level of perceived achievement. Q.8 Have you any expectation of change in the achievement of the course this year? A. Yes 8; No 2 One member of staff considered that he had insufficient information to give an opinion. Q.9 What proportion of the student handout material have you read? A. 0-10% (8); 10-50% (—) 50+% (3). The three who had read 5 0 + % of the handout material will all be involved with the teaching or management of the course next year. Aims and Objectives Q.lO.Is the philosophy of the course (with an emphasis on student independence and use, as well as acquisition, of knowledge), perpetuated in years 3 and 4? A. Yes 5; To a limited extent 2; No 4. There was general agreement that these were desirable student attributes which should be perpetuated and developed in later years. However, as the figures show, there was some doubt as to whether this objective was being achieved. (W.H.D., external evaluator, notes: My own experience of attending four consecutive sessions of course 535 strongly reinforces these doubts. Had these attitudes extended into year 4 they would undoubtedly have been in evidence during these seminars, but on the whole they were singularly absent.) Q.l 1 Does this philosophy conform to departmental policy? A. There was unanimous agreement that there was no departmental policy, and some questioned whether there could/should be one on matters of this kind. It was clear that departmental opinion was sharply divided regarding course philosophy probably with a slight majority in favour of the approach of Course 200. Assessment and Evaluation Q.l 2 Can we attempt to assess all kinds of objectives, including attitudes? A. Yes 9; No 2. The difficulty of assessing this kind of objective was fully appreciated, but the view widely held was that it was worth trying even if assessment could be only subjective. Those who answered negatively were not prepared to drop an objective just because it could not be evaluated. 40 W.H. Dowdeswell and H.M. G o o d Q.13 Do you consider an evaluation of the sort now being carried out on Biology 200 of high, little or no value? A. High 9; Little 2; No Q.14 Would you be willing to have a course you teach subjected to the same kind of evaluation? A. Yes 11; No One member of staff added the proviso that the evaluation process must not detract significantly from academic activities. Q.15 Do you agree with the designers of Biology 200 that the process of student assessment can also be a valuable part of their learning? A. Yes 11; No Several staff felt strongly on this matter. One made the point that the act of assessing a student also helps him to prepare himself for assessment. Another pointed out that excessive assessment pressure may detract from the benefits which can otherwise accrue. Resources Q.16 Do you consider that an advanced course required more staff resources per student than an elementary course? A. Yes 6; No 5 Among those who said "yes", the basic argument was that although there were fewer students at advanced level they were studying in greater depth, often for a longer time, and were therefore more in need of access to staff. The alternative argument was that it is important for an introductory course to reach a high standard thereby setting students on the right track. Classes are inevitably large, therefore more staff are required in order to achieve the necessary objectives. Q.17 Do you consider that the staff resources deployed in Biology 200 are appropriate? A. Yes 9; Uncertain 1; No 1 The negative answer indicated that they were excessive. Q.18 Do you consider that the allocation of more senior staff time than is usual to the process of student assessment is justified? A. Yes 5; Uncertain 1; No 5 Of those who answered "yes", two had reservations but were of the opinion that, on balance, the experiment was worthwhile. It is difficult to summarize succinctly the results of a heterogeneous set of questions such as those asked to the sample of staff interviewed. Rather, I (WHD) feel it is better to let the answers and my brief comments upon them stand on their own. As a generalization, it can be said with some confidence that a considerable fund of good will towards Biology Course 200 existed among those interviewed. Moreover, in spite of a certain amount of adverse feeling, expectations for the course this year are, on the whole, high, 41 Course Evaluation for Academic Management: A Case Study in Biology as the answers to Q's 6 and 8 clearly show. Criticisms there undoubtedly were, but I have not found it easy to determine the extent to which they represent a carryover from last year's course. The fact that such decided views on the course were held by 8 out of the 11 staff interviewed in spite of the fact that they had read less than 10% of the student hand-out material available, seems to enhance the view that a dubious reputation gained last year (as indicated by Q.7) may be dying hard. As the answers to Q.'s 4 and 5 show, there is an appreciable onward flow of students from Biology 200 into the course of Year 3. It might, therefore, be a useful exercise to assess their performance in the light of their previous experience, next year. Questionnaires to the Department Head and to other Departments were not developed in advance but were added as the evaluation proceeded. Neither questionnaire has been adequately developed or tested but both are considered necessary to a complete evaluation process. The questionnaire to the head is included here as the answers provided significant evidence for the evaluation. The questionnaire to other departments is briefly summarized together with one set of responses received. Questionnaire to Head of Department The following questions were put to the Head of Department with the answers being given as shown. It should be pointed out, however, that the Head interviewed here was a member of staff, but not Head, when this course was designed and introduced. Q.l Does Biology 200 play an important role in the offerings of the Department? A. Yes — part of a mandatory core. Q.2 Did those who redesigned Biology 200 receive from the Department any (a) general (b) specific statement of departmental aims for the course? A. Not that I am aware of. Some is implicit in the history of course - e.g. its predecessors. Q.3 Is there in your Departmental organization a unit which might provide such a statement of aims? A. Yes - the Curriculum Committee. Q.4 Was the general approach of Biology 200 made explicit to the Department during the stages of design? A. Not effectively — though the designers thought they had made adequate efforts in this direction. Q.5 Were individual staff members invited to give to the course designers specific statements of their expectations? A. Yes Q.6 A. Do you support the general educational philosophy of Biology 200? Yes Q.7 A. (a) Are you satisfied with the coordination between Biology 200 and other courses? No 42 W.H. Dowdeswell and H.M. G o o d (b) If 'no' is the failure to coordinate primarily due to the failure of i) the Biology 200 staff ii) the staff of other courses iii) both — in roughly equal measure? A. Both — plus an inadequacy in departmental procedure. Q.8 Are you concerned with the reaction of other departments to Biology 200? A. Yes. Q.9 What mechanism is there now for integrating Biology 200 with the needs of outside Departments? A. No effectual mechanism though formally the Faculty Curriculum Committee would be presumed to have this responsibility. Q.10 What is the relative cost of (a) space, (b) staff, (c) supplies and (d) services for Biology 200 relative to other courses in this Department? A. Not far from average in each. Q.l 1 Are you satisfied with the review-revise provisions in Biology 200? A. Plans are adequate. It is t o o early to judge results. Q . l 2 Are there any problems in the grade distributions in Biology 200? A. Very substantial number of C. grades, coupled with the low number of failures, raises questions about whether all those who passed are really ready for higher work. There appears to be considerable support here for the stress Biology 200 places on certain general educational objectives coupled, however, with a question of whether it is fully meeting the departmental expectations as a foundation course. The resolution of this appears to be in Departmental and Faculty procedures which are in place, but not always effective. It is interesting to note that the overall costs of the course are about average. This means that the closer contact with students has been provided essentially by a reorganization of the teaching time budget, not by an add-on of extra resources. Obviously some staff members were misinformed about the relative extent of resources allocated to the course. Questionnaire to other Departments This questionnaire is in summary only since it was not possible to give much time to its development or to use it widely in the university. Questions were asked to determine whether the course was mandatory or optional^how many students took it, how the department concerned got information about the course,' how effective information and interdepartmental liaison systems were, how the Faculty Curriculum Committee was viewed in this context, what intradepartmental machinery existed to monitor such courses in other departments, what expectations the department held of the course in question, and whether and how these expectations had been communicated to the department mounting the course. The answers indicated that the information flow was informal and its effectiveness 43 Course Evaluation for Academic Management: A Case Study in Biology spotty. The Faculty Curriculum Committee was judged ineffective in providing co-ordination and it was questioned whether improved inter-departmental liaison might not be effective. Formal statements of expectations are not exchanged between departments. Overall the co-ordination system appears to be weak. Curriculum Committees are nominally responsible but ineffective. The systemworks tolerably well but some more careful and more effective liaison is needed. This last point echoes that in the comments of the Head of the Department of Biology re intra-departmental co-ordination. Formal mechanisms for co-ordination are in place but they do not function adequately. SUMMARY This paper presents a comprehensive model of course evaluation as it is needed in academic management, plus an example of one evaluation done to illustrate the second stage of this model — a general and comprehensive survey of what aims are being met, what methods seem to be working, what problems exist, how the course is co-ordinated with Departmental and university programs, and what are its demands on resources. The questions which now have to be asked are: 1. Did the evaluation provide the teachers of the course with dependable information which can provide a useful guide to changes which could be desirable in 1977-78? 2. Did the evaluation clarify departmental thinking about priorities and programs? 3. Did the evaluation provide useful information on the adequacy of departmental and Faculty co-ordinating systems? 4. Did the evaluation provide the department head with useful data on intra-departmental consensus or its lack? 5. Did the evaluation give data, however crude, on cost effectiveness? We are satisfied that the process did do all of these things albeit with less precision than one might wish. We feel that the exercise has gone some way to validate the model and that what is most urgent now is the acceptance of the model and the continued refinement of instruments to make it effective. It is worth pointing out that this model is a very far cry from present practices. It argues for a very careful approach by "due process" and for avoidance of incomplete assessment. By implication, and because of the time involved in doing the j o b well for even one course, it argues against casual evaluation of every course every year without guarantee of follow up. It provides, however, for a considerable "ripple e f f e c t " since all staff are involved in the assessment of those one or two courses which a department might do in any one year. It is, we would argue, a vastly more effective model than that currently in place. It is important for sound academic management that we cease to spend significant amounts of student and staff time in a process which is so incomplete as t o be quite as likely to lead in the wrong direction as in the right. We should espouse what we believe t o be an important part of the university ethos "What is worth doing is worth doing well". The follow-up proposed for 1977-78 at Bath will involve a further testing of this model in another and quite different department. One new course at Bath has already (1976-77) 44 W.H. Dowdeswell and H.M. G o o d been evaluated carefully at the stage 3 level but with less attention to the broader departmental involvement than is proposed here. It will, therefore, be an especially useful course for further development of the model. At the same time it is hoped that one or more highly traditional courses which have been rather constant for some time can be examined and that at least one department representing a quite different discipline can be induced t o test the evaluative process as we have outlined it here. REFERENCES 1. Anderson, S.B., et al., Encyclopedia of Educational Evaluation: Evaluating Education and Training Programs. Jossey-Bass 1975. Concepts and Techniques 2. Falk, Barbara and Kwong Lee Dow, The Assessment of University Teaching. by the Society for Research into Higher Education. November 1971. for Monograph published 3. Parlett, M. and Hamilton, D., "Evaluation as Illumination: a new approach to the study of Innovatory Programs". Occasional paper - 9 . Centre for Research in the Educational Sciences. University of Edinburgh, 1972. 4. University of Michigan. Centre for Research on Learning and Teaching. " M e m o to the F a c u l t y " . October 1976.