Evaluation of a Training Course for University Teachers A. I. ROTHMAN* AND SARAH ROBINSON** ABSTRACT This report describes the evaluation of a training course for university teachers offered by the University Teaching Methods Unit of the University of London Institute of Education. The procedures used required the collection of information from the course participants at the conclusion of the course and again after a four month interval. A post course questionnaire asked the participants to assess the success of the course in achieving its stated objectives and to indicate the extent to which the course's scheduled activities contributed to the achievement of the course objectives. Cross tabulation of the results of the two lines of questions identified which course activities contributed to the achievement of specific objectives. The data collected four months later indicated the extent to which and in what areas participants felt the course did have significant impact on their teaching behaviours. RÉSUMÉ Évaluation du cours de formation pour les enseignants universitaires Le rapport présent décrit l'évaluation d'un cours de formation pour les enseignants universitaires offert par l'Institut pédagogique de l'Université de Londres (University of London Institute of Education). La méthode suivie exigea la cueillette de renseignements chez les participants juste à la fin de leurs cours et une deuxième fois quatre mois plus tard. Un questionnaire à la fin du cours demanda aux participants d'évaluer si leur programme atteignit, oui ou non, ses objectifs et d'indiquer jusqu 'à quel point les activités inscrites au cours contribuèrent à la réalisation de ces objectifs. Une distribution des résultats avec des renvois aux deux séries de questions permit d'identifier les activités du cours qui contribuèrent à la réalisation d'objectifs précis. Les données cueillies quatre mois plus tard signalèrent jusqu'à quel point et dans quels domaines les participants estimaient que leur cours avait une influence significative dans leur conduite en tant qu'enseignant. "Professor and Director, Division of Studies in Medical Education, Faculty of Medicine, University of Toronto **Research Officer, University Teaching Methods Unit, University of London Institute of Education 20 A. I. R o t h m a n and Sarah R o b i n s o n It is the purpose of this report to describe the evaluation of a major training course for teachers in higher education and in so doing to suggest a model that may have application with other college and university teacher development programs. The system derives from three basic characteristics of an effective program evaluation system. 1. In order that the evaluation concern itself with the description of course effects viewed and defined in the context of reality, the process should be representative rather than experimental (Snow, 1973). The course did not occur in a rigidly controlled environment that would allow for the testing of isolated and highly specific effects. In fact the course represented a perturbation of an ecosystem consisting of many and various people and institutions where the effects of this perturbation were short term and long, planned and unexpected, profound and trivial, specific and broad. The possible range of effects suggests that ideally the process of evaluation should involve observations from a representative sample of people and institutions from the relevant ecosystem at various points in time following the educational perturbation and at differing "institutional distances" f r o m the point of administration of the educational perturbation. For example, the effects on students of their teachers' previous training as well as the effects on the trainees themselves might be observed. 2. Obviously, the feasibility of the evaluation process depends on its demands on time, people, and finance and requires some compromises with the position stated above. What information can be readily reacted to by the course organizers? When is this information required? What information can signal the success or inadequacy of the course? What is measurable? The answers to these and similar questions should determine to a great extent the nature of the compromises suggested above and the actual characteristics of the evaluation system. 3. The evaluation report should provide the course organizers with understandable information that responds to their concerns about the course and that facilitates rational modification of the course. The information should reach those concerned in a time that is appropriate to its use. The following brief review of literature is intended to: a Describe the status and scope of systematic teacher training in institutions of higher education in the U.K. and elsewhere. b Provide a sample of statements concerning the purposes of courses of this type. c Document the extent to which courses of this type have been evaluated. d Generally, to place the UTMU course for lecturers, its objectives and in particular its evaluation, in an appropriate context. According to Bligh (1975) in the late 1960's and early 1970's training programs were introduced on a large scale in response to widespread concern and discussion about the inadequacy of teaching in institutions of higher education. This response was substantiated by Greenaway's (1971) survey that reported 41 English universities either offering courses in teaching methods to their staff or planning them in the near future. Main (1975a) described this change even more dramatically. "Whereas only 3 (English) universities offered a centrally-organized course of instruction for new teaching staff in 1965, only 1 did not make such provision in 1975." In England the legitimacy of university teacher training was firmly established by the recent agreement of probationary appointments between the 21 Evaluation of a Training Course for University Teachers Association of University Teachers and the University Authorities' Panel. According to Main (1975b), "The institution now has an obligation to offer suitable guidance and training for its inexperienced staff, and these new staff in turn have an obligation to take advantage of the help offered, if the concept of probation is to have any meaning." The world situation, according to Bligh (1975), is characterized by a widespread but uneven introduction of university teacher training courses. He reports vigorous activity in Australia, Sweden and Canada, somewhat less activity although long established in the United States, only isolated development of this kind in Eastern Europe, Asia and South America, and growing interest in Africa. In a more detailed report, Sheffield (1973) cites considerable activity in the Netherlands, the Federal Republic of Germany, Denmark and the Soviet Union in addition to the efforts in Australia, Canada and Sweden cited above. In a similar vein but almost totally independant of the university based activity the world health science community and in particular medicine has devoted considerable resources to the development of teachers. The survey of literature reveals a curious phenomenon. As reported above, considerable training activity has been and is occurring at the university level and within health sciences community. Whereas "scientific reportings" of courses and their evaluations are not uncommon in the medical education journals, similar explications concerning the university based activity have not appeared in recent editions of the general higher education literature. In line with this, a succint prescription relating to the evaluation of health sciences based training programs appeared in a recent World Health Organization (WHO) publication (1973). The WHO author proposed that data be gathered about participants prior to training, during training, immediately upon completion of training, and at some later date and that information also be collected about the relevant performances before and after training in the participants' own setting and that evaluation be carried out by internal and external assessers. Table 1 describes the essential characteristics and the evaluation procedures associated with six training courses for university teachers. These courses should be viewed in terms of their intended effects and documented effects, or at least in terms of their potential for effectiveness. It is felt that an absolute minimum with respect to potential utility required some indication of follow-up investigation beyond just a gathering of immediate post-course data. It may be that these cases accurately represent the state of things with respect to the evaluation of university teacher training efforts. The most recent surveys of trends in teaching and learning in higher education provide no further relevant information (Bligh 1975, Goldschmid 1975). The University Teaching Methods Unit Course for Lecturers — 1975 The focus of this report is the Course for Lecturers — 1975 offered by the University Teaching Methods Unit of the University of London Institute of Education. This course yearly attracts approximately 150 beginning and experienced teachers from institutions of higher learning from the British Isles and abroad. In general, the program is intended to lead participants toward the answers to a series of five questions. These questions are listed in Table 2. 26 Table 3 Six training programmes: Aims, Duration, Number of Participants, Evaluation Procedures Programme Aims Duration Number of Participants Evaluation G. M. Arsham Medicine (1971) refine skills in use of common teaching methods 1 week 25 observations by group leaders during the course, interviews with staff and participants, post-course questionnaire, no follow-up reported. E. D. Prentice and W. K. Metcalf Medicine (1974) knowledge of educational theory; design and justify a complete course; demonstrate skills in using common teaching techniques, aids, and assessing learning and teaching situations 1 l h hours twice a week for 12 weeks 25 post-course questionnaire, no follow up reported J. Anderson et al. — medicine (1972) alter attitudes related to student learning; increase knowledge of educational techniques; demonstrate ability to describe educational processes; develop expertise in curriculum planning, and the planning of instruction, the design of achievement assessment methods; increase interest in medical education 3 days 16 end of course evaluation, and attitude questionnaire. Nine m o n t h follow-up questionnaire directed at documenting attempts to implement course objectives. J. Gale et al (1976) — further follow-up based on questionnaire that enquired of participants' present involvement in medical education. Programme Aims L. R. B. Elton and J. M. King general ( 1 9 7 5 ) " T o introduce new entrants to the profession to the teaching side of their work and to discuss problems in teaching and ' learning in higher education, with special reference to mathematics, science and engineering." E. G. Cantrell Medicine (1972) Directed at teaching performance and skills rather than theory. I. D. Gregory and Berenice Hammar — medicine ( 1 9 7 6 ) " T o help staff to develop further their knowledge and use of teaching methods and skills presently used in their teaching." Duration 7 days days 21 hours over three weeks Number of Participants Evaluation 32 —pre-course questionnaire, daily evaluation sheets, post-course questionnaire; and follow-up interviews during the year following the course. 28 —a questionnaire administered on the last day of the course and a second questionnaire distributed several weeks after the course. 21 —end of course questionnaire and a discussion of the questionnaire, —no long term follow-up. 24 A. I. R o t h m a n and Sarah Robinson These broad aims are expressed more explicitly in the course handbook as a series of 14 specific objectives. These objectives appear in Table 3. Participants were assigned to groups reflecting the extent of their teaching experience (beginners/experienced) and their academic discipline (arts, bio-medical, physical sciences, history). Each group followed a time table that was a mixture of activities unique to that group, for example, a discussion of outlines of units of work provided by the group members and the opportunity to present a brief lecture that was videotaped, played back and discussed by the group, the discipline group tutor and a speech tutor; and activities intended for all participants, for example, a lecture on "Student Problems." In addition, there was a schedule of optional evening activities including discussions on games and simulations, student counselling and the use of the student health service, and group interactions as demonstrated in the film "Twelve Angry Men." The course was residential, and occupied four and one-half days. The Evaluation Process Two levels of evaluation were employed. The first was an end of course questionnaire after the design described by Byrne and Rothman (1975). The questionnaire required participants to indicate the extent to which they achieved the general and specific course goals. In addition, participants were asked to indicate whether the times spent in discussing various topics were adequate and to indicate the extent to which the various scheduled activities had contributed to the achievement of the course objectives. The results of this questionnaire indicated which of the activities were being considered by the participants to be relevant to the achievement of the objectives. In addition, an examination of the relationships between the objectives items and the activities items identified which activities contributed to the achievement of specific objectives. Even more vital to the course organizers was evidence that would indicate that the training model did have some impact on teaching behaviour in the longer term. The second level of evaluation was intended to deal with this issue and was based on a questionnaire sent to the participants approximately four months after the course ended. The questionnaire asked participants whether what they learned in the course had been of practical relevance to their teaching, had a significant effect on their style of teaching, on their organization and planning of teaching, on their relationships with students, on their relationships with their colleagues, on their styles of lecturing, on their confidence as teachers, and on the extent to which and the way they used the various instructional media. They were asked a short series of questions that focused on the issue of continued learning relative to teaching. Beginners were asked specifically whether what they had learned in the course significantly helped them adjust to the realities of teaching in higher education. The participants were asked to identify which parts of the course were most important for them and which parts of the course they felt were well taught. Finally, the participants were asked to describe the overall aim of the course. 25 Evaluation of a Training Course for University Teachers Results and Discussion Part 1 - Results of the Post Course Questionnaire The Post-Course Questionnaire was responded to by 114 of the 132 course participants (83.3%). Of the 114 respondents 46 were beginners and 68 were teachers with some teaching experience. The first section of the questionnaire asked the participants to rate as excellent, good, fair or poor the success of the course in answering five questions that defined the course's general aims. Table 2 reports the proportions (percentage) of the total group, and the beginners and experienced teachers sub-groups that rated these items either excellent or good. Table 2 Proportions (%) of Total Group, and Beginners and Experienced Teacher Subgroups responding either Excellent or Good to General Aims Items < L > c c 'ob a > 03 a. x w H o 1. What different learning outcomes should we try to achieve with our students? 46 42 43 2. How do students learn? 34 21 27 3. What different teaching activities are available for achieving learning outcomes? 79 63 68 4. How can learning and teaching be evaluated? 52 39 45 5. What realistic steps can you take to develop your abilities as a teacher? 61 61 61 54 45 48 Mean Percents It is not possible to assign meaning in an absolute sense to the data presented in this and the following tables. There is the implication that excellent or good ratings suggest adequacy whereas fair or poor ratings suggest inadequacy. However, whether or not 48% of the responding participants found the course adequate in dealing with its major aims is a statement of fact that has significance when considered alone, is a question better left to the reader or better still left alone. The investigators propose that the listed values have meaning and 26 A. I. R o t h m a n and Sarah Robinson can be used to identify the strengths and weaknesses of the course when examined relative to other data appearing in the tables and to the mean values at the b o t t o m of each table. For example, the data in Table 2 indicate that the participants found the course reasonably successful in identifying the different teaching activities available to the teacher (Item 3), and in the practical steps they could take so as to improve their own abilities as teachers (Item 5). In comparison, participants reported the course less successful in defining appropriate learning outcomes (Item 1), and in dealing with the evaluation of learning and teaching (Item 4); and least successful in its handling of the topic "How do students learn? (Item 2). The mean values in the b o t t o m row indicate that the beginners reported the course more successful than the experienced teachers in achieving its major aims. The data in Tables 3 to 5 summarize the participants' responses to the items relating to the achievement of specific objectives, the appropriateness of the times spent discussing a series of topics, and extent to which the scheduled learning activation contributed to the achievement of the courses activities. The first two sections in the post-course questionnaire asked the participants to rate the success of the course in achieving its objectives. The second two sections asked the participants to indicate the appropriateness of the various discussions and activities they had experienced during the course. By calculating the extent of relationships between items in the two main parts of the questionnaire, that is, by correlating each of the "general aims" and "objectives" items with each of the "activity" and "discussion topic" items, it was possible to identify which discussions and activities in the participants' opinions contributed to the successful achievement of particular general aims and objectives. In brief, cross tabulations were calculated for each possible pairing of items from the two major parts of the questionnaire. Where a significant relationship was indicated (Chi Square) the extent of the relationship was calculated (Cramer's V)*. Where a relationship for a pair of items was positive and significant ( a = 0.05, one tailed) it was concluded that the discussion topic or activity being considered contributed to the achievement of the related objectives. In total 380 objectives by activities item pairs were considered. Fifty-one relationships were significant at p < .05. The complete results are available on request. Table 6 gives two examples of these results and permits demonstration of the type of information made available by this analysis. Table 6 indicates that four of the discussion topics and one activity were linked with the general aim "How do students learn?". Under ideal conditions it should be assumed that the general aims would be widely represented among the discussion topics and activities. The data indicate that this may have happened. However, Table 2 reports that only 27% of the participants may have felt that the question "How do students learn?" had been adequately answered during the course. If the course organizers were committed to the achievement of this general aim, it may be that this topic should be treated more explicitly during the course. Also, some specific references to the issues involved should appear in several of the scheduled activities. Whereas the general aims should be broadly represented among the discussion topics *Cramer's V is a measure of association for contingency tables that varies between 0 and 1 for all tables regardless of the number of rows and columns. It is identical to the Phi Coefficient in t h e case of two by two tables. Table 3 27 Beginners Proportions (%) of Total Group, and Beginners and Experienced Teachers Subgroups Responding either Excellent or Good to Specific Objectives Items -a < u o c <t> aj o, X W 1. be thinking increasingly about your own learning needs in respect of teaching, and how they might be met. 66 77 72 2. be familiar with some of the technical terms of educational theory and practice. 36 35 36 3. be able to specify both specific and general objectives relating to your teaching. 67 53 60 4. be aware of a variety of teaching methods and be able to discuss them critically. 76 63 68 5. be aware of various methods of assessing students and be able to discuss them critically. 67 50 57 6. be aware of various methods of evaluating your teaching and be able to discuss these critically. 50 38 44 7. be aware of some of your own strengths and weaknesses in presenting a teaching session. 82 70 75 8. be able to identify some of the factors affecting a student's ability to profit from his study. 37 39 39 9. have developed some understanding of group behaviour. 38 67 54 By the end of the course you should: "3 — t -o> H Should Have Gained Some Skill In: 10. selecting appropriate teaching methods for the achievement of given objectives. 55 50 50 11. use of audio-visual aids. 58 49 52 12. presenting an effective lecture 74 64 69 13. leading group discussions 44 54 45 14. planning the whole or part of a course, or a single session of teaching. 54 59 57 Mean Percents 57 55 56 28 A. I. R o t h m a n a n d Sarah R o b i n s o n Table 8 Beginners Experienced Total Proportions (%) of Total Group, and Beginners and Experienced Teachers Subgroups Indicating Appropriate Time Allotment to Discussion Topics 1. questions concerning the underlying philosophies of courses 47 58 54 2. questions about teaching directly relevant to your areas of interest 52 65 59 3. how to determine and describe what you want students t o learn 65 59 61 4. the common learning and emotional problems in students and h o w to cope with them 67 47 56 5. the assessment of students 65 56 60 6. the evaluation of teaching 44 48 47 7. the applications of learning principles to teaching in your field 44 47 45 8. the rational use of educational media (e.g. television, overhead projector, tape-slide programmes) 62 63 63 9. ways of arousing and sustaining student motivation 53 49 51 individual differences in learning 33 35 34 Mean Percents 53 53 53 10. and activities, it would seem that the specific objectives should be related to only one or two of the program's activities or discussions. For example, by the end of the course it was expected that the participants should be aware of various methods of evaluating their teaching and be able to discuss these critically. Table 2 reports that 44% of the participants may have felt that they had adequately achieved this objective. Table 6 indicates that only the discussion of individual differences in learning 29 Evaluation of a Training Course for University Teachers Table 5 Beginners Experienced Total Proportions (%) of Total Group, and Beginners and Experienced Teachers Subgroups Responding either Excellent or Good and in ( ) brackets did not participate to Items Relating Activities to the Achievement of Objectives 1. Preparation of the short lecture 74( 2) 54( 3) 63( 3) 2. Videotape and critique of the short lecture 94( 2) 76( 3) 81( 4) 3. Preparation of the course unit outline 51( 9) 49(11) 49(11) 4. Critique of the unit outline 41(11) 56( 9) 49(10) 5. Reading of background papers 44( 2) 26( 1) 33( 3) 6. Advanced reading of prescribed book/s 18(52) 19(50) 19(50) 7. Introduction to the A. V. Centre and subsequent discussion with Centre staff 39( 6) 32(23) 36(16) 8. Lecture on student programs 72( 4) 67( 5) 69( 5) 9. Critical summary of the short lectures — speech tutor 64(20) 77( 8) 71(13) Evening Programmes 31(35) 45(24) 40(28) Mean Percents 53(14) 50(14) 51(14) 10. was related to the achievement of this objective. Furthermore, it is apparent that this discussion was not primarily directed at the teaching of this particular objective. If the course organizers are committed to the successful achievement of this particular objective, then this topic should receive explicit attention in a discussion and/or a scheduled activity. The array of relationships can also be examined to assess effectiveness of particular features of the course relative to the achievement of the course aims and objectives. For example, the course organizers were particularly interested in assessing the contribution of the background papers to the overall success of the course. Example data relative to this question appears in Table 7. Reading the background papers was significantly related to seven of the 19 aims and objectives. In absolute terms the relationships were low, and on the average they were in 30 A. I. R o t h m a n and Sarah R o b i n s o n Table 8 Activity and Discussion Topic Correlations of Selected Course Objectives — (Cramer's V) Objectives Activities General Aim — How do students learn? • Discussion — concerning the underlying philosophies of courses (0.22) • Discussion —how to determine and describe what you want your students t o learn (0.18) • Discussion —ways of arousing and sustaining student motivation (0.33) • Discussion —individual differences in learning (0.21) • Activity —reading the background papers (0.27) Specific Objective — By the end of the course you should be aware of various methods of evaluating your teaching and be able to discuss these critically. • Discussion —of individual differences in learning (0.26) the lower part of the distribution of the 51 significant relationships. The background papers were intended to serve as the major written resource for the course. Significant relationships with a larger number of the aims and objectives and with larger absolute values, particularly in the cases of those objectives where background papers serve as the only relevant input, might have indicated the adequacy of the background papers in their designed role. Part 2 - Results of the Follow-Up Questionnaire Approximately four months after the end of the September course, a follow-up questionnaire that was intended to sample the longer term effects of the course was sent to all participants. This questionnaire consisted of 16 open-ended questions that were directed primarily at identifying those aspects of the course that had had some practical effects on the participant's teaching. The questionnaire was also intended to elicit the participants' retrospective definitions of the important parts of the course and the course objectives, and their assessments of the quality of teaching in the course. This in effect was to constitute the summative evaluation of the course. 31 Evaluation of a Training Course for University Teachers Table 7 Aims and Objectives Whose Achievements are Related to Reading the Background Papers (Cramer's V) Activity Reading Background Papers Aims and Objectives - How do students learn? (0.27) - What realistic steps can you take to develop your abilities as a teacher? (0.22) - Be familiar with some of the technical terms of educational theory and practice (0.16) - Be aware of a variety of teaching methods and be able to discuss them critically (0.24) - Be aware of various methods of assessing students and be able to discuss them critically (0.21) - Be able to identify some of the factors affecting a student's ability to profit from his study (0.19) - Be able to select appropriate teaching methods for the achievement of given objectives (0.17) Two mailings provided an overall response rate of 61% (81/132). Table 8 lists the first 14 questions from the questionnaire and provides the number of participants responding yes or no to each question. The judgement as to what number of participants responding affirmatively to any question constitutes success for the course is surely dependent of the focus of the question. For example, a total of eleven people responded yes to the question concerning an involvement with innovations or changes in teaching practice that could be attributed to the course. If one considers the constraints on innovation in teaching in institutions of higher learning then this number of yeses should in fact reflect favourably on the September course. By contrast it seems somewhat unfortunate that only thirteen beginners responded affirmatively to the question that asked whether the course significantly helped them to adjust to the realities of teaching in higher education. 32 A. I. R o t h m a n and Sarah R o b i n s o n Table 8 Numbers of Beginners and Experienced Teachers Responding Yes or No to Questions 1-14 of the Follow-Up Questionnaire Question Beginners Yes' No Experienced Yes No Has what you learned at the course: 1. been of practical relevance to your teaching this year? 18 13 36 11 2. had a significant effect on your style of teaching this year? 14 17 26 20 3. had a significant effect on your organization and planning of teaching this year? 9 22 18 29 4. had a significant effect on your relationships with students? 3 28 10 34 5. had a significant effect on your relationships with your colleagues? 2 29 10 35 6. had a significant effect on your style of lecturing? 14 17 20 25 7. had a significant effect on your confidence as a teacher? 17 14 25 20 8. had a significant effect on the extent to which and the way you have used the various instructional media such as television, tape slides, etc. in your teaching this year? 4 27 17 30 9. Have you enrolled in any courses or conferences relating to teaching in higher education since the September Course? 1 30 8 41 10. Have you introduced or been involved with innovations or changes in teaching practices since the September course that you could attribute to the course? 3 28 8 38 11. Have you contacted any of the course tutors over the past few months? 2 29 11 38 12. Have you read or consulted any of the background papers since the course? 7 24 14 35 33 Evaluation of a Training Course for University Teachers Beginners Yes No 13. Have you had any contact with any of the other course participants since the course? 14. For beginners 16 15 13 Experienced Yes No 19 30 18 only. Has what you learned at the course significantly helped you adjust to the realities of teaching in higher education? Summary of the Open-Ended Responses to All Seventeen Questions Many of the course participants find themselves in established teaching situations which are rigid and conservative. The majority, and especially the beginners, are junior members of the staff, who feel unable to introduce change or plan at the course level at this stage in their careers. Clearly this has implications for the design of the course, in that too great an emphasis on innovatory teaching methods may prove to be frustrating. The videotape lecture and its critique was frequently mentioned as being of great help. The respondents reported that this exercise built confidence and enabled them to see their weaknesses and strengths in the lecturing situation. It would seem advisable therefore to organize the course in future in such a way that all participants would have maximum exposure to this activity. The participants reported that they did not use audiovisual aids to any great extent. Some remarked that their home facilities were primitive, or that access to them was difficult. Simple and inexpensive devices like the overhead projector were mentioned several times. It seems that it would be best to concentrate on these simple aids, like the overhead projector, 35 mm slides and projector and flip charts and ensure that all participants have the opportunity to become familiar with them. There was considerable variation in the answers to the open-ended questions concerned with identifying the parts of the course that participants saw as most and least important. It may be that choice of activity and areas of skills and knowledge development should be a significant aspect of the design of future courses. Participants reported little ongoing contact with their course tutors and with other course participants after the course. Where such contact occurred it tended to be in chance social encounters between colleagues in the same college or department. This suggests that if continued contact is deemed advisable then some formalized system encouraging such contact should be developed. The long term impressions of the course appeared to be favourable. When asked to reflect upon the overall aim of the course the majority of participants indicated that the aim had been to make them better teachers, and most of them also indicated that this had been achieved in their cases. 34 A. I. R o t h m a n and Sarah Robinson Summary The basic assumptions underlying any teacher development program are that the efforts involved should lead to relatively permanent teacher behaviours that can be empirically linked with " b e t t e r " student learning. Furthermore, that the rational modification of these programs in response to measures of participants' learning or opinions will increase the probability of appropriate teaching behaviours and hence "even b e t t e r " student learning. There is no evidence in the literature that these assumptions have in fact been tested and verified. Nor, because of the practical considerations listed in the introduction to this report, did this study use procedures that would in the most direct sense (competent systematic observations of teaching linked with appropriate measures of student learning) approach these issues. However, because of the explicite nature of the information provided and the inclusion of perspectives derived from the longer term examinations of impact, the results of the evaluation contributed to rational planning decisions for the September 1976 course. For example, the course organizers concluded that in the time available all participants could not and would not want to learn everything. Since it was reasonable to expect that participants should develop expertise and skills only in areas relevant to their interests and teaching situations, it was proposed that the participants should have the opportunity to select a major part of the course from a series of options. However, it was intended that certain philosophical positions should pervade all parts of the course. In conclusion, the investigators suggest that the objectives, learning activities, long term follow-up model as described is realistic in terms of its financial and personnel costs, and is justifiable in terms of its ability to provide within a reasonable period of time sufficiently detailed and useful information to course organizers. Bibliography R. E. Snow (1974). "Representative and Quasi-Representative Designs for Research on Teaching," Review of Educational Research, 44, 265-291. Donald Bligh et. al. (1975). Teaching Students. Exeter University Teaching Services: Devon, England. Harriette Greenaway (1971). Training of University Teachers. Society For Research Into Higher Education Limited: L o n d o n . Alex Main (1975 a). The Training of University Teachers in the United Kingdom. Publication of the Co-ordinating C o m m i t t e e for the Training of University Teachers. Alex Main (1975 b). Impetus. Newsletter no. 2 of the Co-ordinating C o m m i t t e e for the Training of University Teachers. E. F. Sheffield (1973). " T o w a r d the Training of University Teachers." Address to recipients of the degree of Bachelor of Education, University of T o r o n t o . G. M. Arsham (1971). " A n Instructional Skill Workshop for Medical Teachers: Design and E x e c u t i o n . " British Journal of Medical Education, 5, 320-324. E. D. Prentice and W. K. Metcalfe (1974). " A Teaching Workshop for Medical E d u c a t o r s , " Journal of Medical Education, 49, 1031-1034. J. Anderson et al. (1972). " T h e Workshop as a Learning System in Medical Teacher E d u c a t i o n , " British Journal of Medical Education, 6, 296-300. L. R. B. Elton and J. M. Kilty (1975). "Courses in Higher Education at the University of Surrey," in Issues in Staff Development. University of London Teaching Methods Unit, L o n d o n . 35 Evaluation of a Training Course for University Teachers World Health Organization Technical Report Series No. 521 (1973). Training and Preparation of Teachers for Schools of Medicine and Allied Health Sciences. World Health Organization Geneva. E. G. Cantrell (1972). "A Course in Teaching Methods (A Consumer R e p o r t ) , " British Journal of Medical Education, 6, 37-43. Janet Gale et al. (1976). "Changing Attitudes of Medical Teachers Towards Medical E d u c a t i o n , " Medical Education, 10, 250-254. I. D. Gregory and Berenice Hammar (1974). "Case Study of First Course in Teaching Skills and Methods for University Medical S t a f f , " British Journal of Medical Education, 8, 92-98. M. L. Goldschmid (1975). "Teaching and Learning in Higher Education: Recent T r e n d s . " Address presented at The Third International Conference on Higher Education, September 1-5, University of Lancaster, Lancaster, England. P. N. Byrne and A. I. R o t h m a n ( 1 9 7 6 ) . " A n Approach t o Program Evaluation in Medical E d u c a t i o n . " Paper presented at The Annual Meeting of the Association for the Study of Medical Education, L o n d o n , September, 1975. Abstracted in Medical Education, March 10, 137-138.