Instructional Development—What? Why? How? H. M. GOOD* I. Introduction Instructional development is a process with compelling internal logic and proven applicability to the management of academic systems. Yet it is widely ignored, or misunderstood, or misapplied. Indeed the misunderstanding is so pervasive that it is worthwhile pointing out what instructional development is not as a preamble to saying what it is. It is not a movement to sponsor wider use of audio visual aids, or television, or computers, or independent study, or any other specific instructional methods. It is not a system aimed primarily at saving money. It is not a system for sponsoring innovations for their own sake. It is not a discipline or a management science requiring new or highly technical competencies. It is quite simply a systematic way of applying to the design of instructional materials and programs the care and standards of critical thought taken for granted in all the scholarly disciplines. This claim will be expanded by dealing with three questions: What is instructional development? Why is it likely to be useful? How can its value be realized? However, because it is a process the what and how are almost inseparable, and because the process concerns itself very much with a critical linking of the objectives of a program to the methods, none of these questions stands alone. It is perhaps presumptuous to include How in the title since this paper in no way sets out to be a guide to the design of instructional programs. The steps in the process have been written about in detail elsewhere. All that is attempted here is to deal with certain general and central problems not dealt with in some of the more technical papers on course development, and to offer the reader an outline of the essential logical steps and relationships. Although instructional development can only be defined by a detailed examination of the whole process a condensed initial definition must be attempted. As used in this paper instruction refers broadly to any facilitation of learning. Development refers to change over time with progression toward some more mature or more adapted condition. Adapted may refer to greater achievement, or fitting in more realistically with cost constraints, or having a more lasting influence on students. The criteria for adaptation will be complex and will vary with the circumstances. *Professor of Biology, Queen's University. Currently on secondment as Director, Ontario Universities Program for Instructional Development. 34 H. M. Good In practice instructional development involves conscious effort to proceed "systematically." Systematically as used here implies two things. First, a logical progression from careful enunciating of objectives, to development of a plan for achieving these, to an evaluation of success, to reconsideration of the objectives, to an application of the results of the evaluation to improving the process. Second, a conscious effort to ask all the relevant questions, i.e., to see the whole system into which the program being developed fits. The need to consider the whole system in which a program operates arises from the possibility that important feedbacks may occur producing effects which are either unexpectedly amplified or produce influences the very reverse of those intended. The emphasis on asking all the questions, i.e., developing a holistic or systems-conscious approach, and watching carefully for side effects, is a vital part of the concept of systematic development. The whole process is admittedly easier to describe than it is to follow. However, although there are formidable difficulties, the system can be made rigorously logical. Most current procedures in our universities fail signally to give the attention to internal logic and consistency implied in the systematic approach to design of teaching materials and programs. Given the ostensible, indeed often ostentatious, devotion of the university to "critical thinking," this deficiency represents an inconsistency which should give us real concern. Deficiencies in the process for planning and developing teaching materials and approaches are not of course fully generalizable. This paper is written in the context of the Canadian university system where the opening statements are, by and large, justified. However, in many universities (most notably in Britain, Australia and the U.S.A.) Centres for Instructional Development have been established. These vary but normally provide advisory services on course development, evaluation, or program design. A few sources of information about these centres are listed at the end of this paper. II. Why? - The Potential Contribution Instructional Development as defined herein can contribute to a university in a variety of ways. Indeed from its emphasis on a holistic and internally consistent approach it can contribute to all phases of the operation. But there are costs, since the process is not a simple one and there are risks that it will be followed in a partial way which may be useful but may equally leave the whole system at the mercy of one key illogicality. With a caveat about the dangers of partial application, it is relevant to look at some areas in which there is evidence that the systematic approach holds out promise of a return which is considerably larger than the outlay. The areas considered fall into three general categories which are of course not entirely discrete. The first of these concerns the logic and consistency of institutional purposes and actions viewed in relation to the responsibilities of staff and their perceptions of these responsibilities, i.e., with accountability to Society. The second concerns Instructional Development and adaptability. The third deals with internal benefits stressing accountability of staff and administration to other members of the University Community. Consistency of Institutional Purposes and Actions: There is a failure, widely recognized 35 Instructional Development by students and the public, that universities do not now achieve logical consistency in setting and pursuing broad policy. A look at certain aspects of staffing policy and rewards systems will illustrate this and point generally toward the kinds of action needed to correct any discrepancies which might be diagnosed through the kind of analysis proposed. A university teacher may be said to have a responsibility to contribute to each of three interconnected sets of institutional objectives which we may designate as the Institutional-Societal, the Scholarship-Discipline, and the Student-Teaching. These may be represented by the three apices of a triangle. INSTITUTIONALSOCIETAL STUDENTTEACHING SCHOLARSHIPDISCIPLINE Any one staff member will weigh these responsibilities differently so that one might be represented by a dot in the middle of the triangle, another well toward the Scholarship-Discipline apex, and still another toward the Institutional-Societal apex. The main thrust of the past few decades, widely discussed in the literature but to some extent unrecognized by those who might have modified the trends, has been toward a heavy emphasis on the Scholarship-Discipline loyalties. In terms of our graphic representation the figure has become like this for most universities. INSTITUTIONALSOCIETAL SCHOLARSHIPDISCIPLINE STUDENTTEACHING 36 H. M. Good There is nothing inherently wrong with this kind of distribution if it arises from a deliberate weighting of the three responsibilities for that university which is willing to justify this weighting to students and to society. However, the distribution of staff in this diagram is now, for many universities, appreciably out of balance with respect to either that pictured in statements of senior university officials or that considered acceptable to students or public. The argument that the Discipline-Scholarship loyalty feeds directly into the TeachingStudent loyalty making the distinction between these responsibilities unnecessary needs to be considered. However the frequent, and perfectly open, decisions by staff to economize on teaching effort in order to provide research output because that is what will be noticed and rewarded makes it clear that the distinction is in practice a sharp one. This is not, of course, to say that active research interests do not have a valuable spin-off into teaching. It is simply to argue that effort going into research does not automatically lead to good teaching. Trying to rationalize teaching responsibilities on the basis of the argument that it does is simply dishonest even if there are a few whose responsibilities are to teach only at a very advanced level and for whom the distinction does admittedly become blurred. Instructional Development as defined herein can be applied to consideration of staffing the overall system and can provide for a corrective feedback aimed at achieving an acceptable objectives — decisions match when the purposes of the institution and the distribution of staff in the triangle are visibly inconsistent. It should be stressed, however, that balance does not imply uniform distribution of dots but only distribution which is rational and effective in terms of the priorities of the institution being considered. This simple triangular schema can be used to sharpen questions about the weighting of institutional objectives, the criteria for selection of teaching or administrative staff, the priorities for different kinds of programs, and the design of appropriate evaluation criteria for rewards as well as for appointments. The distribution of staff in the triangle is clearly a function of both selection and rewards, and it can be altered significantly through adjustment of the rewards system. The schema is, of course, intended to be conceptual rather than a basis for a numerical formula. It is intended to present graphically the kind of analysis of general policies through which Instructional Development or "systematic" analysis could make a contribution to the effective and efficient operation of an institution. Instructional Development and Adaptability: The relation of instructional development to adaptation of the university to changing political or economic conditions, and especially to the maintenance of the more important values when hard times impose specially rigorous financial constraints should be obvious; but from the common tendency to stand firm on traditional ground under these conditions it is clear that this connection has been generally missed. In times of affluence there is no pressure to be critical or experimental and in times of economic strain there is a tendency to take a defensive position behind the arguments — demonstrably false — that instructional development is a frill and impossible to apply without supplementary resources. Yet instructional development as defined in this paper is the sine qua non of effective adaptation and adjustment to minimal resources. Biological evolution provides a useful 37 Instructional Development analogy here. It leads to continued change and adaptation to the environment through mutations acted upon by rigorous selection. In instructional development critical appraisal of what we are doing leading to new proposals is the equivalent of genetic mutability and evaluation with corrective feedback is the equivalent of continuous natural selection. Leave either of these functions out of the adaptive system and it fails in logic and in effectiveness. The process of academic adaptation is therefore another way of defining instructional development. The need rises with the rapidity of change in the environment. In times of financial crisis it is more important, not less. The risks of getting one's values out of order, or of pursuing policies for historic rather than rational reasons becomes greater under conditions of stress. The systematic approach to reappraisal of basic assumptions, to the ordering of values, and to the questioning of traditional methods then becomes of vital importance. In arguing the importance of systematic evaluation and reappraisal one is not, of course, arguing that what the universities now do will need wholesale revision. The logic of development of many specific disciplines is clear and demonstrably effective. But with respect to the deployment of teaching resources, the logic is less clear and indeed in some cases appears faulty. Since this is where the bulk of resources is expended, it is one area which invites prompt and critical reappraisal. It would, however, be entirely unsystematic to rush into the reconsideration of method without relating this critically to objectives and values just as it is unsystematic to hold so dogmatically and uncritically to values that methods cannot be looked at freely. Only out of an integrated reappraisal of values and methods can come improvement and adaptation. It is just such an approach which is the key to, and core of, instructional development. Instructional Development and Internal Accountability: To shift now toward the implications of Instructional Development for the improvement of internal accountability, a further advantage of the systematic approach is the pinpointing of the real basis for decision making. Because systematic development requires that the rationale for each option be enunciated the basis for decisions can be challenged on logical grounds — often with the discovery that a decision to support project A has been rejected while there are projects of lower priority being supported. The decision may, therefore, have been made by invoking a self-imposed or imaginary constraint perhaps to sidestep the need to eliminate programs which have been established but cannot be justified competitively. Admittedly, there are problems of responsibilities to personnel and of fair notice in discontinuing established programs but failure to do so puts the university in an evolutionary strait-jacket which can only lead to disaster. Another useful by-product of instructional development is improved effectiveness. Here it is appropriate to draw on an example in which the Ontario Universities Program for Instructional Development has been involved and to cite the cooperative development of the first year chemistry course by Professor David Humphries of McMaster University and Professor Ronald Martin of the University of Western Ontario. Achievements of this project in 1973-74 were impressive. The failure rate of first year studies was reduced, average standing as judged by a comparable or identical exam was increased, and it was judged feasible to increase the content of the course significantly if this were 38 H. M. Good desired. There was also some evidence of development of better study habits. One more advantage is increased efficiency. Again the McMaster-Western chemistry project will serve as example. Quantitative data for claiming increased efficiency are less clear than for effectiveness but in the judgment of the departments involved the evidence is adequate. The increase in efficiency can be seen at University of Western Ontario in the substitution of one self-paced non-lecture course for nine parallel traditional lecture courses. The judgment of the department was that the parallel preparation and delivery of nine essentially identical lectures was not an efficient way to use teacher time. The self study material, while costly to develop, could, once developed, permit the redeployment into interactive teaching or research, or whatever, of a significant part of the teacher time previously assigned to didactic presentation. Yet another advantage is improved student attitudes. Again this was demonstrated in the chemistry program and it has been shown in several redeveloped courses at Syracuse and at other universities. A final advantage is that some carefully redeveloped courses have shown considerably improved retention of learning. Indeed, successes have been fairly consistent when projects have been given adequate support and staff assigned who were competent managers and either familiar with the essentials of systematic development or knowledgeable enough to know when they needed help. Projects which did not meet these conditions have all too often failed. The moral is that the whole process works but fragments of it usually do not. This view is strongly supported by the experiences of Britain's Open University, discussed in a later section. Summary of Advantages: The advantages claimed for systematic development range over a broad spectrum of the universities' activities. Indeed they must do so if the whole system is kept in view. A corollary is that the benefits derived are not for any specific sector of the university but are distributed throughout and in a manner appropriate to the objectives of the institution. Thus, in the example cited above, students clearly profited since they preferred the revised chemistry course and learned more. Moreover, they appreciated the fact that there was in the method of development a clear-cut way in which student comments and achievements or failures were cycled back into the revision process. This removes a major student concern that their efforts in evaluating courses are less effective than they might be because the information is not systematically used. Faculty profit because the pre-planned use of student reaction removes much of the threat of evaluation imposed by ad hoc student evaluations not linked to a formal and planned revision process. Faculty also profit by release of time from unproductive effort or from a type of instruction which they dislike. Finally, administrative officers profit because, as the rationale becomes clearer, waste can be reduced and brush fire or ad hoc administration replaced with more rational and more long range planning which is required to keep the university in a harmonious relationship to the society which supports it. III. How? — Steps in the Process and Some Central Problems If university programs are to be developed in a fully systematic way then there will have 39 Instructional Development to be an evolution of new working arrangements which, given the nature of the university, must be conceived as coming gradually. Some of these must develop in relation to the individual teacher, some at the departmental level, some at the institutional level. To attempt to discuss each level of the process lies outside the scope of this paper. Therefore, although differences in emphasis are recognized as appropriate to different levels, this section has been written without distinction between issues of primarily individual or departmental or institutional responsibility. The section covers: essential steps in the systematic method, the evaluation problem, and teaching — the choice of method (which includes discussions of the distinction between the textual and interactive systems). These are topics selected to illustrate the process more fully and to explain the basis for the conclusions and recommendations section. They do not attempt to provide comprehensive coverage of the method. Steps in the Method: The essential steps of the method described herein as the essence of instructional development are: (1) Careful enunciation of objectives. (There is a temptation here to say precise instead of careful but this can lead to preoccupation with unimportant or even trivial objectives just because they can be stated and measured precisely). (2) Definition of the population to be instructed with special attention to assumptions regarding prior knowledge. (3) Enunciation of the criteria for success in achieving the objectives. (4) Development of a system for determining whether the criteria for success have been met. This is essentially the development of the process for evaluation of both the instructional material and the students. In terms of instructional development, it is the evaluation of the materials rather than of the students which is the more important. This type of evaluation is almost inseparable from the evaluation of teaching and hence of teachers. This will be dealt with in more detail in the section dealing with the evaluation problem. (5) Examination of all ways of moving the population to be instructed from the level implied in the assumptions to the level implied by the statement of objectives and selection of the most effective overall mix of methods given the applicable constraints. These steps imply the making of all the decisions involved in designing an instructional program to move the students from where they are to where it is hoped they will be. (6) Delivery of the instructional package to the students. (7) Application of the system of evaluation. (8) Iteration of steps 1 to 6, to eliminate unrealistic or trivial objectives, modify badly stated ones, or introduce new ones; and to change the methods of instruction or modify materials as needed. This is the corrective feedback loop of the systematic approach. It stresses the importance of evaluation of the instructional materials for modification and it implies a continuous rather than a one-shot process. Two ideas in these steps are of special importance. The first is the instruct-evaluatemodify cycle which is a central concept in instructional development. Indeed without careful attention to this cycle the word development has no meaning. The second concept which is central to Instructional Development is imbedded in step 4 which calls for an examination of all possible methods and selection of an optimal 40 H. M. Good mix. This requires the holistic or systems-conscious viewpoint referred to in the Introduction and implies a careful ordering of decision making so that the inevitable trade-offs can be made wisely. It involves a form of academic management which is today sadly lacking in universities. There is no implication in any of the steps outlined that new methods should be used although there is an implication that new as well as old methods will be looked at critically to determine which are most appropriate. Much "innovation" in teaching is little better than isolated and ad hoc experimentation. Although such experimentation may on occasion score a spectacular success it lies outside instructional development as defined in this paper. In this brief account it is not possible to deal with each step of the development process fully. Much has been written about the formulation, use, and abuse, of objectives. A pious and formal enunciation of objectives cannot of itself do much. But without reasonably clear objectives there can be no effective collaboration between staff members, or between staff and students, and there can be no valid judgements about success either of students or teachers. Clarification of objectives is essential to effective communication. Important questions about whether duplication of effort is legitimate, about what is best eliminated if the pinch is really on, and a list of other decisions requiring critical discussion compel careful attention to objectives or the discussion becomes diffuse and useless. Participants are talking at, rather than to, each other. Regrettably, the enunciation of objectives has been developed by many as a way of formalizing the teaching and examining in a specific course and this formalization has often been carried to an extreme which stressed trivial matters. "Clarification of objectives" is a phrase which often produces a negative reaction. Yet the decision making process has no logical integrity if this is omitted or done sloppily. Stressing the major objectives or aims as an aid to effective communication between colleagues who are, at least in their own disciplines, trained to smoke out inconsistencies and intellectual inadequacies, seems a useful and valid aspect to stress in developing this part of the system. Several of the problems inherent in steps 3, 4, and 5 need to be considered since they arise frequently and bear on the areas in which Instructional Development may be most valuable. They are important in themselves, and are appropriate starting points for a systematic review. It is true that, once started, the iterative systematic method expands to cover the system. However, the selection of points of entry is by no means inconsequential. Hard thought is required and it is not likely to be provided unless the issues initially looked at are seen as of immediate interest and practical significance. The topics selected for illustrative discussion are Evaluation which bears directly on steps 3 and 4, and Textual Materials and Interactive Teaching which deal with the use of teacher time and bear on step 5 in the process. The Evaluation Problem and a Possible Model: The case for careful and critical evaluation is simply that without it we have no basis for distinguishing progression from retrogression. Unless one is prepared to argue that university instruction is perfect both for the present and the foreseeable future and that change is to be actively discouraged, one must argue that some measure of change aimed at improvement is needed. Improvement implies not just one evaluation but two: of what we have been doing before and what 41 Instructional Development we are doing now. Only a comparison of these two evaluations can indicate whether things are better or worse. Change not aimed at improvement is mindless and frivolous. Claims of improvement without evaluations are empty. We have simply got to have evaluation or one cannot hope for improvement. Certainly we cannot operate a systematic approach to development. The above is likely to be widely misunderstood. It is likely to be interpreted as support for a movement, particularly strong in the United States, to develop and count heavily upon specialists in evaluation. But it will be clear from the model set out below that I believe that the delegation of responsibility for evaluation to experts should be very restricted. A small number of specialists in evaluation, preferably developed from experienced academics, will almost certainly be needed as consultants. I shall not attempt here to develop an argument about the nature of duties and the requisite experiences and talents of the occasional specialist in evaluation I would like to see developed. I would here simply ask that the preceding paragraph be read as an uncomplicated and non-technical approach to the logic of improvement and development. The case for dispersal of the major part of the responsibility for evaluation is, however, made below. It is easy to explain why evaluation is a key to sound instructional development and difficult to propose adequate methods for doing the evaluation. Unfortunately, although we are already devoting considerable energies to this problem and we suffer great anxieties lest unfair evaluation systems be introduced, we do not have any very useful conceptual framework within which a sound system is likely to develop. The purpose of this section is to suggest a framework which, while incompletely worked out, seems to raise the right kinds of questions and may be susceptible to useful development. In this account there is quite deliberately no strict line drawn between the evaluation of instructional materials, of teaching, and of teachers. There are, of course, instances in which a superb teacher is defeated by the circumstances in which he works, or a poor teacher can appear to be doing a good job because the system carries him. But it is the success of the total system which matters, and if a teacher is a bad lecturer but his students achieve a great deal because of the thought and work he has put into alternative styles of teaching, then he is a good teacher. In a systematic and holistic approach the evaluations must be equally comprehensive. One of the most important aspects of a model for an evaluating system is the assignment of responsibility for developing and operating the system. One possibility is the current development of specialists in evaluation. However, this trend appears to be with us largely because this is the common approach to perceived needs in our society, rather than because it has a clear rationale. Calling in an established specialist, or developing a new kind to make judgments for one, could be called a delegation of responsibility model. However, this is not the only choice. In matters of communication between scholars we give only modest powers and responsibilities to editors and consider that each scholar must be capable of doing what*In commenting on this paper Mario Creet has said "Comparative evaluation is predicated on the assumption that absolute objectives and criteria do not exist. If they did exist, one evaluation would be enough. Those who proclaim the freedom of the university in the West might get a jolt if they are made to realize that the implicit acceptance of what they do, or the single evaluation of progress, are characteristic of orthodoxy no less stringent than Marxism-Leninism, or the Holy Inquisition." 42 H. M. Good ever is needed in his field, including communicating it. We use here what might be called a disperse responsibility model. One of the theses of this paper is that while there is a movement toward the delegation model for evaluation, the disperse one is more appropriate. This proposal, and the ideas developed from it, are based on a suggestion by Professor L.R.B. Elton of Surrey that evaluation of teaching, including of all instructional methods and materials, might well work best if done by a system closely comparable to that used for research and scholarship. The remainder of this section is devoted to an exploration of this idea. The model of evaluation used in research, if less than perfect, has had a great deal of success if success be defined not in terms of fixed criteria but rather in adaptive terms of providing the corrective feedbacks necessary to keep a process operating with acceptable and improved effectiveness. It can be described as a peer judgment model. The definition of a peer is generally in terms of someone of roughly equivalent background in the field who is devoting a high proportion of his or her energy to the subject in question. However, because one important feature of the model is its openness — with the requirement that evidence for judgments be available for all to see — there.is built into the model the potential for a much wider definition of a peer, i.e., anyone who has sufficient interest to follow through the argument in detail. The peer judgment model of evaluation as used in research has certain preconditions for successful operation: (1) widely shared assumptions and values which are generally understood within the framework of the research in question. (2) a substantial population of willing and highly qualified evaluators. (3) complete openness of the system with evidence freely available to all. Is the research model applicable to teaching? An evaluation system for teaching and learning will be more complex than one for research since the research normally focuses on a specific problem and may be able to concern itself only with outcomes rather than with the way in which outcomes arise. Teaching is an interactive affair in which process and outcome are never independent. Moreover, if we attempt to develop a peer judgment system in evaluating teaching we shall have to use a very broad definition of a "peer." In matters of scholarship the scholar's peers are distributed on at least a national, perhaps on an international or world wide, basis. In teaching, the peers are chiefly intra-institutional though they are not strictly so. Inter-university judgments of teaching are widely made in relation to preparation of students for graduate study. However the peer is probably properly regarded in teaching as coming essentially from the same institution. But, the peer need not be defined in terms of the university rank hierarchy or even in terms of fellow-teachers. In some situations students will properly be regarded as peers and will participate in the evaluation process. In other situations they would not be so regarded. This will, at times, be a contentious issue but the debate about how it should be resolved should be instructional to those concerned. There is nothing in the above which makes the peer judgment model, as used in research, fundamentally inapplicable to the teaching process. Indeed it appears to be a promising model and one with which the university teachers at least are very familiar. Acknowledging that the systems differ in detail and complexity, it seems eminently 43 Instructional Development worthwhile to try to understand the preconditions for success of the research system and the applicability of these preconditions to the evaluation of teaching and instructional programs. Three of the preconditions are discussed below. 1 — "widely shared and understood assumptions and values" This precondition is very largely met in the field of research and also in the field of teaching. However, there are exceptions which will, initially at least, be more marked in teaching. There is evidence from disciplines with several fundamentally different schools of thought that the system will work with only some of the assumptions and values shared. There is, however, little hope that it can work without their being widely understood. For a clear understanding of assumptions and values it is essential that these be made explicit and that staff members feel a responsibility both to make their own objectives and values clear and to understand those of their colleagues. This precondition is not adequately met now. In both the re-examination of existing programs and the design of new ones, much academic debate is little more than a statement of viewpoints. This applies to all levels: course, department, faculty and senate. Yet somewhere, if not indeed in several places, it should be possible to provide encouragement (or even gentle coercion) for the debators to apply to their academic discussions the same level of critical thought that they would insist on in a research paper. Two specific approaches to achieving the objective can be suggested. One involves the insistence by academic administrators that programs be argued critically in, for example, curriculum committees. The other involves encouragement of those who have already indicated an interest in examination of the assumptions and values of their discipline which receive little attention by their colleagues. It is now easier to find discouragement. There must also be a well publicized change in the rewards system of the university. All of these steps could be achieved in large measure by senior administrative personnel asking the right questions or, when affairs are complicated, they might assign consultants to committees to see that the right questions are posed and debated critically. Assignment of such consultants would provide an exercise for those inclined to this type of analysis of their subjects and a stimulus to their continued study in this difficult but essential area. 2 — "A substantial population of willing and highly qualified evaluators" A major factor in the success of the research evaluation system is that virtually every academic is willing - indeed eager - to play a part. Evaluation of other persons' research is the very essence of the intellectual game and to be asked to play an important role in the evaluation hierarchy as a member of a committee on research, as a journal reviewer or thesis examiner, is a much sought after distinction closely related to the rewards system. Even for those who do not aspire to promotion within this evaluation hierarchy there is a feeling, based on years of indoctrination and training, that willingness to play this role is part of one's duty to the discipline. The situation is very different for teaching. There has been (though it is much less marked now) the feeling that this is not a primary responsibility, that to attempt to eva- 44 H. M. Good luate someone else's teaching is an impertinence, that there are no useful or valid criteria, and that the whole idea is better forgotten. In contrast to research evaluation there is a negative indoctrination, a lack of training, and a lack of incentive or even a disincentive. There is also a fairly conspicious lack of expertise. The establishment of criteria for good teaching, and the development of methods of evaluation are not quite as unexplored fields as most university staff believe. It can, therefore, be argued that there are two parts to this precondition: willingness and competence. Both are deficient but it should be made clear that any lack of competence is based simply on the fact that the total effort put into study of how to evaluate teaching has been to date quite trifling in comparison with the effort which has gone into the study of how to evaluate research. This two-part precondition is emphatically not met. Yet it is a, if not the, central, issue. What action could be taken to increase the extent to which it is met? In trying to answer this question it is worth noting that the research evaluation model does not involve any "experts." It depends upon widespread, essentially universal, participation. University teachers have, probably correctly, been suspicious of the expert in evaluation or in education generally. But they have been unwilling to accept that avoidance of "experts" in a difficult field requires a considerable level of expertness and effort on the part of most practitioners. 3 — "Openness of the system" Traditionally, university teaching has been far from open and it remains essentially a private operation between a staff member and his or her students. This traditional pattern is, however, changing fairly rapidly and, in those cases where a course is produced by a team working together to produce the whole course (utterly different from a series of course fragments given seriatim and independently by several professors), there has to be openness about exactly what each proposes to do, why he has chosen that way, its implications for all other parts of the course, criteria for success, etc. To achieve openness requires a major effort in which there is pressure toward fulfilling all the preconditions here discussed. While very time consuming the team approach to course design is a major change in the style of university teaching and one which can contribute a great deal to meeting the preconditions for successful evaluation. If one is going to apply the research model with regard to openness to teaching, it is worth noting that the stress on openness in research is greatest in the evaluation of the plan and in the judgment of final success. There is little if any attempt to sample the way a scholar is working by looking over his shoulder in the laboratory or library to see whether he is doing a good job in the detail of day to day work. In some ways, but not all, such minute supervision (which would be regarded as both unnecessary and improper) is a parallel to the sampling of a teacher's performance in one or two lectures. Openness of lectures is probably desirable but it gets at only superficial technical aspects of the teaching process. Therefore two important approaches used extensively in evaluation of research which could be more used in teaching are: 1. Examination and evaluation of the plan for courses including objectives, plan of attack, and plans for evaluation. 45 Instructional Development 2. Student examination papers reviewed in the light of stated objectives. Evaluation of teaching is also more complex than is evaluation of research in that the objectives of teaching are more varied. They must cover discipline, institutional and student goals. Evaluation to provide a feedback for improvement in a systematic process cannot afford to ignore any of these though they may be weighted differently in different institutions. Teaching — The Choice of Method: Instruction depends upon the two distinguishable, if often overlapping, processes of presentation of material and of interaction. Presentation is one directional. Interaction is two directional. The totality of the one directional system for presenting material is herein referred to as the textual system regardless of the nature of the method of delivery. These two parts of the style the textual and interactive components will now be discussed. The variety of ways in which material can be presented is large — books, printed handouts, tapes, slides, T.V., films, computers, lectures. This variety has been increasing steadily as a result of technological advances. It brings with it increasing flexibility with regard to time, and place, and number — all problems which have been critical to universities in recent years. It is not, therefore, surprising that the system for presenting material, which absorbs an enormous part of universities' total resources, has been a particularly fertile ground for experimental teaching projects, and that technology-flexibility-cost issues, as they relate to textual materials, have provided the principle incentives for experimentation. Indeed this relationship is the basis for the popular misconception that instructional development is only the use of technologically novel materials. Many of the experimental projects supported by universities over the past decade have been directly concerned with the lecture which has been included above in the list of undirectional or textual materials but must, of course, also be considered potentially at least as a part of the interactive system. This dual nature of the lecture is a major reason for the arguments about its virtues and shortcomings. Admittedly, there are difficulties in separating the textual and interactive aspects of a lecture since in some circumstances they are closely interwoven and interact in subtle but important ways. However, if one is to choose textual materials according to their advantages for the specific job to be done the lecture must be looked at in relation to presentation of material and again in relation to interaction. Not only do different forms of texts have very different advantages and disadvantages, they have also widely different costs. The book, for example, is compact, tightly organized, easily indexed, accurate, portable, durable, and cheap if published in sufficient numbers. It is, however, lacking in pacing and emphasis, may be too concentrated, is limited in the range of visual presentations possible, and has no personality impact. The lecture (and let us suppose for the sake of argument that we are considering one which is purely textual) has pacing, emphasis and personality impact but is often loosely organized, inaccurate as remembered or recorded, ephemeral, inflexible as to place and time, difficult to index, and expensive-to-outrageous in cost per unit of instruction. However, even if summary statements like this can be accepted, the choice of the medium in which specific textual materials can best be provided is a complex one and needs 46 H. M. Good to be looked at very critically; the more so since different forms are appropriate for different materials and also for different students. There is, at present, a trend away from lecturing and there are many experimental programs, and some established ones, using the "new media" of film, television, tapeslide, etc. to replace lectures. This is a sector of instructional development which is very active and there have been both successes and failures. It is a legitimate area, although in many cases programs have been based on a superficial analysis of the range of possibilities and relative advantages of each method and have been aimed at a narrow and incomplete set of objectives. Quite predictably many were found wanting when looked at comprehensively and have been discarded; the baby being routinely thrown out with the bath water. The long term savings and the increased successes of the iterative developmental approach would, over the past ten years, have been enormous. Because different kinds of textual materials have different advantages, because each instructional program has a complex set of objectives, because classes are heterogeneous, it follows that most programs can profitably use a mix of materials and/or offer students alternative modes of access to the textual materials. The mix will be based on the merits of methods for specific jobs and/or cost constraints. Trade-offs aimed at optimizing the effectiveness of the total package will, therefore, be necessary. The choice of an appropriate mode of presentation — book, lecture, etc. — is not, however, the end of the textual issue, which is a vital part of the overall methods issue. How does one get the best materials in whatever modes are chosen and how does one get them published? Implicit, but rarely expressed, in efforts to improve the textual package is a criticism of the current system. To some extent this is based on scholars writing for colleagues instead of for students and to some extent to an almost complete dependence on commercial publishing which understandably is constrained by the need to turn a profit. But if the text-producing industry is doing a less than ideal job and if this is reducing the effectiveness of the educational process significantly then we need to consider whether we might not gain greatly from an integrated instructional program with texts of all kinds eligible for subsidy from general educational funds. The only case in which this has been done systematically is Britain's Open University — an institution which is having an enormous impact in areas which may appear quite separate from its role as an "open" (i.e., no prerequisites) institution. The Open University was designed to serve students at a distance whose principal personal contact with staff would be with tutors rather than with the course designers and the writers of textual materials. The need to separate out clearly the textual and interactive phases of teaching was, therefore, especially marked. In addition, the question of how to choose the best blend of textual materials, how to get the best possible quality produced, and how to publish this material all had to be faced. The approach used by the Open University involved: (a) adoption of the principles of systematic course development. (b) creation of a course team for each project, the team to include subject experts, educational technologists, editors, media experts, etc. (c) use of a mix of modes of presentation — books, T.V., lectures, tapes, etc. — chosen after careful examination of costs and advantages. 47 Instructional Development (d) publication of much of the material by the Open University itself. (It has become the largest educational publisher in Britain.) Commercially available materials are, of course, used when suitable. Thus, the Open University, in its effort to think the educational process through systematically, found itself deviating very greatly from traditional patterns in the important areas of what textual materials to use, how to get the best possible ones prepared, and how to publish them. The high quality of their textual materials has been generally recognized. They have in fact made the case for the application of the systematic method to the textual component of university teaching. But there has, as yet, been little recognition of the wide range of implications of their success. The range of approaches to interactive instruction is much more limited than that for presentation of material. Interactive instruction implies an essentially one-to-one responsive situation though this can be achieved in a group if it be small enough to provide a series of one-to-one encounters. The interaction is perhaps greatest between teacher and student but the student-student relations can be extremely productive. There is considerable literature on small group and interview techniques and it is clear that the value of time spent in interactive activities can be increased greatly by systematic planning of what specific components of the total instructional program are best done in this way and how to get the best results with given resources, especially of time. Instructional development can therefore contribute to both the effectiveness and efficiency of the interactive component. Equally it can contribute to the organization of the over-all program so that this vital component of instruction is not short-changed as it sometimes is by invoking unnecessary constraints. Present resources could quite easily provide for more interactive instruction if this were considered a high priority. In an earlier section on the advantages of Instructional Development, a chemistry course revision was cited in which a redevelopment of the textual materials could permit the redeployment of some half dozen senior staff into interactive instead of presentational activities. This example merits careful study. It focusses on the possibility that the small-to-medium sized lecture section is rationalized on the basis of its having an interactive component which is lacking in the more costeffective large lecture system. But the rationale only holds if the interactive teaching delivered this way is more effective than it would be if the textual and Interactive teaching were separate clearly as in the Open University, or the cited Chemistry Program. This comparison is rarely attempted. There is at the moment a great deal of talk about the need to do more re-deployment of this type. But so long as there is no willingness to look at possible savings in the effort put into purely presentational or textual instruction this will be little more than empty rhetoric. University teachers are busy now. They cannot simply be loaded with more work. The answer can only come from a reappraisal of where the time resource can do most good. IV. Conclusions and Recommendations It would be unfair to suggest that Canadian universities never go through careful and systematic development of programs. But only from a position of profound ignorance can it be argued that a substantial proportion of day to day decisions regarding their 48 H. M. Good academic management is made according to any tightly reasoned plan. Because vested interests and politics are so prominent, decisions have their roots in history and in chance concatenation of forces. But chance will not do. We need choice — and careful choice at that. Careful choice, after review of alternatives, is the essence of management. Universities today demand the right to be self determining. This right carries with it the responsibility to be both honest and precise about what the institution is trying to do and how it is going about it — a credible match between rhetoric and practice. Much harm is now being done to the universities because this match seems less than credible to both students and public. The mismatch arises from a failure to use a logical and systematic approach in the matters of staffing, rewards system, and instructional programs. The failures of the rewards system and the instructional program will be discussed briefly in relation to remedial action. The staffing problem was dealt with at some length in Section II and will be discussed only as it inter-relates with the rewards system. The rewards system provides the most direct, continuous and powerful influence shaping the attitudes and priorities of staff. These attitudes are, in sum, essentially those of the university. A rewards system without critical evaluation is a farce, and rewards not related to institutional objectives are dishonest. There is no logical way of sidestepping the evaluation issue — difficult as it may be. It is an essential part of the systematic approach to instruction at the lesson, unit, or course level. It is an essential part of institutional administration through the rewards system. There cannot be institutional accountability without critical attention to evaluation and rewards, and evaluation implies consideration of effectiveness of the use of teachers' efforts. It is not unreasonable to claim that this aspect of institutional accountability deserves as much attention as does financial accountability. In this connection it is interesting to analyze university management systems in terms of accountability for the dollar and accountability for the hour. Even Deans and Vice Principals are concerned primarily with dollar accountability and the management of, and accountability for, the academic process is largely ignored. Nor is this situation likely to change until the objectives-rewards systems are matched in a highly visible way. As evaluation is improved and applied to the rewards system there must be consistent and courageous use of the results. However, the development process cannot be rushed. Fear of inappropriate evaluation and a deep mistrust of existing systems is currently a more important bar to teachers undertaking systematic course development than is disinterest or ignorance of textual resources, or lack of understanding of interactions of the components of the system. The questions of the specific form of rewards or penalties (salary adjustments, rank, leave, teaching loads) which now concern many in the universities are details which have to be faced, but they can be left — or handled as at present in an ad hoc way — until the evaluation problems are, if not solved, at least incorporated into the system in a way likely to evolve a more satisfactory working basis. This working basis clearly has to involve wide participation by the university community if the disperse responsibility model implicit in the democratic management of the university is to be honored. 49 Instructional Development What is needed is an approach which is pervasive at all levels, i.e., encouragement of analytical questions at departmental level, a better staff evaluation process perhaps modelled on that used in research, a system of critical review of departmental briefs at Faculty level with firm rejection when the rationale is sloppy, and so on throughout the university. This "all levels" approach will not work without an initial cadre of people who have an interest in the work and a willingness to devote a part of their study time to the process of systematic development. It seems probable, therefore, that in spite of the arguments for a disperse responsibility (with its implication for a moderate level of disperse competence and interest) there will have to be a seeding operation in which some university staff are in effect seconded to the duties suggested. This will require resources but is not a revolutionary proposal. If a modern university found itself without accountants it would not hesitate to hire them and if none were available it would send staff out for a crash program to qualify them for the job. No such action is visible with regard to the accounting for the deployment of educational resources. In summary, the recommended actions are: 1. Development of a cadre of part time "consultants" to work in departmental or faculty contexts as critics of programs and proposals. It is essential that these be part-time persons who continue to be directly involved in teaching and desirable that they be established and respected members of staff. This proposal could so easily lead to the development of a new brand of academic administration based on "specialists" that care should be taken at the outset to design a program which fosters the disperse responsibility model. 2. Modification of departmental and faculty administration to focus more sharply on instructional issues and strategies, probably through a revised evaluation and rewards system. Both of these steps require strong leadership at the higher levels of the university but they do not attempt to work simply through upper administrative channels. All levels are involved as they must be if the universities are to justify the wide use of the democratic decision making which they espouse so warmly. The importance of the evaluation process has been stressed above almost to the exclusion of the instructional process. It is true that if the systematic approach is to be followed fully then consideration of any problem leads into consideration of the whole system. Thus it can be argued that if staffing has been carried out with careful regard to institutional objectives and a good evaluation and rewards system operates the instructional process will be looked after. However, from a practical viewpoint, the reverse route is more likely to be profitable. The support of individual projects to develop instructional materials carefully is probably the most effective way of interesting staff in a more critical approach to teaching. An increase in interest in academic management as a challenging problem should create an interest in the research model for evaluation, and in a clearer enunciation of objectives. This might influence staff towards a distribution in the responsibility triangle more justifiable in terms of institutional objectives. 50 H. M. Good Instructional development in the narrow sense of production of specific material is therefore a useful first step in the overall process - perhaps the most useful one. But it is important to reiterate that haphazard experimentation and random data collection are no more effective in development of instruction than they are in the acquisition of knowledge. Most funds now allocated to "innovation" are wasted. They are given in a pious gesture by university officials who lack either an adequate measure of the problen or the courage to act on the measures which they do have. To paraphrase Robert Diamond, Assistant Vice Channcellor of Syracuse: the widespread encouragement of small projects (as opposed to the careful selection of a few major ones) is an excellent method of advertising administrative commitment to instruc tional improvement but many small projects will be of low priority and the results are usually unsatisfactory. As argued earlier in relation to the Open University, and to the more successful projects in all jurisdictions, a thorough systematic approach works well, but half-way measures are costly and likely to be unproductive. The general conclusion is that there are huge gaps in the academic management system and that simple accountability for use of the major university resources — staff and student time — requires more critical accounting than the university now gives or is capable of giving on short notice. What is needed is strong leadership toward developments which will progressively improve the university's ability to account for its use of these resources. In providing leadership there must be recognition that instructional development is a continuing iterative process. As such it must have institutional support to assure it of continuity. Without continuity it remains largely haphazard innovation — possibly successful but probably wasteful. In no area is the old adage more applicable "If a thing is worth doing it is worth doing well." 51 Instructional Development Some Related Readings Beard, Ruth M., Teaching and Learning in Higher Education, Penguin, Second Ed. 1972. Beard, Ruth & Bligh, DonaldResearch into Teaching Methods in Higher Education, Society for Research into Higher Education, London, Third Ed. 1971. Beard, Ruth, et al., Objectives in Higher Education, Society for Research into Higher Education, London, Second Edition, 1974. Becher, R. A., et al., Innovation in Higher Education, Society for Research into Higher Education, London, 1972. Diamond, R. M., et 'A., Instructional Development in Higher Education, Syracuse University, 1973. Embling, Jack, A Fresh Look at Higher Education - European Implications of the Carnegie Commission Reports, Elsevier, Amsterdam, 1974. Good, H. M. & Trotter, Bernard, "Accountability for Effective and Efficient University Teaching," Stoa, Vol. 4, pp. 43-53,1974. Falk, Barbara & Kwong Lee Dow, Assessment of University Teaching, Society for Research into Higher Education, London, 1971. Light, D. W. et al., The Impact of the Academic Revolution on Faculty Careers, American Association for Higher Education, 1972. Nuffield Foundation (London) Group for Research and Innovation in Higher Education, Newsletter, Becher et al, Eds. Trotter, Bernard, Television and Technology in University Teaching, Council of Ontario Universities, 1970. Society for Research into Higher Education (London) Research into Higher Education Abstracts. Wood, Robert, "Academe Sings the Blues," Daedalus, Vol. II (Winter 1975), pp. 45-55. Yarmolinsky, A., "Institutional Paralysis," Daedalus, Vol. II (Winter 1975), pp. 61-67. Zeckhauser, Sally and Richard, "Encouraging Improved Performance in Higher Education," Daedalus, Vol. II (Winter 1975), pp. 97-107.