The Canadian Journal of Higher Education, Vol. XIII-3, 1983 La revue canadienne d'enseignement supérieur, Vol. XIII-3, 1 9 8 3 Systematic Reviews of University Programs and Units WALTER E. HARRIS and EDWARD A. HOLDAWAY* ABSTRACT Because universities in Canada, the U.S.A., and other Western countriesarefacing declining real income and increasing demands for greater accountability, they need to use resources more carefully and undertake planning activities more seriously. By obtaining information from reviews of academic and support units, universities are in a better position to meet these challenges. Most experts agree that reviews aimed at improvement of programs and activities are more successful when conducted under the authority of the university rather than an external agency. Attention needs to be paid to the establishment of an appropriate administrative mechanism that can maximize benefits relative to costs. This article describes the policies and procedures adopted in 1980 by The University of Alberta to allow for a continuous system of reviews of all academic and support units. Establishment of a President's Advisory Committee with a minimum of formal procedures has led to a successful system under which 51 units are now in various stages of the review process. The procedures used at The University of Alberta may be worthy of scrutiny by other universities contemplating such activity. RÉSUMÉ Au Canada, aux Etats-Unis et en d'autres pays occidentaux, les universités doivent faire face aux problèmes de la diminution de leurs revenus réels, des exigences en matière de comptes rendus plus satisfaisants de l'utilisation de leurs fonds et de la nécessité de faire de plus grands efforts de planification. Pour être plus en mesure de relever ces défis, les universités peuvent profiter des informations recueillies au cours des examens de l'état actuel de leurs unités académiques et * Walter Harris is Professor Emeritus of Chemistry and Chairman of the President's Advisory Committee on Campus Reviews at The University of Alberta. Edward Holdaway is Professor of Educational Administration and Director of Institutional Research and Planning at The University of Alberta. 56 Walter E. Harris and Edward A. Holdaway de leurs services de soutien. La plupart des experts reconnaissant que de tels examens périodiques visant à l'amélioration des programmes et des activités connaissent un plus grand succès quand c 'est l'université elle-même qui les effectue plutôt qu 'un organisme externe quelconque. Il faut avoir grand soin de mettre au point un mécanisme administratif propre à apporter le maximum d'avantages par rapport aux coûts. Le présent article décrit la politique et les procédures adoptées en 1980 par l'Université de l'Alberta dans le but de faciliter un système permanent d'examen de l'état actuel de toutes ses unités académiques et de tous ses services de soutien. La mise sur pied d'un Comité consultatif du Recteur ayant le moins possible de procédures formelles a donné naissance à un système d'examen qui produit de bons résultats et grâce auquel 51 unités sont en voie d'être soumis d l'examen ou l'ont déjà été. Toute autre université qui envisage une telle entreprise aurait intérêt d étudier soigneusement le système en vigueur à l'Université de l'Alberta. SYSTEMATIC REVIEWS OF UNIVERSITY PROGRAMS AND UNITS Universities and colleges throughout North America have become increasingly involved during the 1970s and 1980s in reviews of their activities. This trend has resulted from (1) the need for more and better information on which to base budget and planning decisions, ( 2 ) the need to demonstrate to governments and the public that the institutions take accountability seriously, (3) the recognition that reviewing should be an integral part of the operation of an organization, and (4) the recognition that reviews can have important positive consequences. Although some universities and colleges outside North America, notably The University of London and The Australian National University, have carried out program reviews, most of the activity has occurred in the United States and Canada. The pertinent literature mainly reports procedures used in the United States, while those in Canada have received little formal academic attention. Further, most universities still tend to conduct reviews on an ad hoc basis in response to specific demands rather than on a systematic, on-going basis organized by a standing committee or administrative group. This article reviews selected literature on the topic of academic reviews and describes the review system at The University of Alberta. Beginning in 1980 all academic and support units are required to be reviewed under procedures organized by a President's Advisory Committee on Campus Reviews. The Alberta approach, which was developed after lengthy deliberation, is seen by us to be a useful model that could be adapted by other Canadian universities and colleges. The awareness and possible adaptation of viable models can help to reduce the time that institutions contemplating reviews spend in developing their procedures. Without appearing to be presumptuous, we think that the following recommendation from Craven (1980) is appropriate: 57 Systematic Reviews of University Programs and Units A n u m b e r of institutions currently have e x e m p l a r y approaches t o academic program evaluation. T h e concepts, guidelines, and operating e x p e r i e n c e that a c c o m p a n y these evaluation approaches should be s h a r e d m o r e w i d e l y w i t h o t h e r i n s t i t u t i o n s , (p. 4 5 2 ) LITERATURE REVIEW A substantial amount of literature on evaluation is available. Although much relates to elementary and secondary education, a growing number of books and articles focus on the postsecondary sector. For example, Craven (1980) has provided an excellent article dealing with the concepts and issues involved in program review as well as different approaches which can be undertaken. However, little has been published on evaluation and reviews in Canadian universities and colleges. One exception is an article by Bélanger and Tremblay (1982): they discussed the role of committees at the Universite' de Montréal in examining both academic and administrative units and making recommendations for reductions in course offerings and teaching personnel in response to diminished provincial funding. Another Canadian experience is reported in The Shrinking Maze which outlines the review procedures used in 1979-80 by The University of Calgary (1980). Definitions Certain terms should be defined early in this discussion. Evaluation is assumed to be an assessment of the extent to which a performance meets preselected criteria. Scriven's distinction between summative evaluation (conclusion-oriented) and formative evaluation (development-oriented) is well known. A review is usually meant to be much more comprehensive than an evaluation, since it encompasses a detailed examination of historical and current aspects as well as of proposals for future development. Program is taken to refer to either a particular academic program, such as the B.A. program, or more commonly to a budgetary unit such as a department or group of departments, as proposed by Gilmour (1980): T o m y w a y o f t h i n k i n g , a c a d e m i c d e p a r t m e n t s u l t i m a t e l y are a b e t t e r u n i t o f a n a l y s i s t h a n d e g r e e p r o g r a m s , b e c a u s e t h e y are t h e a c t i v i t i e s t o w h i c h i n s t i t u t i o n a l r e s o u r c e s are a l l o c a t e d a n d f o r m t h e b a s i c b u i l d i n g b l o c k o f t h e o p e r a t i o n s a n d m a n a g e m e n t s t r u c t u r e at m o s t h i g h e r e d u c a t i o n i n s t i t u t i o n s , ( p . 1) Probably the use of program review by Arns and Poland (1980) would be widely accepted, that is, "a searching comprehensive evaluation of an existing, coherent set of academic activities" (p. 269) involving both a self-study phase and a formal review phase conducted by a team of experts that may include outsiders. Purposes The following list of purposes behind the conducting of program reviews has been compiled from several different sources: 1. To inform departmental and institutional decision makers about a program's 58 Walter E. Harris and Edward A. Holdaway strengths and weaknesses, as the first step in a process designed to improve its quality (Council of Graduate Schools in the United States, 1979, p. 2; Peterson, 1980, p. 154; Barak and Berdahl, 1978, p. 56; Dressel and Cammack, 1972, p. 288). 2. To establish a program's status relative to specified standards or in relation to the performance of other similar programs (Council of Graduate Schools in the United States, 1979, p. 2). 3. To assist the planning process (Peterson, 1980, p. 124). 4. To arrive at judgments about a range of decision alternatives concerning the installation, continuation, modification, or termination of that program (Jany and Keller, 1981, p. 3; Peterson, 1 9 8 0 , p . 154;Barakand Berdahl, 1 9 7 8 , p . 56). 5. To assist an institution when it is forced to reduce expenditures and/or increase efficiency (Barak and Berdahl, 1978, p. 56). 6. To demonstrate accountability and responsiveness (Barak and Berdahl, 1978, p. 56). 7. To protect students (Barak and Berdahl, 1978, p. 56). Several writers have noted that internal and external reviewers usually have different purposes: For example, Jany and Keller compared different emphases by internal and external reviewers: I n s t i t u t i o n a l e f f o r t s in p r o g r a m e v a l u a t i o n h a v e g e n e r a l l y f o c u s e d o n learning or k n o w l e d g e o u t c o m e s while external efforts have emphasized fiscal austerity, m a n a g e m e n t malpractice, c o n s u m e r p r o t e c t i o n , a n d a c c o u n t a b i l i t y t o g o v e r n m e n t a g e n c i e s a n d s o c i e t y , ( p . 5) Their opinion was supported by Folger (1980): T h e m a j o r f a c u l t y m o t i v a t i o n f o r p r o g r a m r e v i e w is t o d o c u m e n t t h e n e e d f o r a d d i t i o n a l r e s o u r c e s , w h i c h is in o p p o s i t i o n t o a s t a t e m o t i v a t i o n t o u s e p r o g r a m r e v i e w as a basis f o r u s i n g e x i s t i n g r e s o u r c e s m o r e e f f e c t i v e l y , (p. 5 4 ) Procedures Craven (1980, pp. 444-449) has provided one of the most comprehensive listings of matters that need to be considered when program evaluations are to be conducted. He classifies these matters under five general headings: 1. Defining issues, participants, purposes and processes, for example, what information is needed and by what processes are judgments made? 2. Designing information analysis and data collection, for example, what are the sources of the data and how will the data be analyzed? 3. Deciding on interpretation and reporting, for example, what are the criteria to be used for evaluation of the program, and who receives the reports? 4. Planning an audit, that is, how are the program evaluations themselves to be evaluated? 59 Systematic Reviews of University Programs and Units 5. Implementing the design of the program evaluation. The effectiveness of program evaluation can be assessed, according to Craven (1980, p. 434), by these criteria: Are the "evaluation results useful in arriving at judgments about decision alternatives for a given purpose"? Is the quality of these decisions better because information from the reviews was considered? Are the selected decision alternatives actually implemented? Dressel and Cammack (1972, pp. 282-283) have paid particular attention to selection of the review committees. Are the members to be elected or appointed or a combination of both? What criteria should be used in their selection? Should peer respect and convictions be considered? They also mention many of the matters raised by Craven, but in addition emphasize wise publicity and clarity of role expectations for members of the review committees. Another important aspect was raised by Dressel and Cammack (1972): "Because self-study [review] usually avoids pinpointing individual weaknesses, it, therefore, is not likely to correct problems created by poor leadership" and that a review is " n o substitute for lack of administrative willingness to deal forthrightly with a well-defined issue" (pp. 277-278). That is, reviews should not be used to solve problems that should be handled by routine administrative procedures. Trends The extent of use of program reviews in U.S. colleges and universities was surveyed by the National Center for Higher Education Management Systems (NCHEMS) in 1980. Barak (1982) noted that the Carnegie Council's 1975 speculation that "institutional administrators would be relying much more heavily on program and course review in the f u t u r e " has been borne out by data obtained in the NCHEMS survey (pp. 34-35). Of 1,082 institutions surveyed, 882 (82 percent) stated that they had some formal program-review process, with 43% of these having adopted their present policies and procedures since 1975. Barak also observed that "large research universities are most apt to conduct internal reviews" and that "universities that rank high in the various peer ratings are likely to show a similarly high usage of internal program reviews" (p. 35). McCorkle and Archibald (1982) concluded that "program review is now more formal, more frequent, and more comprehensive in scope and depth than in the past," that the criteria for judging programs have changed, and that "the procedures and personnel involved have grown in number and complexity" (p. 143). Similarly Miller (1979) stated that evaluation will continue to increase in higher education during the 1980s and cautioned that "if public postsecondary institutions do not undertake rigorous evaluation, likely it will be done for t h e m " (pp. 1-3). The continuing and increasing need for reviews was also emphasized by Arns and Poland (1980); they built their case upon the requirement that universities, which are complex organizations of "considerable inertia," must evaluate to decide how to change in response to societal changes (p. 268). Another important aspect emphasized by Shirley and Volkwein (1978), is that, though the need to set priorities has always existed in universities, it was 6 0 Walter E. Harris and Edward A. Holdaway not recognized until "'steady-state' financing" appeared (p. 473). "Hastily constituted committees" then were faced with the setting of priorities. Arns and Poland (1980) also observed that during the 1960s universities faced few program choices because new programs received the necessary resources and constraints were rare (p. 273). With changed circumstances, the need to incorporate some ongoing evaluation mechanism seems obvious. In the NCHEMS review, Barak (1982) noted that most institutions initiated reviews to improve their programs because knowledge of strengths and weaknesses was necessary before improvements could be planned (p. 39). Redistribution of resources and reduction of programs was the second most commonly cited reason, especially for contracting institutions. Barak also reported that "a number of colleges and universities are actively implementing planning processes that use program review to set priorities among institutional programs" (p. 38). Benefits and Costs What are some of the specific benefits from reviews? Hoyt (1980) suggested the following: they force examination of frequently ignored matters, provide qualitative balance to normal quantitative indices, provide healthy involvement of program participants, point to constructive changes in response to negative findings, and create a positive institutional image (pp. 3-4). Dressel and Cammack (1972) considered that the strength of reviews done by an institution "lies in the involvement of those affected by the subject under study; gradual molding of receptivity for the changes proposed is a usual result of the study" (p. 274). Some academics have found the self-study phase to be at least as beneficial as the formal review and the review-phase activities at least as important as the reports. This relevant assessment has been made by Russo, Brown, and Rothweiler (1977): The act of asking questions of graduates, drop-outs, current students, and faculty a b o u t goals, objectives, educational process, and their relationship to each other causes each to pause to contemplate these m a t t e r s . S p e c i f i c p r o g r a m s t r e n g t h s a n d w e a k n e s s e s are o f t e n i d e n t i f i e d . Curricular r e v i s i o n s h a v e b e e n i n i t i a t e d o r e x p e d i t e d as a r e s u l t of the review process. Student information systems have been f o u n d s e r i o u s l y l a c k i n g a n d are b e i n g m o d i f i e d . T h e f o l l o w - u p o f g r a d u a t e s h a s b e e n e n c o u r a g e d a n d is t a k i n g p l a c e in s o m e d e p a r t m e n t s f o r t h e first t i m e . Q u e s t i o n s and r e c o m m e n d a t i o n s a b o u t a c a d e m i c q u a l i t y and student f l o w have p r o d u c e d positive program revisions, (pp. 297-298) Nevertheless, the value of the formal review phase should not be minimized. Arns and Poland (1980) stated that reviews contribute to the information, understanding, mutual trust, and sense of stability needed to make wise decisions, and that "The most important outcome may well be the strengthening of confidence in decisions, their evolution, and their implementation" (p. 283). Another benefit was identified by Russo et al. (1977), who expressed the opinion that review information allows selective program termination rather than across-the-board 61 Systematic Reviews o f University Programs and Units cuts that may threaten "the qualitative survival of our academic programs" (p. 292). This conformed with their view that " F u t u r e program decisions must be based on an integration of financial and academic data" (p. 292). McCorkle and Archibald (1982) have strongly argued in favor of routine reviews: occasional ad hoc reviews intensify stress, whereas "continued self-study, encouraged by senior administrators. . . can uncover opportunities for improved service in a positive c o n t e x t " (p. 148). Similarly, Craven (1980) urged that "as a matter of good academic practice . . . a self-study of each academic program be conducted every five or six years" (p. 443). As to cost, certain administrative expenses in operating the review process are obvious — for example, salaries, honoraria, travel, subsistence, and telephone. Dressel and Cammack (1972) cautioned that much of the cost of reviews is "hidden or absorbed by unpaid overtime" (p. 274). But other costs and negative consequences may also be involved. Hoyt (1980) identified these to be (1) the poor planning and decision-making based on erroneous evaluations, which can weaken the institution and unjustifiably disrupt employees' lives;(2) the paranoia resulting from adverse effects on morale; and (3) the replacement of intrinsic by extrinsic factors as motivators of behavior. To increase the benefits and reduce the costs, Hoyt advocated these approaches (pp. 4-6): 1. Because the investment in time, effort, and money must be reasonable, the university should concentrate upon major objectives and collect only the most relevant data. 2. The review system must be credible and acceptable to each segment of the institutional community. 3. The validity of the program evaluation must be clear. 4. Effective communication, especially with respect to early preparation of the report and avoidance of premature announcements, is essential. This list is expanded by Dressel and Cammack (1972) who felt that the impact of program review is affected by the state of readiness for the study, the clarity of the charge provided to the committee, the tact and leadership of the committee chairman, the availability and quality of staff assistance, cooperation in data collection, effective use of consultants, and the quality of reporting (p. 288). They also emphasize that continuing communication among all affected groups during the review is critical (p. 289). Concerning appraisal of the effectiveness of reviews, Dressel and Cammack cautioned that this appraisal should not be based upon how many recommendations are adopted, but upon changes made as a result of the overall review activity which may generate ferment leading to adoption of modified recommendations (p. 289). Some Specific Practices The best known North American experience with program reviews is that of The University of Illinois at Urbana-Champaign (1980). Its procedures have 6 2 Walter E. Harris and Edward A. Holdaway influenced many other universities, especially through the involvement of its staff members in consulting and in conducting conferences. The review processes at The University of Vermont and Ohio State University have also been well reported. Arns and Poland (1980) noted that for these two universities "we made it clear that our purpose was program improvement, nothing more and nothing less" (p. 269). The Vermont and Ohio processes incorporated these major aspects: flexibility in collecting information, a self-study report, involvement of all department members, open communication, feedback, use of a coordinating committee composed of senior faculty from outside the department being reviewed, and development of a Memorandum of Understanding. DiBiasio and Ecker (1982) described this Memorandum, which concludes the review, as containing a mutually accepted statement of program objectives, steps to achieve these objectives, the time frame, and estimated costs (p. 15). The University of Saskatchewan took a different approach (Holmlund, 1980). In 1976 it began reviews of colleges (faculties) when the term of a dean had expired. The Board of Governors started the reviews to provide both it and senior administrators with information about relations, accomplishments, adequacy of resources, opportunities, and problems. A Supervisory Committee on College Reviews controlled the reviews, which were organized by the University Studies Group (equivalent to an office of institutional research). At The University of Calgary, which had earlier conducted some ad hoc reviews of faculties, the President in 1979 appointed a University Program Review Committee " t o analyze the current status of the University and to develop an appropriate program for the decade ahead" (University of Calgary, 1980, p. i). This Committee consisted of the President (as Chairman) and eight academics of diverse rank from different faculties. Each faculty was studied in depth in 1979-80 by the Committee members who were each assigned to two faculties. A draft analysis was prepared of organization, role, programs, opportunities and problems. Faculties were then requested to respond to this draft. A different practice occurs at Queen's University where annual reviews of the Faculty of Engineering have been conducted for the last 25 years. Teams of three people — from industry, government and another university — spend two days each year reviewing departments in the Faculty as well as five departments which service Engineering, namely, Biology, Chemistry, Geology, Mathematics and Physics. The reviewing of geology departments was reported by Neale and Armstrong (1980) to be a common practice in Canada, but only Queen's, Memorial and Toronto used this practice on a yearly or biennial basis. Neale and Armstrong criticized the ad hoc nature of these reviews and lack of follow-up visits, and drew these conclusions from comments made at Queen's, Memorial and Toronto: A d m i n i s t r a t o r s at t h e s e u n i v e r s i t i e s s t a t e t h a t t h e r e g u l a r v i s i t s o f such c o m m i t t e e s and their frank unvarnished appraisals give t h e m c o n f i d e n c e in the o p e r a t i o n o f their g e o l o g y d e p a r t m e n t s . Departm e n t heads state that these c o m m i t t e e s introduce fresh n e w ideas i n t o t h e i r p r o g r a m s , a c t as s a f e t y v a l v e s w h i c h d e f u s e p o t e n t i a l l y e x p l o s i v e situations, and generally carry a great deal of w e i g h t in 63 Systematic Reviews of University Programs and Units their c o n s t r u c t i v e r e c o m m e n d a t i o n s to senior a d m i n i s t r a t i o n (p. 2 4 ) . In some cases more than one campus is involved. For example, in 1975 the Board of Regents of The University of Wisconsin System delegated responsibility to its President for "formal ongoing program planning and review processes on its thirteen campuses" (Jany and Keller, 1982, p. 7). Other variations are also noted. Some universities use both internal and external reviewers on the same team, whereas others, for example, The University of Minnesota (Zimmerman, 1981), use separate internal and external review committees. At Minnesota, where graduate reviews started in 1972, the internal committee produces a set of questions that the external committee can ask so as to make more efficient use of its time. Based upon experiences gained at these universities and elsewhere, several writers have identified aspects needed for successful implementation of program evaluation. Munitz and Wright (1980) named selective rather than comprehensive evaluation of programs, trust and genuine participation, a clear definition of evaluation objectives, and an overall framework of institutional planning priorities within which administrators can interpret the review outcomes (pp. 41-42). Opinions differ as to the extent to which the self-study report should concentrate upon data and details. Arns and Poland (1980) recommended that interpretation of the data should be emphasized rather than the data themselves (p. 279). Most experts, however, agree that both objective data and qualitative assessment are required (for example, Miller, 1979; Gentile, 1980). Based upon the 1980 NCHEMS study cited, Barak (1982), pp. 36-47, extracted these generalizations: 1. A few institutions include reviews of research programs and support departments. 2. Reviews are becoming more rigorous, a trend leading to improved results. 3. More data are being incorporated into the review process. 4. Reviews are becoming more important in institutional decision-making and planning. 5. Outside peer faculty are usually involved in the review teams, spending one to five days on campus. 6. Deans of faculties usually organize the reviews. 7. Students, graduates, and former students usually serve as participants in surveys rather than as members of organizing or reviewing committees. 8. The support of the president and trustees for the review process is essential. 9. Programs are selected for review at convenient times (such as on retirement of a dean) or when their turn comes up in a cycle. Some institutions favor combining reviews with accreditation visits, whereas others discourage or forbid this practice. 10. The concept of "quality" and its assessment in reviews leads to considerable disagreement. 64 Walter E. Harris and Edward A. Holdaway 11. Most reviews are perceived to be beneficial. In contrast to the well-established procedures used for academic program review at many universities, reviews of administrative/support units are in their infancy. The University of California provides a notable exception to this conclusion (Griffin and Burks, 1976). At the Twin Cities Campus of The University of Minnesota, a Task Force for Research and Planning Information, set up by the Vice-President for Student Affairs, recommended adoption of the University of California's structured review team procedure for support units which would have these key elements: 1. Administration — assessment steering committee of four to six campus officers; 2. Review teams — three to seven persons having a mixture of managerial skill, technical expertise, and campus knowledge; 3. Staff — 0.5 to 1.5 FTE would administer the review and organize data collection; 4. Scope — a rolling schedule of reviews of functional areas or departments; 5. Procedures — reviewing data and conducting interviews was expected to take 12-15 weeks; 6. Report — clear recommendations would be specified; and 7. Actions — line managers would react and develop implementation plans. This recommended procedure has not yet been implemented mainly because of "a severe budget crisis" (Matross and Delmont, 1982, pp. 17-21). However, The University of Minnesota (Twin Cities Campus) did review four other support units in the early 1980s using self-assessment and internal and external committee reports based on procedures developed by its Graduate School (Pflaum, 1982). Further evidence of interest in reviews of support/administrative units is provided by the 1981 decision of The University of Illinois at Urbana-Champaign to undertake comprehensive reviews of these units. Each Vice-Chancellor is to design an evaluation approach appropriate to his own units, as the procedure used for academic reviews was deemed to be unsuitable for the support/administrative area. Four types of reviews — desk, focused, strategic and comprehensive — can be used (McManus, Todd and Wilson, 1982). Classification of Models for Program Review A scheme for classifying program-review models has been proposed by Holdaway (1980). These range from the participating governance model, in which the senior elected decision-making body on campus selects the program-review committee, to the state model, in which the state/provincial education department does the selection. In between lie the executive model and the governing-board model, in which the president or board selects the committees and oversees the process. Universities must decide how the organizing committee for reviews is to be constituted and how this committee will relate to the committees actually conducting the reviews. One extreme position is exemplified by The University of Colorado at Boulder where in 1977 the fifteen members of the University Review Committee separated themselves into five teams to review all campus units in one year. 65 Systematic Reviews of University Programs and Units At the other extreme, (the more usual approach) the organizing and review committees are clearly separated and have no common members. PROCEDURES AT THE UNIVERSITY OF ALBERTA Background The University of Alberta was founded in 1908 in Edmonton. It operated a campus in Calgary from 1951 until 1966, when The University of Calgary became an autonomous institution. Current enrolment at The University of Alberta is about 20,500 full-time undergraduate students enrolled in 16 faculties, 2,500 full-time graduate students, and 4,000 part-time students. Figure 1 shows the relationships among its senior administrators, the Board of Governors, the General Faculties Council (GFC), and senior GFC planning committees. The GFC of 136 members consists of ex-officio administrators, other staff members, and students. Under The Universities Act, GFC has overall responsibility for most academic matters, although the Board of Governors has ultimate authority. In 1977 The University of Alberta, being concerned about the need to ensure that its allocation of resources to campus units was appropriate, established a University Priorities Committee (UPC). The UPC concluded that planning during the rapid growth of the 1960s and 1970s concentrated on incorporating new programs and that "We had neither a system of priorities nor adequate data on which priorities could be established." One UPC recommendation was that "The Board of Governors and General Faculties Council jointly approve in principle a system of reviews of academic programs, academic units, and administrative and service units." After extensive discussion, the university endorsed two initiatives: (1) the Office of Institutional Research and Planning was to develop an improved data base to assist decision making, and (2) a system of reviews of academic programs and service units was to be implemented. A Joint Committee of the Academic Development Committee and Deans' Council was established to develop terms of reference for the system of reviews. This Joint Committee, which consisted of the Vice-President (Academic) as chairman, two deans, and four professors, met for several long sessions during which the practices of other universities were evaluated, especially those of The University of Illinois. It recommended procedures that would be as simple as possible and would not interfere with the university's established decision-making structure. Its final recommendations, which were approved with only one change by GFC and the Board of Governors, were these: 1. The goal of the reviews should be production of information that serves as a reliable basis for planning and improvement of programs. 2. The administrative structure should be readily understood, effective, efficient, responsive, flexible, credible, and realistic. 3. A President's Advisory Committee on Academic Reviews (PACAR) should be established, rather than a committee of General Faculties Council. 4. PACAR should consist of a chairman, the Vice-President (Academic), and 66 Walter E. Harris and Edward A. Holdaway Figure 1. Relationships among senior administrators and committees at The University of Alberta. two other members selected by the President because of their experience, judgment, and reputation. 5. The primary tasks of PACAR should be as follows: a. to advise the President on the order in which units should be reviewed and on the composition of reviewing teams; b. to advise reviewing teams on the requirements of a particular review; c. to transmit reports of reviews to the President; and d. to keep the university community informed about its work. 6. Individual Unit Review Committees should be appointed, consisting of one member from outside the University but of the same discipline, one from a related discipline within the University, and one from the University at large. (The number of "outsiders" was later changed to two by the General Faculties Council and the Board of Governors.) 7. Each unit to be reviewed should prepare a self-study report as a starting point for the review process. 8. The reviews should be conducted over a period of three to five years, with the order of selection of units being determined by various considerations such as change of dean or chairman. 9. Each Unit Review Committee should be expected to consult actively with undergraduate and graduate students, academic and non-academic staff, and recent graduates and employers of graduates where appropriate. 67 Systematic Reviews of University Programs and Units 10. Various sets of recommendations should be sent to different administrators and committees on campus for action or discussion. 11. Responsibility for monitoring action taken with respect to recommendations of the Unit Review Committees should rest with the Vice-President (Academic). In response to these approved recommendations, the President in 1980 appointed a PACAR consisting of (a) full-time Chairman — a former Chairman of the Department of Chemistry who had recently retired, (b) the current Dean of the Faculty of Graduate Studies and Research, (c) a former Dean of Engineering, who was still on staff, and (d) the Vice-President (Academic). A deliberate decision was made to select a chairman who was not a member of the university's central administration. The Director of Institutional Research and Planning also attended the planning meetings of PACAR. Concurrent discussions were held by the President and Vice-Presidents concerning how the administrative and service units should be reviewed. Three possibilities were discussed: 1. A separate review committee could be established. 2. A subcommittee of PACAR could be established to which appropriate people could be seconded temporarily. 3. PACAR could assume this extra responsibility. The third possibility was adopted. The Director of Technical Services was added to PACAR, and the name modified to the President's Advisory Committee on Campus Reviews (PACCR). The benefits of this unified approach of having all reviews coordinated by PACCR were that (1) a global view could be obtained of all interunit relations and (2) administrative costs would be minimized. However, this added responsibility increased the workload of PACCR members. During 1981-82 the Deans of Engineering and of Graduate Studies and Research were succeeded by two other professors. Now that the review procedures have been well established, the Vice-President (Academic) attends meetings only when his presence is requested. Operating Principles Several important principles were endorsed by PACCR and the President and publicized during the early phases of organization. These principles were based in part upon opinions expressed in the literature cited above and the experiences of other universities. 1. The University's attitude in conducting reviews was to be one of "forwardlooking change and the encouragement of high-quality scholarship" (University of Alberta, 1980b), that is, the reviews were to be positive exercises and not witch-hunts. 2. Multiple benefits were to be expected, such as improved planning and awareness of operations. 3. Considerable flexibility in the different phases was to be emphasized, with a minimum of directions being provided for the self-study and review committees. 68 Walter E. Harris and Edward A. Holdaway action Figure 2. Simplified representation of t h e overall review process at The University of Alberta. 4. While the procedures were being developed, a limited number of units were to be reviewed. 5. The PACCR and accreditation reviews should coincide whenever possible to avoid duplication. 6. Sensationalism was to be avoided and confidentiality respected, especially as to comments submitted to the members of the Unit Review Committee and the responses of a unit and others to preliminary and final reports. The importance of the use of numerical data in the review process should be recognized but not overemphasized. For a self-study report to be useful, it must contain data relevant to such aspects as staff workload, student enrolments and registrations, budget, research, publications, and graduate activity. Such data need to be coupled with qualitative descriptions of activities before evaluation can take place. Consequently, a balance of data, descriptive information, and evaluation is necessary if reviews are to be fully effective. A requirement to provide detailed information in response to specific questions means that there is likely to be less time for serious reflection. The Overall Process Many aspects of the overall review process at The University of Alberta are reported in the terms of reference listed earlier. The specific approach taken is illustrated in Figure 2. When the self-study report is complete, it is sent to PACCR and the President. A recommendation for members of a Unit Review Committee is sent to the President by PACCR after consultation with the unit. Copies of the 69 Systematic Reviews of University Programs and Units report are sent to members of the Unit Review Committee. Typically, about a m o n t h later the committee meets on campus for an on-site visit of two, three, or four days. A series of interviews is arranged, after which the members write a report, which is expected within one to two months after the campus visit. This review report is sent to PACCR, and with the clearance of the President, the unit is invited to submit a response. The final official step occurs when the self-study report, the Unit Review Committee report, and the unit's response are discussed by the Planning and Priorities Committee. Discussions between the President and the head of the unit can occur at any phase of the review process. Action can be taken by the President, usually in consultation with the VicePresidents and other appropriate administrators, at any stage of the process. That is, the President need not wait until the Planning and Priorities Committee has discussed all the reports, but can take appropriate action at any time after receiving the self-study report. The unit can make changes and take action before completion of the self-study phase, and possibly long after formal completion. PACCR provides advice sheets to units to be reviewed concerning the selfstudy and its report, the Unit Review Committee, and the on-site review. It also transmits standardized information provided by the Office of Institutional Research and Planning showing academic and other data for the previous ten years and budget data for five years. Advice sheets are also provided to students concerning how they can be effective in providing input to a Unit Review Committee. Further, invitations to provide confidential written input are printed in Folio, a weekly publication of The University of Alberta. The self-study phase. The self-study report of the unit should include these aspects: history, current status, impending changes, future prospects, strengths, weaknesses, and limitations. Because program improvement is the preeminent objective, global rather than specific instructions are forwarded from PACCR. Examples of self-study reports from other departments are provided in confidence. To obtain the maximum benefit, the unit is asked to be open both to itself and the review team. The self-study phase has usually taken about six months. It involves a considerable amount of staff time, especially from the chairman. (This has been a common experience at other universities, as noted by Seagren and Bean in 1981.) The reports have usually been realistic and balanced with respect to statements of strengths, concerns, and problems. Our experience is that more comprehensive and revealing reports make the work of the reviewers more effective. Unit Review Committees. The Unit Review Committees consist of four people. Two are from the discipline (or similar support units) outside the university, one is an internal member from a related department, and one is from the universityat-large. This balance has proven excellent. The unit prepares a slate of about six possible external members and submits this list with resumes to PACCR. For each nominated member of a team there must be evidence of a high level of excellence in a chosen field and evidence of objectivity and trusted judgment. The unit is also invited to nominate candidates for the member from a related 70 Walter E. Harris and Edward A. Holdaway department. One important principle is that all reviewers must be acceptable to the unit as well as to PACCR and the President. Each reviewer is supplied ahead of time with several items, including the self-study report. In 1981 the instructions provided to reviewers were broad and general, and they were not changed much in these 1982 instructions: W i t h o u t i n t e n d i n g to restrict the s c o p e of the review, w e s h o u l d like t o h a v e d r a w n t o o u r a t t e n t i o n w h a t , in t h e o p i n i o n o f t h e m e m b e r s o f t h e U n i t R e v i e w C o m m i t t e e , are t h e s t r e n g t h s a n d w e a k n e s s e s o f t h e d e p a r t m e n t ' s t e a c h i n g a n d r e s e a r c h p r o g r a m s , an a s s e s s m e n t o f the staff and staff workloads, the a d e q u a c y of resources provided, t h e e f f e c t i v e n e s s o f t h e o r g a n i z a t i o n o f t h e d e p a r t m e n t , a n d t h e relat i o n s o f t h e d e p a r t m e n t w i t h o t h e r s . In p a r t i c u l a r , w e s h o u l d a p p r e ciate having drawn to our a t t e n t i o n any opportunities the U R C p e r c e i v e s f o r i m p r o v i n g t h e d e p a r t m e n t ' s p r o g r a m s , its r e l a t i o n s w i t h b o t h internal and external organizations, and for better utilization of r e s o u r c e s a v a i l a b l e . W e a l s o ask f o r y o u r j u d g m e n t o f t h e d e p a r t m e n t ' s proposed plans for the future. The Unit Review Committee has a critical role. Its members must be wise and responsible as well as credible. Involvement of the department in nominating potential members carries the danger of collusion — that the reviewers will state that the quality of the department is better than it is, that deficiencies in support are worse. This danger is in reality minimal. The reviewers meet an exceedingly high standard if they are carefully selected. A team with such qualifications has a great deal of collective judgment to be tapped. Both general wisdom and scientific knowledge are desirable characteristics for all reviewers. Each team member brings a special viewpoint to the deliberations. The review team should provide a strong message and maximum impact. This objective is more likely to be attained if members are given freedom to investigate and report on what they find. Current status. Members of PACCR were selected in June 1980. In September 1980 the review process started with a set of five diverse academic departments, which served well in the development of procedures. In the second year 15 more reviews were started. As of 31 March 1983, 51 units were in various stages: 12 had been completed, 8 had completed the site visits, 8 had completed their selfstudy reports, and 23 were in the self-study phase. The process for reviewing any single unit has taken longer than anticipated particularly during the self-study phase. Nevertheless, completions are now to be expected at the rate of about 15 per year. For the future, continuing involvement with all major parts of the university is expected until completion. The present focus is to remain primarily with reviews of departments and smaller faculties. Later it may be appropriate to look at the larger faculties as organizations comprising departments that have been reviewed. ISSUES IDENTIFIED IN REVIEWS During the review process at The University of Alberta from 1980-83, several issues have been identified. The major issues discussed below — time management, 71 Systematic Reviews of University Programs and Units reallocation of resources, teaching load, the teaching-research balance, and reviews of service/administrative units are referred to in the literature cited above or in articles dealing with university administration. The Time Problem Review teams are brought to the campus for a limited time, usually two days but sometimes three or four. About one third of the time is spent in the department, one third outside the department, and one third in committee discussion. Departments normally want as much review-team time on site as can be allowed. The problem of limited time is expected to be a subject of continuing comment, particularly by academic departments. The time scheduled for on-site activity depends on several considerations. The larger and more complex units obviously require more time. Differences among faculties may warrant different review time on campus. The quality of the self-study report can also affect the time needed for adequate reviewing. Possibly not so obvious in the matter of on-site time are the conflicting interests of the department and the reviewers. Most of those nominated are highly respected, mature scholars who have many demands on their time. Our experience has been that for two-day reviews external reviewers remain for the two full days. Assembling a team for three-or four-day reviews can be more difficult. Reallocation of Resources In common with other universities, The University of Alberta has suffered decreased real support for.its operations, leading to increased teaching and research workloads. These concerns are obvious from statements made in the self-study reports. Claims from the unit are understandably often supported by the review team. But how can university administrators balance claims from "competing" units? What workload is fair and what is equitable? The University of Alberta has reserved a sum of money to allow the President to react positively to matters identified by the reviews. Even though the main purpose of the reviews is program improvement, and many types of improvement can occur without resource reallocation, it is recognized that legitimate cases exist where improvement cannot occur without additional resources (Craven, 1980, pp. 451-452). Further, in view of the complexity of operations of the university, the President and Vice-Presidents are unable alone to assess the comparative validity of claims for additional resources and therefore understandably are turning more and more to members of PACCR for advice. This function, which goes beyond organization of the reviews, was not anticipated in the planning stages. The Teaching-Load Problem Most of the self-study and Unit Review Committee reports present points of view on the teaching load of unit staff. Even though numerical measures of teaching load are used in most universities, they need to be supplemented by informed judgment that comes to grips with these questions. Does load increase directly 72 Walter E. Harris and Edward A. Holdaway with increase in number of hours of lectures? How does load vary as size of class varies? How does load in conducting seminars and laboratories compare with that in lecturing? How does load vary with course level — junior undergraduate, senior undergraduate, and graduate? Is some combination of hours of class contact and size of class desirable and if so how can an appropriate measure be derived? At The University of Alberta, as in other universities, a broad distribution of teaching loads occurs across departments. These differences arise or persist because of factors such as (a) the nature of the discipline, (b) the inability of the university to respond quickly to shifting loads with time, (c) differences in the aggressiveness with which chairmen have historically pursued resources, and (d) the impact of external funding for research, postdoctoral fellowships, and graduate students. The Research-Teaching Problem No sharp dividing line can or should be drawn between the two primary functions of teaching and research. In the broad context, when one is considering workload, the separation of teaching and research may be introducing an inappropriate artificiality. As those instructed become more and more competent in their subject, the activity grades imperceptibly, at least in many disciplines, from formal teaching through informal instruction to pure research. Thus, while teaching includes much more, in a narrow context it can be defined to include only the activities associated with the conduct of regularly scheduled classes. Research then includes the nonscheduled teaching of graduate students and postdoctoral fellows as well as other activities normally called research. We have found, however, that both the self-study and Unit Review Committee reports tend to consider teaching and research as separable functions. Such separation has also been sensed by Gonyea (1980, p. 369). In defence of this position, we recognize that when undergraduate teaching duties become excessive, the energy and time required to conduct research is no longer available. Similarly, the extent of commitment to research may preclude much effort devoted to the teaching function. From personal experience we are aware of some individuals who carry heavy teaching loads and are among the most productive in research. A reduction in teaching load does not necessarily improve research productivity. For most university staff, at least three months are available between winter sessions for research unimpeded by undergraduate teaching. Nevertheless, an important question remains, "How heavy must a teaching load be before it starts to interfere with research productivity?" Service/Administrative Units As only two service/administrative units have been reviewed to completion, it is difficult for us to generalize about reviews of this type. Reviews of these units, such as Purchasing, Personnel, Registrar, Library, Security, Student Services, Computing Services, and Housing and Food Services present problems different 73 Systematic Reviews of University Programs and Units from those experienced with academic units. In particular, because they usually have interaction with nearly every academic unit on campus, a larger number of groups are interested in providing input to the review team than with academic reviews. These units frequently deal with a more diverse range of activities than do many academic units, and this diversity can vary considerably across different universities. Not surprisingly, universities and colleges have typically concentrated more upon reviews of academic units. The attention given to reviews of service/ administrative units by The University of California and The University of Alberta is far from being common. Our experience, however, is that reviews of the support and administrative units are necessary and valuable. Our faith in the advantages of one committee organizing both types of reviews remains strong. CONCLUDING COMMENTS AND RECOMMENDATIONS For reasons related to accountability and self-interest, we need to examine critically the operations of our postsecondary institutions. Consequently, organization of reviews of their programs will probably become a more common practice. These reviews need to be conducted with care and sensitivity so that long-term damage does not occur and maximum benefits are obtained. This need can be partially met by adoption of an appropriate, continuing review mechanism rather than ad hoc procedures initiated when trouble is perceived. The procedures adopted at The University of Alberta provide a " p u r e " example of the executive (presidential) model for conducting reviews. For three main reasons its approach is worthy of detailed consideration by other universities and colleges contemplating adoption of a system of continuing reviews: (1) the procedures were carefully developed over approximately 30 months and benefited from the experience of many other institutions, (2) the procedures are simple, and the administrative/manpower costs are minimal, and (3) the procedures have required little alteration over three years as they have been well received by the departments which have been reviewed. Based upon expert opinion and the experience of The University of Alberta and. other universities, the following recommendations are proposed as some guidelines for universities or colleges contemplating program reviews as a means of helping to improve programs. These should be considered in association with the list of questions proposed by Craven (1980: 444-449). Control. The committee organizing the overall process should be selected by the President, be small in number (five or fewer), and consist of knowledgeable, respected and sensitive staff members representing both faculty and support/ administrative units. It will organize reviews of both the academic and support/ administrative units of the institution. It should keep in close contact with the department/unit at all stages of the review process. Purpose. While program improvement is usually the main purpose of reviews conducted by an institution, we should recognize that multiple benefits can result 74 Walter E. Harris and Edward A. Holdaway from reviews and that program improvement may require reallocation of resources. By placing genuine emphasis upon improvement, a common initial perception that the main purpose of reviews is cutting programs, courses and staff can be corrected. Program review should not be used to deal with routine administrative matters such as poor leadership or poor teaching. These matters should be handled by invoking existing procedures. Procedures. The review procedures must be acceptable to the university/college community. This usually means that they are approved by the governing bodies of the institution. Procedures should be flexible, easily understood, and have minimum prescription. Although not all universities follow this approach, separation of the committee organizing the reviews from the unit review committees is desirable, i.e., they should not have common members. The unit review committee should be composed of appointed internal and external experts who possess sound judgment. All units on campus should be reviewed on a rolling cycle over a period of the order of a decade. A small number of units should be reviewed in the first year, the review process should commence with a self-study activity conducted by the unit. Standardized relevant data should be made available from central sources; these can be supplemented by other data specific to the unit's operations. The self-study phase should take approximately six months and result in a detailed report of history, current operations, and plans for the future. Information should also be collected by the unit review committee from affected groups such as students, recent graduates, employers of graduates, and staff members in other departments/units. All reports resulting from the reviews — the self-study report, the unit review committee's report, and the unit's reaction to the review report — need to be distributed to a restricted audience. Sensationalism is to be avoided. Senior administrators should be responsible for ensuring that approved recommendations are implemented and for monitoring progress of such implementation. Our conviction is that a continuous system of program reviews involving academic and support/administrative units is essential and beneficial, that such reviews can help postsecondary institutions meet their financial challenges, and that the information from these reviews is a necessary adjunct to sound university planning. REFERENCES Arns, R.G. & Poland, W. Changing the university through program review. Journal of Higher Education, 1980,51,268-284. Barak, R.J. Program Review in Higher Education: Within and Without. Boulder, Colorado: National Center for Higher Education Management Systems, 1982. Barak, R.J. & Berdahl, R.O. State Level Academic Program Review in Higher Denver, Colorado: Education Commission of the States, 1978. Education. 75 Systematic Reviews o f University Programs and Units Bélanger, C.H. & Tremblay, L. A methodological approach to selective cutbacks. Journal of Higher Education, 1982, 13, 3, 25-35. Canadian Craven, E. Evaluating program performance. In P. Jedamus and M.W. Peterson (Eds.), Improving Academic Management, San Francisco: Jossey-Bass, 1980. pp. 4 3 2 - 4 5 7 . Council of Graduate Schools in the United States. Assessment of graduate programs. Communicator, October 1979, 12, 1-2. DiBiasio, D.A. & Ecker, G. Academic program review as a loosely coupled system. Paper presented at Annual Meeting of American Educational Research Association, New York, March 1982. Dressel, P.L. & Cammack, E.F. Conducting self-study. In P.L. Dressel and Associates (Eds.), Institutional Research in the University. San Francisco: Jossey-Bass, 1972. Folger, J.K. Implications of state government changes. In P. Jedamus and M.W. Peterson (Eds.), Improving Academic Management. San Francisco: Jossey-Bass, 1980, pp. 48-64. Gentile, A.C. A model for internal review. Communicator, February 1980, 12, 4-7. Gilmour, J.E. Using marketing to incorporate environmental considerations into program review processes. Paper presented at Annual Forum of Association for Institutional Research, Atlanta, April 1980. Gonyea, M.A. Determining academic staff needs. In P. Jedamus and M.W. Peterson (Eds.), Improving Academic Management. San Francisco: Jossey-Bass, 1980, pp. 364-372. Griffin, G. & Burks, D.R. Appraising Administrative Operation: A Guide for Universities and Colleges. Berkeley, California: University of California Systemwide Administration, 1976. Holdaway, E.A. The review process in universities. Australian 2 5 , 2 , 106-120. Journal of Education, 1981, Holmlund, B., Vice-President (Special Projects), University of Saskatchewan. Personal communication, October 1980. Hoyt, D.P. Program evaluation: Is it worth it? Paper presented at The University of Illinois at Urbana-Champaign Conference on Planning and Conducting Program Evaluations and Reviews in Higher E d u c a t i o n , St. Petersburg, Florida, J a n u a r y 1980. Jany, G.O. & Keller, R.J. An evaluation of a state university systemwide academic program review process. Paper presented at Annual Forum of Association for Institutional Research, Minneapolis, May 1981. Matross, R.P. & Delmont, T.J. Planning and evaluation in student services: The design and implementation of new information systems at The University of Minnesota. Paper presented at Annual Forum of Association for Institutional Research, Denver, May 1982. McCorkle, C.O. & Archibald, S.O. Management and Leadership in Higher Education. San Francisco: Jossey-Bass, 1982. McManus, J., Todd R. & Wilson R. An evaluation plan for administrative units at The University of Illinois at Urbana-Champaign. Unpublished paper, January 1982. Miller, R.I. The Assessment of College Performance. San Francisco: Jossey-Bass, 1979. Munitz, B.A. & Wright, D.J. Institutional approaches to academic program evaluation. In E.C. Craven (Ed.) New Directions for Institutional Research: Academic Program Evaluation. San Francisco: Jossey-Bass, 1980. pp. 21-42. Neale, E.A.W. & Armstrong, J.E. Geology and geophysics in Canadian Universities. Paper 80-6, Canadian Geoscience Council, part 1, 1980. 76 Walter E. Harris and Edward A. Holdaway Peterson, M.W. Analyzing alternative approaches to planning. In P. Jedamus and M.W. Peterson (Eds.), Improving Academic Management. San Francisco: Jossey-Bass, 1980, pp. 113-163. Russo, J.R., Brown, D.G., & Rothweiler, J.G. A model for internal program review. College and University, 1977, 52, 2 8 8 - 2 9 8 . Pflaum, A.M. The use of program reviews to assess efficiency and effectiveness in four academic support units. Paper presented at 1982 AIRUM Fall Conference, LaCrosse, Wisconsin, September 1982. Seagren, A.T. & Bean, J.P. Evaluating academic programs: alternative purposes, procedures and results. Paper presented at Annual Meeting of Association for Institutional Research, Minneapolis, May 1981. Shirley, R.C. & Volkwein, J.F. Establishing academic program priorities. Journal of Higher Education, 1978,49,472-488. University of Alberta Report of the University Priorities Committee. Edmonton: 1978. University of Alberta A report to General Faculties Council on proposed terms of reference for a system of reviews of academic programs. Folio, 14 February 1980a, Suppl. 1-3. University of Alberta President's Advisory Committee on Academic Reviews. Folio, September 1980b, 1. 11 University of Calgary. The Shrinking Maze. Calgary: The University of Calgary, 1980. University of Colorado. Final Report of the Committee for Program Review and Resource Allocation. Boulder: 1978. University of Illinois. Council on Program Evaluation: Second cycle evaluation procedure Urbana-Champaign, August 1980. Z i m m e r m a n , K.C., Associate Dean, Faculty of G r a d u a t e Studies, University of Minnesota. Personal c o m m u n i c a t i o n , May 1981. ACKNOWLEDGMENT WEH wishes to acknowledge the comments, advice, and interest of the other members of PACCR, Professors F. Enns and H. Kreisel and Mr. W. Jopling.