The Canadian Journal of Higher Education, Vol. XV-3, 1985 La revue canadienne d'enseignement supérieur, Vol. XV-3, 1985 A Survey of University Policy Makers' Preferences and Expectations for Provincial Examinationst ROSS E. TRAUB and LESLIE D. McLEAN* ABSTRACT A survey was conducted of persons responsible for making undergraduate admission policies at Ontario universities to ascertain their preferences and expectations for provincial examinations. Fifty-eight individuals, at least two from each university, responded to a series of questions by telephone interview or questionnaire. Strong support was expressed for the reintroduction of provincial examinations for mathematics and first language (English or français) courses in the final year of the secondary school program. Most respondents rejected the use of scores on province-wide examinations for rating secondary schools and adjusting school marks. Instead, they expressed a preference for having applicants report both teacher-assigned course marks and provincial examination scores. RÉSUMÉ Nous avons mené une enquête auprès des personnes responsables des règlements d'admission au premier cycle dans les universités de l'Ontario afin de connaître leurs attentes et préférences au sujet des examens provinciaux. Cinquante-huit personnes, dont au moins deux de chaque université, ont répondu à une série de questions par voie téléphonique ou par questionnaire. Les répondants ont montré une forte tendance en faveur de la ré-introduction d'examens provinciaux pour les mathématiques et la langue première (l'anglais ou le français), examens qui seraient administrés à la fin de la dernière année d'école secondaire. La plupart on rejeté l'idée de se servir des résultats de ces examens pour l'évaluation des écoles secondaires et pour le réajustement des notes scolaires. En fait, ils préféreraient que les candidats présentent les notes accordées par leurs enseignants ainsi que les résultats de leurs examens. *The Ontario Institute for Studies in Education tThis research was funded by a Small-Scale Research Grant from OISE. We gratefully acknowledge the contribution of Cathy Allen. 10 Ross E. Traub and Leslie D. McLean INTRODUCTION Whether some of the achievements of students in their final years of secondary school should be assessed by provincial examinations, by which we mean examinations developed and administered under the auspices of the Ministry of Education, is a question under active consideration in Ontario. That this is so is a consequence of governmental response to a variety of pressures: 1. The Council of Ontario Universities (Council of Ontario Universities, 1984) has called for a return to school-leaving examinations in language - English for Anglophones, français for Francophones - and mathematics. 2. The Commission on the Future Development of the Universities of Ontario was to "address a number of specific issues related to accessibility (to Ontario universities) such as the need for, and form of, general and specific entrance examinations to the Ontario university system, with reference to the new secondary school curriculum structure" (Minister of Education's statement to the Legislature of Ontario, 15 December 1983). Recently this Commission submitted its final report, in which it was recommended: ".. .that admissions direct from secondary schools be based on a combination of teachers' marks and school reports and of province-wide admissions examinations assessing achievement in at least language (English or français) and mathematics, but that alternative arrangements for the admission of mature students be continued" (The Commission on the Future Development of the Universities of Ontario, 1984, p. 37.) (No rationale for this recommendation was provided.) 3. There has been considerable editorial support for a return to provincial examinations. For example, in December 1983, editorials calling for school leaving examinations appeared in two major Toronto newspapers ("Expensive Ignorance," 1983; "Province-wide," 1983). These were in response, ostensibly at least, to a report out of Carleton University that most students admitted to the first-year program in computer science had failed mathematics. The editorial writers speculated that too many of the students admitted to Carleton's computer science program had been assigned marks in secondary school that were spuriously high compared to the marks assigned to other, more deserving, students whose teachers had espoused tougher marking standards. In addition, the editorial writers expressed the opinion that standards of education in Ontario have fallen since the discontinuation of the Grade 13 examinations. It seems they believe school-leaving examinations would reduce, if not eliminate, inequities in university admissions due to variation from school to school in standards of marking, and in addition would raise educational standards across the province. 4. Informed critics of education have called for a return to school leaving examinations. For example, Holmes (1984a, b) advanced the case for examinations, in part for the reasons given in the aforementioned newspaper editorials. In addition, Holmes argued that examinations could be used to satisfy the public's need for accountability and to monitor changes in educational standards over time. 11 A Survey of University Policy Makers' Preferences and Expectations for Provincial Examinations (It is not clear what Holmes and the editorial writers meant by educational standards, but that large issue is not considered here.) Other jurisdictions have either seen fit to return to a system of examinations (e.g., British Columbia and Alberta) or never abandoned them (e.g., Newfoundland and Quebec). An indication of the Ontario Government's thinking in response to these pressures was revealed in the Speech from the Throne, delivered 20 March 1984: In consultation with the Council of Ontario Universities and the Ontario Teachers' Federation, the Government will work to design a province-wide testing program necessary to assess the effectiveness of our curriculum and the performance of our students. The teacher in the classroom is the cornerstone of excellence in education, and, to a great extent, the promise of Ontario. However, to assist the Government in meeting its responsibilities, and parents in participating in their children's education, such tests will help all of us maintain the high quality of our educational system. The form of the initiative promised in the Throne Speech has yet to be defined. A search is under way for the design and supporting rationale of an acceptable testing program. University Admissions and Provincial Examinations It has been suggested that universities need provincial examination scores to make equitable admission decisions. As some see it, the problem is to reduce, if not eliminate, differences among schools and among teachers within schools in marking standards. Approaches for doing this fall into two broad categories, here called direct and indirect. With direct approaches, the applicants' examination scores are used in the process by which admissibility is decided. Either the examination scores are the sole criterion of admissibility or they are considered along with (unadjusted) school marks or they are used to adjust school marks, which then form the sole criterion of admissibility. With indirect approaches, the examination scores of students admitted in a previous year or series of years are used to adjust the school marks of current applicants, these adjusted marks then forming the criterion for admission, but the examinations cores of applicants are not considered. Why is it that anyone would use an indirect approach? One reason is that some examination systems cannot provide scores early enough for universities to use them directly. One of the arguments for discontinuing Ontario's Grade 13 examinations in 1967 was that the universities could not wait until August for the examination results in order to decide whom to admit for the term beginning the very next month. For many years, Ontario universities have admitted students to the term beginning in September in at least two stages. The first stage begins after mid-April, by which time the secondary schools have sent such marks as are available for students applying for university admission to the Ontario University Application Centre (OUAC). In the first stage of the admission process, the information assembled by the OUAC is sent to the universities, and by mid-June 12 Ross E. Traub and Leslie D. McLean preliminary offers of admission have been sent to applicants. Preliminary offers are made this early to allow for a second stage in the process, during which offers of admission are sent to enough additional applicants to make up the shortfall between the size of first year class that is desired and the number of applicants who accept the preliminary or first-stage offer of admission. As far as we have been able to determine, the preliminary offers of admission are in effect final offers of admission for virtually all the students who choose to accept them. It seems that final school marks rarely affect the decision to admit a student, even when they are noticeably lower than the marks on which the preliminary decision was made. Given this system for admitting undergraduate students to Ontario universities, provincial examinations administered at the end of courses taken in the final year or the final semester of the final year of secondary school could not play a direct role in the process by which decisions are made about preliminary offers of admission. They might, however, play an indirect role, or some other kind of province-wide testing system might be adopted, one that would provide scores early enough for them to be used in the process of deciding on preliminary offers of admission. Purpose The purpose of the survey described in this report was to ask those individuals in Ontario universities who are responsible for policy on undergraduate admissions whether they agree on the need for provincial examinations and, if they do, to ascertain the preferred type of examination and preferred use of examination scores in the admission process. It may be argued that university professors and administrators will respond predictably to the question of whether or not there should be provincial examinations. Still, it is well, from time to time, to test our conceptions of the predictable. And even if there is agreement on the need for provincial examinations, it does not follow that there will be agreement on the form those examinations should take or on the uses that should be made of the examination scores. METHOD Respondents The target population was defined initially as those individuals responsible for making undergraduate admission policy at 15 of Ontario's 17 degree granting institutions. (The names of the 15 institutions appear in Table 1; Royal Military College and Ryerson Polytechnical Institute were excepted.) Subsequently, the definition of the target population was narrowed to include only those individuals who are members of university senate committees (or the equivalent) on undergraduate admissions and who are either central administrators (e.g., registrars, vice-presidents) or members - administrators or teaching staff - of the faculties of arts, science, engineering or pharmacy. (Attention was focussed on 13 A Survey of University Policy Makers' Preferences and Expectations for Provincial Examinations these faculties so as to tap opinions from representatives of faculties that are highly selective and of faculties that, relatively speaking at any rate, are less highly selective.) The foregoing conception of the target population could not be realized at three universities, which had no senate committee or equivalent responsible for undergraduate admission policy. In these cases, the target group consisted of the chairpersons of the committees responsible for undergraduate admission policy in each target faculty plus a central university administrator responsible for undergraduate admission policy. Procedure The 1983-84 membership list of the Ontario University Registrars Association was used to identify a central administrator at each institution likely to be knowledgeable about the undergraduate admissions process. Telephone calls to these individuals elicited the names of the persons in the target population. Information was collected in two phases, by a different means in each phase. The first was a structured telephone interview. A member of the target population was interviewed at 12 of the 15 institutions. The registrar or associate registrar for admissions or director of admissions was the person interviewed at eight of the institutions. At the other four institutions, the chairperson of the senate committee on undergraduate admissions (or equivalent) was interviewed. The second means of collecting information was the questionnaire. The questions in this instrument were based on those asked in the structured interview. There were 11 questions, three with more than one part. Eight questions or parts thereof could be answered by choosing one of several predefined responses. All but three questions provided space for written responses or comments. The questionnaire was mailed to all the persons in the target population who had not been interviewed. One follow-up request was sent to questionnaire recipients who had not responded within three weeks of the initial mailing. Characteristics of the Respondents The intended and realized number of respondents for each university are reported in Table 1. The total number of persons asked to respond was 80, while the number of persons who actually did respond was 58, for a response rate of 73 percent. This compares favourably with the response rates reported for other surveys (Dillman, Dillman & Makela, 1984). The total group of 58 individuals who responded, either to the interview or the questionnaire, can be further described as follows: 1. Twenty-six described their positions as chiefly administrative, 24 as chiefly teaching and research, and 7 as involving both administration and teaching and research. (One person did not reply to this question.) 2. Forty-one of the 58 respondents indicated they were members of particular faculties or departments. When divided on the basis of professional (engineering, 14 Ross E. Traub and Leslie D. McLean Tablet: I n t e n d e d and R e a l i z e d Numbers of R e s p o n d e n t s f r o m Each of 15 O n t a r i o U n i v e r s i t i e s University N u m b e r of R e s p o n d e n t s Intended Realized Brock U n i v e r s i t y 4 2 Carleton University 5 4 U n i v e r s i t y of G u e l p h 4 3 Lakehead University 6 3 Laurentian University 7 5 McMaster University 5 4 U n i v e r s i t y of O t t a w a 5 5 Queen's University 4 4 U n i v e r s i t y of T o r o n t o 4 3 Trent University 6 6 U n i v e r s i t y of W a t e r l o o 5 5 10 3 Wilfrid L a u r i e r U n i v e r s i t y 3 2 U n i v e r s i t y of W i n d s o r 5 5 York U n i v e r s i t y 7 4 80 58 U n i v e r s i t y of W e s t e r n O n t a r i o Totals pharmacy) versus arts and science programs, 11 respondents fell in the professional category and 30 in the arts and science category. When divided into three groups - arts, science and professional - the numbers were 16, 14, and 11, respectively. 15 A Survey of University Policy Makers' Preferences and Expectations for Provincial Examinations 3. Fifty-one persons reported that they had been required to take some kind of examination to get into university, while only six said they had not. (One person did not respond to this question.) The number of individuals in the intended population who chose not to respond (N = 22) was substantial in relation to the number who did respond. We possess insufficient information about the positions most of these individuals filled to categorize them either by chief responsibility (administration versus teaching/ research) or by faculty. Thus, we cannot compare the respondent and nonrespondent groups in any meaningful way. We do note, however, that with respect to the question, "Do you think your colleagues would answer these questions as you have, or are there considerable differences of opinion about this topic?," 53% of the respondents thought there would be differences of opinion. This underlines the need to exercise caution in generalizing the results of the respondent group to the intended population. RESULTS The principal results can be summarized easily: Most respondents (49 of 58 or 84%) were in favour of provincial examinations at the end of secondary school. The preferred proposal was end-of-course examinations for Honour Graduation (Grade 13) Courses or, under the new secondary school system, Ontario Academic Courses. (47 or 81% rated this type of examination favourably; 33 or 57% said they preferred it to any other examination model.) Most respondents (40 or 49%) thought that the examinations should be written by all students, not just those going to university. As for the use of examination results in the admission process, most respondents (47 or 81%) approved of considering the scores along with school marks. No significant associations were found between the responses that generated the foregoing summary and the positions of the respondents (administration versus teaching/research) or their faculties (professional versus arts and science). A number of points not brought out in the foregoing summary merit further attention. The Type and Timing of Examinations Answers to the first question revealed that respondents were in favour of some type of mandatory examinations at the end of secondary school by a margin of about six to one. Most of those who favoured such examinations (36 of 48) felt that all students should take them (question two). In the several sections of question three, we tried to determine respondents' preferences among different types of examinations and different times the examinations might be given. The first alternative offered was "End-of-course examinations for Ontario Academic Courses or, under the present system, Honour Graduation Courses". As would be expected, responses were consistent with 16 Ross E. Traub and Leslie D. McLean question one, only six people changing from favour to oppose and three from oppose to favour. Seven people commented that end-of-course examinations would be too late for universities to use in admissions decisions, but two of these respondents were still in favour of examinations at the end of secondary school courses. Other comments made it clear that respondents were in favour of examinations in general, whether or not they would be useful to the universities. Seven of the ten respondents who were opposed to examinations at the end of Honour Graduation or Ontario Academic Courses listed themselves in "chiefly administrative" positions, one each from seven different universities. Five of the seven commented that the exams would come too late to be used in admissions decisions. As we have noted elsewhere in this report, reliance on examinations given at the end of courses would require a major change in the present admissions process, so it is puzzling that more people did not mention it. One interpretation consistent with the data is that respondents assumed that admissions would continue to be handled as at present but that province-wide examinations would ensure that secondary school students paid attention to their academic studies through to the end of the academic year. The matter of timing had come up in pilot testing, so question 3b offered the alternative of "Early in final year tests of achievement." A majority still favoured this alternative, but the margin was smaller than in question 3a (62% vs. 81%). The comments of those who were opposed stemmed from a concern that the examinations should be related to the curriculum and should reflect achievement in the final year. Those in favour commented again along the line of general usefulness, but some did remark that such scores could be used in admission decisions. Two people pointed out a potential complication, that the timing of the examinations might give advantage to (or work against) semestered schools, since the classes in these schools would have covered different amounts of the course at the time of the examination than the classes in unsemestered schools. The most popular university entrance examination in the United States is the Scholastic Aptitude Test (SAT), administered by the Educational Testing Service for the College Entrance Examination Board. It is not advertised as an achievement test, being made up of verbal and numerical sections not related to any particular curriculum. Students normally take the examination in the late fall of their final secondary year, when most of them are 17 years of age. Question 3c asked whether respondents favoured "Early in final year tests of scholastic aptitude, e.g., SAT." Fifty percent of respondents favoured this option. Approximately half of those opposed offered comments, which showed either a clear preference for achievement tests or a skepticism for the capacity of SAT-like tests to judge student ability to succeed in university. The final alternative presented was "End-of-course examinations given the year before graduation to students who are planning to attend university". The percentage approving was 52, but the comments revealed many qualifications. This alternative got only one vote when compared directly to the other alternatives (see below). The comments in favour were variants on the idea that practice would 17 A Survey of University Policy Makers' Preferences and Expectations for Provincial Examinations be a good thing for the students or that the results would be useful to students, rather than citing any advantage to the universities. Respondents were then asked, "Are there other examination systems you would consider? Please describe." Only 13 people responded, five of them to say "no". Another four said universities should set their own examinations. The Ontario Assessment Instrument Pool was mentioned twice. It was clear that the respondents had no strong preferences not mentioned, nor was there anything like consensus. The final question in this series was "If you had to pick one of the examination systems outlined, which would you prefer?" Thirty-three (57%) chose endof-course examinations, with early-in-final-year achievement tests and SAT-like tests tied for second, each being chosen by six respondents. Given all the alternatives suggested by the previous questions, this strong majority is a clear indication of preference. Which Courses? Respondents were asked to name the courses that should have examinations. Ten different combinations of courses were named. One person indicated that a test of mathematics would suffice, but everyone else who answered this question (52 of 58) either included both mathematics and first language (English or français) on their lists or implied that these should be included (e.g., by indicating that "all courses" or "as many courses as possible" should be examined). The following question was also asked: "If examinations were given the year before graduation (from secondary school), which courses should be examined?" Forty-eight individuals expressed the opinion that the courses examined should be the same as those that would be examined were the examinations administered at the end of secondary school. More on Timing Respondents were asked to indicate the time in the year by which examination scores would have to be available if they were to be useful in choosing among applicants for September admission. The answers were surprisingly varied, ranging from as early as the preceding January to as late as August, the month before the start of the academic year. Most of those who answered this question specified a deadline in the interval late-April to early-June (N = 23). Seven respondents specified a deadline in the earlier period (January to mid-April) and 13 indicated a later period (mid-June to August). (Fifteen respondents declined to answer, several indicating that they had no knowledge on which to base a response.) Many respondents, including all but one of the persons who were interviewed, were aware that examination scores are required by mid-June or earlier if they are to be considered in the award of preliminary offers of admission. This deadline would rule out the use of end-of-course examinations taken by students 18 Ross E. Traub and Leslie D. McLean completing courses in late May or early June, yet this is the very type of examination that the majority of respondents preferred. An attempt was made by one person to resolve this contradiction with the suggestion that the examination scores could be considered during the summer, as part of a confirmatory phase in the admissions process. The introduction of a serious confirmatory phase into the process would appear to represent a significant departure from much of present practice. Use of Examination Scores in University Admissions The quantitative results for this issue are reported in Table 2. It is apparent that, of the three suggested methods for using examinations scores in the admission process, most respondents favoured that in which the university has available for consideration both examination scores and teacher marks; majority sentiment was strongly against the other two methods. Comments written on the questionnaire provide a measure of insight into the basis for the responses summarized in Table 2. The strong preference for method 6a derives mainly from the notion that the reporting of both examination scores and school marks would give universities maximum information. The very strong rejection of the use of examination scores to adjust school marks (question 6b) seems to rest, at least in part, on the idea that it would be difficult to obtain widespread agreement on a suitable mechanism. There were, however, several expressions of interest in the use of examination scores to monitor school marks, with the added suggestion that appropriate follow-up action should be taken when a school's marks were found to be seriously out of line. Two main reasons were used to explain the rejection of the method of rating schools described in question 6c: The number of students going from a particular secondary school to a particular university would be too small in the main for the rating to be reliable, and any changes in a school's teaching staff, marking policies and student body would not be reflected in the school's rating. In question 6d respondents were asked to describe what they thought the best approach would be for using examination scores in the university admission process. A substantial number (N = 34) did not respond; most who did suggested approaches similar to one or another of the three named in 6a, 6b, and 6c. A very few responses were unique: "Make examinations the sole basis for admission." "My impression is that 90% of university admissions and denials are clear cut. So why an elaborate system for a few borderline cases?" "Identify those who are unquestionably admissible; subjectively assess the information on the other applicants." Other Benefits of Examinations Respondents were asked to describe benefits other than those involving admissions that a provincial examination system would confer on the applicants, the universities and the secondary schools. The list of different suggestions was long, 19 A Survey of University Policy Makers' Preferences and Expectations for Provincial Examinations T a b l e 2: S u m m a r y of R e s p o n s e s to Q u e s t i o n 6: W h a t is y o u r o p i n i o n of t h e f o l l o w i n g w a y s of u s i n g e x a m i n a t i o n s c o r e s in t h e a d m i s s i o n p r o c e s s ? Type of Response Method Approve Disapprove No Response N % N % N % 47 81 7 12 4 7 8 14 45 78 5 9 18 31 35 60 5 9 (a) I n c l u d i n g e x a m i n a t i o n scores a l o n g w i t h school m a r k s in a n e q u a t i o n for d e c i d i n g w h o is admissible. (b) U s i n g e x a m i n a t i o n s c o r e s to a d j u s t school m a r k s , w i t h o n l y a d j u s t e d m a r k s r e p o r t e d to universities. (c) U s i n g t h e e x a m i n a t i o n s c o r e s of s t u d e n t s a d m i t t e d in p r e v i o u s y e a r s to r a t e schools, t h i s r a t i n g being t a k e n into account when current applicants are admitted. and contained the expected entries: motivating students to work hard, eliminating the need for literacy examinations now common in universities, assuring a common secondary school curriculum in the examination subjects, and the like. There were no surprises and no additional information was gained. Minority Report This was "written" by seven respondents who stood in moderate opposition to any form of provincial examinations. (No respondents were strongly opposed.) Stated reasons for opposition were varied. One reason was possibly based on facts: 20 Ross E. Traub and Leslie D. McLean "Studies indicate that school marks are good predictors of university performance; provincial examination scores would add little." Another reason was based on a belief that could be put to empirical test if examinations were introduced: "Examinations reduce curricular flexibility." Of perhaps greater significance was a reason rooted in the political realities of Ontario education: "The antagonism of teachers toward examinations should be considered." The issues raised in these comments were either of no concern to those who provided the basis for the majority report, or they were of insufficient importance to outweigh the factors that prompted most respondents to agree on the need for provincial examinations. A further point of clarification about the minority view: Several adherents of this view took pains to be clear that they were not against examinations per se. They stated that in their view examinations are an integral and important part of the instructional process, and yield many beneficial effects (e.g., enhanced student motivation, need to integrate material). What these respondents were opposed to was provincial examinations, preferring instead exclusive reliance on teacher-set class examinations. CONCLUSIONS The main conclusion we draw from the results of this survey is that there is strong support for provincial examinations among those responsible for the undergraduate admission policies of Ontario universities. Another conclusion, which strikes us as surprising, is that support is strongest for provincial examinations at the end of Honour Graduation or Ontario Academic Courses. This preference is surprising because the scores of at least some applicants on these examinations would become known far too late to influence university admission decisions, at least as the admission process now operates. Some of our respondents seemed unaware of this fact; others seem to have been thinking of the other educational benefits that provincial examinations might confer. An important conclusion to be considered by those who contemplate mounting an examination system concerns the respondents' preferences for examination subjects. First language (English or français) and mathematics appeared on all lists, and some lists contained no other courses. This preference accords with that expressed by the Council of Ontario Universities. Finally, attention must be drawn to possible uses of examination scores in the admissions process. If examination scores can possibly be considered in this process, the preference of the respondents was solidly behind having both examination scores and school marks available for consideration. This is another factor that those who contemplate implementing an examination system should bear in mind. 21 A Survey of University Policy Makers' Preferences and Expectations for Provincial Examinations REFERENCES Commission on the Future Development of the Universities of Ontario. (December, 1984). universities: Options and futures. Toronto: Author. Council of Ontario Universities. (1984, April). Developments interface (Briefing Notes No. 7). Toronto: Author. of the Ontario secondarylpostsecondary Dillman, D . A . , Dillman, J.J. & Makela, C.J. (1984). The importance of adhering to details of the total design method (TDM) for mail surveys. In D.C. Lockhard (Ed.), Making effective use of mailed questionnaires: New Directions for Program Evaluation, No. 21 (pp. 49-64). San Francisco: Jossey-Bass. Expensive ignorance. (1983, December 27). The Toronto Globe and Mail. Holmes, M. (1984, May 22). Provincial testing and the university. University of Toronto Bulletin, p. 11. Holmes, M. (1984, May 3). Putting students to the test. The Toronto Globe and Mail, p. 7. Province-wide exams needed. (1983, December 13). Toronto Star. Staff. (1984, April). Developments at the secondary/postsecondary interface. Briefing Notes, No. 7. Toronto: Council of Ontario Universities.