The Canadian Journal of Higher Education, Vol. XXI-2, 1991 La revue canadienne d'enseignement supérieur, Vol. XXI-2, 1991 Outcomes Assessment: How Much Value Does University Education Add?t FREDRICK T. EVERS* & SID N. GILBERT* Abstract Data from outcomes found two independent resulting to produce added development. However, development much educational outcomes, characteristics research from formal studies university value on a number on a number less value is added it is argued, which future university are used instruction. of important of other represent graduates is dimensions of student of student university courses. some of the more will student education dimensions important by formal to assess University The latter crucial require. t Data reported in this paper are from the "C.E.A.S.E.", "Educational Outcomes," and "Making the Match" projects. The Career and Education Achievement in the Student Environment project and the Student Characteristics, Institutional Structures and Educational Outcomes project (Sid Gilbert, principal investigator) are supported by funds from the Social Sciences and Humanities Research Council, the Department of the Secretary of State of Canada, the Ministry of Science and Technology Canada, and the University of Guelph. The Making the Match Between University Graduates and Corporate Employers research project (James Rush and Frederick Evers, principal investigators) is supported by funds from the Social Sciences and Humanities Research Council, the Department of the Secretary of State of Canada, the University Research Incentive Fund (Ontario Ministry of Colleges and Universities), and by research awards from the School of Business Administration, University of Western Ontario and from the University of Guelph. * University of Guelph, Sociology and Anthropology. Paper presented at the Annual Conference of the Canadian Society for the Study of Higher Education, Victoria, B.C., June 3-5, 1990. The authors would like to thank Jasna Krmpotic, Joanne DuncanRobinson, and Caryl Durst for the research assistance on the projects reported in this paper. 54 Frederick T. Evers and Sid N. Gilbert Résumé Les informations provenant de deux recherches indépendentes sont utilisées pour évaluer les résultats d'une éducation universitaire formelle pour les étudiante)s. On trouve que l'éducation universitaire produit une valeur ajoutée dans nombre de dimensions importantes du dévelopement des étudiant(e)s. Cependant, les cours universitaires formels ajoutent beaucoup moins de valeur dans un certain nombre d'autres dimensions importantes du dévelopement des étudiant(e)s. Ces derniers résultats de l'éducation représentent, pense-t-on, quelques unes des caractéristiques les plus cruciales dont les futures gradué(e)s universitaires auront besoin. Introduction A great deal of research takes place at Canadian universities. However, Canadian universities do very little research on themselves in terms of the characteristics of the students admitted and their progress within universities or within the post-secondary system. Apart from the type of university degree and the final grade average, universities know little about other specific outcomes of education such as cognitive and affective consequences . One of the major conclusions emerging from the National Forum on Post-Secondary Education held in Saskatoon, October 1987 was that more detailed research and data were needed on educational outcomes and that this should be one of four priorities which required immediate federal-provincial co-operation (Access to Excellence: A Status Report by the Secretary of State of Canada, 1988). Brian Segal, Chairperson of the National Forum, said in his Report to the Secretary of State of Canada and to The Council of Ministers of Education for Canada, that at the present time, we really cannot say with any confidence what works and what does not work; there are no student profiles or adequate performance measures (Segal, 1987:12-13). Other participants at the National Forum on Post-Secondary Education expressed similar sentiments: "Since no national data are available for comparison, we have not examined changes in failure and drop-out rates, program switches or length of study ... or calculated what percentage of them actually obtain a degree. Yet the success of a genuine policy of accessibility should be measured by individual progress, by the 'value added' by the system, rather than by admissions alone" (Fortin, 1987a: 18). Outcomes Assessment: How Much Value Does University Education Add? 55 "Might there be a role here for a neutral intermediary, an agency whose responsibility is to construct and publish comparative national indicators of institutional performance?" (Cameron, 1987:11). "How ... can we respond to the needs of students keeping in mind their varied backgrounds and respecting their values, and at the same time, supply them with quality training and education? How do we simultaneously define quality, results and added value?" (Fortin, 1987b:71, emphasis ours). If student learning is an essential goal and purpose of university education, then it is remarkable that so little university research is conducted on the structures and processes which facilitate student learning and development. As the Task Force on Post-Secondary Education in Manitoba noted in 1974 "Is it not strange that the universities, which stress so much and so rightly their research function, do so little to find out how to perform better their own chief r e s p o n s i b i l i t y , t e a c h i n g and l e a r n i n g ? " ( R e p o r t of the Task Force on Post-Secondary Education in Manitoba, 1974:77). The Task Force also noted that the essence of an evaluation of university education was an assessment of how much student learning had taken place (p. 81). This paper includes an investigation of a range of outcomes of university education focusing on the skill development process. The paper also provides an examination of the perceived competency levels of university students and graduates and relates these perceived performance levels to various sources of added value. What skills, abilities and qualities might be enhanced by post-secondary education? A consensus has emerged around the following basic aspects: reading and communication skills, thinking and reasoning skills, critical intellectual and analytical skills, q u a n t i t a t i v e or c o m p u t a t i o n a l skills, substantive knowledge of a field of study, sensitivity, creativity, wisdom and integrity (Bowen 1977; Astin 1985; Boyer 1987). One university has officially stated its learning objectives and published these at the beginning of its undergraduate calendar together with textual description. The stated learning objectives are: literacy, numeracy, sense of historical development, global understanding, moral and aesthetic maturity, understanding of forms of inquiry, depth and breadth of understanding, independence of thought and love of learning (University of Guelph, 1989). Furthermore, education is recognized as not merely a question of intellectual growth but "...also includes growth in the emotional, spiritual, social, and physical aspects of the human character" (University of Guelph, 1985). If university programs and courses are intended to produce graduates with these kinds of qualities, it is necessary to assess, first, whether or not graduates, 56 Frederick T. Evers and Sid N. Gilbert in fact, have these qualities and, second, whether or not they acquired them as a result of university teaching and instruction and not merely on account of maturation, selection or other extraneous influences. It also would not be wise to assume that these very specific skills, competencies and abilities are indicated by course grades (Gilbert, 1989). Ideally, appropriate assessment would involve examination of the gains between pre-education and post-education measures of student development on the desired qualities that graduates should possess. To examine exactly what benefits are derived during the university years, Astin (1987:95) argues that it is necessary "to determine both entering and exit levels of competence." This concept, which Astin labels "talent development," provides a useful framework with which skill acquisition may be studied. It would be nice to have objective, pre- and post-university outcome measures. In this paper, however, we will be relying upon students' and graduates' estimates of their development on a number of dimensions and upon their specification of the source of that development. Data from two separate studies of the consequences of education are employed. First, longitudinal data concerning the general and specific outcomes of f o u r years of study, that is, the proportion of educational o u t c o m e s attributable to university study, for institutional persisters are considered (CEASE Project data, University of Guelph). Second, data concerning a wide range of skill development and the source of skill development for university students from five Ontario universities and university graduates working in 20 major Canadian companies, are analyzed (Making The Match Project). The analyses, including variation by degree program, provide a preliminary estimate of how much value university education contributes to student development. Methods: CEASE Project The C E A S E Project (Career and Education Achievement in the Student Environment) was conceived initially to test Vincent Tinto's (1987) model of attrition from university, with a tighter research design and better data than existing American studies. The project features a longitudinal design rather than a cross sectional approach and, consequently, captures current measures of the variables as opposed to retrospective measures. The project also contains a more precise operationalization of the original dependent variable, student departure, by differentiating among transfers, those required to withdraw, stopouts, and system leavers. Outcomes Assessment: How Much Value Does University Education Add? 57 In the Fall of 1986, all new first semester students at the University of Guelph were surveyed on their background characteristics and their aspirations and expectations regarding university life. These same students were contacted again after two semesters (in the Winter 1987 semester) and data were obtained concerning actual university experiences, problems, learning and knowledge acquisition along with various student satisfaction measures. In the Fall 1987 semester, the same cohort of students was again contacted for an even more detailed evaluation of how well studies were progressing in terms of k n o w l e d g e and skill acquisition and in terms of intellectual development. In the Winter 1988 semester, all leavers from the University, from the start of the project to that date, were telephoned (interview) to ascertain their exact location in the post-secondary/ occupation systems and to ascertain the reasons for any change of plans. The study design permits comparisons between stayer and leaver groups or, for that matter, among those required to withdraw, stayer and departure groups, based upon differential experiences at the University of Guelph and based upon a variety of social background characteristics. A wide range of behavioural and attitudinal items is contained in the total data set. The focus in the initial research was upon student progress and student departure and a number of papers have reported the findings in that regard (e.g., Gilbert and Evers, 1989; Gilbert, et al„ 1989). A f o u r t h s u r v e y has r e c e n t l y been f u n d e d by S S H R C C ( S t u d e n t Characteristics, Institutional Structures and Educational Outcomes). In the Winter semester of 1990 students received a final questionnaire which measured prior-to-graduation educational outcomes. Data exist on the same cohort of students/ leavers over the four year course of their educational or occupational careers. With this type of longitudinal design (four over-time questionnaires and with the information contained in the student record system i.e. prior-to-entry background characteristics including gender, high school p e r f o r m a n c e measures, university program and university performance measures), the C E A S E p r o j e c t data are w e l l - s u i t e d to a s s e s s i n g the d e v e l o p m e n t of educational outcomes. Response rates to the total population (not sample) s u r v e y s h a v e been e x c e l l e n t to good i.e. 7 0 % , 70%, 4 8 % , 4 5 % , 55% respectively, and the absolute numbers are large: Ns = 1,937; 1,626; 906; 264; 800. 58 Frederick T. Evers and Sid N. Gilbert Methods: Making the Match Project The Making the Match Project ( M T M ) was designed to investigate the education and training experiences of university students and graduates in Canada. The first phase (Rush and Evers, 1986a; 1986b) was commissioned by the Corporate-Higher Education Forum (C-HEF), a group of Chief Executive officers of major corporations and presidents of universities who are interested in issues common to both types of organizations. The purpose of that study was to examine the perceptions of managers and university-educated employees of large Canadian corporations about the adequacy of university education for corporate employment. The second phase of the MTM Project looks at skill acquisition over the fifteen year period from early university until people are ten years out in their careers (Rush et al., 1989). It is designed to gather information on the skills developed: by students and graduates from different programs of study; at various career stages; due to formal instruction as well as practical experience; and in the context of changing organizational systems and structures. R a t h e r than f o l l o w university students over a f i f t e e n year period, a p r o x y - m e t h o d was chosen which requires selecting groups of people to represent key career stages and then following each of these groups over a three year period. This cross-sequential design has some of the benefits of true longitudinal designs, yet it is more manageable. Five stages were chosen as a framework for the study: 1. Early university; 2. Pre-graduate; 3. Job entry; 4. Job change; 5. Stabilized career. The early university group or "cohort" represents students who are in their second or third year of university. At that stage in their university education, students may have completed two years in their chosen program or they may be deciding whether to apply to a specific program for their last two years. The pre-graduate cohort represents students in their last year of an undergraduate program. The remaining three cohorts represent university graduates working in corporations. People who have been out of university for six to eighteen months belong to the job entry cohort, those who graduated two to six years ago are part of the job change cohort, and those who have been out of university for more than seven years are in the stabilized career cohort. Each student and graduate was asked to nominate a professor or manager who could provide further information. The professors and managers were surveyed to provide a validity check on the self-assessment of the students and graduates and to provide additional information on the process of skill development in universities and corporations. Outcomes Assessment: How Much Value Does University Education Add? 59 In the first year, questionnaires were returned by 1,548 students in five Ontario universities; 580 nominated professors; 1,873 graduates working in 20 Canadian corporations; and 1,056 managers of the graduates. The cohort breakdown of the first year data is: 1. Early university = 526 students; 2. Pre-graduate = 1,013 students (cohort could not be determined for 9 students); 3. Job entry = 434 graduates; 4. Job change = 758; and 5. Stabilized = 639 (with 42 where cohort could not be identified). The data presented in this paper constitute two aspects of skill development: perceived competency levels and the source of skill development. Skill levels and the source results are analyzed by career stage and academic program. Findings: CEASE Project What specific student outcomes are produced by four years of study at university? Students were first asked how they rated their level of competence or performance on the following dimensions: thinking and reasoning skills, problem solving skills, decision making skills, planning and organizing skills, time management skills, communication skills, interpersonal and social skills, quantitative/ mathematical skills, independence and supervisory skills. Generally students considered themselves to have high levels of competence on each of these aspects (Table 1). Thinking and reasoning, problem solving, decision making, planning and organizing, interpersonal and social, and independence were estimated as very high, with independence the highest. Time management, communication, supervisory, and quantitative/ mathematical skills were seen by students to be less developed, with quantitative/ mathematical the least developed. These levels of competence or performance may have arisen as a result of a number of activities or processes, formal university instruction being only one possible source of student development. Accordingly students were asked "The levels of competence you have specified above may have resulted from formal or informal university structures and procedures, external (or outside university) structures or procedures or maturation. For each dimension please indicate the "most important" influence (1) and the "least important" influence (4)." Students were not only able to indicate the most and least important sources of their development; approximately 30% of respondents estimated from most important to least important, or in other words ranked all four sources of possible development. 60 Frederick T. Evers and Sid N. Gilbert Table 1 CEASE Project: Skill Competency Levels (N = 800) Mean 1 Rank 2 Thinking and Reasoning Skills Problem-Solving Skills 4.03 3.83 (2) Decision-Making Skills 3.82 3.94 3.46 Competency or Performance Level Planning and Organizing Skills Time-Management Skills Communication Skills (5) (6) Quantitative/Mathematical Skills 3.91 3.24 (3) (9) (7) (4) (10) Independence Supervisory Skills 4.36 3.76 (1) (8) Interpersonal and Social Skills 1 2 3.80 Students were asked: "How do you rate your level of competence or performance in e a c h of t h e f o l l o w i n g : . " R e s p o n s e c a t e g o r i e s w e r e : 5 = E x t r e m e l y h i g h c o m p e t e n c e ; 4 = Very high c o m p e t e n c e ; 3 = S o m e c o m p e t e n c e ; 2 = Little competence; 1 = Very low competence Ranked in descending order, with 1 representing the category with the highest mean perceived competency level The results are displayed in Table 2 and summarized in Table 3. Formal university education is considered to be the most important source for development of thinking and reasoning skills, problem solving skills, planning and organizing skills, time management skills and quantitative/ mathematical skills. Formal university education, on the other hand, is considered to be the least important source of decision making, communication, interpersonal and social, supervisory, and independence. Students consider communication skills to have been developed due to the informal structures and processes involved in university life as opposed to the formal aspects of learning. Students also indicate that interpersonal and social skills and supervisory skills are produced through external or outside of university sources. Decision making skills and independence were felt to have emerged simply on account of maturation. Outcomes Assessment: How Much Value Does University Education Add? 61 Table 2 CEASE Project: Sources of Skill Competencies 1 fN = 800Ì University Formal Informal Outside University Maturation Most Important Least Important Solving Skills Most Important 51.4 11.0 16.6 24.4 13.9 40.2 16.5 14.1 57.7 14.1 14.2 10.0 % Least Important Decision-Making Skills 10.0 24.1 32.5 22.9 % Most Important % Least Important Planning and Organizing Skills % Most Important 17.2 27.1 17.0 26.4 30.7 20.6 31.9 15.0 35.1 21.6 23.1 16.0 % Least Important Time Management Skills % Most Important 16.4 21.2 26.0 25.0 40.6 21.0 16.9 19.5 % Least Important Communication Skills 18.4 18.9 25.2 26.2 % Most Important % Least Important Interpersonal and Social Skills 25.6 33.7 28.5 15.0 28.7 21.2 18.1 17.6 % Most Important % Least Important Quantitative/Mathematical Skills % Most Important % Least Important Independence % Most Important % Least Important Supervisory Skills % Most Important % Least Important 5.6 65.2 31.4 9.9 37.6 7.4 25.0 8.0 70.4 6.9 4.2 17.4 15.4 22.1 4.4 47.7 7.0 57.5 13.9 17.4 28.0 9.0 53.4 5.2 13.2 44.4 17.1 20.4 47.1 12.5 18.9 39.8 Competency Thinking and Reasoning Skills % % Problem % 1 Students were asked: "The levels of competence you have specified above may have resulted from formal or informal university structures and procedures, external (or outside university) structures and procedures, or maturation. For each dimension, please indicate the "most important' influence (1) and the 'least important' influence (4) 62 Frederick T. Evers and Sid N. Gilbert Table 3 CEASE Project: Summary of Sources of Skill Competencies (N = 800) University Competence Formal Informal Outside University Maturation Thinking and Reasoning 1 4 Problem-Solving 1 4 Decision-Making 4 Planning and Organizing 1 Time Management 1 Communication 4 Interpersonal and Social 4 Quantitative/Mathematical 1 4 Independence 4 1 Supervisory 4 4 1 4 4 4 1 1 1 What about the broader intellectual goals and consequences of the university experience? In the CEASE Project, students were asked to indicate their development on the University of Guelph's stated Learning Objectives, namely: literacy: reading, writing and oral communication skills; numeracy: quantitative or c o m p u t a t i o n a l skills; sense of h i s t o r i c a l d e v e l o p m e n t / h i s t o r i c a l consciousness; independence of thought; desire to continue learning; creativity; global understanding: a sense of wider international and cultural contexts; moral maturity: an understanding of moral and ethical choices; aesthetic maturity: acquaintance with literature and the arts; understanding of forms of inquiry: an appreciation of science and other methods of inquiry and their limitations; and, depth and breadth of understanding: substantive in depth knowledge of a field of study. Students considered their development, as a result of their education at the University, to be very high on the following: independence of thought (highest), Outcomes Assessment: How Much Value Does University Education Add? 63 d e s i r e to c o n t i n u e l e a r n i n g , d e p t h of u n d e r s t a n d i n g a n d l o w e r o n a e s t h e t i c maturity (lowest), historical consciousness, numeracy, creativity and global u n d e r s t a n d i n g (Table 4). Apparently, university e d u c a t i o n d o e s better with the c o n c e p t u a l / a n a l y t i c a l , s u b s t a n t i v e and m o t i v a t i o n a l a s p e c t s of d e v e l o p m e n t than with b r o a d e r artistic, cultural, historical sensitivities and with quantitative or c o m p u t a t i o n a l skills. T h e latter aspect is e x p e c t e d to vary greatly b y p r o g r a m of study, as w e shall see later in the paper. Table 4 C E A S E Project: A t t a i n m e n t of L e a r n i n g O b j e c t i v e s ( N = 800) Desired Characteristic Mean 1 Literacy: Reading Skills 3.42 Writing Skills 3.66 Oral Communication Skills 3.69 Numeracy: Quantitative or Computational Skills 3.18 Sense of Historical Development/Historical Consciousness 2.82 Independence of Thought 3.95 Desire to Continue Learning 3.88 Creativity 3.25 Global Understanding: A Sense of Wider International and Cultural Contexts 3.44 Moral Maturity: An Understanding of Moral and Ethical Choices 3.57 Aesthetic Maturity: Acquaintance with Literature and the Arts Understanding of Forms of Inquiry: An Appreciation of Science and Other Methods of Inquiry and Their Limitations 2.74 3.65 Depth and Breadth of Understanding: Substantive In-Depth Knowledge of a Field of Study 1 3.82 Students were asked: "Below is a set of desired characteristics of educated graduates, used in part to guide educators in their development of courses and programs. How much would you say you have developed these characteristics as a result of your education at the University of G u e l p h ? " R e s p o n s e c a t e g o r i e s were: 5 = Greatly; 4 = Very Much; 3 = Somewhat; 2 = Hardly; 1 = Not at all 64 Frederick T. Evers and Sid N. Gilbert Similarly, students were asked "How much has your university experience contributed to your personal development in a number of areas. Generally, the ' s o f t e r ' aspects were not recognized as being fostered by the university experience, namely: caring for others, social and political awareness, concern for others, ability to establish relationships and dealing with conflict (Table 5). Table 5 C E A S E P r o j e c t : C o n t r i b u t i o n of U n i v e r s i t y E x p e r i e n c e to P e r s o n a l Development (N = 800") Area of Personal Development Mean 1 Self-confidence 3.90 Motivation 3.65 Ability to Handle Stress 3.70 Ability to Deal with Conflict 3.55 Ability to Understand Others 3.81 Responsibility 3.85 Social Skills 3.66 Social and Political Awareness 3.31 Concern for Others 3.32 Caring for Others 3.21 Ability to Establish Relationships 3.47 1 Students were asked:"How much has your university experience contributed to your personal development in the following areas:?" Response categories were: 5 = Greatly; 4 = Very Much; 3 = Somewhat; 2 = Hardly; 1 = Not at all Findings: Making the Match Project Skill Levels and Source of Development The skill competency levels are based on more than 60 items evaluated on a scale of 1 ("Very High Competence") to 5 ("Very Low Competence") and then condensed into summary measures of competence representing 18 broad skill areas for graduates and 17 broad skill areas for students. The skill composites are listed in Tables 6 through 12.1 The sources were obtained by including a question with the competency question to ascertain the source that the respondents felt was the 'most useful' in helping them to develop the skill. Students were given the following choices: parents and peers, primary and secondary school, community college, university courses, extra-curricular Outcomes Assessment: How Much Value Does University Education Add? 65 activities, work experience, and other. It should be noted that these sources, especially community college, are not applicable to all students, but the intention was to cover all possibilities. Graduates were presented the following choices: before university, university courses, extra-curricular university activities, on-the-job experience, formal job training, and other. 2 In this paper we are p r i m a r i l y i n t e r e s t e d in u n i v e r s i t y c o u r s e s as a s o u r c e of skill development. The means of the competency composites and the percentages by source are presented in Table 6 for students and Table 7 for graduates. The students and graduates have a generally positive attitude about their skill levels, with the average competence ratings falling between "3" (Average) and "2" (High) for each skill area examined. For both the graduates and the students, interpersonal skills, personal organization/time management, and personal strengths 3 are among the skills with the highest perceived competence levels. Both the graduates and the students consider risk-taking as a skill in which they are relatively less competent. In addition, students feel that their technical and oral communication skills are least developed while graduates feel that their competence levels in visioning 4 , managing conflict, and leadership ability are among the least developed. Turning to the source of skill development, university courses are the major contributor for 9 of the 17 skills presented to the students. In particular, university courses are the primary source for the ability to conceptualize, technical, problem—solving/analytical, personal time management, and learning, with university courses being rated as the most useful source by more than 40% of the students for each of these five areas. University courses did not rank as the primary source of development for risk-taking, listening, interpersonal, managing conflict, leadership/ influence, coordinating, creativity/ innovation, and personal strengths. The primary source for listening, interpersonal, managing conflict, and personal strengths was parents and peers. The primary source for risk-taking, leadership/ influence, coordinating, and creativity/ innovation was students' work experience. Work experience even accounted for a surprisingly high proportion of students' written communication skills. For the graduates, the university courses category was rated by the majority for only one of the 18 skills: conceptualizing. On-the-job experience comes through as a predominant force as a source of skill development. Although university courses are an important aspect of many of these skills such as problem-solving, oral communication, and written communication, a larger percentage of graduates list on-the-job experience as the most useful source. Table 6 M T M - Year 1 - Students f N = 1548) Source of Skill Development Percentages Averaged Across Items ^ Primary/ Composite Mean Competency^ Parents &Peers Secondary School Extra Community College University Courses curricular Activities Work Experience Problem-Solving 2.31 11.2 15.9 0.7 47.5 4.5 16.9 Decision-Making 2.46 30.3 6.2 0.3 30.6 6.1 20.2 30.7 Planning & Organizing 2.41 9.2 9.1 0.6 34.2 11.7 Time Management 2.17 15.4 11.1 0.5 44.3 5.4 18.7 Risk-Taking 2.48 18.9 3.7 0.3 22.9 5.8 41.3 Oral C o m m u n i c a t i o n 2.52 16.3 22.4 1.0 29.3 11.1 16.2 Written C o m m u n i c a t i o n 2.42 5.4 18.2 0.9 35.5 2.5 35.2 Listening 2.19 35.0 14.7 0.5 29.1 5.9 11.2 Interpersonal 2.05 37.0 9.8 0.4 9.4 8.5 30.8 Managing Conflict 2.36 44.8 5.5 0.6 10.8 10.9 21.2 Leadership/Influence 2.37 13.4 9.3 0.5 12.6 18.5 42.4 Coordinating 2.46 11.7 10.3 0.5 24.4 18.2 32.3 Creativity/Innovation 2.45 13.8 9.2 0.4 28.5 7.7 34.0 Conceptulization 2.41 3.0 11.0 0.7 69.0 2.1 11.2 Learning 2.46 15.1 2.9 0.3 41.9 5.1 23.3 Personal Strengths 2.27 34.1 8.0 0.4 22.8 9.1 16.8 Technical 2.74 4.2 11.1 0.8 52.9 1.8 24.6 1 Percentages are based on an a v e r a g e for the items within each scale. The "other" category is not included, therefore percentages d o not total 100 2 Higher m e a n s imply lower levels of perceived competence Table 7 MTM - Year 1 - Graduates (N = 1873) Source of Skill Development Percentages Averaged Across Items ' ExtraMean Composite Before University curricular On-the-Job Formal J o b Competency^ University Courses Activities Experience Training Problem-Solving 2.17 10.1 29.8 3.5 49.0 4.8 Decision-Making 2.43 10.1 17.2 4.0 59.5 3.4 Planning & Organizing 2.37 6.2 16.5 4.1 64.7 6.0 Time Management 2.19 14.2 22.0 3.0 50.6 6.2 Risk-Taking 2.48 8.1 12.2 2.9 69.4 3.1 Oral C o m m u n i c a t i o n 2.33 18.4 22.9 10.0 33.4 9.2 Written C o m m u n i c a t i o n 2.23 11.6 32.1 1.2 46.5 6.1 Listening 2.28 30.7 18.4 5.6 31.5 6.2 Interpersonal 2.09 29.0 4.8 9.3 45.7 2.7 M a n a g i n g Conflict 2.60 18.8 7.1 9.1 49.1 6.0 Leadership/Influence 2.48 7.8 7.2 71.2 4.8 Coordinating 2.47 5.5 7.0 6.3 75.6 2.4 Creativity/Innovation 2.43 12.0 14.3 3.1 59.4 4.3 Visioning 2.90 5.0 15.9 1.4 64.7 3.0 Conceptualization 2.30 9.1 46.7 1.6 37.2 1.3 Learning 2.29 10.0 16.3 2.2 55.9 4.9 Personal Strengths 2.15 34.5 14.6 6.9 29.2 1.3 Technical 2.30 1.9 36.0 0.8 47.3 10.1 4.8 1 Percentages are based on an average for the items within each scale. T h e " o t h e r " category is not included, therefore percentages d o not total 100 2 Higher m e a n s imply lower levels of perceived competence 68 Frederick T. Evers and Sid N. Gilbert For example, 49% of graduates specified on-the-job experience as the most useful source of development for problem-solving/ analytical as opposed to 30% who indicated university courses. Similarly, 60% versus 17% specified on-the-job experience over university courses for decision-making skills; 65% of the graduates versus 17% gave o n - t h e - j o b experience over university courses for planning and organizing; 71% versus 5% specified o n - t h e - j o b experience as opposed to university courses for leadership/ influence; and 59% versus 14% selected o n - t h e - j o b experience over university courses for creativity/innovation. Skill Levels by Career Stage Grouping the students and graduates on the basis of their career stage uncovers some interesting aspects of the skill development process within the university and corporate settings. The pre-graduates in the university sample rated their competence either as high as or higher than the early university students for all of the skill and ability areas we examined. This suggests that there is some value added during the university years in terms of the skill areas we are tracking. Fewer differences exist among the three graduate employee groups in the corporate sample. The older graduate cohorts (those who have been out of u n i v e r s i t y f o r a l o n g e r p e r i o d of t i m e a n d t e n d to b e f u r t h e r a l o n g in t h e i r careers) do not always feel more positive about their competence in comparison to the younger graduate cohorts. For example, the graduates who have been out of university for a longer period of time have lower opinions regarding their competence in technical areas compared to those who graduated more recently. On the other hand, they have more positive opinions regarding their ability to solve problems/ analyze, make decisions, take risks, and be creative/innovative. These findings suggest that some of the skills are sufficiently developed regardless of career stage, while others need to be improved during certain career stages more than others (Table 8). Table 8 MTM - Year 1 - Students (N=1548) and Graduates (N=1873) Skill Competence Composite Means 1 by Cohort 2 Students Composite Problem-Solving Decision-Making Planning & Organizing Graduates 3 4 5 ** 2.27 2.15 2.13 ** 2.43 ** 2.42 2.40 ** 2.38 * 2.52 2.41 2.37 2.36 1 2 2.39 2.27 2.52 2.45 Outcomes Assessment: How Much Value Does University Education Add? 69 Table 8, continued Students Composite Time Management 1 2.26 Risk Taking 2.48 Oral Communication Written Communication Listening Interpersonal 2.60 2.53 2.22 2.10 Managing Conflict Leadership/Influence Coordinating Creativity/Innovation 2.36 2.38 2 2.15 2.48 ** 2.50 2.37 * 2.18 2.03 2.36 2.47 2.32 ** 2.28 * 2.26 2.05 2.62 2.20 2.25 2.08 2.36 2.45 2.44 2.53 2.53 2.54 2.48 2.38 2.96 2.33 2.50 2.34 2.44 2.24 2.81 2.70 2.49 2.47 Learning Personal Strengths Technical Graduates 4 5 2.20 2.22 2.58 2.34 Visioning Conceptualization 3 2.15 ** 2.59 2.44 2.45 2.33 2.23 2.33 2.13 2.62 2.43 2.40 2.90 2.51 2.49 2.41 2.86 * 2.28 2.29 2.30 2.15 2.21 2.13 2.27 2.19 2.39 ** 2.31 2.27 ** ** ** * Statistically significantly different means at the .05 alpha level (t-test for students and F-test for graduates). ** .01 alpha level 1 Higher means imply lower levels of perceived competence. 2 Cohorts: l=Early University, 2=Pre-Graduate, 3=Job Entry (0-1 year), 4=Job Change (2-6 years), and 5=Stabilized Career (7 years or more) Skill Levels by Program and Degree C o m p a r e d to s t u d e n t s c u r r e n t l y e n r o l l e d in b u s i n e s s or e n g i n e e r i n g undergraduate programs, those in arts programs (i.e., arts, social sciences and humanities) have more positive views of their oral communication, listening and interpersonal skills, and their ability to manage conflict. Arts students also express higher levels of competence than business students in decision making and written communication skills. Students enrolled in engineering programs, however, perceive their problem solving, risk taking, creativity, and technical skills more positively than either arts or business students (Table 9). 70 Frederick T. Evers and Sid N. Gilbert Table 9 M T M - Year 1 - Students ( N = 1 5 4 8 ) Skill C o m p e t e n c e C o m p o s i t e M e a n s 1 Arts/ Business Engineer Other Problem-Solving 2.36 2.34 2.18 2.34 ** Decision-Making 2.40 2.52 2.45 2.58 ** Composite Soc. Sci. F-test) Planning & Organizing 2.42 2.40 2.37 2.46 Time Management 2.25 2.12 2.19 2.16 Risk Taking 2.51 2.50 2.39 2.47 * Oral Communication 2.41 2.61 2.58 2.52 ** Written Communication 2.34 2.47 2.43 2.56 ** Listening 2.04 2.29 2.25 2.30 ** Interpersonal 1.94 2.15 2.09 2.07 ** Managing Conflict 2.26 2.42 2.37 2.52 ** Leadership/Influence 2.35 2.34 2.39 2.45 Coordinating 2.48 2.43 2.44 2.57 Creativity/Innovation 2.44 2.56 2.2 T 2.57 ** Conceptualization 2.36 2.54 2.29 2.50 ** Learning 2.42 2.46 2.46 2.55 Personal Strengths 2.29 2.27 2.21 2.35 Technical 3.30 2.65 2.17 2.66 ** * Statistically significantly different means at the .05 alpha level ** .01 alpha level 1 Higher means imply lower levels of perceived competence Table 10 M T M - Year 1 - G r a d u a t e s ( N = 18731 Skill C o m p e t e n c e C o m p o s i t e M e a n s 1 b y Bachelor Degree Arts/ Composite Soc. Sci. Problem-Solving Decision-Making Math/ Business Engineer Science Other (F-test) 2.13 2.20 2.16 0.163 2.27 * 2.36 2.42 2.46 2.52 2.45 ** Planning & Organizing 2.33 2.34 2.41 2.43 2.37 Time Management 2.09 2.15 2.27 2.30 2.20 Risk Taking 2.47 2.50 2.47 2.50 2.53 ** Outcomes Assessment: How Much Value Does University Education Add? 71 Table 10, continued Business Engineer Math/ Science 2.38 2.35 2.39 2.29 * Written Communication 2.05 2.26 2.25 2.40 2.15 ** Listening 2.16 2.26 2.35 2.42 2.16 ** Interpersonal 2.01 2.07 2.14 2.16 2.02 * Managing Conflict 2.50 2.56 2.67 2.72 2.65 ** Leadership/Influence 2.45 2.46 2.48 2.56 2.58 Coordinating 2.39 2.47 2.46 2.61 2.46 * * Composite Arts/ Soc. Sei. Oral Communication 2.22 Other (F-test) Creativity/Innovation 2.39 2.49 2.39 2.45 2.42 Visioning 2.92 2.91 2.84 2.92 3.01 Conceptualization 2.15 2.41 2.28 2.30 2.46 Learning 2.22 2.31 2.29 2.31 2.31 Personal Strengths 2.14 2.13 2.14 2.23 2.17 Technical 2.46 2.45 2.17 1.91 2.42 ** ** * Statistically significantly different means at the .05 alpha level ** .01 alpha level 1 Higher means imply lower levels of perceived competence For comparison purposes, the graduate employees are categorized by the field of study of their undergraduate degree 5 . The arts group has the lowest opinion regarding their technical skill, while the engineering and math/science groups have the highest (Table 10). Ignoring the ranking of the competence scores and comparing the actual scores for each of the skills, there is a tendency for the arts graduate group to have slightly higher s e l f - a s s e s s m e n t s of decision-making, personal organization/time management, oral communication, written communication, listening, interpersonal skills, conflict management, coordinating, and the ability to conceptualize. University Courses as a Source of Skill Development by Program and Degree Tables 11 and 12 contain the percentages for university courses as a source of skill by program and degree. Students from business programs list university courses as the most useful source more often than those in engineering and arts/social science programs for many of the skills. The differences are not as great among the graduate respondents. 72 Frederick T. Evers and Sid N. Gilbert Table 11 M T M - Year 1 - S t u d e n t s ( N = 1 5 4 8 ) U n i v e r s i t y C o u r s e s as the Skill D e v e l o p m e n t S o u r c e by P r o g r a m Percentages Averaged Across Items 1 Composite Arts/ Soc. Sci. n=545 Total n=1548 Business n=466 Engineer n=515 Problem-Solving 47.5 41.0 52.2 52.2 Decision-Making 30.6 21.6 40.3 26.2 Planning & Organizing 34.2 32.8 38.3 33.5 Time Management 44.3 43.7 38.9 51.5 Risk Taking 22.9 18.9 31.2 19.0 Oral Communication 29.3 29.2 32.9 25.6 Written Communication 35.5 33.2 43.9 32.3 Listening 29.1 31.5 28.1 27.3 9.4 7.5 10.5 11.0 Managing Conflict 10.8 9.7 14.8 7.5 Leadership/Influence 12.6 24.4 7.6 14.2 15.6 16.7 33.6 30.4 Creativity/Innovation 28.5 23.2 33.8 31.1 Conceptualizaton 69.0 75.4 66.5 63.9 Learning 41.9 46.9 36.1 42.6 Personal Strengths Technical 52.9 52.9 23.2 37.6 19.7 57.5 27.5 66.7 Interpersonal Coordinating 1 Percentages are based on an average for the items within each scale Table 12 M T M - Year 1 - G r a d u a t e s ( N = 1 8 7 3 ) U n i v e r s i t y C o u r s e s as the Skill Development Source by Bachelor Degree Percentages Averaged Across Items' Composite Total n=1873 Arts/ Soc. Sci. n=437 Problem-Solving 29.8 27.7 31.5 30.7 Decision-Making 17.2 16.2 21.5 16.2 12.1 Planning & Organizing 16.5 15.4 17.8 18.0 13.2 Time Management 22.0 19.2 21.4 24.4 23.3 Risk-Taking 12.2 9.3 17.4 11.5 8.3 Business n=543 Engineer n=515 Math/ Sci. n=273 27.8 Outcomes Assessment: How Much Value Does University Education Add? 73 Table 12, continued Percentages Averaged Across Items 1 Math/ Sei. n=273 Composite Total n=1873 Arts/ Soc. Sei. n=437 Oral Communication 22.9 24.2 32.6 15.9 12.8 Written Communication 32.1 36.1 36.1 29.5 21.7 Listening Business n=543 Engineer n=515 18.4 18.8 22.5 14.3 15.1 Interpersonal 4.8 4.4 6.0 4.7 3.5 Managing Conflict 7.1 7.7 10.8 3.0 5.0 Leadership/Influence 4.8 7.3 6.5 4.3 3.0 Coordinating 7.0 5.3 10.9 6.4 2.9 Creativity/Innovation 14.3 13.7 15.5 14.2 13.5 Visioning 15.9 14.9 24.9 11.1 13.5 Conceptualization 46.7 52.8 48.5 41.1 43.6 Learning 16.3 16.3 13.7 18.3 15.8 Personal Strengths 14.6 15.1 13.6 15.7 14.4 Technical 36.0 22.7 30.4 49.5 41.6 1 Percentages are based on an average for the items within each scale According to both the students and graduates, university plays a major role in the development of the ability to conceptualize. And among both students and graduates, the arts/social science category is the highest. Skills Needed in the Future In order to summarize the MTM results on skill development, let us look at what skills are t h o u g h t to be in g r e a t e s t d e m a n d in f u t u r e c o r p o r a t e employment. When presented with the list of 18 skill areas, the managers participating in the MTM study listed creativity, visioning, and leadership as the top three. It is appropriate to add written communication to this list since the professors surveyed gave it as the most important skill. In a separate survey of 265 companies in Canada, Evers (1990) found that corporate leaders felt that university graduates in their employ were in greatest need of upgrading in the areas of written communication, visioning, and leadership. These findings are also consistent with Useem (1989) who concludes that liberal arts graduates have become more valued by corporate employers because of communicative skills and their ability to learn. 74 Frederick T. Evers and Sid N. Gilbert Conclusions Data from two independent research projects concerning educational outcomes show that the perceived skill competency level of university students tends to be high on most dimensions. Students, however, rate themselves lower on quantitative/ mathematical skills, supervisory skills, communication skills, risk-taking, leadership, and creativity/ innovation. Formal university instruction is c o n s i d e r e d to be a m a j o r source of development for thinking and reasoning skills, problem solving skills, planning and organizing skills, time management skills, ability to conceptualize, learning skills and quantitative, mathematical and technical skills. Consequently, students consider university study to add value to their skills and abilities in these important areas. Formal university instruction, however, is not considered to be the major source of development for independence, interpersonal and social skills, supervisory skills, risk-taking, managing conflict, leadership/ influence, and creativity/ innovation. When the list of student development outcomes is broadened to include social, cultural, artistic, global, historical, political sensitivities/ awareness, and caring and concern for others, university experiences are not regarded as important influences. When university graduates are asked about the sources of their development, the only dimension where university instruction played the most important role was for "ability to conceptualize.' Consequently, students consider that much less value is added by university instruction for these important aspects of development. It seems appropriate to suggest that, in the future, universities should ensure that their graduates have better communication skills, are more creative and have greater leadership potential. Universities need to avoid developing those characteristics which matter least: rote, petty conformity and narrow competencies. As Ernest Boyer (1987, p. 283) put it so well: Throughout our study we were impressed that what today's college is teaching most successfully is competence — competence in meeting schedules, in gathering information, in responding well on tests, in mastering details of a special field. Today the capacity to deal successfully with discrete problems is highly prized... But technical skill, of whatever kind, leaves open essential questions: Education for what purpose? Competence to what end?... What a monumental mistake it would be if students, Outcomes Assessment: How Much Value Does University Education Add? 75 during the undergraduate years, remained trapped within the organizational grooves and narrow routines to which the academic world sometimes seems excessively devoted. Notes 1 Each measure is the sum of the competence rating for the items listed under a major skill area divided by the number items under that area. Given the rating scale from which the means are derived, a higher mean represents a lower perception of competence. There are f e w e r skill areas for students because they were not asked to rate their competence on visioning ability. Also, the students' coordinating ability was measured by a single item. 2 Clearly skill development is a multi-faceted process, however we felt that it would be instructive to force respondents to select a single category. The results on source of skill development are the average percentage of respondents across the items for each of the scales. For example, referring to Table 6, 11.2% is the average across the eight items that make up the 'problem solving/analytic' composite of the percentage of respondents that gave 'parent & peers' as the most useful source of skill development. 3 'Personal Strengths' are a variety of traits (eg. self-motivation, stress management, dealing with criticism) which help people to deal with day to day pressures. 4 Visioning is defined as the ability to conceptualize the future of company growth. 5 S o m e of the e m p l o y e e s also had g r a d u a t e d e g r e e s , but f o r this analysis the undergraduate degree is used to permit comparison with the students in our sample, all of whom are undergraduates. References A s t i n , A l e x a n d e r W. ( 1 9 8 5 ) . Achieving educational excellence. San F r a n c i s c o : Jossey-Bass. Astin, Alexander W. (1987). Assessment, value-added, and educational excellence. In D.F. Halpern (ed.) Student outcome assessment: What institutions stand to gain (pp. 89-107). San Francisco: Jossey-Bass. Bowen, H. (1977). Investment in learning. San Francisco: Jossey-Bass. Boyer, Ernest L. (1987). College: The undergraduate experience in America. New York: Harper & Row. Cameron, David M. (1987). The framework for managing and financing post-secondary education in Canada. The Forum Papers, National Forum Secretariat. Halifax: The Institute for Research on Public Policy. Evers, Frederick T. (1990). Corporate training in Canada survey: Summary Guelph, Ontario: University of Guelph. of results. 76 Frederick T. Evers and Sid N. Gilbert Fortin, Michele. (1987a). Accessibility to and participation in post-secondary education system in Canada. The Forum Papers, National Forum Secretariat. Halifax: The Institute for Research on Public Policy. Fortin, Michele. (1987b). Accessibility to and participation in post-secondary education. National Forum on Post-Secondary Education, Saskatoon, Saskatchewan Proceedings. Halifax: Institute for Research on Public Policy. Gilbert, Sid. (1989). The forgotten purpose and future promise of university education. Canadian Journal of Community Mental Health 8 , 103-122. Gilbert, Sid N., and Frederick T. Evers. (1989). Accessibility and quality in higher education. Services Industries Journal 9, 44-62. Gilbert, Sid N., Frederick T. Evers, and Marion S. Auger. (1989). University attrition differentiated: Rates and institutional influences. Paper presented at the CSSHE Meetings, Laval University. Rush, James C. and Frederick T. Evers. (1986a). Making the match: Canada's university graduates and corporate employers. Montreal: Corporate-Higher Education Forum. Rush, James C. and Frederick T. Evers. (1986b). Making the match: Canada's university graduates and corporate employers. Business Quarterly 50, 42-47. Rush, James C., Frederick T. Evers, and Jasna A. Krmpotic. (1989). Making the match: Phase II: Year I Report. London, Ontario: The University of Western Ontario. Secretary of State of Canada. (1988). Access to Excellence. Ottawa: Ministry of Supply and Services. Segal, Brian. (1987). Report to the Secretary of State, Canada and The Council of Ministers of Education, Canada on The National Forum on Post-Secondary Education, Saskatoon, Saskatchewan. Ottawa: Institute for Research on Public Policy. Task Force on Post-Secondary Education in Manitoba. (1974). Report of the task force on post-secondary education in Manitoba. Winnipeg: Province of Manitoba. Tinto, Vincent. (1987). Leaving college: Rethinking the causes and cures of student attrition. Chicago: University of Chicago Press. University of Guelph. (1985). Aims and objectives of the University ofGuelph: Toward 2000: Challenges and responses. Guelph, Ontario: University of Guelph. University of Guelph. (1989). Undergraduate Calendar 1989-90. Guelph, Ontario: University of Guelph. Useem, Michael. (1989). Liberal education and the corporation: The hiring advancement of college graduates. New York: Aldine de Gruyter. and