The Canadian Journal of Higher Education La revue canadienne d'enseignement supérieur Volume XXIX, No. 1,1999 pages 111-142 Using Surveys to Measure 'Value Added' in Skills in Four Faculties J. PAUL GRAYSON Institute for Social Research, York University. ABSTRACT S o m e u n i v e r s i t i e s in C a n a d a a n d o t h e r c o u n t r i e s are i n t e r e s t e d in k n o w i n g the v a l u e - a d d e d to skills — e.g., critical and c o m m u n i c a t i o n skills — by the university experience. T h e best w a y of obtaining information on this m a t t e r is through longitudinal analyses with appropriate control g r o u p s that utilize both subjective and objective m e a s u r e s of skills; unfortunately, such designs are costly and studies based on t h e m t a k e a long t i m e to c o m p l e t e . In this article an alternate strategy that involved c o m p a r i n g t h e s k i l l s of e n t e r i n g a n d g r a d u a t i n g s t u d e n t s at Y o r k U n i v e r s i t y is d e s c r i b e d . It is a r g u e d that i n f o r m a t i o n c o l l e c t e d in this f a s h i o n can be of assistance in assessing the value added to specific skills a n d c a n b e u s e d in a d i a g n o s t i c f a s h i o n by f a c u l t i e s c o n c e r n e d w i t h skills d e v e l o p m e n t . RÉSUMÉ C e r t a i n e s universités au C a n a d a et d a n s d ' a u t r e s p a y s sont intéressées à c o n n a î t r e les c o m p é t e n c e s à « v a l e u r a j o u t é e » , p a r e x e m p l e , les c o m p é t e n c e s a n a l y s t i q u e s et c o m m u n i c a t i v e s , q u e l ' e n s e i g n e m e n t supérieur permet aux étudiants d'acquérir. Pour obtenir des r e n s e i g n e m e n t s à c e sujet, la m e i l l e u r e f a ç o n est de f a i r e d e s a n a l y s e s longitudinales à l ' a i d e des g r o u p e s d e contrôle appropriés en utilisant des 112 J.P. Grayson méthodes subjectives et objectives. Malheuresment, de tels projets sont toujours coûteux et les études qui s'y rapportent prennent beaucoup de t e m p s . D a n s cet article, aune autre stratégie de c o m p a r a i s o n de compétences entre les étudiants qui entrent à l'Université York et ceux qui en s o r t e n t est e x p o s é e en détail. On c o n s t a t e q u e l ' i n f o r m a t i o n recueillie de cette manière peut être utilisée pour l'évaluation des compétences à «valeur ajoutée»ainsi que pour l'analyse de l'acquisition de ces compétences par les facultés qui s'y intéressent. INTRODUCTION An increasing number of Canadian universities are carrying out surveys of their graduates that a m o n g other things ask students to c o m m e n t on d e v e l o p m e n t in various skills, such as analytic and communication skills, over the course of their education. To date, however, f e w if any attempts have been m a d e to measure the value added to skills by the university experience. There are several reasons for this. First, there is no consensus regarding skills that should be developed over the course of a student's university career. Second, even if agreement were reached on which skills should be developed, there is no agreement on ways to measure them. Third, the longitudinal studies required to measure value added in skills are expensive and take a long time to complete. At York University, in order to measure the value added in skills over the course of an undergraduate education, steps were taken to deal with each of the three p r o b l e m s . First, an iterative process was established in which m e m b e r s of the university c o m m u n i t y identified skills that, ideally, would be improved over the course of a university education. Second, survey questions were developed to measure such skills on the part of students entering, and graduating from, four faculties. Third, a research design was used in which skills of entering students were compared to those of graduating students after adjustments had been made through an analysis of covariance for Ontario Academic Credit marks, gender, ethno-racial origin, and language spoken in the home while growing up. This article focuses on the rationale for conducting an analysis of the value added to skills in this relatively cost e f f e c t i v e fashion and the results of the research. The Canadian Journal of Higher Education Volume XXIX, No. 1,1999 Using Surveys to Measure 'Value Added' in Skill in Four Faculties 113 BACKGROUND TO RESEARCH In a r e v i e w of exit s u r v e y s used in C a n a d i a n universities, E v e r s and O ' H a r a ( 1 9 9 6 ) d e f i n e k n o w l e d g e as, " T h e u n d e r s t a n d i n g of a b o d y of i n f o r m a t i o n in a p a r t i c u l a r f i e l d ; f o r e x a m p l e , electrical e n g i n e e r i n g . " Skills are seen as, " T h e abilities or p r o f i c i e n c i e s d e v e l o p e d in certain areas; f o r e x a m p l e , written c o m m u n i c a t i o n . " Finally, values are d e f i n e d as, " T h e c o m m o n l y held positive attitudes t o w a r d an abstract c o n c e p t ; f o r e x a m p l e , respect f o r d i v e r s i t y " (p. 44). E v e r s and O ' H a r a n o t e that w h i l e a g r o w i n g n u m b e r of C a n a d i a n universities use self-reports in exit surveys to m e a s u r e their g r a d u a t e s ' k n o w l e d g e , skills, a n d v a l u e s ( K S V ) , " t h e r e is little e v i d e n c e of t h e d e v e l o p m e n t of m o r e sophisticated m e t h o d o l o g i e s such as b e h a v i o u r a l m e a s u r e s or o b j e c t i v e testing of K S V " (p. 54). In addition to d e v i s i n g such m e a s u r e s , like m a n y other researchers, particularly in the U n i t e d S t a t e s , t h e y c o n c l u d e that it is n e c e s s a r y to d e v e l o p i n d i c a t o r s of t h e v a l u e a d d e d by the institution to s t u d e n t s ' k n o w l e d g e , skills, a n d values. T h i s o b j e c t i v e can only be achieved if i n f o r m a t i o n is collected on both e n t e r i n g and g r a d u a t i n g students. T h e r e are a n u m b e r of steps that m u s t be t a k e n if t h e s e d e s i d e r a t a a r e to b e a c h i e v e d . F i r s t , to f o c u s o n l y on s k i l l s , it is n e c e s s a r y to d e v e l o p a g r e e m e n t r e g a r d i n g the t y p e s of skills that s h o u l d b e f o s t e r e d d u r i n g an u n d e r g r a d u a t e career. A s a t t e m p t s to a c h i e v e this o b j e c t i v e at t h e n a t i o n a l level h a v e b e e n less than s u c c e s s f u l in the U n i t e d S t a t e s ( J o n e s , 1994; Pike, 1995a), it is highly unlikely that w e will h a v e any m o r e luck north of the border. A s a result, it m a k e s s e n s e f o r i n d i v i d u a l u n i v e r s i t i e s to d e c i d e f o r t h e m s e l v e s the skills with w h i c h they are m o s t c o n c e r n e d . If this can b e d o n e in c o n j u n c t i o n with other universities, so m u c h the better. O n c e i n d i v i d u a l i n s t i t u t i o n s r e a c h s o m e f o r m of a g r e e m e n t o n w h i c h skills s h o u l d b e d e v e l o p e d , the s e c o n d s t e p is to f i n d w a y s to m e a s u r e t h e m . In the c r o s s - s e c t i o n a l exit s u r v e y s e x a m i n e d by E v e r s a n d O ' H a r a , s t u d e n t s are t y p i c a l l y a s k e d to s e l f - r e p o r t on h o w m u c h u n i v e r s i t y e x p e r i e n c e s h a v e i n c r e a s e d skills, such as c o m m u n i c a t i o n or i n t e r p e r s o n a l skills. A s well as u s i n g s e l f - r e p o r t s to m e a s u r e skills, s t a n d a r d i z e d tests, such as the C a l i f o r n i a Critical T h i n k i n g Skills Test, the W a t s o n - G l a s e r Critical T h i n k i n g A p p r a i s a l , the E n n i s Weir T h i n k i n g E s s a y Test, and t h e C o r n e l l C r i t i c a l T h i n k i n g T e s t , c a n b e u s e d to m e a s u r e s k i l l s ; The Canadian Journal ofHigher Education Volume XXIX, No. 1, 1999 114 J. P. Grayson however, in many instances users must pay to utilize instruments such as these. Moreover, incentives must frequently be given to students to complete lengthy test m o d u l e s . F o r e x a m p l e , P a s c a r e l l a and A s s o c i a t e s (1995) report that in their longitudinal study of university outcomes in the United States they started by paying students $25 U.S. to complete tests and questionnaires and that the stipend increased by $5 each time additional tests were taken. Even if standardized tests are used, questions can be raised regarding their validity, particularly for measuring shortterm gains in skills (Jacobs, 1995; McMillan, 1987). Independent of whether or not self-reports in surveys or standardized tests are used to measure skills, when examining how much students' experiences in any given university contribute to the development of skills or any other desired outcome, it is important to obtain measures of the outcome under consideration at entry and again at graduation so that the value added can be assessed (Astin, 1991). Similar measures should be taken of a same-age control group that does not attend university. Increases in skills between entering and graduating students that are not also observed in the control group can be attributed to the university experience. In longitudinal research such as this, because many students might leave the university or refuse to participate in both the entry and exit measurement of skills, large samples are required to ensure that at graduation sufficient numbers are still involved in the study to facilitate meani n g f u l a n a l y s i s . E v e n if a l a r g e s a m p l e r e m a i n s at g r a d u a t i o n the possibility exists that non-participants are different f r o m students who remain in the study. In addition, control group attrition is a potential problem. Also, when relying on longitudinal studies, institutions must wait until a c o h o r t has m a d e its way through the university s y s t e m before a measure of value added is available. A COST EFFECTIVE ALTERNATIVE Clearly, identifying and measuring skills, and determining the value added by the university experience to skills in a longitudinal study is a complicated, potentially costly, and time consuming process. There is, h o w e v e r , a w a y to d e c r e a s e the cost and a m o u n t of time n e e d e d to obtain i n f o r m a t i o n on the v a l u e added c o m p o n e n t of the university experience. Subject to qualifications regarding question wording to be discussed later, this approach involves using surveys in which entering and graduating students provide self-assessments of skills in particular The Canadian Journal of Higher Education Volume XXIX, No. 1,1999 Using Surveys to Measure 'Value Added' in Skill in Four Faculties 115 areas. T h r o u g h an analysis of covariance, after controlling f o r variables h a v i n g the potential to i n f l u e n c e skills acquisition, such as p r e v i o u s levels of a c h i e v e m e n t (high school marks), gender, l a n g u a g e spoken in the h o m e , a n d e t h n o - r a c i a l origin, c o m p a r i s o n s b e t w e e n the g r o u p s can then be m a d e on various m e a s u r e s of skills. If c o m p a r i s o n s s h o w that g r a d u a t i n g students h a v e skills not evident a m o n g entering students and a s a m e - a g e control group, d i f f e r e n c e s b e t w e e n the entering and graduating students can be attributed to the university experience. Variations of this m e t h o d have been used s u c c e s s f u l l y by Keeley, B r o w n e and K r e u t z e r ( 1 9 8 2 ) and Steele (1986). O n e p r o b l e m with this approach is that when they entered university, the graduating students may have been different f r o m the entering g r o u p with w h o m they are being compared. For example, if in recent years an increased e m p h a s i s had been placed on certain skills in high schools, entering students would demonstrate higher levels of skills than would have been displayed by graduating students when they were in first year. A second problem is that students with certain characteristics, low motivation f o r e x a m p l e , m a y leave the university b e f o r e c o m p l e t i n g their studies. A s a result, g r a d u a t i n g students m a y be d i f f e r e n t f r o m t h o s e entering the university in ways that are hard to detect and control. Despite these potential difficulties, Pascarella and Terenzini (1991) point out that in studies carried out on reflective judgment, longitudinal and cross-sectional research produce similar results. A similar conclusion was reached by P a c e (1979). A s a result, provided that adjustments are m a d e for possible c o n f o u n d i n g i n f l u e n c e s , cross-sectional studies involving entering and graduating students can be used by institutions in assessing the valued added in particular skills. A l t h o u g h b o t h longitudinal and cross-sectional studies require an external control g r o u p if the possible effects of the university experience are to be disentangled f r o m those of maturation, external control groups are most o f t e n absent f r o m studies of university o u t c o m e s . W h i l e this presents difficulties if the intent is to assess the effect of the university e x p e r i e n c e per se, provided s a m e - a g e groups are under consideration, it is less p r o b l e m a t i c if the r e s e a r c h o b j e c t i v e is o n e of a s s e s s i n g t h e impact of different institutional contexts on outcomes. For example, if the o u t c o m e s of o n e university, faculty, or d e p a r t m e n t are being c o m pared with t h o s e of other universities, faculties, or departments, provided that s a m e - a g e g r o u p s are involved in the study and that pre-entry characteristics are held constant, an external control group is not required. The CanadianJournalof Higher Education Volume XXIX, No. 1, 1999 116 J.P.Grayson SELF-REPORTS Self-reports are an integral part of the relatively more cost and time effective response to measuring valued added in generic skills currently under discussion; however, some may view the use of techniques such as this as less desirable than more objective measures of skills. As Pike (1995a) points out, however, in the United States, the National Education Goals Panel Resources Group on Adult Literacy and Lifelong Learning noted that the development of assessment tools of use at the national level to measure university outcomes would cost several million dollars and take many years to complete. In the interim the Group recommended that self-reports be used as proxies. To what extent can self-reports be viewed as reasonable proxies of k n o w l e d g e and/or skills, and/or values? After examining a number of studies in which the results of self-reports were compared to test results, Pascarella and Terenzini (1991) conclude that correlations between selfreports and other measures of the same phenomena range f r o m .25 to .65. M o r e importantly, the literature they draw upon in their examination indicates that the validity of self-reports varies with the specific k n o w l e d g e and/or skills, and/or values under discussion. For example, Berdie (1971) found a high correlation between self-professed knowle d g e of p u b l i c f i g u r e s and the results of tests d e s i g n e d to m e a s u r e k n o w l e d g e of the same figures; however, the relationships between selfreported and tested knowledge of authors and artists were not as high. Similarly, P o h l m a n n and Beggs (1974) discovered that self-reports of academic growth in the affective realm were supported by test results. Growth in simple and complex cognitive realms, however, did not correlate highly with self-reports. In a review of the literature on the utility of self-reports, Baird (1976) cites examples of both high and low relat i o n s h i p s with external measures of various p h e n o m e n a . Still others h a v e p r e s e n t e d i n f o r m a t i o n s u g g e s t i n g that w h i l e s e l f - r e p o r t s h a v e some uses, they should not be viewed as substitutes for other measures of various college or university outcomes (Dumont & Troelstrup, 1980; M c M o r r i s & Ambrosino, 1973). Overall, despite their limitations and inconsistencies in the research, Kuh and Associates (1997) report that self-reports are valid w h e n : r e s p o n d e n t s know the information requested; questions are clear and unambiguous; and respondents treat questions seriously. The Canadian Journal ofHigher Education Volume XXIX, No. 1,1999 Using Surveys to Measure 'Value Added' in Skill in Four Faculties 117 In a study c o n d u c t e d by P i k e ( 1 9 9 5 a ) c o n c l u s i o n s w e r e r e a c h e d that s u p p o r t t h e v a l i d i t y of t h e s e c o n d c o n d i t i o n of K u h a n d A s s o c i a t e s . B r i e f l y , P i k e s t u d i e d 1,568 g r a d u a t i n g s t u d e n t s f r o m ten c o l l e g e s a n d u n i v e r s i t i e s in t h e U n i t e d S t a t e s w h o c o m p l e t e d t h e C o l l e g e B a s i c Academic Subjects Examination (College B A S E ) adapted for graduating students. T h e C o l l e g e B A S E tests p r o f i c i e n c y in English, m a t h e m a t i c s , s c i e n c e , a n d s o c i a l s c i e n c e . S t u d e n t s w h o c o m p l e t e d the e x a m i n a t i o n also participated in a survey in w h i c h they rated their ability on exactly the s a m e m a t t e r s c o v e r e d in the e x a m i n a t i o n . T h e correlations b e t w e e n test p e r f o r m a n c e a n d self-reports w e r e sufficiently high f o r P i k e to c o n c l u d e that self-reports can be used as general p r o x i e s f o r traditional m e a s u r e s of a c a d e m i c a c h i e v e m e n t . N o n e t h e l e s s , in an earlier p u b l i c a t i o n P i k e ( 1 9 9 5 b ) n o t e d that self-reports w e r e g o o d p r o x i e s f o r p r o f i c i e n c y in m a t h e m a t i c s , r e a s o n a b l y w e a k proxies f o r English and science c o m p e tency, a n d very w e a k indicators of p r o f i c i e n c y in social studies. A s a general rule P i k e e m p h a s i z e s that if self-reports are to be valid m e a s u r e s of, in his case, k n o w l e d g e m e a s u r e d in standardized tests, there must be a high content correspondence between the self-report questions and those asked in tests. ( W h i l e t h i s m a y s e e m s e l f - e v i d e n t , o t h e r research has f o c u s e d on the relationship b e t w e e n s o m e general self-report m e a s u r e s and the results of specific tests.) If this rule is not f o l l o w e d , it m a y be d i f f i c u l t to specify exactly w h a t self-reports measure. A l t h o u g h this a n d other w o r k by P i k e (1994, 1995b) is e n c o u r a g i n g , it s h o u l d b e stressed that the f o c u s of his research has been on k n o w l e d g e ( n o t s k i l l s ) as m e a s u r e d t h r o u g h t h e C o l l e g e B A S E . A l t h o u g h r e s e a r c h to b e a n a l y s e d in this r e p o r t a s s u m e s a s i m i l a r r e l a t i o n s h i p b e t w e e n s e l f - r e p o r t e d skills and the results of tests d e s i g n e d to m e a s u r e such skills, r e s e a r c h is n e c e s s a r y to c o n f i r m the c o n n e c t i o n . A n e n c o u r a g i n g step in this direction has b e e n taken by E v e r s and A s s o c i a t e s ( 1 9 9 3 ) in a study in which they c o m p a r e d the self-reported skill levels of r e c e n t g r a d u a t e s of a n u m b e r of C a n a d i a n universities to a s s e s s m e n t s of the s a m e skills by their e m p l o y e r s . T h e a v e r a g e correlation f o r statistically s i g n i f i c a n t r e l a t i o n s h i p s b e t w e e n s e l f - r e p o r t s and e m p l o y e r s ' a s s e s s m e n t s w a s .16. T h e correlation w a s highest f o r written c o m m u n i c a t i o n skills, .26, and lowest f o r d e c i s i o n m a k i n g skills, .07. A n e x a m i n a t i o n of d a t a d e r i v e d f r o m The International Adult Literacy Survey (IALS) ( A n o n y m o u s , 1995) jointly undertaken by S t a t i s t i c s C a n a d a and T h e E d u c a t i o n a l T e s t i n g S e r v i c e l o c a t e d in t h e The Canadian Journal of Higher Education Volume XXIX, No. 1, 1999 118 J.P. Grayson U n i t e d States, also sheds light on the relationship b e t w e e n self-reported and o t h e r m e a s u r e s of skills. A s well as o b t a i n i n g s e l f - a s s e s s m e n t s of reading, writing, and quantitative skills, this study required participants to c o m p l e t e a n u m b e r of p e r f o r m a n c e tasks of v a r y i n g levels of difficulty related to p r o s e literacy, d o c u m e n t literacy, and quantitative literacy. F o r C a n a d i a n s a g e d 16 to 25 w h o h a d c o m p l e t e d p o s t s e c o n d a r y e d u c a t i o n (the g r o u p m o s t c o m p a r a b l e to the subjects of the current study) the correlations b e t w e e n self-assessed reading and test m e a s u r e d p r o s e and docu m e n t literacy w e r e .45 and .36 respectively. T h e correlations b e t w e e n s e l f - r e p o r t e d writing skill and test m e a s u r e d p r o s e and d o c u m e n t literacy w e r e .29 a n d .22; however, the last m e n t i o n e d w a s not statistically significant. Finally, there w a s a correlation of .29 b e t w e e n self-assessed quantitative skills a n d quantitative literacy as m e a s u r e d in tests. T h e study b y E v e r s and A s s o c i a t e s and the results of the I A L S survey i n d i c a t e on the basis of C a n a d i a n d a t a that there d o a p p e a r to b e statistically s i g n i f i c a n t correlations b e t w e e n s e l f - a s s e s s m e n t s of skills a n d skills m e a s u r e d in other w a y s . W h i l e the m a g n i t u d e of these correlations is i n s u f f i c i e n t f o r m a k i n g a c a d e m i c d e c i s i o n s about, f o r e x a m p l e , individual u n i v e r s i t y g r a d u a t e s w h o m a y be p r e s u m e d to h a v e m a s t e r e d this or that skill, t h e y are s u f f i c i e n t f o r a n a l y s e s of d i f f e r e n c e s in skill d e v e l o p m e n t a m o n g g r a d u a t e s of d i f f e r e n t d e p a r t m e n t s or universities. THE IMPORTANT SKILLS A l t h o u g h t h e r e h a v e b e e n s o m e a t t e m p t s in C a n a d a to identify the s k i l l s t h a t u n i v e r s i t y g r a d u a t e s s h o u l d a c q u i r e ( C o n f e r e n c e B o a r d of C a n a d a , 1992; E v e r s & R u s h , 1996), there has b e e n no general a c c e p t a n c e of the results of t h e s e e n d e a v o r s . C o n s e q u e n t l y , in this study, a list of d e s i r a b l e skills w a s d e v e l o p e d through an inductive process that c o u l d b e utilized easily in o t h e r u n i v e r s i t i e s . In an iterative p r o c e s s , f a c u l t y m e m b e r s with k n o w l e d g e of skills d e v e l o p m e n t , along with researchers f r o m t h e I n s t i t u t e f o r S o c i a l R e s e a r c h at York University, i d e n t i f i e d a n u m b e r of tasks, t h e p e r f o r m a n c e of w h i c h ideally w o u l d b e i m p r o v e d o v e r t h e c o u r s e of a university career. 1 N e x t , q u e s t i o n s w e r e d e v e l o p e d that f o c u s e d on the d i f f i c u l t y students w o u l d h a v e in c o m p l e t i n g s p e c i f i c tasks. Finally, q u e s t i o n s w e r e g r o u p e d into logical categories. A l t h o u g h in t h e I A L S s t u d y r e a s o n a b l e c o r r e l a t i o n s w e r e f o u n d b e t w e e n s o m e g e n e r a l q u e s t i o n s on skills a n d test m e a s u r e d skills, in k e e p i n g w i t h The Canadian Journal of Higher Education Volume XXIX, No. 1, 1999 Using Surveys to Measure 'Value Added' in Skill in Four Faculties 119 P i k e ' s f i n d i n g noted earlier, tasks, and their related very specific and relevant to students ' questions, were experiences. S u r v e y q u e s t i o n s w e r e tested in a small pilot survey i n v o l v i n g stud e n t s a b o u t to g r a d u a t e f r o m P u r e and A p p l i e d Science, Arts, F i n e Arts, a n d A d m i n i s t r a t i v e S t u d i e s . F o r e a c h q u e s t i o n , on a f i v e p o i n t scale, r e s p o n s e s of 1 indicated a high, and 5 a low, level of d i f f i c u l t y in c o m p l e t i n g the s p e c i f i c task r e f e r e n c e d in the question. O n the basis of the pilot, s o m e q u e s t i o n s w e r e d r o p p e d and a s s e s s m e n t s w e r e m a d e of the reliability of indices c o m p r i s e d of various questions. T h e relevant skill c a t e g o r i e s a n d q u e s t i o n topics used in the c o n s t r u c t i o n of indices are disp l a y e d in T a b l e 1. THE SAMPLE I n f o r m a t i o n f o r the analysis of value a d d e d in generic skills b e t w e e n first y e a r and g r a d u a t i o n w a s collected in t w o surveys c o n d u c t e d in the Fall of 1995 and o n e in the S p r i n g of 1996. In o n e of the Fall 1995 surveys, q u e s t i o n n a i r e s assessing skills w e r e mailed to all students w h o had j u s t e n t e r e d t h e f a c u l t i e s of F i n e A r t s , S c i e n c e , a n d A d m i n i s t r a t i v e S t u d i e s and to a s a m p l e of s t u d e n t s e n t e r i n g t h e F a c u l t y of Arts. T h e n u m b e r s of r e t u r n s w e r e 171, 2 2 3 , 100, a n d 8 1 2 r e s p e c t i v e l y . T h e r e s p o n s e rate w a s 5 5 % . T h e s e c o n d survey carried out in the Fall of 1995 f o c u s e d on all stud e n t s eligible to g r a d u a t e in the Fall c o n v o c a t i o n . In total, 4 8 0 Arts stud e n t s r e t u r n e d c o m p l e t e d q u e s t i o n n a i r e s as did 25, 30, and 6 s t u d e n t s f r o m t h e F a c u l t i e s of F i n e A r t s , S c i e n c e , a n d A d m i n i s t r a t i v e S t u d i e s respectively. L o w n u m b e r s reflect the facts that c o m p a r e d to A r t s other f a c u l t i e s are small and relatively f e w students g r a d u a t e in the Fall. T h e r e s p o n s e rate f o r the survey w a s 5 8 % . T h e s u r v e y c a r r i e d o u t in t h e S p r i n g a n d e a r l y s u m m e r of 1 9 9 6 i n c l u d e d all s t u d e n t s f r o m the s a m e faculties eligible to g r a d u a t e in the J u n e c o n v o c a t i o n . In total, 1,835 A r t s g r a d u a t e s , 180 g r a d u a t e s f r o m F i n e Arts, 165 f r o m S c i e n c e , and 7 0 g r a d u a t e s f r o m the Schulich S c h o o l of B u s i n e s s c o m p l e t e d the survey f o r a r e s p o n s e rate of 5 1 % . Overall, the three s u r v e y s included 1,306 entering and 2 , 7 9 1 g r a d u a t i n g students. T h e results of all three surveys w e r e m e r g e d t o g e t h e r and with i n f o r m a tion, s u c h as grades, obtained f r o m a d m i n i s t r a t i v e records. The Canadian Journal ofHigher Education Volume XXIX, No. 1, 1999 120 J.P. Grayson Table 1 Skill Categories and Question Topics Analytic Skills • • • • • • identifying the main points in lectures in your major clearly identifying the pros and cons of controversial issues like abortion figuring out the main arguments in articles written on topics in your discipline identifying flaws in positions given by other students in classes or seminars explaining your strengths and weaknesses to a potential employer defending a position you have taken in a classroom or seminar against the criticisms of other students Communication Skills • taking an article you read for a course this year and summarizing it in no more than two pages • verbally presenting your ideas on a topic of your choice to a group of ten strangers • verbally communicating to other students in your classes the flaws in their positions or arguments • writing a letter to a friend • writing a letter of application for a job • expressing yourself clearly in written English in an essay • correcting the grammar and spelling in the essay of a friend Personal Skills • assessing the feelings of people you have worked with in part-time or summer jobs • 'cooling out' a friend who is annoyed with you • being able to apologize to someone if you said something wrong about them • being able to admit to yourself when you are wrong • being able to get along with other people • enjoying meeting new people • helping friends patch up disagreements • considering the feelings of others • knowing yourself Organizational Skills • planning a job search strategy for a friend • organizing priorities to prevent rushing at the last minute • recruiting and organizing twenty strangers to collect money for the Heart and Stroke Fund The Canadian Journal of Higher Education Volume XXIX, No. 1, 1999 Using Surveys to Measure 'Value Added' in Skill in Four Faculties 121 Table 1 (continued) Comparative Skills • comparing what is going on in Canada to what is going on in a third world country • comparing what is going on in Canada today to what was going on fifty years ago • comparing what is going on in Canada to what is going on in any European country Basic Numeracy Skills • determining change from a $10 bill for a $2.75 pen • calculating 15% discount on a $9.36 book • determining percentage Liberal vote when 15 students vote Conservative, 10NDP, and 20 Liberal • solving x in the equation: 3x - 5 = 56.7 • explaining the meaning of 'square root' Basic Computer Skills • using a word processing program • using a spread sheet program • using a statistical analysis program The characteristics of survey respondents are summarized in Table 2. Consistent with the sampling procedures outlined above, for both the e n t e r i n g and g r a d u a t i n g surveys, the vast m a j o r i t y of students were enrolled in the Faculty of Arts. For both surveys nearly two thirds of respondents were female. Administrative records indicate that this gender distribution is comparable to that for undergraduate students as a whole. With regard to ethno-racial origin, the majority of both entering and graduating students identify themselves as of European origin; however, there are more students of European origin among graduating than entering students. Finally, most students in both surveys spoke English in the h o m e while growing up. Nonetheless, considerably more graduates than entrants reported English as their home language. The Canadian Journal ofHigher Education Volume XXIX, No. 1, 1999 122 J.P. Grayson Table 2 Characteristics of Survey Respondents Entrants Graduates Group Total 7.7% 104 3.1% 70 4.8% 174 Arts 61.7% 839 81.6% 1835 74.1% 2674 Fine Arts 13.3% 181 8.0% 180 10.0% 361 Science 17.3% 235 7.3% 165 11.1% 400 Col % 100.0% 100.0% 100.0% Count 1359 2250 3609 Female 61.0% 831 68.0% 1531 65.4% 2362 Male 39.0% 531 32.0% 719 34.6% 1250 Group Total Col % 100.0% 100.0% 100.0% Count 1362 2250 3612 Ethno-Racial Origin Black 5.3% 72 6.1% 135 5.8% 207 South Asian 8.0% 108 4.2% 94 5.6% 202 Chinese 16.6% 225 5.7% 128 9.9% 353 Other 14.8% 200 10.6% 235 12.1% 435 European 55.3% 749 73.4% 1635 66.6% 2384 Col % 100.0% 100.0% 100.0% Count 1354 2227 3581 Faculty Group Total Gender Group Total Business The Canadian Journal of Higher Education Volume XXIX, No. 1, 1999 Using Surveys to Measure 'Value Added' in Skill in Four Faculties 123 Table 2 (continued) Entrants Home Language English Graduates Group Total 63.9% 76.4% 71.7% 868 1722 2590 12.8% 3.0% 6.7% 174 67 241 3.5% 6.4% 5.3% 48 144 192 19.7% 14.2% 16.3% 268 320 588 Col % 100.0% 100.0% 100.0% Count 1358 2253 3611 Chinese Italian Other Group Total CONTROL GROUPS While it is not possible to have an external control group for the current study, a means of estimating the extent to which maturation affected the skills under examination was devised. This objective was accomplished by using the responses to 6,974 questionnaires completed for d e p a r t m e n t a l r e v i e w s in t h e W i n t e r and S p r i n g of 1996 and 1997 (response rate approximately 65%). A m o n g other things, at the departmental level, such surveys measure the same skills as in the survey of entering and graduating students. In brief, the logic of the procedure for estimating the effects of maturation is based on the fact that in all faculties but one at York University, m o s t s t u d e n t s e n t e r f i r s t y e a r f r o m high s c h o o l . T h e e x c e p t i o n is Atkinson College, the evening operation at York, that caters to mature students, most of w h o m are older than traditional students in other undergraduate faculties. W h i l e the vast majority of students entering Atkinson have some prior postsecondary education, a minority have, at best, c o m p l e t e d high school. As a result, d i f f e r e n c e s in skill levels between students entering Atkinson with no more than high school education, and traditional entrants to other faculties, can be attributed to the maturation process. The Canadian Journal of Higher Education Volume XXIX, No. 1, 1999 124 J. P. Grayson Unfortunately, as with most of the research focussing on value added by the university experience, resources did not permit the establishment of an external control group. Even if resources had not been an issue, given how some of the specific skill constellations under discussion were operationalized, it is doubtful that a comparison with an external control group would have been meaningful. For example, being able to identify the main points in lectures in your major is irrelevant to someone with no postsecondary experience. Moreover, as the objective is to measure the value added in skills among graduates of various faculties, a control group is not necessary. RELIABILITY OF INDICES The reliability of the various indices of skills as outlined in Table 1 was tested in the combined surveys of graduating and entering students by using Cronbach's alpha which: . . . can be viewed as the correlation between this test or scale and all other possible tests or scales containing the same number of items, which could be constructed from a hypothetical universe of items that measure the characteristic of interest. (Norusis, 1992, p. 149) In general, indices having alphas at or above .7 can be regarded as reliable. With one exception, alphas were well above .7. The coefficient for analytical skills was .84; for communication skills .80; personal skills had a coefficient of .81; for comparative skills the alpha was .85; the coefficient for basic numeracy skills was .79; and for basic computer skills alpha was .78. Only organizing skills, with a coefficient of .63, must be treated with caution. 2 CORRELATIONS AMONG SKILLS AND GRADES Correlations among each of the skills and Ontario Academic Credit (OAC) marks are reported in Tables 3 and 4. The first thing of note is that for entering students (Table 3) correlations between OAC marks and various skills are low. Of those that are statistically significant, there is a positive correlation between, on the one hand, OACs and, on the other, communication skills (.102), organizing skills (.076), numeracy skills (.101), and basic computer skills (.065); however, for personal skills the The Canadian Journal of Higher Education Volume XXIX, No. 1, 1999 Using Surveys to Measure 'Value Added' • 5 f * * * r— .N i O N oo O in o o o i < C L e> X co O o < o * * o * * * * * * * * * 125 *N O oo I N O O < N o < N — O •r N rC O o o 1 * * in Skill in Four Faculties * — o * * * o in o N oO o * * o o o O N (N N O N o oo i H O N O O O C (N O N o N o O in O o C N o S 2 0 > p p O 1 1 p C D — C O * * * * * * * * * * * # * o o o * * r- C rN o C O r — 1 co N < N O (N N O O O O oN O fxf O co o * t-- N O O (N >n O p T 3 -a jy 2 S N C NI — * * o r-; o o *—' '—' > „ »o O o o o OO OO t D ( I )J • C C a: a c c a u a 0 C '5 c a .5?.2} w m c a 0 X > s E u 3 t z 0 0 w u u EO 00 c c B 'c 3 M 13 c < o a Û eÛ 'N aj S a rE S o o I D U & H o U c 'c o !S1 S- 60 3 a < Ë U < C o O U V- -X- Z -X-X- The Canadian journal of Higher Education Volume XXIX, No. 1, 1999 5= S 1 & w a S B- Table 4 Skill and Grade Correlations for Graduating Students Skills Analytic Communication Personal Organizing Comparative Basic Computer OAC Marks GPA Numeracy ^ » Analytic Communication I Personal 1.000 .700** 1.000 .454** .454** .428** .154** .169** .065** .151** .470** .456** .393** .166** .200** .132** .193** .496** .303** .127** .102** -.042 -.044* .361** .095** .200** .018 .041 .127** .187** -.058* -.045* 1.000 Organizing Comparative 1.000 1.000 Basic Numeracy Computer O A C Marks N O T E : N u m b e r of cases on w h i c h correlations are based range from 1,641 to 2,250. * Correlation is s i g n i f i c a n t at the 0.01 level (2-tailed) ** Correlation is s i g n i f i c a n t at the 0 . 0 5 level (2-tailed) 1.000 .315** 1.000 .100** .055* .074** .082** 1.000 .636** Using Surveys to Measure 'Value Added' in Skill in Four Faculties 127 correlation is negative (-.114). If first year GPA is examined, it is evident that the highest statistically significant correlations is with O A C marks (.519). Statistically significant correlations of GPA with skills are weak. Analytic, communication, and organizing skills have correlations of .063, .087, and .059 with first year GPA. Personal skills correlate negatively ( - . 0 9 1 ) with GPA. T h e r e are two ways in which these general finding can be interpreted. First, if one expects that generic skills examined here are a necessary condition for high school achievement as measured in O A C marks, the low correlations can be taken as an indication that the current measures of generic skills are invalid: students with high O A C marks must possess high generic skills. On the other hand, if it can be accepted that O A C marks do not necessarily reflect generic skills the findings make sense. Certainly many faculty who bemoan the lack of preparation on the part of first year students despite high O A C marks would be amenable to this interpretation. At the same time, if employers are to be believed, it is equally likely that there is little relationship between high university grades and skills (Jones, 1994). Unfortunately, support for employers concerns is found in Table 4. For graduating students there are statistically significant, but weak, correlations (.151 and .193 respectively) between analytical and communication skills and c u m u l a t i v e g r a d e point a v e r a g e (GPA). S m a l l e r yet statistically significant correlations of .055 and .082 are found between basic numeracy and basic computer skills and GPA. Correlations between personal and comparative skills ( - . 0 4 4 and - . 0 4 5 respectively) are small and negative, but statistically significant. In essence, there is at best a weak relationship between some skills and cumulative GPA of graduating students: marks may measure subject knowledge, but not skills. Unfortunately, once again, we are left with a quandary. If we assume that university graduates must possess the generic skills analysed here, then w e must also c o n c l u d e that the skill measures e m p l o y e d in the study are invalid. If, on the other hand, we accept the views of critics that university curricula do not necessarily instill skills and that grades do not also measure skills, the findings help validate the skills measures e m p l o y e d in the study. As s o m e other r e s e a r c h e r s (Franklin, 1995; Money, 1996) have also reported weak relationships between measures of various skills and grades, it is tempting to side with employers. The Canadian journal of Higher Education VolutneXXIX, No. 1, 1999 128 J.P. Grayson While the link between skills and GPA may be weak, in Table 4, the correlation between cumulative GPA and OAC marks is a statistically significant .636. This relationship is comparable to the findings of studies of graduating students in the United States (Astin, 1993). In essence, O A C marks maintain a reasonably strong relationship with academic achievement over the students' university careers. Indeed, the correlation of OACs with cumulative GPAs of graduating students is higher than the correlation with first year grade point averages. This observation may be explained by the probability that students with low GPAs are less likely than others to finish their education. CONTROL VARIABLES In the analysis of net gains in various skills, it is important to control for possible confounding variables such as gender, racial origin, and language spoken in the home while the student was growing up. That each may have implications for particular skills is evident from Tables 5, 6, and 7 that include unstandardized and unadjusted skills scores and grades for entering and graduating students combined. Unfortunately, space constraints prevent a detailed examination of data in the tables. At every general level, however, it is obvious that considerable differences in skills are associated with differences in gender, racial origin, and home language. For example, from Table 5 it is seen that males score highest on basic numeracy, and computer skills. Females do better than males on analytic, personal, and organizational skills. There are no statistically significant differences for comparative skills. If ethno-racial origin as summarized in Table 6 is examined, black students score highest on analytic and communication skills. Students of European origin have the highest scores on personal and organizing skills while South Asian origin and other students do best on comparative skills and South Asian origin students score highest on computer skills. The highest scores on basic numeracy skills are reported by students of Chinese origin. It must be stressed that absolute differences among some scores for different ethno-racial groups are very low. One cause for alarm is the low rating for students of Chinese origin on analytic, communication, and comparative skills. As the majority of these students did not speak English in their homes, these figures likely illustrate the impact of language on various skills areas. In a relatively The Canadian journal of Higher Education Volume XXIX, No. 1, 1999 Using Surveys to Measure 'Value Added' OC MO O NC M tf C N •—1 ro N r ^ rOC tf C L in C N M — C M O < o < o o in Skill in Four Faculties tf r-; in > n 00 ro 00 tf ro C O O O C M tf 00 O N N O r- O N ro _ N OO O t-^ r- O 0 O N r- tf tf C M O O NN N O < N ro tf ro ro o on ro C OC M N O P- N O N O -H O N tf tf O N O C M O N 00 (N N O ~ C O ^ C M o tf CO tf O l ^ m m C M N O C O ^ m tf tf N O tf ( — N O on E O N C O C O C O tf O N O C O C C N tf tf On Ci 2 ' > n cx 129 C O C O <N C O tf tf O C O tf N O O O in C O ro to N O tf < Nm in o 10 cm 00 C O C N C M o m O C O C CN O O O N C O O n v o C O tf ,-sl UO rm C O O N tf O O C O O ^ 10 W C M tf ON O N C N N O ON ro in O 1 tf 0 tf ~ no o o w J C O E o O O C O in r- tf O C O O — I N O tf CN _ ^ i. V T 3 S V o u 13 B a > O h tf O tf o tf m no in Z S u -I C O C O C M c C 3 U tf N O c u ^ Q C S Z 00 C3 c a < u C 3 O H The Canadian Journal of Higher Education Volume XXIX, No. 1, 1999 130 J.P. Grayson oo —i—I O ON O -Sf O O t— (N •sf V On O O N O Tj" O O O ^ r -a- v> N on co C O O N ^î" • in oi 4 OO 00 CO r - V) co X co • m i ^ ^ V CN m co < Q. O O < o C L e o O — co ^tO 00 m co m no N Tf N O (N <N O O<N N O VI ON O NO O -H co >n (N O C O no N O ^f O co r-J C O N O ^ N O H " N oo 2 « £ S ^ O rr-r- N O T l - OC — C — v > v> m o O •sf ON O ^ t "C O N ^ <N ' ^ O • V N O C O Tt" O O N O N N O^ C Tt- (N O N ^ ^H O N Tt" O N O N co co" C C O C N O O < u v> O in r ^ o E o ^ O E ^ vi t vo V) O V) in c N ON vi C ^ "tf o O ^f Tf in O N no N ^ CO ' ^ O N oo N O ^ t " IN O O N O — — m —O i N N ^t 0O cn — ^r on O -<t C O O N co • r X< N ' C O r^ ^ • O in -h <N o\' Tf 00 N O ON ON ^t" (N N O Tf N N O C O >-1 00 C v> o ~C O no on co C N vo N O no Tj Tt O N ^ £ O ^ < O— O oo (N ON O co ^ • M "û o o rA C O r- C O C N O O in v) oo ' r X ^ ' co co C N O I oo t— t - ON NO O O VI V -h on >n t C m co • •sf N C O T j "O (N O O V co C O N ^ r- ^t- ^tV) (N M (N co in 1 ' O N V) ^ C M — -h cn m u « C O Tj- co o\ r - >n ON C O 00 ^r r — ' O 'n O On no ^iO N (N N O ro ^t- E E o X! SB O) •O es s- o 73 V C MN V Tt H o O H O — T f N O ~C O ^< N O N O C O O N S, c o c < — . O >n ' O N C O r- ^t- r- o V) O (N N O O N O N N O • C O rt N O ^ ^ r XC O ' O (N —N ^ J •t O M C O O —I V V ~C O ^ < N C es VI « u Z S o Z S u N O t» <u a I <J1 Z S Z S Z S c S '3d a m o a S J3 n u a o o <2 on < W 3 .2 The Canadian journal of Higher Education Volume XXIX, No. 1, 1999 3 ^ Q Z oo Using Surveys < Q. o o < o to Measure "Value Added' O r- O O O N ;<n C N O N (N in C N oo ;cn O rO O O N C O C O N O oo O ; in N >/-i (N : wi CN — wi iri " i in — I" C O r- N O O N O in C O C o in 1 — 1 oo oo IT) r - 00 N O N O rr- N O IT) O O O £ s N O rin co m >o Tf tN O N Tf cn on o in Skill in Four Faculties ^ £ • iN O O N — 00 : O O N O m r- (N N O W > rTf N O O N on- co r M <» ^ £ • E o O —; (N in cn N O O N O C N N iNO o o no ;no O N O N • Ç f ^ O. O N C O O Tt" 00 N O Ti- , , , 1 N —• in oo rO N O N ol C O in C O m C O C O ^t n •} h ^ ^ 0\ « oo no co ^t ^ < N TO C D oo co 131 tN oo oo ; i/-> ^ O r-- O in m N O N O C O C O NO , ,NO NO O N t-- ON O m C O O C O in C O i co O : "3- O N C O O N C O in :NO O N C N o 00 C N C O ON O m oo O C O :NO 00 00 C O r- oo : T T in oo in m C N : C O C O C O C O C O m C N C O o rO N r- oo oo r- N O , , 00 oo in in — C NC O in ; r-; C N 10 C N N O IT) in m N C N : co C C O ^ 4 > b£ « s C J j B : . '—• C 3 nJ O TtO O O C r- C N oo in in o O N o C i—i in C O C O C-; i-H 0 O oo N O o 0 NO O in N C O in C O C O) s o a & C N E o O vi a > T3 a i* O C O in N O O C O N O m N O O C N <N O N N m in C N C O N O < — I O O in O N r- N O o in C N in C O C O •sr C N O •v a a VI at h o o CO JU — 2 H co in C O ^ Q ZC O z S z S ^ Q Z oo O m N O in C O Z S a "3d 'E O 00 c W u The Canadian Journal of Higher Education Volume XXIX, No. 1,1999 132 J. P. Grayson language neutral area, like basic numeracy, students of Chinese origin score high. The importance of language is further illustrated when language spoken in the home is examined. Table 7 shows that students who spoke Italian while growing up score highest in analytic, personal, organizing, comparative, and computer skills. Students who spoke English do best in communication skills. Finally, students who grew up speaking Chinese have the highest scores in basic numeracy. ASSESSING VALUE ADDED As noted earlier, in order to determine value added, it is necessary to compare the scores on various skill indices of graduating to entering students. While standardized scores are not necessary to achieve this object i v e , t h e r e w e r e t w o r e a s o n s f o r c a l c u l a t i n g t h e m in the c u r r e n t undertaking. (For most practical purposes standardized scores, or z-scores, have a mean of 0 and a standard deviation of 1.) First, in order to compare changes at York to changes reported in the literature, and to make changes understandable to the average reader, it is necessary to specify differences between graduating and entering students in terms of percentile point differences. Second, standardized rather than actual scores will be reported in order to protect the confidentiality of the York data. In essence, while readers will know the amount of change that can be attributed to the university experience, they will not know the absolute value of entering and graduating scores (however, such data are available to the York community). From a value added perspective the former information is more important than the latter: the concern is not with the level of skills of graduating students but with how much the university experience has contributed to the development of skills. Percentile point difference in the skills of graduating as compared to entering students in the Schulich School of Business (Business), Arts, Fine Arts, and Pure and Applied Science (Science), after adjusting for OAC marks, gender, ethno-racial origin, and home language, are presented in Table 8. The specific skills analysed are listed in column one. Column two, Entering Students, lists standardized scores for each skill by faculty for students entering the university. Similar measures for graduating students are found in column three, Graduating Students. For columns two and three the number of cases on which the scores are calculated are found The Canadian Journal of Higher Education Volume XXIX, No. 1, 1999 Using Surveys to Measure 'Value Added' in Skill in Four Faculties 133 Table 8 Skills' Z-scores by Faculty for Entering and Graduating Students (Controls on OAC marks, gender, ethno-racial origin and home language) Entering Students Analytic Graduating Students Percentile Difference Skills Business*" Arts"* Fine A r t s ' " Science*" -0.55 -0.34 -0.18 -0.56 (95) (760) (158) (207) 0.13 0.22 0.40 -0.29 (62) (1229) (138) (122) 26 22 23 10 Communication Skills Business*" Arts'" Fine Arts"* Science"* -0.38 -0.38 -0.23 -0.69 (92) (758) (160) (211) 0.44 0.23 0.34 -0.13 (61) (1295) (137) (121) 32 24 22 20 Personal Skills Business*** Arts*" Fine Arts Science -0.56 -0.17 0.07 -0.28 (90) (755) (151) (199) 0.02 0.16 0.09 -0.23 (62) (1300) (138) (123) 21 13 0 2 Organizing Skills Business*** Arts'" Fine Arts"* Science*** -0.32 -0.30 -0.43 -0.52 (92) (758) (158) (206) 0.44 0.20 0.08 0.09 (62) (1306) (138) (123) 29 20 17 20 Comparative Skills Business Arts"* Fine Arts Science 0.05 -0.10 -0.13 -0.50 (94) (765) (161) (215) 0.08 0.16 -0.21 -0.47 (62) (1304) (138) (123) 0 10 -3 1 Numeracy Business Arts"* Fine Arts*** Science* 0.41 -0.28 -0.65 0.45 (96) (753) (160) (215) 0.53 0.05 0.08 0.56 (63) (1290) (137) (122) 4 11 24 4 Computer Skills Business"* Arts"* Fine Arts*** Science"* 0.14 -0.26 -0.38 0.06 (90) (755) (155) (206) 1.05 0.03 0.08 0.66 (62) (1292) (136) (124) 30 10 15 25 Basic Significance for F for differences between entering and graduating students: * LE .05; ** LE .01 ; *** LE .001 Number of cases in parenthesis. The Canadian Journal of Higher Education Volume XXIX, No. 1, 1999 134 J. P. Grayson in parenthesis. In column four, differences in scores for entering and graduating students are presented as percentile point differences. To begin with analytic skills, the highest scores for entering students are found among students entering Fine Arts ( - . 1 8 ) and Arts (-.34). Among graduating students, those from Fine Arts (.40) and Arts (.22) have the highest scores. In terms of the difference between the scores of entering and graduating students, however, the greatest improvement, or value added, is observed for Business students (26 percentile points). With communication skills, students entering Fine Arts have the highest scores ( - . 2 3 ) and Science students the lowest (-.69). Among graduating students those from Business have the highest scores (.44). M o r e o v e r , the value added in c o m m u n i c a t i o n skills (32 percentile points) is higher once again in Business than in any other faculty. Students entering Fine Arts report the highest scores on personal skills (.07). Among graduates the highest scores are for Arts students (.16). In terms of value added, however, Business students show the greatest improvement (21 percentile points). A m o n g entering students, those going into Arts have the highest organizing scores (-.30); however, among graduates, those from Business score highest (.44). Business students also show the greatest value added (29 percentile points). When it comes to comparative skills, entering Business students do better than others (.05). Upon graduation, however, Arts students score highest (.16). Arts students also display the greatest value added (10 percentile points). It is worth noting that for Fine Arts students there is a net loss in terms of comparative skills between first and final year ( - 3 percentile points). As it is unlikely that a Fine Arts curriculum relates to comparative skills as measured here, this is not surprising. With scores of .45, students entering Science have the highest basic numeracy scores among first year students. Science students also have the highest scores among graduating students (.56). In terms of value added, however, the greatest improvement is found for Fine Arts graduates (24 percentile points). It should be noted, however, that the final Fine Arts score is only .08. Among entering students, those in Business have the highest scores on basic c o m p u t e r literacy (.14). This score increases to 1.05 f o r Business graduates and represents a gain of 30 percentile points, the highest increase of students in any faculty. The Canadian journal ofHigher Education Volume XXIX, No. 1, 1999 Using Surveys to Measure 'Value Added' in Skill in Four Faculties 135 There are two ways in which the data in Table 8 can be examined. First, it can arbitrarily be decided that for graduating students a score of, say, .10 is too low in an absolute sense for any skills under discussion. If this yardstick is accepted, the scores of graduating Science students for analytic (-.29), communication (-.13), personal (-.23), organizing (.09), and comparative skills (-.47) are unacceptable. Similarly, the personal (.02) and comparative (.08) scores of Business graduates are problematic. The problem areas for Fine Arts graduates are personal skills (.09), organizing skills (.08), comparative skills (-.21), basic numeracy skills (.08), and computer skills (.08). Arts students are low in the areas of basic numeracy (.05) and computer skills (.03). The second way in which the data can be examined is in terms of value added. If this tack is followed, it can be seen that the greatest gains in analytic, communication, personal, organizing, and computer skills are made by students in Business. The value added in comparative skills is greatest for Arts graduates and Fine Arts students show the greatest increases in basic numeracy. Overall, the information in Table 8 indicates first, that some skill levels of graduating students in certain faculties are too low. Second, faculties differ in the amount of value added to particular generic skills. In order to obtain an overview of value added in the skills under discussion, we can simply take the average over the four faculties. When this is done, in descending order average gains are: communication skills (25 percentile points), organizing skills (22 percentile points), analytic and computer skills (20 percentile points each), basic numeracy skills (11 percentile points), personal skills (9 percentile points), and comparative skills (2 percentile points). THE EFFECTS OF MATURATION As noted earlier, it was not possible to have an external same age control group of individuals who did not go to university. As a result, there is a possibility that the skills gains noted in the previous section could be totally or in part a result of maturation. In order to test for this possibility, using the results of surveys carried out for departmental reviews, the skill scores of first year students in Atkinson College were compared to those of students in other undergraduate faculties. The average age of first year Atkinson students was 32; in other faculties it was 21. The mean number The Canadian Journal ofHigher Education Volume XXIX, No. 1, 1999 136 J. P. Grayson of courses registered in or completed for Atkinson students was .8; for other students, 4.4. The analysis was restricted to first year students who had no prior postsecondary education at either the college or university level and skill scores were adjusted for the effects of gender, ethno-racial origin, and language spoken in the home while growing up. Adjustments were not made for O A C marks because admission to Atkinson is not necessarily based on high school performance. T h e results of analyses of covariance indicate that with one exception there were no statistically significant differences between first year students in Atkinson and in other faculties or colleges on the measures of skill used in this study. The exception was organizing skills for which the mean score of Atkinson students was 11 percentile points higher than scores of other students. F i n d i n g s such as these give support to the n o t i o n t h a t d i f f e r e n c e s b e t w e e n e n t e r i n g and g r a d u a t i n g s t u d e n t s analysed earlier are not simply the result of maturation. This said, a note of caution must be sounded. Surveys for program reviews are carried out between January and March. As a result, it is possible (but not probable) that the skills of students in colleges or faculties other than Atkinson could have differentially increased since the beginning of the year. AN OVERALL ASSESSMENT H o w can we evaluate the differences in generic skill scores of students entering and leaving York University? Fortunately, in their tome, How College Affects Students, Pascarella and Terenzini (1991) have summarized findings from a number of studies on freshman to senior gains in a number of domains similar to those analysed here (see Table 9). T h e first c o l u m n of the table lists the skills under consideration. C o l u m n t w o lists the effect size that can be viewed as the difference between the mean score for graduating and entering students divided by the entering student standard deviation. This value is equivalent to the unadjusted mean in the analysis of covariance conducted for this report. E l s e w h e r e (Grayson, 1996) it has been shown that for the Faculty of Arts at York, differences between unadjusted and adjusted means for the skills scores are minor. Column three contains information on effect size translated into area under the normal curve. The Canadian Journal ofHigher Education Volume XXIX, No. 1, 1999 Using Surveys to Measure 'Value Added' in Skill in Four Faculties 137 Table 9 Summary of Estimated Freshman-to-Senior Changes: Learning and Cognitive Development Outcome Effect Size Percentile Point Difference G e n e r a l verbal skills .56 21 G e n e r a l quantitative skills .24 10 Specific subject matter knowledge .84 31 Oral c o m m u n i c a t i o n skills .60 22 W r i t t e n c o m m u n i c a t i o n skills .50 19 Piagetian (formal) reasoning .33 13 1.00 34 informal reasoning) 1.00 34 Ability to deal with conceptual complexity 1.20 38 Critical thinking U s e o f r e a s o n a n d e v i d e n c e to a d d r e s s ill-structured problems (reflective j u d g m e n t , Taken f r o m Pascarella & Terenzini (1991), p. 558. There are two important observations that can be made on the basis of the data in Table 9. First, overall gains between first and final year are modest ranging from a low of 10 percentile points for quantitative skills to a high of 38 percentile points for ability to deal with conceptual complexity. Second, while different measurement techniques preclude direct comparisons, there is enough similarity between differences observed at York and those seen in studies conducted elsewhere to inspire confidence in measures used for the York study. For example, Pascarella and Terenzini report gains of 21 percentile points for general verbal skills, 22 points in oral communication skills, and 19 percentile points in written communication skills (the average for these three is 21 percentile points). At York, average gains in communication The Canadian Journal of Higher Education Volume XXIX, No. 1, 1999 138 J. P. Grayson skills, that included some general verbal as well as written and oral components, were 24 percentile points. The same is true of quantitative skills: Pascarella and Terenzini report gains of 10 percentile points and the average gain in basic numeracy at York was 11 percentile points. It must be stressed that the studies summarized by Pascarella and Terenzini used many different measures of the skills under consideration (some involved the use of standardized tests of various skills). As a result, it would be wrong to place too much emphasis on specific comparisons between their and the York findings. What is important is the fact that on the two gains on which it was possible to comment, increases observed at York were similar in size to those measured elsewhere. BROADER IMPLICATIONS Students graduating f r o m four faculties at York University report higher skills than entering students. Unfortunately, the degree to which value a d d e d in generic skills at York is c o m p a r a b l e to that in other Canadian universities is contingent upon the latter opening comparable avenues of inquiry and publicizing the results. In the e v e n t that similar research w e r e initiated, what m i g h t be expected? Research carried out in the United States can be of assistance in a n s w e r i n g this question. As Pascarella and Terenzini (1991), after their review of approximately 3,000 U.S. studies related to university outcomes, argue: There are clear and unmistakable differences among postsecondary institutions in a wide variety of areas, including size and complexity, control, mission, financial and educational resources, the scholarly productivity of faculty, reputation and prestige, and the characteristics of the students enrolled. At the same time, however, American colleges and universities also resemble one another in a number of important respects. It may be that despite their structural and organization differences, their similarities in curricular content, structures, and sequencing; instructional practices; overall educational goals; faculty values; out-of-class experiences; and other areas do in fact produce essentially similar effects on students although the 'start' and 'end' points may be very different across institutions. (p. 589) The Canadian Journal ofHigher Education Volume XXIX, No. 1,1999 Using Surveys to Measure 'Value Added' in Skill in Four Faculties 139 Given that U.S. universities are far more diverse than those in Canada, it is likely that in this country there is even a smaller difference in outcomes such as those studied here than south of the border. In other w o r d s , w h i l e i n s t i t u t i o n s like York, M e m o r i a l , Q u e e n ' s , R y e r s o n P o l y t e c h n i c U n i v e r s i t y , the U n i v e r s i t y of T o r o n t o , W e s t e r n , the University of Manitoba, the University of Regina, and the University of Victoria may vary in things like resources and the entering average of students, based on findings from U.S. research, it is unlikely that great differences exist in the value added to skills similar to those studied at York. CONCLUSION The objective of this study was to assess the value added to generic skills by the university experience. In the best of all possible worlds such a study would have measured the skill levels of entering students using both self-reports and standardized tests and then, after the students had completed their education, skills measured with the same instruments could be compared to those at entry. Provided that increases in skills were not also observed in a same age control group that did not attend university differences between entry and exiting skill levels could be attributed to the university experience. Unfortunately, this type of research is both costly and time consuming and unlikely to be carried out in Canadian universities; however, the type of research described in this report, in which, after adjusting for potentially confounding influences, self-reports of skill levels of entering students were compared to those of graduating students, is a cost e f f e c - , tive and timely alternative to collecting information in longitudinal studies. The justification f o r taking this tack can be found in previous research indicating acceptable levels of agreement between self-assessed and other measures of skills and the similarity in results of cross-sectional and longitudinal research carried out in related areas. Although information collected via the process described in this study should not be used in decisions regarding individuals, it does have utility in assessing change in groups. Information collected in this fashion can have several practical applications. First, once benchmarks have been established, within institutions, individual faculties can determine whether or not relative to other faculties their students emerge from their courses of studies with acceptable skill levels. Second, faculties that make deliberate attempts to The Canadian Journal of Higher Education Volume XXIX, No. I, 1999 140 J. P. Grayson e n h a n c e certain skills can see if there are d i f f e r e n c e s in skill levels between entering and graduating students. In either the first or second cases, if skill levels are lower than desired, steps can be taken to remedy the p r o b l e m . Third, if similar information were available f r o m other institutions, universities could compare the value added by their institution to that of o t h e r u n i v e r s i t i e s . O n c e again, if c o m p a r i s o n s w e r e unfavourable, changes could be introduced to deal with deficiencies. Consistent with the foregoing possibilities, the most important finding of the current study was that, in keeping with previous research, students graduate f r o m York with better developed skills than students who enter the institution. There are, however i m p o r t a n t differences between and a m o n g faculties in terms of the value added to particular skills and in some faculties the skill scores of graduating students may be too low. M o r e o v e r , s o m e e v i d e n c e s u g g e s t s t h a t the d i f f e r e n c e s o b s e r v e d between entering and graduating students are not the result of maturation. Furthermore, the magnitude of two average gains in skills are comp a r a b l e to gains r e c o r d e d in the United States. G i v e n the results of research conducted south of the border, similarity in results such as these is to be expected. Unfortunately, we will not be able to c o m p a r e the value added at York to that of other Canadian universities until the latter conduct the required research. References Anonymous (1995). Literacy, international adult literacy A s t i n , A . ( 1 9 9 1 ) . Assessment economy survey. for excellence. A s t i n , A . ( 1 9 9 3 ) . What matters and society: results of the first Ottawa, O N : O E C D & Statistics C a n a d a . in college? Toronto, ON: Collier Macmillan. San Francisco, CA: Jossey Bass. Baird, L. (1976) Using self-reports to predict student performance. New York, NY: College Entrance Examination Board. B e r d i e , R . ( 1 9 7 1 ) . S e l f - c l a i m e d a n d t e s t e d k n o w l e d g e . Educational and Psychological Measurement, 31(3), 629-636. C o n f e r e n c e B o a r d o f C a n a d a . ( 1 9 9 2 ) . Employability skills profile. Ottawa, ON. D u m o n t , R., & Troelstrup, R. (1980) Exploring relationships between objective a n d s u b j e c t i v e m e a s u r e s of i n s t r u c t i o n a l o u t c o m e s . Research Education, in E v e r s , F . , R u s h , J., K r m p o t i c , J., & D u n c a n - R o b i n s o n , J. ( 1 9 9 3 ) . Making match: Higher 72(1), 3 7 - 5 1 . Phase II. G u e l p h , O N : U n i v e r s i t y of G u e l p h . The Canadian Journal of Higher Education Volume XXIX, No. 1, 1999 the Using Surveys to Measure 'Value Added' in Skill in Four Faculties 141 E v e r s , F., & O ' H a r a , S. ( 1 9 9 6 ) . E d u c a t i o n a l o u t c o m e m e a s u r e s of k n o w l e d g e , skills, and values. Education Quarterly Review, 3(1), 43-56. E v e r s , F . , & R u s h , J . ( 1 9 9 6 ) . T h e b a s e s of c o m p e t e n c e : S k i l l d e v e l o p m e n t d u r i n g t h e t r a n s a c t i o n f r o m u n i v e r s i t y t o w o r k . Management Learning, 27(3), 2 7 5 - 2 9 9 . F r a n k l i n , M . ( 1 9 9 5 ) . T h e e f f e c t s of d i f f e r e n t i a l c o l l e g e e n v i r o n m e n t s on a c a d e m i c l e a r n i n g a n d s t u d e n t p e r c e p t i o n s of c o g n i t i v e development. Research in Higher Education, 36(2), 127-153. G r a y s o n , J. ( 1 9 9 5 ) . D o e s r a c e m a t t e r ? — O u t c o m e s of t h e f i r s t y e a r e x p e r i e n c e in a Canadian university. The Canadian 25(2), Journal of Higher Education, 79-109. Grayson, J. (1996). Value added in generic skills between first and final year: a. pilot project. Toronto, O N : Institute for Social Research, Y o r k University. J a c o b s , S. ( 1 9 9 5 ) . T e c h n i c a l c h a r a c t e r i s t i c s a n d s o m e c o r r e l a t e s of t h e C a l i f o r n i a critical t h i n k i n g skills test, f o r m s A a n d B . Research Education, in Higher 36(1), 8 9 - 1 0 8 . Jones, E. (1994) Essential skills in writing, speech and listening, and critical thinking for college graduates: perspectives of faculty, employers, and policy makers. National Centre on Postsecondary Teaching, Learning, and Assessment. K e e l e y , S., B r o w n e , M . , & K r e u t z e r , J. ( 1 9 8 2 ) A c o m p a r i s o n of f r e s h m e n a n d s e n i o r s in g e n e r a l a n d s p e c i f i c e s s a y t e s t s of critical t h i n k i n g . Research Higher Education, K u h , G., V e s p e r , N., C o n n o l l y , M . , & P a c e , R . (1997). College questionnaire: in 17(2), 139-154. Revised norms for the third edition. student experiences Bloomington, IN: Center for Postsecondary Research and Planning, Indiana University. M c M i l l a n , J. ( 1 9 8 7 ) . E n h a n c i n g c o l l e g e s t u d e n t s ' critical t h i n k i n g : a r e v i e w of studies. Research in Higher Education, 26(1), 3 - 2 9 . M c M o r r i s , R., & Ambrosino, R. (1973). Self-report predictors: A reminder. Journal of Educational Measurement, 10(1), 13-17. M o n e y , S. ( 1 9 9 6 ) . The relationship between critical thinking scores, achievement scores, and grade point average in three different disciplines. Ph.D. dissertation draft, Michigan State University. Norusis, M. (1992). SPSS for Windows: Professional Statistics. Chicago, IL: SPSS. P a c e , C . ( 1 9 7 9 ) . Measuring outcomes of college. San Francisco, C A : Jossey Bass. Pascarella, E., & Terenzini, P. (1991). How college affects students. San Francisco, CA: Jossey Bass. P a s c a r e l l a , E . , B o h r , L., A m a u r y , N . , & T e r e n z i n i , P . ( 1 9 9 5 ) . I n t e r c o l l e g i a t e a t h l e t i c p a r t i c i p a t i o n a n d f r e s h m a n y e a r c o g n i t i v e o u t c o m e s . The Journal Higher Education, of 66(4), 3 6 9 - 3 8 7 . The Canadian journal ofHigher Education Volume XXIX, No. 1,1999 142 J. P. Grayson Pascarella, E., Nora, A., & Bohr, L. (n.d.). Longitudinal and documents. panel study instruments National Center on Teaching, Learning, and A s s e s s m e n t . Pike, G. (1995a). Limitations of using students' self-reports of academic development as proxies for traditional achievement measures. Paper p r e s e n t e d to the A n n u a l M e e t i n g of the A s s o c i a t i o n f o r Institutional Research, Boston, M A . Pike, G . (1995b). T h e relationship b e t w e e n self-reports of college experiences a n d a c h i e v e m e n t test s c o r e s . Research in Higher Education, 36(1), 1 - 2 1 . Pike, G . (1994). T h e relationship b e t w e e n self-report and objective m e a s u r e s of s t u d e n t a c h i e v e m e n t . P a p e r p r e s e n t e d to the A n n u a l M e e t i n g of the A s s o c i a t i o n f o r the Study of H i g h e r Education, Tucson, A Z . P o h l m a n n , J., & B e g g s , D . ( 1 9 7 4 ) . A study of the validity of s e l f - r e p o r t e d measures of academic growth. Journal of Educational Measurement, 2(2), 115-120. Spector, P. (1992). Summated rating scale construction. N e w b u r y Park, C A : Sage Publications. Steele, J. ( 1 9 8 6 ) . Assessing postsecondary students. reasoning and communication skills of P a p e r p r e s e n t e d to the m e e t i n g of the A m e r i c a n E d u c a t i o n a l R e s e a r c h Association, San Francisco, C A . Notes ' R o n Sheese, N i c k Elson, Darla Rhyne, T a m m y Chi, David Northrup, and P a u l G r a y s o n p a r t i c i p a t e d in t h i s p h a s e of t h e s t u d y . ^ F o r N o v e m b e r 1995 graduates of the Faculty of Arts, a principal c o m p o n e n t a n a l y s i s u s i n g all of t h e s k i l l s v a r i a b l e s w a s c o n d u c t e d in w h i c h t h e n u m b e r o f f a c t o r s s p e c i f i e d f o r r e t e n t i o n , 7, w a s e q u a l t o t h e n u m b e r of i n d i c e s t h a t h a d b e e n c o n s t r u c t e d . W i t h f a c t o r l o a d i n g s l o w e r t h a n .4 s u p p r e s s e d , t h e a n a l y sis y i e l d e d f a c t o r s s i m i l a r t o t h o s e o f b a s i c n u m e r a c y , c o m p u t e r skills, p e r s o n a l s k i l l s , c o m p a r a t i v e skills, a n d j o b p r o c u r i n g skills. T h e o t h e r f a c t o r s c o m b i n e d , in n o s y s t e m a t i c f a s h i o n , v a r i a b l e s t h a t c o m p r i s e d t h e a n a l y t i c a l a n d c o m m u n i c a t i o n skill c a t e g o r i e s ( s e e S p e c t o r [ 1 9 9 2 ] f o r a r a t i o n a l e ) . O n t h e b a s i s o f t h e s e f i n d i n g s , it w a s d e e m e d a p p r o p r i a t e t o r e t a i n t h e o r i g i n a l i n d i c e s . The Canadian Journal of Higher Education Volume XXIX, No. 1, 1999