The Canadian Journal of Higher Education La revue canadienne d'enseignement supérieur Volume XXVII, Nos. 2,3,1997 pages 91-112 Pitfalls in the Assessment of Postgraduate Scholarship Programs: The Need for New Indicators F.-PIERRE DUSSAULT & ANDRÉ MANSEAU Scienscript FPD Inc. National Research Council Canada Abstract Very few published studies have examined the outcomes of postgraduate scholarship programs. Basing our analysis on these studies and on internal reports from U.S. and Canadian organizations involved in scholarship programs, we have compiled an overview of the wide variety of indicators and methods that have been used, and conducted a comparative study of outcomes using the four most commonly used indicators: awarded diploma, obtained job, obtained related job, and pursuing studies. Our analysis revealed several methodological pitfalls in comparing the results. Although the use of available data limits the depth of a comparative analysis, our results show that scholarship programs tend to increase the rate of awarded diplomas. Résumé Très peu d'études publiées ont examiné les retombées des programmes de bourses d'études supérieures. En fondant notre analyse sur ces études, mais surtout sur des rapports internes d'organismes américains et canadiens, T h e authors are grateful for the support that they have received from the Institut de recherche en santé et en sécurité du travail du Québec (IRSST) to prepare this research. 92 F.-P. Dussault & A. Manseau nous avons répertorié une variété des méthodologies et d'indicateurs utilisés puis nous avons réalisé une étude comparative avec les quatre indicateurs les plus fréquents : le taux de diplômation, le taux d'emploi, le taux d'emploi en relation avec les études et le taux de poursuite des études. Notre analyse a révélé plusieurs difficultés méthodologiques dans la comparaison des résultats. Malgré le fait que les données disponibles limitent la portée d'une analyse comparative, nos résultats montrent que ces programmes favorisent l'obtention d'un diplôme. What are the outcomes of postgraduate scholarship programs? In Canada, the three major research councils (MRC - Medical Research Council, NSERC - Natural Sciences and Engineering Council, and SSHRC - Social Sciences and Humanities Research Council) spend up to $100 million (Cdn) per year to support postgraduate university students (master's, doctoral and postdoctoral work). By including all other organizations involved in such programs (governmental agencies, private business corporations and nonprofit organizations), we can estimate that more than $200 million is spent annually. In the United States, this figure could reach several billion dollars per year, considering the population and the tuition fees, although it is difficult to estimate, due to the very large number of private foundations. Seventeen university foundations had more than one billion dollars each in assets in 1994 (National Association of College and University Business Officers, 1995). The objectives of scholarship programs for master's and doctoral degrees are different from those for undergraduate or postdoctoral ones. The basic objective of the grants at the postgraduate level is not to provide access to university, as it is at the undergraduate level, but rather to produce highly qualified professionals or researchers who will work in the field in which they have been educated. Postdoctoral programs are aimed at developing research expertise and are not related to any form of degree. Our study focuses on master's and doctoral programs since they are related to formal studies leading to a diploma. Furthermore, there are significantly fewer studies on these programs than on postdoctoral ones, which underlines a particular need for a better understanding of the benefits of master's and doctoral scholarship programs. Carrying out an analysis of outcomes from master's and doctoral scholarship grants entails many pitfalls since there are several sources of funds, no cumulative statistics available, and granting agencies seldom The Canadian Journal of Higher Education Volume XXVII, Nos. 2,3, 1997 Assessment of Postgraduate Programs 93 conduct systematic assessments of their programs. These assessments, which would involve regular surveys of former fellows, do not appear to be part of current operations. In addition, the lack of specific criteria could raise many questions in the assessment process, as Beck (1985) concluded in the case of undergraduate programs. Recent Studies of the Assessment of Scholarship Program Outcomes Very few published studies have examined the outcomes of master's and doctoral scholarship programs; 1 however, there are a number of interesting studies of undergraduate programs which have identified indicators, and some others which have assessed postdoctoral programs. Evaluation studies of undergraduate financial aid programs were briefly examined to identify which indicators of outcomes were used. St. John (1990) analyzed five types of potential impacts: access to university, choice of school, persistence, choice of major, and macroeconomic outcomes. He found that financial aid programs increase general attendance and persistence, and have a greater influence on the choice of school than do tuition fees. This study also concluded that the choice of major seems to be more influenced by family and type of college than by financial aid, especially since up to now, undergraduate scholarships have seldom been awarded for a specific field of study. Finally, the economic outcomes were evaluated by predicting the effect of graduating on the total lifetime income of the students (with or without financial aid); in this regard, the study estimated a return of $4.30 for each dollar spent, over a 40-year period. This return is equivalent to a 3.7% simple interest rate per year on a deposit for 40 years, which is a rather low rate of return. The indicators of outcomes used for assessing undergraduate scholarship programs could be adapted to the analysis of graduate scholarship returns and become: access to graduate studies, choice of school, persistence, choice of major or thesis subject, and the total lifetime income effect. However, we will see that these indicators are not frequently used for the assessment of graduate scholarship programs. Some studies assessing postdoctoral scholarship programs have been published (Fogarty International Center, 1989; Institute of Medicine, 1986; N S F - National Science Foundation, 1988b; Teichler, 1991). These studies analyzed academic position, research grants, scientific publications and citations of former fellows. Such indicators of outcomes are strictly related to academic career paths* and consequently are The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3, 1997 94 F.-P. Dussault & A. Manseau not sufficient for assessing the more varied career paths following master's and doctoral studies. Indeed, these indicators are rarely used to assess graduate scholarship programs. We found only two published studies that present results of postgraduate scholarship program outcomes. The NJSDHE study (New Jersey State Department of Higher Education, 1989) presents many indicators of outcomes, based on data from a survey of former fellows of the NJSDHE postgraduate scholarship program. Sheridan and Pyke (1994) summarized the question of the time to complete graduate studies, including its relationship with financial support. The results from these studies will be discussed in this paper. Some studies have assessed program characteristics and/or activities but not program returns. Golding, Lang, Eymard, and Shadish (1988) conducted a longitudinal study of the American Psychological Association Ph.D. Fellowship Program. They determined absolute and relative values of the grants in relation to overall income over the years. However, they did not analyze program outcomes. A report prepared for Supply and Services Canada (Programs Evaluation Teamwork Study Group, 1985) presented an administrative evaluation of several postgraduate fellowship programs in various federal departments and agencies. In this report, administrative procedures and goal setting were assessed, but, as in Golding's study, no data on program outcomes were presented. Method Assuming that several evaluation reports on postgraduate scholarship programs remain unpublished, we contacted directly some officers of the two major American agencies involved in such programs, the National Science Foundation (NSF) and the National Institutes of Health (NIH), officers of the NJSDHE, and officers of major Canadian agencies. In Canada, there are hundreds of postgraduate scholarship programs. However, if we consider only those that offer a full annual scholarship (on average, more than $10,000 Cdn. per year, to cover the minimum annual cost), the number drops significantly. For example, there are about twelve Canadian postgraduate scholarship agencies, federal or provincial ones, that offer more than 25 full grants per year to Canadian students in the province of Quebec (Table 1). Few organizations, either American or Canadian, have systematically assessed postgraduate scholarship programs. We obtained only one The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3, 1997 Assessment of Postgraduate Programs 95 Table 1 Main Canadian Postgraduate Scholarship Programs Available for Quebec Students in 1992-1993 Organization Master's Rank Value ($Cdn) 4 14,154 Canada National Defence 3 12,000 to 15,000 Canadian Cancer Society to 53,000 (not offered) Canada Mortage & Housing Corp. Doctoral Rank Value ($Cdn) Postdoctoral Rank (not offered) 2 Value ($Cdn) (not offered) 15,000 to 21,000 (not offered) (not offered) 21,800 to 53,000 Canadian Diabetes Association 2 15,295 4 15,295 1 max. 60,000 Canadian Heart Foundation 2 15,295 4 15,295 3 27,100 to 42,585 FCAR 6 11,000 7 13,000 9 22,000 FRSQ 5 13,000 7 13,000 5 21,800 to 34,650 IRSST 5 13,000 5 15,000 1/special 20,000 to 30,000 6 24,000 to 30,000 MRC 2 15,295 4 15,295 4 27,100 to 42,000 NSERC 1 15,600 3 17,400 7 29,000 SSHRC (not offered) 6 14,436 8 27,984 Woodrow Wilson Foundation 5 13,000 (not offered) (not offered) The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3, 1997 96 F.-P. Dassault & A. Manseau internai report from NSF and NIH agencies and seven from five Canadian organizations (the three major councils previously mentioned: MRC, NSERC and SSHRC, and FCAR - Fonds pour la formation de chercheurs et l'aide à la recherche, and the IRSST - Institut de recherche en santé et en sécurité du travail). We found no such evaluation study done by the Ontario Graduate Scholarship Program. A recent study conducted for NSERC (1993) could not be used for our analysis, since it was an assessment of the effects of scholarships on the fellows during their studies, rather than an assessment of the consequences after this period. Using data from these internal reports and from the published studies, we conducted a comparative analysis.2 Although we carried out an extensive literature search for both American and Canadian programs, we obtained much more from the Canadian ones. This leads us to suggest that the evaluation of postgraduate scholarship outcomes is more of a preoccupation in Canada than in the U.S. We also examined the most recent general follow-up surveys of Quebec and Canadian postgraduate students, with or without scholarships, to identify which indicators were used to assess returns and to compare the findings on students in general to those on students having received a scholarship (Audet, 1991, 1993; Bastien, 1992; Statistics Canada, 1989, 1991, 1996). The amount of the scholarship does not vary greatly from one program to another, except in the case of special awards and postdoctoral studies. However, since the survey method varies from one study to another, one has to be cautious in comparing results. For example, the relative size of the sample (as compared to the potential population) is not available in some cases (MRC, 1989; NSERC, 1985; SSHRC, 1991), the years covered by the surveys vary from two to eleven years, and the time between the end of the period covered by the grant and the date of the survey also varies widely, from six months to ten years. Finally, response rates cannot always be assessed due to incomplete information on the sampling method. The IRSST (1986 and 1992) obtained the highest response rates (90% and 87%) with a sample size consisting of the total population of former fellows since the program began in 1981, while others had response rates that varied between 24% and 79% (Table 2). Our aim was to conduct a comparative analysis of the outcomes of master's and doctoral scholarship programs, using available data. In establishing a comparative base, we determined which indicators are The Canadian journal ofHigher Education Volume XXVII, Nos. 2,3,1997 Assessment of Postgraduate Programs 97 Table 2 Recent Surveys on Master's and Doctoral Scholarship Program Assessment: Survey Date, Years Covered, Research Sample, and Response Rate Agency and year of survey Years covered Sample of population (in %) Response rate of sample (in %) F C A R - 1989 1981-1982 60 75 I R S S T - 1986 1981-1986 100 90 I R S S T - 1992 1981-1992 100 87 M R C - 1989 1970-1979 n/a 62 NJSDHE - 1989 1978-1988 100 51 N S E R C - 1985 1975-1976 n/a 74 NSF - 1988 1967-1976 100 24 and 79 1 S S H R C - 1981 1972-1975 100 39 SSHRC - 19912 1981-1985 n/a 31 Note "n/a" means data not available ' This study used existing files to trace former fellows, and had data on 79% of those applying for NSF grants, and on 24% of those applying for NIH and ADAMHA grants; 2 Study conclusions not approved by the agency, but permission given to use raw data. most frequently used in assessing the outcomes of these programs. Then, we compiled survey results for the four most commonly used indicators of outcomes: awarded diploma, obtained job, obtained related job, and pursuing studies, and we developed a preliminary integrated indicator to obtain an overall value for comparing the results from different surveys. The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3,1997 98 F.-P. Dassault & A. Manseau In some cases, we calculated indicators from available raw data in order to obtain comparative results. We presented all indicators as ratios where the denominator is the survey's total number of respondents, except for an additional "related job" indicator where the denominator is the "obtained job" value (related job/obtained job) (Table 3). Table 3 Results of Surveys on Master's and Doctoral Scholarship Program Outcomes Agency & survey date FCAR - 1989 Awarded diploma M. Ph.D. (in%) 78 Obtained job 52 (in %) Related Job Pursuing total related4 studies obtained (in %) (in %) 77 64 83 12 2 IRSST- 1986 78' (M+Ph.D.) 62 46 74 35 IRSST- 1992 79 (M+Ph.D.) no n/a 82 (M+Ph.D.) 84 572 68 7 54 80 52 59 96 74 42 15 95 n/a 88 n/a 71 50' 93 n/a 5' n/a 25 37 MRC 1989 NJSDHE-1989 NSERC - 1985 NSF 1988 SSHRC - 1981 SSHRC-1991 3 99 no no no 92 73 75 50 79 57 90 88 Note "no" means there is no master's degree scholarship; "n/a" means data not available 1 Estimated by authors from available data 2 Related to a field of application, while in other surveys it is related to scientific discipline. 3 Raw data received from the,agency (SSHRC) without previous analysis ^ "Related job / Obtained job" ratio The Canadian journal of Higher Education Volume XXVII, Nos. 2,3,1997 Assessment of Postgraduate Programs 99 We did not distinguish between full- and part-time jobs or studies because this information was not always available. Indeed, the relative allocation of time between work and study was often overlooked, probably due to the fact that questionnaires are not yet fully adapted to reflect the complex working situation of recent Ph.D.s, who can sometimes be involved in several activities (research assistant, postgraduate student or research fellow, part-time teaching, etc.). Results Program Objectives Graduate scholarship program objectives are more often described in general terms, rarely underlining the precise results to be achieved. The most common formulation is: "to assist in the training of highly qualified . . .". The NSERC (1993) adds to this general objective the one of "attracting excellent students," and SSHRC (1991) and the I R S S T (1992) add the objective of meeting the work force demand for the expertise. FCAR (1990) is looking for reducing the duration of studies as well as the diploma award rate. Frequently Used Indicators Our results show that the most frequently used indicators are not linked to specific objectives. Table 4 is a compilation of the outcome indicators for master's and doctoral scholarship programs used in the different studies and reports that were analyzed. As can be seen, the most commonly used indicators were: awarded diploma, obtained job, obtained related job (in relation to previous studies), and pursuing studies. Duration of study and the diploma award rate at a given time are related indicators. With full scholarships, duration of study is generally expected to be equal to the time of the scholarship period, with the student obtaining a diploma at the end of the award period (including renewals). As a result, "awarded diploma" is the more widely used indicator. The awarded diploma rate gives the percentage of former fellows having received the diploma corresponding to the level of studies (master's or Ph.D.) for which the scholarship was awarded. Career development is subdivided into three elements: obtained job, obtained job related to the diploma, and pursuing studies. Former fellows could be pursuing The CanadianJournalofHigher Education Volume XXVII, Nos. 2,3, 1997 100 F.-P. Dassault & A. Manseau Table 4 Commonly Used Outcome Indicators of Master's and Doctoral Program Evaluations Indicators ++ used often; + used occasionally; o rarely used Comments ++ Awarded diploma (Table 3) Data easy to obtain ++ Obtained job (Table 3) Data easy to obtain, except on whether it is a part-time or a full-time job ++ Related job (to studies) (Table 3) Some interpretative variation ++ Pursued studies (Table 3) Data easy to obtain, except on whether it is part-time or full-time studies + Duration of studies (NSF-88, NJSDHE-89, SSHRC-81 & 91, FCAR-89. Related to "awarded diploma" + Actual position by: Economic sector (NSF-88; SSHRC-81 & 91; FCAR-89, MRC-89, NSERC-85); Job status (MRC-89, NSERC-85); and Scientific Discipline (SSHRC-81) Data easy to obtain, but usefulness depends on program objectives, and relative value of job status + Annual income (Audet 1991 & 1993, SSHRC-81, NJSDHE-89) Relative value vs. job context + Bibiometric indicators: publications communications, citations (FRCAR-89 MRC-89, NIH-89, SSHRC-91) Require extensive surveys and analysis relative value vs. job context and scientific field o Research funding and grants MRC-89, NSF-88) Relative value vs. job context o Related awards and promotion (MRC-89) Some interpretative variation and depends on career path. o Satisfaction and perception of former fellows (SSHRC-91) Could be unrelated to other program returns, and insufficent to assess returns The Canadianjournalof Higher Education Volume XXVII, Nos. 2,3,1997 Assessment of Postgraduate Programs 101 studies because they still have not been awarded the diploma corresponding to the scholarship, or because they are continuing with other studies, without being supported by the agency conducting the survey. Another indicator currently used in program evaluation studies is the employment sector (university, government, private) of former fellows. Some studies also specify the nature of the job (teaching, research, related professional or administrative position) and the selected fields of specialization. We did not use these indicators for comparative purposes because some programs focus on specific employment sectors, positions or disciplines, while others do not. Some studies present data on the annual income (Audet, 1991 and 1993; SSHRC, 1981; NJSDHE, 1989), research funding (NSF, 1988; MRC, 1989), professional status (MRC, 1989) or scientific publications and communications (MRC, 1989; NIH, 1989; SSHRC, 1991; FCAR, 1989) of former fellows. These data would be interesting for a comparative evaluation if they were generally available, although they would have to be used with caution since they are weighted differently according to the discipline and employment sector. Finally, one survey (SSHRC, 1991) measured the degree of satisfaction of former fellows and their perception of the effectiveness of the scholarship program. This survey obtained a high degree of satisfaction for factors such as "reducing time to complete" and "enhancing career opportunities," even though other factors such as "awarded diploma" and "obtained job" had the lowest rates (Table 3). Former fellows are satisfied even though comparative indicators show lower rates. However, Teichler (1991), in his assessment of a postdoctoral scholarship program, found that a fellow's good perception of a host lab and supervisor correlated with higher output results such as job status, papers published, and projects carried out. From these two cases, we cannot conclude that the satisfaction or perception of former fellows could be appropriate outcome indicators for scholarship programs. These indicators could be used to help improve relationships between students and the administrators of the programs, but would not be directly related to the assessment of a program's outcomes. In order to perform a comparative study of outcomes, we retain only the four most commonly used indicators: awarded diploma, obtained job, obtained related job, and pursuing studies. The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3, 1997 102 F.-P. Dussault & A. Manseau The "awarded diploma" indicator The NSERC-1985 survey obtained a high average rate (95%) of students graduating in natural sciences and engineering. However, these data were from fellows of ten years previously who therefore had had sufficient time to complete their studies, even if these were not completed by the end of the scholarship period. IRSST-1992 and NJSDHE-1989 are the only surveys that reached recent former fellows, six to twelve months after their scholarship period, and despite this short time lag, they achieved fairly high rates for "awarded diploma" (79% and 82% respectively) for students in a broad spectrum of disciplines. Despite a six-year time lag between the end of the scholarship period and the survey date, SSHRC-1991 explains their low 50% rate by the length of doctoral studies in their field (social sciences and humanities). Other surveys (FCAR, N S F and SSHRC-1981) obtained rates within the 50%-75% range, or had no such data available (MRC-1989). From these results, we observed that graduation rates may depend on a variety of factors such as the discipline, the study level (master's or Ph.D.), and the time lag between the scholarship period and the survey. However, comparing across all disciplines for Ph.D. students only, and for a 6 - to 10-year time lag between the scholarship period and the survey, full annual scholarship grantees (receiving more than $10,000 Cdn.) obtained a higher "awarded diploma" rate than students in general. D a t a on P h . D . f e l l o w g r a d u a t i o n r a t e s f r o m the F C A R - 1 9 8 9 , N S E R C - 1 9 8 5 , N S F - 1 9 8 8 , S S H R C - 1 9 8 1 and S S H R C - 1 9 9 1 surveys (which are 52%, 92%, 73%, 75% and 50% respectively; Table 3) are all higher than the 38% obtained in the Bastien (1992) survey for all Ph.D. students in Quebec (Canada), after their seventh year in the program. It is clear from these results that full annual scholarships are associated with a higher rate of degree obtention. One explanation is that such students can allocate more time to their studies, but another possible explanation is that students receiving a full scholarship are better ones, and consequently complete their studies more easily; however, we did not find any data comparing the academic marks of students with and without scholarships. Some authors have noted the importance of financial support at the graduate level, and particularly its effect in decreasing the time to completion (Sheridan & Pyke, 1994), but this effect is generally associated with some kind of financial support (internal funds, assistantships, scholarships, The Canadian Journal ofHigher Education Volume XXWI, Nos. 2,3,1997 Assessment of Postgraduate Programs 103 and others). Our study has stressed more specifically the major influence of scholarship granted from funding agencies. The "obtained job" indicator The "obtained job" indicator follows approximately the same pattern. Rates vary from 54% to 95% and may depend on the discipline, the study level (master's or Ph.D.), and the time lag between the scholarship period and the survey. NSERC's high result of 95% cannot be explained solely by the time lag (10 years), because it is almost the same time lag as in MRC-1989, which had the lowest rate. The difference between the results of these two surveys could be explained by the difference in discipline (natural sciences and engineering for the former, and medical sciences for the latter program) and/or the study level for which grants were awarded (MRC program is for the Ph.D. degree only). IRSST-1992 and NJSDHE-1989 obtained the second and third highest results (84% and 80%) for "obtained job" (Table 3). These programs are addressed to all disciplines and study levels, and both conducted their surveys within one year after the scholarship period. Other surveys obtained rates between 57% and 79%. Unfortunately, few data are available to compare these results with those for graduate students not having received a full scholarship. Statistics Canada (1991) has shown that 84% of all master's degree graduates and 92% of all doctoral graduates obtain a job within 2 years. Audet (1991, 1993) found similar results in Quebec with 80% and 90% for master's and doctoral graduates respectively. Data from scholarship programs rarely separate former fellows who have graduated from those who have not, in estimating the "obtained job" rate. Such data are only available in the SSHRC-1981 and the MRC-1989 surveys, and their "obtained job" rates are 82% and 93% respectively for their former fellows who have graduated, a finding which is rather similar to that for all graduates. It is interesting to note that more former fellows obtain jobs than diplomas, according to five of the seven surveys having data permitting this comparison (Table 3). Detailed data on the different job situations of students who do graduate and those who do not would be quite useful in assessing the importance of the diploma on job situation. The Canadian Journal of Higher Education Volume XXVII, Nos. 2,3,1997 104 F.-P. Dussault & A. Manseau The "related job" indicator Between 46% and 88% of former fellows have a job related to their studies, and if we look at the "related job'V'obtained job" ratio, this figure increases to between 68% and 96% (Table 3). This latter indicator gives a much better position for surveys having a low "obtained job" rate, such as MRC-1989 and SSHRC-1991 (it goes from rates of 52% and 50% to 96% and 88%). An explanation for this improvement could be that when it is hard to get a job (54% and 57%), it is easier to get a job related to studies (96% and 88%). The opposite relation exists for IRSST-1992 and NJSDHE-1989, where it is easier for former fellows to get a job (with rates of 84% and 80%) than to have a job related to their studies (68% and 74%). This relation does not depend only on the discipline, as NSERC-1985 is really outstanding on both indicators, obtaining the highest "obtained j o b " rate (95%) and a very high "related job'V'obtained job" ratio (93%). FCAR-1989, which relates to all disciplines, is in the middle range for "obtained j o b " as well as for the "related job'V'obtained job" ratio (77% and 83%). The lower "related job" rate observed in the IRSST survey may also be explained by the fact that the term "related" is interpreted here as being to a specific field of application (occupational health and safety), whereas for the other surveys, it refers to a scientific discipline. This more restrictive interpretation of the word "related", in the case of the IRSST survey, could be partly responsible for the lower rate obtained for this indicator. However, although former fellows may or may not have completed their degrees, they appear to perform better in obtaining a related job than all graduate students; 68% of master's and 76% of doctoral graduates had a job related to their studies, 5 years after graduation (Statistics Canada, 1996). The "pursuing studies" indicator Data from MRC-1989, SSHRC-1981 and SSHRC-1991 indicate that a large proportion (42%, 25% and 37%) of former fellows were still studying more than six to ten years after leaving the scholarship program. SSHRC explains this situation by the extra length of doctoral studies in their field (social sciences and humanities), reflected in their low "awarded diploma" rates (75% and 50%). For MRC, with 42% still studying more than ten years after being doctoral fellows, the questions The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3, 1997 Assessment of Postgraduate Programs 105 that remain to be answered are whether they are still pursuing their doctoral studies in their field (medical sciences), or are pursuing other studies, since the "awarded diploma" rate is not available. Consequently, the rate for this indicator could depend, as with other chosen indicators, on the discipline, the level of studies, and the time lag between the grant period and the survey date. I R S S T - 1 9 9 2 obtains the lowest rate in "pursuing studies" (7%). NSERC does not have data on this indicator, but we could assume that with 95% of former fellows having obtained a job, the rate of those purs u i n g s t u d i e s is p r o b a b l y a r o u n d 5% or l e s s . O t h e r s u r v e y s ( F C A R - 1 9 8 9 , I R S S T - 1 9 8 6 and N J S D H E - 1 9 8 9 ) have rates between 12% and 35% (Table 3). Developing an "integrated" indicator In an attempt to compare the different programs or eventually to evaluate one program over time by means of a single combined indicator, we developed a formula for this "integrated" indicator which we applied to some of the surveys that we have analyzed. This indicator was obtained by compiling, for each individual survey, the results for each of the four reviewed indicators, namely: awarded diploma (AD), obtained job (OJ), related job (RJ), and pursuing studies (PS). The AD, OJ and RJ indicators are assigned a positive value since they reflect the common broad objective of postgraduate scholarship programs to assist in the training of highly qualified personnel. The PS indicator is assigned a negative value because pursuing studies implies that former fellows have either not had their scholarship renewed due to poor performance, have gone beyond the scholarship period without completing their degree, have changed their field of study, or have switched to another granting agency. The last reason could be interpreted as a positive one if the student was awarded a more prestigious scholarship, but no data are available to document this possibility and, in any case, for five of the nine analyzed surveys (FCAR-1989, MRC-1989, NSERC-1985, SSHRC-1981 and SSHRC-1991), six to ten years after being a fellow is a rather long time to be still studying for any postgraduate diploma related to the awarded scholarship. To take into account the fact that a longer time period between the end of the scholarship and the survey date is usually associated with higher results for the four above-mentioned indicators, a negative time The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3,1997 106 F.-P. Dassault & A. Manseau lag (TL) variable was added to the formula used for compiling our integrated indicator. Since the maximum time lag period is ten years in the reviewed studies, we took this number as the limit, and assumed that this time lag has a linear effect on the results. The formula does not stem from any theoretical model, but is instead a practical attempt to compare different surveys. The formula is: I = (AD + OJ + RJ + (100 - PS) + (100 - (TL/10 yrs.) X 100)) / n where: n = the number of available indicators in a survey including time lag TL = time lag in years and all other values are in %: I = integrated indicator of performance, AD = awarded diploma, OJ = obtained job, RJ = related job, and PS = pursuing studies. We tested this formula with some survey results obtained previously and listed in Table 3. If we combine master's and doctoral degrees, we obtain the following rates: 66% for FCAR - 1989; 70% for IRSST1986; 83% for IRSST - 1992; 79% for NJSDHE - 1989; and 77% for NSERC -1985. It might be more appropriate to separate master's programs and doctoral ones, as the "awarded diploma" rate is normally higher for the master's degree. Unfortunately, only two surveys (FCAR 1989 and NSERC - 1985) offered such specific data. Only two organizations carried out surveys at two different times (IRSST and SSHRC). It would have been interesting to compare the improvement in the indicator rates of each program over time, but this is impossible since the survey methods are different: IRSST surveys used all the former fellows and SSHRC conducted their surveys on two different groups of former fellows (Table 2). An "integrated" indicator could be a better means of comparing returns from postgraduate scholarship programs, if there were more uniformity in survey methods. The Canadian journal ofHigher Education Volume XXVII, Nos. 2,3,1997 Assessment of Postgraduate Programs 107 Discussion Not only are there very few studies that have examined the outcomes of postgraduate scholarship programs, but these studies vary greatly in their survey methods; consequently, readily applicable guidelines cannot easily be extracted. Even applying the methods used for assessing undergraduate scholarships to postgraduate ones did not seem appropriate, since indicators such as access to university, choice of school, persistence, and choice of major are not readily applicable to postgraduate programs. Moreover, the economic outcome indicator estimating the higher income of former fellows has not been mentioned in any postgraduate survey. Our analysis of the studies on the assessment of postgraduate scholarship programs raised several methodological points which deserve attention, showing that a comprehensive comparative analysis of the outcomes of graduate scholarship'programs is limited by the use of available data. Indeed, three of the nine fully usable studies in our analysis (Table 2) did not clearly identify the size of the sample and the size of the potential population. Six studies have a time lag of more than six years between the time of the survey and the scholarship period, thus reducing the ability to evaluate the direct effects of these programs. It is also important to collect periodic data on each indicator, taking into account career evolution and the incompatibility of comparing former fellows of one year ago to those of ten years ago. Teichler (1991) observed that career paths change as researchers get older, with senior researchers becoming more involved in management positions. With longitudinal studies, in which former fellows are surveyed over several years, it would be possible to define how indicator rates change over time, as careers evolve, and also to introduce economic outcome indicators that follow the evolution in wage income and job quality. Furthermore, by using control groups for these longitudinal studies, the outcomes of postgraduate studies with and without scholarships could be compared. By evaluating the perception that former fellows have of the effectiveness of their scholarship program, we could focus our attention on other potential indicators and extend the study to other major actors in the program (student directors, program officers, etc.). A high degree of satisfaction on the part of the major actors is important for the good reputation and effectiveness of the program, but perceptions also need to be supported by documented facts. Furthermore, these indicators The Canadian journal ofHigher Education Volume XXVII, Nos. 2,3, 1997 108 F.-P. Dassault & A. Manseau could be used in an ongoing evaluation during the scholarship period, providing additional data that could help in improving program performance and outcomes. Conclusion Our results suggest that postgraduate scholarship programs may contribute to an increase in the rate of awarded diplomas, although the effect of selecting the best students for such programs remains to be verified. However, these programs do not appear to have much influence on the rate of employment, but their effect on income and job quality could be interesting to assess in the future. The complexity of the education-towork transition is far from being capture. Finally, the rates obtained for all chosen indicators may well depend on the discipline, the level of studies, and the time lag between the scholarship period and the survey date. We have pointed out some differences in the three major fields of discipline (medical, social sciences and humanities, and natural science and engineering), but we have been limited in doing so because many results are not broken down by discipline. It is the same situation for the level of studies. We pointed out that the most commonly used indicators were: awarded diploma, obtained job, obtained related job, and pursuing studies. Other indicators, such as bibliometric ones (publications, citations, etc.), personal income, and responsibilities, may not be useful in a comparative analysis since they are weighted differently according to the discipline and employment sector, but may be useful in assessing performance in relation to some specific objectives of the programs. We have seen that postgraduate scholarship programs may achieve their general objective of assisting in the training of highly qualified personnel. However, an identification of more precise objectives would be a prerequisite for a better performance assessment. Most of the commonly used indicators are not linked to specific objectives and we showed that many methodological pitfalls limit the use of these indicators in performing a broader comparative analysis. With the growing need of society to provide an economic justification for its own structures, the objectives of postgraduate scholarship programs will have to be more precisely defined, and criteria for assessing the performance and outcomes of these important programs will have to be developed. In this regard, the different organizations involved The Canadian journal ofHigher Education Volume XXVII, Nos. 2,3,1997 Assessment of Postgraduate Programs 109 in delivering and/or assessing scholarship and education programs would need to increase their information-sharing and possibly develop some common guidelines for assessing these p r o g r a m s . ^ References Audet, M. (1991). Qu'advient-il des diplômés et diplômées universitaires? (What has happened to university graduates?). Québec, PQ: Les Publications du Québec (Ministère de l'Enseignement Supérieur et de la Science du Québec). Audet, M. (1993). L'intégration au marché du travail des titulaires d'un doctorat. (Job market integration of people with doctoral degrees). Québec, PQ: Les Publications du Québec (Ministère de l'Enseignement Supérieur et de la Science du Québec). Bastien, A. (1992). Cheminement des étudiantes et étudiants au doctorat. (Progress and follow-up of doctoral students). Québec, PQ: Les Publications du Québec (Ministère de l'Enseignement Supérieur et de la Science du Québec). Beck, N.E. (1985). Financial aid today: An economic perspective. College Board Review, 137, 2-5 & 29-30. FCAR (Fonds pour la formation de chercheurs et l'aide à la recherche). (1990). Relance des candidates et candidats à une bourse du Fonds FCAR (19811982). (Survey of FCAR former scholarship candidates). Québec, PQ: Gouvernement du Québec. Fogarty International Center. (1989). The scientific and professional accomplishments of former international research fellows of the Fogarty International Center, NIH: A review of the last three decades. Report prepared by S.J. Fitzsimmons. No. FIC 85-104, Washington, DC: National Institute of Health. Golding, J.M., Lang, K„ Eymard, L.A., & Shadish, W. (1988). The buck stops here: A survey of the financial status of PhD graduate students in psychology. American Psychologist, 43, 1089-1091. Institute of Medicine. (1986). The career achievements of NIH postdoctoral trainees and fellows. Report prepared by Garrison, H.H., & Brown, P.W. Washington, DC: National Academy Press. IRSST. (Institut de recherche en santé et en sécurité du travail du Québec) (1986). Programme de formation de chercheurs: Bilan 1981-1986 et éléments de prévision 1987-1990 (Researcher training program: Review 1981-1986 and forecasting elements for 1987-1990). Montréal, PQ: Working paper. The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3, 1997 110 F.-P. Dussault & A. Manseau IRSST. (Institut de recherche en santé et en sécurité du travail du Québec) (1992). Le programme de bourses d'études supérieures en santé et en sécurité du travail de l'IRSST; 12 ans d'impact (The IRSST postgraduate scholarship program; 12-year impact). Montréal, PQ: Internal Report. MRC (Medical Research Council of Canada). (1989). Evaluation of MRC programs for training biomedical (Ph.D.) scientists. Ottawa, ON: Internal Report. NACUBO (National Association of College and University Business Officers). (1995) Chronicle. NJSDHE (New Jersey State Department of Higher Education). (1989). The Garden State graduate fellowship program: A survey of current and former fellows, 1978-1988. Report prepared by L.K. Berkner & C. Lee. Trenton, NJ: New Jersey State Department of Higher Education, Cat. No. 540. NSERC (Natural Sciences and Engineering Research Council of Canada). (1985). Final report on the evaluation of NSERC's highly qualified manpower programs. Internal report prepared by Currie, Coopers & Lybrand Management Consultants, Ottawa, ON. NSERC (Natural Sciences and Engineering Research Council of Canada). ( 1993). Final report on the evaluation of the scholarships and fellowships programs of the NSERC. Prepared by ARA Consulting Group Inc., Ottawa, Canada. NSF (National Science Foundation). (1988). Early career achievements of National Science Foundation graduate fellows, 1967-1976. Technical Paper prepared by J. Snyder,Washington, DC. NSF (National Science Foundation). (1988). Career progression of NATO postdoctoral fellows. NSF Report 88-130 prepared by J. Maher & D. Williams, Washington, DC. Sheridan, P.M., & Pyke, S.W. (1994). Predictors of time to completion of graduate degrees. The Canadian Journal of Higher Education, XX/V(2), 68-88. SSHRC (Social Sciences and Humanities Research Council of Canada). (1981). Five years later: Doctoral fellows in the labor market. Internal Report, Ottawa, ON: SSHRC. SSHRC (Social Sciences and Humanities Research Council of Canada). (1991). Evaluation of the SSHRC doctoral fellowship program. Prepared by The Angus Reid Group. Internal Report, Ottawa, ON. St. John, E.P.(1990). The impact of student financial aid: Review of recent research. Paper presented at the Annual Meeting of the Mid South Education Resources Association, New Orleans, LA. Statistics Canada. (1996). The class of 86 revisited. Statistics Canada. (1992). The class of 1986. Statistics Canada. (1991). The class of 82 revisited. The Canadian Journal ofHigher Education Volume XXVII, Nos. 2,3,1997 Assessment of Postgraduate Programs 111 Supply and Services Canada. (1985). Services au public: Éducation et recherche, (Public Services: Education and Research). Report prepared by Programs Evaluation Teamwork Study Group. Ottawa ON, Cat. No. CP3250/16. Teichler, U., Lewinson, G., & Massino, L. (1991). Research fellowships: Surveys of European community transnational research fellows. Research Evaluation, 1, 137-147. Notes ' We reviewed the literature on graduate scholarship program evaluation using several Canadian and American data bases: CISTIMON from the Canadian Institute on Scientific and Technology Information, NLCATBN from the National Library of Canada, MICROLOG from the Canadian government and public agencies, Psychinfo from the American Psychological Association, ERIC from the Educational Resources Information Center, and NTIS from the National Technical Information Service of the U.S. Department of Commerce. These data bases cover most American and Canadian publications on education research, as well as reports from government and nonprofit organizations. ^ Internal reports have been graciously supplied with verbal agreement to use raw data. The agencies involved have not reviewed our interpretation of their data in this study. The Canadian Journal of Higher Education Volume XXVII, Nos. 2,3, 1997