T h e Canadian Journal of Higher Education, Vol. X X I - 3 , 1991 La revue canadienne d'enseignement supérieur, Vol. X X I - 3 , 1991 G r a n t i n g A g e n c y C r i t e r i a for A w a r d i n g G r a d u a t e Research Scholarships MICHAEL O'BRECHT* and ROBERT O. PIHL1" Abstract Eleven Canadian research granting agencies were surveyed concerning the weights that they apply to various criteria when reviewing applications for graduate student awards. Nine agencies responded, seven providing useful data. The undergraduate academic grades of applicants for research training awards were found to carry high weight in selection decisions, especially in competitions that were either large or were open to students at the master's level. However, the literature on early career indicators suggests that the undergraduate academic achievement of graduate students is not an effective predictor of their future research productivity. Strategies through which agencies might improve the use of more predictive criteria, such as applicant characteristics and quality of the graduate training environment, are discussed. Résumé Onze organisations subventionnaires canadiennes ont fait l'objet d'une enquête concernant le poids qu'elles attribuent à divers critères dans leur évaluation de demandes de bourses. Neuf questionnaires ont été retournés; sept d'entre eux ont pu être utilisés. Les notes de premier cycle obtenues par les candidats de bourses de recherche revêtaient une grande importance dans les décisions, surtout dans les concours ouverts aux étudiants de deuxième cycle. Cependant, la littérature sur les indicateurs précoces d'une carrière scientifique suggère que les notes de premier cycle obtenues par les étudiants gradués ne sont pas un bon indicateur de leur productivité future en recherche. Les stratégies susceptibles d'augmenter l'utilisation des critères plus fiables, comme les caractéristiques des candidats et la qualité de leur environnement de recherche, sont discutées dans cet article. Coordinator of Program Evaluations, Medical Research Council of Canada, Ottawa. t Professor, Department of Psychology, McGill University. 48 Michael O ' B r e c h t and Robert 0 . Pihl Introduction Research scholarships provide graduate students with both financial support and societal recognition of the importance of their work and are vital to the development of a national cadre of highly qualified research personnel. For 47% of Ph.D. graduates holding appointments in health science faculties, research scholarships were the principal source of support during graduate study (MRC, 1988a). The importance of graduate research scholarships in the development of a nation-wide research capability justifies careful examination of the criteria by which they are awarded. This study presents information on the weightings that granting agencies assign to various selection criteria and discusses the appropriateness of those weightings given the theoretical predictive strength of each criterion. Criteria often used in the selection of graduate student award winners include the undergraduate academic grades of applicants, assessments by sponsors, the quality of the proposed graduate research project and training environment, and the applicants' traits and research experience. The validity of some of these criteria is questionable. The correlation between the undergraduate academic grades of graduate students and their eventual research productivity is very low (Clark and Centra, 1982; O'Brecht, Pihl and Bois, 1989). While students with high grades may be more likely to obtain admission to prestigious universities, and graduates of prestigious universities may be more likely to end up in careers where research is a major activity, the link between undergraduate academic performance and career productivity is tenuous at best. Since the range of u n d e r g r a d u a t e grades in a graduate student population is very narrow, essentially restricted to A and B records, it is not surprising that grades are a poor discriminator of research potential in an applicant pool comprised of graduate students. Sponsors' assessments of applicants for graduate awards also appear to be poor predictors of the level of post-training research activity. This is possibly because sponsors' assessments are highly correlated with applicant's undergraduate academic performance (Davis, 1965; O'Brecht, et al., 1989). The appropriateness of assessing the quality of the graduate research proposal when reviewing applications for research training awards has received little attention in studies of research career indicators. However, a positive correlation between number of publications resulting from graduate work and post-training research productivity suggests that the scientific import of the graduate research project is a relevant criteria. Further, many studies have found a positive correlation between the quality of the graduate training environment and subsequent Granting Agency Criteria for Awarding Graduate Research Scholarships 49 research activity (Folger, Astin and Bayer, 1970; Clemente, 1973; and Zumeta, 1983). In the health sciences, the research orientation of the immediate environment for graduate training appears to be especially critical (O'Brecht, et al., 1989). That is, students training under active researchers in an atmosphere where research is highly valued and often talked about, where travel to scientific meetings is expected and f r e q u e n t , where guest lecturers and postdoctoral trainees abound, are more likely to become productive researchers than students who train in less stimulating environments. A number of personality traits also appear to be correlated with research productivity. Independence, critical ability and determination all seem to be good predictors (O'Brecht, et al., 1989). Pre-university interest in science also showed a positive correlation with research career outcome in a population of biological scientists (Segal, Busse and Mansfield, 1980). Research experience during u n d e r g r a d u a t e study was f o u n d to bear a s i g n i f i c a n t c o r r e l a t i o n with post-training research activity in a sample of health professionals (O'Brecht and Bois, 1988). However, in a population of graduate students in the health sciences (a population which includes persons with science, social science, engineering or arts undergraduate degrees) research experience during undergraduate study does not appear to be an important predictor, possibly because undergraduate research awards tend to be distributed on the basis of academic course results (O'Brecht, et al., 1989). In summary, the literature on early indicators of research careers would suggest that agencies wishing to identify graduate students who are likely to become productive researchers should focus their attention on the applicants' traits, interests and training e n v i r o n m e n t . The p r e s e n t study p r o v i d e s information on the current selection practices of a variety of granting agencies in both the public and private sectors. Method Selection of Sample The population of interest included agencies with a specific mandate to support research (e.g., research granting councils). Programs intended to support graduate study under an educational mandate (e.g., provincial graduate scholarships programs) were not included. The Reference List of Health Science Research in Canada and other public directories were examined to identify granting agencies that operate graduate student award programs. The sample included three private non-profit agencies (the Heart and Stroke Foundation of Canada, the National Cancer Institute of Canada and the Muscular Dystrophy Association of Canada), four provincial 50 Michael O ' B r e c h t and Robert 0. Pihl agencies (the Alberta Heritage Foundation for Medical Research, the Fonds de recherche en santé) du Québec, the Ontario Mental Health Foundation and the Ontario Ministry of Health), and four federal agencies (the National Health Research and Development Program, the Medical Research Council, the Natural Sciences and Engineering Research Council, and the Social Sciences and Humanities Research Council). Collection of Data Q u e s t i o n n a i r e s , in both English and French, were developed to solicit information from agency managers on the number of applications received, awards granted, procedures for reviewing applications and weightings assigned to various selection criteria. Separate data were collected for programs open to master's students and those restricted to doctoral students. Questionnaires were mailed in August of 1989 and agencies that had not responded by October were telephoned and their participation again requested. Agencies were asked to complete the questionnaire and to provide copies of their application material as well as any documentation on the process by which they select graduate student award winners. Definition of Data The six categories of selection criteria presented in the questionnaire were as follows: 1) quality of the graduate training environment, a category that included such variables as the productivity of the a p p l i c a n t ' s graduate supervisor, the reputation and research funding of the training laboratory and opportunities for collaboration with other trainees and investigators; 2) the quality of the project proposed by the applicant and the level of his/her input into d e v e l o p m e n t of the p r o j e c t ; 3) traits of the a p p l i c a n t , i n c l u d i n g characteristics such as the candidate's independence, determination, interest in research, and so forth; 4) academic achievement of the applicant, primarily reflecting grades achieved in university courses; 5) research experience of the applicant, especially the extent to which the applicant's undergraduate (or early graduate) research experience is considered in the selection process; and, 6) other, a free response category to ensure that agencies would report all selection criteria. Results Nine of the research granting agencies agreed to participate in the study and useful data were received from seven. (One of the provincial agencies had Granting Agency Criteria for Awarding Graduate Research Scholarships 51 recently closed its program to new applicants and a federal government agency was not able to estimate criteria weights because of the nature of its review process.) A statistical profile of the seven agencies appears in Table 1. Table 2 shows the data provided by seven granting agencies in response to the question on weighting of each criterion. (Two agencies, A and D, preferred to group several criteria together when providing estimates.) Results indicate that assessment of the applicant's personal file (traits, grades obtained and experience) carries approximately 70% of the weight in a typical selection, while a s s e s s m e n t of the p r o p o s e d g r a d u a t e research training (training environment, research project and other) receives a weight of about 30%. However, there is wide variation among agencies in the relative importance attributed to assessment of the individual and the project. At one extreme, a small agency assigns only a 40% weight to assessment of the applicant; at the other extreme, a large agency assigns 100% weight. Academic marks tend to be the most important criterion. For awards programs that are open to master's students, course results account for about 68% of the r a t i n g . A c a d e m i c grades r e m a i n an i m p o r t a n t criterion in competitions restricted to Ph.D. students, but carry noticeably less weight (47%) than in competitions open to master's students. In competitions restricted to Ph.D. students, more weight is placed on the personal traits of candidates and the quality of research proposals. These differences are illustrated in Figure 1. As an examination of the raw data suggested that larger agencies attribute more weight to academic grades than do smaller agencies, correlations between size of agency (number of applicants) and criteria weights were computed. The results appear in Table 3. Data confirm a positive correlation between agency size and the amount of emphasis placed on academic course results. In comparison with smaller agencies, larger agencies will tend to assign more weight to the applicant's grades and less weight to the research project. As the intent of the data collection exercise was primarily to expose the range of weights applied to criteria by the various agencies, and since the number of agencies in the study was not large, further statistical analysis of the data was deemed inappropriate. Discussion Selection Theory and Agency Practice Research granting agencies may pursue a variety of objectives through their graduate awards programs. The awards may be intended to recognize academic excellence, to produce master's or doctoral students or to support the training of future researchers. However, regardless of the stated program objectives, or lack Table 1 Profile of Agencies that Provided Data on Weightings of Selection Criteria Characteristics Agency A B C D E F G Sector 1 P-n-p P-n-p Prov Prov Prov Fed Fed Programs^ M/PhD PhD M;PhD M/PhD M/PhD M;PhD M/PhD Applicants^ 122 44 302; 189 38 188 2545; 1174 877 Award rate^ 19% 11% 9% 21% 29% 49% 20% External Prescreening^ none unknown unknown minor minor major none Internal Prescreening^ no yes yes no no no no Interviews no yes no no no no no 1: P - n - p , Private-non-profit; Prov, Provincial government; Fed, Federal government. 2: M/Ph.D., program open to master's or doctoral students; M, open only to master's degree students; PhD, open only to doctoral students. 3: Total number of applicants for each program. 4: Overall approval rate for the agencies' graduate award programs. 5: External pre-screening of applications: minor, agencies report that university departments may screen applications before they are submitted for consideration; major, universities review applications and submit a ranked list. 6: Internal pre-screening of applications, i.e., a two-stage selection process within the granting agency, an initial screening followed by a final selection. Granting Agency Criteria for Awarding Graduate Research Scholarships 53 Table 2 Weighting of Various Criteria in the Selection of Graduate Research Award Recipients Criterion' (by program type) Agency and Percentage Weights for Criteria A C D E F 16.7 16.7 B G _ 20 15 0 0 5 0 10.4 7.9 10 80 0 10 5 90 0 0 9.2 67.5 2.5 2.5 0 5 28.8 95 71.2 Average Weight Programs Open to Masters Students (1) Environment (2) Project (3) (4) (5) (6) Traits Academic Experience Other 40 20 _2 - 0 16.7 50 0 0 - 5 50 10 0 60 33 40 35 (3), (4), (5) and (6) 3 40 67 60 65 (1) and (2) - - 100 Programs Open Onlv to PhD Students (1) (2) (3) (4) (5) (6) Environment Project Traits Academic Experience Other 10 20 25 35 5 5 25 25 25 25 0 0 0 0 10 80 0 10 11.7 15.0 20.0 46.7 1.7 5.0 ( D a n d (2) (3), (4), (5) and (6) 30 70 50 50 0 100 26.7 73.3 1: 2: 3: Environment, graduate training environment; project, graduate research project; traits, personal characteristics of applicant; academic, academic grades; experience, previous research experience. The agency did not estimate weights for the specific criterion but did estimate the weight for a grouping of criteria that included the criterion. "Other" criteria related to assessment of applicant qualities. In one case "other" was a small score related to the impact of an interview with the candidate; in another c a s e , " o t h e r " was the a s s e s s m e n t and r a n k i n g p r o v i d e d by the c h a i r of the applicant's department of major study. 54 Michael O'Brecht and Robert O. Pihl Table 3 C o r r e l a t i o n C o e f f i c i e n t s B e t w e e n S e l e c t i o n C r i t e r i a a n d S i z e of C o m p e t i t i o n Criterion Correlation ^ with Size of Competition by Program Type Open to Master's Students (max n=6) -.77 -.81 -.02 .65 -.49 .95 -.81 .80 1) Training Environment 2) Research Project 3) Applicant Traits 4) Academic Grades 5) Research Experience 6) Other Proposal (1 and 2) Applicant (3, 4, 5 and 6) PhD Students Only (n=3) -.73 -.99 -.99 .95 -.60 .80 -.86 .86 1 : R values calculated using the correlations procedure of S P S S - P C . Figure 1 Selection Criteria a n d W e i g h t s W e i g h t in S e l e c t i o n P r o c e s s ( % ) 8 0 >1 70V !] M a s t e r ' s level * (n=4) jDoctorallevel CZ71 (n=3) 60 501/ 40./ 30./ 20./ 10./ <171 ¿7 GRADES ENVIRONMENT PROJECT * Master's or master's/doctoral ** Research experience and "other" TRAITS MISC.** Granting Agency Criteria for Awarding Graduate Research Scholarships 55 thereof, it is reasonable to assume that research granting agencies will be interested in supporting students who are likely to become active researchers following their graduate and postdoctoral training. For example, in a survey of panel chairmen and program administrators concerning the graduate Studentship program of the Medical Research Council, the objective "to support the training of future researchers", although not formally stated in the program guidelines, received the highest mean rating of importance of five statements of program objectives (Medical Research Council, 1986). Findings from the research literature would suggest that agencies which are interested in identifying and supporting future researchers should not restrict their review of applicants to a consideration of undergraduate academic records but should also carefully consider the personal traits of applicants and their graduate training environment. In practice, research funding agencies weight undergraduate academic grades more heavily than other criteria. For all but one of the agencies in this study, academic grades carried the largest weight in the selection process. At least two factors influence the weight that awards committees attach to academic grades: the level of study of the applicants and the number of applications to be reviewed. First, if the awards competition is open to master's level students, reliance on academic grades will tend to be high. Candidates for master's level awards, often applying while still in their undergraduate program are, arguably, not in a good position to develop a sophisticated research proposal. In fact, some may not have finally decided upon the location for their graduate work. In addition, undergraduate students have not had a full o p p o r t u n i t y to d e m o n s t r a t e traits that r e l a t e to r e s e a r c h p r o d u c t i v i t y (independence, critical ability and determination). In effect, inadequate data on the graduate research environment and applicant traits are a probable cause of heavy reliance on grades in the assessment of applications from students at the master's level. For awards competitions that are restricted to doctoral degree students, assessment of the applicants' research proposals and research-related traits becomes more feasible. A second influence on the weighting of academic grades in the award selection process is the number of applicants that must be reviewed. The larger the number of applications, the greater the reliance on academic marks as a selection criterion. Decisions based primarily on academic grades, as compared with decisions that take into account the individual qualities and training situations of applicants, can be made quickly and efficiently and yet will appear to be fair and rational. It is therefore not surprising that agencies which must review a large number of applications will tend to assign high weight to the academic achievements of candidates. 56 Michael O ' B r e c h t and Robert 0. Pihl Importance of Optimal Weighting of Selection Criteria The number of Canadian students now in or entering the science and technology research training pipeline appears to be insufficient to meet expected future demand (Natural Sciences and Engineering Research Council, 1989; Medical Research Council, 1988b). Data on the projected balance between the supply and demand of scientists in the United States indicate a future undersupply that should be quite evident by 1995 and of major proportion by the turn of the century (Atkinson, 1990). A shortage of scientists in the U.S. would probably worsen the problem in Canada. If Canadian and U.S. predictions of demand for research personnel are accurate, the identification of the most promising future scientists within pools of applicants for graduate student awards becomes an increasingly important exercise. The effectiveness of research training programs is directly dependent on the quality of awardee selection processes. A less than optimal selection process implies that some resources will be awarded to trainees who are unlikely to attain program objectives, while better prospects, or applicants to other programs, must be turned down for lack of funds. Agencies that proclaim the pursuit of excellence as their mission must regularly examine the relevance of the selection criteria that underlay their concept of excellence. The literature suggests that excellence in one area (e.g., academic course performance) does not necessarily correlate with excellence in another (e.g., post-training research activity). Improving the Selection Process The research trainee selection process may be improved through a variety of measures: agencies may assign more weight to criteria that appear to predict desired program outcome; application forms may be revised to collect better information on predictor variables; and, the structure of programs or selection processes may be changed to facilitate better use of effective predictors. Agencies should critically examine the criteria used in their selection of graduate student award winners. There is a tendency to assume that historical use of predictors validates their effectiveness. Once an agency recognizes that criteria may be added, reweighted or dropped, selection processes may be rapidly improved. For example, in the mid 1980s, Agency C completely revised its graduate student selection criteria, using an intuitive approach rather than studying the research literature on early career indicators, but nevertheless developing criteria weights that closely approximate theoretical optima. From an almost total reliance on academic grades, the agency moved to a selection Granting Agency Criteria for Awarding Graduate Research Scholarships 57 process in which that criterion accounts f o r only half of the score for applications from master's students and only one quarter of the score for applications from Ph.D. students. A better information base on relevant criteria should also improve the quality of selection committee decisions. To enable an assessment of the graduate training environment for Ph.D. degree students, application forms should collect data on the graduate research supervisor: productivity; funding; and, trainees. To obtain complete data on the student's research proposal, agencies should request clarification of authorship (applicant or research supervisor), indicate the points to be covered in the project description, and provide enough space for a full description of the proposed work. Assessment of applicant traits is particularly challenging because sponsors' ratings of the personal characteristics of undergraduate students will tend to be influenced by their perception of the student's academic performance. Agencies should consider various techniques to enhance the quality of data on applicant characteristics. Sponsors' assessment forms should be carefully structured. The forms should ask for data only on key research-related traits: critical judgment, independence, determination, and interest in research. Presentation of a long list of traits (the typical assessment form lists seven) may encourage a superficial response. For each trait, there should be a separate rating scale and a space for sponsors to state the reason for their rating. Also, agencies should consider obtaining reasonably extensive (e.g., one page) statements from candidates about their interests and career goals. The present study suggests that when agencies use one program to support both master's and doctoral students, the selection process tends to favour undergraduate grades, a criterion which is even less appropriate for doctoral than for master's students. Agencies should operate separate competitions for students with no graduate experience and those that have already completed some graduate work. For large agencies, the logistics of reviewing hundreds or even thousands of applications may prohibit a detailed examination of each case. These agencies should adopt a pre-screening procedure, eliminating on a first round of review t h o s e a p p l i c a t i o n s which are c l e a r l y not c o m p e t i t i v e . T h e r e m a i n i n g applications should then be assessed using all criteria found to be predictive of the desired program outcome. 58 Michael O'Brecht and Robert 0. Pihl Conclusion T h e r e a p p e a r s to be c o n s i d e r a b l e o p p o r t u n i t y f o r a g e n c i e s to e n h a n c e the e f f e c t i v e n e s s of their g r a n t i n g p r o g r a m s b y i m p r o v i n g selection p r o c e s s e s . Agencies m a y revise criteria, adjust importance weights, r e w o r k application forms or stream a p p l i c a n t s to s h a r p e n t h e j u d g m e n t p r o c e s s a n d t h u s i n c r e a s e the p r o b a b i l i t y of s u p p o r t i n g i n d i v i d u a l s w h o are m o s t likely to a c h i e v e program objectives. References Atkinson, R.C. (1990). Supply and Demand for Scientists and Engineers. A National Crisis in the Making. Science, 248 (April 27, 1990), 425^132. Clark, M.J., and Centra, J. (1982). Conditions ofPh.Ds Influencing the Career Accomplishments (GRE Board Report 76-2R). Princeton, NJ: Educational Testing Service. C l e m e n t e , F. (1973). Early career determinants of research productivity. Journal of Sociology, American 79 (2), 409-419. Davis, J.A. (1965). What college teachers value in students. College Board Review, 56, 15-18. Folger, J.K., Astin, H.S. and Bayer, A.E. ( 1 9 7 0 ) . Human Education. Resources and Higher New York: Russell Sage Foundation. Medical Research Council of Canada. (1986). P re-evaluation the training of health professionals report: MRC programs for in research. Ottawa: MRC. Medical Research Council of Canada. (1988a). Impact of MRC programs on the training of university faculty. Ottawa: MRC. Medical Research Council of Canada. (1988b). Research training supervised by MRC grant holders. Ottawa: MRC. Natural Sciences and Engineering Research Council. (1989). Graduate Students in Demand. Contact, 14 (4), 1-3. O'Brecht, M. and Bois, P. (1988). Characteristics of active and less active researchers in a group of Canadian health professionals. Journal of Medical Education, 63 (4), 327-329. O ' B r e c h t , M., Pihl, R.O. and Bois, P. (1989). Criteria for granting research training awards to graduate students. Research in Higher Education, 30 (6), 647-664. Segal, S.M., Busse, T.V., and Mansfield, R.S. (1980). The relationship of scientific c r e a t i v i t y in t h e b i o l o g i c a l s c i e n c e s to p o s t d o c t o r a l a c c o m p l i s h m e n t s a n d experiences. American Educational Z u m e t a , W. ( 1 9 8 3 ) . Extending Postdoctoral Research Journal, the Educational 17 (4), 491-502. Ladder: The Changing Role of Training in the United States (Report to National Science Foundation on Grant SRS 80-11953).