What is Your Degree Worth? / B. Fenesi & F. Sana 440 CSSHE SCÉES Canadian Journal of Higher Education Revue canadienne d’enseignement supérieur Volume 45, No. 4, 2015, pages 440 - 456 The Efficacy of Key Performance Indicators in Ontario Universities as Perceived by Key Informants Vivian Chan Ryerson University Abstract The Ontario Ministry of Education and Training’s Task Force on University Accountability first proposed key performance indicators (KPIs) for colleges and universities in Ontario in the early 1990s. The three main KPIs for Ontario universities are the rates of (1) graduation, (2) employment, and (3) Ontario Student Assistance Program loan default. This exploratory and descriptive study examined the perceptions of 12 key informants from 11 participating universities about the efficacy and effectiveness of these KPIs. The results of this study demonstrate that a clear majority of participants believe these KPIs are not having the intended impact. This paper analyzes the evidence and makes recommendations designed to foster efficient collaboration between stakeholders; it also asks all parties to clarify their goals, agreed expectations, and requirements, in order to develop effective measures of institutional performance and accountability and address the political needs of the government, the universities, and the public. Résumé Au début des années  90, le Groupe de travail sur la responsabilisation des universités du ministère de l’Éducation et de la Formation de l’Ontario a d’abord proposé des indicateurs de performance pour les collèges et les universités de l’Ontario. Ces trois principaux indicateurs clés de performance (ICP) pour les universités ontariennes sont (1) les taux d’obtention des diplômes, (2) les taux d’embauche et (3) les taux de défaut de remboursement des prêts du Régime d’aide financière aux étudiantes et étudiants de l’Ontario (RAFEO). Cette étude exploratoire et descriptive a examiné la perception CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 441 de 12 répondants clés (provenant de 11 universités participantes) envers l’efficacité de ces ICP. Les résultats de cette étude démontrent qu’en majorité absolue, les participants croient que ces ICP n’ont pas l’effet envisagé. Ce rapport analyse les preuves et émet des recommandations ayant pour but de favoriser une collaboration efficace entre les intervenants. Il demande aussi à chaque partie de clarifier les objectifs, les attentes convenues et les exigences établies dans le but d’élaborer des mesures efficaces de la performance et de la responsabilisation des établissements, ainsi que d’adresser les besoins politiques du gouvernement, des universités et du public. Introduction The Ministry of Education and Training’s Task Force on University Accountability first proposed performance indicators for colleges and universities in Ontario in the early 1990s (Ministry of Education and Training, 1993). The decision to develop and use key performance indicators (KPIs) was made that same decade, to help students reach informed program decisions (MET, 1998). The required KPIs for the Ontario universities were graduation rates, employment rates, and Ontario Student Assistance Program (OSAP) loan default rates. The research project behind this paper studied the impact of KPIs on a large group of Ontario universities—specifically, the efficacy of KPIs in the context for which they were adopted, 10 years following their implementation, as “key informants” saw them. This investigation further sought to identify areas of concern among the participants and to solicit suggestions for either altering or improving the KPIs’ effectiveness. The informants from the Ministry of Education and Training’s Task Force on University Accountability, known as the Broadhurst Task Force (named after the task force’s chair, Dr. William Broadhurst), included a member appointed by the ministry, a Council of Ontario Universities (COU) representative, and an observer/consultant of the task force. This paper presents a report based on the responses to online survey statements and one-on-one follow-up interviews with key informants from the participating universities. Collectively, the results provide a measure of the strength of the respondents’ agreement or disagreement with a series of statements on a four-point, forced-choice response scale (from strongly disagree to strongly agree). The Ministry of Training, Colleges and Universities (MTCU) has declared that the KPIs are published to enable parents and students to make an informed choice of program and institution for the student’s post-secondary education, to help universities evaluate the quality of programs, and to improve the universities’ performance (MTCU, 2012). Having examined the history of why and how KPIs were introduced in Ontario, I was motivated to research the following lines of inquiry: (i) to assess whether these KPIs are having the intended effect and (ii) to reflect on whether changes could improve their effectiveness. As this paper will demonstrate, the findings of the study provide a better understanding of the impact of the KPIs, as well as suggestions for alternative models and/or adjustments to improve future performance. The collected data may also be valuable resources to inform evidence-based MTCU policy decisions that will directly affect Ontario’s universities and their stakeholders. If the KPIs were initially effective gauges after the collection of KPIs was imposed in September 1999, are they still? Is there a need to re-examine other accountability measures that may be more effective in the current and future context for CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 442 Ontario universities? Addressing these general research questions, this paper is part of a larger study driven by seven research questions, of which I consider the following four for discussion here; these share a common theme and were selected for their relevance to Ontario universities. Based on the observation and experience of key informants: 1. To what extent does each of the KPIs impact institutional planning in participating Ontario universities? 2. To what extent do universities use KPI results for recruiting students, setting tuition fees for specific programs, or other purposes? 3. What is the effectiveness of KPIs in improving the performance of universities at the institutional and program levels? That is, what is the impact of each of the KPIs, if any, on academic planning and delivery, according to this group of stakeholders? 4. How are the three KPIs currently in place appropriate measures of institutional performance and accountability? What other indicators may better serve all stakeholders—i.e., the government, the universities, and the public? Background and Literature Review Since the 1960s, Ontario universities have relied on government grants for the majority of their institutional revenue, and the universities have been accountable for the expenditures of these funds. This accountability involves multiple groups—i.e., university governing boards, faculty and students, government, and taxpayers. Specifically, Ontario universities were required to file externally audited financial reports to the government. Monahan (2004) observes that, based on the rules around eligibility for receipt of operating grants and capital funding, these requirements satisfied the need to monitor accountability. However, in the early 1980s, the provincial auditor questioned the adequacy of these simple requirements. A decade later and subsequent to three inspection audits at Trent University, the University of Guelph, and the University of Toronto in 1988–1990, the Ontario auditor concluded there was inadequate accountability for university funding (Guelph, 1992). The decision was influenced by a CCAF (Canadian Comprehensive Auditing Foundation ) paper, which quoted a statement made by one of Canada’s leading educational experts that they (the universities) had to, and wanted to, be more accountable; they needed defined indicators to measure performance (CCAF, 1988). Gaither, Nedwek, and Neal (1994) observed that policy makers and the academic community explored using a system of indicators to raise warning signs about the efficiency and effectiveness of higher education. Outcomes, results, and performance thus would determine funding allocation. Depending on the results desired, the role of performance indicators could vary from a minor role as a supporting tool to an expanded, crystallized role to measure results and thereby satisfy the public appetite (Gaither et al., 1994). In 1997, Queen’s University defined a performance indicator as any parameter, measure, or quantitative instrument intended or used for: (i) evaluating, measuring, ranking, or comparing the performance, productivity, or standards of an academic unit, program, faculty, school, or institute, in whole or in part; and (ii) making decisions that affect the terms or conditions of employment for a faculty member (QUFA, 1997). Burke (1997) compared indicators of performance-funding programs in eight states in the United States, observing that indicators among institutions differed in number, content, and terminology. McMil- CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 443 lan and Datta (1998) claimed that data envelopment analysis (DEA) is more reliable than performance indicators in terms of comparisons. DEA includes output variables such as the number of graduate enrolments in different degree-level programs, total sponsored research expenditures, and input variables such as total number of full-time faculty members eligible for Tri-council or Canada Council grants (McMillan & Datta, 1998). However, the Ontario government did not include these among the KPIs that it ultimately selected. Frink and Klimoski (2004) claimed that shared expectations are important for an organization’s response regarding its members’ accountability. In order to anticipate and model agent behaviour effectively, it is essential to determine the nature and clarity of the expectations of members and of the role set, and the degree of alignment with these expectations (Frink and Klimoski, 1998). Ansell and Gash (2007) agreed that collaborative governance brings public and private stakeholders together in common fora to engage in consensus-oriented decision making. Frink and Klimoski (2004) also stated that shared expectations are important for an organizational response to accountability. Similarly, Hernandez and Hodges (2001) stressed the necessity of seeking all stakeholders’ input. Ultimately, there remains a need both to clarify and to satisfy the goals and expectations of the ministry and the universities in order to improve KPI-related institutional accountability processes. In this respect, the observations of Knouse (1979) remain valid: accountability requires the clarification of role expectations by the parties involved; both parties should be clear on each other’s goals and expectations. As Burke and Minassians (2003) appropriately recommended for the American system, the Ontario government and universities should work together to develop a set of indicators that incorporate both quality and efficiency indicators in a way that responds to the concerns of both parties. Lang (2013) commented that performance funding should be tied to quality, significance, and influence—that is, “the difference that performance makes” (Lang, 2013, p. 15). He also added that the “one size fits all” homogenizing effect of performance indicators impaired the essence of performance funding. The incapability of the current performance indicators in measuring quality has led the commercial surveys and rankings of universities to substitute reputation for quality (Lang, 2013). In Ontario, the performance funding program tied with performance indicators has changed four times. These indicators do not capture each university’s total performance, and other sources of funding for universities are significantly more than the current performance funding program, resulting in diminished reliability of both performance measurement and the influence of performance funding (Lang, 2013). According to Salmi (2009), with respect to characteristics indirectly related to research, a “world-class” university (as defined by national and international rankings published by third-party organizations) must have (i) a sound financial base and (ii) a diversified source of income, including research income; (iii) provide a high-quality and supportive research environment (e.g., high-quality buildings and facilities); and (iv) have a first-class management team with strategic vision and implementation plans. Although the KPI initiative does not claim explicitly to identify “world-class” universities, nor are KPIs intended overtly to rank universities, their data may encourage ranking. There are currently a few well-known national and international ranking publications (e.g., the Maclean’s University Rankings and the Times Higher Education Supplement) that claim to report performance measures. In the case of Maclean’s, the methodology has changed CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 444 from originally using a single set of indicators for all universities to currently using three separated ranking groups: namely “medical doctoral,” “comprehensive,” and “primarily undergraduate.” Issues of data integrity, differentiation of universities’ missions and characteristics, and differences in students’ preferences in the Maclean’s ranking system jeopardize the accuracy and relevance of the rankings (Smith, 2000). For example, the Maclean’s University Rankings do not consider the KPIs; instead, Maclean’s uses measures that are distinctively different from KPIs. However, the stakes for universities that may struggle to meet MTCU’s KPIs are high. MTCU has created a benchmark (on benchmarking, see Alstete, 1995, p. iii) for each of the three indicators of graduate employment rate six months after graduation, graduate employment rate two years after graduation, and graduation rates for calculating the distribution of funds to universities. The MTCU set the benchmark for the particular indicator at a certain percentage below the system average; institutions that fall below the benchmark do not receive funding (MTCU, 2009). In 2000, MTCU introduced performance-based funding for colleges and universities, allocating $16.5 million to universities based on graduation rates, employment rates six months after graduation, and employment rates two years after graduation (MTCU, 2001). Research Design This study explored the efficacy of the three KPIs for Ontario universities over a 10year period (1993–2012). The perceptions of key informants at participating universities and an analysis of KPI reports from 1993 to 2012 provide a deeper understanding of the efficacy of the KPIs in driving performance improvements at these universities during the period of study. In evaluating these data, this study used a sequential mixed methods, and exploratory descriptive research design. To answer the research questions, I collected data in four phases. The first phase focused on document analysis as well as interviews with individuals who had participated in the initial development of the KPIs; the latter included interviews with three members of the Broadhurst Task Force. In the second phase, documents reporting KPI performance were analyzed and trends were identified. The primary research sources were publicly available mission statements and strategic plans of all Ontario universities, as well as MTCU policies related to KPIs, to see how they were related to the KPI findings. Additional sources for this phase were published data on graduation rates, employment rates, and OSAP default rates for 1998–2010, provided by MTCU. Two officials from the Postsecondary Accountability Branch of MTCU provided further primary evidence on KPI performance and trends. Phase three of the project invited 20 universities in Ontario to participate in an online survey. The universities covered the spectrum from primarily undergraduate to comprehensive and medical/doctoral institutions. Eleven universities accepted the invitation to participate in this study (a 55% survey response rate). Participating university presidents identified each of the key informants. The latter were preferred, in this study, to other academic bodies, including faculty and students, for their greater authority over policy and administrative decisions, their inside knowledge of data related to KPIs in their respective institutes, and their interest in leveraging this knowledge to attract more funding. The sample of participating universities was fairly representative of the total population, based on size, urban/rural location, and programming. The survey CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 445 structure centred on the themes identified in the data collected in the first two phases. The selection criteria for the invited participants/respondents of this study were their direct roles in the areas of institutional planning, budgeting, and institutional performance, which would enable them to provide a perspective from the institutional level. The survey posed questions for evaluation on a four-point response scale, with the option of providing additional comments. The fourth phase of this project explored further themes generated by this survey via one-on-one interviews with seven key informants from the university respondents. Further interviews with two officials from MTCU, mentioned above, also provided a more comprehensive perspective on how the ministry perceives and uses the data. Overall, 17 key informants contributed to this study: 12 from universities; two from MTCU, selected by the Minister; and three from the Ministry of Education and Training’s Task Force on University Accountability. Usefulness of KPIs for Institutional Planning Most of the university respondents (10 of 11) agreed that the impact of the “graduation rates” KPI on their universities’ institutional planning is high (Table 1). Six and eight of the 11 respondents, respectively, disagreed that the impact of the “employment rate” and “OSAP loan default rate” KPIs is high with respect to their universities’ institutional planning. Since the majority of the respondents either disagreed or strongly disagreed that the OSAP default rate informs institutional planning, a follow-up interview question investigated why they did not perceive the OSAP default rate to be helpful to universities. Of the seven university participants interviewed, a majority (five of the seven) stated that other factors beyond the institution’s control impact these KPIs, such as employment opportunities, labour market needs, and the state of the economy. Table 1. Percentages of Positive Responses (Agree and Strongly Agree) from Online Survey on KPIs, their Impact, Usefulness, and Effectiveness   Graduation Rates Employment Rates OSAP Loan Default Rates High impact on institutional planning 91% (10 out of 11) 45% (5 out of 11) 20% (2 out of 10) Used in student recruitment 73% (8 out of 11) 45% (5 out of 11) 9% (1 out of 11) Used in setting tuition fees for specific programs 0% (0 out of 11) 18% (2 out of 11) 0% (0 out of 11) Used for other purposes 100% (11 out of 11) 82% (9 out of 11) 40% (4 out of 10) An effective factor in improving institutional performance 73% (8 out of 11) 55% (6 out of 11) 0% (0 out of 10) An effective factor in improving performance at program level 80% (8 out of 10) 20% (2 out of 10) 0% (0 out of 10) Source: Responses from the online survey provided by universities’ key informants. CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 446 In reference to the funding associated with KPIs and its impact, one of the respondents said that “based on the relatively small amount ($23 million performance funding), nobody likes them, nobody pays attention to them or uses them directly or very much in their programming [evaluation] processes.” The same person also stated that due to changes in the funding allocation formula, there are “fewer true winners and losers,” as the threshold point of the funding rewarded is relatively minor. Another respondent raised the point that the university environment is so complex that having three indicators is not formative enough to direct an institution. While most participating universities use graduation rates to a limited extent for institutional planning, the universities use neither employment rates nor OSAP default rates during institutional planning processes to any considerable degree. One respondent commented that the approach of pooled funding for KPI performance hinders KPI usefulness in institutional planning, since most of the decisions that would impact program excellence are made at the program level. For internal purposes, all of the participating universities reported that instead of the KPIs, they use other institution-specific and “plan-focused” indicators, which enable drill-down analysis related to strategic priorities that each university sets in its academic plan. The advantage of institution-specific performance indicators, as opposed to Ontario’s KPIs, is that the former are more helpful for managing performance and planning, more relevant to measuring outcomes, outputs, and impact, and more useful for evaluation and communication purposes. One of the university participants mentioned that his/ her institution used more than 100 indicators for different purposes. Interestingly, most of these internal indicators reported by the survey participants are similar to those in the list of indicators that the CUPA (Council on University Planning and Analysis) Committee on Accountability, Performance Indicators and Outcomes Assessment initially recommended to the Broadhurst Task Force (CUPA, 1993). Hernandez and Hodges (2001) have stressed that clear linkages between identified goals, strategies, and implementation plans are essential to inform strategic planning. The gulf between the scope of the KPIs and institution-specific approaches to measuring performance suggests that these linkages between KPIs and the institutions affected by them are not strong. Usefulness of KPIs for Student Recruitment Among MTCU respondents, there was a perception that although universities were using KPIs in their active student recruitment, they did not know the impact of KPIs on student preferences. In contrast, a majority of university respondents (eight) agreed that graduation rates are useful data for their institutions with respect to student recruitment efforts; however, they disagreed (six and 10 out of 11, respectively) with the proposition that employment rates and OSAP loan default rates serve the same purpose (Table 1). Most key informants commented in the follow-up interview that the KPIs are of limited use to student recruitment. While KPIs are useful as a public relations tool in two interviewees’ institutions, this is obviously only the case when the KPI results are favourable. In general, respondents indicated that most participants’ universities use graduation rates for student recruitment but do not use employment rates or OSAP default rates widely for that purpose. One of the university participants stated that in his/her opinion, the major factors influencing students’ decisions are program availability, quality, reputation, and CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 447 location. Given that the initial purpose of the KPIs was to inform the public—particularly so as to assist parents and prospective students in selecting preferred, appropriate institutions for post-secondary studies—this project conducted a review of student recruitment-related online resources for 20 universities to identify whether these sites included information on KPI performance. None of these resources included reports on KPIs or web-links related to KPIs for any of their online recruitment efforts, indicating that the importance of KPIs as a tool for current recruitment campaigns is, at best, minimal. Usefulness of KPIs for Setting Tuition Fees for Specific Programs With respect to the usefulness of KPIs in setting tuition fees for specific programs, all 11 university participants disagreed with the proposition that the graduation and OSAP default rates were useful. Furthermore, only two respondents (18%) agreed that employment rates are one of the factors considered by their respective institutions when determining tuition fees for specific programs (Table 1). In follow-up interviews, the respondents explained that the province regulates tuition fees. Specifically, MTCU has regulated the Tuition Fee Policy for publicly assisted universities since 2006 (MTCU, 2010). However, universities were able to impose tuition increases during the inception of KPIs. Furthermore, four of the seven interviewed respondents felt that these KPIs were disconnected from setting tuition fees for specific programs. They report that high-level indicators, such as employment rates, cannot translate or properly reflect students’ ability to find employment in relation to specific programs. In the respondents’ views, because students often do not pursue careers within disciplines most closely related to the field of their degree and/or program, the employment rate KPI is of limited use. For example, one respondent stressed that “you offer a [university] program, [designed] not to equip individuals for the workplace, but to equip them for life, so it’s not as if you are going to tie directly into some sort of employability or retention data.” Another respondent commented that the typical delay in reporting employment data about the labour market (five years) limits the usefulness of employability statistics. Both the scope and the applicability of data were concerns—for example, that “the more granular the level of planning for specific programs is, the less it is useful to look at the global employment and graduation rates.” Finally, participants’ institutions did not use OSAP default rates at all. Only two participants reported that their universities used employment rates for setting tuition fees, and then only for very specific programs; they viewed OSAP loan default rates as irrelevant for this purpose. Stressing that the province regulates tuition fees, most key informants stated that the KPIs are “disconnected” from their control. Usefulness of KPIs for Other Purposes All respondents agreed that graduation rates are useful for other purposes, and 82% of the respondents (i.e., nine) agreed that employment rates are also useful for other purposes. However, only four agreed that OSAP loan default rates are useful for other purposes in their institutions (Table 1). In the comment section of the survey, three respondents clarified that they use the KPIs as content for “communication messages” and “background data scans to support the review of existing and new programs.” CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 448 Usefulness of KPIs for Improving Performance With respect to improving institutional performance, eight and six of the respondents, respectively, agreed that graduation and employment rates are effective factors in improving institutional performance (Table 1). However, none felt the OSAP loan default rates were useful in this regard. At the program level, a majority of respondents (80%) agreed that graduation rates have an impact on performance improvement (Table 1). However, they did not view either employment or OSAP loan default rates as impactful: only two out of 10 respondents agreed that employment rates affected performance improvement efforts, and none agreed with this statement with respect to OSAP default rates (Table 1). None of the respondents felt that the OSAP default rates have any impact on improving performance at the institutional or program level. As identified earlier, this is due, in the respondents’ opinions, to factors beyond the institutions’ control. Thus, there is a perception that this KPI is “disconnected” from institutional and program performance. Based on the input from the university participants surveyed and interviewed, the efficacy of the KPIs as a means to affect university behaviour and performance funding is minimal at best, and then only for very different purposes than the initial accountability goals. The participants recognized some usefulness of graduation and employment rates for communication and recruitment purposes. However, the participants were concerned about what they viewed as a disconnect between the KPI data, the universities’ mission, and universities’ ability to impact these outcomes. Further, respondents also expressed reservations about what they perceived to be “flawed” data collected. What was striking was the disconnect between their perceptions and that of the two MTCU informants, the latter of whom expressed confidence that the KPIs and performance funding have had a positive impact, given that the graduation and employment rates remain consistently high. The MTCU informants felt that the KPI requirement realized the government’s goal of encouraging university behaviour to support the government’s priorities. The Broadhurst members also stated that the KPI initiative had had a positive impact, but only by increasing the awareness of the accountability of governing boards and perhaps by providing them with needed information. Ten out of 11 respondents concluded that the current three KPIs are not the best indicators to demonstrate accountability, which runs counter to the intent of the Broadhurst Task Force during their initial task of identifying performance indicators. In the follow-up interviews, key informants stated that the current KPIs are disconnected from the performance of the institution. In addition, they claimed that their generic nature does not work well when institutions are so diverse— e.g., with different sizes, mandates, program mixes, and student demographic groups. Indeed, respondents questioned the overall value of the current KPIs because of the unreliability of data, including the method of calculation and the response rates. Moreover, it was impossible to identify under/over-performance, as the results across the system are close to the mean. Burke and Minassians (2003) pointed out that information can be powerful if it is used appropriately. When these performance indicators aligned with performance funding, there was certainly an expectation of improvement or change in performance. However, universities were not able to improve the values for these indicators, as the relevant factors were not under their control. To address this “disconnect,” some university respondents suggested replacing “the graduation rates methodology with CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 449 Consortium for Student Retention Data Exchange (CSRDE) methodology to ensure inter-institutional consistency.” In the CSRDE methodology, students of the same cohort are tracked from their first year to their graduation year, whereas the MTCU calculates graduation rates using a single entering cohort of students and by determining whether or not they graduated within seven years. The MTCU methodology does not isolate students who transferred between universities and thus understates the graduation rates for students affiliated with the original university. The university respondents suggested that the KPIs should “certainly be re-assessed and/or recalculated” and be disassociated from performance funding, due to the expressed disconnect. In terms of other key performance indicators, the respondents recommended the use of the Common University Data Ontario indicators, student satisfaction, research, indicators identified in the Broadhurst Report, and institution-specific reporting. One respondent suggested that the Strategic Mandate Agreement (SMA) with the ministry was a more appropriate tool, as it respects university autonomy while differentiating institutions by absolute measures. Respondents also recommended the indicators included in the Multi-Year Accountability Agreement (MYAA). Another suggestion was to define the general expectations at the system level. This would enable the development of key indicators consistent with these expectations and would include both a standard section and a variable section, with the latter capturing the uniqueness of the institutions. With respect to the impact on institutional planning, contrary to the perception of MTCU participants, most university key informants claimed that KPIs do not have a high impact on their institutional planning process. Most participating universities use graduation rates for institutional planning but not employment rates and OSAP default rates. For internal purposes, all the participating universities use other indicators. In discussing performance indicators and the impact of performance funding, the universities may desire to effect improvement or changes in performance in order to obtain more funding from the ministry. However, universities believe that they are not able to improve these indicators, as the contributing factors are not under their control. The university sector, in general, does not believe that KPIs are having much of the intended impact. From the ministry participants’ perspective, however, the perception was that since the graduation and employment rates have remained high across the system, they are having a positive impact on performance. The participants did not indicate whether these indices were high or not prior to the introduction of KPIs. Most university participants felt that the graduation rates had some but not much impact in improving performance at both institutional and program levels. About half of the university respondents agreed that employment rates were somewhat effective in improving institutional performance, but only two university participants felt that employment rates were effective in improving performance at the program level. On the other hand, OSAP loan default rates were not used at all because factors such as employment opportunities, labour market, and the economy were beyond any given institution’s control. Clark et al. (2009) stated that one of the roles of MYAAs was to subsidize the limitations of KPIs in terms of institutional performance. However, the majority of university participants disagreed that the introduction of the MYAA process has altered either the role or the impact of KPIs in their universities. They felt that the MYAA and KPIs should be consolidated under one accountability framework. When the decision was made to re- CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 450 quire that KPI reporting be made public, the government’s intent was to provide potential students and parents with the basis to make informed decisions in the selection of institutions and programs (MTCU, 1999). On the sector side, colleges and universities could use the data to demonstrate their achievements and improve their programs and services. However, both MTCU and the sector participants agreed that they did not use existing KPIs fully for active student recruitment. MTCU participants confirmed that currently the government also expects KPIs (i) to monitor progress in key areas at the institutional and system levels in order to assist with evaluating programs and institutions, with internal benchmarking, and with comparison of institutions, and (ii) to have sufficient integrity for the ministry to attach funding. The findings in this study suggest that the three KPIs implemented currently do not achieve these goals. These ministry expectations aligned with the desire of most of the university participants but differed from what they perceived to be the ministry’s expectations. The participants claimed that the government focused on simplistic measures as a surrogate accountability scale, expected to identify key issues of institutional performance, and expected the KPIs to highlight institutionspecific problems and to reflect institutional accountability. In summary, the performance that the university respondents considered important is not captured by the current KPIs, and consequently they are not reliable indicators of performance for either accountability purposes or performance assessment. In contrast, the ministry participants perceived the high graduation and employment rates to have a positive impact on performance. The responses further demonstrate that funding allocations attached to KPIs are minimal and do not influence performance. Implications for Practice In order to gain broad acceptance and demonstrate effectiveness and accuracy, performance indicators must be based on valid and reliable tools specifically designed to measure performance metrics over which universities can exert control. The results of this study demonstrate a need to revisit Ontario’s KPIs. The following discussion will offer several recommendations for specific KPI characteristics. Consolidate the MYAA and KPIs As universities are supported by public funding, the outcomes of university education service should be measureable and reported in an accountable way. If the reporting of performance is to be at all useful, there is a need to ensure the integrity and comprehensiveness of the data collected to measure performance. Mutually agreed upon performance measures should assist in decision making for both the institutions and the government. To this end, these performance measures should also allow institutions to build institution-specific indicators as recommended originally by the Broadhurst Task Force, to enable institutions to improve their performance. Participants in this study recommended the consolidation of the MYAA and KPIs under one accountability framework. Clarify the Relationship between Funding and KPIs There is a need to explicitly address the question of whether or not to continue performance funding. Furthermore, performance monitoring should allow institutions (i.e., CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 451 their governing bodies) to evaluate operational efficiency, effectiveness and cost effectiveness, and any reward funding should be sufficient to make it worthwhile for the institutions to implement appropriate changes. Adopt the CSRDE Methodology Of the current KPIs, the university participants viewed only graduation rates to be most helpful for institutional planning, student recruitment, and other purposes. The study participants saw them to be effective factors in somewhat improving both institutional and program performance in that context. However, participants pointed out that because the current methodology does not isolate the students who transferred between universities, graduation rates for the originating university are understated. Participants recommended the adoption of the CSRDE methodology (i.e., students of the same cohort are followed through from the first year to the year they graduate). Disseminate COU and MTCU Data to the Public Currently, only the KPI-related data in the Ontario Graduate Survey (i.e., employment rates) are published. Some of the data that would be useful to students and parents are not published by the universities, COU, or MTCU but should be made publicly available. For example, the proportion of respondents working part-time and the number of respondents attending further post-secondary education would be useful information for students and parents (OUAC, 2009). Implications for Public Policy The findings of this study have the potential to enable more effective, evidence-based decision making on public policy by the government and on institutional policies by the universities. In instituting the KPIs, originally the Ontario government attempted to indirectly influence the behaviour of the universities to support its socio-economic goals of increasing (i) student participation in post-secondary education and (ii) employment rate after graduation. The government’s ultimate goal in instituting the KPIs and additional accountability measures, such as the MYAA, was to build a highly qualified workforce and monitor how effectively the universities use the public support of scarce resources. However, the goals and missions of the universities are broader than that. Government policy for post-secondary education has become increasingly more market-oriented because of the unstable economic climate, not only in Ontario but worldwide. The KPI model for accountability has been in place and unchanged since the 1990s. The government has implemented the MYAA and SMA as additional accountability measures. The findings in this study suggest that the KPI model is not working; universities perceive it to have had only a minimal impact on their behaviour and consider the KPIs to be more of a “nuisance” than a driver for program change, which may be partly due to high barriers, such as faculty tenure. There is a need to update government policy in order to achieve accountability goals and align institutions more closely with the universities’ core missions, empowering them to impact positively both their own goals and those of the government. Also, the new policy framework must take into account diversity within the university sector. In addition, I recommend consolidating the current measures of accountability—KPIs, MYAA, and SMA—into one coherent accountability framework. CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 452 Limitations of the Study Several factors, such as difficulty in accessing information and the reluctance of university representatives to participate in the survey, have influenced this study’s results through the various stages of data collection and analysis. Methodological Assumptions and Limitations A major challenge was locating the members of the Broadhurst Task Force to identify the original intent of the KPIs. Related to this was the methodological limitation of collecting historical data based on informants’ recall of past activities. This was especially true when members of the Broadhurst Task Force were recounting what happened two decades ago. However, what they did recall must have been important issues if they remembered them so clearly that many years later. Furthermore, since the participants in the survey and interviews had nothing to gain or lose, it is reasonable to assume that their responses reflected what they actually perceived or remembered the issues to have been. Scope This study explored the perceptions of key informants in 11 (55%) of the 20 Ontario universities who participated in this study with respect to the implementation and subsequent impact of KPIs in the universities. The extent to which current KPIs were aligned with individual universities’ mission statements and other tools or mechanisms was also explored. Participation in the study was not by random sampling, and because of the considerable diversity among Ontario universities in terms of size, geographic location, program mix, and intensity of teaching and research, the findings are not generalizable to all Ontario universities. This study involved a voluntary sample (N = 11; 55%) of English language universities in Ontario, and the findings cannot be generalized to the other Englishlanguage universities, nor to the francophone university in Ontario, or to other universities in the rest of Canada or elsewhere. This study did not address existing research related to performance indicators for post-secondary institutions in other Canadian provinces (since under the Canadian constitution, education is a provincial matter) or in international jurisdictions, which may be similar to or very different from the Ontario experience. Key Informants The invited participants/respondents in this study were personnel directly involved in the areas of institutional planning, budgeting, and institutional performance and provided a perspective from an institutional level. On the other hand, the deans of faculties, who are probably most directly involved in directing program changes, were not included and may well have had a very different viewpoint. Recommendations for Further Research This study confined itself to the evaluation of the KPI model more than 10 years after its implementation in Ontario, based on feedback from key informants in 11 of 20 public universities in Ontario. Further research in the following areas would build on these data to address the issue of having valid and reliable measures of accountability in publicly funded educational institutions: CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 453 1. What are the common elements and differences in the indicators that all of the 20 universities currently use for different purposes? According to the findings, for internal purposes such as gauging student satisfaction, retention, accessibility, research, and time to completion, all the participating universities use other indicators. The participants mentioned that these institution-specific indicators are helpful for different purposes, such as managing performance and planning. They are also applicable to outcomes, outputs, and impact, and for evaluation and communications. A thorough analysis of the commonalities and differences among the indicators that the university sector employs would be helpful for all universities. It would be especially productive for the institutional planning offices to share best practices, identify the norms or standards of different programs, evaluate success, and encourage improvement. 2. How and when do the institutions actually base their decisions on their KPI results? This study has examined the usefulness and effectiveness of KPIs for certain areas, such as institutional planning, student recruitment, setting tuition fees for specific programs, and improving institutional performance. The findings vary among the three KPIs and in different areas. The MTCU participants thought that the universities indeed paid attention to and used the KPI results at the institutional level, and that the universities were committed to transparency and accountability as represented by the reporting of the KPIs. However, the university respondents did not share that view. For instance, one university interviewee specifically mentioned that his/her institution’s short-term plans did not relate to the KPIs at all, because it took seven years to receive data on graduation rates. Given the different perspectives of the MTCU and university participants, as reflected in the analyses of the findings, it was not clear exactly how and when the institutions actually based decisions on their KPI results. Further research on this topic is recommended in order to assist the ministry in evaluating the usefulness of each of the three KPIs. Implications for the Theory of Accountability as Used in the KPIs By the time the Ontario government instituted the KPIs, accountability had become the driving force in initiating the requirement for regular and public reporting by Ontario universities. This priority was already evident in the formal title of the Broadhurst Task Force: The Task Force on University Accountability. Accountability is also a core focus of other similar approaches, such as Oxford University’s White Paper on University Governance (2006), which declares that accountability is the first and most fundamental principle of governance. The governing bodies of Ontario’s publicly funded universities have a fiduciary responsibility to ensure that their universities operate effectively and efficiently. They also have a responsibility to ensure that the interests of their stakeholders are well served and that the funds provided by the public purse fulfill their declared institutional missions. As Trow (1996) stated, sustaining or improving institutional performance is one of the functions of accountability. He also stressed that institutions need to be examined internally and externally in order to sustain or raise their quality of performance. However, as this study has shown, there is a strong perception among many university administrators that the current KPIs fall well short of meeting this goal. Consequently, the province needs new indicators and processes that more accurately reflect performance concern- CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 454 ing both governance and accountability. Furthermore, Clark, Moran, Skolnik, and Trick (2009) demonstrated that accountability and quality are closely interconnected, remarking that demanding accountability without considering the quality of the performance is limiting. The findings in this study suggest that the initial interpretation of accountability as defined by the three required KPIs is too narrow to be practically useful. To address this failing, there is a need for a comprehensive review and redefinition of the theory of accountability as applied to the governing boards of Ontario’s universities. Conclusion This study addressed important issues arising from the implementation of KPIs for Ontario universities. KPIs provide data that affect core decision making among multiple stakeholders in higher education: students, parents, universities, government, and taxpayers. Hence, the efficacy of KPIs is either directly or indirectly relevant to all citizens of Ontario. The findings of the study highlight the questionable effectiveness of the KPIs as they currently exist and point to possible alternatives for measuring university performance. Although not generalizable, the results of this research also provide valuable insights that may assist in evidence-based decision making in the revision or creation of MTCU policies that affect both Ontario’s universities and their stakeholders. Acknowledgements I received invaluable advice and guidance on this study from Katharine Janzen, Dan Lang, and Tim McTiernan. I also wish to thank the participants for their time and insightful knowledge. References Alstete, J. (1995). Benchmarking in higher education: Adapting best practices to improve quality. Washington, DC: ASHE-ERIC. Ansell, C., & Gash, A. (2007). Collaborative governance in theory and practice. Journal of Public Administration Research and Theory, 18, 543–571. Burke, J. (1997). Performance Funding Indicators: Concerns, Values, and Models for Two- and Four-year Colleges and Universities, Albany: Nelson A. Rockefeller Institute of Government. Burke, J., & Minassians, H. (2003). Performance reporting: “Real” accountability or accountability “lite”: Seventh annual survey. Albany, NY: Nelson A. Rockefeller Institute of Government. Canadian Comprehensive Auditing Foundation (CCAF). (1988). Accountability, accounting and audit – responding to a decade of experience. Archives, University of Toronto. Clark, I. D., Moran, G., Skolnik, M. L., & Trick, D. (2009). Academic Transformation: The Forces Reshaping Higher Education in Ontario. McGill-Queen’s University Press. Council on University Planning and Analysis (CUPA). (1993). Report of the Committee on Accountability, Performance Indicators, and Outcomes Assessment to the Minister’s Task Force on University Accountability. Archives, University of Toronto, Toronto, ON. CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 455 Frink, D., & Klimoski, R. (1998). Toward a theory of accountability in organizations and human resources management. In G. Ferris (ed.), Research in Personnel and Human Resources Management, vol. 16 (pp. 1–51). Stamford, CT: Elsevier Science/JAI Press. Frink, D., & Klimoski, R. (2004). Advancing accountability theory and practice: Introduction to the Human Resource Management Review special edition. Human Resource Management Review, 14(1), 1–17. Gaither, G., Nedwek, B., & Neal, J. (1994). Measuring up: The promises and pitfalls of performance indicators in higher education. Washington, DC: ASHE-ERIC. Hernandez, M., & Hodges, S. (2001). Theory-based accountability. In M. Hernandez & S. Hodges (eds.), Developing outcome strategies in children’s mental health (pp. 21–40). Baltimore, MD: Paul H. Brookes Publishing. Knouse, S. (1979). Toward a psychological theory of accountability. Interfaces, 9(3), 58–63. Lang, D. (2013, February 26). Performance funding: Past, present, and future. Paper presented at the Higher Education Seminar of the Ontario Institute for Studies in Education, Toronto, ON. McMillan, K., & Datta, D. (1998). The relative efficiencies of Canadian universities: A DEA perspective. Canadian Public Policy, 24(4), 485–511. Retrieved from http://ideas. repec.org/a/cpp/issued/v24y1998i4p485-511.html Ministry of Education and Training (MET). (1993). University accountability: A strengthened framework. Report of the Task Force on University Accountability. Archives, University of Toronto, Toronto, ON. Ministry of Education and Training (MET). (1998, February 13). Changes to OSAP for 1998–99. Memorandum from the Assistant Deputy Minister to executive heads, provincially-assisted universities; presidents, colleges of applied arts and technology; and presidents, private vocational schools. Archives, University of Toronto, Toronto, ON. Ministry of Training, Colleges and Universities (MTCU). (1999). Ministry of Training, Colleges and Universities 1999–2000 Business Plan. Toronto, ON: Author. Ministry of Training, Colleges and Universities (MTCU). (2001). Accountability framework for university funding. In MTCU, Office of the Provincial Auditor, 2001 annual report of the Provincial Auditor of Ontario (section 4.13). Toronto, ON: Author. Ministry of Training, Colleges and Universities (MTCU). (2009). The Ontario operating funds distribution manual – a manual governing the distribution of Ontario government operating grants to Ontario universities and university-related institutions (formula manual), 2009–10 fiscal year. Toronto, ON: Post-secondary Finance & Information Management Branch, MTCU. Retrieved from http://www.uoguelph.ca/ analysis_planning/images/pdfs/2009-10-Operating-Manual-Sept09.pdf Ministry of Training, Colleges and Universities (MTCU). (2010). Guidelines for implementation of the tuition fee policy for publicly-assisted universities, 2010–11 to 2011–12. Toronto, ON: Author. CJHE / RCES Volume 45, No. 4, 2015 What is Your Degree Worth? / B. Fenesi & F. Sana 456 Ministry of Training, Colleges and Universities (MTCU). (2012). Post-secondary key performance indicators. Toronto, ON: Author. Monahan, E. (2004). Collective autonomy: A history of the Council of Ontario Universities, 1962–2000. Waterloo, ON: Wilfrid Laurier University Press. Ontario Universities’ Application Centre (OUAC). (2009). Ontario university graduate survey 2006: Final report on response rate and survey results. Unpublished report, provided by MTCU. Oxford University. (2006). White paper on university governance. Oxford University Gazette, 36 (Supp. 5). Retrieved from http://www.ox.ac.uk/gazette/2005-6/supps/ whitepaper.pdf Queen’s University Faculty Association (QUFA). (1997). The collective agreement between Queen’s University Faculty Association and Queen’s University at Kingston. Retrieved from http://archive.qufa.ca/ca/archives/1997-1999.pdf. Salmi, J. (2009). The challenge of establishing world-class universities. Washington, DC: World Bank. Smith, D. (2000). “How will I know if there is quality?” Report on quality indicators and quality enhancement in universities: Issues and experiences. Toronto, ON: Council of Ontario Universities. Trow, M. (1996). Trust, markets and accountability in higher education: A comparative perspective. Higher Education Policy, 9(4), 309–324. University of Guelph. (1992). Auditor’s Report Prompts Task Force. At Guelph. September 30, 1992. Archives, University of Toronto. Contact Information Vivian Chan Finance and Administration Ryerson University [email protected] Vivian Chan has over 20 years of experience in university administration. Her current position as the Senior Director, Research Planning, Finance and Administration at Ryerson University requires her to contribute to the university’s decisions about the development of increasingly effective, efficient, and accountable administration of the university’s scarce resources and opportunities. As a senior member of the central administration team of one of the fastest growing universities in North America, she has found that a deeper understanding of issues related to KPIs enables her to contribute more knowledgably to policy making and decision making at the university. She is a Chartered Professional Accountant (Certified General Accountant) and holds a PhD (higher education), Master of Business Administration, and Bachelor of Arts (economics). CJHE / RCES Volume 45, No. 4, 2015