Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 57 CSSHE SCÉES Canadian Journal of Higher Education Revue canadienne d’enseignement supérieur Volume 46, No. 1, 2016, pages 57 - 75 Early Contributions to the Evolution of the Canadian Scientific Integrity System: Institutional and Governmental Interaction in the Policy Diffusion Process Jordan Richard Schoenherr Carleton University Bryn Williams-Jones University of Montréal Abstract Academic institutions and research funders have in the last decade devoted considerable effort to developing policies to support academic integrity and prevent misconduct. In this study, we consider the extent to which various initiatives of Canadian federal and provincial (Québec) funders have affected the development of institutional research integrity/misconduct (RIM) policies. Examining the creation and modification dates of 32 institutional RIM policies, we find that federal but not provincial initiatives appear to have the greatest impact on the development of RIM policies. Idiosyncrasies in the creation dates, as well as lack of evidence of a systematic pattern in modification dates, suggest a complex system that is often insulated from certain government initiatives. These results lead us to conclude that there should be greater consistency in the development or updating of RIM policies to ensure the appropriate treatment of misconduct and to encourage behaviour that meets the highest standards of research integrity. Résumé Au cours de la dernière décennie, les institutions académiques et les bailleurs de fonds de recherche ont consacré beaucoup d’efforts à l’élaboration de Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 58 politiques pour soutenir l’intégrité scientifique et prévenir les inconduites en recherche. Dans la présente étude, nous analysons dans quelle mesure les diverses initiatives des bailleurs de fonds canadiens, tant fédéraux que provinciaux (Québec), ont affecté l’élaboration de politiques institutionnelles sur l’intégrité et les inconduites en recherche (IIR). En examinant les dates de création et de modification de 32 politiques institutionnelles sur l’IIR, nous constatons que ce sont les initiatives fédérales, et non provinciales, qui semblent avoir le plus grand effet sur l’élaboration de politiques. Des idiosyncrasies dans les dates de création, ainsi que l’absence de preuves d’un modèle systématique de dates de modification, suggèrent un système complexe, souvent isolé de certaines initiatives gouvernementales. Ces résultats nous amènent à conclure qu’il devrait y avoir une plus grande cohérence dans l’élaboration ou la mise à jour de politiques sur l’IIR pour assurer le traitement approprié des inconduites et encourager un comportement qui répond aux normes les plus élevées d’intégrité en recherche. Introduction Concern on the part of academics, institutions, funders and the public regarding the integrity of research findings and practices has been stimulated in part by high-profile and mediatised cases of misconduct (e.g., Callaway, 2011; Jasanoff, 2010; Lock, Wells, & Farthing, 2001; Medawar, 1976; Normile, 2014; Snyder & Loring, 2006). In the North American context, the US government initiated concerted efforts in the 1980s to study the governance mechanisms (i.e., policies and procedures) of research integrity systems within American research institutions (CHPS Consulting, 2000; Greene, Durch, Horwitz, & Hooper, 1985; Lind, 2005). This led in the United States to the development of an active research community studying integrity (also known as the responsible conduct of research), and to the development and implementation of formal governance structures at both the federal (e.g., the Office of Research Integrity) and state levels, as well as in institutions, to address research integrity/misconduct (RIM; e.g., LaFollette, 1994; Steneck, 1994). In comparison with the United States, Canada’s RIM policy landscape is much less clear, in large part because this subject has received far less attention from scholars and only in the last decade (Hickling Arthurs Low, 2009; Pencharz, 2007; Schoenherr & Williams-Jones, 2011). This situation is problematic given that Canada ranks high among nations in terms of investment in science education and there is a considerable degree of return on investment in scientific research (King, 2004). Like their American counterparts, Canadian academic institutions all have relevant RIM policies and procedures (regarding, for example, conflict of interest, intellectual property, and research ethics), but the connection between these institutional policies and provincial or federal government initiatives (e.g., the Tri-Agency Framework: Responsible Conduct of Research, TriCouncil, 2011) is not at all clear. Research on the state of research integrity in Canada is thus needed (Master, McDonald, & Williams-Jones, 2012). Whereas the scientific integrity system in the United States represents an important landmark for laying out a set of interactions between academia and government, the Canadian context presents a unique set of considerations, due to differences in public poli- Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 59 cies as well as government and academic relationships. In this paper, we examine the extent to which the actions of research institutions or central authorities affect the development of RIM policies at Canadian higher education institutions. To narrow our focus, we have limited our investigation to those policies developed by federal granting agencies (i.e., by the Tri-Council members) and one provincial funder (i.e., the Fonds de recherche du Québec [FRQ]) to study the impact on the development of RIM policies. Québec was chosen as a case study for a number of reasons: (1) the FRQ funds a substantial amount of research (mirroring the scope, if not the depth, of funding by the three agencies); (2) the health-funding branch of the FRQ, the Fonds de recherche en santé du Québec (FRSQ, now the FRQ-S), has been very active in promoting research ethics and integrity and in providing guidance regarding RIM policies for the Québec academic community (e.g., by developing a key document in 2003); (3) French is the official language in Québec, so provincial government documents might have particular influence on institutions; and (4) the Québec judicial system is based on a civil code, which could add force to certain elements of RIM policies. The Tri-Council and FRQ require academic institutions that receive funding from these sources to develop and implement research integrity policies. However, as noted in the literature on policy implementation (e.g., Palumbo & Calista, 1990; Palumbo & Oliverio, 1989; Scheirer, 1987), local institutional contexts can significantly change the nature of a policy due to the need to accommodate different stakeholders (Cohen, March, & Olsen, 1972; Kingdon, 2002; Kraatz & Zajac, 1996, 2001). Without detailed evaluation, it remains unclear what is actually included in these policies (Schoenherr & Williams-Jones, 2011), whether and how they are implemented, and how effective they are at preventing and managing instances of misconduct. Despite the potential variability in an organization’s approach to addressing a particular concern, organizations often appear to adopt highly similar solutions to policy problems (DiMaggio & Powell, 1983; Pfeffer & Salancik, 1978/2003). For instance, Starr’s (1982) history of American medical education demonstrates a shift from considerable organizational diversity and competition to a contemporary model that is far more homogeneous and centralized. Rather than being an institution responding to the needs of its academic community in isolation, it must also adapt to the larger social and policy landscape. Thus, change can also result from a desire for legitimacy rather than a wish to solely or specifically improve organizational operations (Kraatz & Zajac, 2001; Meyer & Rowan, 1977; Suchman, 1995). Given our previous policy analysis (Schoenherr & Williams-Jones, 2011), it does not appear that the content of RIM policies reflects a move toward homogeneity. Instead, it is likely that policy innovations reflect some form of a diffusion process (Kadushin, 2012; Rogers, 2003; Valente, 1995). Specifically, we suggest that the RIM policy adoption pattern in North America reflects both a social contagion process within research communities (e.g., fields, disciplines) and a response by organizations (i.e., universities) to perceived external demands that reflect the concerns of federal funders (e.g., for greater institutional accountability).1 This observation has important implications for the modelling of policy-creation behaviour. In the present study, we examine the dates in which 47 Canadian university RIM policies were created (between 1985 and 2006) and compare them to federal and provincial (specifically, Québec) governance policies developed during the same time period. These Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 60 policies were previously sampled from a listing of the 50 highest ranked universities, using RE$EARCH Infosource (www.researchinfosource.com), and then analyzed in terms of content to determine the features of institutional mechanisms developed to address research misconduct (Schoenherr & Williams-Jones, 2011). In our previous study, we noted that the release of a research misconduct policy by a federal granting agency was cited in some university policies as an impetus for their creation. To examine whether the evolution of university policies supports this claim, the current study assessed RIM policies before and after the release of key federal and provincial policies. Policy dates were summarized in a cumulative distribution function, wherein the frequency of policy creation for each date represented the number of policies created in that year, in addition to policies created for all years preceding that date. This procedure allowed us to identify and test a number of functions that represented theoretically important patterns of institutional policy-creation behaviour (Rogers, 2003; Valente, 1995), to distinguish between the various factors that affect policy adoption (e.g., DiMaggio & Powell, 1983). For instance, we could examine whether policies were randomly adopted over a period of time (linear increase) or whether there were periods of rapid policy creation (non-linear increase). We took the latter as evidence for the importance of factors that were external to an institution (i.e., the release of government policies). The pattern of policy creation dates obtained by our study clearly shows a period of rapid policy creation following the release of a Tri-Council policy in 1994. These findings support the view that in the context of Canada’s higher education system, an important role is being and can be played by central authorities in promoting a comprehensive scientific integrity system for the research community (see, e.g., Bechner & Kogan, 1992). Canadian Research Integrity/Misconduct Policies Initiatives on the part of American government agencies, such as the Public Health Services (1986) and the National Science Foundation (1987), led to the development of centralized mechanisms (notably the Office of Research Integrity) capable of investigating alleged cases of scientific misconduct, and these developments have had an important impact on the American research integrity system and on the policies implemented by academic institutions. But the Canadian context is quite different, lacking as it does such a central authority; nor is it clear what the key motivating factors were in the creation of current RIM policies and governance mechanisms in the Canadian academic community. It is intriguing to note that in Canada, federal policies—such as those developed by Agriculture and Agri-Foods Canada in 2005, or the Canadian Food Inspection Agency in 2010—vary in terms of whether they outline explicit measures for investigating cases of misconduct, and the extent to which these policies can perform an integrative function so that multiple existing policies are explicitly linked together. Other federal policies related to integrity, such as the Values and Ethics Code for the Public Sector (Government of Canada, 2012b), the Federal Accountability Act (Government of Canada, 2006), the Policy on Conflict of Interest and Post-Employment (Government of Canada, 2012a), and the Directive on Conflict of Interest and Post-Employment (Government of Canada, 2014), or the Public Servants Disclosure Protection Act (Government of Canada, 2005; updated in 2012) are limited to addressing concerns related to scientists and personnel working in these government organizations and so are unlikely to have an impact on the behaviour of Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 61 the academic community working in non-government research centres or in universities. Similarly, while documents produced by the Council of Science and Technology Advisors (CSTA), such as Science Advice for Government Effectiveness (SAGE; CSTA, 1999) and S&T Excellence in the Public Sector (STEPS; CSTA, 2001), have developed a best-practice framework for the use of scientific results in evidence-based decision-making (see also Health Canada, 1999), they have not outlined norms for the regulation of scientific activity. Nonetheless, broad policy statements have been made by professional organizations, such as the Canadian Federation for the Humanities and Social Sciences (CFHSS) with their Statement on Research Ethics and Scholarly Integrity (2006). Such general policies can be contrasted against more specific policies, such as the National Research Council’s Research Integrity Policy (National Research Council, 1996/2013), or reports by the Canadian Association of University Teachers (2003; see also Savage 1994/2003) and the Council of Canadian Academies (2010), wherein specific forms of misconduct and regulatory considerations are discussed. Two recent reviews of the Canadian policy landscape have provided insight into the general structure of RIM governance. The first, broader review was conducted by the Hickling Arthurs Low (HAL) consulting group (2009) and examined the policies of academic, government, and private-sector research institutions. This review concluded that the RIM system in Canada is characterized by a decentralized approach that is defined by codes of conduct, guidelines, and policies that function at an institutional level. Importantly, the HAL report identified one policy of a national granting agency, i.e., the Tri-Council Policy Statement: Integrity in Research and Scholarship (TCPS-IRS; Tri-Council, 1994), as the locus of the RIM system and concluded that this structure was adequate for addressing concerns of misconduct in the Canadian academic community. In a more focused analysis of RIM policies developed by academic institutions (Schoenherr & Williams-Jones, 2011), we documented the elements contained within these policies (e.g., applicability statements, procedures for submitting allegations, inquiry and investigation mechanisms) as well as their objectives. Of the institutions that had RIM policies (87.2%), the majority stated their objectives (92.7%). Interestingly, though, only 50% explicitly referred to Tri-Council initiatives as the impetus for their policy, with an additional 31.6% stating funders in general. Taken together with the finding that policy areas were not consistently covered in all policies (e.g., ranging from 2.6% to 100%), it is unclear to what extent institutions were influenced by the release of the TCPS-IRS in 1994, in the same way that American institutions were influenced by the PHS and NSF initiatives in the late 1980s. Local Responses: Institutional Initiatives The Canadian RIM system has been characterized as decentralized but with “an influential locus of policy coordination and leadership that resides with the three federal granting councils” (HAL, 2009, p. 7). Specifically, institutions are each responsible for the development and administration of their own RIM policies. Information on institutional responses to reports of research misconduct is, however, limited, making it difficult to assess the factors that are responsible for the development of these policies. In Canada, one of the only documented cases of misconduct leading to policy development was provided by Pencharz (2007). Commissioned by Memorial University of Newfoundland, Pencharz Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 62 conducted an independent review of six existing Canadian policies to determine whether those of Memorial University were comparable to those of other institutions and to the standards set out by the TCPS-IRS (1994, updated 2005) more generally. This review occurred in response to the repeated falsification of research by Dr. Ranjit Chandra (Smith, 2005). In the end, the case as well as the review motivated the development of a more comprehensive institutional policy (Pencharz, 2007). In assessing this case, it is important to note that the response of Memorial University could be non-representative. Institutions might be more or less proactive in their development of policies and other measures. For instance, the 1985 publication date of McGill University’s RIM policy might suggest a response to ensure compliance with American policies (e.g., of the NSF) so that the university could continue to receive funding, rather than a response to unpublicized cases of research misconduct. Moreover, even though misconduct might be detected, institutions could also choose not to develop a comprehensive response, preferring instead to treat each instance on a case-by-case basis. This latter approach would likely be a result of believing that instances of misconduct are sufficiently rare that the investment of time and financial resources in developing/updating policies is not justified. Nonetheless, the external funding received by universities—whether Canadian or American—obliges these institutions to adhere to the terms of the agreements made with funders, and these terms now invariably include the promotion of ethical research and academic integrity, and the prevention and management of research misconduct. The creation of institutional RIM policies thus appears, at a minimum, to be influenced by interactions with external stakeholders. External Responses: Federal and Provincial Funding Agencies Unlike the American institutional integrity system—wherein the federal Office of Research Integrity (ORI) acts as a central agency that establishes regulations, procedures, and appropriate sanctions for misconduct—Canada’s integrity system has no equivalent central authority. Instead, a collaborative organization of three federal funding agencies, known as the Tri-Council—i.e., the Social Sciences and Humanities Research Council (SSHRC), the Canadian Institutes of Health Research (CIHR), and the Natural Sciences and Engineering Research Council (NSERC)—provides some central coordination and leadership but without the full investigatory powers of the ORI. Specifically, the Panel on Research Ethics (which is responsible for the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans, second edition [TCPS2]) and the Panel on Responsible Conduct of Research (which is responsible for the Tri-Agency Framework: Responsible Conduct of Research  [TAF-RCR], formerly the TCPS-IRS) set national standards and guidelines for research ethics and integrity.  The key Canadian research integrity guideline in the 1990s was the Tri-Council Policy Statement: Integrity in Research and Scholarship (TCPS-IRS; Tri-Council, 1994), replaced in 2011 by the Tri-Agency Framework: Responsible Conduct of Research (TAFRCR; Tri-Council, 2011). The TCPS-IRS—and the more detailed TAF-RCR—outlined the general principles of responsible scientific practice for researchers receiving Tri-Council funding, as well as the responsibilities of institutions who administer these funds. Principles included appropriately acknowledging others’ contributions (including authorship), Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 63 revealing potential conflicts of interest, and maintaining rigour throughout the research and reporting process. Institutional responsibilities required by the TCPS-IRS included promoting research integrity, investigating allegations of misconduct, and reporting on the investigation outcomes to the appropriate funding council. A companion document to the TCPS-IRS statement, entitled Framework for Tri-Council Review of Institutional Policies Dealing with Integrity in Research (Tri-Council, 1996) contained more specific guidelines. The other major federal document was the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS1, Tri-Council, 1998/updated in 2010 and 2014; TCPS2, Tri-Council, 2014), which was developed specifically to address the ethical complexities of research involving human participants. But provinces have also taken leadership roles in developing resources and guidelines on research integrity for their academic communities. A notable example is Québec, where in 1998 the Ministry of Health and Social Services produced a Plan d’action ministeriel en éthique de la recherché et en intégrité scientifique (Gouvernement du Québec Ministère de la Santé et des Services sociaux, 1998), followed in 2003 by the Guide d’éthique de la recherche et d’intégrité scientifique from the Fonds de recherche en santé du Québec (FRSQ, now the FRQ-S), which outlined comprehensive measures regarding research integrity.2 Both of these documents could be sources of motivation for the creation of institutional RIM policies in Québec. In the same manner as the Tri-Councils, the FRSQ later revised and updated these standards with the publication of Standards du FRSQ sur l’éthique de la recherche en santé humaine et l’intégrité scientifique (2008/2009). These standards make explicit reference to articles in the Québec civil code and associated policy statements while delegating responsibility for the investigation of cases of misconduct to individual academic institutions.3 The major focus of our study concerns whether these standards were cited, so it remains an open question whether these documents were in fact an impetus for institutional policy creation. Consequently, a preliminary examination was conducted of the inception dates of Canadian institutional policies to determine whether they followed the creation of these federal and Québec guidelines and standards. Policy Creation Dates TCPS-IRS Federal Policy Initiative: Creation Date Examination of document inception dates (see Schoenherr & Williams-Jones, 2011) revealed that of the academic institutional RIM policies that indicated this information (n = 32/47), nearly half (n = 15) were approved within a year of the TCPS-IRS release in 1994 (Mode = 1995; Mdn = 1995). The extremely limited range of policy release dates, and the fact that the modal value occurs within one year of the TCPS-IRS, suggest that the TCPS-IRS might have played a crucial causal role in the creation of individual institutional policies. Importantly, as seen in Figure 1, the 1996 release date of the TCPS companion document, the Framework for Tri-Council Review of Institutional Policies Dealing with Integrity in Research (TCPS-FIR), resulted in no policy creation in 1996 or 1997. This is notable given that the TCPS-FIR lays out explicit procedures for investigating allegations of misconduct. Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones Creation Date Modification Date CDF Best-Fit Sigmold 30 Total Document Frequency 64 25 20 15 10 5 0 1985 1990 1995 2000 2005 Figure 1. Bars represent RIM policy creation and modification date frequency. The bestfit three-parameter sigmoid function is plotted against the cumulative distribution function (CDF, represented by triangles). Diffusion modelling: curve fitting to cumulative distribution function. To examine the patterns of policy creation by academic institutions, we used a diffusion modelling procedure (e.g., Rogers, 2003; Valente, 1995). First, policy creation dates were summed, creating a cumulative distribution function. Functions were fitted for the data points using SigmaPlot, and R2 values were obtained to examine the proportion of variance accounted for in the data by the respective curves. Seven candidate models were selected. (1) A linear model was selected to determine whether policies were adopted in a more or less continuous fashion over the period examined (i.e., 1985 to 2006). (2) To consider the possibility that policy-creation behaviour consisted of two different linear trends before and after the publication of the TCPS-IRS in 1994, we also applied a piecewise two-segment linear function. (3) A three-segment linear function was used to reflect differences in policy-creation patterns before, during, and following the release of the TCPS-IRS in 1994. (4) The logarithmic function in Figure 2A was selected to determine whether the behaviour of institutions reflected a rapid onset of policy creation followed by little additional policy-creation activity. The (5) sigmoid function and (6) logistic function reflect variants on a pattern of behaviour consistent with a small volume of policy creation, followed by a rapid increase in creation, then a period of stabilization. As can be seen in Figure 2B and 2C, the sigmoid function would reflect an earlier onset of policy creation followed by a slow period of increase, whereas the logistic function would reflect a later CJHE / RCES Volume 46, No. 1, 2016 Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 65 onset of policy creation and a more abrupt increase. (7) Finally, the sigmoidal hill function in Figure 2D reflects a period of stability and inactivity, followed by rapid policy creation, then another period of stability without any further policy creation. A B C D Figure 2. Representative functions consistent with (a) logarithmic, (b) sigmoid, (c) logistic, and (d) Hill models. Figures adapted from SigmaPlot 11.0. An examination of the R2 in Table 1 suggests that all of the seven candidate functions fit the data reasonably well, but that the three-segment linear function, the sigmoid function, and the logistic function provide the best fit. Despite the high correlation of the three-segment linear function, its superior fit is likely a result of the number of model parameters used (see Table 1) and so does not reflect the best model. To rule out the influence of the number of model parameters in our functions, we used Akaike’s Information Criterion (AIC) as a goodness-of-fit index to compare models and correct for the number of parameters (Burnham & Anderson, 2002). A correction was used to account for the small sample size (32 documents) due to the limited number of policy-creation dates: AICc = n∙ln(RSS/n) + 2∙K+(2∙K∙(K + 1))/(n-K -1)), where RSS is the residual sum of squares, n is the number of data points (policy creation dates), and K is the number of parameters within the model. A comparison of the AICc for each model provided a difference score—Δi(AIC)—reflecting the extent to which the fits of the alternative functions deviated from the best-fit sigmoid function. These scores were then standardized to obtain Akaike weights (wi), a measure providing the probability that a given function accurately reflects the observed pattern in the data (Wagenmakers & Farrell, 2004). The Akaike weights also support the general observation that both the sigmoid and the logistic functions provide the best fit to the data. Moreover, by computing a confidence set (Royall, 1997), we were able to determine which functions were within 10% of the highest Akaike weights, thereby account- CJHE / RCES Volume 46, No. 1, 2016 Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 66 ing for a similar proportion of variance in the data. As both the sigmoid and the logistic functions meet this cut-off, they cannot be distinguished from one another, confirming that they fit the data equally well. The AICcs in Table 1 support the conclusion that both functions provide the best fit to the data, with a small, non-significant advantage for the sigmoid function. Once the number of parameters is accounted for, the three-segment function does not fit the data any better than the other candidate models. Demonstrating that the sigmoid and the logistic functions provide the best fit for the policy creation date data has important implications for understanding the kind of behaviour engaged in by Canadian academic institutions in their RIM policy development. The non-fit of the linear function suggests that the creation of RIM policies was not guided by a progressive increase in institutional policy creation; the two- and three-segment linear functions also provided a poorer fit, suggesting that multiple, linear behavioural patterns were not at work. Similarly, the failure of the logarithmic function to fit the data suggests that RIM policy creation was not characterized by a period of inactivity, followed by a sudden burst of policy development after the release of the TCPS-IRS in 1994, and then a steady rate of policy creation. Finally, the failure of the Hill function to fit the data suggests that a pattern characterized by rapid policy creation followed by a period of stabilization does not explain RIM policy creation in Canadian academic institutions. Our finding that the sigmoid and logistic functions fit Canadian RIM policy creation dates suggests a very specific pattern of behaviour characterizing the Canadian research integrity landscape. Both of these functions show slower initial onsets followed by a rapid series of policy-creation events, which are in turn followed by a slower decline in policycreation activity. The fits of these functions with the data suggest a pattern wherein some universities (e.g., McGill) preceded the general trend in Canada and therefore were likely responding to external influences (e.g., American or Tri-Council initiatives). Similarly, the later development of policies after a rapid period of policy creation suggests that institutions either resisted change or could not mobilize policy responses. The fact that there is at least a measure of variability in these data conforms to the general suggestions of the policy implementation literature that macro-level policy creation can produce varied responses at the local level. This conclusion is also reflected in the content of these policies (Schoenherr & Williams-Jones, 2011), which shows an evolutionary process wherein policies were developed to fit a particular niche within the academic culture of a particular institution. Deviation from Tri-Council inception date. To examine whether the trend of policy creation around the 1994 release of the TCPS-IRS was statistically significant, all creation dates were entered into a chi-squared test. This analysis revealed a significant difference for the frequency of policy creation dates, Χ2(8) = 42.813, p < .001, confirming that there was systematic variability in terms of when RIM policies were created. Moreover, a onesample t-test was performed on the data to examine whether the mean policy creation date (= 1995.2, SD = 4.03) differed from 1995, respectively. No significant difference was found when 1995 was used as the criterion year, t(31) = .263, p = .794, suggesting that policy creation dates were not significantly different from 1995. By contrast, a significant difference was observed when both 1997, t(31) = –2.546, p = .016, and 1999, t(31) = –3.950, p < .001, were used as the criterion years. Taken together, these results suggest that the TCPS-IRS creation date represents a specific landmark, but that the TCPS-FIR in 1996 and the TCPS1 in 1998 did not have a significant impact on RIM policy creation in Canadian institutions. Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 67 Table 1. Curve Estimates from Functions for Policy Creation Dates Model 1-Segment Linear (2) Logarithmic (2) Sigmoid (3)* Logistic (3) 2-Segment Linear (4) Hill (4) 3-Segment Linear (5)+ RSS 260.35 260.18 47.30 47.32 218.41 253.42 24.27 AICc 36.28 36.28 25.73 25.74 46.70 48.04 62.93 Δi(AIC) 10.55 10.54 0.00 0.00 20.97 22.31 37.20 wi(AIC) 0.005 0.005 1.000 0.998 0.000 0.000 0.000 R2 0.80 0.80 0.96 0.96 0.82 0.80 0.98 CI 0.05 - Proportion of variance accounted for by the respective curves and critical value. The superscripted plus indicates the model with the highest correlation, and the asterisk indicates the best-fitting function in terms of AICc and weighted difference scores. The number of model parameters are contained within parentheses, in the first column. FRSQ Provincial Policy Initiative: Creation Date The significance of the TCPS-IRS release in 1994 seems clear, given the mean and modal values of policy dates. An alternative hypothesis, though, is that provincial policies could have a significant and maybe even more important impact on RIM policies. To test this hypothesis, we examined the policy creation dates of institutions in the province of Québec, for two reasons. First, given that French is the official language in Québec—as compared with English in the rest of Canada, with the exception of New Brunswick, which is officially bilingual—this linguistic difference could make provincial policies more relevant than federal policies. Thus, although the culture of science might be identified by at least some universal norms (see, e.g., Merton, 1942/1973, 1957; cf. Ziman, 2000), their instantiation and communication through institutional policies are likely to be shaped by the relationship between higher education institutions and regional government. Second, during the period of RIM policy creation examined in our study, a similar but far more inclusive set of standards related to the TCPS-IRS was developed by Québec’s health research funding agency, the FRSQ. In addition to the Plan d’action ministeriel en éthique de la recherché et en intégrité scientifique (PAM; Gouvernement du Québec Ministère de la Santé et des Services sociaux, 1998), comprehensive measures were outlined by FRSQ in the Guide d’éthique de la recherche et d’intégrité scientifique (Guide; Fonds de la recherche en santé du Québec, 2003). Finally, given that the basis of the legal system in Québec (civil law, modeled on that of France) differs from what is used in the rest of Canada (common law, following that of the UK and other Commonwealth countries), we might expect legal standards in Québec to be accommodated in RIM policies in a different manner than elsewhere in Canada. However, when subjected to the same general analysis as for the other RIM policies, our data do not suggest that the FRSQ policy had a statistically significant impact. Of the policies implemented by Québec institutions in our study (n = 8), the majority were created in the same period as in all other Canadian institutions (Mo = 1995; Mdn = 1995), with only Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 68 one policy created after this date. Given that neither the date of the PAM (Gouvernement du Québec Ministère de la Santé et des Services sociaux, 1998) nor the date of the more specific Guide (Fonds de la recherche en santé du Québec, 2003) aligns with these numbers, it is reasonable to conclude that these provincial documents were not the primary impetus for institutional policy creation in Québec. This suggests that federal rather than provincial guidelines are the principle reason for Canadian institutions’ creation of RIM policies. In a similar fashion to the overall analysis, all creation dates for Québec institutional RIM policies were entered into a chi-squared test. Due to the restricted range and small sample size, no significant difference was found between the policy creation dates, Χ2(3) = 6.00, p = .112, nor was any significant difference found between the mean policy creation date (= 1996, SD = 4.11) and 1995 as the criterion year, t(7) = .689, p = .513. Analyses of the years immediately following the PAM release in 1998 revealed a marginally significant difference for 1999, t(7) = –2.067, p = .078, and a significant difference for 2004, t(7) = –5.511, p < .005. Thus, while it might be reasonable to assume that the Québec Ministry of Health’s PAM could have had an impact on policy creation dates, this does not appear to have been the case. Again, our data indicate that the creation of the TCPS-IRS in 1994 represents a specific landmark for scientific integrity policies across Canada. We must interpret these data cautiously, given the small sample size available for analysis. Institutional policies are rarely static but are instead occasionally (although not frequently) modified to respond to emerging challenges. So, it is possible that provincial guidelines that followed federal policies might have informed later changes to the RIM policies of Québec institutions. Moreover, if, as our data suggest, federal initiatives are the principle drivers of policy creation and modification, then the release of the TCPS-FIR and revisions to the TCPS1 should also be visible within institutional policy modification dates. The analysis presented in the next section considers what factors might have affected later RIM policy modifications. Policy Modification Dates The first notable property in the distribution of modification dates in our study sample is the relative lack of variability. Of the 32 documents with creation dates that we assessed, only half (n = 16) appear4 to have been modified in the date range examined (i.e., 1985 to 2006), as evidenced by the lack of modification dates cited within these documents. In this case, the mode (Mo = 2004) and median (Mdn = 2000) modification dates differ more substantially than in the previous creation-date analysis. Supporting the lack of variability in the modification of Canadian RIM policies, a chi-squared analysis did not reveal a significant difference for these frequencies, Χ2(10) = 3.25, p = .975. Given that no general difference was observed, no further analyses were conducted. The analysis of Québec policies is discussed below only in descriptive terms. The lack of subsequent policy events affecting Canadian institutional RIM policies is also evidenced by the fact that only two policies were modified in the 1996–1997 period, suggesting that the release of the TCPS-FIR had little effect on Canadian institutions. This finding is particularly surprising given the greater clarity that these guidelines appear to provide in terms of the content (recommendations for specific procedures) for institutional RIM policies. In Québec, however, there is some evidence for the influence of provincial policies. The majority (n = 5) of policy modification dates were during or after the release Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 69 of the PAM in 1998, with two documents being revised in 2004 after release of the Guide in 2003. Drawing solid conclusions about the influence of the PAM is nonetheless somewhat problematic; due to the small number of policies in our sample that were modified, we cannot determine whether there are statistically significant trends. If, however, we extrapolate from the general trend evidenced in policy creation dates, then we would not expect considerable modification in Québec institutional policies to have followed the release of the TCPS-FIR, although provincial standards might still have affected policy modifications. Considerations and Criticisms Some important limitations of our analyses should be noted. Our study only examined the creation of (1) federal research integrity policies and (2) the policies of a single province, Québec, the latter as a case study that might demonstrate a divergent trend in policy-creation activity. This was a reasonable approach given that some institutional RIM policies explicitly cite the policies or guidelines of research funders as a key consideration (Schoenherr & Williams-Jones, 2011). However, we in no way wish to claim that the TCPS-IRS is the only factor that affected the creation of RIM policies. One of the first RIM policies in Canada, that of McGill University, was created in 1985, a date that coincides with the American federal initiatives noted above. Nonetheless, the fact that aside from this one outlier, the majority of RIM policies were created after the TCPS-IRS in 1994 is strong evidence that federal granting agencies represented a central motivation for institutional policy change. When cautiously interpreted, the pattern of results evidenced in Canadian RIM policies provides evidence for models of organizational change (e.g., DiMaggio & Powell, 1983; Kraatz & Zajac, 1996). With the exception of a few institutions, the majority of Canadian universities produced policies within a year of the TCPS-IRS release. This is consistent with models of organizational change that emphasize adaptive similarity as a response to institutional competition (DiMaggio & Powell, 1983; Meyer & Rowan, 1977). In the present case, this competition represents compliance in order to continue receiving research funding, while also likely reflecting the desire to maintain institutional legitimacy (Meyer & Rowan, 1977; Suchman, 1995). However, in our previous study of RIM policy content (Schoenherr & Williams-Jones, 2011), we found evidence of considerable variability in terms of what policy elements were included (e.g., types of behaviour classified as research misconduct, explicit reference to national standards). Such deviation can be taken as evidence that institutional decision makers have attempted to accommodate the stakeholders and competencies of their organizations (e.g., Cohen, et al. 1972; Kraatz & Zajac, 2001), and it supports the role of institutional culture in organizational change within higher education (e.g., Kezar & Eckel, 2002). In general, idiosyncrasies evidenced within institutional policies, and pressure to adapt to global features of the policy landscape, suggest that RIM policy development can be accounted for by evolutionary models. Before drawing generalizable conclusions about the impact of federal agencies on science policy, we would need further evidence in other related policy areas, such as research ethics involving human participants, or conflict of interest—e.g., following the creation of the TCPS1 in 1998, its major revision (TCPS2) in 2010, and the update in 2014. Moreover, studies of the decision-making process are required to discern what factors determine and define this process (e.g., Volberda & Lewin, 2003). Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 70 A related concern stems from the quantitative technique that we employed, i.e., diffusion modelling (Rogers, 2003; Valente, 1995). One might argue that directly contacting university administrators involved in the creation of RIM policies to investigate their motivations would have been a more effective means of assessing the impact of the TCPS-IRS or other federal and provincial initiatives (HAL, 2009). While such an approach would clearly provide rich and pertinent information and certainly warrants further research, it does not undermine the utility of our approach, which was consciously chosen because of its more distal nature. By assessing policy-creation dates, we removed ourselves from the assumptions that an informant would provide accurate and unbiased information about the underlying motivations for policy creation. Moreover, using techniques similar to those we employed in our study, it would be possible to conduct meaningful comparisons between international policy-development initiatives on research integrity. Added to these studies, research should also assess the extent to which policy implementation has been effective in ameliorating cases of misconduct. In the Canadian context, this would benefit from the retention of statistics by an autonomous organization similar to the ORI in the United States. Conclusion In the present study, we considered one possible factor in the creation of RIM policies at Canadian academic institutions: the actions of government funding agencies at both the federal and the provincial (specifically Québec) levels. Our results provide compelling evidence that soon after the release of the TCPS-IRS in 1994, most Canadian universities quickly followed suit by creating RIM policies in or around 1995. Moreover, we observed that provincial guidelines released by the province of Québec, i.e., the PAM in 1998 and the Guide in 2003 (which provide comprehensive policies on research integrity and research ethics), did not have as large an impact on the creation of policy documents in Québec-based institutions but might nonetheless have affected subsequent modifications to policies within the province. Taken together, this suggests that the 1994 federal initiative to establish national guidelines addressing research integrity played an important role in changing the RIM policy landscape in Canada. In this sense, the Tri-Council has performed a role similar to that of the ORI in the United States. This similarity, however, must not be exaggerated, given that the ORI’s role has been one of enforcement (i.e., it has policing powers, including an investigative branch) and the development of a comprehensive research program and educational initiatives. Neither do we mean to suggest that the inception of the TCPS-IRS is the only or primary reason for policy creation. It is possible that a need for change was present within the higher education community and that the actions of the Tri-Council proved to be the final impetus for creation. Nevertheless, institutional responsiveness to federal policy development indicates a potentially strong and positive role for national agencies (e.g., Kondro, 2007). The finding that universities are affected by federal policy does not, of course, in any way imply that national standards are sufficient or appropriate. Steneck (1999) articulated this point particularly well when he noted that “‘change’, however, is not the same as ‘accomplishment’. Change simply means that conditions before and after are not the same. Accomplishment is effecting change for a purpose” (p. 162). Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 71 Notes 1. These correspond to what DiMaggio and Powell (1983) referred to as coercive and normative pressures in isomorphic organizational change. 2. It is worth noting that in contrast to the Tri-Councils, which separated policies on research integrity (i.e., the TCPS-TAF) from those dealing with research ethics for human subjects (i.e., TCPS1/2), Québec policies have tended to address both mandates within the same document. 3. With the reorganization the three Québec funding councils in 2010—i.e., the creation of a centralized Fonds du recherche du Québec, but with individual councils still in charge of grant programs—there has been a move to harmonize policies, including those for research ethics and integrity. A new policy on the responsible conduct of research—Politique sur la conduite responsable en recherche—was published in September 2014. 4. Based on explicit modification dates included in the published policies. Acknowledgements The authors would like to thank Tyler Burleigh for his advice in obtaining corrected measures of Akaike’s Information Criterion, and Zubin Master for his constructive feedback on the manuscript. References Bechner, T., & Kogan, M. (1992). Process and structure in higher education (2nd ed.). London, UK: Routledge. Burnham, K. P., & Anderson, D. R. (2002). Model selection and multimodel inference: A practical information-theoretic approach (2nd ed.). New York, NY: Springer. Callaway, E. (2011). Report finds massive fraud at Dutch universities. Nature, 479, 15. Canadian Association of University Teachers. (2003). CAUT policy statement on fraud and other misconduct in academic research. Canadian Federation for the Humanities and Social Sciences. (2006). The statement on research ethics and scholarly integrity. Canadian Food Inspection Agency. (2010). Policy on the responsible conduct of research and development. Retrieved from http://inspection.gc.ca/aboutthe-cfia/organizational-information/vision-and-mission/responsible-conduct/ eng/1319825850028/1319826099857 CHPS Consulting. (2000). Analysis of institutional policies for responding to allegations of scientific misconduct. Rockville, MD: Office of Research Integrity. Retrieved from http://ori.dhhs.gov/documents/institutional_policies.pdf Cohen, M. D., March, J. G., & Olsen, J. P. (1972). A garbage can model of organizational choice. Administrative Science Quarterly, 17, 1–25. Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 72 Council of Canadian Academies. (2010). Honesty, accountability and trust: Fostering research integrity in Canada. Retrieved from http://www.scienceadvice.ca/uploads/ eng/assessments%20and%20publications%20and%20news%20releases/research%20 integrity/ri_report.pdf Council of Science and Technology Advisors. (1999). Science advice for government effectiveness (SAGE). Ottawa, ON: Industry Canada. Council of Science and Technology Advisors. (2001). S&T excellence in the public sector (STEPS). Ottawa, ON: Industry Canada. DiMaggio, D. J., & Powell, W. W. (1983). The iron cage revisited: Institutional isomorphism and collective rationality in organizational fields. American Sociological Review, 48, 147–160. Fonds de la recherche en santé du Québec. (2003). Guide d’éthique de la recherche et d’intégrité scientifique. Retrieved from http://www.frqs.gouv.qc.ca/ documents/10191/186005/Guide_ethique_integrite_2003.pdf/111344c8-1208-42b5a0d5-2495f555ed12 Fonds de la recherche en santé du Québec. (2008). Standards du FRSQ sur l’éthique de la recherche en santé humaine et l’intégrité scientifique. Retrieved from http://www. frqs.gouv.qc.ca/fr/ethique/pdfs_ethique/Standards.pdf Gouvernement du Québec Ministère de la Santé et des Services sociaux. (1998). Plan d’action ministeriel en éthique de la recherché et en intégrité scientifique. Retrieved from http://ethique.msss.gouv.qc.ca/lethique-en-bref/plan-daction-ministeriel.html Government of Canada. (2005). Public servants disclosure protection act. Retrieved from laws-lois.justice.gc.ca/eng/acts/P-31.9/ Government of Canada. (2006). Federal accountability act. Retrieved from laws-lois. justice.gc.ca/eng/acts/F-5.5/ Government of Canada. (2012a). Policy on conflict of interest and post-employment. Retrieved from http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=25178&section=text Government of Canada. (2012b). Values and ethics code for the public sector. Retrieved from http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?section=text&id=25049 Government of Canada. (2014). Directive on conflict of interest, gifts and hospitality, and post-employment. Retrieved from http://www.cra-arc.gc.ca/crrs/wrkng/ cnflctntrstcdgdlns-eng.html Greene, P. J., Durch, J. S., Horwitz, W., & Hooper, V. (1985). Policies for responding to allegations of fraud in research. Minerva, 23, 203–215. Hannan, M. T., & Freeman, J. (1984). Structural inertia and organizational change. American Sociological Review, 49, 149–164. Health Canada. (1999). The Best Practices Initiative: Best practices for the conduct, management and use of science in the Government of Canada. Ottawa, ON: Government of Canada. Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 73 Hickling Arthurs Low. (2009). The state of research integrity and misconduct policies in Canada. Retrieved from http://www.nserc-crsng.gc.ca/_doc/NSERC-CRSNG/HAL_ Report_e.pdf Jasanoff, S. (2010). Testing time for climate science. Science, 328, 695–696. Kadushin, C. (2012). Understanding social networks: Theories, concepts, and findings. New York, NY: Oxford University Press. Kezar, A., & Eckel, P. D. (2002). The effect of institutional culture on change strategies in higher education. The Journal of Higher Education, 73, 435–460. King, D. A. (2004). The scientific impact of nations: What different countries get for their research spending. Nature, 430, 311–317. Kingdon, J. (2002). The reality of public policy making. In M. Danis, C. Clancy, and L. Churchill (Eds.), Ethical dimensions of health policy (p. 97–116). New York, NY: Oxford University Press. Kondro, W. (2007). Call for arm’s-length national research integrity agency. Canadian Medical Association Journal, 176, 749–750. Kraatz, M. S., & Zajac, E. J., (1996). Exploring the limits of the new institutionalism: The causes and consequences of illegitimate organizational change. American Sociological Review, 61, 812–836. Kraatz, M. S., & Zajac, E. J., (2001). How organizational resources affect strategic change and performance in turbulent environments: Theory and evidence. Organization Science, 12, 632–657. LaFollette, M. C. (1994). The politics of research misconduct: Congressional oversight, universities, and science. Journal of Higher Education, 65, 261–285. Lind, R. A. (2005). Evaluating research misconduct policies at major research universities: A pilot study. Accountability in Research, 12, 241–262. Lock, S., Wells, F., & Farthing, M. (2001). Fraud and misconduct in biomedical research. London, UK: BMJ Books. Master, Z., McDonald, M., & Williams-Jones, B. (2012). Promoting research on research integrity in Canada. Accountability in Research, 19, 47–52. Medawar, P. (1976, April 15). The strange case of the spotted mice. New York Review of Books. Merton, R. K. (1957). Priorities in scientific discovery: A chapter in the sociology of science. American Sociological Review, 22, 635–659. Merton, R. K. (1973). The normative structure of science. In N. W. Storer (Ed.), The sociology of science (pp. 267–278). Chicago, IL: University of Chicago Press. (Original work published 1942) Meyer, J. W., & Rowan, B. (1977). Institutionalized organizations: Formal structure as myth and ceremony. American Journal of Sociology, 83, 340–363. National Research Council. (1996/2013). NRC research integrity policy. Retrieved from http://www.nrc-cnrc.gc.ca/eng/about/policies/research_integrity/index.html Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 74 National Science Foundation. (1987). Misconduct in science and engineering research: Final regulations. Federal Register, 52, 24466–24470. Normile, D. (2014, May). RIKEN to check 20,000 papers for doctored images and plagiarism. Science Insider. Retrieved from http://news.sciencemag.org/ asiapacific/2014/05/riken-check-20000-papers-doctored-images-and-plagiarism Palumbo, D. J., & Calista, D. (Eds.). (1990). Implementation and the policy process. New York, NY: Greenwood Press. Palumbo, D. J. & Oliverio, A. (1989). Implementation theory and the theory-driven approach to validity. Evaluation and Program Planning, 12, 337–344. Pencharz, P. (2007). An examination of research integrity issues pertaining to Memorial University of Newfoundland. Report prepared for Memorial University. Pfeffer, J., & Salancik, G. R. (2003). The external control of organizations: A resource dependence perspective. New York, NY: Harper & Row. (Original work published 1978) Public Health Services. (1986). Policies and procedures for dealing with possible misconduct in science. Washington, DC: National Institutes of Health. Rogers, E. M. (1962/2003). Diffusion of innovations (5th ed.). New York, NY: Free Press. Royall, R. M. (1997). Statistical evidence: A likelihood paradigm. London, UK: Chapman & Hall. Savage, D. (1994/2003). Discussion paper on fraud and misconduct in academic research and scholarship. Scheirer, M. A. (1987). Program theory and implementation theory: Implications for evaluators. In L. Bickman (Ed.), New directions for program evaluation, vol. 33: Using program theory in evaluation (pp. 40–68). San Francisco, CA: Jossey-Bass. Schoenherr, J., & Williams-Jones, B. (2011). Research integrity/misconduct policies of Canadian universities. Canadian Journal of Higher Education, 41, 1–17. Smith, R. (2005). Investigating the previous studies of a fraudulent author. The British Medical Journal, 331, 288–291. Snyder, E. Y., & Loring, J. F. (2006). Beyond fraud—stem-cell research continues. New England Journal of Medicine, 354, 321–324. Starr, P. (1982). The social transformation of American medicine. New York, NY: Basic Books Steneck, N. (1994). Research universities and scientific misconduct: History, policies, and the future. Journal of Higher Education, 65, 310–330. Steneck, N. (1999). Confronting misconduct in science in the 1980s and 1990s: What has and has not been accomplished? Science and Engineering Ethics, 5, 161–176. Suchman, M. C. (1995). Managing legitimacy: Strategic and institutional approaches. Academy of Management Review, 20, 571–610. Federal and Institutional Integrity System / J. R. Schoenherr & B. Williams-Jones 75 Tri-Council. (1996). Framework for Tri-Council review of institutional policies dealing with integrity in research. Retrieved from http://www.nserc-crsng.gc.ca/NSERCCRSNG/Policies-Politiques/frameworkintegrity-cadrereferenceintegrite_eng.asp Tri-Council. (1998/2010/2014). Tri-Council policy statement: Ethical conduct for research involving humans. Retrieved from http://www.pre.ethics.gc.ca/pdf/eng/tcps22014/TCPS_2_FINAL_Web.pdf Tri-Council. (2006). Tri-Council policy statement: Integrity in research and scholarship. Retrieved from http://www.nserc-crsng.gc.ca/NSERC-CRSNG/PoliciesPolitiques/tpsintegrity-picintegritie_eng.asp Tri-Council. (2011). The Tri-Agency Framework: Responsible conduct of research. Retrieved from http://www.rcr.ethics.gc.ca/_doc/Framework-CadreReference_eng.pdf Valente, T. W. (1995). Network models of the diffusion of innovations. Cresskill, NJ: Hampton Press. Volberda, H., & Lewin, A. (2003). Co-evolutionary dynamics within and between firms: From evolution to co-evolution. Journal of Management Studies, 40, 2111–2136. Wagenmakers, E.-J., & Farrell, S. (2004). AIC model selection using Akaike weights. Psychonomic Bulletin & Review, 11, 192–196. Ziman, J. (2000). Real science: What it is and what it means. Cambridge, UK: Cambridge University Press. Contact Information Jordan Richard Schoenherr Department of Psychology Carleton University [email protected] Jordan Richard Schoenherr is an adjunct professor in the Department of Psychology, Carleton University. He has worked as a consultant and principal researcher at Health Canada, developing scientific integrity standards. His research interests include cognition, scientific explanation, research integrity, and evolutionary approaches to the development of epistemic communities. His current research projects include examining the dissociation of multiple learning and decision-making systems in human populations, and the properties of social organization in the evolution of norms in science and medicine. Bryn Williams-Jones is an associate professor and the Director of the Bioethics Programmes, Department of Social and Preventive Medicine, School of Public Health, at the University of Montréal. Bryn is an interdisciplinary scholar who employs analytic tools from applied ethics, health policy, and the social sciences to explore the socio-ethical implications of health innovations. His current research focuses on professional ethics, research integrity, and the management of conflicts of interest that arise in the context of university research, particularly in university–industry relations.