H. Kanuka & J. Kelland / Has e-Learning Delivered on its Promises? 45 CSSHE SCÉES Canadian Journal of Higher Education Revue canadienne d’enseignement supérieur Volume 38, No. 1, 2008, pages 45-65 www.ingentaconnect.com/content/csshe/cjhe Has e-Learning Delivered on its Promises? Expert Opinion on the Impact of e-Learning in Higher Education Heather Kanuka Athabasca University Jennifer Kelland University of Alberta ABSTRACT The purpose of this study was to determine the extent of agreement among experts on the impact of e-learning technology in Canadian higher education learning experiences. Fourteen participants who are experts in e-learning in higher education agreed there are contentions about e-learning technologies in the following areas: (1) a platform for ideal speech; (2) greater opportunities for interactions; (3) the extent to which communities of learners can be created; (4) provision of a new kind of learning environment; (5) a platform for discussions; (6) demand for e-learning by students; (7) the degree to which the environment is equal and equitable; and (8) the quality of the learning experience. The findings of this study indicate that the value of e-learning requires further research before higher education leaders andteacher-practitioners are willing to incorporate them in teaching practices and policy documents. RÉSUMÉ Le but de cette étude était de déterminer le degré d’accord parmi des experts à propos de l’impact de la technologie d’apprentissage en ligne dans l’enseignement supérieur au Canada. Les quatorze experts consultés s’entendent pour dire qu’il existe des controverses au sujet 46 CJHE / RCES Volume 38, No. 1, 2008 des technologies d’apprentissage en ligne dans les domaines suivants : (1) les technologies comme moyen de communication permettant une situation idéale de parole; (2) l’augmentation des occasions d’interaction; (3) la possibilité de créer des communautés étudiantes; (4) la possibilité de fournir un nouveau genre d’environnement d’apprentissage; (5) l’offre d’un moyen de communication pour la discussion; (6) la demande étudiante pour l’apprentissage en ligne; (7) le degré d’équité et d’égalité de l’environnement d’apprentissage offert; et (8) la qualité de l’expérience d’apprentissage. Les résultats de cette étude indiquent que davantage de recherche est nécessaire pour éclairer les décideurs et les praticiens de l’enseignement supérieur sur les politiques et les pratiques d’apprentissage en ligne. As a result of the increased integration of Internet and web-based communication technologies, which we refer to as simply “e-learning,” higher education has moved into a third decade of change in how courses and programs are designed and delivered. During this time, e-learning technologies have produced an intense, immediate, and disruptive transformation on higher education (Archer, Garrison, & Anderson, 1999). Enthusiastic early adopters of e-learning argued that there are many new possibilities offered by these technologies for Canadian educators in higher education. These enthusiastic accounts were soon followed by the creation of task forces to assess e-learning’s potential. For example, a task force comprised of administrators, researchers and other specialists in e-learning was established by the Natural Sciences and Engineering Research Council (NSERC) to explore the (1) opportunities and challenges presented by virtual programs in the postsecondary sector and (2) future policy for research funding for the development of online learning. The following conclusion was made by the task force members about online learning and distributed online research communities: [e-Learning] comes at an excellent moment for Canada. It enables research collaboration to begin to conquer geographic limitations. It provides the key to a radical expansion in the national research effort. And it opens doors for Canadian leadership in innovation for online learning. (Report of the Task Force on Virtual Universities and Online Learning, 2002, retrieved from http://www.nserc.ca/about/researchbase_e.htm) Elsewhere, leaders in the field of higher education asserted that e-learning technologies can respond effectively to accelerating global competition (Daniel, 2000), increase the quality of learning experiences (Garrison, 2002), remove situational barriers (Bates, 2005), and be more cost effective than face-to-face learning (Twigg, 2003). With the continued rise of e-learning technologies, a frequently asked, and investigated, question has been: Can e-learning deliver on these promises? H. Kanuka & J. Kelland / Has e-Learning Delivered on its Promises? 47 Review of Relevant Literature Beginning almost three decades ago researchers have attempted to answer this question (e.g., Hiltz & Turoff, 1978, 1985; Hiltz, Johnson, & Turoff, 1986); more recently, extensive literature reviews have been conducted. Commonly cited reviews of the research on the impact of e-learning technologies have tended to reveal inconsistent results, such as (1) no significant difference with classroom based instruction (Russell, 1999), (2) some indication of effect size differences under specific circumstances (Abrami et al., 2006; Bernard et al., 2004; Pascarella & Terenzini, 2005), or (3) inconclusive results (Rourke, 2005). And although, in general, the literature reviews concluded that the research has been uneven, there is evidence that educators in higher education believe that e-learning technologies (1) have a positive impact on course delivery and student learning, (2) are effective at achieving greater student participation and student interest, and (3) allow opportunities to improve critical thinking (Saundercook & Cooper, 2003). Consistent with these perceptions, a good deal of the literature also suggests that e-learning can transform learning experiences in positive ways, resulting in an increase in the quality of learning experiences (Garrison & Anderson, 2003; Heckman & Annabi, 2005; McKnight, 2001). In particular, it has been argued that e-learning technologies facilitate the development of argument formation capabilities, improve written communication skills, require greater complex problem solving abilities, and increase opportunities for critical and reflective thinking (Abrami & Bures, 1996; Garrison, Anderson, & Archer, 2001; Hawkes, 2001; Winkelmann, 1995). Lapadat (2002) observes, for example, that with asynchronous text-based Internet technology learners have the means to carefully compose their ideas and thoughts into a written form of communication. This attention to writing, in combination with the time-lag inherent in asynchronous communication, provides students with opportunities for critical reflection, which is necessary for higher-order learning (Garrison & Anderson, 2003). And although these, and similar, articles are often cited as fait accompli benefits of e-learning, thorough and rigorous reviews of the literature have yet to show a consistent and reliable body of knowledge indicating that these benefits are an outcome of the use of e-learning technology. For example, with respect to higher levels of learning (e.g., constructivist pedagogy), we continue to know little about what works, why, in what ways, and under what conditions – on campus, or off (Tenebaum, Naidu, Jegede & Austin, 2001). Indeed, much of the research on this topic reveals that perhaps the only aspect that research has shown, with consistency, is that deep and meaningful learning is not easily achieved in the e-learning classroom (Kanuka, 2005; McKlin, Harmon, Evans & Jones, 2002; Meyer, 2003; Nussbaum, Hartley, Sinatra, Reynolds, & Bendixen, 2002; Pawan, Paulus, Yalcin & Chang, 2003; Thomas, 2002; Vaughan & Garrison, 2005). These findings should be no surprise for the e-learning classroom, as prior research has shown that these benefits are not easily achieved in the on-campus classroom either (e.g., Biggs, 1982; Ramsden, 1992). What marks 48 CJHE / RCES Volume 38, No. 1, 2008 the difference between the on-campus classroom and the e-learning classroom, in regard to effective learning environments, is that e-learning advocates have made, and continue to make, claims that e-learning can provide more effective learning environments than face-to-face classroom experiences. For example, the federally funded Pan-Canadian Advisory Committee for Online Learning arrived at the following conclusion: [T]he virtual classroom will offer a high-quality leaning experience. . . These improvements will stem from the ever-expanding depth and breadth of knowledge in our colleges and universities, the innovations unleashed by online learning. . . These will bear fruit in an online learning experience that is enriching, deep and varied, and capable of passing on the most basic skills and capacity for critical judgment and reasoning. . . Learners will find the learning opportunity most suited to their individual needs, situation, income, language and learning styles, whether online at home, at work, or at a public access site. . . they will find online learning opportunities they need as a basis for personal fulfillment. (E-learning e-volution, 2001, retrieved from: http://www.cmec. ca/postsec/evolution.en.pdf#search=%22%22e-volution%22%22) A second possible reason for the incongruence between our perceptions of e-learning’s benefits and the research findings stems from reviews of the literature that have been inappropriately generalized to all aspects of e-learning in higher education environments. For example, the meta analysis by Bernard et al. (2004) investigated quantitative research in the areas of achievement, attitudes, and retention (e.g., drop out rates) outcomes in distance education. The most common generalizations have been (1) to e-learning, when the study focused on distance education (semi-permanent separation of learner and instructor) (2) to higher education, when the study drew on diverse educational environments. A careful examination of the research findings reveals significant heterogeneity in each subset. The authors conclude the following: It is simply incorrect to state that DE is better than, or worse than, or even equal to classroom instruction on the basis of mean effect sizes and heterogeneity. . . The mistake that a number of previous reviewers have made, from early narrative reviews (e.g., Moore & Thompson, 1990) to more recent reviews (e.g., Russell, 1999), is to declare that DE and classroom instruction are equal [or better] without examining the variability surrounding their difference. Wide and unexplained variability precludes any such simplistic conclusion. (p. 406) A more recent review of e-learning research in Canada was conducted by this same group of researchers (Abrami et al., 2006). Reviewing achievement, motivation/satisfaction, interactivity/communication, social demands, attrition/retention and learning flexibility within the post secondary sector, this H. Kanuka & J. Kelland / Has e-Learning Delivered on its Promises? 49 same group of researchers concludes that “there is limited empirical research to assess the benefits” (p. 3) and that “this review. . . does not readily present us with evidence of best practices and ‘what works’ in e-learning” (p. 32). Without question it is difficult to make sense of the research literature on e-learning and, in turn, to use it to make informed decisions. Moreover, most of us do not have the time to review all of the literature in a critical and detailed manner. As we move into a third decade of exploring e-learning in higher education with good evidence that it will continue to be an important part of the learning experience in Canada, it is important to gain a well informed understanding of its impact. How best to assess the impacts has been, and will continue to be, debated. The most frequent way researchers have attempted to gain a general understanding of the effects of e-learning has been by conducting literature reviews, such as meta analyses and/or narrative reviews. Although useful in many respects, these methodologies have limitations. It is possible to address these limitations by using diverse approaches and compare results. As an alternative approach to a review of the literature, we explored the impact of e-learning by bringing together expert researchers in Canada to establish, collectively, where there is agreement on the impact of e-learning technology. Theoretical Framework: The Non-Neutrality of Technology Embedded in our opinions on e-learning technologies are views on the (non)neutrality of technology. The debate revolves around whether technologies are neutral and whether biases can arise only from the ways in which technologies are used by teachers and students – or whether biases can occur through the technologies themselves. An analogy to contextualize and bring relevance to these views can be gained from the catch phrase “people kill people; not guns.” A comparable catchphrase in the field of e-learning might be “educators reshape education; not technologies.” Many educational technologists agree with Jonassen (1996), who asserts that “carpenters use their tools to build things; the tools do not control the carpenter. Similarly, computers should be used as tools for helping learners build knowledge; they should not control the learner” (p. 4). While Jonassen’s argument sounds solid in its rationale, media theorist Marshall McLuhan has suggested otherwise. Specifically, even though the neutrality of a tool speaks to our common sense with respect to the ways in which tools are used, McLuhan and Fiore (1962) maintain that media can transform society and the human psyche profoundly – their famous aphorism, “the medium is the message,” gives pause to the assumption of the non-neutrality of technology. Building on such an assumption, Chandler (1996) postulates that media shapes our experiences, and they do so in part through their selectivity. In particular, Chandler asserts that when we interact with media we act and are acted upon, use and are used. Consistent with McLuhan’s views, Postman (1993) maintains that “embedded in every tool is an ideological bias, a predisposition to construct the world as one thing rather than another, to value one thing over another, to amplify one sense or skill or attitude more loudly than another” 50 CJHE / RCES Volume 38, No. 1, 2008 (p. 13). Postman and McLuhan hold definitive views about the non-neutrality of technology. Others, such as Ihde (1979) and Dahlberg (2004) have adopted moderate views of technological determinism, or a “non-reductionist” orientation. Ihde, for example, suggests that the use of instruments both amplifies and reduces human experiences. The belief that technologies are non-neutralwas an assumption underpinning this study. RESEARCH DESIGN Background of the Study This study builds on the results of a previous study where the aim was to determine how the use of e-learning technologies impact higher education learning experiences. We acknowledge that there are many asynchronous and synchronous Internet communication and social software tools currently being used as e-learning technologies (i.e., Centra/Elluminate, iVisualize/vocalize, Skype, Blogs, Wiki, Podcast, Groove). However, asynchronous group communication tools continue to be the dominant technology for e-learning courses in higher education. As such, this study and the previous study were concerned only with the use of asynchronous group communication tools as the technology for e-learning. Participants selected for the previous study were 12 administrators of elearning programs from Western Canada. Our earlier findings (Kanuka & Rourke, 2008; Kanuka & Rourke, 2006) included the following themes: ideal communication/writing platforms, communities of learners, new platforms for learning, student centred learning, equal access, quality of teaching and course design, and opportunities for facilitating discussions. However, there was a lack of consensus on whether these changes negatively or positively affected the higher education learning experience. For example, the participants agreed that e-learning provides the possibility for a more equal learning environment because we cannot see the skin colour, age, gender, physical disabilities – and so forth of other participants. Some argued the lack of visible physical traits was a positive aspect about e-learning. Specifically, when physical characteristics are not visible, there is less opportunity for discrimination and learning occurs on a more equal platform than in face-to-face learning. Other participants, though, noted that when our physical presence is not seen, we are not forced to confront our biases and as a result, learning to understand and value diversity is reduced as a consequence of e-learning technologies. Although the outcomes of this study provided good insights and consensus about whether e-learning is having an impact, we concluded that the diverse perspectives were likely generated by a small, homogeneous sample who were drawing on institution-specific practices and experiences. Building on these results we brought together expert researchers from across Canada, with crossdisciplinary backgrounds, to determine where agreement could be established in the areas identified in this previous study. H. Kanuka & J. Kelland / Has e-Learning Delivered on its Promises? 51 METHOD When considering a method to use for this study, we found mainstream data gathering processes, such as survey research or individual interviews, too restrictive. Given this circumstance, we made a decision to combine aspects of group interviews and the Delphi technique, a hybrid technique that we came to refer to as a “deliberative inquiry.” The term deliberative was chosen because we asked the participants to deliberate with each other about the issues arising from the previous study and arrive as a consensus on the impact of these issues. Bringing a group of people together can be a powerful way to collect data (Glesne & Peshkin, 1992). In particular, group interviews allow participants “to react to and build upon the responses of other group members. This synergistic effect of the group setting may result in the production of data or ideas that might not have been uncovered in individual interviews” (Stewart & Shamdasani, 1998, p. 509). An additional benefit is that they provide checks and balances on false or extreme views, making it fairly easy to assess the extent to which there is a consistent, shared view (Patton, 1990). The main assumption embedded in group interviews is that the views we hold are socially constructed and grow out of discussions with other people (Patton, 1987). Hence, an assumption of this method is that a group interview is effective at gathering data in a social context, where individual members consider their opinions against the opinions of others. Furthermore, because the aim of this study was to gain consensus on the benefits of e-learning technologies, we also made a decision to integrate aspects of the Delphi technique. The Delphi technique works by soliciting the opinion(s) of experts (usually in some sort of individual format) on an important issue or question in order to determine consensus (Langford, 1972; Linstone & Turoff, 2002). The researcher sometimes provides background materials or suggests reference material that participants can consult to better inform their position. If non-consensus occurs, participants are asked to defend, explain or change their opinions in order to move the group to a single best answer or consensus. An important feature of this process is the facilitation and encouragement of individuals to share the rationales for their opinions. Opinion sharing can be facilitated through distribution of written responses to questions, a structured discussion during a face-to-face meeting, or a real-time distributed conference (Anderson & Kanuka). For this study, we used a structured face-to-face meeting with an experienced group moderator. We were mindful of Mahatma Gandhi’s observation that honest disagreement is often a good sign of progress (Anderson & Kanuka, 2003). Therefore, we acknowledged the possibility that this study may not lead to a convergence of opinions, which is a limitation of this methodology. However, the resulting outcome, irrespective of whether or not an opinion synthesis occurs, may be more valid than other methodologies because of the acknowledgment and accommodation of opposing opinions (Kanuka, 2002). 52 CJHE / RCES Volume 38, No. 1, 2008 The moderator for the deliberative inquiry was an experienced group facilitator with expertise in communication technologies. This process necessitated that participants not just talked about the issues presented but also carefully weighed the alternative possibilities posed by others and assessed the consequences of those alternatives. Therefore, the moderator was key to eliciting meaningful information from each of the participants in a manner that remained respectful and safe when non-consensus arose (Fontana & Frey, 2005). PARTICIPANTS The success of the deliberative inquiry process is dependent on the diversity, expertise, and experience of the participants. With respect to consensus techniques, group members are usually purposely selected because participants are informed, interested and capable of providing high quality opinions about the topic under investigation (Langford, 1972). Participants in consensus groups draw, first, on their own experiences and opinions, and, then, they build upon that knowledge by considering the opinions and expertise of others. As such, the participants selected for this study were interested stakeholders with broad research and practical expertise in the area of e-learning technologies. Participants were selected carefully to ensure all regions in Canada, including Nunavut and Francophones from Quebec, were represented. Funding for this study allowed us to bring together a maximum of 14 participants from across Canada. Participant selection included both internal and external stakeholders (i.e, student, university and government stakeholders). With the exception of one participant (one student), participants had experience in e-learning and higher education for more than 10 years (with eight participants having 20 or more years experience in conducting related research). The search for participants was delimited to 14 individuals who had been recipients of scholarly and peer reviewed federal funding (e.g., SSHRC or NSERC). The final group of invited experts included eight women and six men. At the time of the study, two participants held federally funded Canada Research Chairs related to technology and education; all participants are well known in Canada for their contributions to e-learning. Data Collection and Analysis It was necessary to be sensitive to each participant’s views in order to encourage them to openly share their ideas and perspectives. Although there are advantages of group deliberation, a limitation is that this format makes it difficult to prevent one group member from dominating, and hence, shaping the entire conversation. Additionally, some participants might be uncomfortable sharing personal opinions in a group format. To address these limitations, the moderator played an active role in ensuring a trusting and respectful environment was maintained throughout the process. H. Kanuka & J. Kelland / Has e-Learning Delivered on its Promises? 53 The moderator began the discussion with a brief description of the theoretical framework, the impact of communication technologies, a review of the results of the previous study, and a number of case studies as examples. Also, follow-up interviews via either email or telephone were carried out after we had conducted a preliminary review of the data. The follow-up interviews were also used to begin member checking (Lincoln & Guba, 1985). The member check was used to ensure our understanding of the data was consistent with our participants’ understanding. We did this by summarizing, repeating and paraphrasing the participants’ words. To maintain trustworthiness and credibility of the research process, we kept an audit trail comprised of field notes, memos and observer comments (Bogdan & Biklen, 1998). In addition, we also applied negative case data (Glaser & Strauss, 1967) to reduce researcher selection and interpretation bias. Throughout the data analysis process we also used Becker and Geer’s (1960) recommendations for determining relative strength and intensity of manifestations in our data analysis. This process involved counting the number of times a topic or theme emerged, how often it occurred relative to the negative cases, and how widely the data occurred between participants. Using these techniques, we were able to more accurately analyze the frequency and distribution of data appearance, as opposed to simply counting the number of times a theme or category emerged. We also participated in peer debriefing meetings with our research assistants to help avoid researcher privileging, and to point out potential biases or inconsistent conclusions. FINDINGS The participants spoke often about the issues presented as misconceptions of e-learning technologies and as utopian and dystopian views. The terms myth and mythologies recurred numerous times when participants described e-learning technologies. Perhaps more importantly, the mood of the discussion took on a critical realist approach, in which assumptions and truisms were put forward relating to e-learning technologies and, invariably, challenged by the participants. The challenging of perceived truisms and assumptions (e.g., “best practices” for e-learning) by all participants suggests that consensus on the impact of e-learning technologies in higher education is a contentious topic. While acknowledging the complexity of the impact of e-learning technologies, our findings did suggest that there was consensus among participants on one theme: there are pervasive myths about the benefits of e-learning technologies. Following is a description of the myths about e-learning technology as described by the participants. 1. E-learning Provides a Platform for Ideal Speech One of the issues presented in the previous study (Kanuka & Rourke, 2006) generated considerable group deliberation in terms of myths. In particular, the 54 CJHE / RCES Volume 38, No. 1, 2008 Writing Skills category resulted in the following kinds of comments: There are mythologies about online learning. And one of these myths that’s been promoted, a lot, is that asynchronous text-based communication makes some sort of ideal speech situation available. And I think that’s been contested quite a bit in both research and opinion, especially recently. Likewise, another participant observed that given that so much of what gets written in these contrived online communities is empty ventriloquation, I’m not sure I see how or why this could or would improve people’s written abilities. I’m actually not sure either that good writing, apart from handwriting and grammar, is a matter of skills. Although there was general agreement that the kind of ideal speech discussed in much of the e-learning literature is a myth, the reasons why were not as clear. The group deliberated about the reasons why this type of communication is not occurring, with explanations that ranged from poor pedagogy and unskilled instructors to a need for improved technologies for facilitating e-learning discussions. 2. E-learning Provides Greater Opportunities for Interactions Considerable deliberation between participants revolved around the construct of interaction. It was acknowledged that it is possible that e-learning provides more interaction in terms of many-to-many versus one-to-many that can provide opportunities for learner engagement and knowledge sharing. However, this perspective was qualified by more than one participant: While this is possible – it doesn’t happen when, as is so often the case, students don’t read other students’ postings. It was also suggested by one participant that social interaction is not necessary; rather, interaction with the content is more important. This participant stated that we do not need a teacher – we can learn a great deal through selfinstruction. Although several participants respectfully disagreed with the notion that students do not need social interaction, it was recognized, as another participant stated, that interaction in face-to-face is a myth. . . we imagine a warm environment, talk with people, interact with people, meet people. These are myths about face-to-face learning – but still this myth is attempted online. So, really, we are trying to replicate a myth about an interactive on-campus classroom, which does not exist, in our off-campus classroom. H. Kanuka & J. Kelland / Has e-Learning Delivered on its Promises? 55 There was also considerable deliberation on the topic of interaction and effective pedagogy. Our participants agreed that interactive e-learning is very time consuming. One participant noted further that cost effective e-learning is achieved through economies of scale that results in poorer quality courses, with minimal instructor contact – and interaction. No one disagreed that interaction is time consuming and economies of scale cannot be achieved with highly interactive courses. The topic of interactivity arose several times throughout the deliberation and was clearly a very important aspect of e-learning. But, at present, the participants agreed it is difficult to achieve meaningful interactions under most circumstances with elearning technologies. 3. E-learning Creates Communities of Learners Considerable deliberation also occurred about e-learning communities. It was agreed that it is difficult to create an online community of learners and the notion that e-learning easily creates a community of learners is a myth. As one participant noted, the reality is, Internet communication can be cold and inhuman. . . you know, like we’ve all heard the expression “Crying at the keyboard.” It is the temporal constraints rather than the spatial that are most important for online learners. It was suggested that one reason for the difficulty of creating learning communities is because of the discontiguous nature of asynchronous and textual interaction: Online communication is fragmented, sporadic. How does this affect the students’ ability to learn and their ability to feel part of a community – when their discussions are spread over many days for short periods of time? What is the impact of the fragmented communication that I tend to have when I’m teaching online? For instance I’m teaching an online course right now and this morning I had 10 minutes to check in and do something, post some messages or send an email. I may have another 10 or 20 minutes at lunch and same when I get home tonight. So what impact does that have on the quality of the communication that I can give, and the interaction that I can kind of facilitate? And the students are doing the same kind of thing, if they’re participating in that same kind of fragmented, sporadic way. How does that affect their ability to learn and their ability to feel part of a community when the conversations are spread over many days for short periods of time? We really don’t understand the social dynamics of online learning/communication as well as we think we do. 56 CJHE / RCES Volume 38, No. 1, 2008 One participant, who was a graduate student and had taken a number of e-learning courses, added the following about communities of learning: I can tell all of you that at the end of my online courses I didn’t feel any closer to any of the other students. Overall, little disagreement occurred between the participants about the difficulty of creating a community of learners using e-learning technologies. 4. E-learning Provides a New Kind of Learning Environment Our participants raised an important issue in regard to the myth that e-learning technologies present a new kind of learning environment. Here again, there was agreement that we are still trying to replicate campus-based classrooms. Deliberation between participants centred on the notion that much of what we do with e-learning technologies reflects the old ways of distance learning and oncampus classrooms. One participant summed up this discussion as follows: I see people struggling to replicate conventional classrooms and put it into the electronic classroom by having all these group discussions, which actually don’t really happen face-to-face as everyone’s said. That’s a myth of face-to-face and now an e-learning myth – so, really, just old wine in new bottles. 5. E-learning Provides an Excellent Platform for Discussions Deliberations also evolved about e-learning discussions. It began, like the other topics, on the myths of online discussions: First of all I want to underscore, yeah, I think there are mythologies about classrooms, but I also think that in classrooms, people who normally would not speak to each other, whether the professor is putting effort into it or not, are compelled to speak to each other in a reasonably civilized manner. . . people have to take each other seriously; they can’t just dismiss each other, even if they don’t take each other equally seriously. Although no one disagreed with this point, another participant pointed out that perhaps text-based discussions were not really discussions at all: Are online discussions really discussions? For example, in face-to-face discussions we have to hear, unless we make an effort not to. So, in face-to-face discussions we generally hear most of what is being said, versus in an online environment this is optional. And in face-to-face settings what is said is modified through subtle feedback from those around us. So, I’m not convinced asynchronous, text-based conferencing should be called a discussion. H. Kanuka & J. Kelland / Has e-Learning Delivered on its Promises? 57 Another participant added to this point: Agreed, discussions are more than words. It includes body language, facial expressions. Intonation. In my opinion textual communications are not discussions at all because discussions include both language and paralinguistic cues. It’s just really really fast correspondence – a document delivery system of sorts. Not a discussion at all. . . And if people don’t read or respond to postings, can we still lay claim that a discussion has occurred? In addition to questioning whether text-based communications can be regarded as discussions, another participant questioned the impact of this kind of communication: I wanted to mention that in some ways the word interaction can break down into things like social interaction and learner-content interaction, and that sort of thing. . . it can have a powerful influence on the communication and in some ways I think it’s a form of cyber-colonization or an imposition of a way of understanding. It tends to sort of break things down into discreet exchanges of messages in a kind of cybernetic sense and it takes the influence away from a situation. I think we need to question what it is to be embodied in a particular situation. Hence, it appears we do not have a good understanding of how we are working and communicating with e-learning technologies, and perhaps more importantly, what the impact is on teaching and learning. 6. Students Want E-learning On the topic of e-learning technologies creating a student-centred or learning-centred environment, the discussion tended to be focused on questioning the recurring perception that students want the flexibility provided by e-learning technologies. It was maintained by many of our participants that, in reality, most students want the on campus experience. In fact, both students and their parents have certain expectations of what a university experience should be: Why do my two daughters want to go away to university? Rather than do it by distance ed at home? Because they are on campus. They can interact. They can learn the public transit system. They are challenged. There are a lot of things. More than only content. I want this for them too. Likewise, another participant shared this view: I want to tell a story that kind of moves on the idea of teaching choices that people make. My daughter is now 22. . . and she went into her 58 CJHE / RCES Volume 38, No. 1, 2008 first year of university and came home and told me about her classes and I said, so anybody using the web? And she got quite offended at the idea, she said “No, you know, I’m paying to have someone stand up in front of me”. . . . Her opinion was that she is paying for a certain experience, not an online experience. . . and with all the work I have done on my campus with e-learning over the last ten years I don’t see that attitude changing, at least for undergraduate students. On this topic, participants also ruminated over how we have forgotten that people used to pursue an education as part of an enculturation process, or to become a member of the culture in a certain manner. As one participant commented, the purpose of education was to enter into a culture. . . like learning how to negotiate a discussion. . . become an intellectual of some sort. . . it was about the formation of particular kinds of social subjects within particular cultures and that is not an interactivity element that you can add on to a content base 7. E-learning Provides an Equal and Equitable Environment On the theme of the ability of e-learning technologies to provide an equitable environment, there were a number of diverse perspectives on current myths and misunderstandings. A widely cited benefit of e-learning technologies, for example, is that while relatively few people talk in face-to-face classes, everyone has an opportunity to talk in e-learning classrooms. On this note, one participant stated that some of what we perceive as beneficial in regard to a more equal platform for communication is more myth than reality: While some literature says there’s more opportunity for equal contributions than face-to-face, I don’t think that is necessarily true. . . it is too simplistic to say that there is more equality in participation. It just doesn’t pan out. . . equality is due to the removal of turn taking and time limits. It’s easier for students to participate – or, perhaps, more accurately, to appear to participate. . . online students can, and do, dominate discussions. . . The crude concept of equality is indeed probably adequate in the sense of equality of opportunity but not necessarily of what results out of this opportunity. . . students have different abilities in expressing themselves– which gives them a leg up on learners who are less adept. Another participant noted further that even though textual e-learning technologies remove visual and auditory characteristics, learners, invariably, establish their identity with e-learning technologies anyway: H. Kanuka & J. Kelland / Has e-Learning Delivered on its Promises? 59 Textual markers are being used by learners in social environments to establish identity. Learners also make their presence felt towards others through textual intercourse– lack of postings; amount of postings. . . There are markers in textual forms and there are initiatives invariably made by interlocutors, in textual form that establish identity markers. So in practice, invariably, people do indicate who they are. For example we might not want to self-identify as a person of colour, but as soon as some remark is made we invariably do indicate who we are. But there are also other markers, revealed by an absence of participation, relationships formed, and so forth. There are so many ways in language that we reveal who we are – just in things like the lack of posting, the tone of posting. 8. E-learning Creates Better Quality Learning Environments The topic of whether e-learning technologies are capable of generating more effective learning environments was met with considerable diversity of opinion. In the end, most participants agreed that although textual e-learning communication technologies provide opportunities for added time for critical reflection, most students do not actually use the time to critically reflect on the topic(s) presented. Many participants provided personal anecdotes from their own experiences and examples of research, which reveals that very often the contributions are not at a high level. One participant made a salient contribution to our discussion on effective learning and the need for us to carefully consider the notion of “higher levels of learning”: I’m not sure that in anywhere except institutions driven by so-called accountability procedures, the notion of a higher level of learning actually exists. It’s a kind of fiction that the schools have somehow got us to think is a clear concept; but what’s higher for me at some point might be extremely low for you, it’s situation specific, problem specific, person specific so that is just a really terrible artifact or a terrible institution which has colonized our consciences in very dysfunctional ways. And I want to just refuse the notion of higher levels of learning and again talk substantively about particular learning and particular problems. So, yeah, especially when you’re dealing with something like First Nations’ or Indigenous Peoples’ learning. Which is the higher and lower? DISCUSSION AND CONCLUSIONS The purpose of this study was to establish agreement with experts on the impact of e-learning technology in Canadian higher education. The participants 60 CJHE / RCES Volume 38, No. 1, 2008 took a critical realist perspective and established consensus on beliefs about the positive effects of e-learning technology that are widely held, but continue to be in need of further research. In closing the deliberative inquiry, one participant tried to establish a general consensus with the group about e-learning’s impact: On the Internet and other new technologies, in general, we tend to be either utopic [sic] or apocalyptic. It’s either going to ruin education, or save education. I guess I’d like to throw out the question on how we might measure the success of e-learning. Do we really know if it’s making a positive impact? If it’s being successful? Well could we not say that its success is in its exponential growth? That is, its exponential growth is due to the fact that it’s a good form of learning. Characteristic of the deliberative inquiry process, a reply from another participant to this closing comment was as follows: Respectfully, I’d like to disagree. . . Let’s just think about this as we might about [name of a drug company]. What might they say or do to ascertain their success? Well they might say that 90% of their patients who’ve taken their drug have recovered from chronic renal failure. However, unreported is the 97% who got brain cancer from the drug. . . We can only say there is no significant difference in outcomes if we only look at the apparent production of apparent knowledge outcomes. So, like [drug company], we can produce data that makes us feel good, like the number of students enrolled in e-learning. . . and that could then be a justification for more of this. . . but it wouldn’t indicate that it is good and what the consequences are. It just indicates that a lot of people are prepared to put a lot of money and time into it. This last comment is consistent with Idhe’s (1979) theorem of the nonneutrality of technologies – that technologies amplify and reduce, and there are consequences to our use of them. On this note there was consensus: technologies are non-neutral and have inherent biases as regards to selectivity and intentionality. The participants in this study agreed on (1) a number of pervasive contentions about e-learning technologies in need of further research, and (2) consensus can be reached about e-learning technologies on low impact issues. As one participant stated, “it seems clear that we can only agree on issues of relative insignificance and low impact. So I interpret this to mean that if it’s true– or we all agree it’s true– it is likely trivial.” As such it is important that public policy makers, higher education administrators, leaders and teacherpractitioners who are using e-learning technologies remain skeptical about the truisms presented to about e-learning technologies (e.g., best practices). Specifically, if we can agree on “what works and what doesn’t” it is likely H. Kanuka & J. Kelland / Has e-Learning Delivered on its Promises? 61 about issues that are trivial, and does not reflect the complexity of the real world messiness of teaching with e-learning technologies in higher education. On this topic, Westera (2004) argues that the innovation of education is a complex and toilsome process: “It always involves various parties and many ‘educated’ people, having their own opinions and preferences. If there is agreement at all about the need to innovate, discord about the road to innovation easily arises” (Westera, 2006, p. 502). Researcher Reflections and Limitations of the Study The higher education literature on e-learning technology is replete with research that tinkers with, and then tests the effects of, instrumental practices. The ultimate aim is to determine, once and for all, what works and what does not – passing by the question of why. Upon reflection on the outcomes of this research project, it has become evident to us that the focus on tinkering and testing – or evidence-based practice – with the aim to determine what works, just does not work. The problem is twofold. First, evidence-based practice assumes that the ends of education (or the “why”) are known and agreed upon, and the only relevant research questions are about effective ways of achieving those ends. The problem with this approach is that the ends of education are often not stated, or even known by both researchers and practitioners – and when they are, they are most certainly not agreed upon. The second problem is the assumption that there is a separation between the means and the ends of education. Again, this is simply not the case in education. Moreover, even if we do identify, and agree, on the most effective way of achieving our educational aims and objectives, we still might not choose to act accordingly. As one of our participants noted, while e-learning interventions might result in certain benefits, there can be undesirable consequences which might prevent us from acting accordingly. Hence, although the contentions identified in this study about e-learning are in need of further research, it is important that we keep in mind that e-learning research needs to also account for the potential value of learning opportunities (our aims, assumptions and objectives – or why). When we fail to articulate our desired ends, the result is that our search for direction on what works will fail to provide meaningful information, and agreement, on the means necessary to achieve our desired ends. ACKNOWLEDGMENT We would like to thank the Social Sciences and Humanities Research Council for the funding of this project. 62 CJHE / RCES Volume 38, No. 1, 2008 REFERENCE Abrami, P.C., Bernard, R., Wade, A., Schmid, R., Borokhovski, R. T., Surkes, M., Lowerison, G., Ahany, D., Nicolaidou, I., Newman, S., Wozney, L., & Peretiatkowicz, A. (2006). A review of e-learning in Canada: A rough sketch of the evidence, gaps and promising directions. Canadian Journal of Learning and Technology, 32(3), 1-70. Abrami, P. C., & Bures, E. M. (1996). Computer-supported collaborative learning and distance education. American Journal of Distance Education, 10(2), 37-42. Anderson, T., & Kanuka, H. (2003). E-research. Methods, strategies, and issues. Allyn and Bacon Publishers. Archer, W., Garrison, R. & Anderson, T. (1999). Adopting disruptive technologies in traditional universities: Continuing education as an incubator for innovation. Canadian Journal of University Continuing Education 25(1), 1330. Abrami, P. C., & Bures, E. M. (1996). Computer-supported collaborative learning and distance education. American Journal of Distance Education, 10(2), 37-42. Becker, H., & Geer, B. (1960). Participant observation: The analysis of qualitative field data. In R. N. Adams & J. J. Preiss (Ed.), Human organization research: Field relations and techniques. Homewood, IL: Dorsey Press. Bernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How does distance education compare to classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research,74(3), 379-439. Biggs. J. B. (1982). Student motivation and study strategies in university and college of advanced education populations. Higher Education Research and Development, 1, 33-55. Bogdan, R. C., & Biklen, S. K. (1998). Qualitative research in education (3rd ed.). Boston, MA: Allyn and Bacon. Chandler, D. (1996). Engagement with media: Shaping and being shaped. Retrieved December 13, 2006 from http://users.aber.ac.uk/dgc/determ.html. Dahlberg, L. (2004). Internet research tracings: Towards non-reductionist methodology. Journal of Computer Mediated Communication, 9(3). Retrieved December 23, 2005, from http://jcmc.indiana.edu/vol7/issue1/dahlberg.html. Fontana, A., & Frey, J. H. (2005). The interview: From neutral stance to political involvement. In N. K. Denzin & Y. S. Lincoln (Eds.), The Sage handbook of qualitative research (3rd ed., pp. 695-727). Thousand Oaks, CA: Sage Publications. H. Kanuka & J. Kelland / Has e-Learning Delivered on its Promises? 63 Fuller, T., & Söderlund, S. (2002). Academic practices of virtual learning by interaction. Futures, 34, 745-760. Garrison, D. R., & Anderson, T. (2003). E-Learning in the 21st century: A framework for research and practice. London: Routledge/Falmer. Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7-23. Glaser, B., & Strauss, A. (1967). The discovery of grounded theory. Chicago: Aldine. Glesne, C., & Peshkin, A. (1992). Becoming qualitative researchers. An introduction. NY: Longman. Hawkes, M. (2001). Variables of interest in exploring the reflective outcomes of network-based communication. Journal of Research on Computing in Education, 33(3), 299-315. Heckman, R., & Annabi, H. (2005). A content analytic comparison of learning processes in online and face-to-face case study discussions. Journal of Computer Mediated Communication, 10(2). Retrieved December 13, 2006, from http://jcmc.indiana.edu/vol10/issue2/heckman.html Hiltz, S. R., & Turoff, M. (1978). The network nation: Human communication via computer. Reading, MA: Addison-Wesley. Hiltz, S. R., & Turoff, M. (1985). Structuring computer-mediated communication systems to avoid information overload. Communications of the ACM, 28, 680-689. Hiltz, S. R., Johnson, K., & Turoff, M. (1986). Experiments in group decision making: Communication process and outcome in face-to-face versus computerized conferences. Human Communications Research, 13(2), 225-252. Idhe, D. (1979). Technics and Praxis. London: D. Reil. Jaspers, K. (1931). Die geistige situation der zeit. Berlin: Göschen. Jonassen, D. H. (1996). Computers in the classroom. Mindtools for critical thinking. Englewood Cliffs, NJ: Prentice Hall. Kanuka, H. (2005). An exploration into facilitating higher levels of learning in a text-based Internet learning environment using diverse instructional strategies. Journal of Computer Mediated Communication, 10(3). Retrieved December 13, 2006, from http://jcmc.indiana.edu/vol10/issue3/kanuka.html Kanuka, H., & Rourke, L. (2008). Exploring the Non-Neutrality of e-Learning Technologies. Technology, Pedagogy and Education, 17(1), 5-16. Kanuka, H., & Rourke, L. (2006). Impact of eLearning on Higher Education. Proceedings of the International Technology Based Higher Education and Training (ITHET; IEEE). July, 2006, Sydney, Australia. 64 CJHE / RCES Volume 38, No. 1, 2008 Lapadat, J. C. (2002). Written interaction: A key component in online learning. Journal of Computer Mediated Communication, 7(4). Retrieved December 13, 2006, from http://www.ascusc.org/jcmc/vol7/issue4/lapadat.html Langford, H. W. (1972). Technological forecasting methodologies: A synthesis. New York: American Management Association. Lincoln, Y., & Guba, E. (1985). Naturalistic inquiry. Thousand Oaks, CA: Sage Publications. Linstone, H. A., & Turoff, M. (2002). Introduction. In H. A. Linstone & M. Turoff (Eds.), The Delphi method: Techniques and applications. Information Systems Department at the New Jersey Institute of Technology. Retrieved August 9, 2007, from http://www.is.njit.edu/pubs/delphibook/index.html#toc. McKlin, T., Harmon, S. W., Evans, W., & Jones, M. G. (2001). Cognitive presence in web-based learning: A content analysis of students’ online discussions. American Journal of Distance Education, 15(1), 7-23. McKnight, C. B. (2001). Supporting critical thinking in interactive learning environments. Computers in the Schools, 17(3/4), 17-32. McLuhan, M., & Fiore, Q. (1962). The medium is the message. New York: Bantam. Meyer, K. A. (2003). Face-to-face versus threaded discussions: The role of time and higher-order thinking. Journal of Asynchronous Learning Networks, 7(3), 55-65. Nussbaum, M., Hartley, K., Sinatra, G. M., Reynolds, R. E., & Bendixen, L. D. (2002). Enhancing the quality of on-line discussions. New Orleans, LA: Paper presented at the annual meeting of the American Educational Research Association. Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students (Vol 2): A third decade of research. San Francisco: Jossey-Bass. Patton, M. Q. (1987). How to use qualitative methods in evaluation. London: Sage. Patton, M. Q. (1990). Qualitative evaluation and research method (2nd ed.). London: Sage Publications. Pawan, F., Paulus, T. M., Yalcin, S., & Chang, C-F. (2003). Online learning: Patterns of engagement and interaction among in-service teachers. Language Learning & Technology, 7(3), 119-140. Postman, N. (1993). Technopoly: The surrender of culture to technology. New York: Vintage Books. Ramsden, P. (1992). Learning to teach in higher education. London, UK: Routledge. H. Kanuka & J. Kelland / Has e-Learning Delivered on its Promises? 65 Rourke, L. (2005). Learning through online discussion. Unpublished Ph.D. Dissertation. University of Alberta, Edmonton, Alberta, Canada. Russell, T. (1999). The no significant difference phenomenon. Montgomery, AL: International Distance Learning Certification Center. Retrieved August 14, 2007, from http://www.nosignificantdifference.org Saundercook, J. & Cooper, P. M. (2003). 4th annual technology and student success in higher education. A research study on faculty perceptions of technology and student success. Toronto, ON: McGraw-Hill Ryerson. Stewart, D. W., & Shamdasani, P. N. (1998). Focus group research: Exploration and discovery. In L. Bickman & D. J. Rog (Eds.) Handbook of applied social research methods (pp. 505-526). London: Sage. Tenebaum, G., Naidu, S., Jegede, O., & Austin, J. (2001). Constructivist pedagogy in conventional on-campus and distance learning practice : An exploratory investigation. Learning and Instruction, 11, 87-111. Thomas, M. J. W. (2002). Learning with incoherent structures: The space of online discussion forums. Journal of Computer Assisted Learning, 18, 351366. Vaughan, N., & Garrison, D. R. (2005). Creating cognitive presence in a blended faculty development community. Internet and Higher Education, 8(1), 1-12. Westera, W. (2006) On strategies of educational innovation: Between substitution and transformation. Higher Education, 47, 501-517. Winkelmann, C. L. (1995). Electronic literacy, critical pedagogy, and collaboration: A case for cyborg writing. Computers and the Humanities, 29(6), 431-448. CONTACT INFORMATION Heather Kanuka Athabasca University 1200, 10011-109 Street Edmonton AB T5J 3S8 Email: [email protected] Phone: 1.780.428.7278 Heather Kanuka is Associate Professor in the Department of Educational Policy Studies at the University of Alberta. Her research interests revolve around mediated learning, higher education and faculty development. Jennifer Kelland is a doctoral candidate in the Department of Educational Policy Studies at the University of Alberta. She is currently involved in her thesis research examining how women learning online mediate tensions in their learning environments.