Towards Capturing Complexity: an interactive framework for institutional evaluation
This article discusses the evaluation of the implementation of a virtual learning environment (VLE) at Coventry University, UK. The development and integration of VLEs is demanding significant resources and investment across the sector in the UK. As institution after institution adopts a system for online delivery of their teaching and learning, it is timely to reflect on the process and the possible outcomes of these types of adoption. The availability of a VLE affects many aspects of University life and practices. At Coventry, we are endeavouring to understand the impact of the changes on staff and students in terms of motivation, achievement and satisfaction as well as on the curriculum and modes of delivery. Through the presentation of our evaluation model, the article offers a framework for capturing the complexity of a major development such as this. Examples of the findings from the first two years are used to illustrate the nature of the Coventry evaluation model.
The evaluation strategy
As the title of this paper suggests, we are concerned with complexity. Our research is both multi-disciplinary and multi-method and, similar to the large scale evaluation study carried out by Anderson et al (2000) adopts an "eclectic approach to the selection and combination of research methods". The evaluation research from which our evidence is drawn, is carried out by students, lecturers with educational researchers.
From the outset, there have been three distinct layers to our evaluation of the implementation of a VLE, each with a discrete audience for the results. Firstly, the evaluation is being used to guide the further development of the technology and the associated staff and educational development. This serves the needs of the VLE development team and the users (both teachers and learners.) Secondly, it is informing decision-making around policy and practice in relation to teaching and learning and technology. The primary audience for this layer of evaluation is management and administration. Whilst the infrastructure for online learning continues to be refined and modified, the findings have also been able to influence decisions made at various levels of University administration, both central and devolved. Thus the evaluation can be perceived as a cycle of action-reflection-action-reflection (Deepwell & Cousin 2002), with interim results feeding into relevant forums across the institution. Thirdly, it is a vehicle for us to disseminate our experiences to a wider academic community in this field and thereby offer a contribution to theory-building in the area of technology in teaching and learning. The maintenance of a web site indicating the evaluation findings and approaches, together with conference presentations and articles such as this, serves this third aim.
In 1999, Coventry University launched a virtual learning environment comprising WebCT plus additional web-based facilities (a module information directory, previous exam papers, and study skills materials) to enhance teaching and learning at the University. An automated procedure generated a web presence for all modules based on a common template. Student registration on modules was updated on a nightly basis using an automated routine. Use of the VLE was encouraged and centrally supported, but not compulsory.
With such a large-scale move into a new area of teaching and learning, we needed to ensure that as the implementation was rolled out we were informed of the impact and consequences of the change, including unintended phenomena. The findings needed to be delivered in as timely and insightful a way as possible, representing the many voices touched by the development. This information would enable us to review factors under our control in order to improve matters and provide evidence to others of the need for change elsewhere in the institution. Our intention in conducting an evaluation, therefore, was to inform developments as they progressed and has become more of a process (ongoing reflection and feedforward) than a product.. For these reasons, we felt outsider evaluation would not yield as rich a return as an in-house process. We are seeking a deeper understanding of the changes in the teaching and learning environment both "prospectively" and "retrospectively", therefore our research is evaluation for development and for knowledge rather than evaluation for accountability (Chelimsky, 1997).
In considering the nature of our evaluation, we can position it in regard to the three measures outlined by Oliver and Conole (1998): authenticity, exploration and scale. The evaluation study remains highly "authentic" in that it attempts to capture the effects of the VLE as it operates in real time. The study is "exploratory" in that it observes, describes and interprets events. Finally, it is "large scale"; it compiles the big picture of what is happening, illustrated by university-wide phenomena as well as smaller in-depth studies.
A number of theoretical models have been considered along the way. The introduction of an institution-wide online learning environment can variously be interpreted as institutional change, curriculum development, technology application, cultural shift, learning opportunities, widening participation measure or even product evaluation; each interpretation serving a discrete audience, or group of stakeholders (Pawson and Tilley, 2000). The initial understanding of the nature of the intervention significantly impacts on the evaluation strategy and model adopted. In the case of Coventry University, the introduction of a VLE was clearly directed by senior management. The intentions for such a move were a fundamental component of the University's teaching and learning strategy. Concurrently there was an expectation that the universal implementation of such a VLE would be designed to serve the needs of the teaching and learning processes and new developments at both a macro and a micro level. At the macro level, the institutional procedures and support mechanisms would embrace the VLE. At the micro level individual learners and teachers would be able to integrate their use of the VLE with existing and evolving teaching and learning methods. A lynchpin of our evaluation therefore are the questions on how these intentions, both implicit and explicit, have been met.
In our evaluation we have drawn on the strengths of illuminative evaluation and have sought to study the innovation within the "learning milieu" (Parlett and Hamilton 1977). Its methods, too, are akin to our preferred methods of data collection, namely observation, interviews, questionnaires, document analysis and background information. We were concerned however not to adopt the outsider stance associated with illuminative evaluation. We felt that an illuminative approach might produce a one-off report with recommendations for actions around phenomena but would not support the evolution of the innovation across time. Such an approach was therefore seen as not sufficiently productive for our needs and we were drawn to Robert Stake's "countenance" model (Stake 1967).
Countenance model of evaluation
The "countenance" model of evaluation seemed more appropriate because its suggested matrices for descriptive and judgmental data are able to support the study of an evolving programme across time, looking at the antecedents as well as the intended and unintended consequences of the programme. Robert Stake's "countenance model" (Stake, 1967) was originally formulated for curriculum studies in the late 1960s. The countenance model aims to capture the complexity of an educational innovation or change by comparing intended and observed outcomes at varying levels of operation. The congruence between the intentional and the observational accounts provides the basis for judging the success or otherwise of the innovation, whilst at the same time allowing for the recording of unintended outcomes. A summary model of Stake's data matrix is shown in Figure 1
Our evaluation however deviates from the model in two significant aspects. Firstly in that it is conducted by internal agents who are active and reflective participants in the development. Secondly, it is an inextricable part of the development cycle of the implementation of a VLE in higher education. Therefore, our approach also lies close to the emerging notions of "action evaluation" (Rothman and Friedman) and the collaborative research findings continually feed into the further evolution of the online learning developments. In this action evaluation model, intended outcomes are compared to observed outcomes to reveal the areas for further action and revision of intentions. It is flexible and cyclical in allowing the findings to change the direction of enquiry. This is particularly relevant in the current case, where we were introducing an innovation that has enabled new departures in practice.
Action evaluation is a form of evaluation that has its roots in conflict resolution. In many regards, this is a valid metaphor for our research and evaluation into processes of change at Coventry. Because of the large number of stakeholders and range of their interests, there are inevitable conflicts which are in need of resolution in our higher education context. Action evaluation, like action research, is iterative. It also attempts to give voice to all stakeholders in the spirit of collaborative inquiry. In our case, this involves gathering disparate and uneven data from the widest variety of sources.
Stake's countenance framework, combined with the stakeholder participation inherent in action evaluation, is helping us achieve a long range view of the development. A view that expresses multiple voices and permits a range of data sources and enables us to observe and contrast the actual findings with stated intentions in an iterative and dynamic cycle.
One further aspect of our evaluation is the medium through which we present the results. For such a multi-layered and complex analysis of the intervention, we have deployed the online medium itself. The hypertext environment of the web has enabled a three-dimensional approach to capturing and presenting the analysis and some of the underlying evidence. This inherently dynamic environment also enables the evaluation to grow in depth over time, making it an organic, possibly interactive, entity rather than a static report.
Online Learning at Coventry University: the Evaluation
Since 1999, students at the University have had access to an online environment for each of their taught modules. Our approach to implementing the online facility across the institution has been outlined previously (Deepwell and Syson, 1998). Briefly, whilst the implementation of the online learning environment using WebCT is centrally designed, it has been rolled out in such a way as to provide academic colleagues with as much scope as possible in determining the extent to which, how and whether they adopt the technology in their teaching. Administrative procedures and generic support for students' learning is managed centrally, for example: each module has a link to the module description and regulatory information held centrally. Furthermore, the student class lists are uploaded directly from the student information system and updated nightly. "Ownership" of the online modules rests with the lecturers involved in delivering those modules. Access to the modules is through the University homepage and is simplified for students. A recent upgrade has enabled us to simplify this entry route for staff, too.
A comprehensive evaluation of such a dispersed and flexible implementation is far from straightforward. Further complexity comes from it being a moving target - the evaluation process is under way whilst the "intervention" is taking shape. Software upgrades, changes in legislation and University policy together with the upsurge in Internet usage amongst students and staff all affect the outcomes. Concurrently, attitudes towards the use of computers in higher education are shifting, particularly in non-technical disciplines, as availability of the technology becomes more widespread and awareness of its usage grows across all sectors of the economy.
The methods of data collection we used combine both quantitative and qualitative data. Indeed the evaluation framework is sufficiently flexible to integrate quite diverse data into coherent sets according to a structured framework (see Figure 1). Evaluation data is derived from a wide range of sources, including:
The server statistics were collected in order to give a broad-brush impression of comparative usage across the University. They are not sufficiently detailed to provide a clear map of usage. Due regard is also paid to the limitations of the software as regards data capture. There is no audit trail of designer actions in WebCT, although there is student tracking. With an earlier version of WebCT, the designer account was a shared account and it was not evident whether one person or more had been actively involved in WebCT developments. With the current version we are able to gauge how many people have designer access, but still unable easily to ascertain frequency, duration or nature of their interactions.
Summary of interim research findings and their implications
The findings presented here are illustrative of the evaluation approach and represent a discussion of the congruence between the intended and the observed during the transaction phase and a statement of the actions taken or planned as a consequence. In order to aid the reader, I have given each section a heading with a general theme that encapsulates the issue presented and then listed three subsections: intended transactions, observed transactions and actions. The findings mostly relate to the period between September 1999 and July 2001 and where relevant I have noted indicative sources of the data. Fuller presentation of the findings is available on the Online Learning Research website at http://www.coventry.ac.uk/ched.
Intended transactions: over 10% active use. The initial usage target was set at senior management level to be 10% uptake. From a preliminary analysis of the server statistics, this was seen as an attainable target and therefore a "safe" measure of success in VLE uptake. In the pilot year, where early adopters were able to use the VLE but did not have administrative support, we were already observing usage levels of around 10%.
Observed transactions: 23% use. An analysis of the server statistics showed that around 23% of modules were using WebCT to some extent throughout the second year of its full implementation. It was observed, however, that the percentage distribution of usage varies within Schools between over 50% to under 10% usage. Each School is a separate administrative unit and has a great deal of autonomy over its expenditure. Significantly higher usage is made of the VLE in two of the Schools: Business (CBS) and Computing (MIS - mathematical and information science). This is evidenced by the server statistics and corroborated by quality monitoring and other local reports from the Schools. In the case of the Business School, additional support was obtained to convert teaching materials for the VLE.
Action. Quantitative evidence needs to be supplemented with background knowledge and qualitative research (e.g. case study, student interview) in order to build a larger picture of events and triangulated in order to reconcile discrepancies in findings. The server statistics are used as a baseline snapshot of activity and welcomed by University committees who can then chart the rise or fall of use and wield the data as an argument for more investment or justification of resource expenditure.
Intended transactions: full and equitable access. Access to the VLE was linked to University, and modular, registration. In this way, registered students should have access to all the modules for their current programme of study. Students would have access to all module spaces, irrespective of whether lecturers were using the VLE. The template was designed in particular to promote the communication tools.
Observed transactions: partial access. In the first year of operation, two principal factors prevented full access. Firstly, in order to get a username, students had to be on campus at least twice in person in order to register and this proved to be a barrier to access for all students studying University courses at partner colleges or in the workplace. Secondly, a few departments in the University were not yet using the student record system for registration and enrolment, thus debarring their students from automatic registration onto the VLE. Case studies, participant observations and email reports in particular have provided evidence that student groups took advantage of the VLE as a peer communication vehicle.
Action. Both of these barriers to access were overcome by the second year of operation through changes to the username registration process and developments in the student record system. Access problems remain a prominent feature of complaints by students (data derived from error report logs, interviews and email messages to a shared list), caused either by forgetting passwords, or incorrect details stored on the student record system. Online module registration has emerged as a necessary development of the system to improve access still further.
Ease of adoption
Intended transactions: adoption to be made as easy as possible. Great efforts were made to simplify the entry routes both for staff and students into the VLE. Direct access from the University homepage, module registrations feeding into the student lists within each module space in the VLE, template containing the essential tools with a common look and feel for each School of the University, local points of call for password and technical assistance.
Observed transactions: wide variation in adoption patterns. It is observably easier for some than for others to adopt the VLE. There has been noticeable dissatisfaction expressed by, for instance, some computer scientists who want a more robust and adaptable system, frustration by others who complain (via email messages to a group list, memoranda, questionnaire feedback and in interviews, for example) of the unreliability of the systems generally (e.g. obtaining passwords, student registration, network speed). Local technical support is insufficient to address the tutor needs. There is evidence that the poor interface design and known bugs in the version of the VLE have discouraged adoption.
There are a number of tools available in the VLE, which broadly fall into two camps: communication and presentation. Usage varies widely across the University. Server statistics and questionnaire feedback shows that the Business School has a relatively low usage of communication tools, whereas the School of Health and Social Science exhibits far stronger use of these tools than the presentation ones. There is also variation at the micro-level, for example some of the most interesting case studies of use of the communication tools occur in the School of Art and Design, although this is one of the schools which uses the VLE the least overall.
Action. In the third year of operation the University has upgraded to a version of WebCT with a substantially improved user interface and, although there are still some valid usability issues, feedback from colleagues suggests that it no longer appears to be a disincentive to use. Further action needs to be taken to encourage adoption, for instance, by recommending third party software to create questions for online assessments.
Training and support
Intended transactions: the provision of adequate training and support, both technical and pedagogical. Local support was deemed an essential part of the rolling out of the VLE. To this end, there was a 10-week familiarisation programme for a group of technical support staff who assessed themselves against a competence list of tasks associated with supporting the VLE, including skills in general software used by colleagues. In each School there were also academic colleagues in the Teaching, Learning and Assessment Task Force charged as local educational developers and action researchers to support the adoption process (Beaty and Cousin, forthcoming). Additionally, a team of three people in the Centre for Higher Education Development took on the role of a central support team for WebCT. The computing services general helpdesk supported student users.
Observed transactions: variable support. Studies of the effectiveness of the VLE support shows that the technical experts were rarely able to perform in this role because they had not been given remission from other tasks. In-depth interviews revealed that, in most cases, they did not promote their support role and soon lost the knowledge they had acquired during the training programme through disuse. In a couple of instances, extra, short-term appointments were made to produce teaching materials in a format suitable for the VLE. The temporary appointments were effective for the jobs they were appointed to do, although there is little evidence that they have encouraged adoption beyond the close circle of colleagues who were able to draw on their services. Local academic support on the other hand has developed more successfully, as shown in feedback questionnaires and case studies, for instance. Colleagues have been able to draw on the experience and expertise of members of our teaching, learning and assessment Taskforce, some 25 or so academic colleagues who have been seconded by the University to develop and disseminate innovations in their practice. In addition, other support networks have emerged independently (e.g. buddy systems). The team in CHED has continued to provide central support through online information and discussions, training sessions and a busy telephone and email hotline.
Action. The short-term appointments that were made were "pump priming" the implementation and have now been terminated. Empowering colleagues to work directly with the VLE is the main thrust of the support provided both centrally and locally. The staff development model is responding to the nature of support required and is adopting more of a consultancy approach, with consultations tailored to individual teaching and learning requirements.
Student engagement and staff effort
Intended transactions: online learning to be motivational for student engagement. During the pilot year, analysis of some student groups showed a marked improvement in student motivation and performance (including a 5% rise in end of year results in one module) however these studies were conducted only by those typified as early adopters (Daniel 1996). The intention of the full implementation was to bring these motivational advantages of technology to enhance teaching and learning to the majority of courses.
Observed transactions: required individual effort, time investment and perseverance on part of staff. Case study and interview evidence has shown that effective usage still requires additional individual effort of a kind expected from early adopters. Technical hitches continue to occur, e.g. slow delivery of network applications at peak periods, hardware failures and other blockages. Notwithstanding this, a far wider user group has now become involved in using the VLE. Members of staff who use it even for quite modest enhancements for their face-to-face delivery have recommended the use of the VLE to their colleagues precisely because they feel it does motivate their students. Our student-generated data suggest that engagement with the online medium is particularly beneficial for part-time students, or those with additional learning needs.
Action. Of special interest are those colleagues who have become engaged with the online technology primarily for teaching reasons, despite expressed inexperience in C&IT and expressed scepticism. Further analysis is required to ascertain levels of student engagement.
Intended transactions: anytime, anyplace learning (24/7). It was intended that student learning would be enhanced through wider availability of the learning resources. Through the VLE template all students had links to the central web-based learning resources of the University, namely the library, study skills and to centrally held module information. Module tutors would build the web environment organically, adding learning materials and other information gradually over the academic year.
Observed transactions: anytime, anyplace module resources. There are a growing number of modules using the VLE for specific learning activities. However, in most cases, the VLE has been used as a repository for documents (among other things). Learning resources have been added to the VLE, often in the form of module handbooks, lecture notes or summaries. In a few subject areas, the VLE has become a central rather than a supplemental resource for students. Many colleagues report student pressure to build up the online learning elements of their modules, including student/student and student/tutor communication. These reports are backed up by student survey data.
Action. Further localised staff development is required, including sharing of good practice to enable those colleagues who make comments such as "next year I'll use it for more than just my lecture slides" are able to put their intentions into practice.
Along with an analysis of intentions and how they have been met, the Coventry model of evaluation allows for unintended outcomes emerging from the intervention. In our case, one of the more unexpected outcomes of the implementation of the VLE was the emergence of a trust culture amongst academic colleagues from different disciplines. When the notions of a VLE were first proposed, it was presented as a virtual classroom - the lecturer could control who was allowed to observe their teaching or their students' learning. The fears lecturers expressed at the time (in focus groups and dedicated seminars) were that the VLE would become a means by which they would themselves be controlled and monitored by administrators or managers. More recent evidence shows however that those lecturers who have achieved integration of the VLE into their teaching practice actively want others to observe what they are doing and often set up guest accounts for their colleagues locally and their peers in other institutions.
Other unintended outcomes include re-engineering the student registration process and new staff development opportunities for technicians. The evaluation is also highlighting further areas for action, such as the conditions of contract for academic and academic-related colleagues, wider availability of Internet connections in general teaching rooms etc.
In this paper, I have attempted to offer a framework drawing on Stake and action evaluation for capturing the complexity of online learning across an institution. As the evaluation at Coventry University progresses the stakeholder perspectives are shifting and the evaluation framework we are using accommodates this shift as I have shown above. Through the availability of a VLE there is increasing expertise in the media of online learning and a general growth in familiarity with all forms of online practice. The University is moving towards a natural plateau of usage in mixed mode delivery.
We are now in the fourth year of a five-year cycle of evaluation. An interim report was presented on years one and two and a second report is being compiled for years three and four. The question "does it make a difference?" has been answered through evidence e.g. of joined-up thinking at senior management level, the growth in online facilities, staff engagement, increase in student Internet usage and IT competence levels etc). The more elusive question "does it improve student learning?" is still open - however, the signs are promising.
Copyright by the International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the authors of the articles you wish to copy or email@example.com.