Evaluating information and communication technologies for learning
Eileen Scanlon, Ann Jones, Jane Barnard, Julie Thompson
and Judith Calder
It is part of the new orthodoxy in higher and further education that the effectiveness of teaching needs to be assessed, and many institutions now have policies for the conduct of evaluation of courses, thus creating a link between evaluation and quality assurance.
Over the past thirty years a number of evaluation methodologies have been developed and applied which we review here in order to explain the background of the evaluation framework we have developed.
One way to examine the issues surrounding evaluation is to consider a range of polarised dimensions. A quantitative approach can seem attractive focusing, as it does on those aspects of an evaluation that can be measured and expressed in terms of numbers or percentages. However, there is the danger that those aspects that are difficult to quantify, such as students’ enjoyment or orientation to study, can be neglected. The formative and summative distinction (first made by Scriven and discussed by Kandaswamy, 1990) focussed attention on the purpose of the evaluation: for example, whether the evaluation is for improvement or to assess the success of a project. Attention to the illumination/controlled distinction encourages evaluators to consider non-traditional source of information, rather than simply considering the inputs and outputs of a situation. Parlett and Hamilton (1972) developed the illuminative approach and concentrated on developing an understanding of the processes that students go through while studying. Stufflebeam (1971) also considered process and measures of intended outcomes as well as a consideration of the context in which a course is used.
Another way of distinguishing between models of evaluation is to distinguish between an objective or testing school of evaluation typified by that of Tyler (1986) and alternative schools associated with anthropological models (Parlett and Hamilton, 1972) and connoisseurship (Eisner, 1985). A relatively recent distinction is that between context-based and decontextualised evaluation which arises in part from the situated cognition paradigm (Brown et al., 1989).
Evaluation of information and learning technologies (ILT)
An interpretation of the illuminative/controlled evaluation dimension is the distinction between ideographic and nomothetic approaches, first made by Kemmis (1977). Ideographic approaches involve very open exploration, whereas nomothetic approaches are usually strongly rule based and involve commitment to a particular set of values. In a nomothetic approach, systems are measured against these values. Kemmis was one of the main developers (with MacDonald) of the Understanding Computer Assisted Learning (UnCAL) evaluation based at the University of East Anglia, which produced highly influential evaluations of the National Development Program for computer assisted learning (NDPCAL) in the UK in the 1970’s. The successor to this program in the 1990’s, the Teaching and Learning Technologies Program (TLTP, 1996), also encouraged a number of evaluation approaches, notably the integrative evaluation approach adopted by Draper et al. This approach is described as by Oliver (1999, pp12-13 ) as an approach that
Information and learning technologies have been put at the centre of recent government initiatives to expand the post compulsory sector of education by increasing student numbers or by an increase of training in basic skills training (e.g. Her Majesty’s Stationary Office, 1998). This push towards an increased use of new technology has led to a situation in which there is a much greater interest in and discussion of evaluations of educational technology. The NDPCAL (Hooper, 1977) programme of the ‘70s stressed the importance of rigorously evaluating any learning approach, an opinion echoed in the Mc Farlane report of the early ‘90s. However, Higginson (1996) took a slightly different approach. He reported on the work of Learning and Technology committee set up by the Further Education Funding Council (FEFC), investigating how FE colleges were using technology and how beneficial the use of technology was for learners. Higginson recommended the setting up of a national staff development programme, specialist advice centres and demonstration projects to show the effective incorporation of learning technologies, the Quality in Information and Leaning Technology programme (QUILT) and an advisory committee (jointly with the British Educational Communications and Technology agency, BECTa). Here the stress was more on promoting effective use, but the consequences have included an increased interest in evaluation promoted by the Further Education Development Agency. This is the context for the case study described below, which includes an interest in supporting change and in establishing good practice so that the potential of ILT can be realised. Therefore there is a wide range of different uses of ILT to be examined and a breadth of activities to be explored for the evaluator.
The Context, Interactions and Outcomes (CIAO!) framework
There is a long history of the use of computers in teaching at the Open University (see for example, Butcher and Greenberg, 1991). This history has informed the development of a model for evaluation that is laid out in Jones et al. (1996) and reproduced in Table 1 below.
Table 1. The model for evaluation
A framework for the conduct of evaluations, CIAO!, focusses on three aspects: context, interactions and outcome. Context refers to a wide interpretation of the rationale for use of the software including the aims of the use. This includes considerations of questions such as how the CAL fits within the course, where is it used (at home/in the classroom), who uses it and whether it is used alone or in groups. Interactions refers to a consideration of ways of examining and documenting how students interact with computers and each other, focussing on the learning process. Often, in recording such interactions, inferences can be drawn about the learning process. An outcome refers to a wide interpretation of the changes in students, which result as a consequence of using the program. Restricting consideration of outcomes to cognitive ones is too limited as often other consequences emerge which need exploration. This means that learning outcomes still need to be considered but so also do changes in learners’ perceptions and attitudes. One aspect of measuring outcomes is the fact that it is very difficult to attribute learning outcomes to a particular use of ILT or to a particular use of the computer, when that is only one part of a multi-faceted experience of a course.
Looking vertically down the framework, the data row highlights the different types of data that needs to be collected to inform the work, and the methods row specifies how this is done. Many different types of methods have been used: interviews with staff to inform an understanding of context, questionnaires sent out prior to the start of the course, post course questionnaires and logs of computer use and terminal time. The framework reflects its origin in the evaluation of distance learning courses where questionnaires are often used to collect information from large numbers of students, while small-scale studies can supplement this picture with observations and interviews.
Experience from work at the Open University
The framework evolved from the many years of working with course teams at the Open University on a range of evaluation projects over the past twenty years (see e.g. Butcher and Greenberg, 1991). However, the details of the framework emerged from a focussed project where we applied our experience to a set of programs for using ILT in science and technology projects. The projects ranged from summative testing of a tutorial CAL program for a technology foundation course, formative testing of a simulation of chaos for a higher level physics course and formative testing of a multimedia CD ROM for use in a foundation science course and the summative testing of conferencing applications on a science studies course. All our work had involved a consideration of context as a starting point for the design of an evaluation working alongside software developers and instructional designers. Our experience of working on formative evaluations had taught us the value of observations.
One evaluation project on a program to help with understanding phase diagrams illustrated this (see Tosunoglu et al., 1996; Scanlon et al., 1996). We established relatively quickly, using questionnaire returns, that students found a particular program enjoyable and its game format motivational. We used pre- and post tests to compare students' learning before and after the use of the program in an attempt to establish how the program contributed to students’ learning. It was difficult to show that any observed differences between pre- and post-test scores immediately after use represented what the learning gains might be, as the package was still available for student use at any time throughout the year which meant that further tracking was necessary. In any case it was difficult to establish that shifts in learning were due to the use of the computer-based teaching alone. This package was only one part of a course that consisted of a variety of teaching resources and help from tutors and feedback from written and computer-based assessment packages. In this case, we nevertheless established a considerable amount about how students learn from the program using interviews and observations. During observation sessions students were observed using the software and semi-structured interviews were conducted.
Observations and interviews also featured significantly in a formative evaluation project where a version of multimedia software was piloted (see Taylor et al., 1995, 1996). The aim was to investigate if the format of a multimedia CD-ROM would serve the aim of presenting a course in an interactive environment in which the student has more control than in traditional text presentations.
These examples illustrate some of the parts of the framework we introduced earlier illustrating rationale and the selection of methods we used. To completely outline examples which illustrated the application of the framework in all its aspects would take more space that we have in this paper, so for the sake of brevity we have encapsulated here the results of a range of such studies in the following principles.
Principles for conducting evaluations
This section summaries a set of working principles for the conduct of evaluations which have been developed from the CIAO framework and our Open University experience. It can be seen by our earlier discussion that we value conducting evaluations of ILT in context. However we must note that this leads to certain constraints on the conduct of evaluations. A real world teaching and learning situation cannot be manipulated in the way that laboratory experiments can which makes it difficult to establish cause and effect in teaching situations. In attempting to establish the crucial variables in a particular successful use of ILT, it is tempting to set up the equivalent of a controlled experiment such as a direct comparison of groups of ILT users and non ILT users. We would argue that such an approach is seldom the appropriate response. The conduct of evaluations under a set of working constraints, however, provides us with a very clear idea of who in a particular evaluation is the audience for the findings. In all our experience and in the literature, it appears that establishing the aims or rationale of a particular ILT application is the key to designing an evaluation. Evaluation should involve learners and take account of context. We also feel it important to stress that in conducting an evaluation there is a need to note unanticipated issues which arise and would question evaluations where no surprise findings emerged.
Case study: Evaluating Learning with IT in Further Education
In this case study we describe our participation in one of the phases of a three phase study of the evaluation of learning with IT in the Further education sector, conducted between September ‘98 to July ‘99. This work involved the refinement and development of a tool to be used for the purpose of evaluation and the application of the tool and associated methodology to six case studies in colleges to allow (their) further refinement. A number of colleges were involved with projects where ILT was deployed. These projects included an on-line careers advice project, a pilot for resource based delivery, courses on educational use of IT for teachers, the use of an authoring package to put materials on-line, a multimedia key skill project for disaffected 14-16 year olds, and staff development in ILT.
This situation had a number of challenges; not least to apply the framework and principles described above to this particular evaluation situation. Some of these constraints are worthy of comment. The users of the tool were to be FE practitioners in a pressured environment, and who, therefore, as a result needed ‘off the shelf’ tools which did not require any particular expertise and would be easy and quick to use. The wide range of projects, mentioned above each had their particular aims as well as different types of students varying in age and mode of study. These constraints led to the decision to use questionnaires. These have the advantage that they are familiar, do not need a high level of expertise and are quick to use. They could be used on their own or combined with other tools which would allow the triangulation of information. They also allow a way to gather information from each major stakeholder involved in any specific project whether student, lecturer or manager. It is also possible to produce a template for questionnaires so that they can be used generically but can also be changed to be sensitive to context.
As the aims of each ILT innovation were different, the approach adopted focussed on the development of a generic evaluation tool that was customisable. It was important to focus on the rationale, ensuring that in the tool there were opportunities to report on different perspectives to the development of the particular ILT application. We hoped that the opportunity to customise the tool would cope with the range of projects but would also allow a sense of ownership of the evaluation project to develop among practitioners. We were able to assess how this worked in practice, as we worked on the development consultation and trialling phase of the tool. We found that one issue for all the participants was difficulty in finding time for evaluation and that the three separate strands of the evaluation process needed more integration. An analysis template was therefore developed so that the three questionnaires (learner, manager, and lecturer) could be compared and critical areas could be highlighted and cross-referenced.
In the final phase the participant groups (colleges) used the modified tool to conduct evaluations of their ILT provision and we made final visits to college to collect data and produced a report and guidance manual for use of the tool based on experience with the tool.
There is a great challenge in using a generic questionnaire to probe into specific situations. Some problems were experienced in formulating questions on learning outcomes due to the very variable aims of the colleges’ projects. Some of the feedback suggested that more questions should be included that related specifically to student learning. This was responded to the last phase of the project by introducing a question that needed to be customised by the colleges but this in some cases did not happen. The final version of the manual therefore outlined the importance of customisation and ensuring that the questionnaire reflected the specific aims of the project it was being used to evaluate.
In this case study we confirmed the difficulties of using a generic tool when customisation does not take place in the way we expected. However, the overall message was that the questionnaire template was robust as it stood, and the analysis template that had been developed elicited very positive comments. The use of a questionnaire in this context had a number of advantages, in terms of scale and applicability. This is not to deny our experience described above in the value of observation and interview approaches as a follow up.
Our findings on this project were of two types: about the use of generic evaluation tools and about user involvement in evaluations. We found overall that to encourage of the use of a generic evaluation tool, support for customisation and support for the tool is crucial. A manual for the use of the tool is vital. It is necessary for the evaluation process to be customised and for staff at different levels of involvement with the project (such as the manager) to have strong awareness of the aims and objectives of the project. Some of the staff who were involved in the evaluation work had not been part of the original ILT projects development team and were therefore not aware of the original project aims. We decided that increasing support through the analysis template would be necessary to increase the likelihood of effective use of the tool. The project confirmed that information from different sources (different stakeholders in our project) was extremely valuable in cross-referencing or triangulation which Breen et al. (1998) refer to as “the use of a combination of research methods”. These have the aim of “where various methods leave gaps in understanding or room for misinterpretation, the other methods would provide answers and improve the interpretability of each set of results” (ibid, p 153).
We noted above the trend towards involving practitioners with evaluation and much of this case study was aimed at, primarily, teachers evaluating their use of ILT and our approach was to involve users as part of the development cycle. We found a useful rationale and additional motivation for this involvement by helping users to see how it can lead to useful information that could be included in the self inspection process which has a role the quality assurance processes in the sector.
The aim of the CIAO! framework which informs our work is to encourage the consideration of a variety of methods, rather than offering a prescriptive approach to evaluation. This approach has served us well in influencing work in our own institution, and we have illustrated here how this approach informed our work in a further education context. We have illustrated here that context is crucial and when developing any evaluation tool evaluators must find a way of taking it into account. Given our earlier comments about expansion in the sector with the consequent impact on staff, it is also important to find a motivation or ‘re-use’ for the results of an evaluation process.