Educational Technology & Society 3(4) 2000
ISSN 1436-4522

Evaluating information and communication technologies for learning

Eileen Scanlon, Ann Jones, Jane Barnard, Julie Thompson and Judith Calder
Institute for Educational Technology
The Open University, Milton Keynes MK7 6AA
United Kingdom
e.scanlon@open.ac.uk
Tel: +44 1908 274066

 

ABSTRACT

In this paper we will describe an approach to evaluating learning technology which we have developed over the last twenty-five years, outline its theoretical background and compare it with other evaluation frameworks. This has given us a set of working principles from evaluations we have conducted at the Open University and from the literature, which we apply to the conduct of evaluations. These working practices are summarised in the context interactions and outcomes (CIAO!) model. We describe here how we applied these principles, working practices and models to an evaluation project conducted in Further Education. We conclude by discussing the implications of these experiences for the future conduct of evaluations.

Keywords: Evaluation framework, Further and higher education, Distance education


Introduction

It is part of the new orthodoxy in higher and further education that the effectiveness of teaching needs to be assessed, and many institutions now have policies for the conduct of evaluation of courses, thus creating a link between evaluation and quality assurance.

Over the past thirty years a number of evaluation methodologies have been developed and applied which we review here in order to explain the background of the evaluation framework we have developed.

 

Evaluation approaches

One way to examine the issues surrounding evaluation is to consider a range of polarised dimensions. A quantitative approach can seem attractive focusing, as it does on those aspects of an evaluation that can be measured and expressed in terms of numbers or percentages. However, there is the danger that those aspects that are difficult to quantify, such as students’ enjoyment or orientation to study, can be neglected. The formative and summative distinction (first made by Scriven and discussed by Kandaswamy, 1990) focussed attention on the purpose of the evaluation: for example, whether the evaluation is for improvement or to assess the success of a project. Attention to the illumination/controlled distinction encourages evaluators to consider non-traditional source of information, rather than simply considering the inputs and outputs of a situation. Parlett and Hamilton (1972) developed the illuminative approach and concentrated on developing an understanding of the processes that students go through while studying. Stufflebeam (1971) also considered process and measures of intended outcomes as well as a consideration of the context in which a course is used.

Another way of distinguishing between models of evaluation is to distinguish between an objective or testing school of evaluation typified by that of Tyler (1986) and alternative schools associated with anthropological models (Parlett and Hamilton, 1972) and connoisseurship (Eisner, 1985). A relatively recent distinction is that between context-based and decontextualised evaluation which arises in part from the situated cognition paradigm (Brown et al., 1989).

 

Evaluation of information and learning technologies (ILT)

An interpretation of the illuminative/controlled evaluation dimension is the distinction between ideographic and nomothetic approaches, first made by Kemmis (1977). Ideographic approaches involve very open exploration, whereas nomothetic approaches are usually strongly rule based and involve commitment to a particular set of values. In a nomothetic approach, systems are measured against these values. Kemmis was one of the main developers  (with MacDonald) of the Understanding Computer Assisted Learning (UnCAL) evaluation based at the University of East Anglia, which produced highly influential evaluations of the National Development Program for computer assisted learning (NDPCAL) in the UK in the 1970’s. The successor to this program in the 1990’s, the Teaching and Learning Technologies Program (TLTP, 1996), also encouraged a number of evaluation approaches, notably the integrative evaluation approach adopted by Draper et al. This approach is described as by Oliver (1999, pp12-13 ) as an approach that

aims to evaluate the course as a whole, rather than simply the resource being used, and aims to improve learning by integrating educational technology as effectively as possible into the learning environment and (…) conducting qualitative analyses are complemented by measuring factors with a significant impact on learning. This enables a wider range of explanations to be offered, and allows some results to be generalised (Draper et al., 1994).  Adopting an illuminative approach all but precludes comparative or experimental research designs. TILT's framework is predicated on the assumption that the evaluation must take place in context, since it aims to evaluate the course's use of educational technology, not the educational technology alone.

Information and learning technologies have been put at the centre of recent government initiatives to expand the post compulsory sector of education by increasing student numbers or by an increase of training in basic skills training (e.g. Her Majesty’s Stationary Office, 1998). This push towards an increased use of new technology has led to a situation in which there is a much greater interest in and discussion of evaluations of educational technology. The NDPCAL (Hooper, 1977) programme of the ‘70s stressed the importance of rigorously evaluating any learning approach, an opinion echoed in the Mc Farlane report of the early ‘90s. However, Higginson (1996) took a slightly different approach. He reported on the work of Learning and Technology committee set up by the Further Education Funding Council (FEFC), investigating how FE colleges were using technology and how beneficial the use of technology was for learners. Higginson recommended the setting up of a national staff development programme, specialist advice centres and demonstration projects to show the effective incorporation of learning technologies, the Quality in Information and Leaning Technology programme (QUILT) and an advisory committee (jointly with the British Educational Communications and Technology agency, BECTa). Here the stress was more on promoting effective use, but the consequences have included an increased interest in evaluation promoted by the Further Education Development Agency. This is the context for the case study described below, which includes an interest in supporting change and in establishing good practice so that the potential of ILT can be realised. Therefore there is a wide range of different uses of ILT to be examined and a breadth of activities to be explored for the evaluator.

 

The Context, Interactions and Outcomes (CIAO!) framework

There is a long history of the use of computers in teaching at the Open University (see for example, Butcher and Greenberg, 1991). This history has informed the development of a model for evaluation that is laid out in Jones et al. (1996) and reproduced in Table 1 below.

 


Context

Interactions

Outcomes

Rationale

In order to evaluate CAL we need to know about its aims and the context of its use

Observing students and obtaining process data helps us to understand why and how some element works in addition to whether it works or not

Being able to attribute learning outcomes to CAL when CAL is one part of a multi-faceted course is very difficult.  It is important to try to assess both cognitive and affective learning outcomes e.g. changes in perceptions and attitudes.

Data

Designers' and course teams' aims

Policy documents and meeting records

Records of student
 interactions

Student diaries

On-line logs

 

Measures of learning

Changes in students' attitudes and perceptions

Methods

Interview CAL designers and course team members

Analyse policy documents

Observation

Diaries

Video/audio and computer recording

Interviews

Questionnaires

Tests

Table 1. The model for evaluation

 

A framework for the conduct of evaluations, CIAO!, focusses on three aspects: context, interactions and outcome. Context refers to a wide interpretation of the rationale for use of the software including the aims of the use. This includes considerations of questions such as how the CAL fits within the course, where is it used (at home/in the classroom), who uses it and whether it is used alone or in groups. Interactions refers to a consideration of ways of examining and documenting how students interact with computers and each other, focussing on the learning process. Often, in recording such interactions, inferences can be drawn about the learning process. An outcome refers to a wide interpretation of the changes in students, which result as a consequence of using the program. Restricting consideration of outcomes to cognitive ones is too limited as often other consequences emerge which need exploration. This means that learning outcomes still need to be considered but so also do changes in learners’ perceptions and attitudes. One aspect of measuring outcomes is the fact that it is very difficult to attribute learning outcomes to a particular use of ILT or to a particular use of the computer, when that is only one part of a multi-faceted experience of a course.

Looking vertically down the framework, the data row highlights the different types of data that needs to be collected to inform the work, and the methods row specifies how this is done. Many different types of methods have been used: interviews with staff to inform an understanding of context, questionnaires sent out prior to the start of the course, post course questionnaires and logs of computer use and terminal time. The framework reflects its origin in the evaluation of distance learning courses where questionnaires are often used to collect information from large numbers of students, while small-scale studies can supplement this picture with observations and interviews.

 

Experience from work at the Open University

The framework evolved from the many years of working with course teams at the Open University on a range of evaluation projects over the past twenty years (see e.g. Butcher and Greenberg, 1991). However, the details of the framework emerged from a focussed project where we applied our experience to a set of programs for using ILT in science and technology projects. The projects ranged from summative testing of a tutorial CAL program for a technology foundation course, formative testing of a simulation of chaos for a higher level physics course and formative testing of a multimedia CD ROM for use in a foundation science course and the summative testing of conferencing applications on a science studies course. All our work had involved a consideration of context as a starting point for the design of an evaluation working alongside software developers and instructional designers. Our experience of working on formative evaluations had taught us the value of observations.

One evaluation project on a program to help with understanding phase diagrams illustrated this (see Tosunoglu et al., 1996; Scanlon et al., 1996). We established relatively quickly, using questionnaire returns, that students found a particular program enjoyable and its game format motivational. We used pre- and post tests to compare students' learning before and after the use of the program in an attempt to establish how the program contributed to students’ learning. It was difficult to show that any observed differences between pre- and post-test scores immediately after use represented what the learning gains might be, as the package was still available for student use at any time throughout the year which meant that further tracking was necessary. In any case it was difficult to establish that shifts in learning were due to the use of the computer-based teaching alone. This package was only one part of a course that consisted of a variety of teaching resources and help from tutors and feedback from written and computer-based assessment packages. In this case, we nevertheless established a considerable amount about how students learn from the program using interviews and observations. During observation sessions students were observed using the software and semi-structured interviews were conducted.

Observations and interviews also featured significantly in a formative evaluation project where a version of multimedia software was piloted (see Taylor et al., 1995, 1996). The aim was to investigate if the format of a multimedia CD-ROM would serve the aim of presenting a course in an interactive environment in which the student has more control than in traditional text presentations.

These examples illustrate some of the parts of the framework we introduced earlier illustrating rationale and the selection of methods we used. To completely outline examples which illustrated the application of the framework in all its aspects would take more space that we have in this paper, so for the sake of brevity we have encapsulated here the results of a range of such studies in the following principles.

 

Principles for conducting evaluations

This section summaries a set of working principles for the conduct of evaluations which have been developed from the CIAO framework and our Open University experience. It can be seen by our earlier discussion that we value conducting evaluations of ILT in context. However we must note that this leads to certain constraints on the conduct of evaluations. A real world teaching and learning situation cannot be manipulated in the way that laboratory experiments can which makes it difficult to establish cause and effect in teaching situations. In attempting to establish the crucial variables in a particular successful use of ILT, it is tempting to set up the equivalent of a controlled experiment such as a direct comparison of groups of ILT users and non ILT users. We would argue that such an approach is seldom the appropriate response. The conduct of evaluations under a set of working constraints, however, provides us with a very clear idea of who in a particular evaluation is the audience for the findings. In all our experience and in the literature, it appears that establishing the aims or rationale of a particular ILT application is the key to designing an evaluation. Evaluation should involve learners and take account of context. We also feel it important to stress that in conducting an evaluation there is a need to note unanticipated issues which arise and would question evaluations where no surprise findings emerged.

 

Case study: Evaluating Learning with IT in Further Education

In this case study we describe our participation in one of the phases of a three phase study of the evaluation of learning with IT in the Further education sector, conducted between September ‘98 to July ‘99. This work involved the refinement and development of a tool to be used for the purpose of evaluation and the application of the tool and associated methodology to six case studies in colleges to allow (their) further refinement. A number of colleges were involved with projects where ILT was deployed. These projects included an on-line careers advice project, a pilot for resource based delivery, courses on educational use of IT for teachers, the use of an authoring package to put materials on-line, a multimedia key skill project for disaffected 14-16 year olds, and staff development in ILT.

This situation had a number of challenges; not least to apply the framework and principles described above to this particular evaluation situation. Some of these constraints are worthy of comment. The users of the tool were to be FE practitioners in a pressured environment, and who, therefore, as a result needed ‘off the shelf’ tools which did not require any particular expertise and would be easy and quick to use. The wide range of projects, mentioned above each had their particular aims as well as different types of students varying in age and mode of study. These constraints led to the decision to use questionnaires. These have the advantage that they are familiar, do not need a high level of expertise and are quick to use. They could be used on their own or combined with other tools which would allow the triangulation of information. They also allow a way to gather information from each major stakeholder involved in any specific project whether student, lecturer or manager. It is also possible to produce a template for questionnaires so that they can be used generically but can also be changed to be sensitive to context.

As the aims of each ILT innovation were different, the approach adopted focussed on the development of a generic evaluation tool that was customisable. It was important to focus on the rationale, ensuring that in the tool there were opportunities to report on different perspectives to the development of the particular ILT application. We hoped that the opportunity to customise the tool would cope with the range of projects but would also allow a sense of ownership of the evaluation project to develop among practitioners. We were able to assess how this worked in practice, as we worked on the development consultation and trialling phase of the tool. We found that one issue for all the participants was difficulty in finding time for evaluation and that the three separate strands of the evaluation process needed more integration. An analysis template was therefore developed so that the three questionnaires (learner, manager, and lecturer) could be compared and critical areas could be highlighted and cross-referenced. 

In the final phase the participant groups (colleges) used the modified tool to conduct evaluations of their ILT provision and we made final visits to college to collect data and produced a report and guidance manual for use of the tool based on experience with the tool.

There is a great challenge in using a generic questionnaire to probe into specific situations. Some problems were experienced in formulating questions on learning outcomes due to the very variable aims of the colleges’ projects. Some of the feedback suggested that more questions should be included that related specifically to student learning. This was responded to the last phase of the project by introducing a question that needed to be customised by the colleges but this in some cases did not happen. The final version of the manual therefore outlined the importance of customisation and ensuring that the questionnaire reflected the specific aims of the project it was being used to evaluate.

 

Reflections

In this case study we confirmed the difficulties of using a generic tool when customisation does not take place in the way we expected. However, the overall message was that the questionnaire template was robust as it stood, and the analysis template that had been developed elicited very positive comments. The use of a questionnaire in this context had a number of advantages, in terms of scale and applicability. This is not to deny our experience described above in the value of observation and interview approaches as a follow up.

Our findings on this project were of two types: about the use of generic evaluation tools and about user involvement in evaluations. We found overall that to encourage of the use of a generic evaluation tool, support for customisation and support for the tool is crucial. A manual for the use of the tool is vital. It is necessary for the evaluation process to be customised and for staff at different levels of involvement with the project (such as the manager) to have strong awareness of the aims and objectives of the project. Some of the staff who were involved in the evaluation work had not been part of the original ILT projects development team and were therefore not aware of the original project aims. We decided that increasing support through the analysis template would be necessary to increase the likelihood of effective use of the tool. The project confirmed that information from different sources (different stakeholders in our project) was extremely valuable in cross-referencing or triangulation which Breen et al. (1998) refer to as “the use of a combination of research methods”. These have the aim of “where various methods leave gaps in understanding or room for misinterpretation, the other methods would provide answers and improve the interpretability of each set of results” (ibid, p 153).

We noted above the trend towards involving practitioners with evaluation and much of this case study was aimed at, primarily, teachers evaluating their use of ILT and our approach was to involve users as part of the development cycle. We found a useful rationale and additional motivation for this involvement by helping users to see how it can lead to useful information that could be included in the self inspection process which has a role the quality assurance processes in the sector.

 

Conclusions

The aim of the CIAO! framework which informs our work is to encourage the consideration of a variety of methods, rather than offering a prescriptive approach to evaluation. This approach has served us well in influencing work in our own institution, and we have illustrated here how this approach informed our work in a further education context. We have illustrated here that context is crucial and when developing any evaluation tool evaluators must find a way of taking it into account. Given our earlier comments about expansion in the sector with the consequent impact on staff, it is also important to find a motivation or ‘re-use’ for the results of an evaluation process.

 

References

  • Breen, R., Jenkins, A., Lindsay, R. & Smith, P. (1998). Insights through triangulation. In Oliver, M. (Ed.) Innovation in the Evaluation of Learning Technology, London: University of North London, 151-168.
  • Brown, J. S., Collins, A. & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18 (1), 32-42.
  • Butcher, P. & Greenberg, J. M. (1991). Educational computing at the Open University: the next decade.  Education and Computing, 8 (1), 201-215.
  • Calder, J. (1994). Course evaluation; improving academic quality and teaching effectiveness. In F. Lockwood (Ed.) Materials Production in Open and Distance Learning, London: Paul Chapman Publishing, 361-370.
  • Draper, S., Mc Atteer, E., Tolmie, A. & Anderson, A. (1994) Bringing a constructivist approach to bear on evaluation. In H. Foot, C. Howe, A.Anderson, A.Tolmie & D. Warden (Eds.) Group and Interactive Learning, Southampton: Computational Mechanics Publication, 31-36.
  • Draper, S., Brown, M., Henderson, F. & McAteer, E. (1997). Integrative Evaluation: an emerging role for classroom studies of CAL. Computers in Education, 26 (1-3), 17-32.
  • Draper, S., Brown, M., Edgerton, E., Henderson, F., McAteer, E., Smith, E. & Watt, H. (1994). Observing and Measuring the Performance of Educational Technology. TILT Report no. 1, Glasgow, UK: University of Glasgow.
  • Eisner, E. (1985). The Art of Educational Evaluation, London and Philadelphia: Falmer Press.
  • Higginson, G. (1996). Report of the Learning and Technology Committee, London: FEFC.
  • Her Majesty’s Stationery Office (1998). The Learning Age: a renaissance for a New Britain, London: HMSO
  • Hooper, R. (1977). National Development Programme in Computer Assisted Learning: the final report of the Director, London: CET.
  • Jones, A., O'Shea, T. & Scanlon, E. (1987). Evaluation of computer assisted learning at the Open University. In Jones, A., Scanlon, E. & O'Shea, T. (Eds.) The Computer Revolution in Education. Brighton: Harvester Press, 263-276.
  • Jones, A., Scanlon, E. & Tosunoglu, C. (1998). Reflections on a model for evaluating learning technologies. In Oliver, M. (Ed.) Innovation in the Evaluation of Learning Technology, London: University of North London, 25-40.
  • Jones, A., Scanlon, E., Butcher, P., Murphy, P., Greenberg, J & Ross, S. (1999). Evaluating CAL: a distance education perspective. Interacting with Computers, 11, 499-516.
  • Jones, A., Scanlon, E., Butcher, P., Greenberg, J., Ross, S., Murphy, P & Tosunoglu, C. (1996). Evaluation of computer assisted learning at the Open University-fifteen years on. Computers and Education, 26 (1-3), 5-15.
  • Kandaswamy, S. (1990). Evaluation of instructional materials: a synthesis of and methods. Educational Technology, 20 (1), 19-26.
  • Kemmis, S., Atkins, R. & Wright, E. (1977). How do students learn: the UNCAL Evaluation, occasional publication no 5, Norwich, UK: Centre for Applied Research in Education, University of East Anglia.
  • Mc Farlane, A. (1992). Teaching and learning in an expanding Higher Education System, Edinburgh: Committee of Scottish University Principals.
  • Oliver, M. (1999). A framework for the evaluation of learning technology, ELT Report No 1, London: University of North London.
  • Parlett, M. & Hamilton, D. (1972). Evaluation as illumination: a new approach to the study of innovatory programs. In D. Hamilton (Ed.) Beyond the numbers game: a reader in evaluation and learning, London: Macmillan, 6-22.
  • Ross, S. & Scanlon, E. (1995). Open science: the distance teaching and open learning of science subjects, London: Paul Chapman Publishing.
  • Scanlon, E., Jones, A., Butcher, P., Greenberg, J., Ross, S., Taylor, J., Murphy, P. & Tosunoglu, C. (1998). Learning with computers: experiences of evaluation. Computers and Education, 30 (1/2), 1-9.
  • Scriven, M. (1980). The methodology of evaluation. American Educational Research Association Series of Curriculum Evaluation, 1, 32-83.
  • Stake, R. E. (1967). The countenance of educational evaluation. Teachers’ College Record, 68, 523-540.
  • Stufflebeam, D. (1971). Educational evaluation and decision making, Itasca, IL: F.E. Peacock Publishers.
  • Tosunoglu, C., Butcher, P., Scanlon, E. & Jones, A. (1996). The Works Metallurgist: an evaluation of a CAL package on phase diagrams. Association for Learning Technology Journal, 4 (1), 51-57.
  • Scanlon, E., Ross, S. & Tosunoglu, C. (1997). Simulating chaos: an evaluation of the driven pendulum experiment. Association for Learning Technology Journal, 5 (1), 60-65.
  • Taylor, J., Oliver, M. & Hatzipanagos S. (1995). The Galapagos Tutorial: a formative evaluation, Programme on Learner Use of Media, Report No 55, Milton Keynes, UK: Open University.
  • Taylor, J., Scanlon, E. & Hodgson, B. (1996).Multimedia and science education. Education Research and Perspectives, 23 (2), 48-58.
  • TLTP (1996). Evaluation of the Teaching and Learning Technology Programme, Final Report M21/96, Bristol: HEFCE.
  • Tyler, R. (1986). Another paradigm and a rationale. Changing concepts of educational evaluation, 10 (1), International Journal of Educational Research Monograph.

decoration