Educational Technology & Society 5 (1) 2002
ISSN 1436-4522

A framework for evaluating computer supported collaborative learning

Jim Ewing and David Miller
Northern College of Education
Dundee, Scotland
Tel: +44 1382 464367
Fax: +44 1382 464293
J.M.Ewing@norcol.ac.uk
D.Miller@norcol.ac.uk

 

ABSTRACT

Starting from some central issues of Information and Communications Technology (ICT) in learning and of collaborative learning, this paper postulates a composite framework for evaluating learning environments. Using the framework, one scenario is described and analysed. The framework identifies where ICT in learning (e-learning) may be enhanced and where ICT may be difficult for students. Some conclusions are made relevant to the design of ICT learning environments. Areas for future research are suggested.

Keywords: Collaborative learning, e-learning, Evaluation framework, Personalisation of learning, Social interaction


Introduction and theoretical background

The study examined in this paper focuses on some issues and questions relating to how the use of Information and Communications Technology (ICT) influences teacher-centred learning. Virtually all students in higher education will experience some aspect of ICT in their learning programme and most will be expected to actively participate in some computer or computer-related learning environment. Teacher education students, in particular, are expected to become competent and confident in using ICT in learning before they start teaching in the classroom. This study looks less at the educational performance of the learners (students’ cognitive gains) and more at their perceptions both of the learning environment and of their part within it. Emphasis is on student involvement in interactive learning, with particular regard to inter-student collaboration.

The relationship between the socio-cultural and the cognitive influences on a learning exchange is undoubtedly a complicated one and much about this is unknown. Cobb (1996) suggests that “it is as if one perspective constitutes the background against which the other comes to the fore” (p. 45). The contribution of a constructivist approach to learning goes some way to bring the socio-cultural and cognitive perspectives together as previous writing has already suggested (Duffy & Jonassen, 1992; Ewing et al., 1999; Jonassen, 1994). The application of constructivism to the design of ICT learning environments allows a freer approach to the instructional process than adopting a more process-based approach. Constructivism lays greater store in the involvement of the individual in learning than in the design of a set of instructional sequences focusing on learned outcomes. These differences present a challenge to the promotion of new approaches to ICT supported learning (e-learning) and in the evaluation of their effectiveness in learning.

Learner control is one aspect of this challenge. Learner control was initially an encouragement to adopt personal control over access to learning programs based on the belief that this would promote effective and efficient learners (Reeves, 1993). More recently, the emphasis appears to include the use of information technology as cognitive tools where the learner is more engaged in constructivist learning and personal representations of knowledge (Jonassen and Reeves, 1996). Learner control has also featured in Merrill’s earlier Component Display Theory, to be replaced with advisor strategies in his later Component Design Theory, both of which address the organisation of a learning (or instructional) environment for the promotion of relevant cognitive structures for student learning (Merrill, 1988). In the later theory, Merrill suggests that organisation and elaboration within a learning environment should be part of the instructional design rather than being left to the learner to decide for himself. The theory is structured round a course evaluation model which involves: organising subject matter according to a content-appropriate structure; course organisation of lessons and modules; selection of learner interactions (“transactions”) for the promotion of ideas; and the identification of strategies for progress through a course. This model has been integrated into a computer-based delivery system known as ID Expert (Merrill, 1998). This approach can be viewed along with the constructivist orientation of learning in the cognitive tools approach espoused by Jonassen and Reeves (1996). Together these ideas encourage a closer examination of the individual learner in the electronic learning environment and in particular the relationship between collaborative learning and electronic learning.

The place of computer supported co-operative and collaborative learning is already well established within education (see, for example, Davidson & Worsham, 1992; McConnell, 1994). Some of the difficulties that might be addressed through a collaborative approach using computer technology are elaborated by Koschmann et al. (1994). They suggest that collaborative learning, case-based instruction and a significant emphasis on student autonomy make a substantial contribution to effective learning. Cook and Boyle (2000) also identify some of the key issues in the effective delivery of learning material, focussing on campus-specific networks. They suggest that some key issues, including task appropriateness, learner motivation and the delivery environment, should be taken into account in the planning of networked learning environments. Somewhat similarly, Kearsley (2000) proposes an engagement theory where learners must be actively engaged in relevant tasks for effective learning to take place. He suggests that learning, including learning through computers, requires collaboration among the participants, involvement in problem based activities and relevant, realistic learning resources.

The most widely reported approaches to collaborative learning in general, notably in Dillenbourg (1999), Johnson & Johnson (1989, 1994) and Slavin (1995) refer to learning situations where individuals work together (usually in small groups) towards a common goal. Significant features of learner interaction include shared as well as individual responsibility and accountability, good communication skills, and learners’ ability to identify their role in the learning task. Salomon also stresses the important role of social interaction in learning in the classroom (Salomon, 1992) and suggests that research should look at the relationships between individual learning and group learning. The more focussed writing on computer supported collaborative learning is beginning to address additional issues which include knowledge construction, social learning and the postulation of theoretical models which help in the setting up of interactive electronic learning environments (for example Bentley et al., 1997; Daradoumis & Marquès, 2000). Some interesting work by Salomon has explored how the use of computers might support cognitive and intellectual performance through increasing the learner’s mental effort (Salomon et al., 1989; Salomon et al., 1991).

Different ways of approaching collaborative learning and its use within ICT have  been proposed. For example, the traditional view of educational computing as a means of communication by transmitting knowledge is challenged by Pea (1994) who favours a much more transformative approach. When communication is seen as transformative, it is two-way, interactive and dynamic, with the emphasis on construction of knowledge rather than the receipt of information. When using an existing curriculum resource to create a computer-based learning environment, the teacher does more than simply make the material accessible via a computer. Through using a computer to present curricular information the teacher is able to reformulate the information in a fashion which better meets the needs of the learner. In doing so, the teacher has to understand the learner and how to support him in the building of his own knowledge. The achievement of this through the involvement of ICT presents the challenge of a greater level of teacher and learner interaction because this approach to constructivism recognises not only the roles of each participant (teacher and learner) but also their cultural backgrounds and previous experiences from other learning environments. Pea refers to this as ‘collective’ learning. Jonassen (2000) contrasts this transformative view of educational technology with modern and post-modern views of technology in an effort to explain how it might help to transform learning and to support learners become independent and self-regulating.

Cobb (1996) makes a similar suggestion and emphasises the meanings and understandings that have to be taken as shared within a group for active knowledge construction to take place and which are therefore the core of social learning. The interaction between a teacher and students functions most successfully if there is an open and accepted sharing of social standards, cognitive awareness and even of personal interests.  The sharing of a collaborative learning experience more closely between a learner and a teacher is probably different from peer group collaborative learning, which Johnson & Johnson (1989, 1994) call co-operative learning.

In order to examine the relationship between e-learning and collaborative learning, a model of ICT supported learning that is essentially constructivist in nature has been adopted. This draws on the extensive work of others such as Jonassen (1994, 1996) and Kafai & Resnick (1996) in establishing a theoretical framework for promoting e-learning (Ewing et al., 1999).

A framework is therefore proposed on the basis of looking for relevant points of contact across the two parameters, collaborative learning and ICT supported learning. This framework identifies the key features of both approaches and how they might relate to each other, as indicated in Table 1.

 

Key features of e-learning support

Key features of collaborative learning

  • autonomy in student learning
  • learners have individual responsibility and accountability
  • an environment which promotes collaborative learning
  • learning interaction takes place in small groups
  • moving beyond knowledge transmission to include communication as a real life skill
  • communication during learning is interactive and dynamic
  • promotion of personalisation and reduction of depersonalisation of learning
  • learners can identify their role in the learning task
  • support for learners in development of ITC and personal learning skills
  • participants have a shared understanding within the learning environment

Table 1. A framework for evaluating e-learning environments

 

The purpose of this framework is to guide evaluation of electronically supported learning environments. The effectiveness of the framework in practice will depend on the strength of the links between the pairs of features in the table and this is described further in the context of a study of undergraduate students following a modular degree course in teacher education. The work undertaken by the students was a course on research methods where the learning material was available online and supported by tasks which required students to work together in self selected groups and to submit their responses in an open forum available to tutors and the whole student course cohort (Ewing, 2000).

 

Evidence in support of the framework

Autonomy in student learning

A characteristic of this study was the degree of control students had in their decisions regarding involvement in the learning activities. It meant that they could work on the learning material at a time that was convenient, and they could make decisions about how and when they completed tasks and posted responses (within a recommended time frame). Students falling behind were sent a personal note to ask if the tutor could help them catch up.

Being given the freedom of choice did not appear to compromise the focus on collaboration and all students experienced the same course related pressure points but were happy to find a time to collaborate. In addition to recognising that there were individual differences in relation to personal learning styles, the students also gave a high priority to responsibility to their peers in the learning tasks. They became aware of the value of collaborative learning and of their own contribution to the group’s work and as a testament to such commitment, there was not a single student who did not participate in a group task.

 

Promoting collaborative learning

The promotion of collaboration was encouraged through positive reinforcement feedback from tutors. Although the feedback was content-related, additional supportive and reinforcing comments were focussed on the experiential process of working together. The selection of working groups was left to the students. Some groups were found to be relatively stable where others varied from week to week. The size of these self-selected groups ranged from 2 to 5, with the most common groups comprising 2 or 3 members.

In the end-of-course evaluation, students indicated a high level of acceptance of peer interaction and collaboration as a successful approach to their own learning. This positive view of collaboration is supported in the suggestions made by students to further promote of this approach.

 

Beyond knowledge transmission

It was expected that the students would think about the words and expressions they used to communicate their ideas and suggestions in responding to the tasks. They were expected to have the reader in mind for two reasons: firstly to express their ideas clearly and succinctly and secondly to comment on the suggestions of others in a positively supportive manner. However there was no widespread awareness of this matter and only about half of the students said they were thinking about how they expressed themselves rather than the content of their suggestions when making their responses to the tasks. That group indicated that they would try to make their contribution clear, less formal, or use fewer words.  Some of them took the trouble to prepare a script off line to ensure good communication. Many others suggested that good communication was a function of how much time they spent on making an electronic response. If they spent less time, their response was more likely to be solely task-related.

 

Promotion of personalisation

Students’ personal involvement in learning was felt to be of particular relevance when moving from a traditional learning environment to one where using an electronic medium for course presentation might be seen as depersonalising. In earlier pilot work, students had commented on their unwillingness to become involved computer oriented learning because they saw it as isolating and depersonalising.  A means for electronic communication amongst students and between students and the tutor, was seen as one way of promoting personalisation, or at least of decreasing depersonalisation In this study, therefore, the use of an open electronic conference for student responses and tutor replies, was a core part of the learning environment.

The use of students’ and tutors’ names was considered to be very important and all student work was named by its author or authors and all tutor replies were personally addressed to these authors although the replies were available for all to read.

Because of the complex nature of personal involvement in a learning task, it is very difficult to assess it in any depth. The students have suggested in end-of-course evaluations, however, that promoting personalisation has helped them to identify their own role within a collaborative learning task. They claim to have recognised that their own contribution is as valuable as that of others, perhaps because the exchanging of views and ideas was open to all involved.

 

Shared understanding

Although all students had prior experience using ICT, there were different degrees of competence and confidence. Consequently, an important element was the provision of optional support for those individuals or groups who were uncertain about aspects of the technology. 

Support for personal learning skills was ongoing through the feedback and in face-to-face contact. The students’ work, which was shared openly in the electronic conferences, was evaluated only in an informal way. This was more like a collaborative evaluation such as Sharan & Sharan (1992) suggest, where tutors can learn about their students’ progress through the shared learning environment and can direct them to adopt a reflective approach to their own collaborative learning skills. The avoidance of a formal assessment which might be associated with a competitive element was deliberate and is in tune with other findings which indicate that competition might not be the preferred route to promote personal, affective and social learning skills  (Yu, 2001).

The tutors took great care to appreciate the emotional needs as well as the curricular needs of the students in terms of their online submissions. Earlier trials had indicated that not replying to every student had the effect of some feeling ‘left out’ and subsequently this was scrupulously avoided. The workload for the tutors was significantly reduced through getting students to submit as groups.

 

An evaluation of the framework

The framework for computer supported learning given in Table 1 is an attempt to make meaningful links between ICT and collaborative learning, based on some of the current beliefs as they impinge on aspects of learning and teaching. It was felt that if such a framework could be shown to be a fair representation of our current practices and understanding, it might be useful in evaluating specific learning scenarios which have been designed to reflect ICT and collaborative learning. This study suggests that there are meaningful links between e-learning and collaborative learning.

The evidence from this study shows that it is possible to give students freedom and choice over the conditions and circumstances for their own learning and, in doing so, to sustain a high level of individual responsibility and accountability for collaborative learning. Encouraging learner control in this case has taken the affective and socio-cultural aspects into account as much as the cognitive features of learning and this appears to have helped in encouraging individual responsibility and accountability.

In this study, the encouragement of working in small groups appears to have successfully promoted collaborative learning. More importantly, the students’ willingness to participate in this way, their general satisfaction and approval of collaborative learning in an ICT learning scenario are all at a commendably high level. Asking for more of the same can be taken as an indicator of some success in designing an appropriate learning scenario.

However, there appears to be a substantial element of communication during learning which is utilitarian and content-based rather than concerned with a meaningful person-to-person exchange. It seems  difficult to extend the perceptions of students to view communication within a learning context as a valuable aspect of learning within its own right. Moving beyond knowledge transmission is not an easy aspect of e-learning for teachers or students. We need to examine more fully how interaction within a peer group or between a tutor and the student influences the construction of knowledge by the individual learner. The shared understandings and perceptions of those in a learning exchange may be harder to establish within an electronic learning environment.

Computer supported learning environments may have a particular strength in providing the opportunity to promote personalisation of learning which in turn may help learners play their part more fully in successful collaborative learning. In this study, determined efforts were made in this regard and in the main they have been successful. There is probably a balance to be attained in terms of knowledge construction and feeling positive about a learning event. In the context of designing ICT learning scenarios, recognising a role for personalisation is worthy of some attention.

The evidence from this study seems to suggest that the participants in the e-learning scenario have a significant influence on each other. The learning environment in this study does show that support for learners is a key feature and is certainly presented in some measure. The level of shared understanding may be a bit harder to ascertain in clear cut terms but the present indications are that sharing can meaningfully operate at both the tutor-student and at the student-student levels.

 

Conclusions

While admitting that e-learning is not always easy learning, this study indicates three ways in which e-learning might evolve: promoting collaborative learning within the ICT learning medium; affective learning and interpersonal communication as a learning outcome; and, the personalisation of the learning environment.

There are clear indications from this study that for the majority of students, collaborative learning in an ICT learning medium is both possible and acceptable. Some did, however, have a residual doubt and uncertainty of whether it is ‘acceptable’ to undertake coursework and to submit course related tasks by working in groups. Nor were the personal gains of working collaboratively using computers immediately obvious to all the students.

The residual doubt is not surprising. Communication and its part in the interactive construction of knowledge are not straightforward and uncomplicated. For these students, as perhaps for many others, the links between the socio-cultural and the cognitive aspects of learning are still at a poorly developed level, and ICT may not make this any easier. Like other learning environments, e-learning attempts to embrace the emotive and conative aspects of learning as well as the cognitive ones. Also, computer oriented collaborative learning lays some emphasis on learner interaction where participation is part of the learning, not just a means to an end. These features serve to highlight the complexity of e-learning and it is not difficult to see that are often no easy solutions. 

Ross & Schulz (1999) argue that there is a need for more research into the area of learning styles and human-computer interaction. Kearsley (2000) highlights the importance of social skills in on-line learning and correctly, in our view, suggests that there are significant differences in social behaviour between e-learning and the more traditional face-to-face interaction of classroom based learning. There is also still a lot we have to learn about the processes involved in social interaction in collaborative learning through the computer (Daradoumis & Marquès, 2000). We support these views and suggest that the human-human interaction in this context merits special attention. For instance, we need to know more about the differences between tutor-student interaction and student-student interaction and how much this is influenced by e-learning environment. Research in this area should lead to promoting how best traditional learning might be integrated into ICT based learning.

A second poorly understood aspect of ICT in learning is the promotion of personalisation. It is widely held that ICT is not appropriate for all learners or all learning experiences and that not all students enjoy, or indeed benefit from working with a computer. How much a learner’s sense of isolation or depersonalisation contributes such feelings of inappropriateness appears to be largely unknown. This study indicates a few instances of an unexpected strength of computer supported learning environments in providing the opportunity for promoting personalisation of learning. This might raise the question of the relative importance of different learning outcomes - knowledge, understanding, skills, self-worth, etc. Some of these are easier to assess than others and the context of the learning will influence how important each might be.  In the study reported here there was no formal examination of student knowledge gained from undertaking the module.  Rather, their skills, insights and understandings were ‘put to the test’ by getting each student to engage in a research activity within which small group collaboration was still expected. The level of personal involvement was maintained by permitting a high level of student choice in all aspects of the practical application of their new knowledge and skills. Their success was at least as good as under more traditionally taught courses (Ewing, 2000).

The use of an evaluation framework such as that suggested in this paper could address each of these possible ways forward. At its simplest, the framework could be applied as a broad indicator of the presence or absence of the five features of e-learning or of collaborative learning.

 

References

  • Bentley, R., Appelt, W., Busbach, U., Hinrichs, E., Kerr, D., Sikkel, K., Trevor, J. &. Woetzel, G. (1997). Basic support for cooperative work on the world wide web. International Journal of Human Computer Studies: Special issue on Novel Applications of the WWW,
    http://bscw.gmd.de/Papers/IJHCS/IJHCS.html.
  • Cobb, P. (1996). Where is the mind? A coordination of socio-cultural and cognitive constructivist perspectives. In C. W. Fosnot (Ed.), Constructivism: Theory, perspectives and practice (pp. 34 - 52). New York: Teachers College Press.
  • Cook, J., & Boyle, T. (2000). Effective delivery of on-campus networked learning: Reflections on two case-studies. In M. Asensio, J. Foster, V. Hodgson & D. McConnell (Eds.), Networked Learning 2000. Proceedings of the 2000 International Conference on Innovative Approaches to Lifelong Learning and Higher Education through the Internet (pp. 64 - 71). Lancaster University and the University of Sheffield.
  • Daradoumis, T., & Marquès, J. M. (2000). A methodological approach to networked collaborative learning: Design and pedagogy Issues. In M. Asensio, J. Foster, V. Hodgson & D. McConnell (Eds.), Networked Learning 2000. Proceedings of the 2000 International Conference on Innovative Approaches to Lifelong Learning and Higher Education through the Internet (pp. 72 - 77). Lancaster University and the University of Sheffield.
  • Davidson, N., & Worsham, F. (Eds.) (1992). Enhancing thinking through cooperative learning. New York: Teachers College.
  • Dillenbourg, P. (1999). What do you mean by “Collaborative Learning”? In P. Dillenbourg (Ed.), Collaborative learning: Cognitive and computational approaches (pp. 1 - 19). Amsterdam: Pergamon.
  • Duffy, T. M., & Jonassen, D. H. (Eds.) (1992). Constructivism and the technology of instruction: A conversation. Hillsdale, NJ: Lawrence Erlbaum.
  • Ewing, J. (2000). Enhancement of online and offline student learning. Educational Media International, 37 (4), 205 - 207.
  • Ewing, J., Dowling, J. & Coutts, N. (1999). Learning using the world wide web: A collaborative learning event. Journal of Educational Multimedia and Hypermedia, 8 (1), 3 - 22.
  • Johnson, D. W., & Johnson, R. T. (1989). Cooperation and competition: Theory and research. Minneapolis, MN: Interaction.
  • Johnson, D. W., & Johnson, R. T. (1994). Learning together and alone (4th ed.). Englewood Cliffs, NJ: Prentice-Hall.
  • Jonassen, D. H. (1994). Thinking technology. Educational Technology, 34 (4), 34 - 37.
  • Jonassen, D. H. (1996). Computers in the classroom: Mindtools for critical thinking. Englewood Cliffs, NJ: Prentice-Hall.
  • Jonassen, D. H. (2000). Transforming learning with technology: Beyond modernism and post-modernism or whoever controls the technology creates the reality. Educational Technology, 40 (2), 21 - 25.
  • Jonassen, D. H., & Reeves, T. C. (1996). Learning with technology: Using computers as cognitive tools.  In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology (pp. 693 - 719). New York: Macmillan.
  • Kafai, Y. B., & Resnick, M. (Eds.).  (1996). Constructionism in practice: Designing, thinking and learning in a digital world. Hillsdale, NJ: Lawrence Erlbaum.
  • Kearsley, G. (2000). Online education: Learning and teaching in cyberspace. Belmont, CA: Wadsworth Thomson Learning.
  • Koschmann, T. D., Myers, A. C., Feltovich, P. J., & Barrows, H.S. (1994). Using technology to assist in realizing effective learning and instruction: A principled approach to the use of computers in collaborative learning. Journal of Learning Sciences, 3(3),227 - 264.
  • McConnell, D. (1994). Implementing computer supported cooperative learning. London: Kogan Page.
  • Merrill, M.  D. (1988). Applying component display theory to the design of courseware. In D. H. Jonassen (Ed.), Instructional designs for microcomputer courseware. Hillsdale, NJ: Lawrence Erlbaum.
  • Merrill, M. D. (1998). ID Expert: A second generation instructional development system. Instructional Science, 26, 243 - 262.
  • Pea, R. (1994). Seeing what we build together: Distributed multimedia  learning environments for transformative communications. Journal of the Learning Sciences, 3 (3), 285 - 299.
  • Reeves, T. C. (1993). Pseudoscience in computer-based instruction: the case of learner control research. Journal of Computer-Based Instruction, 20, 2, 39 - 46.
  • Ross, J., & Schulz, R. (1999).  Can computer-aided instruction accommodate all learners equally? British Journal of Educational Technology, 30, 1, 5 - 24.
  • Salomon, G. (1992). New challenges for educational research: Studying the individual within learning environments. Scandinavian Journal of Educational Research, 36 (3) 167 - 182.
  • Salomon, G., Globerson, T., & Guterman. E. (1989). The computer as a zone of proximal development: Internalizing reading-related metacognitions from a reading partner. Journal of Educational Psychology, 81, 4, 620 - 627.
  • Salomon, G., Perkins, D. N., & Globerson, T. (1991). Partners in cognition: Extending human intelligence with intelligent technologies. Educational Researcher, 20 (3), 2 - 9.
  • Sharan, Y., & Sharan, S. (1992). Expanding cooperative learning through group investigation. New York: Teacher College Press.
  • Slavin, R. (1995). Cooperative learning: Theory, research and practice (2nd ed.). Needham Heights, MA: Allyn and Bacon.
  • Yu, F-Y. (2001). Competition within computer-assisted cooperative learning environments: Cognitive, affective and social outcomes. Journal of Computing Research, 24(2) 99 - 117.

decoration