An Evaluation of Computer Mediated Communication to Support Group Discussion in Continuing Education
Dr. Rachel Pilkington
Dr. Catherine Bennett
Dr. Sarah Vaughan
This study evaluates the use of Information and Communication Technology (ICT) to provide flexible learning including Distance Learning support for a Post-Graduate Master of Education course. To meet students’ needs a Virtual Classroom environment was developed using WebCT computer based tools. This environment is illustrated in Figure 1. Use of this environment was evaluated in relation to the quality of work produced by students and the nature of their collaboration and communication through the tools. Some preliminary results and emerging issues will be discussed.
Why Flexible Learning?
The student base is changing - those wanting to take post-graduate courses are no longer only recent graduates. Increasingly, students need to balance work and family commitments with the demands of learning new skills (McCombs, 1991). As a consequence Educational Institutions must respond rapidly and flexibly to meet students' needs.
“Britain towards 2010 published by the Department of Trade and Industry says students no longer need to be resident at universities...The document dismissed objections about face to face interaction between student and teacher - for most students it does not happen under traditional methods - and lifestyle". (Donald MacLeod reporting in the Guardian HE, 25th January 2000).
The argument is that, using ICT, flexibility can be provided for students living and working at locations distant from the University. However, to be effective ICT must meet requirements of quality. How do we meet these requirements given the financial constraints that inhibit face to face teaching? This is the problem for the design of the module critically evaluated here. Students enrolling on the course may be full-time, part-time or fully distant, International or Home students. For these students we not only need a course that delivers informative content but one that builds on existing employability skills, promotes critical thinking and stimulates students to become autonomous learners whilst, at the same time delivering practical ICT skills. Increasingly, society needs workers skilled in the retrieval, manipulation and dissemination of information (particularly but not exclusively through ICT). As a consequence of these shifting trends our Educational Institutions must themselves respond rapidly and flexibly to meet the need, not only for training in ICT, but training in the skills required for autonomous, life-long learning (Cowham, 1998).
Designing for Flexible Learning - Our Approach
Theories of learning suggest that deep conceptual understanding and the development of intellectual skills require active, constructive tasks (Piaget, 1970), that provide scope for verbal interaction (Vygotskii, 1978). Collaborative Learning exposes students to new information and alternative perspectives encouraging them to reflect on and revise conceptions (Blaye, Light, Joiner & Sheldon 1991; Littleton & Light, 1998). Articulation through Exploratory talk (inquiring, explaining and justifying) has been shown to be particularly beneficial for these purposes (Mercer and Wegerif, 1999). From the above discussion we have developed an approach (Barker & Pilkington, 2000) to guide course construction in a way that is grounded in theories of teaching and learning. This attempts to marry approaches based on setting learning objectives (Gagne, 1972; Bloom, 1976; Anderson & Sosniak, 1994) with a more learner-centred and constructive approach to setting learning tasks. This approach suggests that the course needs to provide opportunities for:
These stages may overlap and take place at different levels during the same activity. Acquisition could in principle take place in a variety of different ways, through self-paced individual reading, serendipitously in discussion, or by active 'discovery' (e.g. by using a simulation). However, exposure to resources (human or material) that can help students to grasp key concepts is required. To ensure meaningful learning rather than rote learning there needs to be reflective activity both during and following acquisition. This enables students to test and elaborate their understanding. Ideally there is a reflective cycle (Kolb, 1984) in which both learner and teacher are engaged. Thus, constructive feedback is used to refine understanding, pointing to gaps or inconsistencies in knowledge and prompting new goals to acquire new knowledge. Articulation of understanding in discussion can prompt such feedback from others but assessment addresses the need to present considered pieces of work for more explicit peer or tutor review. Feedback in the form of both student evaluation and student performance provides a similar role for the tutor in refining the setting of appropriate learning tasks.
In applying this approach to the design of Networked Learning we have designed a Virtual Learning Environment using WebCT’s integrated Computer Mediated Communication (CMC) and World Wide Web (WWW) Course Management Tools. Using these tools, flexibility can be enhanced for students living and working at a distance as follows:
The Module and Course
The Module evaluated here "Learning and the New Technologies" is a core module in "MEd in Information Technology, Multimedia and Education" and "MEd in Learning Systems Design" courses. The module aims to develop students' understanding of learning theory and stimulate creative reflection on implications for the design of computer-based learning. A Virtual Learning Environment (VLE) using WebCT’s CMC and World Wide Web (WWW) Tools was used to support the module. Figure 1 shows the home page with icons through which CHAT, Bulletin Board and Course Content were accessed.
Figure 1. The Home Page
The module ran for 12 weeks. It consisted of 11 two-hour face to face classroom sessions and 11 one-hour remote on-line CHAT seminars conducted through WebCT. One tutor and 2 demonstrators taught the module and there were 20 registered students (class of 1998). Half the students were part-time and half full-time students. There were no full Distance Learners enrolled for 1998 (data reported here). Thus, all students attended face to face sessions. Most (15/20) were teachers with an existing role in teaching ICT and seeking continuing professional development. Full-time students were mainly International Students for whom English was a second language but who were resident in Leeds to study. In contrast all part-time students were Home students, but many travelled considerable distances to Leeds.
Lecture notes and Slides were pre-prepared, edited and uploaded into the WebCT environment. Lecture Notes were plain text with hyper-links to glossary definitions. Slides were downloadable as Power Point mixed media presentations. Having the course material on-line enabled acquisition to take place at a distance, as a self-paced and individual activity. The emphasis was on giving maximum opportunity for articulation and application in face to face time. The expectation was that students would preview each week's slide presentation before attending the face to face session for that week. Associated information on schedule, assignments, exercises and group-work was also available on-line.
Figure 2. Course Slides
Figure 3. CHAT Room in Use
On-Line CHAT Seminars
The CHAT seminars were held at a regular time prior to the face to face session in order to discuss issues arising out of students’ reading of set papers. These evening seminars were tutor-led and content oriented. The aim was to provide a reflective and critical forum for discussion in which all students were encouraged to participate. The CHAT provided a useful means for the tutor to check on the understanding of key concepts to be reviewed in the face to face session the following day. A reconstruction of the CHAT Room in use is illustrated in Figure 3. The names of students participating have been changed but the dialogue is reconstructed verbatim. In this example students 'discover for themselves' the issue of whether self-explanation is truly self-explanation if it is prompted by the tutor's inquiries!
Face to Face Sessions
In the expectation that on-line course material was pre-read before attending face to face sessions there was no need for the tutor to give a lecture during face to face sessions. However sessions typically began with up to twenty minutes of orientation, reviewing issues arising out of the notes, CHAT and Bulletin Board discussions. This was followed by either a collaborative practical exercise or (from weeks 5-10) collaborative small-group project work. This structure enabled maximum time to be given to students to articulate their understanding and to apply it to constructive and interactive tasks - activities which are generally preferred in a face to face context.
The Small-Group Project Work
Design issues were made authentic for students by involving them in collaboratively producing their own mini project. From weeks 5 to 10, students organised their time, working in a group of 5 or 6, to produce a web-based resource (set of web pages). Students were free to choose the members of their group and the topic of their group’s resource. Although the assessment of this mini-project did not count directly towards their grade, students were encouraged to produce a resource that would help them complete their written essay assignments. Figure 5 shows an example resource.
Figure 4. Example of Student' Constructed Small-Group Resource (class of '99)
Managing Collaborating Together
Within the WebCT Virtual Classroom, private forums (private Bulletin Boards) were also set up to support small-group project work. This was aimed at helping students to manage their collaboration outside contact hours. Private e-mail and CHAT rooms were also available through WebCT. In week 10 students presented their pages and made their resource public for the rest of the class to view on the Web. They then peer-reviewed each other’s work using the Bulletin Board and a supplied evaluation framework (see appendix). The tutor reviewed the presentation and content of each resource, and the balance and management of the group’s collaboration to give feedback to each group.
The Evaluation of the Module
The evaluation of the module examines the use of the Virtual Classroom both to access course materials and to participate in on-line CHAT seminars on course reading. The quality and the quantity of the discourse produced by students and tutor in the on-line CHAT was also evaluated using dialogue analysis techniques (Pilkington, 1999). In particular the evaluation examined:
Overall Student Impressions
Fifteen out of twenty students returned an evaluation questionnaire. Overall, 80% of those giving a response rated the course quality as good to very good. Some students were supportive of the collaborative group-work and wished that assessment had been based on this, others saw problems in working at a distance. Part-timers did not value the on-line CHAT seminars as much as full-time (mainly) International Students.
Patterns of Use
From students’ questionnaires the main reasons for non-participation in the CHAT seminars was difficulty in accessing the CHAT from home due to other commitments at the scheduled time, or not having access to a Java enabled browser. From Table 1 it can be seen that six students did not participate in the on-line CHAT at all and five of these were part-time Home students. Most Home students showed little use of the CHAT. However, two Home students proved the exception to this rule producing 89% of all Home student’s turns. No International student (all full-timers) failed to participate at all. International Students showed a higher level of participation than Home students (though turn length was shorter). Access difficulties also affected part-time students' approach to group-work. There was evidence from group-work evaluations that these students adopted a more co-operative than collaborative approach to constructing the on-line resource and from questionnaires, of a frustration in not having sufficient common time to manage collaboration outside the face to face sessions. Table 1 is sorted by ranking students according to the number of weeks in which they spoke in the CHAT. It can be seen that this indicates that a low participation in the CHAT is likely to be accompanied by under average performance on the Group-work score. Group-work scores showed a significant positive correlation with number of weeks speaking in the CHAT (Spearman rank correlation rs = 0.74, N = 20, p < 0.01). This result needs to be treated with caution since each individual in the group is allocated the same grade based on rating the group resource produced. However, it supports the impression (from talking to students) that part-time Home students, frustrated by not having time to manage group-work, adopted a more independent approach to group-work which proved both less satisfying and less productive (see appendix for criteria in marking Group-work).
Table 1. Participation in CHAT Hits on Course Content Pages and Group-work Scores
Participation in CHAT Seminars
Table 2 divides students into Home students (resident and studying in the UK) and International (studying away from home in the UK). For the reasons above, most Home and International students showed different patterns of participation in the on-line CHAT. Many (60%) of Home students (who were mainly part-time students) did not take part in the CHAT at all whereas none of the International students failed to participate. Two Home students proved the exception to the rule, between them producing 89% of Home students' turns. These students both began the course as full-time students, although one later converted to part-time status. From a more detailed dialogue analysis of eight CHAT sessions the International students produced a similar number of words and turns as Home students, but these were more evenly distributed amongst them. Thus, although one International student was responsible for about 45% of all International students' words and turns, no International student logged in fewer than two times and no International student produced fewer than 2 turns or 23 words. Many International students also said on their questionnaires that they particularly valued reviewing the transcripts of the CHATs on the following day. Further dialogue analysis reported elsewhere (Pilkington, Treasure-Jones & Kneser, 1999) showed that International speakers had a slightly higher rate of initiating than Home students - indicating that they were not more reticent to speak than Home students were (though turn length was generally short). The proportion of quality content-related talk of an exploratory (critical and inquiring style) was a constant 45% across International and Home students alike, although the two dominant Home students produced a higher ratio of this type of talk than other students did.
Table 2. Home and International Student Participation in CHAT
Inclusiveness of Dialogue
The on-line CHAT was a strongly tutor-led discussion. From a total of 9831 words and 664 turns over eight analysed seminars the tutor produced 4487 words and 248 turns. This represents 46% of words and 37% of turns. One full-time International Student and two Home students accounted for another 3638 words and 284 turns between them - leaving just 17% of the word total (15% of turns) to the remaining students. Looking at the quality of the participation, the proportion of on-topic Exploratory Dialogue (see Pilkington, Treasure-Jones & Kneser op cit.) was a constant 45% for International and Home students alike. The two dominant Home students produced a higher ratio of this type of talk than other students did (25% of the total number of such moves). However, the tutor accounted for 54% of such moves. This raises the issue of how the CHAT could be made more inclusive and whether students would have benefited from this. From an analysis of a private CHAT session between two students in a one-to-one situation, students were more active and inquiring privately than in a tutor-led CHAT on the same topic held later the same day. This may suggest that the tutor should take a more hands-off approach in seminars and encourage students to express themselves more. However, it also indicates that when space for one-to-one peer-to-peer interaction is provided it is spontaneously and constructively used by some students (see Pilkington, Treasure-Jones & Kneser, 1999).
The Role of Assessment
Table 3 is sorted according to mark achieved in the final written assignment. Two students failed the module as a result of not submitting a written assignment. The average group mark is calculated without including a mark for these two students listed at the bottom of table 3. It was noted earlier that there seemed to be a reasonably strong correspondence between participation in the CHAT and a good mark in the group-work assessment. However, students who made no use of the on-line CHAT were not less likely to produce a well-written assignment. Since it is the written assignment (and not the group-work mark) that contributes to the final mark, there was no assessment-based incentive to participate in CHAT or group-work.
Table 3. CHAT use in relation to Group-work and Essay Assignment Marks
Conclusions and Issues Raised
Differences in participation in the CHAT seminars between Home (mainly part-time) and International (all full-time) students emerged from the evaluation presented here. In particular International students showed a more even distribution of participation with more International students participating. The reasons for non-participation in CHAT seminars were mainly pragmatic. Some students (mainly part-timers) had problems attending the evening CHAT seminars. The 6-7.00 p.m. time-slot for the CHAT was a compromise between full-timers staying late and part-timers logging in from home. This proved ideal for few. This year, with 6 students registered as Distance Learners, the solution was to run two seminar slots, a lunchtime slot and an evening slot. This has doubled tutor contact time but has also doubled part-timers' participation in CHAT seminars on set reading.
Difficulties in scheduling common time to CHAT also affected part-time students' approach to group-work. Difficulties in meeting face to face tended to lead towards more distributed and individual patterns of working, in turn limiting the score obtained for group-work. These difficulties continue to affect part-time students' approach to group-work this year. This year, 2 International full Distance Students (not attending any face to face sessions) were able to schedule lunchtime CHAT and Bulletin Board discussions with full-time campus-based students. This enabled them to collaborate in group-work, suggesting that (at least for part-time students) pressure of time and not the functionality of the tools is a more limiting factor affecting the quality of group-work. That two students failed to submit a written assignment and several students required extensions due to work or family problems further suggests that time to study and common time to meet (on-line or face to face) are more critical factors for part-time students. However, in piloting the VLE with a small number of full Distance Students this year, first impressions are that CHAT and Bulletin Board tools have not fully met these students’ need for a sense of presence. One student suggested video-conferencing so that they could really meet their fellow students. Kahl & Cropley (1986) suggest Distance Students feel more 'isolated' than face-to-face students and experience lower levels of self-confidence as a result. This can lead them to drop out. Students who commented on this, rated face-to-face contact above CHAT and CHAT above Bulletin Board in giving them a sense of presence. The Bulletin Board was sparsely used by full-time, part-time or Distance students, despite being seeded by the tutor on each of the topics discussed in the CHAT seminars; despite the difficulties involved the symmetric CHAT and not the asymmetric Bulletin Board was the preferred discussion tool. The speed of response was too low for the Bulletin Board to meet this need.
It was noted that there was a significant correlation between participation in the CHAT and a good mark in the group-work assessment. However, there was no such relationship between speaking in the CHAT and the written assignment mark. Students who made no use of the on-line CHAT (mainly Home part-timers) were not less likely to produce a well-written assignment. However, this is probably true for students well able to frame an argument on the topic. Participating in the CHAT may have assisted students starting from a lower level of knowledge or inexperienced in academic argument in English. From the questionnaires, full-time International students tended to value the CHAT more (even those who were silent) and were also more likely to read CHAT transcripts posted the next day. It was difficult for some International students to keep up (in real time) with discussion in a second language. The transcripts provided an opportunity for students to check their understanding. As already noted, part-time students were under considerable pressure to produce good quality work in the time available to them, indeed, two students failed to submit a written assignment. Since it is the written assignment that contributes to the final course grade and not the group-work score there was no assessment-based incentive for part-time students to participate in the CHAT or the group-work. This raises the issue of whether skills such as oral debate, computer-based textual and graphical design and team-working skills (the skills acquired and applied in the CHAT seminars and group-work), should be formally assessed. For this to be effective a component of the assessment should probably be based on student logs of the collaborative process and involve a private peer-rated as well as a tutor rated component.
The on-line CHAT seminars on set reading were strongly tutor-led. Based on previous research, CMC tools are poorly used unless the tutor actively facilitates discussion (Rohfeld & Himestra, 1995, p.91). From the dialogue analysis (Pilkington, Treasure-Jones & Kneser, 1999) the tutor interacting with two or three active students was effective in modelling Exploratory Dialogue. However, the analysis revealed a need to encourage more inclusive participation. The value of differing roles for the tutor (e.g. explaining, inquiring or encouraging) and the ideal balance between these is now the subject of further work.
All students benefited from some aspects of the Virtual Learning Environment. Part-time and distant students benefited from open and flexible access to course material and those who chose to participate in CHAT seminars benefited from not having to travel to meetings. Full-time International students benefited from flexible access to written transcripts of tutorial discussion as well as text-based lectures. When the VLE is seen as an add-on support to an existing course (a course with problems in meeting the different needs of part-time, full-time, Home and International Students), then the advantages are clear. This additional support was not achieved without costs both in terms of additional development time and additional student contact hours. Whether these benefits will prove sufficient to enable full Distance Learners to achieve comparable results remains to be seen.
Evaluation criteria for group-work. Group-work was marked independently by tutors and demonstrators who rated the work from 1 to 3 on each of the columns in the table below (where three represents very good, 2 good and 1 average an additional discretionary 1/2 mark could also be awarded). The marks from each of the three markers were summed and averaged to give the final overall score.