Evaluation of the Basic Life Support CD-ROM: Its effectiveness as learning tool and user experiences
1. Background to the research
The Interactive Teaching and Learning Project (INTaL) was funded by the NHSE South-West Teacher Education Development Funds in the UK, as a staff development project across the south-west region universities, with the University of the West of England (UWE) as the lead site. As part of the project a basic life support (BLS) CD-ROM was developed, using the findings of previous research (Moule and Knight 1997) and following recognised resuscitation guidelines (American Heart Association and International Liaison Committee for Resuscitation (2000) and Resuscitation Council UK (2000)).
A beta version of the CD-ROM piloted at UWE, received a positive response (see Moule and Gilchrist 2001), and the final version is now in use across the original INTaL partner institutions. The CD-ROM provides a unique resource, which relates to an area of nursing curriculum in which all nursing students must demonstrate competency (UKCC 1999). Basic life support is an essential skill for nurses, as they are often first responders to the collapsed person. Nurses require skills of assessment for cardiac arrest and need to initiate BLS, involving maintaining respiration and circulation for the casualty until emergency services, or advanced life support services, arrive. Basic life support is also an important skill for many health-care professionals and the BLS CD-ROM can facilitate interprofessional teaching and learning (DoH 2000, UKCC 1999) in a variety of settings, such as skill laboratories, clinical practice settings and the users’ own home. Finally, those who complete the CD-ROM also develop key transferable skills in information and communication technology.
This research project aimed to measure the effectiveness of the CD-ROM as a learning aid, and explored students’ experiences of learning knowledge for essential skills. It also sought to explore the lecturers’ experiences of using the CD-ROM as a teaching aid.
The project included two stages of data collection and analysis. Firstly, the effectiveness of the CD-ROM in the attainment of knowledge and skills development was measured, and secondly, students’ and lecturers’ experiences of using the CD-ROM as a learning and teaching resource were interpreted.
2.1. The Sample
Two universities formed the sample. Both were “Making a Difference” (DoH 1999) pilot sites, implementing the new nursing curriculum that required competency development in first aid (which included BLS) for all first year nursing students. A University in the midlands (Site 1) was randomly selected from the 13 “Making a Difference” (DoH 1999) pilot sites. Basic life support skills were taught and tested within their curriculum, however, they had not previously used a CD-ROM for delivery and had had no input to the development of the BLS CD-ROM. The second site in the south-west region (Site 2) had been part of the original INTaL project team. Each site received copies of the CD-ROM for use in delivering BLS knowledge prior to skills practice and testing sessions. Technical advice for supporting the use of the CD-ROM was provided throughout the research. Ethical approval was sought and successfully gained from each institution and a protocol to maintain confidentiality, anonymity and informed consent was applied.
2.2. The CD-ROM
The CD-ROM has four main components. It includes pre- and post-tests, best practice in ‘What, How, Why’ and a selection of four problem-based scenarios within ‘Critical Thinking’. The pre- and post-test questions are drawn randomly from a bank of ninety questions and are presented within five categories: airway, breathing, circulation, safety and responsiveness. Each student is randomly allocated four questions in each of the five categories, for the pre- and post-test. On completion of the test, scores are presented for each section as either 0%, 25%, 50%, 75%, 100%, with 25% being awarded for each correct response.
Using the pre-test, students can check their knowledge of BLS and identify individual learning needs. The students have to complete both pre- and post-tests, with the results of the post-test being e-mailed to lecturers. These can also be printed off for students to keep in their Portfolio of Learning. Best practice is presented in ‘What, How, Why’, which outlines the basic life support algorithm, including the recovery position and BLS in special circumstances, such as, pregnancy, trauma and near drowning. This section presents video footage and photographs alongside essential text. The main screen can be printed or stored for future reference.
The ‘critical thinking’ section supports the application of the principles of best practice. It presents four scenarios: choking, suffocation, and myocardial infarction in the home and in hospital. Realistic scenarios are portrayed through video footage and the students are required to answer multiple choice questions that provide instant feedback on their response. The CD-ROM also includes a glossary of terms and supporting references, both of which can be printed or saved.
2.3. Data collection
The pre- and post-test results were automatically emailed to the research university into a protected account, to be accessed by the researcher and module leaders responsible for BLS testing across both institutions. The sample across both institutions totalled n = 358 (see Table 1). A total of 115 nursing students who had just started their education programme at the university in the midlands (Site 1) completed the CD-ROM. Fifteen of the group participated in two focus group interviews, following their self-selection, which represented males, females and different cultural groups. Two lecturers were interviewed individually to gain their accounts of using the CD-ROM.
Table 1: Students sampled across both institutions
Data was also collected from two groups of students in the south-west (Site 2). Students starting new programmes in September 2001 formed the largest sample n = 167 (Site 2A). Those undertaking BLS for the second time, having started their course in March 2001 completed the sample n = 76 (Site 2B). This group had used a beta version of the CD-ROM six months previously, when first undertaking BLS testing in the University. This site was the only site involved in BLS testing, with n = 45 from the September cohort (Site 2A) and n = 43 from the March (Site 2B) cohort, taking part. This was necessitated as the lead researcher undertook all the skills testing to ensure consistency in the application of the test and limit the effects of any intervening variables, and the constraints of working with pre-programmed timetables across two institutions limited the researcher to working in Site 2 only.
Basic life support skill testing was completed to measure whether competency in the skill of BLS is positively correlated with knowledge attainment. Such testing was already a required component of the course, though for research purposes those students tested by the researcher self-selected themselves into the research group, and completed a second consent form to enable the use of their results within the research. Competency in BLS was measured following the BLS algorithm, using a computerised skill meter manikin, which provided a competency measure in expired air ventilation and external chest compressions. The results were made immediately available to the student and the university, to allow for further practice and testing for those students unable to demonstrate competency.
Eleven students self- selected themselves into the two focus group interviews, whose composition represented males, females and different cultural backgrounds. One lecturer was interviewed individually. Thus, across both institutions, twenty-six students and three lecturers talked about their experiences of using the CD-ROM as a learning and teaching tool.
2.4. Data preparation and analysis
The pre- and post-test results were recorded by the CD-ROM as either 0%, 25%, 50%, 75% or 100%, relating to the number of correct responses within each of the five categories: airway, breathing, circulation, safety and responsiveness. Within Site 2, percentages achieved for expired air respiration and external chest compression was included for those students taking part in BLS skills testing. Each university and group were coded separately (Site 1, Site 2 A, Site 2B) to allow for comparisons. The lead researcher and statistician were involved in data inputting to support accuracy.
The pre- and post-test results were compared using the non-parametric Wilcoxon test, as appropriate for the ordinal level data, to measure for any significant difference between the pre-test scores achieved and post-test results. The total number of students passing both expired air respiration and external chest compressions, achieving 70% and more in both, was calculated. A Pearson’s co-efficient of correlation was calculated to measure any correlation between knowledge attainment and skill performance, with post-test results achieved by students in the south-west University (Site 2), being compared with scores achieved in BLS skills testing, for expired air respiration and external chest compression.
The focus group interviews were conducted within each institution, as were the individual lecturer interviews and verbatim transcripts were produced. To support this exploratory research, data analysis followed Miles and Huberman’s (1994) definition: data reduction, data display and conclusion drawing, or verification. To (re)present the data as true to the experiences of the students’ and lecturers’ at the time of the interview, the transcripts were independently analysed by the lead researcher and a second experienced researcher.
Themes were identified through a process of reduction and display. The initial interpretations corroborated in identifying four themes: student-centred resource, context of learning, skill development and technological support. The themes and copies of interpretations from the full transcripts were sent to the students and lecturers for verification.
3. Results and Discussion
3.1 Knowledge and skills testing results
The results achieved demonstrate that the students from Site 1 scored more highly than the students in Site 2A, in the pre-test, though the students in Site 2B group achieved the highest scores (See Figure 1). Although all groups have included some students who have achieved the maximum possible score of 500, correctly answering all questions in all five sections, the median for both Site 1 and Site 2B group is 400, whereas the median for Site 2A is lower at 350. The range of marks achieved suggests that there are more students scoring lower marks in Site 2A group, (range 75 to 500), whereas Site 1 have achieved between range 125 to 500 and Site 2B have achieved between range 225 to 500.
It is also interesting to note that learning has occurred across all three groups when comparing pre-and post-test results (see Figure 2). The pre-test results show a range of 75 to 500, with four students scoring lower, and the median as 380. The post-test results include a range of 225 to 500, with a median of 400, thus demonstrating an increase in scores across the three groups from the pre- to the post-test.
Looking more specifically at the results, knowledge attainment had significantly increased for students in Site 1 within the category breathing, where post-test scores had significantly improved following completion of the CD-ROM, (z = -2.04, p = 0.041). This was the only category where such results were seen. The students in Site 2 showed differing results. Site 2A group, undertaking BLS for the first time in the university, achieved statistically significant results for all the categories: airway (z = -2.16, p = 0.031), breathing (z = -5.85, p < 0.001), circulation ( z = -3.73, p < 0.001), safety (z = -2.65, p = 0.008), responsiveness (z = -3.65, p < 0.001). Site 2B group showed significant achievement in one category only, that of airway (z = -2.51, p = 0.029).
The comparison of results gained in expired air respiration and external chest compression testing by Site 2A and Site 2B, with post-test scores, failed to show any correlation. There was no correlation of any statistical significance between knowledge and skill attainment, Site 2A ( r = 0.23, p = 0.131) and Site 2B (r = -0.252, p = 1.738). Despite the lack of correlation Site 2B group achieved higher pass rates than Site 2A group. For Site 2A, a total of 24 of the 45 students (53%) had achieved 70% or more in both expired air respiration and external chest compression, whereas for Site 2B, 27 of the 43 (63%) had achieved a pass in both (See Figures 3 and 4).
3.2. Knowledge and skills testing discussion
In considering the pre-test results, Site1 and Site 2B achieved higher scores than Site 2A. It is possible that those students completing BLS for the second time (Site 2B), have been able to draw on existing knowledge in answering the pre-test. However, the reasons behind the high achievement of Site 1, when compared with Site 2A, can only be postulated. It is possible that the students from Site 1, have all previously undertaken BLS, though this is felt unlikely, as during the focus group interviews, many participants confirmed that they had used the CD-ROM to learn about BLS for the first time.
The more likely explanation for the results, is that Site 1 students had more opportunities for learning, before completing the pre-test. Within the focus group discussions, many students revealed that they had had some technical difficulties using the CD-ROMs, which often crashed during use. This necessitated restarting the CD-ROM and taking the pre-test having seen part, or all of, the CD-ROM. This may mean that the pre-test results that were finally submitted on completion of the CD-ROM, reflected answers based on knowledge gained through looking at the CD-ROM many times, and were not a true pre-test result, submitted before seeing any part of the CD-ROM. This identifies one of the problems of researching the CD-ROM, when the researcher is unclear in what circumstances the pre-and post-test are completed. Whilst the intention was to collect pre- and post-test results, submitted by individual students having looked at the CD-ROM once, it appears that this was not always the case. Students may use the resource within a group, or look at the CD-ROM several times before submitting test results.
Considering the results across the groups it is however evident that learning has occurred, with higher post-test results seen. Site 1 showed statistically significant improvement in one category, breathing (p = 0.041), with Site 2B also showing results of statistically significant improvement in the airway category (p = 0.029). In contrast Site 2A showed statistically significant results across all the categories: airway (p = 0.031), breathing (p < 0.001), circulation (p < 0.001), safety (p = 0.0008) and responsiveness (p = 0.029). This outcome compares favourably with previous research testing BLS learning following traditional classroom instruction, where significant improvement in theoretical knowledge has been seen (O’Donnell and Skinner 1993, Moule and Knight 1997, Moule 2000, Badger and Rawstorne 1998). Though these studies have drawn on small convenience samples, all have demonstrated a significant gain in knowledge following BLS instruction, a finding now replicated when using the CD-ROM.
Previous research has evaluated the use of computer-assisted learning in BLS instruction for lay persons, found no difference in learning when compared with traditional teaching methods (Edwards and Hannah 1985). Branas et al (2000) reported that a group of Spanish medical students using a CD-ROM learnt “Advanced Life Support” theory at least as well as those students attending a lecture. A computer-certified BLS training programme in the USA, enabled the objective evaluation of nurses’ skills, found the flexibility of the learning approach to be the greatest benefit, rather than noting any significant learning difference (Shehee 1989).
The results failed to identify any correlation between knowledge and skill attainment, Site 2A (p = 0.131), Site 2B (p = 1.738). This again is not an atypical result. Previous research that has compared knowledge attainment gained through traditional teaching methods, and skill attainment, also failed to show any correlation (O’Donnell and Skinner 1993, Bager and Rawstorne 1998, Moule and Knight 1997, Moule 2000). An extensive literature review of the effectiveness of BLS training (Jabbour et al 1996) concluded that knowledge improvement was evident in three out of eight studies, whereas skill improvement was seen in only one of nine. Thus, the acquisition of knowledge through the CD-ROM is no guarantee of skill acquisition, though the evidence demonstrates that those undertaking BLS practice and testing for the second time achieved higher scores, thus the frequency of BLS instruction may have had some effect on skill attainment.
Achievement of skill acquisition, measured as the achievement of 70% or more in both expired air respiration and external chest compression, was higher for Site 2B group (63%), than Site 2A group (53%). Previous research measuring skill attainment following traditional methods of delivery has shown that skills were significantly improved (p < 0.001) following minimal review at four months (Yakel 1989). The optimal period between training is unclear, as skills have been seen to deteriorate at two (Moser and Coleman 1992) and ten weeks (Broomfield 1996) and six months (Inwood 1996), with populations of trained nurses. Despite the lack of consistency seen in the deterioration of skills, it is suggested that the period between training should be no greater than six months (Dunn et al 1992), the period elapsing between training input for the Site 2B group. Thus, the higher skill attainment achieved by this group might reflect the effects of the six month re-training period. Skill acquisition achieved through previous testing and reinforcement of good practice, may have had an effect on performance, as identified in previous research where the importance of structured teaching, practice, feedback and testing has been recognised (Yakel 1989, Curry and Gass 1987, Morgan et al 1996, Davies and Gould 2000).
3.3. Experiences of using the CD-ROM
The focus group and individual lecturer interviews were analysed to reveal four main themes, verified by the interviewees as: student-centred resource, context of learning, skill development and technological support.
Student centred resource
Both students and lecturers highlighted the flexibility and usability of the CD-ROM. Students particularly referred to the benefits of using the CD-ROM within an environment of their choice and welcomed the opportunity to work at their own pace, at a time convenient to them. These identified benefits were further enhanced as the resource was seen as user friendly in its organisation and presentation.
The students also felt less intimidated when testing their understanding of BLS, than in the classroom setting, and felt more secure in exploring their own learning needs using the CD-ROM. Through interacting with the CD-ROM, they were able to identify their own learning through the pre- and post-test results.
The students were all very supportive of the CD-ROM as a learning resource, commenting favourably on it when drawing comparisons with textbooks and lectures.
There is little research reporting learner satisfaction in the use of multi-media technology (Woo et al 2000), though these findings seem to support the existing body of knowledge. The students welcomed the student-centered approach to learning offered by the CD-ROM, as previously reported by Green et al (2000) and Moule and Gilchrist (2001), and they have valued the opportunity to learn at home (Bradshaw and Dale 1999). Developing the CD-ROM has been a viable alternative to classroom delivery of BLS theory. It was seen positively by the students, who during the course of the discussions identified further areas of the curriculum that could be delivered using CD-ROMs or e-learning, particularly for teaching biological sciences and other psychomotor skills, such as the administration management of intra-venous infusions. The approach has empowered the students to take an active role in the learning process, taking control of their own learning and adopting adult learning principles (Knowles 1990). This approach also requires the instructor to trust the learner to take control over their learning, and requires the learner to take responsibility for learning. Both of these issues were raised as part of the interviews and are discuss within the following theme.
Context of learning
Both the lecturers and the students stated that the CD-ROM facilitated student-centred learning, with the student taking control over the learning situation. This seems to have had both benefits and drawbacks. Whilst one could suggest that students might always control what they learn in any given situation, lecturers within traditional teaching modes are still able to dictate content delivery through their interactions with the students. The INTaL production team decided on the content of the CD-ROM (see Moule et al 2001), empowering the student to take control of their own learning, within the confines of the resource. Both students and lecturers commented on the knowledge gained and the benefits of being able to develop skills during the practice sessions. However in some cases students viewed the CD-ROM merely as a test to get out of the way, and not as a learning resource.
Others were selective in their use of the tool, engaging with what they perceived as necessary content, and because of this some lecturers felt some of the students were inadequately prepared for the practice sessions.
A further group of students had not seen the CD-ROM prior to the practice sessions and one lecturer took the view that they should, and sent them away to look at it.
There seems to be some dichotomy between using the resource in a student-centred way and having some control over the context of learning, ensuring that students use the resource to learn what the lecturers see as essential knowledge for skill development. It is not clear why some students have viewed the resource in this way. It may be, as one student suggested, that the testing component of the CD-ROM was emphasised to the detriment of the learning resource. This may have been as a consequence of this research project, as the introduction of the research highlighted that pre- and post-test results would be collected by the researcher. It may also have been more problematic for these students as the resource was introduced to two of the groups (Site 1 and Site 2A) during Freshers Week, amongst a plethora of instruction, which may have caused them to focus on the need to complete a test using a CD-ROM. It is apparent that students need to be motivated to use the resource as a learning aid, not merely a testing tool, which will require appropriate introduction.
There are two main issues about skills development. Firstly developing BLS skills seems to have been easier after the resource has been used. Students claimed that this was because of the benefit of being able to have visual images on the CD-ROM and being able to draw on those images in the classroom to aid practice and skills testing. These findings would seem to corroborate with Darr (2000), who found benefits in using scenario based, self-directed delivery in developing advanced life support skills.
Previous research has demonstrated that images can bring back the memories of specific information, and images of actions are more easily remembered, as we retain 50% of what we see and hear (Boyer 1999). These results are corroborated through the experiences of the students, who have felt they have learnt about BLS through using the multimedia available on the CD ROM; a feeling that has been supported through the pre- and post-test results achieved.
Secondly, the students were adamant that the practice session was essential. They felt the CD-ROM was a good way of preparing them for the practice session, but wanted to ensure that the practice session was provided and not replaced by the CD-ROM. Whilst viewing demonstrations of BLS has been shown to enable some skill development without practice, manikin practice has proved necessary for best skill acquisition (Eisenburger and Safar 1999), and will be maintained within both universities to support competency testing.
When using a CD-ROM there is a reliance on the performance of the technology and an assumption that students will, with minimal written instruction, be able to use the resource. The students in Site 1 did appear to experience greater problems with the use of the CD-ROM.
These problems may have had an effect on the pre-test results for this group. It not absolutely clear why Site 1 experienced more problems, though it may have been due to the quality with the original discs, which have subsequently been replaced.
Some students also talked with trepidation about the thought of using the CD-ROM, an issue identified by lecturers.
There was recognition that some IT skills were needed to facilitate the use of the CD-ROM and that the level of skills students had was variable. Though both institutions offered drop-in IT skills support sessions, these were little used by the students who tended to rely on existing skills. Some were struggling to cope with using the learning resource, relying on friends or family for support. This may suggest that the universities need to reconsider their provision of IT training, and foster approaches that encourage students IT skill development.
The students are not alone in voicing concern about the use of technology. Studies within nurse education have shown that student’s lack confidence in their use of computers (Wilson 1991, Martyr 1988), this is particularly seen among more mature students (Schwirian et al 1989, van Dover and Boblin 1991). Dewhurst et al (2000) also found that students were initially fearful of using computer technology, but that reservations were dispelled through exposure and use; though there were a minority for whom technology continued to hold serious fears. This was corroborated in the interviewers.
A limitation of the research is that it is clear in the analysis of pre- and post-test results and through the focus group interviews, that students had not always submitted results reflecting their first attempt at using the CD-ROM. In addition, some may have completed the tests without considering the content of the CD-ROM. The results may also be affected as students self-selected themselves into the skills testing and focus group interviews, and as the rationale for self-selection is not transparent, it is not clear how this approach to sampling may have affected the results. Despite these weaknesses, the BLS CD-ROM does appear to have supported a student-centred approach to learning across the groups of students. Though correlation between knowledge attainment and skill performance was not evident, students and lecturers did note a readiness for skill development, with students remembering and applying visual images from the CD-ROM to skills practice and testing situations. The CD-ROM was well received by students and lecturers from both universities and is viewed as a viable resource for delivering BLS knowledge, as a pre-requisite for skill practice and testing. The research has also identified the importance of ensuring that learning resources can be delivered efficiently and used effectively by students, who need to be confident in their IT skills. It has identified additionally, further development opportunities for Higher Education, with consistent requests across the groups for similar resources to be made available in the biological sciences, and for a range of psychomotor skills.
Students valued the flexibility and usability of the resource, particularly the self-testing and the presentation of visual images, which helped students to identify with the realities of BLS situations. In using the CD-ROM students suggested that they have been able to meet individual learning needs. They felt a sense of control over the timing and pace of their own learning, and for some the CD-ROM created a greater sense of self-achievement than seems to have been generated through lecture/seminar sessions or through learning with written materials. Students did not identify any limitations to their learning. The lecturers commented that the students had control of their own learning, noting differences in skills sessions amongst those students who have fully engaged in learning from the CD-ROM and those who have used it to complete the pre-and post-tests and little more. This raised issues for lecturers supporting the development of practice skills who expected students to be safe in practice and testing, having acquired a reasonable knowledge level prior to the session. This suggests a dichotomy between using the resource in a student-centered way, and the need to have some control over the students’ context of learning, ensuring students use the resource to obtain necessary knowledge for practice. Though it might be argued that students will always ultimately determine their own learning whatever the teaching medium, the use of the CD-ROM raises questions about how such resources might be employed to support student learning to a degree that is seen as safe to lecturers. Even though the lecturers have ensured essential content is available on the CD-ROM, they cannot dictate what content the student considers. Using pre- and post-tests and competency skills testing to check learning, also failed to ensure that all students viewed the CD-ROM, and seems to have led some students to concentrate purely on the outcome measures, rather than using the resource to support a process of learning. As more e-learning materials are developed for student use, it will be important to ensure students view them as learning materials, and essential course content, not merely as tasks to be completed.
Copyright by the International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the authors of the articles you wish to copy or firstname.lastname@example.org.