Evaluating the usability of Web-based learning tools
M.-A. Storey, B. Phillips, M. Maczewski and M. Wang
Web-based learning tools provide integrated environments of various technologies to support diverse educators’ and learners’ needs via the Internet. The goal of these tools is to enhance face-to-face instruction and to deliver distance-learning courses. Each of these tools offers similar components, such as course note posting, assignment submissions, quizzes and communication features. The primary motivation for developing these tools is to make it easier for instructors who have little knowledge of HTML and web navigation to put course materials on the web. However, the simplicity of use for novice users has significant drawbacks. For example, these systems force instructors and course administrators to use predetermined navigation models and course formats. These constraints may have a negative impact on their flexibility and usability for administrators, teachers and students. There have been several studies and frameworks developed for analyzing these tools from pedagogical and institutional perspectives (Britain and Liber, 1998; Pantel, 1997). Frameworks such as these provide guidance on what factors to consider and how they may be applied when educators and administrators are considering deploying web-based learning tools. However, they provide no information about how easy or difficult the tools are to use. Very few case studies have been conducted to identify potential usability issues with these tools. One notable exception is an examination of WebCT from a student’s perspective using questionnaires (Morss and Fleming, 1998). In general, they found that the use of WebCT did not place undue burdens on the students in terms of learning to use the tool. There were, however, a significant number who did find the tool difficult to use. This study, while interesting, does not explain why students found the systems hard or easy to use.
Even though several formal and informal case studies have been conducted, none of them have paid attention on both the usability of the tools and the impact on the potential users: students, instructors and course administrators. Software usability is not a new concept. It is usually referred to as the degree to which computer software assists a user in completing a task (Levi and Conrad, 2000).The concept of usability encompasses such attributes as learnability, efficiency, memorability, handling of user errors, and user satisfaction (Nielson, 1993). Various methods can be employed to examine the usability of computer software, such as observation, interviews, questionnaires and expert reviews (Shneiderman, 1998). The results of usability studies are typically incorporated into several stages of the software life cycle, from early analysis and design through final testing and follow-up studies. For software engineers’ perspective, usability concerns center on the user interface and the degree to which it meets various usability heuristics (Nielson, 1993). A user interface designed on sound principles will exhibit natural dialogue between system and user that is clearly expressed in nontechnical terms(Shneiderman, 1998). In addition, features of the interface will be consistent, and the memory load on the user will be minimized (Norman and Draper, 1986).User errors are prevented by good design, but when they occur, error messages follow the guidelines for dialogue mentioned previously. And finally, the system keeps users informed about what is happening through appropriate feedback. When the software is designed for educational use, additional principles become important, such as the design of learning activities and the learner's ability to control sequence, pacing, presentation medium, and level of difficulty (Hannafin, 1989).
Although the emphasis on software usability has grown in the past fifteen years since software designers and developers attempted to incorporate principles of human-computer interaction into their work, some designers have suggested that concerns for usability are not truly integrated into the design and development of educational software (Pavlik, 2000; Levi and Conrad, 2000).To better understand educational software, more usability studies are needed. To address the lack of controlled and situated studies on how the constraints imposed by web-based learning tools affect their usability, we conducted a study to compare two commercially available learning tools. In particular, we wanted to know:
There are many usability considerations one must attend to when designing user interfaces (Norman and Draper, 1986). First, the important elements of a page must be visible. The content of web pages should be formatted and displayed such that users can easily see or access the important elements and navigational aids. Second, a user interface must also provide appropriate feedback to users. For every action a user performs, the system should provide some feedback allowing the user to evaluate the effect(s) of their action. Third, the system should be consistent. Consistency can take many forms, such as consistent sequences of actions, consistent labeling of links and buttons, and a consistent navigation format.
In our study we noted that most of these usability principles were violated by the tools we evaluated and negatively impacted students and their attitudes towards these tools. The rest of this paper presents a selection of the empirical results from our study and the results from a thematic analysis of students’ open-ended comments to our questionnaires.
Fifty-four students enrolled in the spring 1999 section of Human-Computer Interaction at the University of Victoria volunteered to participate in the study and were randomly assigned to two groups of 27. Most of the participants were third and fourth year students in Computer Science. Instructor and administrators overlapped in functions in this study. One university professor, one system administrator and three research assistants participated in recording their experiences with these tools.
Web-based learning tools
We decided to consider tools that would be suitable for diverse courses campus-wide. Accordingly, the web-based learning tools had to provide facilities for course news and announcements, an area to store course notes and related information, synchronous and asynchronous communication tools such as email, bulletin board and chat facilities as well as features for creating and administering online quizzes and assignment submission. Given these criteria, we chose Blackboard 6.0 (updated to 7.0 and 8.0) (Blackboard, 2000) and WebCT 2.0 (WebCT, 2000) for this study.
At the start of the semester each group used the instructors’ generic course web site for four weeks. This web site was constructed using HTML editor tools and provided students with access to course announcements and course material. It had no specialized features central to web-based learning tools such as assignment submission, quizzes or communication tools. After the first four-week period in the term each group used both Blackboard and WebCT, in counterbalanced order, for four weeks. The students were then permitted to pick the site of their choice (WebCT, Blackboard or the generic course site) for the last three weeks of term.
The students completed a series of questionnaires over the course of the term. The first questionnaire (Q1) was administered after the first 2 weeks of using Blackboard/WebCT. A second questionnaire (Q2) was given at the end of the 4-week period with the assigned tool. Q1 and Q2 were completed again for the next tool assigned. A final questionnaire (Q3) was given at the end of term.
The questionnaires had three types of questions: multiple choice questions, scaled-answer questions and open-ended questions. Q1 was designed to gather information concerning the ease of learning and ease of use. Q2 was designed to gather more information about the students' learning curve for the tool and assessment of the tool by the individual features they used. Q2 also had open-ended questions concerning students' assessment of the tool (e.g., 'Which features of Blackboard/WebCT did you consider useful? Why?').
The goal of the final questionnaire, Q3, was to compare the two tools to each other and to the generic course website. There were also several questions in Q3 concerning students' recommendations for using web-based learning tools as a teaching and learning aid (e.g., 'How could the University best enhance your learning through the use of a web-based learning tool?').
All of the questionnaires were administered as online forms. For each question reported in this paper there were 7 possible answers. These answers ranged from rating a tool as being “extremely convenient/easy to use” to “extremely inconvenient/hard to use”. The responses were then converted to numbers such that low numbers reflect positive ratings of a tool. Further quantitative and qualitative data was collected from the experimenters who kept an informal log. These logs were used to gather information about installation issues and difficulties arising from tool use from both the instructors' and course administrators' perspectives.
We found a significant difference between Blackboard (M=2.98) and WebCT (M=4.04) for the participants’ rating for how easy the tools were to navigate. Blackboard was rated as being significantly easier to navigate than WebCT, t(94)=4.16, p=.00.
Our deployment of Blackboard and WebCT used different navigation formats. When students logged onto the Blackboard course web site they were presented with an Announcements page containing the course news. Along the left hand side of the screen was a persistent frame containing buttons for navigating through the site (see Figure 1). These buttons remained on the users’ screen from page to page giving students a consistent navigation tool. Also, Blackboard formatted course content such that little scrolling was required to locate information. The student’s preference for this navigation format is reflected in their open-ended comments. For example:
WebCT, on the other hand, did not provide a consistent navigation format. When students logged onto the site they were presented with a list of links (see Figure 2). These links, when clicked, would open up a new browser window with the new page inside. The pop-up windows had their own set of navigation buttons that replicated, to some extent, the navigation tools provided by the browser itself. The students experienced many difficulties navigating between the pop-up windows and the main window that contained links to other sections in the site. Also, the course content was displayed such that students had to scroll through pages to locate information.When commenting on the ease of navigation for WebCT students wrote:
Navigation of WebCT has been a nightmare, at least three different interfaces used within the tool (e.g. close, back, forward buttons; my WebCT; communication tools - bulletin board, course notes, etc....) made it very confusing.
Both tools provide facilities for preparing quizzes for the students. The quizzes were administered as study aids in the course. Students could write a quiz multiple times and receive feedback on correct and incorrect answers.
There were no significant differences found between Blackboard (M=2.31) and WebCT (M=2.66) for participants’ ratings of ease of use of the online quiz, t (63) = 1.41, p=.16 and their rating of effectiveness of helping them study the course materials, Blackboard (M=1.93) and WebCT (M=1.95), t (85) = 0.059, p=.95. On average participants felt the online quiz facility was easy to use and effective in helping them study.
Both WebCT and Blackboard had facilities for online assignment submission. There was a significant difference found between Blackboard (M=2.91) and WebCT (M=4.43) for the participants’ rating of ease of use of the Assignment Submission tool. Blackboard was rated as significantly easier for submitting assignments than WebCT, t(90)=5.64, p=.00. The main difference between the tools, and the one most commented on by students, was that Blackboard provided appropriate feedback to students when the assignments had been successfully submitted while WebCT did not.
We arbitrarily split the participating students into two research groups which did not intentionally reflect the small groups that worked together within the class. This ensured that students opting out of the study (3 in total) would not be disadvantaged in any way. Therefore students did not use the communication options, such as chat and whiteboards within Blackboard and WebCT.
Students’ overall assessment of Web-based learning tools
On the final questionnaire students provided responses to open-ended questions asking them for general comments on the use of web-based learning tools in a university course. Most students had positive attitudes towards using a well-designed tool.
An analysis of all student comments suggested that they favoured using web-based tools primarily because they were:
Aspects that related to the actual tool design and the content it conveyed mediated the ‘convenience factors’ that supported students studying. Students’ commented that the tools were positive to use, when they were:
Overall, Blackboard was also considered to be easier to use than WebCT by instructors and administrators. Brief assessments of the tools are reported in the following subsections. More detailed results are available in is discussed elsewhere (Wang, 2000).
Instructors and administrators found it easier to navigate in Blackboard than WebCT. Figure 3 shows the screen shot of the Control Panel in Blackboard.
Most labels used for the various components in the Blackboard Control Panel were clear, so it was easy to find the information needed. In addition, most of the components in the Control Panel provided a consistent user interface for navigation. The Control Panel in WebCT caused confusion. The labels used for the various components were ambiguous. Figure 4 shows the screen shot of the Control Panel in WebCT.
The bottom frame in Figure 4 was the main Control Panel used by instructors. Two major problems were detected in the Control Panel. First, the labels of the buttons were not descriptive enough to provide clues as to what laid beneath. Second, there were too many modes in the Control Panel to remember. So much information was hidden behind the buttons and modes that multiple steps were necessary to perform a function and consequently, it was easy for the user to get lost. These administrative tasks did not get easier with practice as it was found difficult by all of the system administrators to remember the sequence of steps needed. Most administrative tasks, such as posting course contents and assignments, or making online quizzes, required fewer steps in Blackboard than WebCT.
There were two ways to create course content in Blackboard and WebCT. One of them uses the built-in editors of the tools to construct course content. The other way involves migrating existing course materials to the tools. Since the editors within both of the tools provided limited features and the content already existed for the generic website, existing content was migrated to the new sites. It was very easy to upload existing HTML and PDF formatted files into Blackboard. However, we experienced difficulties with this task in WebCT. When an HTML format file with internal links was uploaded, internal links were broken. When a PDF format file was uploaded, the file was damaged so that it could not be read later. In addition, the File Manager in WebCT required a minimum of seven steps to upload a file.
Blackboard and WebCT both provide facilities for creating online quizzes. Creating an online quiz proved simpler in both Blackboard and WebCT. However, some minor difficulties and inflexibilities were experienced with both tools.
The user interface in WebCT was more customizable than in Blackboard. Instructors were able to add elements such as "News", "Interesting Links" or "Supplementary Readings". Blackboard did not provide a customizable user interface. Content had to be placed within the preexisting eight categories present in a frame on the left side of the screen. When migrating course materials from the generic website into existing categories this caused confusion, as students experienced difficulties finding the materials under the new category labels.
Registering students in the course within Blackboard was straightforward and batch registering of students is provided. Administrators were also able to batch register students in WebCT, but they had to assign a unique "Global ID" for each registered student which caused confusion and did not seem essential to our purposes.
Throughout this study we were interested in addressing five questions, three of them regarding students’, instructors’ and administrators’ assessments of tool usability. Sections 3 and 4 of this paper address these first three questions. In this section we address the last two questions: we report on students’ perceptions of whether the tools impacted learning positively or negatively and their main recommendations to universities who are considering campus-wide adoption of web-based learning tools. Finally we provide some recommendations for universities from the perspective of instructors and administrators.
Impact on learning
We did not directly measure whether the tools had a negative or positive impact on students’ learning. Indeed, we strove throughout the study to ensure that students could always access the materials and had alternative means to submit assignments, write quizzes and so on. We only intervened occasionally, but felt we had an ethical obligation to ensure that the students’ learning and grades were not negatively affected.
We did, however, solicit feedback on their perception of how these tools affected their learning. Many of the students’ did feel that having access to web-based materials could be positive, as one student wrote:
However, students’ written responses in the questionnaires suggest that when tools are hard to navigate (and frustrate the user), they not only have a neutral but negative effect on learning:
Recommendations for universities by students
In summary, students made the following recommendations for university-wide implementation of web-course tools:
The students’ comments indicate that even a usable tool needs to be located within a specific context. In the following quote a student mentions how hardware and software compatibilities, lack of technical support, financial feasibility and personal relationships are factors influencing tool use:
A further implication of tool use arose in the study debriefing session; there is a tension between the need for tool standardization and individual flexibility in tool use. Students wanted to be able to choose which tool to use, but at the same time frequently asked for the same tool to be implemented university and even nation wide.
When research team members presented initial results at an educational technology conference (Storey, Bayelas, Wang and Phillips, 2000), representatives from both WebCt and Blackboard stated that their tools were moving toward a ‘plug and play’ design that would enable instructors to easily assemble only those features of the tool that they considered relevant for course instruction and content. This is one way of addressing the need for individual adaptation for those inclined to do so, while maintaining the convenience of only learning one tool for all courses.
Lastly, it is important to note that the usability issues that arose in our study stemmed from two sources: (1) functionality and usability of the tool itself and (2) how the instructor used the tool to build the course. For example, WebCT provided the option of having new pages open in a pop-up window or in the main browser. In other words, it was a choice by the course administrator to select this navigation format. Therefore, an important element in ensuring accessibility is the existence of training and support for administrators, instructors and students using the web-based learning tools.
Recommendations for universities by instructors/administrators
During this study, we played the role of both instructors and administrators. The insights we gained lead us to make the following suggestions to universities considering deploying web-based learning tools:
The rapid expansion of the Internet and increasing software capabilities are influencing the dynamics of teaching and learning on many different levels. Web-based learning tools are constantly being re-designed by the developers to improve their effectiveness. Both WebCT and Blackboard have newer versions of their course tools than the ones used for our study. As the results of this study have illustrated, the usefulness and effectiveness of the tool is contextual, depending on many different factors including the design of the tool itself. Feedback from ‘real’ users, such as students, is important to provide input into further tool improvement. Unfortunately, users of these tools in educational institutions are rarely included in this process.
Pragmatically, however, it is important to recognize that there are many obstacles to conducting such studies. For example, one of the problems with evaluating available web-based learning tools is that relatively few are currently being used in academia. Evaluating tools under development in a classroom setting has ethical implications which have to be considered by the researchers (Storey, Phillips and Maczewski, 2001). Consequently, data and feedback from classroom-based users is difficult to collect. In our study, we evaluated the tools using computer science students with a background in user interface design. Obviously, the results from our study may have been different if we had used other groups of students. We purposely chose these students as they are already familiar with technology and hopefully would not be unduly burdened by using these tools. However, the fact that they experienced difficulties further increases the likelihood that students from a non-technical domain would also have experienced difficulties.
Nonetheless, the findings of this user study suggest that web-based course supplements are widely accepted, and at times even expected, as being part of students’ experiences in education. If the tools are not professionally developed, implemented, maintained and administrated, the positive support for learning can be reversed. The tools themselves can have an impact on students’ perceived learning in positive or negative ways. Training, questioning of implementation processes as well as reflection and usability studies on how, when and why web-based tools are used is essential to ensure that chosen tools achieve what they are designed to accomplish: to positively enhance students learning and instructors teaching. Our study did not address how the tools affected learning in the context of offline activities – in fact we strove to ensure that overall students’ learning was not negatively impacted so we could not measure this impact.
From the students’ assessment of the individual tool features and the tools in general, their view of web-based learning tools in educational institutions could be summed up in the following quote:
We would like to thank the students that participated in this study, without their cooperation this study would not have been possible. In addition, we are very grateful to Shohreh Hadian for assisting with the study and administration of the tools. Dr. Janet Bavelas from the Dept. of Psychology gave us invaluable advice on our research methods and study design. This research was supported by a fellowship from the IBM, Pacific Development Center in Vancouver, B.C. and also by an Innovative Teaching Grant from the University of Victoria.
Copyright by the International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the authors of the articles you wish to copy or firstname.lastname@example.org.