Developing and Evaluating a University-wide Online Distributed Learning System: The Experience at RMIT University
IT Alignment Program
RMIT University (Royal Melbourne Institute of Technology)
RMIT University is, like all universities in Australia at present, engaged in a rapid process of change, where terms like ‘niche market’ and ‘productivity’ jostle alongside concerns about ‘generic graduate attributes’ and ‘professional competence’. Quantity and quality are both important considerations in the universities of the 21st century as they seek to maintain important intellectual and physical spaces for their staff to pursue creative research and development, while at the same time needing to provide teaching for escalating numbers of students in all courses in order to shore up funding. These student cohorts have become increasingly diverse (McInnis, James & McNaught, 1995) with more part-time students, and students from a greater variety of backgrounds.
Flexible modes of delivery have been widely viewed as the prime way of meeting the challenges posed by this diversity. There has been a fair amount of naive equating of flexible delivery with production of online materials (‘Plug them into the web’) and insufficient attention to the relationship between flexible modes of operation for students, the use of communication and information technologies, and the design of educationally sound learning environments (Kennedy & McNaught, 1997; Reeves & Reeves, 1997). This is true of all levels in the system. There is pressure on universities to become more ‘efficient’, often to the exclusion of educational effectiveness, and this has translated in too many cases to the placing of text-based materials on the web and a reduction in face-to-face teaching. However, there is no doubt that communication and information technologies will be a major part of future university planning, as several recent reports make clear (e.g. Yetton et al., 1997).
We have conceptualised the process as follows (figure 1):
RMIT university policy
There are two key policy documents which are currently guiding the direction RMIT takes for the next three to five years. The first is the Teaching and Learning Strategy (T&LS).
RMIT Teaching and Learning Strategy
The RMIT Teaching and Learning Strategy aims to provide a student-centred learning environment where:
The latest version provides goals and plans for the period 1998-2000. The documents list six goals, each of which have clearly articulated operational priorities, sub-strategies and performance information/ indicators. A couple of examples are given in table 1.
These should not be seen as empty policy statements. There are resources allocated to implement the T&LS both in human and financial terms. Each faculty has a Director of Teaching Quality position established by secondment of an academic staff member from within the faculty. Over the last couple of years each faculty has received $2-300,000 from the University’s Strategic Investment Fund for curriculum and courseware development, and for staff development. An interactive web site is being developed to support this suite of innovative projects. (RMIT Clearinghouse for Teaching and Learning Projects)
RMIT University has developed this comprehensive Teaching and Learning Strategy to enable and ensure that the University is able to provide students with an effective and efficient learning environment. The direct catalyst for adopting a systematic and corporate approach to the development of such a strategy is the financial pressure on tertiary education due to the change in government attitude following the advent of the Unified National System and the consequent emphasis on ‘user pays’ (Higher Education Contribution Scheme and up-front fees). This has caused the University to seek alternative income streams through increased enrolments of fee-paying international students and simultaneously to reduce costs which are dominated by staff salaries.
While economics may have crystallised the need for a strategy and provided the political critical mass, the general dissatisfaction with the lack of a coherent educational framework based on the needs of students as learners generated the momentum to formulate the strategy and to implement it systematically across the whole University. Consequently the strategy subjugates the needs for efficiency to the imperative of learning effectiveness and relies on the impact of a cultural change to a student-centred approach to learning to generate a collateral efficiency dividend.
Table 1. Excerpts from the RMIT Teaching and Learning Strategy
The need to harness the potential of information technology as a means of implementing this strategy has been evident since the inception of the Teaching and Learning Strategy in 1995 and through its refinement and reaffirmation in 1997. The means of using technology to achieve the University’s objectives are discussed below.
RMIT IT Alignment Program
RMIT University established a project team in 1998 to develop an Information Technology Strategy designed to facilitate the implementation of the objectives of the Teaching and Learning Strategy in respect of electronically mediated flexible learning environments. The Information Technology Alignment Project (ITAP) report (1998) delivered by the team in June 1998 and adopted by the University forms the basis for a $50 million investment by RMIT over the next three years.
The report comprises four elements:
The ITAP report has allowed the University to properly articulate its objectives with respect to the use of IT in teaching and learning by identifying that IT will be used to enrich the learning environment by augmenting traditional methods rather than displacing them. The emphasis is on interactivity and time/space independence and flexibility. The position adopted by RMIT recognises that the delivery of content by electronic photocopying and sophisticated multimedia is not the primary objective. RMIT is mandating corporate standards compliant with the IMS to mitigate the risk inherent in such a large investment.
Enacting the RMIT Teaching and Learning Strategy through the IT Alignment Program: Designing the RMIT Distributed Learning System
We have to deliver on our promise that we can provide a flexible set of tools that will enable staff who are not technological whizz kids to develop pedagogically sound, interesting, and relevant online courses in an efficient and well administered way. Quite a task! How have we designed our Distributed Learning System (DLS)? Here are some of our principles:
Twenty online learning system products were considered, based on published information available via vendor web sites, and a detailed investigation of 13 of them followed.
We chose to go down the path of multiple tools to provide better functionality, so we are not using one product but a set of tools. These are largely commercial tools but we are also using a few tools developed in-house over the last couple of years. We have chosen the following as a benchmark set but we are aware that these tools will change constantly.
Eventually this will seamlessly link with the University’s new Student Management System which is currently being commissioned. We are aiming to have a seamless online environment where students can enrol via the web with student details going directly into the Student Management System. These details can be accessed by teachers in putting online learning environments together. Assessment data held in the Learning Management System can be sent back to the Student Management System.
The semester 1 Benchmark process for the new Distributed Learning System
We are leaping in with gusto with this corporate initiative and had 37 online projects using our new Distributed Learning System set of software tools running in the first semester of 1999. These projects are in all seven faculties of the University across both the VET (Vocational and Educational Training) and higher education sectors of the University. We had 40 staff and 717 students involved in this testing process. In many cases the DLS was used for part of the semester’s work as staff tried out the tools and explored educational ideas.
Subject areas included: futures and options in business; superannuation & retirement planning; research methods for business; introduction to organisational behaviour; professional skills for ESL students; signals & systems in engineering; map work in geography; computer programming; non-western architectural history; design economics and cost planning; cinema, gender & popular culture; dress, role & status; design, art & society; online multimedia modules; and tertiary teaching professional development.
Each faculty has a Director of Information Technology (DoIT) appointed to work with the Director of Teaching Quality (DoTQ) (mentioned earlier) and a Faculty Educational Services Group to provide technical and design support. Few of the staff involved in this benchmarking exercise have much experience with using technology in teaching, and in particular online teaching and learning. The DLS team is working with the faculty groups to provide group and one-on-one training and support for each project.
At the beginning of semester 1, DLS support staff spent some hours with the teachers in each Benchmark subject discussing educational design issues in online learning environments, and key resource and support needs. A questionnaire was used to support this process. The data collected during these preliminary discussions was collated into a spreadsheet for the DLS team to access. A series of information sessions were held for department heads, DoTQs, DoITs, and general staff. Discussions were also held with student representatives to inform students of the developments within the DLS. The website for Online @ RMIT was developed as a major communication tool and as the gateway to the DLS. The services to be offered by the DLS team were clarified and the processes and procedures to establish and run the DLS were developed.
Mechanisms and forums were set up to enable staff to provide regular and on-going feedback. These included: establishment of a developers’ forum; creation of a threaded discussion area for Benchmark staff; an online feedback form for Benchmark staff; an online feedback and help form for other users of the DLS and the website; and monthly ‘Whine & Cheese’ sessions. The DLS team also established a help and support service and developed a set of guidelines and standards for issues related to course and subjects using the DLS, e.g. web publishing standards, intellectual property guidelines, etc.
The development of these technical support documents has occurred alongside the development of course and subject revision and improvement for flexible learning guidelines in each faculty. In this way it is hoped to align educational review and planning with the development of tools and support processes so as to achieve educational goals.
Process of evaluating the RMIT Distributed Learning System
Educationally, we are explaining the functionality of each of the tools in terms of student learning activities. Table 2 matches some student learning needs with examples of the design of suitable student activities with components of the DLS toolset. Several of the tools could be used for most of the activities; examples are used for simplicity.
Table 2. Functions of the DLS toolset
A learning-centred evaluation is being attempted. In order to set up a base line for the teachers’ reflections, teachers in each DLS Benchmark project were asked to articulate the student learning outcomes for their subject and where they thought the online experience would enhance learning. We then asked all teachers to submit a weekly journal entry via an online feedback form to continue this process of reflection. It was quite difficult to extract this stream of continuous reflection from teachers (staff workload is an issue that will be discussed as a major problem), but several teachers did give us formal and informal feedback from time to time. As much of the informal anecdotal feedback was in email messages, the substance could be captured. Twenty staff provided substantial feedback.
We also planned to use the usual evaluation strategies with students, such as online questionnaires, focus groups, analysis of web access data, analysis of support/ help desk records, and analysis of performance on learning outcomes. At this stage we have little student performance data from semester 1 and expect to receive only anecdotal evidence of learning enhancement (or otherwise) from this testing phase.
The feedback data collected from staff was collated and stored in a spreadsheet. Data received by the DLS Help Desk was also included; this Help Desk data was from staff and students. The data was then coded using an iterative method whereby each discrete statement was assigned a descriptive category; these were then reviewed with some reduction of the number of categories occurring. This process led to the identification of 39 categories under which the comments were grouped. The categories were grouped under seven headings. Table 3 contains a list of these codes. This coding was done in mid-May as part of an interim evaluation report.
Table 3. Categories of issues emerging from the DLS interim evaluation
We used these codes to look at all the data received in semester 1. This data included:
Each discrete statement was assigned to a category and the number of statements in each category was tallied. The tally of results for all feedback data received in semester 1 ranged from -14 to +13; this means that one issue received 14 negative comments and one issue received 13 positive comments. Table 4 shows those issues raised which had ratings (positive or negative) of 5 and above. There was little difference in the issues highlighted by staff and students.
Table 4. Benchmark issues with ratings of 5-11 (as at 6 July 1999)
An online form was made available to all Benchmark staff. It sought a rating from staff on the technical reliability of the DLS and on the service provided by the DLS team. It also sought comments on a number of areas such as problems encountered, lessons learnt, successes and general comments. As commented earlier, it was difficult to get weekly feedback from all staff. Table 5 is based on 34 occasional responses.
Table 5. DLS ratings feedback (as at 6 July 1999)
Lessons from the semester 1 DLS Benchmark process
Designing for an online environment requires new educational design skills.
Many staff are aware of and interested in the educational potential of the DLS toolset. They recognise that new methods of teaching are involved, such as how to get full participation from students in a threaded discussion, and this creates a challenge. Technical issues can lessen motivation, and this is particularly true of staff who are observers of the Benchmark process. Detailed case studies of actual educational strategies used are needed and we have started building this into the Resources section of our website.
This is a period where ongoing dialogue is essential in departments.
"During a meeting with other staff members in my department involved in online development, it became clear that some are being encouraged to put courses online without having a clear belief that it can improve the quality of the class. Their experience is that it is a second rate tool for education and they’ll just have to make the best of it and minimise the degradation of the course. I tried to communicate that online delivery has its own strengths and these should be maximised. Also that not all courses suit being fully online. I think it’s important that staff members who are involved in online development should try to experience online learning so that they can adapt their courses in a way that suits the different pedagogy and learning style."
The process of sharing reflection on these new strategies is important.
"Many students have been accessing the discussion area and reading the messages but have not been actively participating. I have tried to introduce a number of discussions of varying seriousness to try to encourage participation as I perceived that students may be intimidated and think they must contribute only high level learning. This problem may be exacerbated by the voluntary nature of student participation, with a fully online course there would be more incentive to become involved (i.e. assessment requirements, individual encouragement from tutor, students, etc).
The most difficult aspect of preparing a course for online delivery is in trying to understand the pedagogical changes that necessitate a restructuring of most aspects of the content. The learning experience is quite different and it’s necessary to adapt the course structure to capitalise on its strengths and minimise its weaknesses."
Many staff were keen on building some of the assessment into activities which use the DLS. A number of the Benchmark subjects took this approach.
"Not surprisingly, 77% of students commented that they tended to use only those parts of the DLS which related to assessment tasks."
For many staff there have been real rewards.
"The most rewarding aspects of my experiences with online delivery have been the ability to communicate one-to-one with all my students and the emphasis it places on student-centred learning. Interaction is primarily between students and asynchronous discussion groups are an excellent way for each participant to contribute ideas."
Students must have easy and fast access to online environments.
The student registration process in semester 1 was very fraught because of several technical issues. Individual rather than automated registrations had to be done. Some staff and students negotiated this hurdle well, but for others, this left a lasting negative impression. The difficulties of the registration process in semester 1 should not be underplayed. These must be avoided in the future. Suitable computer access and network reliability were also real difficulties which must be solved if online education is to be successful at RMIT.
"The difficulties with the registration process has hampered some students from accessing the site and discouraged others. As the usage of the online component of the class is voluntary, there has been minimal use so far, mainly by students with easy web access, prior online experience and enthusiasm for the process and the subject.
Several students have reported anything from mild dissatisfaction to absolute and total frustration with the process of accessing the material. Of the 70 enrolled I currently have about 54 logged in. I don’t know whether this indicates a problem with DLS, a problem with the students or both. Responses from students via the discussion board span between total frustration, to ‘loneliness’ and an apparent lack of direction when no-one is there to do the directing. I have also had some feedback which says this is great, an ideal way to handle this material."
Once students have access and are trained, they appreciate the flexibility.
The level of sophistication and interest in online learning was sometimes on the students’ side! Both staff and students recognised this.
"The students undertaking the course were eager to explore the potential of the technology. I was dragging my feet, as I desired some level of confidence before opening the window to the group.
I think the tutors need to be more familiar with threaded discussions, e.g. like the ones which go in newsgroups. Multiple topics should be started, where replies are in an informal manner. The tutor should throw around personal experiences and what they think about certain things."
The flexibility of access has been a positive aspect for staff and students.
"Greater flexibility for students in studying, which the feedback so far suggests to be true especially for students studying purely by this mode. They do however state that they still like attending some seminars and having actual contact with peers and their lecturer.
I was particularly pleased to receive an email from a student who had contracted glandular fever, but was able to keep up with the class from home because we were using the DLS. In fact, a survey I administered (on the DLS) indicated that 74% of students found the online quizzes useful, 76% found the online tutorials useful and 42% of students used the DLS from both home and RMIT."
One student summed up the positive aspect of the experience as:
First semester examination results are still being processed; it is unlikely we will see significant learning gains, but we are keen to monitor this over time.
Staff workload is a critical issue.
"That writing material for online education is very time consuming (it requires about one to two full-time day equivalents to write one complete online module).
Unable to get immediate support. Working from home after hours is a problem - But that’s the only time I have to do these added duties. Lack of time to develop course work, implement it and support students as well as learn about new technological application so that it can best serve the teaching learning experience - On top of that you are expected to meet the bureaucratic requirements.
I did spend significant extra time in delivering my subject using the DLS, but I’d like to encourage other staff to do so, by viewing it as a long term investment (with some short term risk) that will potentially reap rewards for staff and student alike."
A discussion board needs to be a conversation where people offer opinions about certain topics. A key to generating the discussion and keeping it going is the participation of staff. In some cases, staff were not using the board effectively. They were getting the technical assistant to add comments and post announcements. This implies that the tutors did not take an active role in the discussion and students felt the lack of interaction. A telling piece of feedback in relation to this came from the technical adviser who wrote:
"I found that I had to do the bulk of this work as the academics teaching the subject did not attend DLS training sessions. I set up tutorial groups, uploaded documents, downloaded student assignments, posted announcements, and posted discussion items to the discussion board (on behalf of the lecturer). Decided how we should use the Classroom (i.e. which sections to enable/disable)."
More local support for staff and students is needed.
The support given by the DLS Team has been appreciated. But more support for students and for staff, especially at faculty level, is needed.
"In the last week I have had to address student concerns regarding access to the pages/information and access to the system. I appreciate that it is not possible to have someone on hand all the time - but when students are panicking, work is due and the system doesn’t seem to be happening - it doesn’t help. I think that this does need to be improved, especially before we have even more students participating.
The support from the support team has been very good with queries being answered almost immediately when it comes to the system and student access. Students have reported back to me the same information.
I don’t have the feeling that the Faculty is fully supporting this activity. I know I can get a graphic designer to help me, but I don’t need face-lifting at the moment."
Support means both collegial and technical support.
"What you do require is time to play and explore, support from management and the collegiality of those who have ventured in this direction before you.
I have come to the conclusion that it is essential that there be adequate specialist IT assistance available to academics in the creation of online course materials."
We need more evidence about the performance of the DLS Toolset.
Technical issues have overwhelmed these initial impressions. ‘Wait and see’ is the predominant view at this stage.
"On the positive – I do think that CourseInfo has potential to be very effective. I like the way that it looks and the interface for students, I find it to be far more user friendly than what I was doing last year. From a building perspective, I will not make my call just yet. As we are really only at the beginning of the benchmark I’m sure that there will be many more good things ahead – especially now that we can get students registered!
At the same time, there were some 60 positive responses about ‘what was good about using Online @ RMIT; and not a single student was prepared to nominate anything that was ‘bad’ about using Online @ RMIT. I will be very interested to compare these result with the same questions at the end of the year.
Do not be too optimistic, too early, about the technical side to the process."
The relative ease of the tools released several staff to think about educational issues.
"I found CourseInfo very easy to use. The key question became one of how do I organise the subject? rather than how do I use the software?"
Synchronous chat was found to be a problem; our network is not yet reliable enough.
"You are no doubt aware that my tutorials on Thursday night and Friday afternoon did not run because the server was down. In despair and frustration, and because I am concerned about your learning, I have therefore abandoned the idea of online tutorials. We will continue with the threaded discussions and with the posting of lecture materials and exercises."
We have received valuable feedback about particular tools.
"I have learned that a system needs to be comprehensive if it is to be used. For example, CourseInfo provides a method to send mail but doesn’t provide a method to read mail. This has caused my students grief.
(1) Students took readily to the online test, anecdotal feedback was positive, though will be interested in their comments a week or so ex-post. (2) Students have expressed some reservations about the BSCW collaboratory environment (despite initial enthusiasm), and how I will assess their relative contributions to their group assignments - particularly with regard to how they will ‘impression manage’ their individual contributions. That said, after some spirited discussion in the classroom around this issue, several groups appeared to be applying their minds to the ‘problem’ and demonstrated some teamwork/ delegation/ negotiation skills. I think the point has been made, but I will watch their use of BSCW with interest."
The next phase ...
In semester 2 we will have 100 new subjects online using the RMIT Distributed Learning System. Learning Technology Mentors are being appointed in each department. There are approximately 70 Learning Technology Mentors (LTMs) who are academic staff who have one day a week time release to develop online materials and support their colleagues in their departments to engage with online teaching and learning. These LTMs undertake an extensive staff development program. They all develop a work contract with the Professional Development Team of the IT Alignment Program; if individual staff wish this can be formalised into accreditation for a subject in a Graduate Certificate of Flexible Learning. Another chapter begins.
We wish to sincerely thank: