Educational Technology & Society 2(4) 1999
ISSN 1436-4522

Developing and Evaluating a University-wide Online Distributed Learning System: The Experience at RMIT University

Carmel McNaught, John Kenny, Paul Kennedy & Ross Lord
IT Alignment Program
RMIT University (Royal Melbourne Institute of Technology)
Melbourne, Australia
carmel.mcnaught@rmit.edu.au
john.kenny@rmit.edu.au
paul.kennedy@rmit.edu.au
ross.lord@rmit.edu.au



ABSTRACT

This paper will explain the policies relating to teaching and learning at RMIT University. The need to harness the potential of communication and information technologies as a means of implementing the Teaching and Learning Strategy has been evident since its inception in 1995 and through its refinement and reaffirmation in 1997. There has been an increase in the use of CIT in individual projects over the last few years which has increased flexible access to RMIT courses in several ways. However, it has become increasingly clear that a sustained approach to developing flexible courses for both our on-campus and off-campus students required a university-wide approach. In 1998, the University embarked on a comprehensive and ambitious project to align information technology systems to the principles and goals of the Teaching and Learning Strategy. The components of this IT Alignment Project are described. One of the major components is the RMIT Distributed Learning System (DLS). The choice of the online tools that comprise the DLS is described. A benchmark project to evaluate the toolset, both educationally and technically, has been underway in semester 1 1999. A major focus of the paper will be the description of the evaluation design and the results and insights obtained in semester 1.

Keywords: Online learning system, Online toolset, University teaching & learning strategy, Evaluation



Introduction

RMIT University is, like all universities in Australia at present, engaged in a rapid process of change, where terms like ‘niche market’ and ‘productivity’ jostle alongside concerns about ‘generic graduate attributes’ and ‘professional competence’. Quantity and quality are both important considerations in the universities of the 21st century as they seek to maintain important intellectual and physical spaces for their staff to pursue creative research and development, while at the same time needing to provide teaching for escalating numbers of students in all courses in order to shore up funding. These student cohorts have become increasingly diverse (McInnis, James & McNaught, 1995) with more part-time students, and students from a greater variety of backgrounds.

Flexible modes of delivery have been widely viewed as the prime way of meeting the challenges posed by this diversity. There has been a fair amount of naive equating of flexible delivery with production of online materials (‘Plug them into the web’) and insufficient attention to the relationship between flexible modes of operation for students, the use of communication and information technologies, and the design of educationally sound learning environments (Kennedy & McNaught, 1997; Reeves & Reeves, 1997). This is true of all levels in the system. There is pressure on universities to become more ‘efficient’, often to the exclusion of educational effectiveness, and this has translated in too many cases to the placing of text-based materials on the web and a reduction in face-to-face teaching. However, there is no doubt that communication and information technologies will be a major part of future university planning, as several recent reports make clear (e.g. Yetton et al., 1997).

We have conceptualised the process as follows (figure 1):


Figure 1. Effective policy stages for the development of flexible learning environments


RMIT university policy

There are two key policy documents which are currently guiding the direction RMIT takes for the next three to five years. The first is the Teaching and Learning Strategy (T&LS).


RMIT Teaching and Learning Strategy

The RMIT Teaching and Learning Strategy aims to provide a student-centred learning environment where:

  • subjects and the courses they comprise are designed to develop the following graduate attributes in students: knowledgeable, critical, responsible, creative and with a capacity for life-long learning, leadership and employment and an international outlook;
  • the system is flexible enough to suit the particular learning needs of students in terms of their prior experience and current situation;
  • courses are designed and implemented holistically with coherent connections between subjects comprising the core of a course;
  • students and the community are seen as significant stakeholders;
  • assessment is directly related to the explicitly stated objectives of subjects; and
  • quality improvement and quality assurance based on reflective practice and customer focussed systems design are ubiquitous.

The latest version provides goals and plans for the period 1998-2000. The documents list six goals, each of which have clearly articulated operational priorities, sub-strategies and performance information/ indicators. A couple of examples are given in table 1.

These should not be seen as empty policy statements. There are resources allocated to implement the T&LS both in human and financial terms. Each faculty has a Director of Teaching Quality position established by secondment of an academic staff member from within the faculty. Over the last couple of years each faculty has received $2-300,000 from the University’s Strategic Investment Fund for curriculum and courseware development, and for staff development. An interactive web site is being developed to support this suite of innovative projects. (RMIT Clearinghouse for Teaching and Learning Projects)

RMIT University has developed this comprehensive Teaching and Learning Strategy to enable and ensure that the University is able to provide students with an effective and efficient learning environment. The direct catalyst for adopting a systematic and corporate approach to the development of such a strategy is the financial pressure on tertiary education due to the change in government attitude following the advent of the Unified National System and the consequent emphasis on ‘user pays’ (Higher Education Contribution Scheme and up-front fees). This has caused the University to seek alternative income streams through increased enrolments of fee-paying international students and simultaneously to reduce costs which are dominated by staff salaries.

While economics may have crystallised the need for a strategy and provided the political critical mass, the general dissatisfaction with the lack of a coherent educational framework based on the needs of students as learners generated the momentum to formulate the strategy and to implement it systematically across the whole University. Consequently the strategy subjugates the needs for efficiency to the imperative of learning effectiveness and relies on the impact of a cultural change to a student-centred approach to learning to generate a collateral efficiency dividend.

Goal

Operational priority

Sub-strategy

Performance information/ indicator

1. To graduate students of world class standing who demonstrate leadership by contributing creatively, critically and responsibly to their professions and vocations and to the community.

1.1 (one of three)

To design learning programs which attract and develop graduates who are knowledgeable, critical, creative and responsible, and therefore equipped for professional practice in a global, technological society.

A (one of two for 1.1)

Review of each course by Course Teams, industry and professions through the EQA (quality assurance) process to improve quality and market share. Guidelines for reviews to be developed by Teaching and Learning Strategy Committee.

ii (one of four for A)

All courses to be refurbished to include flexibly delivered courseware within 5 years: 60% by 2000.

2. To maximise learning for all students by creating student-centred environments in all subjects and courses.

2.2 (one of four)

To introduce cost-effective flexible learning modes using a range of educational technologies which expand students’ learning opportunities and encourage staff to become facilitators of learning.

E (one of three for 2.2)

Faculty plans to implement staff development programs to enable the introduction of student-centred, flexible learning environments.

viii (one of three for E)

Increased numbers of quality assured modules and subjects. Delivery modes fully evaluated and results disseminated.

Table 1. Excerpts from the RMIT Teaching and Learning Strategy

The need to harness the potential of information technology as a means of implementing this strategy has been evident since the inception of the Teaching and Learning Strategy in 1995 and through its refinement and reaffirmation in 1997. The means of using technology to achieve the University’s objectives are discussed below.


RMIT IT Alignment Program

RMIT University established a project team in 1998 to develop an Information Technology Strategy designed to facilitate the implementation of the objectives of the Teaching and Learning Strategy in respect of electronically mediated flexible learning environments. The Information Technology Alignment Project (ITAP) report (1998) delivered by the team in June 1998 and adopted by the University forms the basis for a $50 million investment by RMIT over the next three years.

The report comprises four elements:

  • IT infrastructure aligned with the needs of education to deliver the systems and hardware necessary to provide students with an electronically connected learning environment and access to computer based learning resources;
  • a Distributed Learning System (DLS) compliant with the emerging Educom/CAUSE Instructional Management System (IMS);
  • a Student Management System (SMS), fully integrated with the DLS to provide enrolment and subject and course progress records electronically accessible to academics and students; and
  • extensive staff development.

The ITAP report has allowed the University to properly articulate its objectives with respect to the use of IT in teaching and learning by identifying that IT will be used to enrich the learning environment by augmenting traditional methods rather than displacing them. The emphasis is on interactivity and time/space independence and flexibility. The position adopted by RMIT recognises that the delivery of content by electronic photocopying and sophisticated multimedia is not the primary objective. RMIT is mandating corporate standards compliant with the IMS to mitigate the risk inherent in such a large investment.


Enacting the RMIT Teaching and Learning Strategy through the IT Alignment Program: Designing the RMIT Distributed Learning System

We have to deliver on our promise that we can provide a flexible set of tools that will enable staff who are not technological whizz kids to develop pedagogically sound, interesting, and relevant online courses in an efficient and well administered way. Quite a task! How have we designed our Distributed Learning System (DLS)? Here are some of our principles:

  • a suite of tools, not just one;
  • integrating educational principles into the description of the toolset;
  • IMS compliance of all tools;
  • a team approach to all online projects; and
  • involvement of all seven faculties in a benchmarking exercise to evaluate the toolset and the effectiveness of the learning environments we are building.

The key technical selection criteria and priorities for the DLS toolset were:

  1. The DLS platform has a demonstrated commitment and plan to incorporate all the requirements and specifications related to the EduCause IMS Standard.
  2. The DLS platform is built on a non-proprietary, open and modular architecture that permits cost-effective re-engineering via the backend interface to add extra functionality and to substitute components.
  3. The DLS platform supports connectivity to external databases (in particular Oracle databases) and other web-based systems using the ODBC and JDBC protocols.
  4. The DLS platform’s datasets and coding standards, structure and language allows RMIT to contract the broadest range of providers of programming and software re-engineering services.
  5. The DLS platform must have the capacity to be interoperable with existing RMIT enterprise systems such as the existing and future student management system, Novell Directory Services (NDS), Groupwise and the Course Information Data Warehouse (CIDW).
  6. The DLS platform is Year 2000 compliant.
  7. Content is independent of the DLS platform.
  8. The DLS platform supports the assigning of courses to specific groups and individuals, and individuals and groups to courses/subjects.
  9. The DLS platform supports client access via Netscape v3/4 and Internet Explorer v3/4 browsers on Macintosh (System 8.0 or above) and Intel PCs (Windows95 or better).
  10. The DLS platform can manage (track) web-based learning materials as well as content available via third-party CBT applications distributed on CD-ROM, disk and local hard drives.
  11. The DLS platform allows for the customisation and addition of reporting functions.
  12. The associated software must be at least Version 1.0 and preferably beyond Version 1.0.
  13. The vendor must be able to verify that the software is currently operating in a real-world situation at an enterprise level e.g. in university settings and large corporations.
  14. The supplier of the DLS platform must be able to demonstrate that they can provide a high standard of local support either directly or via authorised dealers/software solution providers.
  15. DLS platform provides assessment, tracking and record-keeping functions that can be used for online learning and other on-campus activities.
  16. Documented and verifiable impartial third party technical performance and testing data is available for the DLS platform.
  17. Supports Priorities 1 to 3 under the Universal Access Standards prescribed by W3C.

Twenty online learning system products were considered, based on published information available via vendor web sites, and a detailed investigation of 13 of them followed.

We chose to go down the path of multiple tools to provide better functionality, so we are not using one product but a set of tools. These are largely commercial tools but we are also using a few tools developed in-house over the last couple of years. We have chosen the following as a benchmark set but we are aware that these tools will change constantly.

  • The Campus (Macromedia Pathware was the initial choice; now an in-house tool is in use). This provides a common gateway to the RMIT Benchmark DLS for all registered staff and students and acts as a Learning Management System for the compilation of student results in each subject.
  • The Classroom (CourseInfo). This is a secure area, accessible through Campus, where registered staff and students can access learning materials and facilities connected with a particular subject in which they are enrolled.
  • The Community. This is an area of the website designed to allow RMIT staff and students to be able set up communication and discussion areas. There is a conference area (CourseInfo can be used, but WebBoard is also being provided. Groupwise is the RMIT choice for normal email).
  • The Collaboratory. This is a tool designed to enable groups of students to work collaboratively on projects by sharing documentation, linking to threaded discussion areas and email facilities. (BSCW)
  • Critique quiz tools (Perception Question Mark; and WebLearn, an in-house tool).
  • Problem solving tools (AGILE ToolBook wizards, developed in-house; Kennedy, McNaught & Nicolettou, 1998).
  • Some other supplementary tools, including HelpReader for visually impaired students.

Eventually this will seamlessly link with the University’s new Student Management System which is currently being commissioned. We are aiming to have a seamless online environment where students can enrol via the web with student details going directly into the Student Management System. These details can be accessed by teachers in putting online learning environments together. Assessment data held in the Learning Management System can be sent back to the Student Management System.


The semester 1 Benchmark process for the new Distributed Learning System

We are leaping in with gusto with this corporate initiative and had 37 online projects using our new Distributed Learning System set of software tools running in the first semester of 1999. These projects are in all seven faculties of the University across both the VET (Vocational and Educational Training) and higher education sectors of the University. We had 40 staff and 717 students involved in this testing process. In many cases the DLS was used for part of the semester’s work as staff tried out the tools and explored educational ideas.

Subject areas included: futures and options in business; superannuation & retirement planning; research methods for business; introduction to organisational behaviour; professional skills for ESL students; signals & systems in engineering; map work in geography; computer programming; non-western architectural history; design economics and cost planning; cinema, gender & popular culture; dress, role & status; design, art & society; online multimedia modules; and tertiary teaching professional development.

Each faculty has a Director of Information Technology (DoIT) appointed to work with the Director of Teaching Quality (DoTQ) (mentioned earlier) and a Faculty Educational Services Group to provide technical and design support. Few of the staff involved in this benchmarking exercise have much experience with using technology in teaching, and in particular online teaching and learning. The DLS team is working with the faculty groups to provide group and one-on-one training and support for each project.

At the beginning of semester 1, DLS support staff spent some hours with the teachers in each Benchmark subject discussing educational design issues in online learning environments, and key resource and support needs. A questionnaire was used to support this process. The data collected during these preliminary discussions was collated into a spreadsheet for the DLS team to access. A series of information sessions were held for department heads, DoTQs, DoITs, and general staff. Discussions were also held with student representatives to inform students of the developments within the DLS. The website for Online @ RMIT was developed as a major communication tool and as the gateway to the DLS. The services to be offered by the DLS team were clarified and the processes and procedures to establish and run the DLS were developed.

Mechanisms and forums were set up to enable staff to provide regular and on-going feedback. These included: establishment of a developers’ forum; creation of a threaded discussion area for Benchmark staff; an online feedback form for Benchmark staff; an online feedback and help form for other users of the DLS and the website; and monthly ‘Whine & Cheese’ sessions. The DLS team also established a help and support service and developed a set of guidelines and standards for issues related to course and subjects using the DLS, e.g. web publishing standards, intellectual property guidelines, etc.

The development of these technical support documents has occurred alongside the development of course and subject revision and improvement for flexible learning guidelines in each faculty. In this way it is hoped to align educational review and planning with the development of tools and support processes so as to achieve educational goals.


Process of evaluating the RMIT Distributed Learning System

Educationally, we are explaining the functionality of each of the tools in terms of student learning activities. Table 2 matches some student learning needs with examples of the design of suitable student activities with components of the DLS toolset. Several of the tools could be used for most of the activities; examples are used for simplicity.

Student learning need

(Laurillard)

Examples of student activity

Example of current benchmark Distributed Learning System (DLS) toolset component

information handling skills

  • web searching;
  • using electronic Library databases
 

developing understanding

  • building links between information from various sources;
  • problem solving exercises

CourseInfo/ BSCW

 

 

Question Mark/ AGILE

linking theory to practice

  • working with embedded media and simulations in course material;
  • tutorial programs with feedback

CourseInfo

hybrid systems with CD-Roms

practising discussion and argument

  • online debates using a threaded discussion

WebBoard

practising articulation of ideas

  • role playing using a threaded discussion;
  • sharing essays online

WebBoard

 

BSCW

rehearsing skills and procedures

  • online quizzes with feedback

Perception Question Mark

WebLearn

practising teamwork

  • group projects

BSCW

learning professional practice

  • all of the above!
 

Table 2. Functions of the DLS toolset

A learning-centred evaluation is being attempted. In order to set up a base line for the teachers’ reflections, teachers in each DLS Benchmark project were asked to articulate the student learning outcomes for their subject and where they thought the online experience would enhance learning. We then asked all teachers to submit a weekly journal entry via an online feedback form to continue this process of reflection. It was quite difficult to extract this stream of continuous reflection from teachers (staff workload is an issue that will be discussed as a major problem), but several teachers did give us formal and informal feedback from time to time. As much of the informal anecdotal feedback was in email messages, the substance could be captured. Twenty staff provided substantial feedback.

We also planned to use the usual evaluation strategies with students, such as online questionnaires, focus groups, analysis of web access data, analysis of support/ help desk records, and analysis of performance on learning outcomes. At this stage we have little student performance data from semester 1 and expect to receive only anecdotal evidence of learning enhancement (or otherwise) from this testing phase.

The feedback data collected from staff was collated and stored in a spreadsheet. Data received by the DLS Help Desk was also included; this Help Desk data was from staff and students. The data was then coded using an iterative method whereby each discrete statement was assigned a descriptive category; these were then reviewed with some reduction of the number of categories occurring. This process led to the identification of 39 categories under which the comments were grouped. The categories were grouped under seven headings. Table 3 contains a list of these codes. This coding was done in mid-May as part of an interim evaluation report.

Access

Toolset competence

Support

Student issues

Security

Multi-functionality

DLS Help Desk

Student management

Student access

Course Info

Website

Student attendance

Email access

WebBoard

Notices/ announcements

Student work habits

Stability of system

Question Mark

Training

Student confidence

User registration

BSCW

Guidelines

Student workload

Network reliability

Staff training

Faculty support

Student usage patterns

Effects on flexibility

Student induction

Other software

Email system

Educational outcomes

Communication

Miscellaneous

Achievement of educational outcomes

Email use

Staff workload

Educational assessment used

Threaded discussion

Staff enthusiasm

Educational strategies considered

Real-time chat used

Content development

Planning time

Cost effectiveness

Table 3. Categories of issues emerging from the DLS interim evaluation

We used these codes to look at all the data received in semester 1. This data included:

  1. initial data collected from 20 staff and the Help Desk, up to mid-May;
  2. more staff data from the weekly questionnaires and from email messages received in the second half of May and June;
  3. student surveys conducted by individual subjects (surveys were sometimes developed together with staff; a generic set of questions was also provided; we are still receiving this data);
  4. compilations of student email messages received by faculty support staff; and
  5. reports of four focus groups with students conducted by the first author.

Each discrete statement was assigned to a category and the number of statements in each category was tallied. The tally of results for all feedback data received in semester 1 ranged from -14 to +13; this means that one issue received 14 negative comments and one issue received 13 positive comments. Table 4 shows those issues raised which had ratings (positive or negative) of 5 and above. There was little difference in the issues highlighted by staff and students.

Issues with negative rating

Rating

Issues with positive rating

Rating

Network reliability

-14

Educational strategies considered

13

Faculty support

-11

Staff enthusiasm

11

Registration

-10

DLS Help Desk

7

Staff workload

-9

Multi-functionality

6

Course Info

-8

Effects on flexibility

7

Training

-6

   

Student workhabits

-9

   

Planning time

-6

   

Email system

-5

   

Content development

-5

   

Table 4. Benchmark issues with ratings of 5-11 (as at 6 July 1999)

An online form was made available to all Benchmark staff. It sought a rating from staff on the technical reliability of the DLS and on the service provided by the DLS team. It also sought comments on a number of areas such as problems encountered, lessons learnt, successes and general comments. As commented earlier, it was difficult to get weekly feedback from all staff. Table 5 is based on 34 occasional responses.

Rating of DLS

Technical reliability

Service from DLS team

Excellent

1

10

Very good

0

0

Good

11

19

Satisfactory

16

1

Poor

2

1

Very poor

1

0

Not applicable

3

3

Table 5. DLS ratings feedback (as at 6 July 1999)


Lessons from the semester 1 DLS Benchmark process

Designing for an online environment requires new educational design skills.

Many staff are aware of and interested in the educational potential of the DLS toolset. They recognise that new methods of teaching are involved, such as how to get full participation from students in a threaded discussion, and this creates a challenge. Technical issues can lessen motivation, and this is particularly true of staff who are observers of the Benchmark process. Detailed case studies of actual educational strategies used are needed and we have started building this into the Resources section of our website.

This is a period where ongoing dialogue is essential in departments.

"During a meeting with other staff members in my department involved in online development, it became clear that some are being encouraged to put courses online without having a clear belief that it can improve the quality of the class. Their experience is that it is a second rate tool for education and they’ll just have to make the best of it and minimise the degradation of the course. I tried to communicate that online delivery has its own strengths and these should be maximised. Also that not all courses suit being fully online. I think it’s important that staff members who are involved in online development should try to experience online learning so that they can adapt their courses in a way that suits the different pedagogy and learning style."

The process of sharing reflection on these new strategies is important.

"Many students have been accessing the discussion area and reading the messages but have not been actively participating. I have tried to introduce a number of discussions of varying seriousness to try to encourage participation as I perceived that students may be intimidated and think they must contribute only high level learning. This problem may be exacerbated by the voluntary nature of student participation, with a fully online course there would be more incentive to become involved (i.e. assessment requirements, individual encouragement from tutor, students, etc).

The most difficult aspect of preparing a course for online delivery is in trying to understand the pedagogical changes that necessitate a restructuring of most aspects of the content. The learning experience is quite different and it’s necessary to adapt the course structure to capitalise on its strengths and minimise its weaknesses."

Many staff were keen on building some of the assessment into activities which use the DLS. A number of the Benchmark subjects took this approach.

"Not surprisingly, 77% of students commented that they tended to use only those parts of the DLS which related to assessment tasks."

For many staff there have been real rewards.

"The most rewarding aspects of my experiences with online delivery have been the ability to communicate one-to-one with all my students and the emphasis it places on student-centred learning. Interaction is primarily between students and asynchronous discussion groups are an excellent way for each participant to contribute ideas."


Students must have easy and fast access to online environments.

The student registration process in semester 1 was very fraught because of several technical issues. Individual rather than automated registrations had to be done. Some staff and students negotiated this hurdle well, but for others, this left a lasting negative impression. The difficulties of the registration process in semester 1 should not be underplayed. These must be avoided in the future. Suitable computer access and network reliability were also real difficulties which must be solved if online education is to be successful at RMIT.

"The difficulties with the registration process has hampered some students from accessing the site and discouraged others. As the usage of the online component of the class is voluntary, there has been minimal use so far, mainly by students with easy web access, prior online experience and enthusiasm for the process and the subject.

Several students have reported anything from mild dissatisfaction to absolute and total frustration with the process of accessing the material. Of the 70 enrolled I currently have about 54 logged in. I don’t know whether this indicates a problem with DLS, a problem with the students or both. Responses from students via the discussion board span between total frustration, to ‘loneliness’ and an apparent lack of direction when no-one is there to do the directing. I have also had some feedback which says this is great, an ideal way to handle this material."


Once students have access and are trained, they appreciate the flexibility.

The level of sophistication and interest in online learning was sometimes on the students’ side! Both staff and students recognised this.

"The students undertaking the course were eager to explore the potential of the technology. I was dragging my feet, as I desired some level of confidence before opening the window to the group.

I think the tutors need to be more familiar with threaded discussions, e.g. like the ones which go in newsgroups. Multiple topics should be started, where replies are in an informal manner. The tutor should throw around personal experiences and what they think about certain things."

The flexibility of access has been a positive aspect for staff and students.

"Greater flexibility for students in studying, which the feedback so far suggests to be true especially for students studying purely by this mode. They do however state that they still like attending some seminars and having actual contact with peers and their lecturer.

I was particularly pleased to receive an email from a student who had contracted glandular fever, but was able to keep up with the class from home because we were using the DLS. In fact, a survey I administered (on the DLS) indicated that 74% of students found the online quizzes useful, 76% found the online tutorials useful and 42% of students used the DLS from both home and RMIT."

One student summed up the positive aspect of the experience as:

  • "ability to work at my own pace";
  • "ease of access and no time restraints on when to attend lectures - particularly useful since I work full-time"; and
  • "develops interest and virtual communication skills."

First semester examination results are still being processed; it is unlikely we will see significant learning gains, but we are keen to monitor this over time.


Staff workload is a critical issue.

"That writing material for online education is very time consuming (it requires about one to two full-time day equivalents to write one complete online module).

Unable to get immediate support. Working from home after hours is a problem - But that’s the only time I have to do these added duties. Lack of time to develop course work, implement it and support students as well as learn about new technological application so that it can best serve the teaching learning experience - On top of that you are expected to meet the bureaucratic requirements.

I did spend significant extra time in delivering my subject using the DLS, but I’d like to encourage other staff to do so, by viewing it as a long term investment (with some short term risk) that will potentially reap rewards for staff and student alike."

A discussion board needs to be a conversation where people offer opinions about certain topics. A key to generating the discussion and keeping it going is the participation of staff. In some cases, staff were not using the board effectively. They were getting the technical assistant to add comments and post announcements. This implies that the tutors did not take an active role in the discussion and students felt the lack of interaction. A telling piece of feedback in relation to this came from the technical adviser who wrote:

"I found that I had to do the bulk of this work as the academics teaching the subject did not attend DLS training sessions. I set up tutorial groups, uploaded documents, downloaded student assignments, posted announcements, and posted discussion items to the discussion board (on behalf of the lecturer). Decided how we should use the Classroom (i.e. which sections to enable/disable)."


More local support for staff and students is needed.

The support given by the DLS Team has been appreciated. But more support for students and for staff, especially at faculty level, is needed.

"In the last week I have had to address student concerns regarding access to the pages/information and access to the system. I appreciate that it is not possible to have someone on hand all the time - but when students are panicking, work is due and the system doesn’t seem to be happening - it doesn’t help. I think that this does need to be improved, especially before we have even more students participating.

The support from the support team has been very good with queries being answered almost immediately when it comes to the system and student access. Students have reported back to me the same information.

I don’t have the feeling that the Faculty is fully supporting this activity. I know I can get a graphic designer to help me, but I don’t need face-lifting at the moment."

Support means both collegial and technical support.

"What you do require is time to play and explore, support from management and the collegiality of those who have ventured in this direction before you.

I have come to the conclusion that it is essential that there be adequate specialist IT assistance available to academics in the creation of online course materials."


We need more evidence about the performance of the DLS Toolset.

Technical issues have overwhelmed these initial impressions. ‘Wait and see’ is the predominant view at this stage.

"On the positive – I do think that CourseInfo has potential to be very effective. I like the way that it looks and the interface for students, I find it to be far more user friendly than what I was doing last year. From a building perspective, I will not make my call just yet. As we are really only at the beginning of the benchmark I’m sure that there will be many more good things ahead – especially now that we can get students registered!

At the same time, there were some 60 positive responses about ‘what was good about using Online @ RMIT; and not a single student was prepared to nominate anything that was ‘bad’ about using Online @ RMIT. I will be very interested to compare these result with the same questions at the end of the year.

Do not be too optimistic, too early, about the technical side to the process."

The relative ease of the tools released several staff to think about educational issues.

"I found CourseInfo very easy to use. The key question became one of how do I organise the subject? rather than how do I use the software?"

Synchronous chat was found to be a problem; our network is not yet reliable enough.

"You are no doubt aware that my tutorials on Thursday night and Friday afternoon did not run because the server was down. In despair and frustration, and because I am concerned about your learning, I have therefore abandoned the idea of online tutorials. We will continue with the threaded discussions and with the posting of lecture materials and exercises."

We have received valuable feedback about particular tools.

"I have learned that a system needs to be comprehensive if it is to be used. For example, CourseInfo provides a method to send mail but doesn’t provide a method to read mail. This has caused my students grief.

(1) Students took readily to the online test, anecdotal feedback was positive, though will be interested in their comments a week or so ex-post. (2) Students have expressed some reservations about the BSCW collaboratory environment (despite initial enthusiasm), and how I will assess their relative contributions to their group assignments - particularly with regard to how they will ‘impression manage’ their individual contributions. That said, after some spirited discussion in the classroom around this issue, several groups appeared to be applying their minds to the ‘problem’ and demonstrated some teamwork/ delegation/ negotiation skills. I think the point has been made, but I will watch their use of BSCW with interest."


The next phase ...

In semester 2 we will have 100 new subjects online using the RMIT Distributed Learning System. Learning Technology Mentors are being appointed in each department. There are approximately 70 Learning Technology Mentors (LTMs) who are academic staff who have one day a week time release to develop online materials and support their colleagues in their departments to engage with online teaching and learning. These LTMs undertake an extensive staff development program. They all develop a work contract with the Professional Development Team of the IT Alignment Program; if individual staff wish this can be formalised into accreditation for a subject in a Graduate Certificate of Flexible Learning. Another chapter begins.


Acknowledgments

We wish to sincerely thank:

  • staff at RMIT who participated in the first semester projects in our Distributed Learning System; and
  • other members of the DLS Team who worked on the first semester projects: John Kumm, Jary Nemo, Andrew Thomas, Kate Thomas and April Weiss.

References

  • Instructional Management System (IMS),
    http://www.imsproject.org/
  • Jonassen, D. H. & Reeves, T. C. (1996). Learning with technology: Using computers as cognitive tools. In D. H. Jonassen (Ed.) Handbook of Research for Educational Communications and Technology, New York: Simon & Schuster Macmillan, 693-719.
  • Kennedy, D. K. & McNaught, C. (1997). Design elements for interactive multimedia. Australian Journal of Educational Technology, 13 (1), 1-22.
  • Kennedy, P., McNaught, C. & Nicolettou, A. (1998). Flexible and AGILE. In R. Corderoy (Ed.) FlexibilITy: The next wave? Proceedings of the Australian Society for Computers in Learning in Tertiary Education ‘98 conference, 14-16 December, University of Wollongong, 397-406.
  • Laurillard, D. Technology strategy for academic advantage. Open University, UK,
    http://www2.open.ac.uk/LTTO/internal/tsaa.htm
  • McInnis, C., James, R. & McNaught, C. (1995). First year on campus, A commissioned project for the Committee for the Advancement of University Teaching. Canberra: Australian Government Publishing Service.
  • Online @ RMIT,
    http://www.online.rmit.edu.au/
  • Reeves, T. C. & Reeves, P. M. (1997). Effective dimensions of interactive learning on the World Wide Web. In B. H. Khan (Ed.) Web-based instruction, Englewood Cliffs, New Jersey: Educational technology Publications, 59-66.
  • RMIT Clearinghouse for Teaching and Learning Projects,
    http://wwwtafe.lib.rmit.edu.au/innovate/
  • RMIT Education and Training IT Alignment Project
    http://wwwtafe.lib.rmit.edu.au/online98/main.htm and
    http://www.online.rmit.edu.au/main.cfm?code=ia00
  • RMIT Teaching and Learning Strategy,
    http://www.teaching.rmit.edu.au/
  • Yetton, P. & Associates (1997). Managing the introduction of technology in the delivery and administration of higher education. Evaluations and Investigations Program report 97/3, Canberra: Higher Education Division Department of Employment, Education, Training and Youth Affairs.
    http://www.deet.gov.au/divisions/hed/operations/eip9703/front.htm

decoration