Evaluating the Texas Woman's University Distance Education Program: A Case Study
Professor, Department of Health Studies, and Chair, Task Force on Emerging Instructional Technologies
Texas Woman's University, P.O. Box 425499
Denton, Texas 76204-5499 USA
Mary E. Cissell, MLS, MS
Student Representative, Distance Education Program
Evaluation Advisory Committee and Task Force on Emerging Instructional Technologies
Texas Woman's University
Lynda Murphy, MM
Coordinator Distance Education, Graduate Studies and Research
Texas Woman's University
Recognizing that many professionals in higher education are facing the challenge of evaluating distance education programs, a team - including a faculty member, a staff member, and a doctoral student - from Texas Woman's University (TWU) prepared a case study to present at the 1999 Annual Conference of the Texas Distance Learning Association (TxDLA). During the presentation session, they described the evaluation activities at Texas Woman's University as a case study to stimulate discussion about distance education issues and concerns among those attending the session. The result was an enthusiastic dialog that has continued well beyond the session, itself.
Part I: Case Study
Like many institutions of higher education, Texas Woman's University has found the rapid growth in emerging instructional technologies challenging. Who should teach with the new technologies? How should they be prepared? Who is responsible for helping teachers become skilled in teaching with the new technologies? How can technical support be organized and assured when teachers and programs want to offer courses with new technologies? How can we assure that students participating in distance learning activities are receiving appropriate support? How can we tell if we are succeeding in our efforts to address all of these issues? A well-planned approach to implementation and evaluation is essential to a successful distance education program. Like many other institutions of higher education, Texas Woman's University has been faced with maintaining the quality of its traditional instructional program while trying to address the challenge of integrating emerging instructional technologies.
The Innovative Do Not Wait
While central administrators were addressing the financial challenges of operating an institution of higher education with funding that was chronically below the formula established by the Texas Higher Education Coordinating Board, they found that some faculty members, some staff members, and some students were utilizing personal or outside resources to acquire skills and competence in using emerging technologies. These techies were clamoring for resources to be re-directed to equipment and support services that would permit both synchronous and asynchronous instruction to distant students. Recognizing that the emerging technologies presented new means for providing services to a broader clientele, central administrators sought to support the requests of the technologically advanced faculty, staff, and students.
Since resources were meager, planning was very limited, and the infrastructure was at an early stage of development, faculty and academic administrators resisted extensive evaluation of faculty performance prior to the 1998-1999 academic year. Whereas faculty teaching in the single onsite classroom environment were able to draw upon close personal contact with their students and great familiarity with the technological instructional aids that they were using, the technologically advanced teachers were challenged by environments that provided little personal contact and by equipment and technological support that frequently created inconveniences for their students. They did not want to have the irritation students experienced as a result of failures of the equipment or inadequate technological support reflected in the evaluations of their performance.
During the 1998-1999 academic year, substantial attention was given to the need to plan the distance education program more carefully and establish a sound infrastructure to assure that distance education would be supported by dependable equipment and more complete and competent technological support. As is stated in the introduction of the Texas Woman's University Strategic Plan for Distance Education (TWU, 1998), "Strategic planning lies at the heart of the University's distance education activities, and the specific goals, objectives, and implementing activities for distance education are correlated with the University's strategic plan, Pioneering Our Future..." Among the planning proposals, Objective 6 addresses evaluation of distance education activities. It indicates that Texas Woman's University should "...set performance benchmarks..." and that it "...must develop a more holistic, integrated plan for the evaluation of distance education activities in regard not only to programmatic quality, but also to the overall support structure and student and employer satisfaction." This is consistent with the standard of institutional effectiveness required by regional accrediting agencies (Logan, 1998).
As the 1998-1999 academic year approached, an administrative intern was selected to guide the development of a comprehensive template and action plan for evaluating the distance education program. Previously a task force on emerging instructional technologies representative of all segments of the university community had been appointed and had contributed to the decision to plan the distance education program more carefully. The administrative intern assumed chair responsibilities for the task force and invited members of it to serve on the Distance Education Program Evaluation Advisory Board (DEPEAC). DEPEAC was established to provide advice on the evaluation activities to the Associate Vice for Research and Dean of the Graduate School, who is also the director of distance education.
The first task accomplished by DEPEAC was to define its purpose and outline its goals and objectives, which included the following elements.
Enhance the quality of distance education at Texas Woman's University.
Goal 1. Facilitate development of an ongoing distance education evaluation program.
Objective A. By September 10, 1998, establish a distance education program advisory committee (DEPEAC) to provide counsel to the Office of the Associate VP for Research and Dean of the Graduate School.
(1) By August 25, 1998, identify prospective members of the DEPEAC and solicit them to serve on it.
(2) On or before September 10, notify members of the DEPEAC that they have been confirmed as members and solicit the times they are available to meet.
Objective B. By October 15, 1998, establish the range of guidance the DEPEAC will provide.
(1) Convene at least one face-to-face meeting of the DEPEAC.
(2) Establish a forum and/or a listserv to facilitate dialog among members of the DEPEAC and between the DEPEAC and the Office of the Associate VP for Research and Dean of the Graduate School.
Objective C. By November 30, 1998, develop, with counsel from the DEPEAC, a draft template for evaluation of the TWU distance education program.
(1) By October 30, 1998, an initial review of literature on evaluation of distance education programs operated by educational institutions will be completed and distributed to members of the DEPEAC.
(2) By November 15, 1998, members of the DEPEAC will be polled about the essential components of a template for evaluation of the TWU distance education program.
Objective D. By February 26, 1999, conduct a workshop for administrators, faculty and staff on implementation of an effective institutional distance education program evaluation.
(1) By November 1, 1998, convene at lease one face-to-face meeting of the DEPEAC that addresses planning for the distance education program evaluation workshop.
(2) Schedule the distance education program evaluation workshop in January or February 1999.
(3) Develop objectives for the distance education evaluation workshop.
(4) Establish an agenda for the distance education evaluation workshop.
(5) Solicit evaluation experts to serve as faculty for the workshop.
(6) Develop a workshop evaluation plan and materials
(7) Implement the workshop.
(8) Evaluate the workshop.
Objective E. By April 30, 1999, complete a pilot test of the template for evaluation of the TWU distance education program.
(1) By March 5, 1999, convene at least one face-to-face meeting of the DEPEAC that addresses a plan for pilot testing the template for evaluation of the TWU distance education program.
(2) Develop a plan for pilot testing the template.
(3) Schedule the pilot testing activities.
(4) Produce the pilot testing materials.
(5) Recruit the participants in the pilot testing activities.
(6) Train volunteers conducting the evaluation activities.
(7) Implement the pilot test.
(8) Analyze the results from the pilot test.
Objective F. By May 31. 1999, submit to the Associate Vice President for Research and Dean of the Graduate School a report on a prototypical plan for effectively evaluating the TWU distance education program.
(1) By May 7, 1999, convene at least one face-to-face meeting of the DEPEAC that addresses a prototypical plan for effectively evaluating the TWU distance education program.
(2) By May 18, 1999, post a draft of the report on a prototypical plan for effectively evaluating the TWU distance education program to the DEPEAC forum and/or listserv.
(3) Write and submit the final report on the prototypical plan to the Office of the VP for Research and Dean of the Graduate School.
Goal 2. Facilitate application of a critical thinking model to instruction delivered through distance education technologies.
Objective A. By May 1, 1999, develop a continuing education course on applying a critical thinking model to instruction.
(1) By January 15, 1999, complete a review of literature on critical thinking models and submit a report of it to the Task Force on Emerging Instructional Technologies.
(2) By February 15, 1999, select, with counsel from the Task Force on Emerging Instructional Technologies, a critical thinking model for infusion into instruction delivered by distance education technologies.
(3) By March 31, 1999, complete a draft of a continuing education course for applying the selected critical thinking model to instruction delivered by distance education technologies and circulate it to members of the Task Force on Emerging Instructional Technologies for their critique.
(4) Compile the final report and submit it to the Office of the Associate VP for Research and Dean of the Graduate School.
Meeting the Objectives of Goal 1
Objective A was met virtually before this plan was developed and adopted by the DEPEAC. This action was taken to fulfill an objective in the plan of the administrative intern. As described above, the administrative intern invited members of the Task Force on Emerging Instructional Technologies (EITT) to serve on the DEPEAC. They accepted this invitation. They began meeting in September 1998.
Objective B was met in September 1998, when the plan of activities of the DEPEAC was developed and adopted. The tasks the DEPEAC set for itself to accomplish during the 1998-1999 academic year included:
1. Review pertinent information on evaluation of distance education, including both print and online documents.
2. Summarize the information gathered and develop a matrix to depict the various components of the university to be addressed by a comprehensive plan for evaluating distance learning. This matrix can be accessed at the URL: http://www.twu.edu/committees/eitt/evaluation/matrix.html.
3. Develop a diagram that depicts the process of evaluating all activities in Texas Woman's University's distance education program. This diagram can be accessed at the URL: http://www.twu.edu/committees/eitt/evaluation/diagram.html
Objective C was met in November, when members of the DEPEAC were polled and contributed their perspectives about the essential components of a distance education evaluation template. This template was posted, in December, and can be accessed at the URL: http://www.twu.edu/committees/eitt/evaluation/default.html. The entire campus community was alerted to the URL of the template and encouraged to critique it before a symposium projected for February 28.
Objective D was met with a symposium convened on April 1, 1999, rather than the February 28 date previously projected. The symposium involved three phases, including preparation, participation, and post-symposium activities. Preparation was initiated with an invitation to the key leaders and representatives of the various operating units throughout the university. If a key leader could not participate, they were requested to appoint an able alternate to represent their operational unit.
Also, these key leaders were asked to select a small number of additional representatives from their operating units to participate and to engage these representatives in a meeting to review and critique the portion of the template that addressed their unit and provide feedback to the Distance Education Program Evaluation Advisory Committee (DEPEAC) prior to the date of the symposium.
Implementation involved conducting the symposium, itself.
Objective E was met through the third phase of the symposium activity, which included review and critique of the template and plan for implementation. The deadline for feedback to DEPEAC by symposium participants and other members of the campus community they stimulated to contribute to this process was May 3, 1999. The DEPEAC met on May 13, 1999 to review the recommendations resulting from the review and critique of the template and plan for action. The administrative intern was charged with convening a meeting of a subcommittee to select items from the Flashlight Project and existing evaluation instruments being employed at Texas Woman's University to generate an instrument to be pilot tested during the 1999-2000 academic year.
Objective F will be met by mid-June, when the administrative intern submits the final report of both the actions of the DEPEAC and his internship. The template and action plan will be incorporated in the operating manual for implementation of the distance education program, which will be distributed to administrators, selected staff, and faculty involved in distance education during the 1999 fall semester.
Postponing Action on Goal 2
Goal 2 had been established largely due to a scholarly interest on the part of the administrative intern. When the Associate Vice President for Research and Dean of the Graduate School reviewed the goals and objectives outlined by the DEPEAC, he cautioned the administrative intern that much of the work necessary to accomplish goal 2 would need to be postponed. Achieving goal 1 would require substantial time and energy on the parts of DEPEAC members. While critical thinking skills would be employed in development of the template and plan of action, introducing a particular model of critical thinking for implementation throughout the distance education courses would need to be postponed for a year or more. The administrative intern and the DEPEAC accepted this counsel and postponed any activities designed to achieve goal 2.
Part II: Discussion
Following presentation of the case study, the presenters stated that they believed that Texas Woman's University is very similar to many other publicly supported universities and colleges. They invited members of the audience to describe some of the activities at their universities and colleges and to raise questions of their own. To encourage audience participation, they asked the following questions:
1. How are you evaluating the distance education activities at your institution?
2. Do you have a comprehensive plan for evaluating your distance education program?
3. What do you consider to be the greatest challenge to effective evaluation of distance education in the university or college environment?
The presenters were rewarded by an enthusiastic response from members of their audience. The discussion was lively, with members of the audience responding to questions as well as posing them. As might be expected, the discussion went beyond evaluation to other issues and concerns related to distance education. Some of the major issues and concerns follow.
Distance Education Courses vs. On-Campus Courses
Higher education administrators, state regulatory boards, and accrediting agencies seem to hold distance education courses and programs to higher standards than those on campus. Each new distance education course at Texas Woman's University, for example, is subject to review by the particular departmental curriculum committee and the Academic Deans Council, before it is submitted for final approval to the Associate Vice President for Research and Dean of the Graduate School and to the Vice President for Academic Affairs—this true even if a particular course has been taught by the same instructor via traditional methods for a number of years.
In part, this position regarding distance education courses is the result of demands for assurance of academic quality by higher education administrators and the various regulatory agencies. Many faculty, for instance, have no real experience or training in the delivery of courses via instructional technologies. The TWU Academic Deans' Council, like many other University administrative units, has recently mandated that faculty teaching distance education courses must be trained in the technologies to be employed before their courses can be offered. In addition, adequate support services for students cannot be assumed when offering courses at a distance. Academic departments must work with the various support units in the development of courses for delivery via distance technologies in order to ensure that student needs and demands will be met.
As a result, distance education courses must undergo rigorous approval and evaluation processes. Many of the session participants felt that new sections of on-campus courses are not subject to the same level of scrutiny. The real challenge for distance education administrators is to encourage and help faculty members through these cumbersome and often confusing processes. The session participants, did express some hope that this intensified level of examination for distance education courses and programs may actually lead to improvement in the development and evaluation of on-campus courses as well.
Conversation at the session did not center solely on evaluation. For a distance education program to be successful, session participants suggested that institutions must assist faculty in developing the skills to teach at a distance. Training opportunities too often center on technical aspects, while little consideration is given to pedagogy. Session participants felt that the emphasis in training, however, should be on the art of teaching at a distance. Texas Woman's University has recently established the virtual Center for Innovation and Teaching Excellence (CITE) [http://www.twu.edu/hs/hs/cite/citehome.htm]. One of the goals of the CITE's is to work with TWU's technical support unit in offering ongoing, hierarchical training for faculty on teaching in a student-centered environment with a emphasis on student outcomes. Session participants agreed that technology should be seen as an educational tool, not the driving force in course development.
Faculty Recognition and Rewards
Questions also arose regarding faculty recognition or rewards to support distance education. Should faculty teaching via distance education be given monetary awards, release time, or other recognition for their efforts in setting up distance education classes? The participants suggested that innovative faculty should be encouraged to develop distance education courses through an established faculty reward system. Some institutions pay the faculty member per course or by the number of students enrolled. Other institutions reward online course development in their promotion and tenure reviews. Some TWU departments give faculty members release time while preparing courses to be placed online. This generally occurs the semester before the course is scheduled to go online. Some give faculty members a lighter load when teaching multi-sites via interactive television.
Issues discussed included:
1. Should a monetary bonus be given for teaching classes via distance education?
2. Should preparation of a course to go online be treated as twice the preparation of an on-site single classroom-based course, at least the first time it is offered?
3. Should bonuses, release time, or other recognition or reward be based on preparation time, number of distant sites, or total number of students enrolled?
Examples of policies used at other institutions were cited. One example given was of an East Coast institution, which allows faculty to teach as many as 250 students via one way video, two-way audio, assigns that faculty member to teach only one such class and, additionally, graduate assistants are assigned to assist the instructor. Another gives faculty $100 bonus per class. Still another has already prepared classes—syllabi, lecture notes, etc.—and the faculty member is hired solely to lecture in a class with preparation done by others.
When interactive television is used for multi-sites, should there be a class on the home end? If there is a class on the home end, should the faculty member be given credit for an additional class? This has varied among departments, and various universities handle this situation differently.
The session participants did not reach agreement regarding what an effective reward system should be, but they all emphasized the need to motivate and reward faculty to pursue the addition of technology into course development and implementation.
Services for Distant Learners
Students, like faculty, need to be trained in the technologies used in their courses. Session participants briefly shared technical requirements and indicators of student readiness for distance education courses at their colleges or universities.
One area of concern for the participants was the number and extent of support services to students at a distance for accreditation purposes. According to the Southern Association of Colleges and Schools (1996), "Appropriate student development services must be provided for distance learning programs as well as on-campus programs (section 5.4.1)."
Participants in the session discussed the meaning of "appropriate services" for students beyond admissions, registration, and payment opportunities. The presenters, for example, described attempts by Texas Woman's University to involve students at a distance in departmental clubs and organizations.
Research and Library Support
Under discussion also was the problem of how to deal with library support for classes and research. Distance sites for classes transmitted via interactive television are most frequently within institutes of higher education. In that situation, libraries are available and policies are worked out with those sites. Usually the libraries are involved early in the planning, long before the programs become operational. Items addresses would be interlibrary loan policies, availability of required resources at the remote sites, need for duplicate resources at each site, and need for a dedicated distance education librarian at each site. In one TWU distance education program, libraries at remote sites are reimbursed $100 per student per class for use of services and resources at the location.
Additional issues mentioned included copyright problems in providing information via the Internet or interactive video to these sites. Time did not permit lengthy discussion of these issues. Some of the participants have been in contact with the presenters to extend the discussion on these topics.
Three presenters from Texas Woman's University--a faculty member, a staff member, and a doctoral student--used a case study on evaluating distance education to stimulate discussion about the issues and concerns of participants at the 1999 TxDLA Annual conference. The case study described the development of a distance education evaluation template and a plan of action. The audience responded enthusiastically with issues and concerns related to evaluation of distance education, but also addressed other issues related to distance education. Members of the audience helped answer many of the questions raised during the discussion and volunteered to continue communicating among themselves about this topic.