Educational Technology & Society 3(4) 2000
ISSN 1436-4522

An empirical appraisal of the effectiveness of adaptive interfaces for instructional systems

John Eklund
Access OnLine Pty Limited
112 Alexander St Crows Nest NSW 2065 Australia
Tel: +61 2 94675065
Fax: +61 2 94675020
johne@accessonline.com.au

Ken Sinclair
Faculty of Education, A35
The University of Sydney 2006
NSW Australia
Tel: +61 2 91352601
Fax: +61 2 91352606

 

ABSTRACT

Navigating an information space, particularly in educational hypermedia, has its difficulties. Users may become spatially disoriented, they may be distracted, lose sight of educational objectives, or fail to relate important items of content. The predominant approaches to aid navigation in this in a well-defined information space such as educational software, involves the provision of a range of advanced navigation tools, to employ a strong metaphor and maintain interest through multimedia sequences, or to semantically structure the knowledge in the space according to some cognitively-based theory. However, none of these techniques can account for an individual learner's needs, knowledge, preferences or cognitive abilities. Adaptivity is a particular functionality that may be implemented in educational hypermedia systems in a variety of ways to recognise the importance of an individual discourse with an information space, and to alleviate navigational difficulties on that basis. This paper seeks to provide a broad understanding of some of the instructional and design principles implicit in adaptive educational hypermedia systems, those that use adaptive navigation support techniques and in particular adaptive link annotation. The claim that adaptive techniques can help solve navigation problems is examined through a review of two recent empirical studies that were undertaken to determine the effect of adaptive navigation support on user paths and learning, and a third study, the results of which is being introduced to the literature in this paper. These studies taken together have not shown a clear link between adaptivity and an improvement in learning, but  offer some guidance for ongoing productive research in this field.

Keywords: Adaptive, Evaluation, Instructional, Link annotation


Introduction

The lack of formal empirical studies on systems using adaptive navigation support techniques was the primary motivation for this sequence of studies, and the empirical results presented in this paper make an important contribution to the literature.  The central research question addressed was "does adaptive link annotation improve learning outcomes in educational hypermedia?"  The paper has set out to analyse what adaptive link annotation is, the context in which it may be used, how it has developed, how it may be implemented, and then empirically evaluate the technology in an educational context.

One of the most fundamental and persistent questions in teaching and learning in both real and virtual environments is that of the extent of learner control. How closely should learners be monitored and directed through the knowledge space? What navigational freedom should they be given to follow their interests and learn according to their own interests and needs? How should their progress be modeled and what kind of intervention strategies should be used to guide them in their learning?

Hypermedia can allow students to learn at their own pace (Laurillard, 1993) but some direction may be necessary for hypermedia to be an effective educational tool. There is a growing body of empirical evidence to suggest that learners tend to make poor decisions in learner controlled systems (e.g. Jonassen & Grabinger, 1990; Jonassen & Wang, 1993). Users may become lost, skip important content, choose not to answer questions, look for visually stimulating rather than informative material, and use the navigational features unwisely. Other researchers have demonstrated that systems with significant learner control have serious limitations in accommodating for individual learning styles, prior knowledge and interests (Murphy & Davidson, 1991; Steinberg, 1989). A number of writers of the early 1990s (de La Passardiere &Dufresne, 1992; Duchastel, 1991; Jonassen, 1992; Costa Pereira et al, 1991) have suggested that a hypermedia system with a form of expert assistance or guidance, perhaps as individualised navigational advice or instructional sequences, would provide more structure to the information space and more direction for the learner to help solve these problems of disorientation.

Höök & Svensson (1998) suggest that the metaphor "navigation in an information space" involves both the traditional way-finding activities to a known destination, as well as exploration and object identification (Benyon & Höök, 1997). Way-finding implies that users of an educational hypermedia system have some learning goal, and their task is to move through the information space in an efficient manner, both in a navigational sense and in terms of attaining their learning objectives. Linard & Zeiliger (1995) make the point that navigation occurs at two levels: both in the interface and in the content. There is an important distinction between these two types of navigation that is often not made clear when discussing navigation issues in hypermedia. The first involves users becoming oriented to the navigational tool, such as the basic functioning of a Web browser, while the latter involves users having some understanding of the topography of the domain, or knowledge, in which they are immersed. Other writers believe that hypertext systems can be examined on three levels (Germán & Cowan, 1997 [HREF1]) - the way material is presented, the way it can be navigated, and its semantic structure. Consider an inter-linked network where nodes are knowledge units connected to others in a logical fashion, if the user has some understanding of the content of the current node and its relation to surrounding nodes, this situates them in the hyperspace. Jul & Furnas (1997) make the point that navigation is movement which involves a decision-making process and is inseparable from a consideration of context. Designing for navigation involves the task, a strategy, user knowledge and characteristics of the space. They acknowledge three related levels of structure: the inherent structure of the information, a structure imposed by the author, and finally the user's view of the material which reflects the structure of the knowledge in their mind.

Given that navigation in an information space involves the user's movements toward some learning goal or information retrieval task, it is clear that one method of making navigation easier is to carefully structure the knowledge contained in the information space. Like the layout of a good textbook, the careful sequencing of the material helps navigation through the corpus of information. There is, inherent in implementations of hypermedia courseware, an author/expert defined structure of the material, simply in the sequencing of the nodes and the availability of associative links between them, but it is at the user's discretion that this sequence is followed. This has been the predominant approach to structuring educational hypermedia to date. It is based on the idea that an expert's sequencing and linking of nodes, combined with a domain referenced design of interface ergonomics, will provide a knowledge structure which reflects the way learning typically takes place in the content area. However, a fundamental limitation of this approach is that not all students are typical, nor do they learn in typical ways. The sequencing of the material and the links are fixed, and not dependant on the individual user's responses or actions. Unlike intelligent systems, hypertext-based systems are most often a static, non-adaptive learning medium. They have been described as "user-neutral" (Brusilovsky & Vassileva, 1996) because they know nothing of the characteristics of the individual user. They do not teach, but instead provide students with an excellent opportunity to learn of their own accord, and have been described as a non-pedagogical technology (Duchastel, 1992).

Whether the knowledge structure is imposed on the hypermedia in an informal way or developed through a cognitively based theory (Recker et al, 1995), the environment remains passive, ignorant of the individual knowledge state of the user. The hypermedia is static, inflexible, and presents in an identical manner for each user. This is overwhelmingly the case with the educational hypermedia on the Web. Further, the results of the limited number of empirical studies in the literature underscore weaknesses in learning from internally or semantically structured hypermedia. Such limitations and considerations strengthen the case for a more dynamic external structure to be applied to the hypermedia that accounts for the specific knowledge and tasks of an individual user - these are known as adaptive systems.

 

Adaptive Hypermedia

Adaptive hypermedia is a new direction of research within the area of adaptive and user-model based interfaces, although the goal of adaptivity has featured in the design of intelligent systems for a considerably longer period. Adaptive Hypermedia systems are capable of altering the content or appearance of the hypermedia on the basis of a dynamic understanding of the individual user, so as to adapt the content or presentation to certain characteristics of the user. In a discussion that took place during 1997 on the Adaptive Hypertext and Hypermedia Discussion forum [HREF2], an agreed definition was reached as to what is an adaptive system. "By adaptive hypermedia systems we mean all hypertext and hypermedia systems which reflect some features of the user in the user model and apply this model to adapt various visible and functional aspects of the system to the user" (after Brusilovsky, 1996. By "functional aspects" it is meant those components of a system that may not visibly change in an adaptive system. Curriculum sequencing is a good example: The "next" button will not change in appearance but it will take different users to different pages (Schwarz, 1997). Jungmann (1997a) prefers to emphasise the role of the user model in an adaptive system in his suggested definition that "...by adaptive hypermedia systems we mean all hypertext and hypermedia systems which reflect some features of the user in the way the information is presented to the user".

For a system to be called an adaptive hypermedia system it must be based on hypertext (or hypermedia); Have an explicit user-model which records some features of the individual user; Have a domain model, which is a set of relationships between knowledge elements in the information space; be capable of modifying some visible or functional part of the system on the basis of information contained in the user-model.

Hypermedia Systems that offer intelligent or adaptive interfaces (Maybury, 1993) represent one path in the development of integrated systems, where the position and prominence of the navigational tools provided to the student alter on the basis of student responses to questions posed in the tutorial as well as the student's use of navigational aids (Browne et al, 1990; Norcio and Stanley, 1989, Fischer, 1993). Such systems require a student model, which is a dynamic understanding of student knowledge. These systems provide adaptive advice, and their approach is not directive but use 'soft didactics' (Duchastel, 1991) to guide the learner. For adaptive hypermedia systems to model user and expert knowledge they require a means of collecting data about the student's knowledge state (usually through the use of history trails but also by testing), a means of assessing that data and selecting appropriate tutoring strategies, and a means of representing the domain knowledge. It is widely accepted that adaptivity requires the system to maintain a user-model, so that the terms "user-model based" and "adaptive" are interchangeable in this regard.

 

Adaptive Navigation Support -Methods and Techniques

Adaptive navigation support can be provided by a dynamic user model-driven annotation, and offers a number of techniques that may be classified according to the way they adapt the presentation of links. These are (Brusilovsky, 1996): direct guidance, adaptive ordering, hiding, and adaptive annotation. Direct guidance in a system decides what is the next "best" node for the user to visit according to the user's goal and other parameters represented in the user model. An example of direct guidance is found in ELM-ART (Schwarz, Brusilovsky & Weber, 1996), which generates an additional dynamic link (called "next") connected to the next most relevant node to visit. Direct guidance is quite a relevant technology for the context of the Web. A problem of direct guidance, however, is that it is "too directive" in that it provides almost no support for users who would like make their own choice rather than simply follow the system's suggestion. It also has the difficulty that it does not allow users to make a conscious decision about their path through the learning material, and it thus may not allow the user to have a clear picture of the layout of the hypermedia. However, direct guidance may be successfully used in combination with other technologies, in which case there is a choice for the user - the choice to be directly guided, or use some other adaptive link-level presentation, such as link annotation, described later. The idea of adaptive ordering technology is to sort all the links of a particular page according to the user model using some easily recognisable means of conveying this to the user, such as having the more relevant links closer to the top (Hohl, Böcker, & Gunzenhäuser, 1996). Adaptive ordering has a limited applicability: it can be used with non-contextual links, but it cannot be easily used for indexes and content pages (which usually have a stable order of links), and can never be used with contextual links and maps. Adaptive ordering does not provide the most stable interface for users to work within, as was found in 2L670 (De Bra, 1996).

The technology of navigation support through hiding is to restrict the navigation space by hiding links to irrelevant pages. There are a number of systems that use hiding: SPYROS (Gonschorel & Herzog, 1995); 2L670 (De Bra, 1996; De Bra & Calvi, 1997;) and HYPERTUTOR (Pérez et al, 1995) are examples. A page can be considered as irrelevant for several reasons. Perhaps it is not related to the user's current goal (Brusilovsky & Pesin, 1994; Vassileva, 1996) or it may present material which the user is not yet prepared to understand (Brusilovsky & Pesin, 1994; Pérez et al, 1995). Both HYPERTUTOR (Pérez et al, 1995) and SPYROS (Gonschorel & Herzog, 1995) use a special set of pedagogical rules to decide which concepts and nodes should be visible at the given moment and which should not (Brusilovsky, 1996). Hiding has a wide applicability, it protects users from the complexity of the unrestricted hyperspace and reduces their cognitive load in navigation. It is also rather severe in terms of the lack of control it offers the user.

The final and important technique for navigation support is adaptive annotation technology. This augments links with a comment which provides the user with information about the current state of the nodes behind the annotated links (Brusilovsky, Pesin & Zyryanov, 1993; de La Passardiere & Dufresne, 1992; Hohl, Böcker & Gunzenhäuser, 1996; Schwarz, Brusilovsky & Weber, 1996). This method has been shown to be especially efficient in educational hypermedia (Brusilovsky, and Pesin, 1995; Eklund & Brusilovsky, 1998). Link annotations can be provided in textual form or in the form of visual cues, for example, using different icons, or colours, font sizes, or font types and so forth. Link annotation does not include making link anchors active (or indeed inactive) as the user moves through the learning space, with text on the screen that initially only suggests a link progressively becoming real hypertext or vice-versa. This technique can be considered a "soft form" of hiding, but it presents to the user the full hyperspace, but with some parts unnavigable. Of course none of these annotations is difficult in traditional non-user-model-based hypermedia, but in adaptive hypermedia they are taking place in a user-dependent manner, in which different people are being served different annotations.

The goal of annotation is to provide either orientation or guidance. Orientation informs the user about their place in the hyperspace, where they have been, where they are now, what links are to related nodes or what nodes are ready to be learned, and so forth. In a sense orientation allows users to decide, on the basis of this information, where they might best proceed. This orientation support may be provided in a local or global sense, perhaps by the adaptation of maps (Zyranov, 1996). Guidance in a system is related to a user's goal, which is especially relevant in IR systems. In this case the annotation directs the user's attention to the relevant links, for example suggesting a link to follow on the basis of performance in a test. This guidance may also be offered either globally or locally. Local guidance suggests the very next step that a user should make, perhaps working in a subset of the hypermedia.

 

Categories of adaptive annotation

Within the adaptive annotation technology there are three subcategories which may be implemented alone or in combination. They are history-based, pre-requisite-based, and knowledge based adaptive link annotation (Brusilovsky, Eklund & Schwarz, 1997). Naturally these may be implemented in a variety of forms: on an overview map, on an index, under icons or in contextual links and non-contextual links. A history based mechanism is one which tracks individual users and provides them with answers to navigation questions such as "Where am I now?" and "Where have I been?” A more complex form of this is pre-requisite based adaptive link annotation, the main difference is that the nodes are related to each other in some way, and if one node is visited it may affect the state of another, related node. This again reflects the notion of adaptive systems having a domain structure consisting of a set of semantic relations between nodes, and that these relations may be visualised through the link annotations (Zhao, O'Shea & Fung, 1993). In this form, each of the hypermedia nodes has a number of pre-requisite nodes. Nodes (or "pages") may be annotated as "ready to be learned" when all of nodes pre-requisite to it have been visited. This is precisely the main technique implemented in InterBook (Brusilovsky, Schwarz & Weber, 1996); ELM-ART (Schwarz et al, 1996) and AHM [HREF6]. The problem is that while visited nodes are annotated as "learned" if all of the pre-requisite nodes to it have been visited, this does not account for behaviour such as a student flipping the pages of a book to glance ahead before returning to a serious study of the contents of those pages (Eklund, 1996).

InterBook uses special icons (coloured bullets next to hyperlinks) and different fonts to provide adaptive navigation support. On each link on InterBook pages, either in the table of contents or in the glossary or on a regular page, its font and the colour of its bullet tells the user about the status of the node behind the link. Currently four colours and three fonts are used following a traffic light metaphor. A green bullet and bold font means 'ready and recommended', that is, the node is ready-to-be-learned but still not learned and contains some new material. A red bullet and an italic font annotates a not-ready-to-be-learned node, while white means 'clear, nothing new', i.e., all concepts presented on a node are known to the user. A check mark is added for already visited nodes. Currently, InterBook does not support tests and cannot provide "well-learned" annotation, although this is under development at the time of writing. Another recent development in this system is the use of an incremental interface, where the range and variety of navigation tools that are available to any user is dependent on the level of experience of that user.

The third sub-technique for adaptive annotation is knowledge-based annotation. In this form, the environment annotates the links according to the user's demonstrated knowledge of the content. Information about the current "knowledge" of the student is collected for the student model, and this is the basis for adaptation in the system. However, many of these systems measure the student's knowledge in very different ways: According to where the user has been (history-based); according to where the user has been and how those places are related (pre-requisite-based); and lastly according to a measure of what the user has shown to have understood (knowledge-based).

 

Empirical evidence for the effectiveness of adaptive link annotation

While there is limited empirical work in the literature of adaptive annotation and its effect on navigation, it is beyond the scope of this paper to review that evidence in detail. A full review is available in Eklund & Brusilovsky, (1998). However, three recent studies offer some insight into the usefulness of adaptive link annotation in educational systems. The first by Specht (1998), the second by Eklund & Brusilovsky (1998) and the third being introduced to the literature of adaptive systems in this paper.

A very significant study in the evaluation of adaptive link annotation was published at the ED-MEDIA/ED-TELECOM98 conference by Marcus Specht. Specht's (1998) paper was an account of an empirical study to determine the effect on learning using different forms of adaptive annotation in an experimental and real-world context. In the first component of the study very controlled conditions were applied, with 85 subjects working with an adaptive system in the concise domain of prionic diseases (diseases in cattle) for a short time, with both pre and post tests being 12 multiple choice questions. Students took an online test and were told that they would repeat the same test at the end of the time, the post-test not being available until all the nodes were visited. The students worked in a hyperspace of 16 concepts about prionic diseases, with one of four different forms of annotation, namely:

  1. No annotation, plain hypertext with all hypertext clearly visible and navigable (no annotation + static linking).
  2. Adaptive link annotation, with all hypertext clearly visible and navigable, and bulleted with green=recommended and red=not recommended (adaptive annotation + static linking)
  3. Incremental linking, with links shown and annotated but not navigable. Links become navigable only when they are annotated with a green bullet (adaptive annotation + incremental linking).
  4. Incremental linking with navigable links not annotated and not shown until ready to be learned (no annotation + incremental linking).

Incremental linking is a special form of adaptive hiding where the links become available only when the system calculates that the learner is ready for them. It should be noted that in this experiment, all the treatment forms had a history-based mechanism at work, so that as each node was visited from the main page, a check-mark appeared next to the hyperlink.

Analysis of variance comparing the student's knowledge gains in each of the four treatments showed no significant effect between the two adaptive annotation treatments and none between the two which offered incremental linking. However, a comparison of the group with both adaptive annotation and incremental linking with the group with no annotation and static linking shows a significant effect on both correctly answered questions and time to browse the hyperspace. As Specht (1998) notes in the conclusion to this study, the combination of the two techniques of annotation and incremental linking leaves us unsure as to which had the most profound effect on both the acquisition of knowledge and the total time to browse the hyperspace.

The second component to Specht's study (Specht, 1998) was a field trial using the adaptive system AST (Specht et al, 1997). Over a period of 3 months, a group of 67 subjects who made 20 or more steps in the hyperspace and completed a pre and post test were analysed. The domain consisted of 23 concepts in 8 sections in the area of descriptive statistics. Three treatments were offered: Firstly a static hyperspace; secondly a group with adaptive annotation using coloured balls (green=recommended, orange=all pre-requisites learned, red=not recommended); and thirdly a group with annotation and hiding (not recommended links are hidden). Results showed that students with a good prior knowledge worked best in the annotate group, while those with less knowledge of the domain worked better with a hiding technique. Specht concludes that "...learners with a good working knowledge of the domain seem to prefer more navigational freedom..." and that those "...who do not have much previous knowledge of the domain seem to prefer more directive guidance..." (Specht, 1998, p. 1332).

The second piece of evidence for the usefulness of adaptivity as a navigation aid involves a study of twenty-five undergraduate teacher education students in an educational computing elective at the University of Technology, Sydney (UTS), learning the database and spreadsheet modules of the integrated package ClarisWorks (Brusilovsky & Eklund, 1998; Eklund, Brusilovsky & Schwarz, 1998). The study used a system known as InterBook (Brusilovsky, Schwarz & Weber, 1996), an environment offering adaptive link annotation both with and without the link annotation. The experiment was designed to be in a real-world teaching and learning context, with the use of InterBook as an integral part of a university subject. The goal of this experiment was to assess what impact, if any, user-model based link annotation would have on student's learning and on their paths through the learning space, in this realistic situation. Tests of knowledge were carried out, audit trails and questionnaires were gathered and the results analysed. The study took place over a four-week period. In the first two hour session, students were introduced to InterBook and its features explained to them. They used the system for an hour, and answered a questionnaire about its features. This questionnaire showed that almost all students were familiar with what each of the buttons and annotations meant. They were then free to use the system at any time during the following week. In the second session, students were randomly divided into two groups of equal size, one group receiving the link annotation (12 subjects), while the other group did not (13 subjects). They were allowed access to the chapter of the textbook on databases that had been authored into InterBook, and they completed a questionnaire. Students had access to the database chapter for the following week. In the third session, students took a multiple choice test on the database section of the textbook.

All student transactions with InterBook were recorded by audit trails that were used to examine how participants navigated through InterBook with and without ANS. For each user these trails showed the number of times they selected a green ball (showing recommended links), red ball (showing not recommended links), as well as their use of all the other features of the interface. This information was collated with the test results and the questionnaires. It was initially found using a two-sided t-test that students with adaptive link annotation performed significantly worse in the database section test and that there was no difference in the spreadsheet section. This unexpected result suggested that further investigation was needed. First, students who did not spend a reasonable time with the system were excluded, and once this was done we found no significant difference in the test results for the group with adaptivity and the group without. A clear correlation (R=0.670) was found between an index of the agreement rate and score in the database tests: the more students agree with system's suggestion, the better is the score - for the group receiving link annotation.

 

Experimental design of the new study

On the basis of the inconclusive evidence of these studies, it was decided to repeat the experiments with a modified experimental design. The experimental trials reported by Brusilovsky and Eklund (Brusilovsky & Eklund, 1998; Eklund, Brusilovsky & Schultz, 1998) had the aim of assessing what impact adaptive link annotation had on students' use of educational hypermedia: how they learned from it and how they moved through it.  This work was placed in the context of current evaluation studies that have been conducted on educational hypermedia systems, focussing on the technique of link annotation (Eklund & Brusilovsky, 1998).  Firstly and briefly, the conclusive result from this trial was that adaptive link annotation was useful to the acquisition of knowledge for those users who chose to follow the navigation advice.  In this respect a positive correlation of 0.67 between measures of acceptance of advice and test outcomes was obtained.  It was also inferred, although not conclusively shown, that link annotation had the effect of making user paths less linear, more exploratory in nature.  This was not surprising in some ways, as those students who were offered annotated links were supported in choosing links other than "next page", or "continue". One of the other problems with this study was that the students were able to learn from sources other than the electronic textbook.  This was because the sessions ran over weeks at a time where the students had access to InterBook, but rather than using the Web they found the ClarisWorks documents in closed reserve of the library.  Others had purchased the textbook.  This was clear in the audit trails, as some students made little use of InterBook at all.  As a result it was necessary to eliminate them from the data, creating a "reasonable time" subgroup.  This reduced the effective number of subjects in the experiment.  To avoid the recurrence of this significant problem in the next study it was decided to conduct "closed" sessions in which the students worked in the labs for a total of up to two hours, reading all the necessary material under close supervision.  The altered experimental design also meant that the amount of material had to be reduced, and groups of students with up to two hours free for the experiment would need to be found.

The same domain, namely a chapter on databases from the Clarisworks textbook was used. Prior domain knowledge was a controlled variable in Eklund & Brusilovsky’s (1998) experiment by comparing student results in other aspects of the course.  Previous computing experience and experience with hypermedia also varied little in the first group.  In this trial, these variables were controlled by the use of a modified questionnaire, in which they were asked to provide information about their use of computers in general, and their experiences and knowledge of databases and the web in particular; as well as their formal study of computing studies in senior school.  As noted, the more controlled and supervised implementation in the second trial meant the amount of work covered had to be reduced. It was also decided that for a second experiment a pre-test and post-test structure would be implemented to properly account for prior knowledge, so two separate tests consisting of eight multiple choice questions each were devised, with four common questions.  These were constructed from the same questions as used in the first experimental trial, and had been validated with very respectable Cronbach-alphas.  The common questions were selected from the careful item analysis of the test results in the first trial, and adequate reliability measures were again established.

The purpose of the four common questions was to introduce and isolate an element of information seeking to the experiment.  The pre-test was administered and at this point the students knew which of the questions were to be repeated, as this was marked on both tests (with an asterisk next to the question) and also pointed out to them very carefully at the start of the session.  It was thought that they would recall some of these questions and use InterBook's search features to answer them in the post-test.  They were asked to complete the pre-test then leave that test open next to them while they used InterBook in order for them to be reminded of the four common questions.  Learning relies on memory, but clearly if all the questions in the post-test were the same as those in the pre-test the students with the better memories would perform better in the tests, the tests would be measuring less in terms of gains in knowledge and more of a user's ability to recall questions and use the search features to find the answers to them.  If the tests had been given to the participants at the beginning to complete over the course of the session, it would be more entirely an assessment of their information seeking abilities than the knowledge they had gained from InterBook.  Conversely, had the questions been only presented at the end, the focus would be more on what they remembered from reading the ClarisWorks text through InterBook.  The approach taken for this experiment was that learning in adaptive educational systems and in hypermedia in general relies on a multiplicity of factors such as prior knowledge, concentration, and an ability to make sense of the material presented, as well as short term memory and information seeking/navigational abilities.  Measures of the acquisition of knowledge will depend upon prior domain knowledge, experience with hypermedia environments, motivation, anxiety, a certain amount of luck as to what sections the students paid the most attention to, their paths through the material, memory, and interface functionality.  Sectioning off adaptive link annotation from this great variety of influences is a difficult task indeed, and hence the desire for a tightly controlled experimental procedure.

The sessions involved the following procedure: Explaining to the participants the purpose of the trial, and how the session was to be structured; The basic features of InterBook are demonstrated to the participants; Students sign consent form and answer the eight pretest questions, these forms are collected soon after; Students use InterBook to read the database chapter of the ClarisWorks Book.  The completed pret-test is kept near the students to look at if required, although they were not allowed to alter their answers. At the conclusion of the session, students completed a post-test and the questionnaire.

The use of the analysis of covariance (ANCOVA) design was prompted by a desire to eliminate any initial differences in students' knowledge of databases which may have been present, and which might affect the validity of a positive result using analysis of variance.  Even though students are randomly allocated to one of the two treatments accounting for prior knowledge in knowledge tests is clearly desirable.  Note that the terms 'pretest' and 'posttest' in this study refers to different but very similar instruments. The common component of the pretest and posttest was not analysed separately, as it was thought that the number of questions may be too small.  Instead, the common element was an integrated "information seeking" component of the eight questions.  All of the following analyses were undertaken on the entire eight questions. 

 

Results

The results for the pretest and posttest for the two treatments are shown in the table 1.

 

Annotated group (n=53)

Non-annotated group (n=52)

 

 

pretest

Posttest

pretest

posttest

Mean

3.85

4.81

3.42

4.77

SD

1.23

1.63

1.46

1.50

Table 1. Means and SDs for entire 8 questions (n=105)

 

To determine whether posttest scores differed significantly across the two conditions, an analysis of covariance (ANCOVA) was performed, with pretest scores entered as covariates.  An initial analysis of variance (ANOVA) on pretest scores indicated no significant differences between conditions (F(1,103)= 2.62, p = .11). Initial tests for conformity to ANCOVA assumptions produced satisfactory results.  A visual examination of score plots indicated no substantial deviations from normality for either the pretest or the posttests, and the test for heterogeneity of variance for the posttest scores was also non-significant (Bartlett-Box F(1,31819) = 1.08, p = .30).  The test for heterogeneity of regression slopes across the two conditions was also non-significant ( F(1,103) = .31, p = .58), indicating that use of the pooled within-cells regression coefficient to adjust posttest scores was tenable.  The ANCOVA indicated a significant effect for the relationship between pre and posttest scores, F(1,102) = 14.87, p = .00, indicating that use of the pretest scores as covariates produced a significant reduction in error variance on the posttest scores.  These results, along with the previously mentioned validation of the test instrument allows considerable confidence in the experimental outcomes. However, the test for condition effects on adjusted posttest scores was not significant, F(1,102) < 1, indicating that adaptive link annotation has overall no significant effect on learning outcomes.

The less structured experimental design and the smaller number of participants in the trial in Eklund & Brusilovsky (1998) allowed the negative result to be more open to interpretation than that being presented here.  It is however reasonable to suppose that there are certain subgroups of the population who might benefit from adaptive link annotation.  This was the main purpose of the questionnaire: to gather additional data about users in an attempt to find variables that might account for variances in learning shown by the tests.

 

The effect of Prior Knowledge & Experience on adaptivity

Several researchers (Brusilovsky & Pesin, 1995; Conklin, 1987; Edwards & Hardman, 1993; Linard & Zeiliger, 1995; Specht, 1998) have obtained results suggesting that factors such as students’ prior experience in the knowledge domain and with hypermedia might influence their preference for more or less direction via adaptivity. Student background data gathered in the study allowed a post hoc test of these possibilities. Two composite scores based on such data were calculated, C(exp) was used to identify a subgroup of computer experienced subjects, 21 of whom had been in the annotation treatment group and 21 of whom had been in the non-annotation group. ANCOVA of pre-test and post-test scores this time yielded a significant effect for treatment groups (p<.05) with the non-annotated group performing marginally better than the annotated group. It is doubtful, however, whether this should be interpreted as a negative result for link annotation as it results from a post hoc partition of the data and is not at a high significance level. Rather, it may be interpreted as an indicator of a direction in which further research specifically designed to examine the interaction of interface experience and link annotation might profitably be conducted.

C(know) scores were used to identify a subgroup of students more knowledgable about databases, of whom 16 had been in the annotation group and 17 in the non-annotation group. ANCOVA, however, revealed that the performance differences between the two group were not significant.

 

Conclusion and discussion

The results of the studies reported above provide only partial support for the application of adaptive link annotation. Specht (1998) demonstrated that neither link annotations nor incremental linkage had significant separate effects. However, the composite of link annotations and incremental linking was found to produce superior student performance when compared with students receiving no annotations and static linking. He also found that students with a good working knowledge of the domain to be learned performed best in the annotate group while those with less knowledge appeared to prefer more direct guidance (provided by a hiding technique).

In the first of the studies by Eklund et al (Brusilovsky & Eklund, 1998; Eklund, Brusilovsky & Schwartz, 1998) it was found that adaptive link annotation was useful to the acquisition of knowledge for those users who chose to follow the navigational advice. In this respect a correlation of .67 was obtained between acceptance of advice provided by the annotations and test outcomes. In that study it was also inferred, although not conclusively shown, that annotations had the effect of making user paths less linear and more explorative in nature. In the second study, reported in this paper, link annotation was not found to influence overall performance or performance of subjects experienced in the use of the database technology. When the performance of the subjects with higher domain knowledge was compared, however, those not receiving link annotation did marginally better than those receiving link annotations. However, the overall result confirms what was found for learning outcomes in the studies of Brusilovsky & Eklund (1998) and Specht (1998).  Moreover, it supports the concerns expressed by Höök & Svensson (1998).  They stated that firstly there was little empirical evidence supporting this research direction, and secondly that what work has been reported has produced no positive outcome for adaptive navigation support for a general population of novice learners.

There are other elements of experimental design described in the final study which, when altered, might have produced a positive result.  Users in this experiment might have benefited from greater initial exposure to the environment, although whether it would have been one with or without link annotation raises new questions about them becoming accustomed to the interface.  This was noted as a problem in the experimental design used in Chapter Two, where students first used an adaptive version and then a non-adaptive version of the tool, and vice-versa.  Training sessions on either version were not used because of the 'constancy principle': users becoming accustomed to one or other version of the interface may produce confused results.  However, as a result of some recent unpublished work, there is some evidence that in order for users to benefit from the increased functionality that adaptive link annotation provides in an electronic textbook, they must be quite experienced with computers in general, and should also have significant experience with the tool itself.  This notion is based on a comparison of earlier studies conducted with ISIS-TUTOR (Brusilovsky & Pesin, 1995) and this yet to be announced work with high school graduates.  While this is not supported by the experimental outcomes above, it is clear at least that the level of user experience is having some complex interaction with both navigation and learning.  Further empirical work with groups of experienced users, and with groups with varying experience with the tool itself, is a clear direction for studies in this area as a result of this study.

In the experiment reported in this paper, various subgroups of a population were considered, and it was determined that the critical subgroups of the general novice population would be experienced users and knowledgeable users.  As mentioned early in this paper, navigation takes place at two levels, namely in the information space and in the interface, and the result described above suggests that for most users, interface considerations are the most important.  This is consistent with earlier statements that novice users will concentrate on interface features first, before considering elements of the domain.  Perhaps this also suggests that navigation, for novice users, is less to do with the conceptual association of knowledge and more a concern of 'way-finding' mentioned earlier.  The experiments in this paper used knowledge tests to look for improvements in treatment, rather than some measure of navigational efficiency, such as minimum number of steps.  It may be the case that adaptive link annotation affects some feature of user's navigation but not the acquisition of knowledge for that user.  Further experiments which aim to evaluate the effects of adaptivity using separate 'information seeking' and 'learning' tasks need to be devised and implemented.  This was part of the experimental design in the experiment, but not explicitly evaluated, primarily because it was thought that the four common questions to the pretest and post-test were insufficient to treat as separate variables from the eight questions.

Another factor was the limited nature of the adaptivity in the environment.  InterBook, as in all the systems that use link annotation, reflects its understanding of the individual user only through navigational annotations, and what it actually knows about the student is simplistic by any standards.  Student knowledge is a critical feature of teaching knowledge, an ability of a teacher to understand the student at some meaningful level, and adjust domain and teaching strategies accordingly.  This component of teaching knowledge is what distinguishes an experienced teacher from a novice, or a tutoring system from a learning environment, and was used to justify work in adaptive learning environments.  The experiments described in this paper placed InterBook in a definite teaching role, as students worked independently with it in the trials, and did not use it in a supportive way, but rather the central instrument from which to learn.  And it could be argued that annotating links on the basis of a user's path through an information space is a very poor substitution for the dynamic and complex interactions that take place in classrooms between a teacher and his/her students.  Both the InterBook experiments reported in this paper set out to determine the effects of adaptivity on learning outcomes, yet it appears that the adaptive component is such a small part of the interface, and more generally such a small component in the overall process of learning, as to be insignificant in a practical sense.  Evaluating adaptivity in a system that uses a direct guidance technology would be more likely to show a result: The 'soft didactic' of link annotation is arguably too soft.  In particular, it can also be noted that the lack of detail in the artificial world of the user-model somewhat trivialises the broad range of human responses and motives possible in learning.  Since the start of the 1990s, this has been identified as an "intractable problem" (Self, 1990), although more recent approaches give the learner access to this information as a goal-planning aid and learning-reflective device. Is it possible to make a reasonable suggestion of the next best link for a user to follow with such a paucity of information about that individual?  If it indeed is, is it meaningful when placed against so many other variables that affect learning? This revisits the problem of building practical intelligent tutors as discussed in the AI literature since the early 1980s.  Even in well defined, highly organised and simplified domains, implementing a reliable system to account for user preferences, knowledge and idiosyncratic behaviour appears to be highly problematic.

Taken as a whole, the results of the three studies are quite equivocal and inconsistent and add weight to the concerns expressed by Höök & Svenson (1998). They concluded that there was little empirical evidence supporting this research direction and that results of studies have produced no positive outcome for adaptive navigation support for a general population of novice learners. Of course, link annotation is just one small element of any interactive environment. It is widely agreed that knowledge of the student is a critical component in a learning system, whether it is a classroom or a computer mediated communication tool. If we are to implement these environments as part of the flexible delivery of course materials, there needs to be a careful choice of the tool to match the content, the target students and the course objectives. As well there is a very obvious need for ongoing evaluation of the tool in terms of its effectiveness in enhancing learning outcomes. This is an obvious direction for further research: building the technology of adaptivity into a fully developed multimedia system and evaluating it in context with, and against, a range of clearly identified interactive elements of that system.

 

Acknowledgement

Appreciation is expressed to Dr Peter Brusilovky of Carnegie Mellon University who has collaborated on much of the early research presented in this paper.

 

References

  • Benyon, D. & Höök, K. (1997). Navigation in Information Spaces: supporting the individual, In S. Howard, J. Hammond & G. Linedgaard (Eds.) Human-Computer Interaction, INTERACT97, New York: Chapman and Hall, 39-46.
  • Browne, D., Totterdell, P. & Norman, M. (1990). Adaptive User Interfaces, London. Academic Press.
  • Brusilovsky, P. (1996). Methods and techniques of adaptive hypermedia. User Modeling and User-Adapted Interaction,  6 (2-3), 87-129.
  • Brusilovsky, P., Eklund, J. & Schwarz, E. (1997). Adaptive Navigation Support in Educational Hypermedia on the World Wide Web. In Howard S, Hammond J. & Linddgaard G. (Eds.) Human-Computer Interaction INTERACT97, The 6th IFIP World Conference on Human Computer Interaction, New York: Chapman & Hall, 278-285.
  • Brusilovsky, P. & Eklund, J. (1998). A study of user-model based link annotation in educational hypermedia. Journal for Universal Computer Science, 4 (4), 429-448.
  • Brusilovsky, P., Eklund, J. & Schwarz, E. (1998). Web based education for all: A tool for developing adaptive courseware. Computer Networks and ISDN Systems, 30 (1-7), 291-300.
  • Brusilovsky, P. & Pesin, L. (1994). ISIS-Tutor: An adaptive hypertext learning environment. Paper presented at the Japanese-CIS Symposium on knowledge-based software engineering, May 10-13, Pereslavl-Zalesski, Russia.
  • Brusilovsky, P., Pesin, L. & Zyryanov, M. (1993). Towards an adaptive hypermedia component for an intelligent learning environment. In L. J. Bass, J. Gornostaev & C. Unger (Eds.) Human-Computer Interaction, Lecture Notes in Computer Science, Vol. 753,. Berlin: Springer-Verlag, 348-358.
  • Brusilovsky, P. & Pesin, L. (1995). Visual annotation of links in adaptive hypermedia. In I. Katz, R. Mack & L. Marks. (Eds.) Proceedings of CHI'95 (Conference Companion), Denver: ACM, 222-223.
  • Brusilovsky, P., Schwarz, E. & Weber, G. (1996). A tool for developing adaptive electronic textbooks on WWW. Paper presented at the World Conference of the Web Society, October 15-19, San Francisco, CA.
  • Brusilovsky, P. & Vassileva, J. (1996). Preface. User Modeling and User-Adapted Interaction, 6 (2-3), v-vi.
  • Conklin, J. (1987). Hypertext: An Introduction and Survey. IEEE Computer, 20, 17-41.
  • Costa Pereira, D., Oliveira, A. & Vaz, J. (1991). Hypermedia and ITS. In Oliveira, A. (Ed.) Hypermedia courseware: Structures of communication and intelligent help, Amsterdam: Springer-Verlag, 207-212.
  • De Bra, P. (1996). Teaching Hypertext and Hypermedia through the Web. In H. Maurer (Ed.) Proceedings of WebNet'96, World Conference of the Web Society, Charlottesville, VA: AACE, 130-135.
  • De Bra, P. & Calvi, L. (1997). Creating Adaptive Hyperdocuments for and on the Web. In H. Maurer (Ed.) Proceedings of Webnet 97, Charlottesville, VA: AACE, 149-154.
  • de La Passardiere, B. & Dufresne, A. (1992). Adaptive navigational tools for educational hypermedia. In I. Tomek (Ed.) ICCAL'92, 4thInternational Conference on Computers and Learning, Berlin: Springer-Verlag, 555-567.
  • Duchastel, P. (1991). Integrating hypermedia into intelligent tutoring. In Oliveira, A. (Ed.) Hypermedia courseware: Structures of communication and intelligent help, Berlin: Springer-Verlag, 198-204.
  • Duchastel, P. (1992). Towards Methodologies for Building Knowledge-Based Instructional Systems. Instructional Science, 20 (5-6), 349-58.
  • Edwards, D. & Hardman, L. (1993). 'Lost in hyperspace': cognitive mapping and navigation in a hypertext environment. In R. McAleese (Ed.) Hypertext: Theory into Practice. London: MacMillan Press, 90-105.
  • Eklund, J. (1996). Knowledge-Based Navigation Support in Hypermedia Courseware using WEST. Australian Educational Computing, 11 (2), 10-14.
  • Eklund, J. & Brusilovsky, P. (1998). The Value of Adaptivity in Hypermedia Learning Environments: A Short Review of Empirical Evidence. Paper presented at the 2nd Workshop on Adaptive Hypertext and Hypermedia Held in Conjunction with HYPERTEXT '98: The Ninth ACM Conference on Hypertext and Hypermedia, June 20-24, Pittsburgh, PA, USA.
  • Eklund, J., Brusilovsky, P. & Schwarz, E. (1998). A study of adaptive link annotation in educational hypermedia. In T. Ottman & I. Tomek (Eds.) Proceedings of ED-MEDIA & ED-TELECOM98, Vol. 1, Charlottesville, VA: AACE, 1327-1322.
  • Eklund, J. & Sawers, J. (1996). Customising Web-Based Course Delivery in WEST with Navigation Support. Paper presented at WebNet96 San Francisco, December 2-6, 534-535.
  • Fischer, G. (1993). Shared knowledge in cooperative problem-solving systems- integrating adaptive and adaptable components. In M. Schneider-Hufschmidt, T. KŸhme & U. Malinowski (Eds.) Adaptive user interfaces: Principles and practice, Amsterdam: North-Holland, 49-68.
  • Gonschorel, M. & Herzog, C. (1995). Using hypertext for an adaptive help system in an intelligent tutoring system. In J. Greer (Ed.) Proceedings of AI-ED'95, 7th World Conference on Artificial Intelligence in Education, Washington, DC: AACE, 274-281.
  • Hohl, H., Bšcker, H., GunzenhŠuser, R. (1996). Hypadapter: An adaptive hypertext system for exploratory learning and programming. User Modeling and user adapted Interaction, 6 (2-3), 131-156.
  • Höök, K. & Svensson, M. (1998). Evaluating adaptive navigation support. In N. Dahlback (Ed.) Exploring Navigation; Towards a framework for design and evaluation of navigation in electronic spaces,  SICS Technical Report T98:01, Stockholm: Swedish Institute of Computer Science, 141-151.
  • Jonassen, D. (1992). Designing Hypertext for Learning. In Scanlon, E. & O'Shea, T. (Eds.) New directions in educational technology, Berlin: Springer-Verlag, 123-130.
  • Jonassen, D. & Grabinger, R. (1990). Problems and issues in designing hypertext/hypermedia for learning. In Jonassen, D. & Mandl, H. (Eds.) Designing Hypermedia for Learning, Berlin: Springer-Verlag, 3-26.
  • Jonassen, D. & Wang, S. (1993). Acquiring structural knowledge from semantically structured hypertext. Journal of Computer-based Instruction, 20 (1), 1-8.
  • Jul, S. & Furnas, G. (1997). Navigation in Electronic Worlds - A CHI97 Workshop. SIGCHI Bulletin, 29 (4), 44-49.
  • Laurillard, D. (1993). Rethinking University Teaching, Routledge: London.
  • Linard, M. & Zeiliger, R. (1995). Designing a navigational support for educational software. In B. Blumental & J. Gornostaev (Eds.) Proceedings of EWHCI'95, Moscow: ICSTI, 130-145.
  • Maybury, M. (1993). Intelligent Multimedia Interfaces, Boston: AAAI Press/MIT Press.
  • Murphy, M. & Davidson, G. (1991). Computer-Based Adaptive Instruction: Effects of Learner Control on Concept Learning. Journal of Computer-Based Instruction, 18 (2), 51-56.
  • Norcio, A. & Stanley, J. (1989). Adaptive Human-Computer Interfaces: A Literature survey and Perspectives. IEEE transactions on Systems, Man and Cybernetics, 19 (2), 399-408.
  • Pérez, T., Gutiérrez, J. & Lopistguy, P. (1995). An adaptive hypermedia system. In J. Greer (Ed.) Proceedings of AI-ED'95, 7th World Conference on Artificial Intelligence in Education, Washington, DC: AACE, 351-358.
  • Recker, M., Ram, A., Shikano, T., Li, G. & Stasko, J. (1995). Cognitive media types for multimedia information access. Journal of Educational Multimedia and Hypermedia, 4 (2-3), 183-210.
  • Schwarz, E., Brusilovsky, P. & Weber, G. (1996). World wide intelligent textbooks. Paper presented at the World Conference on Educational Telecommunications, June 1-22, Boston, MA.
  • Specht, M. (1998). Empirical evaluation of adaptive annotation in hypermedia. In T Ottman & I Tomek (Eds.) Proceedings of ED-MEDIA & ED-TELECOM98, Vol. 2, Charlottesville, VA: AACE, 1327-1332.
  • Stanton, N. (1992). Maps as aids in hypertext environments: An empirical evaluation. Journal of educational multimedia and hypermedia, 1 (4), 431-444.
  • Steinberg, E. (1989). Cognition and Learner Control: A Literature Review, 1977-1988. Journal of Computer-Based Instruction, 16 (4), 117-121.
  • Vassileva, J. (1996). A task-centered approach for user modeling in a hypermedia office documentation system. User Modeling and User-Adapted Interaction, 6 (2-3), 185-224.
  • Zhao, Z., O'Shea, T. & Fung, P. (1993). Visualization of semantic relations in hypertext systems. In T.Ottman & I.Tomek (Eds.) Proceedings of ED-MEDIA'93, World conference on educational multimedia and hypermedia, Orlando, FL: AACE, 556-564.
  • Zyryanov, M. (1996). Adaptive local maps in the hypermedia components of intelligent learning environments. In P. Brusilovsky (Ed.) Proceedings of East-West International Conference on Multimedia, Hypermedia and Virtual Reality, Moscow: ICSTI, 213-216.

 

Hypertext References

[HREF1] Germain, D. & Cowan, D. (1997). Towards the Definition of Semantic Hyperstructures to allow Reader-defined Instantiation of Hypertext Systems. Paper presented at The Flexible Hypertext Workshop: A Workshop Held in Conjunction with The Eighth ACM International Hypertext Conference, April 6-11,Southampton, UK,
http://csg.uwaterloo.ca/~dmg/research/flexible.html

[HREF2] The Adaptive Hypertext and Hypermedia Home Page,
http://wwwis.win.tue.nl/ah/

[HREF3] Bell, G. (1997). Adaptive navigation within knowledge-based hypermedia. Paper presented at The Flexible Hypertext Workshop: A Workshop Held in Conjunction with The Eighth ACM International Hypertext Conference, April 6-11,Southampton, UK,
http://www.psychology.nottingham.ac.uk/~grb/papers/ht97-flexitext.html

[HREF4] Miller, M. & Wantz, L. (1998). Computed Web Links: The COOL Link Model. Paper presented at the 2nd Workshop on Adaptive Hypertext and Hypermedia, June 20-24, Pittsburgh, USA,
http://www.contrib.andrew.cmu.edu/~plb/HT98_workshop/Miller.html

[HREF5] WbtSystems Home Page,
http://www.wbtsystems.com/

[HREF6] AHM,
http://miro.cs.kuleuven.ac.be/ahm/


decoration