Student Modelling in an Intelligent Tutoring System for the Passive Voice of English Language
This paper describes an intelligent tutoring system for the domain of the passive voice of the English grammar for Greek students. There is increasing interest in the use of computer-assisted language instruction, especially in cases when the language taught is not the mother tongue of the students (Bos and van de Plassche 1994; Heift 1998; Kunichike et al. 1998). The need for tutoring systems that may provide individualised support to errors is even greater when students are taught more than one foreign languages simultaneously, as is very often the case for Greek students. However, as Murphy and McTear (1997) point out, many packages of computer assisted language learning fall short when it comes to providing the learner with individualised teaching and flexible feedback.
Therefore, the multimedia tutor that we have developed has been based on Intelligent Tutoring Systems (ITSs). In this way, the system combines an attractive multimedia interface and adaptivity to individual student needs. Indeed, many researchers (e.g. McGraw 1994) point out that the integration of AI technologies and hypermedia may offer better support to students than any of these alone. The Passive Voice Tutor complies with the architecture of most ITSs, which consists of the domain knowledge, the student modeller, the advice generator and the user interface (Hartley and Sleeman 1973; Burton and Brown 1976; Wenger 1987). The main focus of the system described is on the student modelling component and especially on the error diagnosis process. The facility of individualised error diagnosis is particularly important for students, who can benefit from advice tailored to their problems. As Nagata (1997) notes, computer-assisted language instruction, in conjunction with contemporary natural language processing technology, holds out the promise of unlimited, immediate feedback pinpointed to the specific grammatical errors made by the student.
However, the underlying cause of a mistake may be difficult for a tutoring system to spot. For example, Hollnagel (1991; 1993) makes an important distinction between the underlying cause or genotype of an error and the observable manifestation or phenotype of the error. In addition, Mitrovic et al. (1996) argue that ambiguity may be a problem, since there may be different explanations of observed incorrect user’s actions. In the case of the passive tutor, error diagnosis is performed by a short term and a long term student model (Rich 1979; 1983). The short term student model is responsible for analysing the student’s mistakes in the current session. On the other hand, the long term model of each student contains information about previous mistakes or correct knowledge of the student. The long term student model is used for ambiguity resolution of competing hypotheses about errors. For example, if a student has been recorded to have been prone to spelling mistakes, then the system may use this piece of information for the interpretation of a mistake by favouring the cause of wrong spelling rather than lack of grammar knowledge at some points of the interaction. In this way we achieve a bi-directional interaction between the long term and the short term student model for the benefit of more individualised error diagnosis and help to the student.
The Passive Voice Tutor uses a categorisation of common student mistakes, so that it may identify the cause of a particular error. The categorisation has resulted from an empirical study that involved both students and teachers. In addition, this empirical study, provided information about the kind of exercises to be encoded into the system, as well as the underlying teaching strategy to be followed by the Passive Voice Tutor.
The domain representation of the Passive Voice Tutor contains knowledge about how to convert a sentence from one voice to another and knowledge about prerequisite grammatical concepts, such as the conjugation of verbs, irregular verbs, etc. In addition, the system is capable of constructing dynamically new exercises for the student to solve. This means that it incorporates syntactic and semantic knowledge concerning the words that may constitute a meaningful sentence. Hence, the system is able to show a wide range of new examples to students and to perform error diagnosis in new exercises.
The ultimate objective for educational software is that it should be educationally beneficial and it is exactly in such environments that it is important to track how usability contributes (or not) to educational goals (Jones et al. 1999). Therefore, in this research, we have evaluated the error diagnosis and advice generation components of the Passive Voice Tutor, so as to ensure that the system can be useful and effective in helping students learn.
2. Related Work
The problem of teaching a language through a computer assisted approach is very significant and has attracted a wide interest from the community of both educators and computer scientists. With regard to second language learning ITSs concerned with formal aspects of natural language, Chanier et al. (1992) distinguish two types of systems: computational-errors and early deeper analysis systems. Computational-errors systems, on the one hand, have a large linguistic coverage, but depth limited error analysis procedures. On the other hand, early systems with deeper error analysis have a narrow linguistic coverage but a quite exhaustive description of the possible errors, which may have occurred in their domain.
The Passive Voice Tutor is a system with deeper error analysis similar with other systems such as the English Tutor (Fum et al. 1992), the VP2 (Schuster 1986), and Spengels (Bos and van de Plassche 1994). In this section we highlight the difference of our approach, as compared to these three systems that are representative of systems with deeper error analysis.
English Tutor is an intelligent system, which comprises a Verb Generation Expert. This expert is devoted to tense generation and is capable of solving fill-in exercises. Each exercise consists of one or more sentences in which some verbs have to be conjugated into the appropriate tenses. In the English Tutor, student modelling is based on a methodology called backward model tracing. This methodology is used to analyse the reasoning process of the student by reconstructing step by step and in reverse order the chain of reasoning that s/he has followed in giving his/her answer. In order to do this, both correct domain specific knowledge and a catalogue of stereotyped mal-rules is used. There is also a meta-bug library that describes generic ways of altering correct knowledge and is used in cases where the catalogue of mal-rules is insufficient for the explanation of the student’s behaviour. One advantage of the Passive Voice Tutor against the English Tutor is that it contains stereotyped mal-rules for more than one levels and types of knowledge. For example, it has mal-rules concerning typos, spelling mistakes, verb transformation errors, etc. In both systems, multiple hypotheses may be generated if an error occurs. However, the hypotheses resolution in the case of the Passive Voice Tutor is performed using the long-term student model, a component which does not exist in the English Tutor. Furthermore, our system has also the ability to construct dynamically new exercises for the student to solve.
One other system that focuses on error diagnosis is VP2. VP2 is an ITS that is aimed at teaching English to Spanish people. The specific domain of interest in VP2 is English verbal constructions formed from a verb plus particle or verb plus prepositional phrase. The system apart from modelling the grammatical concepts of the English language models the Spanish language as well. The two grammar models are used in the process of the error analysis. VP2 first uses the English grammar model in order to parse a student’s answer. In the case that there is an error that cannot be recognised using this model, the system tries to perform the parsing using the Spanish model of grammar. In this way, the system is able to recognise errors that are related to the mother tongue of students. The Passive Voice Tutor is similar to VP2 in that it contains explicit representations of frequent errors of Greek students, which are due to the influence of their mother tongue. In addition, there are explicit representations of errors that are due to the influence of other foreign languages (e.g. French) that a Greek student may learn at the same time as English. However, a limitation of VP2 is that its diagnostic value seems to be minimal, since it allows no representation of the students’ mistakes and misconceptions. In the case of the Passive Voice Tutor, on the other hand, a personal long term model is held for every particular student, allowing the system to provide more individualised tutoring and diagnosis.
Finally, a system that bears a lot of similarities to the Passive Voice Tutor is Spengels. Spengels is a knowledge-based tutor for the conjugation and spelling of English verbs. The system is addressed to Dutch students who have an elementary spelling knowledge of English. The domain knowledge consists of morphological rules, such as:
It also contains spelling alternation rules that are used to alter the standard morphological rules (e.g. y -> ie; such as try -> tried). Spengels contains eight bug classes: 1) syntactic bugs, 2) overgeneralisations of a formula application, 3) application of a morphological mal-rule, 4) application of a buggy spelling rule, 5) typos, 6) analogy bugs, 7) compound bugs, and 8) miscellaneous bugs. It uses a fuzzy set theory to model the degree to which a student masters a feature or rule and it presents a short overview of the student’s performance at the end of each session. The Passive Voice Tutor has a wider domain and buggy knowledge than Spengels. The domain of Spengels (verb transformation) is covered in its prerequisite grammatical concepts. The passive voice generates several hypotheses about possible student errors at many levels. Similarly to the Passive Voice Tutor, Spengels also forms a long term student model. However, in Spengels the long term student model is only used to show to the student a short overview of his/her performance. In contrast, the Passive Voice Tutor uses the long term student model to refine hypotheses about possible student errors, as well as to show a progress report to the student.
3. Architecture of the Passive Voice Tutor
The architecture of the Passive Voice Tutor follows the main line of Intelligent Tutoring Systems (ITS) architectures. It is widely agreed that the major functional components of an ITS architecture are the domain knowledge, the student modeller, the advice generator and the user interface (Hartley and Sleeman 1973; Burton and Brown 1976; Wenger 1987). In this section we will briefly describe the domain knowledge, the advice generator and the user interface, while the student modelling is described in detail in Section 4.
The domain knowledge of the Passive Voice Tutor consists of procedures concerning the conversion of a sentence from active into passive voice and vice versa, a vocabulary of English words used in the construction of exercises, the semantic relations of the words of the vocabulary and procedures and rules about prerequisite grammatical concepts. The domain knowledge of the Passive Voice Tutor is responsible for performing the following tasks:
In order to illustrate a part of the transformation from the active into the passive voice of an exercise sentence, we present some PROLOG predicates.
The tasks performed by the domain knowledge of the Passive Voice Tutor are supported by a knowledge base that represents the vocabulary used in the exercise sentences. For each word, a number of attributes are associated with it. For example, in the case of a noun one attribute represents whether it is uncountable or not, and other attributes concern the Greek translation of the noun and translations to other languages, such as French etc. In case of a verb, attributes represent whether the verb is irregular, along with the irregular form of the past and past participle of the verb, etc. In addition, the words included in the vocabulary are related through a semantic net, so that the system may be able to identify whether a sentence makes sense or not.
The advice generator is responsible for acting when the user makes an error. In such case, it tries to respond in the most appropriate way by informing the user about what the cause of the error has been and by showing to him/her the relevant part of the theory. For example, if a student has made a mistake that is related to the use of verb tenses then the advice generator will show to the student the part of the theory that deals with this subject.
Furthermore, the advice generator is the component that is responsible for constructing new exercises, as well as showing to the learner a set of grammar pages on demand. For example, in one mode of function of the Passive Voice Tutor, the student creates exercises and the system is trying to solve them. The student may create sentences in active voice by selecting subjects, verbs and objects from three lists respectively. When the student has chosen the basic elements of the sentence, s/he may enrich the active sentence using certain supplementing words. Such words include articles, pronouns, and the words “a lot of”, “a few”, “some”, “many”. In addition, the student may also choose singular or plural form for the subject and object as well as s/he can select a tense for the verb. While constructing an active sentence, the system has the ability to semantically check the created sentence, through the semantic net of the domain knowledge that represents the relations between words of the vocabulary. Irrespective of the semantic correctness of the sentence, the system transforms it into the passive voice whenever it is asked to.
Finally, the user interface is quite important for this kind of application, because it can stimulate the student’s interest in learning. In addition, it conveys the functionality of a computer application to the user and translates the user’s input into a machine-specific format (Plass 1998). The user interface of the Passive Voice Tutor, is a multimedia user interface, which involves animations, sounds and a limited form of natural language so that it can attract the student’s interest in the subject.
4. Modelling the Student Knowledge
The student modeller is responsible for preserving the system’s estimation of the learner’s proficiency in the domain as well as his/her proneness to commit errors. The emphasis on the student modelling component of the Passive Voice Tutor has been put on the bi-directional interaction of two sub-components: the long term and the short term student model (Rich 1979; 1983; Jones and Virvou 1991).
The Passive Voice Tutor constructs a student model, which serves as a source of information that can be used for the interpretation of the student’s actions and possible mistakes in solving exercises. The student modeller checks the student’s answer against the expert’s answer and in a case of an error, it performs error diagnosis. While performing error diagnosis, the student’s answer is checked against the set of the erroneous versions that the system is able to identify. One important source of errors (although not the only one) is considered to be the interference of the mother tongue and other foreign languages that the student is familiar with. Error diagnosis is performed by the short term student model.
Another responsibility of the student modeller is to form the long term student model. The long term student model constitutes a history model of the student’s weaknesses and progress. The long term student model influences the process of error diagnosis. For example, if a student has been recorded to have frequently made typographic errors but no grammar errors at all, then in case of ambiguity the former cause is favoured. In addition, as Cook and Kay (1994) have shown, students benefit from viewing their own student models. Therefore, this kind of information is used not only for refining the error diagnosis process but also for presenting it to the user.
4.1. Error Diagnosis in Exercises
An empirical study that we conducted among human teachers and their students showed that both teachers and students were very interested in knowing which categories of error individual students were prone to and which grammatical concepts they had mastered. This result was in accordance with results from other empirical studies as well (e.g. Bos and van de Plassche 1994). Human tutors also considered it important that the type of exercises should be similar to the type used in exam papers of state schools. Therefore, the student, while working with the Passive Voice Tutor, is given three types of exercise to select from (Virvou and Maras 1999a):
As Brown (1997) points out, given the advantages of individual, time-independent language testing, computer-based testing will no doubt prove to be a positive development in assessment practice. The Passive Voice Tutor contains knowledge about how to solve an exercise correctly and in several faulty ways. The student modelling component of the Passive Voice Tutor uses a combination of buggy and overlay techniques to error diagnosis (Wenger 1987). Buggy procedures are related to the conversion of an active into a passive sentence and the opposite, as well as to the prerequisite grammatical concepts. Each one of these procedures is associated with a certain category of error. For example, a common mistake that students seem to make is the one illustrated in Figure 1. In this case, the student has neglected the rule that says that the word ‘with’ is used instead of ‘by’ in the case where the object of the passive sentence is material and not an agent. Similarly, a buggy procedure associated with prerequisite grammatical concepts may be the use of the suffix ‘ed’ while forming the past participle of an irregular verb.
The system is also able to identify not known procedures of transformation. For example, a Greek student may have a difficulty in understanding the following conversion:
Active Sentence: People believe that she is clever.
Passive Sentence: She is believed to be clever.
This is so because there is no such passive voice structure in the Greek language.
Error diagnosis is performed by the Passive Voice Tutor in the Solving Exercises Mode. In multiple choice exercises error diagnosis is simple. For every erroneous answer that the student may select, there is an associated misconception. Therefore, depending on the erroneous selection that the student has made, a corresponding error message is presented, explaining the cause of the mistake.
In the case of exercises where the student is asked to rewrite a sentence, error diagnosis becomes more sophisticated since in this case the student is allowed to be more creative than in multiple choice exercises. In the “rewrite” type of exercise the student is given a sentence in one voice (active or passive) and is asked to rewrite the sentence using another voice. The system incorporates knowledge about how to convert a sentence from one voice to another correctly. However, if the student’s answer differs from the system’s expectation then the system performs error diagnosis.
The cases where the system performs error diagnosis result from the following steps of a parsing algorithm:
Error diagnosis is performed by matching each identified error against the buggy knowledge and corresponding explanations of the system’s knowledge base. In case more than one categories of error and/or explanations match the identified mistake, then ambiguity resolution is performed, using the priorities of the categories of errors and the long term student model.
As Chapelle (1998) points out, when errors are recognised, the process of the learner’s self-correction is also believed to be beneficial, particularly because the items for which self-correction occurs may be those for which learners’ knowledge is fragile. Therefore, after identifying a student’s mistake and providing a corresponding error message, the Passive Voice Tutor expects the student’s correction, instead of providing the correct answer. The answer of an exercise may be presented only on the student’s demand.
4.2. Categories of Error and Explanations
Common student mistakes have been classified into eight broad categories of error that may be recognised by the system (Virvou and Maras 1999b). These errors are associated with the conversion of passive into active voice and vice versa and also the prerequisite grammatical concepts, such as irregular verbs. The categories of error are:
Each category of error (apart from accidental slips) may also be associated with a variety of explanations about the possible cause of the mistake. Explanations have been based on identified strategies that learners may use in order to simplify the task of learning a second language (Richards 1974):
The long-term student model records how often a student has made a certain type of error and what underlying cause was considered relevant. In some cases a mistake of the user may be attributed to more than one categories of error and/or explanations. In such a case, an ambiguity arises and the system should consult the student’s long-term model in order to resolve it. For example, if a user is given the sentence: “Alice brought flowers” and is asked to convert it to passive voice, the correct answer would be “Flowers were brought by Alice”. However, if the student types the sentence “Flowers were brought Alice” where “by” is missing then this mistake may be attributed to two categories of error. It could either be an accidental slip or a conversion mistake. If the particular student has not previously made any conversion mistakes but has frequently made accidental slips then this would have been recorded in his/her long-term student model. In this case the system would favour the accidental slip as the most probable cause of the ambiguous mistake.
Table 1. Tutors’ error diagnosis evaluation results
5. Evaluation of the Student Modelling Component
The correctness of the processes of the error diagnosis and advice generation is considered very important for the overall performance of an intelligent tutoring system. As Dix et al. (1993) pointed out a design can be evaluated before any implementation work has started to minimise the cost of early design errors. They also say that most techniques for evaluation at this stage are analytic and involve using an expert to assess the design against cognitive and usability principles.
The overall performance of the Passive Voice Tutor has been evaluated by human tutors and students (Virvou et al. 2000). However, here we focus on the presentation of the evaluation that was relevant to the student modelling component and its contribution to advice generation. The evaluation methods employed at this phase of the evaluation comprised qualitative techniques, such as questionnaires (Virvou & Du Boulay 1999; Virvou & Kabassi 2000; Virvou & Tsiriga 2000). We compare the Passive Voice Tutor’s error analysis and advice generation with that of 10 human tutors who were asked individually to give their comments on the sample.
Tutors were given a set of erroneous answers to questions relating to the domain of the passive voice of the English language, which were generated using the system’s domain knowledge and bug library. Each one of the questions was accompanied with an erroneous answer. For each question, each tutor had to evaluate the suggestion of the system concerning the underlying misconception of the error. In addition, tutors were asked whether they thought that the mistake was made due to the student’s carelessness or if it was a serious error.
Table 1 illustrates the analysis of a sample of the evaluation results. The first column provides the system’s question along with an erroneous answer of a student. In the second column we show how the Passive Voice Tutor explained the related misconception and in the third column, we give the degree of compatibility of the system’s suggestions to the human experts’ suggestions for each mistake.
The results of the evaluation showed that the Passive Voice Tutor was successful at achieving a high degree of compatibility with the human experts’ opinion. We also found out that there were limitations concerning the ambiguity resolution (for example in question 3). A problem with the evaluation was that the human tutors involved in it lacked experience with other educational software packages. In addition, due to the limited period of time in which the evaluation of the system was conducted, the system’s restrictions related to temporal aspects of the student modelling component could not be spotted by the human tutors. Temporal aspects may be related to the students’ progress in learning over time or forgetting, etc. Therefore, we plan to conduct a second phase of evaluation, which is going to last for a longer period and will focus on the temporal aspects of the long term student model.
The Passive Voice Tutor is an educational program that combines the attractiveness and user- friendliness of a multimedia interface with individualised help that an ITS can provide. In particular, its student modelling component forms a short and a long term student model for each user and performs error diagnosis. The ambiguity resolution of competing hypotheses about students’ errors is based on the long term student model, which contains information about previous mistakes or correct knowledge of the student. In order to perform error diagnosis, the system bears a detailed categorisation of common students’ mistakes along with their explanations. The error diagnosis process of the Passive Voice Tutor is especially focused on errors due to interference of the mother tongue of Greek students as well as of other languages that they may learn at the same time (e.g. French). Given the results of the evaluation, of the student modelling component, it is shown that the explanations that the Passive Voice Tutor can give are considered satisfactory. Furthermore, our system has also the ability to construct dynamically new exercises for the student to solve. For each exercise, the system is performing semantic analysis, so as to ensure that the provided sentence makes logical sense.
Within the future plans of this research, is the improvement of the student modelling component of the Passive Voice Tutor, so that it may cope with temporal aspects of the student learning process. These temporal extensions will include the modelling of the fact that the student is learning some new concepts while working with the Passive Voice Tutor, as well as that s/he may forget some already possessed concepts after some time has passed. In addition, the adaptation of the difficulty of the exercises provided to the student according to his/her proficiency level, will also be addressed in future releases of the Passive Voice Tutor.
The authors wish to thank Katerina Kabassi for her helpful comments.