An Adaptive Geometry Game for Handheld Devices

Introduction

When approaching the concept of adaptive learning materials, we must first have conceptual models of the behaviour of different learners within digital environments. The models can be given, but their implementation is not possible from a contstructive point of view. However, we can find some acceptable estimators for learning results which behave according to constructive rules. According to constructivism, people build up their own knowledge (Resnick, 1989; Phillips, 1997). This means that the construction process is active by nature and understanding is tied up with one’s previous experience. Learning, as well, is a constructive process, in which students actively construct their knowledge through interaction with the environment and through reorganisation of their mental structures (Chi & Bassok, 1989). The high level of students’ activity in comparison with traditional learning situations is the main argument for using information technology in education. Learning theory based multimedia-lessons (Moreno & Mayer, 1999), computerized building environments like Lego/Logo  (Resnick, 1996), and intra- or internet based social knowledge building computer tools like CSILE (Bereiter & Scardamalia, 1993), are all examples of increasing the quality of the students’ learning.

Technology utilization in teaching can be seen as a dimension which starts from information delivery and ends to cognitive tools (Reeves & Laffey, 1999). In this classification a pure cognitive tool is an adaptive and intelligent agent the only purpose of which is to support learner's thinking processes. In reported studies cognitive tools are defined more widely: In fact, many information delivery oriented solutions are claimed to be cognitive tools. The cognitive tool should cause reflection in which a learner is forced to evaluate his or her own conceptual structures and assimilate the new issue to existing conceptual structures (Jonassen, 1994). Lesgold (1998) has presented several dimensions for cognitive tools: 1) Level of cognitive functions, 2) presentation level, and 3) educational level.

1. Level of cognitive functions starts from skill-based functions, which are just skills of knowing how to behave in certain situation. The next level is rule-based function, in which the learner can choose the behavior according to some known rules. The highest level is knowledge-based functions. In this level learner knows how to do some action and why.
2. Presentation level is a dimension between concrete presentation and abstract presentation. For example in geometry Fröbel’s blocs represent a concrete cognitive tool while Papert’s turtle geometry represents an abstract cognitive tool.
3. The educational level can be seen as a dimension between education and training, where education is a holistic view to some issue and training focuses on well specified areas.

Adaptive system can be divided into two categories. Brusilovsky (2001) presents a high level categorization for the concept of adaptive systems: 1) The program’s adaptation to the user’s behavior and 2) the client device adaptation.

1. The system can be adaptive for user behavior by using user models or user data. User models were the most examined area in adaptive systems until 1996, but since 1996 the user data adaptation has been studied. User data adaptive systems do not use categories for modeling, as they accept that the behavior can not be described by specific categories (Brusilovsky, 2001; Kobsa et al., 1999). Artificial intelligence is used in almost every aspect of user behavior adaptive systems: information presentations, user modeling, intelligent agent systems, and user interfaces (Maybury & Wahlster, 1998). The adaptive geometry game is an example of an adaptive system that utilizes statistical decision making rules. The rules control the game according to human-computer interaction during the game. The decision making rules in the geometry game do not require background information about the learner, thus no learning profiles nor other pre-defined variables were used for the implementation of user behavior adaptation.
2. The client device adaptive systems delivers content to different types of platforms. The adaptation within these systems has been implemented with the utilization of environmental data. Though these kinds of adaptive systems are relatively easy to build, there are quite few studies, and solutions in this area. When content and functions are described with XML, new devices or browsers need only a new style sheet for browsing the existing content. (Wehner & Lorz 2001). The adaptive geometry game represents also this dimension of adaptivity: The game can be played both on PC computers and with PDAs (for example Compaq iPaq, HP Jornada).

User modeling is based on facts about users. Facts can be collected in one way or another. These facts are analyzed, and according to the analysis the users are grouped into different stereotypes (Rich, 1998). The validity of Rich's modeling has many threats: 1) The facts do not have to be relevant to the issue and the facts can be presented more like opinions than facts. For example, a person can not evaluate himself with a high validity. 2) The analyzing rules are developed by people, and in most of the cases the rules are not very intelligent. 3) The stereotypes place the learner to some category though constructive learning theories. However, theories on knowledge creation assume that learning styles can not be categorized, but learning is an active construction process.

Kinshuk et al. (2002) presents a classification of learning systems, varying from adaptable to adaptive ones. In this model the system is adaptable if a user can modify his or her settings in the environment. The adaptable system can also be called a personalized system. Personalization is user controlled process contrary to system controlled adaptive processes. It is said that the system is completely adaptive only if the system adopts to the users’ behavior without any user data records. The systems between these ends represent the whole scale of adaptive systems.

The intelligence in the state of the art solutions is based on neural, semantic, or Bayesian modeling. Neural and semantic networks are utilized to model the students' characteristics, learning profiles, patterns of behavior, and skill level in order to support persons performance (Kinshuk et al., 1998; Bollen & Heylighen, 1996; Brusilovsky & Cooper, 1999; Webb et al., 2001). Bayesian modeling is expected to provide more robust user profiles, though the profiles are still utilized as different classifications (Murray, 1999; Nokelainen et al. 2001). Despite the computational diversity of systems, the student modeling also has many obstacles: incomplete recorded information, uncertain observation situations, and non-stable user behavior. In some adaptive systems these obstacles are avoided by giving students the right to choose their own learning profiles (Carro et al., 1999), but those systems are mostly adaptable or personalized systems. There are also other obstacles in statistical, neural, or Bayesian modeling: Data labeling in complex environments is problematic, and even if the data were labeled, the system should be taught to use the data collected. On the other hand, the conceptual models change quickly and it is difficult to react to the change immediately (Webb et al., 2001).

From the constructive point of view, interaction between a computer and a user is the most important issue in computer supported learning materials. To create an adaptive system that uses the user interaction, an observation system must be included into learning material. According to Loomis and Blascovich (1999), the traditional relationship between experimental control and ecological validity of research is negative (Figure 1): when the experimental control of research is high, the ecological validity is low, and when the ecological validity of research is high, the experimental control is low.

Figure 1. The relationship between experimental control and ecological validity

By using a virtual environment as a research and observation environment, both ecological validity and experimental control of research can be quite high. Although the WWW environment is not a virtual environment in the full meaning of the word, we can assume that hidden observation in WWW-based learning materials can give ecologically more valid data about using the material than some traditional observation methods. Naturally, we can only get information about the use of learning materials.

Research description

The present study contains two stages (see Table 1). In the first stage the aim was to find key factors in the user’s behavior which explain the learning results. In the second stage the aim was to develop an adaptive learning system. Both stages included a software development project and an empirical study.

The substudy in the first stage included three groups of six-year-old children: experiment group 1 (n=21), experiment group 2 (n=20) and a control group (n=30). All groups were pre- and post-tested for their geometric skills. Only the experimental groups used the learning materials on geometry, the control group did not get any educational input. The measurements were made during March 2000. In the second stage there was only one experiment group (n=17). That group was compared to the groups of the first stage. The measurements were made during January 2002.

Table 1. Structure of the study

Three different learning games were implemented for this study. For the first stage there were two slightly different games implemented for laptop PC. For the second stage, one adaptive learning game was implemented for PCs and handheld devices. The empirical tests in the second stage were done with handheld devices, Compaq iPaq especially (Figure 2).

Figure 2. The adaptive geometry game on Compaq iPaq (PDA)

The idea of the game was the same for all the implementations. The user was asked to find and mark the required polygon. If the polygons were recognized correctly, the system allowed the user to continue the game. In the other case, the system informed that there were mistakes and user was told to modify the answer.

Arnes and al. (1998) had presented question about the decision making system within interactive interactions. The key points of their problematics was to 1) find rules for machine analysis of user behavior, 2) implement these rules into a system and 3) construct new interactive output. These problems were not solved in general level, the result of the study provides only one solution to one case. The rules, methods and implementations can be used only for the purposes of this kind of learning materials. Naturally the ideas can be transfered to other solutions, but they should be studied as well.

First stage – an interactive geometry game

Two different types of WWW-based learning material on geometry (polygons) were written for this study. Experiment group 1 got learning effect with learning material that requires reflective thinking. Experiment group 2 got learning effect with learning material that does not require reflective thinking all the time. The inputs were given only with the mouse, because not all six-year-old children know the symbols and letters of the keyboard. The observation system in this www-document recorded all the mouse-inputs into the temporary log-file. When a game was played, the system sents the data to a WWW-server, which addes a single piece of log-information to the large data matrix.

The research questions of the first stage include the following:

1. What kind of general learning results have been achieved with different learning materials?
2. In the use of different learning materials, what kind of differences can be found between children with good geometric skills, average geometric skills and poor geometric skills?
3. Can the learning results be explained by the use of the learning material?

Question 1 focuses on improvement between pre- and post-tests. The improvement was viewed through different skill groups, which were defined by the quartiles of the pre-test results including the whole population (experiment group 1, experiment group 2 and the control group). The test variable, the percentage of improvement, was calculated from the primary scores by subtracting the pre-test result from the post-test result and then dividing the difference by the maximum result of the test.

The real learning effect was also examined in this study. The real learning effect can be estimated by subtracting the learning effect of the test (control group) from the improvement percentages, separately in each skill group.

In question 2 the speed of the interaction and the relative errors were analysed and considered within the different skill groups. Question 3 summarizes questions 1 and 2. The explanation was examined through four variables: real learning effect, speed of interaction, type of interaction, and relatively error.

Data collection in the first stage

In the study, the information was gathered in two ways. 1) With a geometry test, which was a paper implemented traditional skill –test and 2) With an observation system that was embedded into a program code. The geometry test was planned to measure the geometric abilities of the pupils from three different points of view. The first point of view was recognizing. The pupils were asked to recognize the requested polygons among other polygons and shapes. The second point of view was analyzing. The pupils were asked to point out a polygon, which did not fit into the concept of other polygons. The third point of view was production. The pupils were asked to draw polygons with the help of different instructions. In this study, the results of these three sections were summarized and used as a primary variable.

The observation system, as well as the whole game, uses the HTTP- and WWW-services. First the client requests a WWW-page from the server. The source code of the page is sent to the client, where the WWW-browser executes the code. The code defines the user interface and connections back to the WWW-server. The WWW-server processes the received data using a cgi-program (a program executed in the server) implemented in PERL and sends the new source code back to the client.

When an event (a mouse click, for example) is made, the www-browser executes the operations that are defined in the html-code. The operations can be addressed to the WWW-server, but they can also be just operations inside the page. For observing the events in the page, there must be an embedded program code which collects all the events, handles the variables and records the data. In this study the observation focuses only on the specific elements that are the most meaningful ones for the evaluation of the learning material.

The results from the first stage tests

When the two learning materials used in first stage were examined for changes between pre- and post-tests, no differences in the learning results could be found. In Table 2 the learning effects in the experiment groups and the control group are viewed by using a percentage of improvement, which is calculated by subtracting the pre-test result from the post-test result and then dividing the difference by the maximum result of the test. According to the percentage of improvement, both experiment groups improved their test results a little more than the control group, but no statistically significant differences can be found. Generally we can say that the learning effect of the test situation was bigger than the learning effect of learning materials, as according to the control group almost 8% of the improvement can be explained by the learning effect of the test.

Table 2. Percentage of improvement between pre- and post-tests (n=71)

Different skill groups are defined by quartiles of the pre-test results including all tested populations (experiment group 1, experiment group 2, and the control group). The lowest quartile, according to the test results, is represented those with ”low skills in geometry”-group. Two quartiles in the middle represent ”average skills in geometry”-group, and the quartile that got the best test results in the pre-test, represents ”high skills in geometry”-group. The distribution of the pupils in the different skill groups seems to follow normal distribution, which means that statistical testing can be done without fittings and classifications.

Table 3. Percentage of improvement in control group (n=30)

From Table 3 we can see that pupils with average skills learned most from the test. Pupils with low skills learned a little less from the testing than averagely skilled pupils. The averagely skilled pupils learned from the test over two times more than the highly skilled group.

Because of the strong learning effect of the testing, it is interesting to focus on the real learning effects of the first learning material. The real learning effect can be estimated by subtracting the learning effect of the test from the percentage of improvement, separately in each skill group.

real learning effect = improvement– improvement_{control group}

From Table 4 we can see that only the pupils with low skills got real learning results from the learning material. When looking at experiment group 1, we can find a statistically significant difference in the real learning results between the low and high skill groups (t = 2,215 p=0,044).

Table 4. Percentages of improvement and real learning effect in experiment groups

This result supports the findings of former studies (Sinko & Lehtinen, 1999), where computer supported learning has mostly helped pupils with low skills. It seems that the learning result in this study is not dependent on different types of illustrations, as supposed earlier, because the same results were achieved with both types of illustrations.

There is an interesting negative result in the real learning effect in the high skill group. The high skill experiment group had not learned much from the learning material, but they havd gained a little improvement in the post-test. For some reason the high skill control group gains more improvement in the post-test and so the negative effect is only a result of the calculation formula.

In the following examination of the results, interaction was defined as the time between two inputs in the use of the learning material. The interaction was recorded by the program, and there could be other interactions between pupil and computer. Relative error shows the number of mistakes divided by the total number of all events.

Table 5. Interaction (seconds in average between two clicks) in different skill groups

In experiment group 1 (Table 5) the interaction in the low skill group was significantly slower than in the average skill (p=0,030) or high skill group (p=0,048). In experiment group 2 which started with learning material suitable for passive thinking, the situation was the opposite. When the low skill group showed slightly faster interaction while playing the game than the other groups, but there were no statistically significant differences between the groups.

Table 6. Relative error (%) in different skill groups

In experiment group 1 (Table 6), all skill groups made approximately an equal number of mistakes. In experiment group 2 (first passive input then reflective), the pupils with low skills made many more mistakes than in the other groups. Although there seems to be quite a big difference, the variance in individual cases was so big that no statistically significant difference can be found.

The estimators for learning result optimizing

Only the low skill group achieved a real learning effect from the learning materials, which in itself is not a surprise according to former studies. The surprise is in the use of the learning materials. When pupils with low skills in experiment group 1 had slow interaction, but made as many mistakes as others in experiment group 1, the pupils with low skills in experiment group 2 used faster interaction and made much more mistakes than other pupils in experiment group 2. Finally, average skilled and high skilled pupils did not get a real learning effect from the learning material, but they used the material approximately in the same way, not depending on the type of learning material in any way. That gives us a good viewpoint to the learning effect: In experiment group 1, where the learning material required reflective input from the very beginning, the learners were forced to think if they did not want to make mistakes. It seems that they were thinking more than the others, because the interaction was slower, but there were no more mistakes than with other pupils. In experiment group 2 the learners got first used to clicking the elements during the passive input material, because the material gave feedback on each single case and the feedback system allowed mistakes. When they started using the reflective input material which did not allow mistakes, the earlier learned way to use the material continued.

Figure 3. Estimation of the learning effect

Reflective thinking material in experiment group 2 forced pupils to think like in experiment group 1, but they still made more mistakes, which led to rethinking of the answer. In general it seems that for learning outcomes it is not important how the thinking effect is achieved, but the most important thing is that the material is so difficult that it requires thinking. It seems that the learning material did not offer new challenges to the pupils with average and high skills, which explains part of their poor learning results.

It seems that in order to achieve a good learning effect, two requirements have to be fulfilled:

1. A good control and feedback system in the learning game. This requirement can also be found from previous studies. Valcke (2002) presents that control and feedback systems in 1980-1990 were based on cognitive load theory, but nowadays all such systems are based on user observation throughout the learning task.
2. A learning task that requires time for thinking or tasks that produce enough mistakes. Learning is information processing and real learning can not be found without reflective thinking. This requirement can also be found from other studies (Valcke 2002; Mayes & Fowler 1999).

In Figure 3 the estimated learning effect is described. Fast interaction with few mistakes has no learning effects. The most obvious reason for this might be that the learning material is too easy. On the other hand, when interaction is very slow and there are lots of mistakes, the learning effect is also small. This can be explained by too difficult a learning material, or by pupils’ lack of motivation. When the combination of interaction and mistakes are in the gray area, there could be a good learning effect with the learning material. For example a pupil who has used a long time to find the requested polygons, but has not made mistakes, has achieved a good learning result.

Second stage – an adaptive geometry game

For the second stage, the geometry game was implemented for a handheld computer.  Handheld computers give new opportunities for teaching and they seem to be very suitable for this kind of solution. In fact, handheld computers with a touch screen are likely to be faster to use than PC’s with a mouse. The second version of the geometry game was implemented for Compaq iPaq PocketPC. When the PDA version of the game was tested, the assumption about platform that could be easily utilized proved to be true. The pupils immediately learned to use the touch screen and the use of the handheld computers was very easy for them.

The PDA’s (Figure 4) were connected to a server with a wireless network, which is the most powerful way to use these terminals. The wireless network behaves as wired and so the network is invisible for http-services.

Figure 4. Interactions in the second stage game

The game works perfectly also in PC environments, because the code was simplified according to the limitations of iPaq scripting. Most of the functions were executed in server. The adaptation process was done in server according to received data and adaptation rules that were set beforehand. All data were collected to database for further studies. Every time a new learning task were requested, the server sents an individual learning task to client with new orders for actions and a new user interface.

The game itself inherits the best parts of the previous reflective input interfaces (Figure 6). The game layout is built as a minimalistone, as the general usability heuristics suggest. There was not any confusing entertaining content (Harp & Mayer 1998; Garner & al 1991). Only one stabile illustration was used, though pupils did not use illustrations at all in the first study. The illustration was also meant for showing the requested shape. The polygons that should be recognized were below the illustration and the ‘check’-button. The educational ideology of the game was to provide an environment for conceptual recognition (Penner 2001). The interface was planned to fulfil the requirements of an economy of action and time (Maes 1998; Raskin 2000, 2) and it proved to be simple and easy to use in the pre-test of the game.

Mayes and Fowler (1999) suggest that there are enormous risks to spoil the learning material with a wrong kind of user interface. When usability is high, the risk to spoil reflective thinking activities are enormous. The testing system should be implemented to be easy to use, but there should not be any risks of passing the test without real knowledge (Figure 5). By utilizing this idea it is easy to implement a user friendly and effective user interface for learning materials.

Figure 5. The conceptualisation process according to Mayes and Fowler

Figure 6. Handheld user interface for geometry game

In the geometry game the good conceptualization process is ensured with ‘multiple right answers’ tasks. There are 64 combinations of the answers, so the risk for getting the right answer by guessing is less than 2% and the risk for getting two right guesses is almost a zero. The requirement for a user friendly interface was ensured in a pre-test of the game.

The adaptive learning game requires naturally more than just scalability to different platforms. The calculation rules (Figure 7) for learning result optimising were developed according to the results of the first substudy. The rules represent a non-linear system where the previous and new interaction was recorded and the learning result (the state of conceptualization process) was estimated according these data.

Figure 7. The calculation rules for learning results optimising

The calculation rules were executed on a www-server and according to the rules, the following display was constructed. The system constructs the new interface partly by randomizing the presented shapes according to calculation rules. With this solution it is almost impossible that the same interface appears twice to one player, no matter how long the game takes.

The results of the adaptive geometry game test

From the very beginning, it seems that the PDA version of the game will be a success. Pupils were very interested in it and motivated playing the game. Even those pupils with low skills, who hardly got any points from the pre-test, played the game without any problems. The motivational issues might be one of the explanations for the good results, though the learning results optimizing rules seem to play the major role.

On average the pupils played the PDA version of the geometry game for 25 minutes, which is a little less than the playing time the PC version. The shortest playing times were under five minutes and the longest playing times were above 40 minutes. The playing time or played tasks do not explain the learning results, which is a good factor. The system was planned to optimize the learning results according to the players’ behavior.

In this test group there were no highly skilled pupils. It seems that the lowly skilled group benefits again more from the learning game than the averagely skilled group (Table 7). There are a statistically significant learning effect reached in the whole test population (t=-2,961 P=0,004) compared to the learning effect of the first game. When comparing the learning effect of the PDA version of the game to the control group, we can see even more significant difference in learning (t=-3,947 p=0,000).

Table 7. The improvement and the real learning effect

Table 8. The number of pupils in the different skill groups

When comparing these results to the results of the first stage, we can see that the PDA version of the game achieves a double learning effect in the low skill group. In the average skill group the PDA version of the game has almost the same effect as the first version on lowly skilled pupils. The most important result was that all lowly skilled pupils reached the group represented by averagely skilled pupils (Table 8). This is important in order to give all pupils the same possibilities in the future.

Conclusions

The aim of this study was to develop a system that adapts to user’s behavior. Although the adaptive system in this study was very limited and the observed behavior was defined as very simple, the PDA version of the software worked extremely well. In general, the development of the adaptive learning systems takes many steps forward during this study.

The most important result was that all lowly skilled pupils reached the level of the group represented by averagely skilled pupils. This is important in order to give all pupils the same possibilities in the future. The tested population in the second stage was quite small. The population was small, because the used technology was new and all systems had to be installed to schools for testing and there were only limited resources for testing purposes. Small population naturally lowers the significance of the second stage results.  On the other hand, the achieved results were quite clear and the results were not meant to be general properties of the learning materials. The results were only meant to fulfill the needs of this study. When transfering these ideas to other learning games, the factors and the limits of the adaptation system should be examined from the viewpoints of the learning material.

Though the PDA version of the geometry game worked well, some development has to be made: When the player gets tired, which is quite usual with six year old children, player’s behavior changes a lot. The system should be taught to understand when the unexpected behavior is due to the lack of skills and when it is due to tiredness. The second issue which needs improvement is the user interface. Though the children used the touch screen well, they still made some mistakes only by accident. The user interface development may be a difficult task because the good usability of the software should be remained.

This study shows that we can easily help lowly skilled pupils with certain technologies, thus the technology was definitely worth developing. The learning effect was very promising with the simple adaptation system and we can only imagine what kinds of effects we could achieve with more complex systems.

References

Arnes, Y., Miller, L., & Sondheimer, N. (1998).Presentation Design Using an Integrated Knowledge Base. In M. T. Maybury & W. Wahlster (Ed.) Readings in Intelligent User Interfaces, San Francisco, California: Morgan Kaufmann Publishers, 131-139.

Bereiter, C., & Scardamalia, M. (1993) Surpassing ourselves - An inquiry into the nature and implications of expertice, Chigago, IL: Open Court.

Bollen, J., & Heylighen, F. (1996). Algorithms for the self-organisation of distributed, multi-user networks - Possible application to the future World Wide Web,
http://pespmc1.vub.ac.be/papers/SelfOrganWWW.html.

Brusilovsky, P., & Cooper, D. W. (1999). ADAPTS: Adaptive hypermedia for a Web-based performance support system.In Brusilovsky, P. & De Bra, P. (Eds.) Proceedings of Second Workshop on Adaptive Systems and User Modeling on WWW at 8th International Word Wide Web Conference and 7-th International Conference on User Modeling, Computer Science Report 99-07, Eindhoven University of Technology, 41-47.

Brusilovsky, P. (2001). Adaptive Hypermedia. User Modeling and User-Adapted Interaction, 11 (1), 87-110.

Carro, R. M., Pulido, E., & Rodriguez, P. (1999). An Adaptive Driving Course Based on HTML Dynamic Generation. In De Bra, P.  & Leggett, J. (Eds.) Proceedings of WebNet’99, NorfolkVA, USA: AACE, 171-176.

Chi, M. T. H., & Bassok, M. (1989) Learning from examples via self-explanations. In L.B. Resnick (Ed.) Knowing, learning, and instruction - Essays in honor of Robert Glaser, Hil­lsdale, NJ: Lawrence Erlbaum, 251-282.

Garner, R., Alexander, P., Gillingham, M., Kluowich, J., & Brown, R. (1991). Interest and learning from text. American Educational Research Journal, 28 (4), 643-659.

Harp, F., & Mayer, R. E. (1998). How seductive details do their damage: A theory of cognitive interest in science learning. Journal of Educational Psychology, 90 (3), 414-434.

Jonassen, D. H. (1994). Computers in the schools: Mindtools for critical thinking, College Park, PA: Penn State Bookstore.

Kinshuk, Oppermann, R., Rashev, R., & Simm, H. (1998). Interactive Simulation Based Tutoring System with Intelligent Assistance for Medical Education. In Ottmann, T.  & Tomek, I. (Eds.) Proceedings of ED-MEDIA / ED-TELECOM 98, NorfolkVA, USA: AACE, 715-720.

Kinshuk, Patel, A., & Russell, D. (2002).Intelligent and Adaptive Systems. In H. H. Adelsberger, B. Collins & J. M. Pawlowski (Eds.) Handbook on Information Technologies for Education and Training, Germany: Springler-Verlag, 79-92.

Kobsa, A., Koenemann, J., & Pohl, W. (1999). Personalized hypermedia presentation techniques for improving online customer relationships,Technical report No. 66, GermanNationalResearchCenter for Information Technology, St. Augustin, Germany.

Lesgold, A. (1998). Multiple Representations and Their Implications for Learning. In M. W. van Somren, P. Reimann, H. P. A. Boshuizen & T. de Jong (Eds.) Advances in Learning and Instructional Series: Learning with Multiple Representations, Oxford: Elsevier Science, 307-319.

Loomis, J., & Blascovich, J. (1999) Immersive virtual environment technology as a basic research tool in psychology. Behavioral Research Method, Instruments, & Computers, 31 (4), 557-564.

Maes, P. (1998). Agets that Reduces work and Information Overload. In M. T. Maybury & W. Wahlster (Eds.) Readings in Intelligent User Interfaces, San Francisco, CA: Morgan Kaufmann Publishers, 525-535.

Maybury, M. T., & Wahlster, W. (1998).Introduction. In M. T. Maybury & W. Wahlster (Eds.) Readings in Intelligent User Interfaces, San Francisco, CA: Morgan Kaufmann Publishers, 1-14.

Mayes, J. T., & Fowler, C. J. (1999).Learning technology and usability: a framework for understanding courseware. Interacting with Computers, 11 (1), 485–497.

Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The Role of modality and continguity. Journal of Educational Psychology, 91 (2), 358-368.

Murray, W. R. (1999). An Easily Implemented Linear-time Algorithm for Bayesian Student Modeling in Multi-level Trees. In S. Lajoie & M. Vivet (Eds.) Proceedings of 9th International Conference on Artificial Intelligence in Education (AI-ED 99), Amsterdam: IOS Press, 413-420.

Nokelainen, P., Niemivirta, M., Tirri, H., Miettinen, M., Kurhila, J., & Silander, T. (2001).Bayesian Modeling Approach to Implement an Adaptive Questionnaire. In Montgomerie, C. & Viteli, J. (Eds.) Proceedings of ED-MEDIA 2001, NorfolkVA, USA: AACE, 1412-1413.

Penner, D. E. (2001). Cognition, Computers, and Synthetic Science: Building Knowledge and Meaning Through Modeling. In W. G. Secada (Ed.) Review of Research in Education 25, Washington, DC: American Educational Research Association, 1-35.

Phillips, D. C. (1997).How, why, what, when, and where: Perspectives on constructivism in psychology and education. Issues in Education, 3 (2), 151-194.

Raskin, J. (2000). The Humane Interface, New Directions for Designing Interactive Systems, Massachusettes: Addison-Wesley Longman.

Reeves, T., & Laffey, J. (1999).Design, Assessment, and Evaluation of a Problem-based Learning Environment in Undergraduate Engineering. Higher Education Research & Development, 18 (2), 219-232.

Resnick, L. B. (1989). Introduction. In L.B. Resnick (Ed.) Knowing, learning and instruc­tion: Essays in honor of Robert Glaser, Hillsdale, NJ: Lawrence Erlbaum, 1-24.

Resnick, M. (1996). Toward a practice “Constructional Design”. In L. Schauble & R. Glaser (Eds.) Innovations in learning: New environments for education, Mahwah, NJ: Erlbaum, 161-174.

Rich, E. (1998). User Modeling via Stereotypes.In M. T. Maybury & W. Wahlster (Eds.).Readings in Intelligent User Interfaces.San Francisco, CA: Morgan Kaufmann Publishers,  329-341.

Sinko, M. & Lehtinen, E. (1999). The Challenges of ICT, Jyvaskyla, Finland: Atenakustannus.

Valcke, M. (2002). Cognitive load: updating the theory? Learning and Instruction, 12 (1), 147–154.

Webb, G. I., Pazzani, M. J., & Billisus, D. (2001).Machine Learning for User Modeling. User Modeling and User-Adapted Interaction, 11 (1), 19-29.

Wehner, F., & Lorz, A. (2001).Developing Modular and Adaptable Courseware Using TeachML. In Montgomerie, C. & Viteli, J. (Eds.) Proceedings of ED-MEDIA 2001, NorfolkVA, USA: AACE, 2013-2018.