A Framework for Enabling an Internet Learning Community
DEMO Lab, Department of Computer Science
Brandeis University, 415 South Street
Waltham MA 02454-9110 USA
Tel: +1 781 736 3366
Fax: +1 781 736 2741
In the last decade, Internet usage has undergone a paradigm shift. While the primary purpose for accessing the Internet in its earliest days was as a vast reference source, the use of the Internet as a communication medium has become increasingly popular (GVU,1994; NUA,1995; GVU,1997). This behavior represents a change in attitude, where the Internet can be seen not only as a “virtual encyclopedia” but also as a “virtual community”.
At the same time, the Internet has grown into a fabulous resource for collecting data on human behavior. The combination of these factors ¾ the ability to join people together in a virtual space and the ability to collect vast amounts of human data ¾ has extremely powerful implications in research communities and represents a new viewpoint: Internet as “virtual laboratory”.
We have developed a framework to support experiments in on-line community learning (Sklar,2000), using the Internet as our laboratory. Our Community of Evolving Learners (CEL) system (Sklar,1998) is open to anyone with access to the Internet and a Java-enabled Web browser (http://www.demo.cs.brandeis.edu/cel). Inside CEL, participants engage in multi-player educational games, with each other or with software agents. Since our target user group is primary school children, we pay particular attention to issues of privacy and anonymity.
CEL joins the rapidly increasing number of interactive learning systems (ILS) appearing on the Internet. Gordin (1996) cites five types of interaction that may be provided at these sites:
The Learning Through Collaborative Visualization project (CoVis) (Pea,1993) was one of the first and most extensive educational on-line communities (http://www.covis.nwu.edu). It includes all five interaction types and centers on uncovering new ways to teach science in classrooms. CoVis is geared towards forming distributed electronic communities dedicated to science learning in kindergarten to 12th grade environments, particularly through scientific visualization. As well, the system provides links to Internet resources and several interaction devices, such as real-time collaborative environments for conversing with teachers and other students. Specific curricular activities are built into the system, for example learning about the weather using graphics tools that let users view climate maps or satellite imagery.
Belvedere (Suthers,1997) is designed to assist students learning critical inquiry skills for science domains (http://lilt.ics.hawaii.edu/belvedere/index.html). The primary element of the system is a collaborative inquiry database, which students can access through a variety of interfaces. The database helps them keep track of the problem they are addressing, their hypotheses, evidence and references. The system also includes a sophisticated coach that can help students during the critical inquiry process.
The Knowledge Integration Environment (KIE) (Bell,1995) is also designed to help students with science learning and is focused on bringing students and evidence together to solve problems cooperatively (http://www.clp.berkeley.edu/KIE.html). The system includes tools like an electronic space for taking notes, on-line discussion tools and a knowledge integration coach that provides hints to students as they work together to answer scientific questions. A more recent version of this system is called the Web-based Integrated Science Environment (WISE) (http://wise.berkeley.edu/WISE/welcome.php).
The Concord Consortium is an international cooperative of high schools that delivers courses over the Internet (http://www.concord.org). Teachers from participating schools contribute teaching time, in exchange for student enrollments. Courses are offered on-line, and interactions can occur between students and teachers.
Educational Object Economy is a project that grew out of a research effort at Apple (http://www.eoe.org). This is a comprehensive community where members can post papers, hold discussions and exchange software, with special emphasis on authoring tools. The target audience is researchers and educators.
MUDs (Multi-User Dungeon/Dimension/Domain) were introduced in the late 1960's, originating with analog text-based role-playing adventure games. Some have used the MUD paradigm to build educational applications (Fanderclai,1995;Gordon,1998). Two examples of educational MUDs implemented on the Internet are Pueblo (Walters,1994) and MOOSE Crossing (Bruckman,1997).
Pueblo, originally called MariMUSE, was first used at a summer camp for primary school children in Arizona in 1993 (http://pcacad.pc.maricopa.edu/Pueblo/index_frame.html). The focus in Pueblo is on learning through writing, programming and simulation. Participants create their own virtual world and take on new identities, using the language of the MUD to invent and describe places and creatures, defining appearances and enacting behaviors. Researchers found that Pueblo served to draw otherwise uninterested children into literacy activities and that the paradigm helped to break down traditional classroom and social boundaries.
MOOSE Crossing is a constructionist environment designed for late primary and middle school students (http://www.cc.gatech.edu/fac/Amy.Bruckman/moose-crossing/). Again, this is a text-based MUD, however MOOSE Crossing uses a new language called MOOSE which was the first MUD language designed explicitly for children. MOOSE Crossing is enabled on the Internet, however it is not accessible from inside a browser. Participants must download software onto their computers and connect to a server using this software. Participants enter MOOSE Crossing with a user name and password, but must apply for membership off-line by sending in forms signed by the child member and a parent/guardian. Adults monitor discussion on the site, and anyone found misbehaving is denied future access. MOOSE Crossing has been extremely well-received by researchers as probably the first widely available environment of its kind, and analysis of participation has revealed positive results similar to those found in Pueblo.
CEL offers an alternative to traditional interactive learning systems, taking advantage of the Internet to provide a setting where students can succeed and fail incognito, safe from the typical social pressures of face-to-face classroom encounters. The work presented here puts forth the CEL system as the basis for future experiments. The environment is described, some sample activities are outlined and various types of data collection are discussed.
Several aspects of CEL distinguish it from other Internet learning communities:
This article is divided into four sections. First, we provide a brief tour of the CEL environment. Next, we review the architecture of the CEL system. Then we describe a pilot study that was used to validate the CEL mechanism and data collection opportunities. Finally, we conclude with a discussion of current work and future directions.
A Brief Tour of CEL
CEL is located on a free World Wide Web site and is accessible to anyone, via a browser. Students log into CEL with an individual user name and password. In order to maintain the levels of anonymity and privacy that CEL demands, the user name (and password) is never shown to others. While the environment is geared toward young users, participation is not restricted ¾ and it doesn't matter who enters CEL, because like a samba school (Papert,1980), participants of all ages and abilities may learn from interacting with each other.
Inside the system, participants are represented only by two-dimensional graphical icons called IDsigns, which users create themselves using a tool called the IDsigner (see figure 1). They are given a palette of 13 colors and a straightforward point-and-click interface with which they can set the color of each pixel.
Figure 1. The IDsigner
After logging in, students are shown a simple menu page, containing a list of available activities. Clicking on a game icon selects that activity. Next, users are placed in an open playground, a page that contains a matrix filled with IDsigns belonging to other users who are currently logged into CEL and are playing the same game (figure 2). These are a user’s playmates; together they comprise a user’s playgroup.
Figure 2. The CEL Playground
By clicking on a playmate’s IDsign, a student invites that playmate to join her in a match. The match begins when the browser displays a game page, containing a Java applet that facilitates play. Both players participate according to the particular format of the selected game. When the match is over, each player is returned to his playground and is then free to engage in another match with another (or the same) playmate.
Inside CEL, there is no chat room - all communication takes place through the moves of the games the children are playing. While the absence of conversation diverges from typical computer supported collaborative learning (CSCL) systems, we feel that open chat on the Internet is not appropriate for young children. The CEL environment provides us with the opportunity to study non-conversational collaborative learning.
For the prototype implementation of CEL described in this article, we built very simple games so that we could evaluate the feasibility of hosting various types of activities. We created several two-player, educational games aimed at reinforcing skills introduced in classrooms: keyboarding (typing), spelling, arithmetic and construction.
The pilot study (discussed in a later section) used two keyboarding games: Keyit and Pickey. While these games follow an old-fashioned drill-and-practice paradigm, they proved fun and interesting to the students involved in the pilot study and gave us the opportunity to develop the system design. Our present work involves collaborations with researchers in human learning to build games with a sound pedagogical basis.
Keyit is pictured in figure 3. This is a competitive game in which participants are given ten words to type and are scored based on speed and accuracy. For each player, a timer begins when she enters the first letter of a word. Time is measured using the system clock on the client’s computer, and a score is calculated, which is simply the time in hundredths of a second. During the course of a game, feedback is provided to both players as words are typed. The match need not be synchronized. For example, a network link may be slow or one user may be interrupted. In this case, the system provides to each user any moves that are available from their playmate.
Figure 3. The game of Keyit
The game of Pickey is pictured in figure 4. This game is very similar to Keyit, except that users start with the complete list of ten words, and they pick which ones they want to type.
Figure 4. The game of Pickey
A map of the CEL site is shown in figure 5, indicating the relationship between the games menu and the playground and match pages. The unshaded boxes represent the static portion of the site. The shaded boxes illustrate playground and match pages, which are created dynamically for each activity in CEL, as users enter and exit playgrounds and initiate matches.
Figure 5. Site map
The CEL system employs a client-server architecture, as illustrated in figure 6. One central server maintains a dynamic database indicating who is logged into the system and which games they are playing. This server also acts as a message passer, sending and receiving commands that go between clients and matchmakers. For each activity in CEL, a matchmaker keeps track of the users that are currently active and their state ¾ either “lonely” (where players see the playground page, as in figure 2) or “playing a match” (where players see a game page, as in figure 3 or 4).
Figure 6. System architecture overview
The client is designed to meet two fundamental needs:
As such, we use small footprint Java applets for games and implement the playgrounds using CGI-bin programs that generate HTML and refresh periodically in order to update playgroup content.
The system is modular and extensible. A new activity is added to CEL by extending Java classes to implement a matchmaker and a game applet. Eventually, we will place these classes and documentation on our web site in order to allow others to contribute to the community by creating their own games. Although all matchmakers currently reside on our server, they could be instantiated on any networked computer, so contributors could choose to host their games and collect data locally if desired. See (Sklar,2000) for further discussion of the extensibility of the CEL system.
In early 1999, a pilot study was conducted with CEL in order to ensure that the system was accessible from a real school setting, that the client software was usable by children, that CEL’s data collection facility was appropriate for the types of analyses we anticipate performing and that the children enjoyed their experiences with the system. During the pilot study, CEL was used by 44 fourth and fifth grade children at a public primary school. Participation centered around a computer lab in the school, containing 15 networked iMac computers. Children in this school make regular use of this lab for various activities, thus the setting for the study was familiar and comfortable. All participants had signed parental permission.
For this study, the activities in CEL were the simple keyboarding games described earlier, Keyit and Pickey. It is important to note that the purpose of this study was to validate the mechanics of the CEL system, and so little effort was put into the design of these games. We found it to be extremely encouraging that, despite the archaic nature of the games and the relative sophistication of our student subjects, the students genuinely and thoroughly enjoyed their experiences with CEL. To us, this indicates the potential power of the system as a platform to support sophisticated activities.
Children from each of the classes visited the computer lab for an hour once a week and “did CEL” under the supervision of the first author. In most sessions, due to scheduling constraints, only children from the same grade were in the lab at the same time. Our study took place over a six-month period. During the first few weeks, adjustments were made to the system to accommodate the bandwidth and memory limitations at the client site, as mentioned earlier. The remainder of the period (a total of 19 days) was spent collecting data, in order to verify the data capture and storage components of the CEL system. We used this data for the sample analysis shown here. At the conclusion of the study, the participants were asked to complete a written survey; this is also discussed.
There is no simple standard for evaluating interactive learning systems. Mark & Greer (1993) outline methodologies commonly used, describing two general categories of assessment: formative analysis, generally performed by computer scientists, used to assess the design and behavior of a system in-progress; and summative analysis, generally performed by educators and/or psychologists, used to assess the effectiveness of a completed system. The purpose of the pilot study was to perform formative evaluation of the CEL system.
We focus on three avenues of analysis for the pilot study data, in accordance with some of the types of evaluations frequently described in the ILS literature (Sklar,2000). These are:
Our overall goal is to demonstrate that the CEL system is capable of supporting the types of activities, experiments and data collection common to the ILS field.
In multi-player educational games, researchers and teachers are interested to find out who participants are interacting with. As well, if software agents are involved, system builders want to know if students are interacting with the agents and to be able to compare human-human and human-agent encounters. During the pilot study, the students interacted with both human playmates and software agents. To test the various modes, we varied the interactions on each day, as summarized below:
Table 1 shows which participants were involved during each test day. On some days, more than one type of participation occurred. Internal participants (IN) were always present. Note that on days 3, 4 and 6, we had to curtail data collection due to various system problems.
Table 1. Summary of participation
The next set of figures illustrate some common types of interaction analysis, such as the change in activity rate under different interaction conditions, the amount of interaction between different types of participants and the specifics of who interacts with whom. Note that these graphs do not take into account the number of participants available in each category during the pilot study, nor was a control study performed, so the reader is cautioned not to draw specific conclusions concerning the comparative success or failure of the different interaction conditions.
Figure 7 illustrates the rate of game play under the various conditions, i.e., the number of games played per minute, averaged across all the students participating that day. The days with the lowest rates correspond to the days in which system problems occurred (days 3, 4 and 6); day 19 had a lower rate because a new math game was introduced, and students required some additional instruction for playing this game.
Figure 7. Number of games played per minute, for each data collection day
Figure 8 shows the breakdown of games played by the kids amongst themselves (IN), with the secret agents (SA) and with others who were not in the room (EX).
Figure 8. Interactions between types of participants
Figures 9 and 10 illustrate the interplay between various groups of participants, namely “who challenged whom”. In both graphs, the vertical axis shows the players who initiated matches and the horizontal axis lists the players who accepted. A point indicates that at least one encounter occurred between the given pair of players. In figure 9, the children are grouped according to grade level and gender; the same order is used on both axes. It is clear to see that the two classrooms of children were infrequently in CEL at the same time (this was due to scheduling constraints), thereby verifying our methodology.
Figure 9. Who challenged whom, grouped by age and gender
Figure 10 contains the same data, but organized differently. The four “kids” sections of figure 9 are intermingled, and in figure 10, the children are ordered by typing speed. The fastest typers are nearer to the origin.
Figure 10. Who challenged whom, ordered by typing speed
Naturally, everyone wants to know if participants’ performance improves when using an interactive learning system. CEL can track changes in performance. As an example, we look at the change in the children’s typing speed as calculated at the beginning and the end of the pilot study. The children engage in many keyboarding activities, so we cannot attribute their progress exclusively to their usage of CEL. However, we can measure their progress, as illustrated in figure 11.
Figure 11. Change in typing speed
The graph indicates that most of the children (85%) improved, shown as a (positive) increase in typing speed. However, some children’s speed decreased. Various factors may account for these decreases, such as children leaving the room in the middle of a session, insufficient data collection or inappropriate increase in challenge level (i.e., the words given to players were too easy at first and too hard later on). Being able to read this type of information is valuable to researchers who may, for example, want to conduct controlled experiments in which they can compare various algorithms for selecting which words to present to typers as they progress.
Determining participants’ interest levels in an interactive learning system is quite important to researchers and system builders. At the end of our data collection period, we asked each of the children to complete a short, anonymous survey. The survey contained six questions, and the wording was geared toward fourth and fifth graders. 90% of the surveys were returned, from 22 of the fourth grade children and from 18 of the fifth grade children. The survey was very open-ended, first asking general information about the student’s grade level and which games the student had played. Next the students were asked to name three things they liked about CEL and three things they did not like. Then they were asked to name three things that could be changed to make CEL better. Finally they were asked if there was anything else they wanted to tell us.
The survey results for three of the questions are tallied in tables 2 through 4, respectively. The tables should be read as follows: if 21 students said they liked creating IDsigns, this means that 21 of the children chose to include this feature in their list of things they liked (it does not mean that the remaining 19 children did not like creating IDsigns).
Table 2. Name 3 things that you like about CEL
Table 3. Name 3 things that you don’t like about CEL
Table 4. Name 3 things that could be changed to make CEL better
It was gratifying that, along with creating IDsigns, the most common answer to “things you like about CEL” was learning. As well, many of the things the children didn’t like reflected how much they did like the system. For example, fifteen children said there was nothing they didn’t like and eight said they didn’t like when the system broke (and they couldn’t use it).
The children suggested a few interesting improvements, such as allowing users to select from a set of ready-made IDsigns, in case they do not want to make their own. We found it ironic that, despite our sensitivity to open competition in an educational setting, several children’s suggestions involved making the site feel more competitive: adding skill levels, ranking pages and prizes. Although the general consensus for “Name 3 things that you don’t like about CEL” was that the matches were too hard, for “Name 3 things that could be changed to make CEL better”, the same number of children commented that the matches were too easy as commented that the matches were too hard.
For “Is there anything else you would like to tell us?”, 29 of the children wrote either “no” or did not respond. A few suggested improvements, which we tallied along with the responses to “Name 3 things that could be changed to make CEL better.” Seven children wrote statements emphasizing how much they liked the system (“It was really fun.”).
Here are some highlights from the survey. (Note that the school where the pilot study took place practices “Write to Read”. With this method, early writers are taught first to express themselves and second to focus on mechanics like spelling, punctuation and grammar.)
Overall we found that the children quite enjoyed using CEL. Indeed, the most common response to “Name 3 things that you don’t like about CEL” was the statement that there was nothing they didn’t like about CEL.
We conclude by first making several observations and then discussing current and future work.
Overall, we observed that the children would rather play with each other than with players external to the computer lab (either human or artificial). However, when external partners were available, the children did select external partners, both humans and software agents. This is important because in order to sustain an on-line community like CEL, some members must be present all the time. This result encourages us to build sophisticated software agents in our future work, as artificial playmates who can fulfill this omnipresent role.
In examining which players the children chose to interact with, we observed few trends. There was no gender disparity: girls were just as likely to engage boys as other girls, and vice versa. When grouped by age, we found that the older children were more likely to initiate interactions with a player that they didn’t know. There was no correlation between invitation and ability; and it was more common for friends to invite each other, regardless of ability.
For most of the children, participation in CEL represented their first experience interacting with other children on-line. Typically, activities in school or at home involving computers mean multiple children sharing one machine. With CEL, each child sits at his own computer, so everyone can be an active participant in the learning activity. This feature makes CEL very attractive for children, as well as teachers who favor constructive learning environments.
Our latest work involves building new games to support complex activities. We highlight two of them. Monkey (figure 12) is a collaborative word game in which players are given one long word and they work together to find as many smaller words as they can, using the letters from the long word. The long word is referred to as the “monkey’s word” and players “monkey around” with the letters in the monkey’s word to create new words.
Figure 12. The game of Monkey
Loois (figure 13) is a collaborative construction game in which players work together to create structures out of building blocks. This is a turn-taking game. The player who initiates the match goes first. He selects a block from a bank of building blocks and uses his mouse to drag it onto the building area. When he releases the mouse, his move is sent to his partner. Every move is checked for structural integrity, using a physical simulator for Lego developed by Pablo Funes (1998). If any blocks are deemed unstable, they are highlighted in black in the building area. When the players have finished constructing, they may print out a schematic, containing plans for building the structure they have designed on-line.
Figure 13. The game of Loois.
This process promotes transference of information from the virtual world to the physical world and helps teach students about visualization, projection and dimensionality. For young children, learning how to assemble physical structures by following paper instructions is a valuable skill. Also, because the children design the structures themselves, they may more easily understand the relationship between the elements of the structure as they are represented on paper (and/or on a computer screen) and their physical instantiations. As well, this game serves to introduce children to the field of computer-aided design.
In future work, we are building an expanded version of this game, where children can work together to design entire cities. The buildings in the cities can be created by anyone who plays the game, so participants in different places can contribute to the cooperative project. Children can learn from their peers by observing structures built by others. A web page can illustrate all the buildings in the city and their locations in relation to each other. Visitors to the web page can print schematics of selected buildings. In this way, children all around the planet can reconstruct the same city, physically, in their own classrooms.
As an enabling platform, CEL could host a wide range of activities for learners and experiments for researchers. While CEL eases implementation of typical studies, performed within single homogeneous classrooms, the system also facilitates more comprehensive studies, across heterogeneous groups of participants. Imagine pairing two students with similar skills but disparate ages, genders and locations. An advanced nine-year-old girl in Boston could interact with an average fifteen-year-old boy in London, and both could benefit because the logistical constraints and social ramifications that would normally surround such an encounter in a physical setting are lost in the virtual community that CEL provides.
Please visit us at http://www.demo.cs.brandeis.edu/cel.
The authors would like to acknowledge John Abercrombie, Pablo Funes, Robert Gebhardt, Travis Gebhardt, Matthew Hugger, Louis Lapat, and Maccabee Levine for implementation help. Special thanks to Tom Banaszewski and Jackie Kagey for their support during the pilot study, to Pete Downey at the University of Arizona for hosting the first author during the completion of this article, to Alan Blair for constructive comments and to Suzanne Rozier for her willingness to test anything and everything. This work was supported by the Office of Naval Research under N00014-98-1-0435.