Classroom Conundrums: The Use of a Participant Design Methodology
In developing a ‘classroom of the future’ for primary children, several interwoven aims (technological, cognitive, and social) have been embedded in this project’s conception. A whole classroom approach is taken to knowledge management. Co-operative technologies are married to intelligent ones such as ‘anthropomorphic’ agents. High technology interfaces (a large 50 inch touchscreen and Wacom PL-300 tablets) are used in real classrooms, with applications designed to encourage literacy and story writing. These applications share a common desktop especially designed for small children who are able to share and jointly create multimedia stories and exchange ideas, text, pictures and sound.
Figure 1.The NIMIS Classroom in Glusburn
Central to the project’s aims is the smooth interaction between human and electronic communication, the digital complementing and facilitating the human and supporting flexible and wide-ranging human interaction. Across the project, evaluation of these aims exploits the flexibility and contextually differentiated nature of participant design, and the claims approach mean that the different contexts and time scales of software development can be addressed individually.
Designing a system through prototyping requires ongoing evaluation so that improvements can be made. At Glusburn, children and teachers were involved in the prototyping process from the outset. Children were interviewed for their thoughts on how they might be supported in their story writing. Teachers too made many suggestions and were involved in the design of the system and the classroom layout. Regular meetings were held to ensure that all parties were informed of the progress of the project.
The classroom was installed gradually to lessen the learning burden. Teachers began keeping diaries and keeping records of lessons and examples of work. The teachers were keen to use the system as soon as possible and probably over-used it initially. Though the teachers were supported throughout both technically and educationally, an intrinsic part of the project was to allow them to have control over how they used the system and when.
The class studied is shared by two experienced teachers. One was ICT literate and the other nervous about using ICT. When the new class arrived in September 1999 they deliberately introduced the children (mostly five years old) gradually to the system and various forms of commercial software. However by half-term the children were coping well and the system began to be used for large parts of the day. The first prototype of the story-writing software T'riffic Tales, was installed in November 1999 and most children have now used it several times.
The educational goals of the project are complex, and although we may explain and clarify various identifiable strands of these goals it needs to be recognised that they are inter-related. Separating the cognitive from the social and emotional, for example, is a false division (Damasio, 1994; Goleman, 1995). Excellence in teaching and learning has often been related to the quality of teacher/pupil relationships and the school and classroom ambience (Aspy, 1992; Kyriacou, 1986).
Vygotsky argues that cognitive development is embedded in social relations: “all the higher functions originate as actual relations between human individuals” (1978, p.57). Such relations are not only influenced by cognitive factors but also by instinctual and emotional elements, which give depth and meaning to learning, provide motivation, raise self-esteem and improve the ability to feel for and understand others. Positive relationships where trust is created and success achieved provide the security for children to make those leaps into the unknown that new learning, of whatever nature, entails. A positive, nurturing and enabling atmosphere, which supports all children, provides the model for their own personal development and relationships with others. We believe that flexible classrooms designed to meet children's needs, to encourage a wide range of interaction and collaboration, to enable the co-construction of ideas, presentation of ideas and subsequent reflection, can help to support and nurture children's emotional, social and intellectual development.
It is the inter-related nature of learning, and the myriad daily interactions between individuals and groups, which creates complexity and makes it particularly difficult to evaluate learning in the constantly changing classroom. The nature of the hidden curriculum in the classroom (Rutter et al., 1979; DES, 1989) can also subvert the more visible and tangible aspects of learning and makes any straightforward evaluation difficult. Differing social and academic relationships between a child’s peers and other adults in school make each child’s learning experience unique and therefore difficult to assess. There are four key areas which NIMIS is interested in evaluating: The whole classroom; Collaboration; Interfaces and how they support learning; and Cognitive skills involved in literacy development. This paper examines the first three of these aspects. Literacy development and story creation will be discussed elsewhere, as will the effect of intelligent agents.
The Whole Classroom
We are interested in understanding how this distinctive whole classroom works, and whether the introduction of the carefully designed technology under the theoretical influences mentioned above, in conjunction with a participant design, can improve and augment the interactions and learning. We are particularly interested in the influences that the classroom may have on the class ambience, teacher time and the interactions within the classroom, and whether these have any effect on children’s self-esteem, perspective taking, engagement and learning.
Perspective taking is a complex issue, but research shows its extensive importance in the development of human relationships and the personal, social and moral development of pupils (Hoffman, 1970). Perspective taking enables pupils to understand the world outside of and in relation to themselves. Carl Rogers (1975) and Dorothy Heathcote (1971) argue that we are all born with this capacity to empathise, but it needs nurturing, modelling and developing by others around us. Heathcote claims it is a human skill much underused in the classroom. Research suggests that schools can play a part in developing perspective taking, both by the educational processes they encourage, by the systems they develop (Bottery, 1992), and through human models, in the form of teachers and others who work with the children (Kozeki & Berghammer, 1992).
Large groups in traditional classrooms can diminish self-esteem and restrict learning and the development of quality relationships. Children with low self-esteem are in a perpetual state of comparison with pupils with higher level interpersonal and academic skills. They are also competing for limited teacher attention. Computer based learning systems can provide one-to-one support (Elsom-Cook, 1988) which can raise self-esteem and boost confidence. The NIMIS classroom will eventually follow this route by offering personalised assistance in the form of intelligent agents providing individualised feedback and supporting step-by-step success.
Pupils motivated by the use of technology (Thompson, 1994; Sharples, 1985), by its practical, flexible, and often, exciting potential, are able to take greater control of their learning. The teachers freed up in the same process are released from more traditional classroom management issues and are able to take a more facilitative role, devolving responsibility for learning to pupils. This places them in a more empathic position with pupils, with less need for traditional teacher domination and in a better position to work with and understand individuals, thereby modelling an empathic approach which pupils are likely to imitate. However, the quality and effectiveness of the technology and the training of teachers are important factors in the success of ICT. An irate teacher, struggling with temperamental systems, will find it difficult to model empathy to anyone. For this reason teacher involvement and support was built into the NIMIS project from the outset.
Pupils’ self-esteem is also raised when they have opportunities to share and display their work with other audiences and to take a pride in their achievements. The high quality reproduction and multimedia nature of the NIMIS technology was expected to support high quality outcomes for pupils, and the shared workspaces and large screen were expected to facilitate the display of children’s work along with peer appreciation and assessment.
NIMIS aims to provide an educational environment which can enhance collaborative working. The social constructivist approach to learning stresses the importance of interactivity, of language and communication, to the development of understanding and the higher cognitive processes (Wood, 1988; Vygotsky, 1978). Pupils enjoy working and discussing with their peers (Bennet & Dunne, 1994). Paired and group arrangements during various kinds of investigative, exploratory and creative work can have considerable benefits for pupils (Wray, 1994). Peer tutoring has been recognised as advantageous to both tutor and tutee, although the choice of peer must be made carefully. Sometimes an older pupil from another class can be more successful than a child from the same class in this role (Kyriacou, 1986). Collaborative working also encourages social and inter-personal skills such as listening, responding, awareness of others, turn-taking, perspective taking and the interleaving of skills between people.
We believe that the NIMIS project supports the beneficial nature of collaborative work in several ways, and that the whole classroom approach is an improvement over developing individual applications. Software emerges not as an isolated application but as a contextualised entity through careful observation and analysis of existing learning scenarios. We have sought to support teacher’s skills and experience by developing collaborative technologies and by supporting the normal classroom teaching and learning dynamics through building on existing relationships. This should enhance communication and improve the quality of exchanges through software which supports shared workspaces, electronic communication and the use of hardware which supports group interaction and presentation via a large touch-sensitive screen. The arrangements support pair and group work as well as whole class communication via the touchscreen.
Interfaces and How They Support Learning
We have chosen to concentrate on how the various hardware and software interfaces directly support learning. In this sense, with all the claims we have formulated, it is their educational aspects which interest us most. However, this often involves a relationship with other issues. As a result, some claims appear quite complex. Initially this appeared problematic but, as we have analysed the data in relation to the claims, now appears to be less so. This is because the claims reflect the complexity involved in learning found in real classrooms.
The Pedagogical Claims Method in NIMIS
The complexity of evaluating the use of technology in education is well-documented, and there are both established and evolving schemes to support and illuminate the process (Ehrmann, 1998; Oliver & Conole, 1998). In NIMIS we adapted Carroll and Rosson's participatory design approach (Carroll & Rosson, 1992; Chin et al., 1997). This methodology is organised around the identification and exploration of key scenarios. Additionally, it uses a form of design rationale known as claims analysis (see below for details). We have extended the claims concept to incorporate the pedagogical intentions underlying the design, and termed this pedagogical claims analysis. As a consequence of the use of participatory design, the evaluation has both formative and summative aspects, and the initial claims have been revised and validated throughout the formative prototyping phase.
These claims help support the understanding of the design process and make design decisions explicit. Generating many claims in a complex project also helps the team determine which issues need to be prioritised. Like the Evaluation of Learning Technologies (ELT) approach by Oliver & Conole (1998), the claims approach is flexible enough for varied and complex situations, giving the research teams the flexibility to address their own questions. However, unlike ELT, it offers no specific guidance on which methods or analysis would be most appropriate.
Identifying and Understanding Classroom Scenarios
Initially, observations and video recordings were made of literacy and drama lessons. At an early NIMIS meeting, teachers and researchers worked intensively together to explain and understand the nature of each others schools and curricula. Explanations were given of how stories were created in the different contexts with different age groups, from the earliest formation of words through the creation of simple stories to story dramatisation. Prototypical classroom scenarios were identified by the teachers.
Story creation, one of several scenarios developed, is outlined in Figure 2. The diagram breaks down the scenario into sequenced activity frames, any of which can be part of the process of creating a story. This helped us decipher the classroom dynamics and utilise the teachers’ expertise to reconsider what was happening and how it could be supported or enhanced.
Later, teachers envisioned the ways in which the technology might be incorporated into classroom activities and how it might enhance and improve story-creation. As a result of considerable analysis of existing practices, combined with consideration of the potential new ways of working, a list of pedagogic claims was drawn up which reflected what was believed could happen in the ‘new classroom’.
Figure 2. The Story Creation Scenario
The Nature of Pedagogical Claims
With this method of evaluation, each aspect of the design process is linked to possible pedagogic outcomes, which raises issues and suggests how each particular claim might be checked (see Table 1 for an example). The format is:
Table 1.An Example Claim
During the initial collection of pedagogical claims, children and teachers were consulted and interviewed, and also engaged in low-technology design (Scaife et al., 1997). Such claims represent the understanding of the teaching and learning process. The claims help the project team to focus on issues rather than providing a formal and reliable means of evaluation. As the software and its various features were developed more specific claims were made.
The Evaluation of Pedagogical Claims
We chose to evaluate a subset of the claims. This selection was determined partly as issues increased in significance but also as the educational issues came to the fore. Initially, the claims fell into 7 categories, which arose from the envisaged classrooms and learning scenarios (with some degree of overlap). The first set of 39 claims is categorised in Table 2.
Table 2. Breakdown of the Initial Claims
New and more specific claims were made as the project progressed, with some being revised or rejected. For example, a claim that a fingerprint recognition device might provide an easy and reliable login mechanism for small children was rejected since the technology turned out to be unable to utilise children’s relatively unformed fingerprints.
After the first year, there were 118 claims (Table 3). The original category 3 ‘claims regarding writing conferences’ was subsumed into category 1, which expanded rapidly as aims for the detailed elements of the software became clearer. A category was added for claims about agents. Interviews with children and observations of the software in use led to a greater understanding of what the agent’s role might be. This gradual developmental process led to a greater inquisitiveness about and understanding of the teaching and learning process in a given context. In addition, the claims both make the intentions and means of checking the results explicit.
Table 3. Claims in October 1999
By January 2000, the number of hardware claims increased by four but the rest remained stable. It was clear that the number and specificity of claims could increase indefinitely as the software developed. It was also clear that they could not all be tested. Consequently, we stopped creating new claims and prioritised a subset of claims. Though each learning context had overlapping claims, there were also some which were specific to the applications being developed. Hence the German team were more interested in evaluating the ‘learning to read’ claims, which were more specific to their application. The Portuguese and the English applications focussed on the category 1 ‘story creation’ claims. The claims relating to the whole classroom concept, the role of the interfaces in supporting learning and the hardware and software environments were common to all partners.
For the interim results outlined here, the Leeds team focussed on the claims concerning the whole classroom, collaboration, and how interfaces support learning.
Data Collection Methods and Analysis
Many factors can affect the ambience and mood of the classroom, which in turn affects the interactions and learning. Lessons rarely follow set patterns and, in order to teach contingently (Wood, 1988), teachers need to adapt spontaneously. Therefore the complex issues required in evaluating a claim need a variety of data collection methods (Robson, 1993; Burgess, 1985).
The small numbers involved, and the open-ended nature of story-writing, means that the data is more qualitative than quantitative in nature. Detailed and multi-sourced data collection was undertaken to make it possible to understand the processes more clearly. Patterns and significant features could be triangulated by different collection methods even with small numbers. With two very busy teachers and for the most part only one data collector we chose a variety of methods which were sensitive to the work loads experienced by those involved. We chose to collect more data than absolutely necessary to increase the chance of capturing significant events.
Given that prototyping the software has been continuous through the project, it was also important to identify teacher and pupils’ needs as well as their reactions to the software. This involved considerable researcher time.
Time pressures on the teachers made it impossible to have regular interviews without multiple interruptions. So an electronic diary/log was provided for the teachers to fill in every month. This was used to obtain personal and experienced reactions to the classroom and, through this, to the childrens’ experiences. The diary could be quickly written up, and provided an opportunity for the teachers to reflect on important events during the previous month.
The diary was structured with questions organised around key aspects of the claims. Recently, new questions were added to address new software claims. Towards the end of the project the diaries will be discussed in some depth with the teachers to enable issues to be clarified and explored.
Field notes were made from school visits on conversations with teachers and children. These recorded specific events which were also relevant to the claims, but which also caught instantaneous reactions to events as they arose and gave opportunity for the researcher to reflect on what was taking place.
To gain the childrens’ perspectives, members of the class were interviewed in the pre-NIMIS, the early NIMIS and the later NIMIS classroom to ascertain their reactions to different possible forms of computer support, the current situation and their reflections. Again, these interviews were organised around the issues raised in the claims. The decision to use interviews also reflected the fact that children were not able to produce significant amounts of data in a written form.
We also video recorded many classroom sessions before and after the introduction of the computers. This was intensified after the software became robust enough to be used regularly. These videos were then available for analysis in different ways with transcriptions being made where necessary. We developed a coding scheme (Table 4) for classroom interactions to understand events and examine the various claims we had made.
Video Coding Scheme
For the analysis reported here, approximately 14 hours of video recording were coded in segments obtained by looking for natural breaks or obvious changes in activity or interaction. The coding scheme used three five-point scales to assess features of any situation from the point of view of each participant (see Table 4). These included the quality of the interaction, the degree of control exerted over other participants and the degree of engagement in the task. These codes were applied to teachers, children and any researchers present.
Focussing on the children meant losing information about what was happening on the screen. This made the computer logs a good supplement to the video data.
As the children’s work was being assessed, data was obtained on their progress. We monitored reading ages and national curriculum writing levels to create a fuller picture of literacy achievements, and developed our own story evaluation scheme to assess what was happening in detail in the text itself. We also monitored the assessments and reading ages of a parallel class. Ten children were chosen from each class whose reading ages could be matched with someone in the other class. These children were then chosen for more detailed assessment. Writing assessments were taken before and after using the new software to record and assess any changes. The results of this will be reported elsewhere.
Which Claims, Which Data Collection Methods?
We expected that particular data collection methods would be more relevant for some claims than for others. While partly true, we are wary of advocating any particular method for types of claims exclusively, as we did expect (and discover) that evidence to support claims came from a range of different sources.
Table 4. Video Analysis Codes
To capture the mixture of emotional and cognitive factors that support children’s learning, there is a need for data collection methods which focus on a range of different people’s perspectives, on processes and outcomes, and on recordable and interpretable events.
To attempt a simple mapping of claim to type of data collection method would fail since the claims have always reflected the complexity found in the classroom. A claim about the giant touchscreen might be substantiated by evidence from an analysis of video recordings of lessons with and without the screen, the children’s interviews and responses to the screen, the teachers’ diaries, the researcher’s field notes, the stories produced by the children on the giant screen, the comparison of the log obtained from work with the giant touchscreen’s log with other logs and maybe even from a comparison between children in the class using the screen and those from another class without it.
However, there are some stronger relationships. The evidence about interactions, collaboration and the whole classroom is more clearly found in the video recordings, the field notes, the diaries and interviews. Detailed evidence about how collaboration worked on the software itself may also be found in the computer logs.
The evidence about interfaces and how they support learning could be drawn from the interviews, diaries, the video recordings and field notes.
The evidence about literacy development are mainly found in the standardised assessments, the evaluation of stories the children created, and also in the computer logs which chart precisely how their stories were created. However, if the stimulation and intervention they receive is an important factor in their literacy development, then motivation and support may well be evident on videos and through field notes.
Analysis and Results
Results are based on the evaluation of the classroom from September 1999 to April 2000 relating to claims which focus on the whole classroom, collaboration and interaction, and the way the interfaces support learning. This work includes data from the teacher diaries, field notes, interviews with the children and the video analysis.
Reactions to the Classroom
The teacher diaries, videos, interviews and field notes all testified to the enthusiasm with which the children and teachers greeted and made use of this classroom. Remarks by the teachers describing the classroom as “wonderful” and children describing it as “brilliant” and “really good” were commonplace. One teacher said about the children, “they only have eyes for the computers”, and both teachers explained how the children's excitement for the system often led to problems with them rushing normal work so that they could have more time to work on the computers. The system proved to be so easy to use and so well-loved by both teachers and children that within a few weeks, both teachers were including it in their lessons for large parts of the day.
After a gradual introduction in the first half-term, the computers began to be used for around five hours a day - i.e. most of the available classroom time. The teachers had successfully integrated the novel technology into a normal working classroom and across the curriculum.
Video evidence provided evidence of high levels of teaching performance. In one video, the previously nervous teacher moves easily between traditional and digital modes of teaching. She includes the use of both large and small screens, engaging the whole class at one moment through the clarity and versatility of the large screen and then using the network around the octagonal table to motivate and support low attaining pupils. Emotional excitement on the part of a teacher when using particular media transmits itself to the children and draws them into the learning, increasing interaction and engagement.
The large screen can also be used more traditionally and with a less beneficial effect. Used as a traditional blackboard and with greater teacher control, it appears to result in less active involvement, less engagement and less emotional involvement in learning.
One good example of how the interfaces can support learning is provided by the large touchscreen which is very popular with both teachers and pupils. There is considerable evidence (from videos and diaries) of high quality teaching and learning which maximises the advantages of the touchscreen. Because of the interactivity and the lowered height, children can be actively involved in whole class sessions, easily taking on the ‘teacher role’ (see claim 6.5, Table 1). The adaptability of the children to using computers makes them eager and confident to use the screen. It also allows the teachers to value their skills and give them opportunities to demonstrate features or complete tasks in front of their peers.
Sensitive teacher handling of learning situations, plus the easy ability to operate the screen and alter or make revisions to work allows children to explain and clarify their understanding in front of a group and for others to listen, think and suggest. Even low-attaining children seemed eager to demonstrate their knowledge, with children who find writing difficult leaping up to type in words or draw shapes. Whole sessions (up to 30 minutes at a time) were conducted where the teacher scarcely touched the board, but during which the children took turns to explore or demonstrate software. One example clearly shows a child taking a very ‘overt’ teacher role in which she models encouragement, affirmation and suggestions to other children. There were also one or two sessions with less pupil engagement in which the screen was used more traditionally, with the teacher more in control. These show up clearly in the video analysis and the stance taken by the teacher in this respect can clearly effect the involvement of the children.
Table 5. Claim 6.6
Claim 6.6 (Table 5) is supported through some good examples on the video tape of teachers modelling software or software features, and then allowing children to experiment afterwards on the small screens and the large screen. They also allowed children to show the software first on the large screen before others had a go. The combination of the large and small screen seems powerful in this respect, and at any time teachers can reiterate features again on the large screen when necessary.
Table 6. Claim 6.7
Claim 6.7 (Table 6) is supported through teacher diaries, videos and field notes which show examples of children jumping up and down in pleasure and excitement in front of the large screen, arrays of hands begging to be allowed to have a go, and children drawing and creating stories and using commercial software in great excitement. The first use of the large screen, the first introduction of the story-writing software, produced massive excitement for the whole class. Even nine months on, children could be heard saying, “it’s magic”, when stories or clocks appear at the touch of the finger. Engagement levels are very high in the video analysis in such sessions (see Figures 4 and 5).
Detailed Video Analysis
The positive comments recurring in the diaries, conversations and interviews have recently been supported by a more detailed but preliminary analysis of the video recordings. Some interesting results have emerged.
One exceptionally clear outcome from both videos and diaries is the motivating effect of this equipment in the classroom. Children seem to be highly engaged, talk avidly about what they are doing with other children and adults, and have rapidly taken on the ‘mantle of the expert’ with the computers. This allows them to gain confidence and self-esteem. The more detailed analysis substantiates this to some extent, although we are aware of the limited nature of our data to date.
The Pre-NIMIS Classroom
Both small group and whole class sessions involved very high levels of teacher intervention. Out of fifty segments analysed, many showed high levels of quality in interaction between the teacher and their pupils. Forty-two percent of interactions were at the two highest levels. These high levels were most noticeable in whole class interactions during literacy hour, in which pupils were often highly engaged in learning and enjoying high quality discussions and opportunities to voice their thinking. Moments of intense emotion in stories produced high levels of engagement of the greatest number of pupils. This was also true in the group sessions. High quality interactions were also seen when teachers engaged individuals in one-to-one discussion.
The quality of interaction was lower when the teachers needed to manage the whole class, reorganise seating arrangements, organise large numbers of individuals to different tasks, reprimand children or manage interruptions from incoming children and adults.
When children are engaged on individual activities they have minimal focussed interactions with other children. Interactions are not specifically linked to the learning task, and the teacher is only able to interact individually with a small number of pupils during this time. Consequently, many pupils work independently but with periods of distraction or lack of progress. The degree of engagement varies from child to child and moment to moment. However, the engagement is generally lower in sessions where the teacher is not involved with a large number of individuals, is concentrating on class management rather than learning issues, or is not displaying enthusiasm and excitement about the learning task.
Noticeably in these videos, there were no opportunities for children to collaborate around a specific shared task in pairs or groups without teacher intervention. However, we did observe a few limited opportunities for this at other times in the classroom, in particular when a pair of children worked on the original classroom computer. The social skills of children of this age require considerable teacher intervention and modelling to help them manage cognitive demands alongside interpersonal learning. At the start of the project, the teachers felt that without teacher intervention children would find it very hard to write stories together. The NIMIS project provided an environment in which collaborative work was facilitated and encouraged.
The NIMIS Classroom
With the arrival of the computers the children had frequent opportunities to work around a shared task, both in pairs and in bigger groups, and both on the large touchscreen and the small screens positioned around the octagonal table. They also had opportunities to participate very actively around the large screen as part of whole class activities. These two activities appeared to be qualitatively different from previous activities in terms of the interpersonal skills needed. In conjunction with the engagement and motivation created by the coming of the computers, there appeared to be very high levels of engagement even in collaborative situations. Though these situations were not totally unproblematic in their nature (e.g. there were arguments about control), of the 74 coded segments, over 94% were at the highest levels of engagement, levels 4 and 5.
These segments also tended to be of a greater length than in the pre-NIMIS classroom because they usually involved more interactions with partners while working on a common task. Though we will need to consider very carefully the details of the conditions and particular situations in which this engagement took place, it does seem that the children are very highly engaged in learning when they are working around the computers, whether they are having to cope with additional social skills or not. Their absorption with the task may account for some difficulties with social interaction, but high levels of engagement seem to be present most of the time regardless of other factors.
The few instances where children’s engagement did drop to very low levels were interesting. There were three instances in total where the engagement level dropped to level two. Here the child who lost interest also had a dominating partner with poor quality interaction. Even then the children were very resilient in the more passive role. Two of these three managed to maintain interest despite the circumstances for virtually 40 minutes before appearing to lose interest. The third lost interest after about half that time, and had a particularly unresponsive partner. However, the passive nature of their learning and any resentment building up about inequality might have other undesirable effects. One child working with a very uncompromising and dominant partner in one episode later modelled precisely the same dominant behaviour with a different, more passive partner.
Generally, relationships between partners varied tremendously between high and low control. The equilibrium level of give and take (level three) was rare amongst children. The quality of the interaction was rarely at level five between the children. More typically, this quality occurred when teachers were modelling high quality interactions to the group as a whole, or when teachers or researchers were interacting sensitively with individuals. One example of this rare occurance of high quality interaction, equal control and engagement involved two girls. This was noticeable for the interleaving of support and, to the viewer, looked to be a good example of high quality collaboration. However, even this ended with an argument later.
A rare example of a pair with high engagement, reasonably equal control but poor interaction produced an extremely disjointed piece of work in which the two parts of the story were totally different. Given that this was all a fairly new experience for these children, the results were interesting and analysis of data gathered recently should provide more substantial data since further opportunities to work together may increase children’s collaborative skills.
The videos also showed considerable passing interactions from children sitting around the octagonal table in neighbouring seats, and also from other interested passers-by. These tended to be positive, complementary or helpful as they were not clouded by the control or ownership issue. They tended to confirm the comments of the children in the interviews who felt they were able to help each other with the computers and the work done on them.
Figure 3. Helping around the Table
Figure 4. Teacher Enthusiasm for the Big Screen
Figure 5. Children’s Enthusiasm for the Big Screen
Reliability of Results
Researcher effects also plays a part in this project, especially in the early days of prototyping the software, since we had to cope with technical issues as well as using the video camera more openly. Inevitably, this extra adult support and attention affected the children’s learning and the classroom dynamics. Measuring this would be difficult, but the effect can be seen in the videos, where children pose to the camera and teachers visibly shy away. Often, however, children, teachers and researchers are so absorbed in their tasks that they mainly forget about the camera.
The videos of sessions in which commercial software was used are likely to be a reasonably accurate reflection of how the classroom currently functions. However, in developing the new story-writing software, videos show one to three researchers present. Additional adults in the classroom can have a powerful effect on both individual responses and the general learning climate. The quality of design feedback was enhanced, but the data was made less reliable than it could be. However, this problem will be addressed by subsequent evaluations in which the researchers will take a more background role.
Teacher/Pupil Influence on the Design of Classroom and the Software
Both teachers and children were quickly excited by the possibilities, when faced with a system that seemed both flexible and easy to use. They also soon discovered minor problems and difficulties, which were documented in the data. The teachers immediately adapted the plan of the NIMIS classroom once the furniture had arrived to make it more convenient. They insisted on shortening the legs on the octagonal table, changed the chairs, gave advice about earphones and requested commercial software for the system. They noted that although they found the screen very useful it was still slightly too tall for some of the smallest children in the class, and ideally needed a hood or shade to prevent glare. Also, an infra-red keyboard or a stand for the keyboard would make it much more user friendly.
The new story-writing software has generally been received by the children with excitement and interest. In the initial interviews children had suggested many features that they thought would help them write stories, and many of their ideas (supported by the teacher’s) were inbuilt from the start. These included features such as different pictures to choose from, drag and drop word banks with speech synthesis to help them write, help with ideas for their stories and access to the common words they have on the classroom walls (such as days of the week, colours and common words). Figure 6 shows a story created by one child using the software before she had access to the word banks.
Figure 6. A Pupil’s Story
The NIMIS software has also changed directly as a result of feedback from classroom trials, e.g. the width of scroll bars has altered to take into account usability factors related to the mouse pen and Wacom tablets, and the goal of facilitating collaboration has also led to changes in the collaborative aspects of the software. Children have also requested features which we hope to add to the software (e.g. the ability to resize and flip characters and props).
Contrary to the teachers’ expectations, the children have been able to create stories collaboratively, although not without difficulties. Many of these sessions reveal high engagement. However we need the more detailed evidence from the computer logs and story evaluation to understand better the effect of this software on story-writing and literacy development.
Reflections on the Use of Participant Design and the Claims Approach
Involving teachers in the design process resulted in a more realistic and accurate assessment of the potential of the new technology. Involving children helped provide knowledge about how well their needs were being met. The apparent success and usability of the classroom and software are probably the best testament to the usefulness of participant design.
The children and teacher’s expressed and observed reactions to the classroom were very positive, but in part, this may reflect the fact that they had helped to create this. The feeling of ownership in the process can help to bridge the gap between theory and practice. As a result, the teachers benefited from the project personally and professionally.
Researchers with more experience and understanding of the educational context were able to build relationships with the teachers and communicate effectively. This helped to ensure flexibility in arrangements, and made the collection and interpretation of data easier and more appropriate.
Problems arose when time for communication was limited, sometimes leading to misunderstandings. Benefits accrued the more time the programmers spent directly with the children, which helped them understand how very small factors could affect the software’s usability. There is a need for sufficient time together, having a commitment to improve understanding, and remaining open to each other’s viewpoints and needs. Where these conditions were met, the process was smoother, more enjoyable and more relaxed. Social events played an important part in this process.
Finding time to do this was problematic, especially for teachers and children: the demands of the National Curriculum are extensive, and using supply teachers is expensive. The teachers were very patient, but it was hard to cope with frequent filming, interruptions and consultations as well as the many demands from government and other agencies. Detailed research in the current climate is made harder by days filled with predetermined curriculum and tasks, and, quite often, a predetermined teaching methodology.
Projects using participant design will need to ensure that participants are released frequently from their regular duties to engage in the design process, and program designers should be required to discover the needs of their users by evaluating the software in the classroom. Finding sufficient initial discussion time is also vital.
From a claims perspective, we were able to identify what initially we believed happened during story creation in existing classrooms, what we expected to happen in the computerised classrooms and then what actually happened, by formulating and testing many different claims. The ongoing formulation of claims helped us to identify design decisions and to track changes in the design process as claims were rejected or altered. Each claim contained a suggestion for which data source would probably provide evidence for that claim, but in reality most of them anticipated evidence from multiple sources. This led to complicated cross-referencing during analysis.
There were problematic issues around the creation, adaptation and documentation of the claims as the project progressed. Participants who were not working in an educational context found it difficult to write the claims. Keeping track of such large amounts of complex information was inevitably difficult. The educational claims were the most ‘messy’, but also the ones which were most significant. A systematic store of the claims needs to be set up from the outset with resources allocated in the early stages.
As part of the NIMIS project, we have used various data gathering techniques to examine the whole classroom, collaboration, and how interfaces support learning. The results here concentrate on the initial 8 months of the NIMIS classroom’s working existence. The software is still under development, although we have reactions to the first prototype. Teachers and children are still influencing the ongoing design. Later analysis will cover the period between April and July 2000, and will also draw on computer log data, a range of normative assessments for reading and writing and comparisons with another similar class. This will also involve the detailed evaluation of stories written both by hand and using the software, and will include initial data on the prototype of the intelligent agent.
Unpredictability in the classroom demands multi-faceted evaluation techniques such as those utilised here. Collectively reflecting on the process has resulted in a collaborative progression from design to final evaluation. The claims methodology lends itself to different levels of evaluation, both specific and more general, and supports the design process, clarifying the means of evaluation from the outset. The combination of participatory design and claims analysis provided a method of managing the complex data over the project’s lifetime,
Key aspects of these findings which merit deeper investigation are the significance of the teacher in modelling and facilitating good quality peer interactions, the high levels of engagement/motivation in such a classroom, and the significance of dominance/control issues and quality of interaction issues in both teacher/class and child/child interaction in relation to motivation. Also, the role of the large screen in facilitating active pupil involvement in whole class sessions and the function of the octagonal table and the non-intrusive Wacom tablets in supporting collaboration and learning.
Maintaining high levels of information distribution, engagement and involvement amongst all the participants in a large project is a time consuming aspect of participant design. However, as the results emerge hope the difficulties will have been worth the effort, and that the NIMIS classroom is ‘owned’ by all concerned and continue to be an exciting learning experience for children, teachers and researchers alike.
This work is supported by EU project 29301. Thanks to the NIMIS team, specially to Alex Martins, Keith Holder, Bill Rudling, Judy Robertson, Sally Freytag, Ilona Gates, Dawn Liversidge, Andrew Kellington, Val Ashdown and the teachers and children at Glusburn Primary School.