The Mathematics performance of students in the UK is neither stellar nor disastrous. On the 2004 TIMSS, students in England scored significantly lower than those in such countries as Singapore, Hong Kong, Japan, Belgium, and the Netherlands, but ahead of pupils in countries such as the US, Australia, and New Zealand. These results do not justify complacency. Success in Mathematics is believed to be a key factor in countries’ long-term economic success, and there is no reason that students in the UK could not achieve as well in this subject as those in such similar countries as the Netherlands and Belgium. However, in order to improve standards of performance in this subject throughout the UK, the quality of teaching and learning will have to improve on a broad scale. In the US, a recent distinguished panel of educators carried out a detailed review, and concluded that real improvement in Mathematics will require changes in daily teaching practices, not just improved curriculum or standards (National Mathematics Panel, 2008). The same is true in the UK, or in any other country committed to improving Mathematics outcomes A recent technological development in the UK may offer a means of making meaningful replicable improvements in pupils’ Mathematics performance. This is the appearance and widespread adoption in the educational charity, the United Learning Trust of Electronic Response Devices (ERDs) typically used in conjunction with interactive whiteboards, which are also used extensively across the group. This study examines the effectiveness of these devices in raising attainment in Mathematics.
The Mathematics performance of students in the UK is neither stellar nor disastrous. On the 2004 TIMSS, students in England scored significantly lower than those in such countries as Singapore, Hong Kong, Japan, Belgium, and the Netherlands, but ahead of pupils in countries such as the US, Australia, and New Zealand. These results do not justify complacency. Success in Mathematics is believed to be a key factor in countries’ long-term economic success, and there is no reason that students in the UK could not achieve as well in this subject as those in such similar countries as the Netherlands and Belgium. However, in order to improve standards of performance in this subject throughout the UK, the quality of teaching and learning will have to improve on a broad scale. In the US, a recent distinguished panel of educators carried out a detailed review, and concluded that real improvement in Mathematics will require changes in daily teaching practices, not just improved curriculum or standards (National Mathematics Panel, 2008). The same is true in the UK, or in any other country committed to improving Mathematics outcomes (see Askew, Brown, Rhodes, Johnson, & Wiliam, 1997; Ma, 1997).
A recent technological development in the UK may offer a means of making meaningful replicable improvements in pupils’ Mathematics performance. This is the appearance and widespread adoption in the United Church Schools Trust of Electronic Response Devices (ERDs) typically used in conjunction with interactive whiteboards, which are also used extensively across the group. An interactive whiteboard is a large screen that makes it possible for teachers to manipulate and present to entire classroom groups anything that can be shown on a laptop or desktop or designed using the software accompanying the whiteboard. An electronic response device (ERD) is a small handheld unit that enables all students to indicate a given answer in response to any question posed by the teacher or written on the whiteboard. ERDs allow students to key in their own answers using a keypad similar to those on cell phones, freeing the devices from the limitations of multiple choice assessments. Teachers can display students’ responses immediately on a whiteboard in a variety of formats, from summary graphs to displays of individual students’ responses. In theory, ERDs provide teachers with a powerful tool to accelerate student achievement. A great deal of research has established that providing frequent formative feedback, to give both the teacher and the students themselves immediate indicators of students’ current levels of understanding and that of the class as a whole, can have a substantial impact on student learning. Studies in the UK (e.g., Black & Wiliam, 1998) and the US (e.g., Natriello, 1987; Crooks, 1988; McMillan, 2004) have shown that frequent formative assessments in daily classroom instruction can accelerate students’ learning by several processes:
- Giving teachers immediate information on students’ learning, so that they can regulate the pace and content of lessons according to current levels of mastery and identify students in need of additional help;
- Giving students feedback on their own learning, enabling them to regulate and evaluate their own learning efforts;
- Giving students routine opportunities to participate actively in lessons, a key precursor to achievement success (e.g.; Good, Grouws, & Ebmeier, 1983);
- Giving students routine opportunities to see how their peers are solving problems and incorporating effective learning approaches used by peers (Slavin, 1995);
- Motivating students to learn academic content and skills by giving them a stake in the outcomes of instruction, as students are more interested in academic content when they have had to take a public position or give a public answer.
The widespread adoption of electronic response devices creates new opportunities to help teachers incorporate various formative assessment strategies into their daily lessons. However, the use of ERDs has never been evaluated in a rigorous experiment. We therefore carried out a randomised experiment to evaluate the use of ERDs in Key Stage 3 (years 7-10) Mathematics. This was conducted as a joint research project between the United Learning Research Institute (the research arm of the United Church Schools Trust) and the Institute for Educational Effectiveness (University of York).
Detailed Research Questions
1. What is the effect of the use of Electronic Response Devices on the Mathematics achievement of students in Years 7-8, in comparison to students taught without the use of ERDs?
2. What is the effect on student learning of the extent of use of ERDs within experimental groups using these devices?
Research Methods: Subjects and Design
The subjects were approximately 60 Year 7-8 classes in ten two-form secondary schools and academies in England. Schools already had in place interactive whiteboards and were willing to give optional SATs (standardised national Mathematics tests). Teachers were asked to volunteer to participate. At each year level, classes were paired based on achievement levels. One paired class was randomly assigned to use electronic response devices for 12 weeks starting in the spring term 2009, while the second class served as a control. In autumn term 2009, the control teachers received the ERDs and training, making this a delayed treatment control group design.
- Electronic Response Devices.
Experimental teachers were given ERDs to use with all of their students. They were given a half-day specific training in the effective use of the devices in Mathematics, as well as monthly follow-up visits to assess implementation and give advice.
Control teachers were asked to continue using their usual teaching methods during the experiment. After the experiment was over, these teachers received ERDs and training in how to use them.
Measures and Analyses
The measure of Mathematics learning was the optional SATs given routinely to most students in England. These were was transformed within grades into z-scores to facilitate pooling across year levels. Data was analysed using Hierarchical Linear Modelling (HLM; Raudenbush & Bryk, 1998). Students were nested within classes, and degrees of freedom were based on the number of classrooms. Using Optimal Design with an average of 20 pupils per class and intraclass correlation of 0.10 and a pre-post correlation of +0.70, an effect size of 0.25 was detected with 87% certainty.
Each experimental and control class was observed by trained observers four times over the course of the study to learn how they differed in terms of teacher and student behaviours. An observation protocol adapted from that developed by Pianta (Pianta, 2009) was used.
ERD devices and software were sourced on loan at no cost to the project from Promethean to enable all participating students to have one.
The proposed research produced important outcomes in two major arenas. First, it carried out a rigorous randomised evaluation of a theoretically interesting intervention, the use of Electronic Response Devices to provide ongoing formative assessment to students. Second, as the treatments were successful, it provided practical educators, head teachers, and policymakers with information on a readily replicable treatment. The design enabled us to find out who benefits from the use of ERDs under what implementation conditions, and on which subskills.
Patterns of Implementation
Drawing on the evidence from support visits, evaluation visits and teacher interviews, reviews of implementation were conducted regularly by the research team over the duration of the study. Implementation was found to be variable within and across schools.
While the implementation period was to start immediately after the administration of the pre-tests in early January 2009, implementation was significantly delayed in several cases, was inconsistent in some cases, and did not meet the requirement of the study in others. Reasons for variability in implementation included the following:
- Teachers re-assigned to teaching groups not in Year 7 or Year 8
- Insufficient training in the use of ActivExpression
- Limited access to the ActivExpression devices
- Teacher lack of confidence in introducing co-operative learning
- Perceived incompatibility with curriculum content
- Other curriculum development priorities
The research project required that the ERDs would be used in most if not all Mathematics lessons during the implementation period in the intervention classes. In some cases, ERDs featured in less than half the Mathematics lessons taught to classes assigned to the intervention.
The research study required a high level of implementation fidelity; there was an expectation that the combined use of ActivExpression and the co-operative learning strategies introduced in the training would feature in the Mathematics lessons of classes assigned to the intervention. In some cases co-operative learning strategies featured insufficiently in lessons. Similarly, in some cases ActivExpression was under-used or not used at all.
Review of patterns of implementation was conducted prior to administration of the post-test measures. Levels of implementation were evaluated separately for the use of ActivExpression, the use of co-operative learning strategies, and for teaching of Mathematics in which the two components were combined as required by the study and exemplified in the training, support visits and resources. The review suggested that actual implementation was limited to a subset of 6 classes of the 12 classes assigned to the intervention. In the remaining 6 classes, implementation was considered to be developmental but insufficient to contribute to a valid and reliable evaluation of the intervention.
The composition of the intervention and control subsets was as follows:
The exclusion of classes in which implementation of ERDs was not adequate made the random assignment design difficult. Instead, remaining classes were matched with control classes at the same level and with similar scores, as shown above. Intervention and control classes were matched within the school subsamples.
Outcomes on Optional Tests of Mathematics (in z-scores)
|Independent Schools||N (classes)||Pre||Post||Adjusted Post||Effect Size|
|Use of ERDs (SD)||77 (6)||+0.14
The findings of the evaluation of the use of ERDs were mixed, but a great deal was learned about the programme and how it can be improved in the future.
Among the Independent Schools, the implementation of Activexpression was generally very good. Most teachers were sophisticated with technology, used interactive whiteboards well, and in some cases were familiar with co-operative learning.
Implementation ratings were made on a 5 point scale from 0 to 4. Four of the Independent School teachers had implementations rated as “3” and two recorded ratings of “4”.
Some School teachers reported that team work and co-operative learning strategies made a positive difference in the engagement and learning of weaker students. They also reported that students were able to work better with harder conceptual material
Conclusions and Recommendations
The results suggest that in the Independent Schools in the present study, the use of ActivExpression response devices combined with co-operative learning strategies impacted significantly on the Mathematics achievement of students in comparison to students taught without ActivExpression.
In order to secure future improvement of the programme, the following recommendations are made:
- The programme should start at the beginning of the school year, to give teachers and students more time to learn the procedures.
- More prepared flip charts should be provided.
- Support high levels of implementation with increased coaching and leadership support.
- Develop a culture of team work and students’ working together to help each other to succeed in learning.
- Provide additional training and practice for teachers in the use the learner response devices and co-operative learning strategies prior to implementation in lessons.
- Provide more time for teachers to establish an appropriate level of implementation, and for teaching and learning strategies to become embedded prior to any future evaluation.
- Support implementation by integrating the programme into teachers’ existing schemes of work and lesson plans, and taking account of opportunities for assessing pupil progress (APP).
Activexpression continues to be evaluated in UCST schools, and although future implementation problems may arise, the classes are off to a good start. Instead of comparing classes within schools, schools are now implementing the programme with all their Year 7 and Year 8 students, so teachers can help and support one another. Training placed more emphasis on co-operative learning, and most classes are doing a good job with this element. Problems with registering and using ERDs are being solved. A broader array of flip charts has been provided, and teachers are finding them useful. The 2009-2010 evaluation results will be available in autumn, 2010, but based on our observations we expect that schools using the programme will obtain even better outcomes than those found in the 2008-2009 evaluation.
Askew, M, Brown, M, Rhodes, V and Johnson D (1997). Effective Teachers of Numeracy in Primary Schools: Teachers’ Beliefs, Practices and Pupils’ Learning. British Educational Research Association Annual Conference, York. Black, P. & Wiliam, D (1998a) Assessment and Classroom Learning. Assessment in Education, 5(1), pp 7-71. Crooks, TJ (1988) The Impact of Classroom Evaluation Practices on Students, Review of Educational Research Winter, 58 (4), pp. 438-481. Good TL, Grouws, DA and Ebmeier H (1983) Active Mathematics Teaching, Longman, New York. Effects of Electronic Formative Assessments on Achievement in Years 7-8 Mathematics . ACEC2010: DIGITAL DIVERSITY CONFERENCE 6-9 APRIL 2010, MELBOURNE, AUSTRALIA Ma, X (1997). Assessing the Relationship between Attitude toward Mathematics and Achievement in Mathematics: A Meta-Analysis. Journal for Research in Mathematics Education, 28 (1), pp. 26-47 McMillan, JH (2004). Understanding and Improving Teachers’ Classroom Assessment Decision Making: Implications for Theory and Practice, Educational Measurement: Issues and Practice, 22(4) pp 34 – 43 Natriello, G (1987). The Impact of Evaluation Processes on Students. Educational Psychologist, London. Pianta, RC (2009). Conceptualization, Measurement, and Improvement of Classroom Processes: Standardized Observation Can Leverage Capacity, Educational Researcher, 38, (2), 109-119 Raudenbusch, R and Bush, SW (1998). Hierarchical Linear Modelling, Congdon, Chicago. Slaven, R (1995) Learning to Cooperate, Cooperating to learn, International Association for the Study of Cooperation in Education, New York.