National literacy and numeracy tests will now have ‘high stakes’ attached to them as a result of the decision of Australian education ministers, at the initiative of the Rudd Government, to publish the results of individual schools.
It means that league tables are now inevitable in Australia. This will put schools under enormous pressure to maintain reputations and enrolments. The future of some schools will also be threatened because the Prime Minister has stated that sanctions will be applied to schools that don’t improve their performance.
The high stakes attached to school results creates a strong incentive for schools to improve their ranking on league tables by rigging their results. Long experience with reporting school results and league tables in England, the United States and elsewhere shows that schools respond by manipulating their results to improve their results, ranking and reputation. Rigging school results is a much easier and quicker way of improving school performance rankings than the hard work of effective school improvement.
There is now an extensive literature on the different methods schools use to manipulate their results and, thus, their ranking against other schools. They include:
- Reducing the participation of low achieving students in tests used to measure and report school results;
- Increased resort to special dispensations for students doing assessments and tests;
- School-sanctioned cheating;
- Using admission procedures to enrol high achieving students; and
- Change teaching and curriculum practices.
Reduce participation of low achieving students in tests
Schools can resort to several ways to exclude low achieving students from tests used to report and rank school performance. They include:
- Increasing classification of low achieving students as special education students who can be exempted from tests;
- Increasing suspensions of low achieving students or encouraged absences on test days;
- Holding students back in non-tested years; and
- Encouraging students to drop out or leave a school before key graduation exams or assessments.
Several studies have demonstrated that the placement of students in special education categories increased with the introduction of reporting on school outcomes.
For example, following the introduction of the Florida Comprehensive Assessment Test, there was a significant increase in the proportion of low performing students and students from low socio-economic backgrounds re-classified into disability categories exempted from the reporting system.
Exemption rates were also much higher in high poverty schools than more affluent schools, indicating a greater tendency for low performing schools to exclude students from tests as a way of increasing their reported outcomes.
A similar pattern in exemption rates has occurred elsewhere following the introduction of school reporting, including in Chicago, Texas and North Carolina. There is also evidence of increases in special education placements in England with the introduction of school league tables.
In Australia, students with severe disabilities or students from a non-English speaking background who have been learning English for less than one year are eligible for exemption from the National Assessment Program Literacy and Numeracy (NAPLAN) tests which will be used for reporting school results. Thus, there is some potential for schools to use this provision to exclude students from the tests.
Other overseas studies also show that schools alter the test-taking pool by strategically assigning long suspensions to low-performing students subject to disciplinary action near the test-taking period. These studies show that this response is stronger for students in tested grades. For example, a study of Florida schools demonstrated that many re-shape the testing pool of students by use selective discipline in response to accountability pressures.
Another common response is to hold students back in non-tested years so as to better prepare them for the tested years. There is also evidence of schools encouraging low achieving students to leave the school or to drop-out of school so as to maintain good graduation rates.
Some schools remove students from secondary school courses whose outcomes are used to compile school results. For example, a common practice in England is to remove low performing students from academic courses and put them into less demanding vocational courses whose results are not used in assessing school performance.
Increased resort to special dispensations
Schools are able to give students in particular circumstances special dispensations for exams or assessments. This includes allowing alternative assessments, extra time for exams or submission of assignments, exemption from some assessment, sitting tests at alternative times
There is evidence from several systems that schools increasingly resort to these special help provisions for special education students and English language learners during tests so as to increase school results. For example, the New York City school reporting system, so admired by Julia Gillard, has encouraged schools to use special dispensations for students during tests to improve their results. Last year’s revelations about excessive use of dispensations for the HSC exam by some NSW elite private schools to boost their results is a harbinger of what can be expected here.
Cheating
School results can be manipulated by cheating. The high stakes of reputation and status associated with reporting school test results create incentives for teachers to help students with answers in tests and change answers. This is an endemic problem in the UK and US where school results are published.
Cheating can take various forms. Pre-test cheating occurs where teachers alert students to test questions or they use access to the test to prepare students for particular questions. Cheating during tests occurs where teachers directly help students with their answers during tests or allow students to cheat. Post-test cheating involves teachers changing answers before submitting test sheets for marking.
Researchers concur that the higher the stakes, the more likely are schools and teachers to cheat. Education departments across the US are spending millions of dollars in trying to monitor and deter cheating that has become epidemic in some places.
Even with the wide availability of security measures, some researchers are pessimistic that cheating can ever be completely eliminated as long as someone has something to gain from a high score. They say that schools and teachers, rightly or wrongly, feel that they are being judged on the basis of these instruments and some of them cave in and do things they shouldn’t.
Controlling school admissions
An easy way for schools to improve their performance is to replace low achieving students with those who generate better test results. This involves actively poaching, or ‘cream skimming’, high achieving students from other schools while denying access to low achieving students.
In the UK, league tables have led to the introduction of selective entry into high ranking schools as schools attempt to recruit higher attaining students in order to sustain their ranking. Formal processes are used to select students for ‘ability’, ‘aptitude’ or ‘motivation’ and informal or ‘covert’ selection processes are used to discourage ‘undesirable’ students. The latter include the less ‘able’, children with emotional or behaviour problems, students from low socio-economic communities, children with learning difficulties and other special needs.
Even if schools do not engage in ‘cream skimming’ of high achieving students they may deny entry or fail to provide programs to students whose special learning needs may result in lower average test results. Rather than ‘cream skimming’ students they ‘crop-off’ services to students whose language or special education needs make them more costly to educate.
Expelling low achieving students for disciplinary breaches also offers a quick and easy way for schools to manage ‘problem’ children and remove those who might perform badly in tests.
Change curriculum and teaching
Another way to improve school results is to devote more time to the tested subjects and test preparation at the expense of other subjects. In these circumstances, schools achieve an increase in the performance in the subjects tested at the expense of others not included in assessing school performance.
In the United States many schools have directed more resources into the tested subjects of literacy and maths while untested subjects such as science, history, social studies, arts, physical education and health are given much less time. Arts and cultural activities are cut back considerably and excursions are also being dropped. Many schools in England suspend the broader curriculum in the lead-up to the assessment period.
Another effect is to turn classroom experience into test preparation. Weeks and months can be devoted to test preparation at the expense of other parts of the curriculum and other learning areas. Focus on practice tests is a priority now in many English and US schools. This takes the form of repeated drills and practising on test items with the aid of test-prep booklets. This can inflate test scores by detracting from teaching in other subject areas and by teaching skills that are limited to very specific tests, such as in answering multiple-choice questions.
Another common practice is to increase teacher time and resources devoted to students performing just below benchmarks (the ‘bubble kids’) at the expense of very low achieving and high achieving students. These students are referred to as the ‘bubble kids’ because their scores form a bubble just below the passing score. Improving their score is seen as the most efficient way to raise a school’s average score or the proportion of students achieving a benchmark.
Test score inflation
Several of these strategic responses to high stakes tests artificially inflate school results. This means that school average results and gains are larger than the true level or change in student learning. Moreover, the extent of test score inflation is highly variable between schools as some schools manipulate there results more than others.
This has serious implications. School results then provide misleading information to parents and the public. It not possible to tell which results are legitimate and which are bogus. It makes it difficult to identify effective schools and programs. It also undermines the goal of monitoring trends in student performance. In effect, it corrupts the whole system of national testing.
Daniel Koretz, Professor of Education at Harvard University, says that test score inflation is the “dirty secret of high stakes testing”.
“Scores on the tests used for accountability have become inflated, badly overstating real gains in student performance. Some of the reported gains are entirely illusory, and others are real but grossly exaggerated. The seriousness of this problem is hard to overstate. When scores are inflated, many of the most important conclusions people base on them will be wrong, and students – and sometimes teachers – will suffer as a result.” [Measuring Up: What Educational Testing Really Tells Us, p.233].
Conclusion
An inevitable outcome of reporting school results is that the national testing system will be corrupted by the strategic responses that are now commonplace overseas where standardised tests are used to rank, reward and punish schools.
Corruption of test scores occurs in two broad ways. One is to distort the measure itself by cheating, increasing the use of special dispensations for tests or by increasing the time devoted to areas that are subject to testing. The other is to undermine the sample on which the test is based by finding ways to exclude lower achieving students.
The decision of education ministers to report school results has opened up national testing in literacy and numeracy to the operation of a well known phenomenon in social science research called Campbell’s law. Campbell’s law states:
“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor”.
The evidence from overseas is that using school test scores for public accountability purposes invokes Campbell’s law.
Trevor Cobbold