The Australian Curriculum, Assessment and Reporting Authority (ACARA) has taken on the role of King Canute in the vain hope of stemming the tide of cheating on national literacy and numeracy tests. It has just released an edict to principals and teachers that “cheating is not permitted”.
The edict is being presented as ACARA’s new tough approach to cheating. However, it has little prospect of deterring cheating. The only threat of action is that allegations of cheating will be investigated. This is likely to prove as ineffectual as investigations of cheating in the United States over the past 20 years which have utterly failed to quell cheating and rorting of school results. Every year brings a new round of cheating incidents in the US despite innumerable investigations and staff firings.
ACARA’s prohibition on cheating is part of a code of conduct it has issued for educators to follow in administering the national tests to be conducted in May. The code says that providing unauthorised assistance to students during the test is inappropriate and that the content of tests must not be disclosed prior to the tests [ The Advertiser, 15 March 2011, The Herald-Sun, 23 March]. It also says that any attempt by school staff to unfairly or dishonestly manipulate test results is inappropriate and that dishonest and inappropriate practices will not be tolerated.
The code of conduct has been issued following widespread allegations of cheating in last year’s national tests in New South Wales, Queensland, South Australia, Western Australia and Victoria. Last year, authorities launched investigations into 16 incidents of cheating in NSW [ The Daily Telegraph, 18 March 2011]. In Queensland, there were 23 allegations of cheating by teachers of which eight were dismissed, seven have been referred to disciplinary action including referral to the Crime and Misconduct Commission and several are still under investigation [ The Courier-Mail, 9 March 2011].
The allegations concerned giving students clues to answers during the tests, helping students with answers, using posters on classroom walls to help students, changing answers after the tests, revealing test questions to teachers before the tests were taken, and excluding low achieving students from the tests.
These incidents came in the first year of My School. Before My School there were no reported incidents of cheating on the national literacy and numeracy tests because school results were not published. Now, there is such pressure from education departments on schools to improve their results that some teachers and principals inevitably succumb and resort to cheating and rorting.
The President of the Victorian Association of State Secondary Principals, Frank Sal, said the new code of conduct was unlikely to make a difference to schools that chose to cheat.
Once we have such high stakes of testing with such publicity on it, there will be different levels of preparation by different schools and in some circumstances one can probably expect some form of cheating will probably occur. [ The Herald-Sun, 23 March 2011]
This has long been the case in the US, where testing irregularities abound. Last month USA Today [6 March] and some partner newspapers reported over 1,600 examples of inexplicable test score gains in schools that testing experts say suggest cheating. The examples were compiled from school results in Arizona, California, Colorado, Florida, Michigan, Ohio and Washington DC. This follows one of the largest cheating episodes ever in the US last year, where nearly 400 schools in the state of Georgia were under investigation for changing student answers on test sheets or were facing increased monitoring during tests.
The analysis of Ohio’s test scores for grades 3-8 from 2005 through 2008 showed unusual spikes in grade level scores at an average of 87 schools per year [ Cincinatti Enquirer, 6 March 2011]. The analysis of Michigan scores for 2007 to 2009 found 83 schools with very large increases in test scores [ Detroit Free Press, 6 March 2011]. In Arizona, 93 schools were found to have made statistically improbable increases in one year that were likened to a “miracle in a bottle” [ The Arizona Republic, 6 March 2011]. The analysis for Colorado found 69 schools where students posted unusually dramatic growth in test scores [ Education News Colorado, 6 March 2011].
Cheating appears to be widespread in the US despite all the investigations. In 2009, a survey of public school teachers in Chicago by the Chicago Sun-Times (29 August 2009) and the teachers’ union revealed that one-third of all teachers had been pressured in the last year by principals and boards to change student grades. Twenty per cent said that they had actually raised grades under this pressure.
Last year, a survey of 3,000 Arizona educators by Arizona State University researchers found that one per cent of teachers admitted to cheating outright on high-stakes tests, including pointing to correct answers or asking low-achieving students to be absent on test days [ The Arizona Republic, 6 March 2011]. Seven per cent admitted to what researchers call second-degree cheating, including coaching students during testing or encouraging them to redo problems. Eighteen per cent admitted to third-degree cheating, such as leaving materials such as multiplication charts on classroom walls.
All this is still going on 20 or more years after public reporting of school results was first introduced.
Testing experts say that because there is such pressure on teachers, principals and superintendents to produce better school results there is little incentive to fully investigate cheating. One expert, Gregory Cizek, a professor at the University of North Carolina-Chapel Hill who studies cheating, told USA Today [6 March 2011] that no one has incentives to vigorously pursue testing irregularities. “In fact, there’s a strong disincentive”, he said.
There is already evidence of this in Australia. One Victorian education department official recently resigned in protest over doubts about the validity of an unusual increase in the results of a school in his district and the “lack of rigour” in the department’s investigation [ The Age, 11 February 2011]. It has also been alleged that the South Australian Department of Education tried to cover up the extent of cheating in one school where tests were widely circulated to staff early, clues given to students during the tests, answers changed and students were given additional time to complete the tests [ The Advertiser, 8 March 2011].
If ACARA is serious about protecting the integrity of the NAPLAN tests as its spokesman told The Herald-Sun (23 March 2011], it has to do more than play King Canute. At the very least, it should undertake annual audits of school results to look for large spikes from year to year. Statistically improbable school results should be sufficient grounds to trigger an investigation.
Professor Brian Jacob, co-author of a path-breaking study of cheating in Chicago schools, says that large year-to-year jumps in test scores by an entire grade should be enough to raise red flags. Such fluctuations by themselves do not prove there was cheating, but Jacob says they offer “a reasonable way to identify suspicious things” that should be investigated [ USA Today, 6 March 2011].
The methodology used by USA Today in its recent investigation offers ACARA one place to start. It compared year-to-year changes in test scores and singled out grades within schools for which gains were three standard deviations or more from the average state-wide gain on that test. In layman’s language, this means the students in that grade showed greater improvement than about 99.9 percent of their classmates state-wide.
This methodology is widely recognized by mathematicians, psychometricians and testing companies. Professor Cizek says even the most powerful curriculum changes or educational interventions usually produce gains of only a quarter or half a standard deviation. “None of our educational interventions – none – produce three standard deviation gains,” he says, “We’ve never seen that” [ USA Today, 6 March 2011]
Thomas Haladyna, a professor emeritus at Arizona State University, says that test gains of three standard deviations or more for an entire grade are “so incredible that you have to ask yourself, how can this be real?” [ Cincinatti Enquirer, 6 March 2011] It is so rare for a whole classroom or grade to score among the worst in the state one year and then among the best the following year that it would be like finding “a weight-loss clinic where you lose 100 pounds a day,” he said.
ACARA should look at other options as well. For example, several US states now contract testing security companies to do random erasure analysis of test papers of schools. This involves computer scanning to identify high numbers of changes of answers from wrong to right. Data-forensics software which uses sophisticated statistical techniques to detect cheating has also been developed by testing security companies [ The Economist Online, 3 March 2011].
Some US school districts randomly rotate teachers to different classrooms during the testing period so that teachers do not supervise tests for their own students. However, this may simply transfer the pressure to cheat to other teachers because school reputation and the careers of colleagues are on the line.
Other US school districts have resorted to independent monitoring of the tests. For example, this month the Baltimore city school district placed independent test monitors in every school taking the state tests over coming weeks [ The Baltimore Sun, 8 March 2011]. This may well be the most effective way to minimise cheating as long as school results continue to be published.
Without effective deterrence measures, the tide of cheating can only increase because My School is now the maker and breaker of reputations and careers. Education departments are also putting principals under incredible pressure to improve school results. School NAPLAN results now dominate departmental regional planning sessions and figure prominently in the evaluation of principals’ performance. This pressure feeds down to individual classroom teachers and causes some to cave in.
More cheating will undermine public confidence in test results and school reports. Parents will not even be able to be confident about the accuracy of their children’s results, let alone the school reports on My School. The integrity and accuracy of My School will come under increasing question as it becomes even more unreliable as an indicator of school quality.
ACARA and education departments are digging their own grave for NAPLAN and My School. They have created the pressures that lead to cheating which, in turn, will inevitably undermine public confidence in the tests and school reports. This will be hastened by the failure to put effective deterrence measures in place. Edicts not to cheat and rort will be no more successful in stemming this tide than was King Canute.