Sunday August 31, 2008
The Rudd Government’s “education revolution” is looking more and more like an extension of the Howard Government’s school policies. All the same elements are there – choice and competition, reliance on markets, and now public reporting of school results.
The model for the new school reporting scheme comes direct from New York. Julia Gillard has been enthusing about the New York system ever since her audience with the New York Schools Chancellor, Joe Klein. She says she is “inspired” and “impressed” by Klein’s model.
It is a pity that Gillard did not look more closely. She would have seen major flaws.
The New York system produces unreliable and misleading comparisons of school performance and student progress. It is incoherent. It can be used to produce league tables. It fails to compare like with like and it is statistically flawed.
Diane Ravitch, Professor of Education at New York University, a former US Assistant Secretary of Education and advocate of school reporting now says that New York’s school reporting system is “inherently unreliable”, “dubious” and produces “bizarre results”.
Jennifer Jennings from Columbia University describes it as “statistical malpractice”, “a mess”, and based on “highly questionable methods”. The New York Sun columnist, Andrew Wolf, says that it is “an overblown grading system that already seems to be sinking from its own weight”.
New York uses an incredibly complicated scoring system, requiring two 30-page technical guides to explain. It combines a wide range of information on student achievement, student progress, student composition and school features to obtain a school grade of ‘A’ to ‘F’ and an overall performance score out of 100.
The process by which all this information is combined, weighted and assessed involves many highly arbitrary and subjective judgements. According to Andrew Wolf, it involves “a bagful of subjective adjustments, bonus points and bureaucratic discretions”. It is riddled with inconsistencies.
Amongst the most bizarre results of the system is that high performing schools can be assessed as failing and be closed down. For example, the elementary school PS 35 on Staten Island was graded last year as failing even though 85% of its students passed the reading test and 98% passed the mathematics test.
It was failed because its students had shown insufficient improvement from 2006. As a result, it is a candidate to be closed if it fails to improve further.
Julia Gillard says that she doesn’t want “simplistic and silly” league tables, and will only compare schools with a similar student population. This is disingenuous. It is not possible to use the New York model to report student results in like schools without providing the scores for all schools. The New York Times and the New York Post list each school’s grade and overall performance scores. It is a simple matter to rank all schools on their grade and scores in league tables.
Despite what Gillard says about the New York system, it fails to consistently compare like with like. Jennifer Jennings has pointed out that school peer groups include schools with very dissimilar demographic profiles. For example, the percentage of high-achieving Asian students in the schools of one peer group ranges from 1% to 69%. In another, the percentage of low income students ranges from 12% to 94%.
Another major problem is that New York school progress reports do not report measurement errors for school scores and grades. As a result, its comparisons of school performance are likely to be inaccurate and misleading.
Many studies of school performance reporting in England, the US and Australia have shown that a large proportion of school results are statistically indistinguishable when measurement error is taken into account. The problem is magnified for measures of student progress, or ‘value added’ comparisons, where measurement error is inevitably larger.
Astoundingly, Gillard’s preferred model assesses student progress on only one year’s data. Yet, a study published by the US National Bureau of Economic Research shows that 50 to 80 percent of the year-to-year fluctuation fluctuations in average school test scores are random and have nothing to do with school quality. School comparisons of progress over one year are therefore highly unreliable.
These and many other criticisms mean that the New York reporting system is deeply flawed. Any system based on it will severely mislead the public and parents.
Its adoption will subject school principals and staff to substantial risks of being punished or rewarded on the basis of dubious and unreliable data and for factors beyond their control. It will not accurately identify best practice in schools as Gillard wants.
State and Territory Governments would be well advised to reject the New York model. It will only do harm to a largely successful education system.
Australia and Finland are two of the highest achieving countries in the world in school outcomes according to the PISA surveys conducted by the OECD. Neither country got there by reporting school results.
Why the Rudd Government is choosing to emulate the reporting policies of much lower performing countries like the United States and England can only be explained as a triumph of ideology over evidence.
Trevor Cobbold
Convenor, Save Our Schools
Save Our Schools is a Canberra-based public education advocacy group.
Previous Next
Author: 
//