Save Our Schools has published a research paper assessing the claims of the Prime Minister and the Federal Education Minister that reporting school NAPLAN results on the My School website will lead to improved school performance.
The paper reviews three streams of research studies relevant to the Government’s claim: studies of the impact publishing school results, studies of “high stakes” school accountability measures and studies of the impact of greater choice and competition between schools.
The following is a summary of the paper.
A review of the major research studies on reporting school results, school accountability measures and policies to promote greater choice and competition between schools shows that the Govenment’s claim that publishing school results will improve student achievement is unsubstantiated. At best, the evidence is mixed. This is not a robust foundation for education policy.
Studies of the impact of reporting school results show show no significant effect on student achievement. Even one of the most frequently cited studies on school accountability concluded that reporting school results itself has no effect on student achievement.
The evidence from the major studies of so-called “high stakes” accountability measures (which include reporting school results, rewards and sanctions for schools, exams for students and grade promotion standards) is mixed. Some find positive effects on student achievement, others find no effect or negative results. Often the results differ within studies as well as between studies. Nearly all of this evidence comes from studies conducted in the United States over the past decade or so.
The effects on reading and mathematics achievement were sometimes modestly negative and sometimes modestly positive. The effects varied among studies depending on the nature, types, and timing of accountability policies. They varied also among subjects, grades, and time periods chosen for the analysis. The methodology used also affected the results.
There are several reasons to believe that the positive results shown in some studies are over-stated.
The two most frequently cited studies which show that accountability measures increase school achievement significantly over-stated the effect. The studies measured the effect as a proportion of the variation in state results instead of the variation in student results, which is much larger and is the more relevant benchmark. When measured against the variation in student achievement the positive effect in these studies is shown to be very small.
Most of the studies showing a positive effect from high stakes accountability include student as well as school accountability measures. This also causes the effect to be over-stated.
The studies take into account school-based measures such as reporting school results, monetary rewards for improved performance and sanctions for low performance such replacing the principal or teachers, re-organising or closing schools and permitting students to enrol elsewhere. They also include requirements for students such as grade promotion standards and tests; end-of-course exams; and a high school graduation exam.
Student-based requirements like these have a significant effect on the final results of many high stakes accountability studies; and their effect is larger than the effects of rewards and sanctions relating to school performance. Indeed, a study which re-worked the data used in one of the most frequently cited studies finding positive results from high stakes accountability found that the results were attributable to the student requirements.
There is also substantial evidence that the positive results in some studies may be due, at least in part, to schools manipulating their results in various ways. Studies show that schools respond to high stakes accountability tests by moving resources from subjects not tested to the subjects that are tested, re-classifying students as special education students so they can be exempted from high stakes tests, suspending low achieving students during the testing cycle and outright cheating.
A few studies of the impact of high stakes accountability regimes have examined their distributional consequences. They found that students just below proficiency standards tend to make greater test score gains as schools tend to concentrate resources on improving the results of this group. There is conflicting evidence concerning the effects on students scoring at either the highest or lowest end of the performance spectrum. There is also little evidence to suggest that accountability measures reduce achievement gaps between low and high income students and between white students and those from other racial groups.
Some studies using international test results have found small positive effects of school accountability measures on student achievement. However, these studies have significant methodological caveats attached to them.
The major reviews of high stakes accountability studies conclude that the evidence is mixed and provides little scientific foundation for these policies. This has been acknowledged by the chief executive of the Australian Curriculum, Assessment and Reporting Authority in a stunning contradiction of his Minister’s claims.
Another stream of studies analyse the impact of choice and competition between schools on student achievement. The weight of evidence from the best designed and most comprehensive of these studies is that increasing choice and competition between schools does not improve student achievement once student and family background characteristics are taken into account.
Greater choice and competition between schools in public education systems does not appear to lead to increased student outcomes. Increased choice and competition from different types of schools within public education systems such as charter schools does not appear to increase student achievement. Nor does competition from private schools appear to increase achievement in public schools. At best, the evidence from these studies is also mixed and therefore does not provide a solid foundation choice and competition policies in education.
In the face of this evidence, even former advocates of “high stakes” school accountability measures and greater choice and competition between schools, such as former US Assistant Secretaries of Education, Chester Finn and Diane Ravitch, have been forced to admit that the results of these policies have been disappointing and that they have not worked.