Reporting School Results Does Greater Harm Than Good

A sense of righteous outrage has pervaded some responses to the decision of the NSW upper house of Parliament to prohibit published comparisons of school results. Some of it has bordered on the hysterical. Federal Education Minister, Julia Gillard, called it “ridiculous” and “crazy”.

What is ridiculous is the intransigence of the Federal Education Minister and her State and Territory counterparts in proceeding with reporting school results, and the inevitable league tables that will follow, without proper consideration of the harm it will do to students and the school system overall.

Behind the outrage and the intransigence lies an assumption that the public has an absolute right to all information about schools, including aggregate school results, without regard to its impact on students, schools and families. Gillard says that parents have a right to information about school performance. As Geoff Masters, chief executive of the Australian Council for Educational Research (ACER), wrote in the latest issue ACER News:

Fundamental to the government’s transparency agenda is the belief that parents and members of the public should be able to compare schools. [Masters 2009]

The right to information is not absolute

The right to information is a very important principle in a democracy. It is critical to keeping governments and government agencies accountable. However, there are some circumstances in which the provision of information can do greater harm than good. Reporting school results, and the inevitable league tables that follow, is one among many such circumstances. Eminent overseas and Australian commentators on the use of performance indicators have acknowledged that publishing school results is not an unambiguous benefit to society and that there is a strong case for withholding such information. For example:

bq.As a reaction to unreasonable secrecy the belief in open access to information seems wholly healthy and undoubtedly has led to many benefits. Yet public disclosure of information cannot be upheld as an absolute principle. This is recognized by governments, for example, when they reserve the right to withhold information they deem to threaten national ‘security’. Likewise, if publication of information in likely to harm individuals unfairly, or to mislead, then there is a case for refusing to publish. It is our contention that some published performance indicators which make statements about schools or other institutions fall into this category. Their capacity to reflect reality accurately may be extremely limited and their publication may cause inappropriate inferences to be drawn about institutions. [Myers & Goldstein 1997; see also Goldstein & Myers 1996]

…if the publication of certain information has the potential for harming individuals, or may be seriously misleading, then a justifiable case can be mounted for refusing its publication. It could be contended that much of what might be described as educational performance indicators based on measures of student achievement falls into this category. Its ability to reflect objective reality may be extremely limited, and its publication may therefore cause both misleading and incorrect inferences about schools and ‘school effectiveness’ to be drawn. In such circumstances, there is strong case for withholding publication. [Rowe 2000; see also 2004]

Decisions on the disclosure of information have long had regard to the public harm and benefit. This is widely acknowledged by governments, the courts and many organizations with an interest in government information and data. Gillard herself would be the first to argue that Cabinet documents and minutes should not be released to the public because it could inhibit or damage government processes. It is a pity that she will not recognize the damage releasing school results can do to the education of children.

Freedom of Information (FOI) laws around the world contain provisions setting out categories of information that can be withheld from release [Banisar 2006; see also Independent Review Panel 2008]. There are a number of common exemptions found in nearly all laws. These include the protection of national security and international relations, personal privacy, commercial-in-confidence, law enforcement and public order, information received in confidence, and internal government discussions. Cabinet documents, for example, are kept secret for 30 years in Australia. These exemptions are based on the general assessment that public or private harm could be incurred which exceeds any benefits that may be obtained by releasing such information.

Courts also often apply restrictions on the release of information. In some types of case the courts issue restraining orders on publicity or, which may come to much the same thing, conduct some proceedings in private. This occurs when a trial in open court would render the proceedings nugatory. Examples are when the subject matter affects national security or is a trade secret. Another instance is proceedings concerning the welfare of a child, where a public hearing would undermine the object of the proceedings. For the same reasons the courts sometimes make orders restraining publicity concerning certain aspects of cases.

Now, reporting school results hardly falls within these typical exemptions to the principle of freedom of information. However, they serve to remind that this worthy principle is not absolute, as Gillard’s rhetoric implies.

Most national laws also include a “public interest test” that requires governments to balance the interest in withholding information against the public interest in disclosure. School results are the property of State and Territory governments which have explicit or implied statutory requirements for public interest tests for the release of information. Good public policy also demands that the costs and benefits of major new proposals be fully assessed.

The basic question at issue in deciding whether school results should be reported is whether it will cause undue harm to some people and/or seriously mislead the public compared to the benefits.

This elementary principle was not followed before the governmental agreement to publish school results. The Federal and State/Territory governments assume that transparency of government data on schools is absolute. There is no evidence that governments have properly assessed the public harm against the benefits. It was a decision taken on ideological and political grounds and not one based on evidence. It is time the evidence was examined carefully before proceeding further.

Reporting school results will cause considerable harm

The potential harm, socially and to individual students and families, from the publication of school results and league tables is manifold. Overseas experience shows that it is likely to:

  • Mislead parents, policy makers and the public about school performance and quality;
  • Reduce the quality of education received by students;
  • Increase social segregation and inequity.

Misleading impression of school quality

Raw school results and league tables provide misleading and inaccurate comparisons of school performance because the results are significantly affected by the socio-economic background of school communities. For example, the Tasmanian league table published last May shows that schools in higher income areas tend to have better average results than schools in low income areas. This does not necessarily reflect differences in the quality of teaching and curriculum, just the impact of differences in the social background of students.

Many other factors outside the control of schools also influence a school’s results. These include student absenteeism and mobility between schools, the extent of parent involvement in learning at home, and the extent to which students are engaged in after hours tutoring. For example, if a higher proportion of families engage private tutoring in any one year a school will receive a boost to its measured performance and league table ranking even though there was no change in teaching effectiveness during the year.

School results and league table rankings are also subject to much manipulation. Overseas experience shows that many schools resort to rorting their results by poaching high achieving students from other schools, denying entry to, or expelling, low achieving students and suspending low achieving students on test days. Cheating also becomes rampant under the pressure to improve a school’s ranking.

League tables also mislead when measurement and sampling errors on school results are not reported. Many technical studies of school results and school league tables have demonstrated that chance differences account for a significant proportion of the differences in school test scores. The margin of error can be exceptionally large in measuring improvement which means that the results of the large majority of schools are indistinguishable from each other. In addition, there can be considerable fluctuations in student achievement between years, especially in smaller schools.

This level of error wreaks havoc when comparing school results. It is not possible to make reliable comparisons or rankings of schools because they may reflect chance differences in school performance rather than real differences. Such comparisons are mostly identifying lucky and unlucky schools, not good and bad schools.

For all these reasons, parents choosing a school on the basis of published results may be misled. Some schools may be wrongly recognised as outstanding while others are identified as unsuccessful simply as the result of chance factors or the nature of the student intake. It may lead parents to choose a school of lesser quality than its results indicate.

These factors also make it difficult to identify effective school practices. Decision-makers and schools may be misled in recommending and adopting particular educational programs. Action taken to assist less successful schools may appear more effective than it is in practice.

Reduces the quality of education

A second source of harm from reporting school results and league tables is that it can reduce the quality of education for students by distorting the curriculum and teaching practice.

The pressure to perform well on the public league table encourages schools to devote more of their resources to the curriculum areas that are tested. Overseas experience shows that schools direct more resources into the tested subjects of literacy and maths while untested subjects such as science, history, social studies, arts, physical education and health are given much less time. Students receive less learning in the non-tested subjects and the effect is a less rounded education.

Of major concern is that the tests used to measure school performance largely resort to multiple-choice questions and schools increasingly constrict what is taught to skills that are most conducive to testing by multiple-choice. In addition, schools increasingly focus on teaching test-taking skills. All this has the effect of reducing the teaching of more complex analytical and writing skills.

Another effect is to turn classrooms into test preparation factories. Weeks and months can be devoted to test preparation at the expense of other parts of the curriculum and other learning areas. Focus on practice tests is already a priority now in many Australian schools. Only last April, the head of the Victorian Department of Education, Peter Dawkins, sent a memo to all principals suggesting more time be spent on preparing students for the NAPLAN tests.

The pressure to improve school results can also create incentives for schools to ignore the low-achieving students. There is extensive evidence from overseas experience with reporting school results that schools concentrated on improving the results of students who are on the border of accepted benchmarks rather than the lowest achievers. Improving their score is seen as the most efficient way to raise a school’s average score or the proportion of students achieving a benchmark. However, while this strategy gives schools a better chance of improving their ranking, it may lead to worse outcomes for low performing students.

Public reporting outcomes of individual schools and public rankings of schools can result in the humiliation of schools, teachers, students and their families. Students and teachers in particular years will be easily identifiable as the “culprits” when a school gets a low ranking, especially in small schools, of which there are many in rural areas of Australia. They will be humiliated and demoralised. Students who are humiliated for their lower learning accomplishments are unlikely to respond positively in their future learning. Public ‘tar and feathering’ is hardly the most effective path for student remediation and improvement.

Increases social segregation and inequity

A third major source of harm from reporting school results and league tables is that it leads to increased social segregation between schools. Extensive international studies show that increasing choice and competition between schools by reporting school results and publishing league tables tends to increase social segregation in schools. White well-off families tend to use choice regimes to enrol in schools with a similar racial and socio-economic background.

This contributes to greater inequity in education and social intolerance.

Schools with high concentrations of educationally disadvantaged students often have detrimental effects on student achievement. A student attending a school where the average SES of the student body is low is likely to have lower outcomes than a student from a similar background attending a school where the average SES of the student body is high. Thus, increasing social segregation between schools is likely to reduce average student achievement. Furthermore, while the social composition of schools has a significant impact on the achievement of all students, studies show that the impact is greater on low SES and immigrant/minority students.

Social segregation in schools is also socially divisive as it undermines social understanding and tolerance between different social groups. Social and religious segregation in schools breeds social intolerance in communities and workplaces and undermines social understanding and cohesion. Schools segregated by class, religion and race make it more difficult for children to develop a real understanding of people of different backgrounds and to break down barriers of social intolerance. Socially segregated schools can feed a lack of social empathy, indeed, social intolerance and an inability of people from different backgrounds to effectively work together and live together.

The benefits of reporting school results are dubious

Against these costs, the benefits of reporting school results appear less consequential. What does the right to information deliver in this case?

First of all it is seen as necessary to support freedom of choice of school. Freedom of choice of school is also seen as an absolute right without regard to its costs to others in society. The other main benefit appears to be largely a management issue. South Australian Education Minister, Jane Lomax-Smith has explained the case as mainly “…so that we, as managers of the system, can look at our schools and make sure that best practice is followed…”

The fundamental belief is that freedom of choice and more public information about schools will drive school improvement. Julia Gillard put the case succinctly on her most recent trip to the United States:

This transparency is crucial. To improve schools that are failing their students, we need information. And we want parents to drive the change. [Gillard 2009]

However, there is little evidence that reporting school results increases student achievement by promoting choice and competition. As Professor of Economics at the University of Chicago and the co-author of Freakonomics, Steven Levitt [2007], says of school choice, “the theory sounds great, but evidence confirming it has been hard to find”. A range of sophisticated research studies support this conclusion.

The real danger is that student achievement deteriorates under the new regime as discussed above. Australia has achieved amongst the highest levels of student outcomes in the world without reporting school results and league tables. The countries that report school results have student outcomes well below Australia’s. It would disastrous if Gillard’s new reporting regime sends Australia backwards.

Apply a public interest test to reporting school results

This imbalance in the harm and benefit of reporting school results demands a re-consideration by Australian governments. The introduction of reporting school results has been developed in secrecy. The Federal Education Minister wants schools to be open and transparent about their performance but she has been totally unwilling to subject her proposal to full public scrutiny. It is time the full costs and benefits of her scheme are fully examined before it is too late. It should be subjected to a ‘public interest’ test.

The way forward is a full public inquiry, including an examination of alternative approaches to monitoring school performance. This could be done by a full parliamentary inquiry (Federal and/or State) or by an independent public inquiry.

Trevor Cobbold

References

Banisar, David 2006. Freedom of Information Around the World: A Global Survey of Access to Government Information Laws. Privacy International, London. Available at: http://www.privacyinternational.org/article.shtml?cmd347=x-347-543400&als[theme]=FOI%20Home%20Page

Gillard, 2009. A New Progressive Consensus for Australian Schools. Presentation to the Brookings Institution, Washington DC, 19 June.

Goldstein, Harvey and Myers, Kate. 1996. Freedom of Information: Towards a Code of
Ethics for Performance Indicators. Research Intelligence, July, 12-16.

Masters, Geoff 2009. Masters Warns Against League Tables, ACER News, Issue 78, June.

Myers, Kate and Goldstein, Harvey 1997. Failing Schools in a Failing System. Institute of Education, University of London.

Independent Review Panel 2008. Enhancing Open and Accountable Government. Discussion Paper. Review of the Freedom of Information Act 1992 (Queensland), January.

Levitt, Stephen 2007. More Evidence on the (Lack of) Impact of School Choice. New York Times, 4 October.

Rowe, Ken 2000. Assessment, League Tables and School Effectiveness: Consider the Issues and ‘Let’s Get Real’, Journal of Educational Inquiry, 1 (1): 73-97.

Rowe, Ken 2004. Analysing and Reporting Performance Indicator Data: ‘Caress’ the Data and User Beware! Paper presented at the Public Sector Performance and Reporting Conference, Sydney, April.

Previous Next

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.