According to the new Melbourne Declaration of national goals for schooling, Australian governments are committed to ensuring that information published on school results will be accurate and fair. However, to date, governments have not indicated what action will be taken to implement this commitment. The current draft Action Plan on the Declaration merely repeats the commitment. It needs to do better than this to ensure accuracy and fairness in reporting school results.
Considerable uncertainty surrounds the accuracy and reliability of school results because of measurement and sampling error. These errors are inevitable in testing and reporting regimes.
Many technical studies of school results and school league tables demonstrate that chance differences account for a significant proportion of the differences in school test scores. In the case of gains from one year level to the next or annual changes in the results of a given year level, the margin of error can be exceptionally large. Several studies, including one Australian study, show that the results of up to 80% or more of schools are indistinguishable from the average school outcome. Real differences in school results can be only identified for a small minority of schools.
This level of error wreaks havoc when comparing school results. It is not possible to make reliable comparisons or rankings of schools because they may reflect chance differences in school performance rather than real differences. Such comparisons are mostly identifying lucky and unlucky schools, not good and bad schools.
To date, this has not been recognised in government statements on reporting school results. Yet, it has major implications for government objectives in reporting school results and could have pernicious consequences for many schools.
Parents may be misled in choosing a school. Some schools may be recognised as outstanding while others are identified as unsuccessful simply as the result of chance and not because of actual programs and teaching practice. It also means that current school performance is highly misleading as a guide to future school performance.
Choosing schools on the basis of a chance variation in a school’s results can have far reaching consequences. A school may be wrongly labelled in the public eye as unsuccessful and exiting parents may initiate a spiral of decline in an otherwise successful school.
Random errors in school results also make it difficult to identify effective school practices. It may mislead decision-makers and schools in recommending and adopting particular educational programs. Action taken to assist less successful schools may appear more effective than it is in practice.
The large degree of uncertainty about school results also makes the Prime Minister’s threat to reward or sanction schools based on their published test results especially fraught. It is likely to result in much unfairness in the treatment of schools and their staff. In particular, using test results to reward teachers on the basis of student progress creates considerable potential for rewards to be misdirected and not actually reward good teaching.
Reporting the results of individual schools should be statistically valid and reliable so as not to mislead parents, schools and the public. Australian governments should commit to reporting margins of error for each school result including the associated score range. Published tables of school results should contain ‘health warnings’ about the limitations of the data and advice on how to interpret the data.
These steps to better inform the public about the accuracy and reliability of school results have been recommended by expert statistical authorities such as the Statistics Commission and the Royal Statistical Society in the United Kingdom as well as the National Center for Research on Evaluation, Standards and Student Testing in the United States.
Trevor Cobbold