The new national system of reporting school test results has failed to meet the standards set for it by education ministers.
The My School website unveiled by the Federal Education Minister, Julia Gillard, last month is in breach of the national ministerial agreement for reporting school results.
The agreement states that statistical error margins should be published with school results to ensure accurate interpretation. It also states that the data used should be “valid”, “reliable” and “fair and accurate”. My School fails on all counts.
The My School sample school report does not include estimates of the statistical errors in school results. Not only is this in breach of the national agreement, but it will mislead parents and the public in comparing schools and in comparing results from one year to the next. It has potentially very significant consequences for parents and their children and for education policy decisions and programs.
Much uncertainty surrounds the accuracy and reliability of school results because of measurement and sampling error which is inevitable in testing. Many technical studies of school results and school league tables demonstrate that chance differences account for a significant proportion of reported differences in school test scores.
Melbourne University statistician, Professor Margaret Wu, has shown that a class average score on national literacy and numeracy tests can vary by around 10% from year to year due to random fluctuations of student cohort and inaccuracies in test scores and that the margin of error could be even larger. This is a substantial error margin, being equivalent to about one year in learning growth.
In the case of gains from one year level to the next or annual changes in the results of a given year level, the margin of error can be exceptionally large. Several studies, including an Australian study, show that the results of up to 80% or more of schools are statistically indistinguishable from the average school outcome. Real differences in changes in school results can be only identified for a small minority of schools.
Large margins of error make school results unreliable as an indicator of school performance and quality. It wreaks havoc in comparing school results. It means that is not possible to make reliable comparisons or rankings of schools because they may reflect chance differences in school performance rather than real differences. Such comparisons are mostly identifying lucky and unlucky schools, not good and bad schools.
Parents may be seriously misled in choosing a school. Some schools may be recognised as outstanding while others are identified as unsuccessful simply as the result of chance and not because of actual programs and teaching practice. It also means that current school performance is highly misleading as a guide to future school performance.
Choosing schools on the basis of chance variation in a school’s results can have far reaching consequences. A school may be wrongly labelled in the public eye as unsuccessful and exiting parents may initiate a spiral of decline in an otherwise successful school.
Random errors in school results also make it difficult to identify effective school practices. It may mislead decision-makers and schools in recommending and adopting particular educational programs. Action taken to assist less successful schools may appear more effective than it is in practice.
It appears that the Australian Curriculum, Assessment and Reporting Authority (ACARA) only intends to issue a general disclaimer that all tests are subject measurement error and to provide some information about margins of error related to the number of students tested (see My School fact sheet).
This is not enough. The error margins should be reported for each test result to ensure accuracy, assist proper interpretation of results and ensure that parents are not misled about what appear to be differences in school results.
Apart from unreported statistical error, My School reports will also be unreliable as a guide to parents and the public because school results are influenced by factors outside the control of schools and are subject to manipulation by schools. None of these factors is mentioned in the My School guide to interpreting national test results.
Even apart from the impact of the socio-economic background of school cohorts on school results, there are many other factors outside the control of schools which influence their results. These include student absenteeism, illness, mobility between schools, the extent of parent involvement in learning at home, and the extent to which students are engaged in after hours tutoring.
For example, if a higher proportion of families engage private tutoring in any one year a school may receive a boost to its measured performance even though there was no change in teaching effectiveness during the year.
School results and league table rankings are also subject to much manipulation under the competitive pressure to improve rankings.
Overseas experience shows that many schools resort to rigging or fudging their results by poaching high achieving students from other schools, denying entry to low achieving students, suspending low achieving students on test days, encouraging low achievers to take courses not used to rank schools, helping students with answers during tests or changing answers.
The Ministerial Council for Education, Early Childhood Development and Youth Affairs should act promptly to protect the integrity of its agreement on reporting school results. It should ensure that the national protocols for reporting school results are adhered to by ACARA. It should require ACARA to report the statistical margin of error for each test result reported in each school report on the My School website.
ACARA should also be required to provide a detailed guide for interpreting school results published on My School so that parents and the public are properly informed about the range of factors that may influence school results.
Failure to act on these matters by the Council will see it condemned as ineffectual and complicit in misleading parents and the public about the reliability of reporting school results.