The Federal Government claims that publishing school results on the My School website will better inform parent choice of school. However, parents can be misled by using published school results to inform their choice of school because school results are not a reliable measure of school quality.
The school results published on My School are likely to be an inaccurate and misleading measure of school quality because:
1. Differences in school composition affect school results;
2. Many other factors outside schools influence school results;
3. They are a selective measure of education;
4. They are subject to manipulation and rorting; and,
5. There may be significant statistical errors on school test results.
These factors may lead parents to choose a school of lesser quality than its results indicate. They also make it difficult to identify effective school practices. Decision-makers and schools may be misled in recommending and adopting particular educational programs. Education practices and programs could be falsely identified as successes while successful programs in reality are ignored or even falsely condemned.
Differences in student composition influence school results
School results are significantly determined by the socio-economic background of school communities. School results and league table rankings are often more a measure of the family background of a school’s students than the quality of its teaching.
Schools with a high proportion of students from high income families generally have higher average results than schools with a high proportion of students from low income families, but this says nothing about school quality. Test results of mediocre schools with privileged intakes can look good compared to schools with high quality teachers serving less privileged communities.
My School allows unfair and misleading comparisons between some of the most advantaged and disadvantaged schools around Australia, despite government assurances this would not happen.
Ethnic and other differences in enrolment profiles also affect school results. For example, schools with a high proportion of Chinese students generally have higher average results than those with more students from other ethnic backgrounds. Schools with higher proportions of students with disabilities participating in tests or higher proportions of Indigenous students may have lower results than other schools.
Other outside factors influence school results
School results are also strongly influenced also by other external factors such as student absenteeism, student turnover, school size, school funding, parent involvement in learning at home, and the proportion of students receiving private tutoring.
Some schools may have lower results because they have a high proportion of students who often change school. Studies show that these students tend to have lower average results than students who remain at the same school.
School results and league table rankings may significantly distorted by the results of a few students in small schools. Studies show that small schools are much more likely to report large changes in average results from one year to the next, both positive and negative.
A school may achieve higher results than another simply because a larger proportion of its families use private tutoring. A school’s results may improve significantly because a higher proportion of families engage private tutoring. This says nothing about the quality of teaching and curriculum at the school.
School results are a selective measure of education
Comparing and judging schools on their test results for literacy and numeracy does not provide a full picture of the work of schools. It fails to consider that education is more than literacy and numeracy, even though these are critical skills. The purposes of school are the intellectual, social, civic, and ethical development of individuals. Literacy and numeracy tests are not necessarily an indicator of the success of schools in pursuing these broader purposes of school.
Different schools often do better in some year levels, in some curriculum areas, and for some schooling objectives. Some schools can do well in some other important areas of learning such as supporting the personal and social development of students, arts and music and science, but not so well in literacy and numeracy. The focus on the results of literacy and numeracy tests fail to take account of the diversity of educational programs offered by schools.
Failure to take account of the contribution schools make in other important areas of childrens’ learning may give a distorted and inaccurate view of school quality.
School results are subject to manipulation and rorting
School results may be artificially boosted by being manipulated in various ways. Overseas overseas experience shows that many schools resort to poaching high achieving students from other schools, denying entry to, or expelling, low achieving students, suspending low achieving students on test days, holding back students in grades not tested, increasing use of special dispensations for tests, encouraging students to take courses whose results are not used to compare schools and outright cheating. Extensive academic studies also show that test results are manipulated by schools in various ways to improve their ranking.
Several of these ways of manipulating school results are already being used in Australia. During the recent NAPLAN tests there were many instances of schools encouraging low achieving students to stay home during the tests, leaking of tests beforehand to alert teachers about questions and teachers helping students with answers and changing answers.
The overseas experience with publishing school results and league tables suggests that manipulation of school results is likely to increase in the future as a result of the pressure placed on teachers and principals to improve school results.
School results are subject to statistical error
Considerable uncertainty surrounds the accuracy and reliability of school results because of measurement and sampling error. These errors are inevitable in testing and reporting regimes.
Many technical studies of school results and school league tables demonstrate that chance differences account for a significant proportion of the differences in school test scores. In the case of gains from one year level to the next or annual changes in the results of a given year level, the margin of error can be exceptionally large.
Several studies, including one Australian study, show that the results of up to 80% or more of schools are indistinguishable from the average school outcome. Real differences in school results can be only identified for a small minority of schools.
This level of error wreaks havoc when comparing school results. It is not possible to make reliable comparisons or rankings of schools because they may reflect chance differences in school performance rather than real differences. Such comparisons are mostly identifying lucky and unlucky schools, not good and bad schools.
The Australian Curriculum, Assessment and Reporting Authority has failed to implement the decision of the national ministerial education council that statistical error margins should be published with school results to ensure accurate interpretation.
The My School website only reports the margins of error for various cohort sizes taking the test. This is meaningless because the size of the cohort taking each test at each year level in each school is not reported.
Little prospect that the reliability of My School can be improved
There is little prospect that making changes to My School will improve its accuracy and reliability in any substantive way and reduce the scope for parents and the public to be misled in comparing school results.
It is unlikely that sufficiently detailed information can be obtained to accurately measure the socio-economic status of schools. It will also be difficult to obtain information on the detailed ethnic make-up of schools.
Little can be done to counter manipulation and rorting of school results. The “high stakes” attached to published school results and league table rankings mean that rorting and cheating is inevitable. It continues to be a feature of systems that have been publishing school results and league tables for the past 20 years.
At the very least, My School should report the margin of error for each test at each year level for each school so as to more accurately inform parents and the public about school results published on the website. It would ensure compliance with the principles and protocols for reporting school results promulgated by the national education ministers council.