The imminent teacher boycott of national literacy and numeracy tests gives cause to assess how well the My School website informs comparisons of school quality. Apart from creating incentives to narrow school curriculum and manipulate school test results, a major flaw is that its so-called “like school” comparisons are systematically biased in favour of private schools.
My School purports to compare the test results of schools with similar socio-economic status (SES) student populations. However, it makes private school results look better than their “like” government schools by comparing higher SES private schools with lower SES government schools.
This bias exists because the Index of Community Socio-Educational Advantage (ICSEA) used to measure school SES is based on the average SES of geographical areas in which students live rather than their family SES. Studies by the Australian Bureau of Statistics show that some high income families live in low SES areas and vice versa.
However, there is a greater leakage of higher income families into private schools. For example, 55% of higher income families in Australia choose private secondary schools compared to 26% of low income families.
The greater leakage of higher income students from each area into private schools causes the ICSEA rating of private schools to under-estimate their actual SES because these students are classified by their (lower) area SES measure rather than by their (higher) family SES.
The leakage also causes an over-estimate of the SES of government schools. Government schools take a greater proportion of low SES students, but many are classified at their (higher) area SES rating rather than by their family SES.
This mismeasurement of school SES generates comparisons between higher SES private schools and lower SES government schools. It favours private schools because higher SES students tend to have higher average results than lower SES students.
The My School “like school” comparisons are also misleading because they ignore other factors beyond the control of schools which strongly influence test results.
My School ignores differences in the ethnic composition of schools. For example, average test results of Chinese students are well above those of Middle Eastern and Pacific Islander students. Schools with a similar ICSEA rating could have quite different average results because some have a high proportion of students of Chinese origin while others have a high proportion of Middle Eastern or Pacific Islander students.
Differences in the proportion of students with disabilities between schools are also ignored. Schools with higher proportions of students with disabilities participating in tests may have lower average results than other schools with a similar ISCEA value. This omission further disadvantages government schools because they have double the proportion of students with disabilities in private schools.
“Like school” comparisons between large and small schools are misleading. Small schools are much more likely to report large changes in average results from one year to the next because their results can be heavily influenced by only 4 or 5 students.
Student selection by many private schools and some government schools is ignored. Some schools may achieve higher results than others with a similar ICSEA rating because they can select higher achieving students and exclude lower achieving students.
Some schools with a similar ICSEA rating may have lower results because they have a high proportion of students who often change school. Students who move school often tend to have lower average results than other students.
My School also ignores major funding differences between schools which affects comparisons of “like schools”. For example, recurrent expenditure per government primary school student in the ACT is 48% higher than in Victoria. Total funding per student in many high SES private schools is double or more that of high SES government schools.
All these flaws and omissions mean that My School does not consistently compare like with like. They also mean that differences in quality between “like schools” are exaggerated.
The socio-economic variables incorporated in ICSEA explain nearly 70% of the variation in primary school test results, leaving about 30% attributable to differences in teaching, curriculum, pastoral care, etc. If school SES was more accurately measured and the other factors outside the control of schools affecting results were included in ICSEA its explanatory power could possibly increase to 85-90% of the variation in school results.
This would leave only 10-15% of the variation in school results as explained by differences in school quality. Even part of this may be due to statistical errors on test results which are also not properly reported by My School.
My School is a travesty of “like school” comparisons. It will mislead parents in choosing schools and mislead policy makers about which schools use best practice. It should be thoroughly reviewed by an independent public inquiry.
Trevor Cobbold
This article was originally published in the Canberra Times, 15 April 2010.