The Prime Minister claimed last week that what is “truly powerful” about My School is the comparison of like schools. In making this claim, she has rejected the advice of the COAG Reform Council that the data used to construct like school groups has “significant problems”.
The Council says that the data is inadequate and not robust because many parents refuse to disclose information about their education and occupation. This data, which is drawn from school enrolment forms, will be used in My School 2.0 to calculate the Index of Community Socio-Educational Advantage (ICSEA) for each school. Schools with similar ICSEA values are grouped as “like schools’
In its report to the Prime Minister last month on the National Education Agreement, the Council says that there are “significant problems with non-response rates” (p.58), that reporting educational outcomes by socio-economic background is one area where data collections “are at their least robust” and that analysis using this data at the State and Territory level “is often limited by poor or missing data” (p.115).
The non-response rates are high and volatile from year to year. They increased from 30-37% in 2007 to 40-47% in 2008 and then declined to 17-25% in 2009. This is still unacceptably high. Further, there will be quite large variations in response rates between individual schools.
The Council’s report states:
Collectively, educators and researchers are supporting efforts to link measures of socio-economic status at the individual student level with schooling outcome measures in preference to area based measures of socio-economic status. However, low levels of disclosure of personal data such as parent education attainment and parent occupation need to be greatly increased for this to be a viable approach across the performance reporting framework. [pp.111-112]
The consequence of this missing data for reporting is that it can only be done at the national level according to the Council:
For NAPLAN, disaggregations for socio-economic status – measured by parental education and occupation – can only be reported at the national level due to high variability in reporting rates between jurisdictions and sectors, although the national response rate has improved greatly between 2008 and 2009. (p.58)
Yet the Government is using this data to measure the socio-economic status of individual schools in My School.
This missing data has significant implications for the comparisons of so-called like schools because the national literacy and numeracy (NAPLAN) results suggest that the non-responses are concentrated in the lowest SES groups.
The NAPLAN results are reported for various categories of parent education and occupation and also for the non-response group. The average results of the non-response group are similar to those of students from the lowest parent education and occupation groups.
This suggests that there is a bias against low SES government and private schools in “like school” comparisons because their ICSEA value will be over-estimated on average and these schools unfairly compared with higher SES government and private schools. Overall, the bias will be stronger against government schools because they enrol the vast majority of low income students, nearly 80 per cent nation-wide.
ICSEA values estimated using area-based Census data in My School 1.0 were fatally flawed and were biased against government schools. But, there can be no assurance that the new method is any better because of high non-response rates on the parent education and occupation questions on school enrolment forms. In My School 2.0 there is an additional bias against low SES private schools as well as low SES government schools.
The bias against low SES government schools could even be worse in many circumstances than under the area-based estimates of ICSEA values in My School 1.0. For example, the ICSEA values of schools where low SES students comprise a majority of enrolments could be very much over-estimated if a quarter to a half of these low SES families do not state their education and occupation.
Thus, there can be no assurance that like school comparisons in My School 2.0 are more accurate than in My School 1.0 as claimed by the Prime Minister and the Education Minister last week in their joint press conference.