Misclassification Errors in My School Comparisons: A Response to Professor McGaw

The Chairman of the Australian Curriculum, Assessment and Reporting Authority, Professor Barry McGaw, has rejected criticisms made by Save Our Schools (SOS) that the “like school” comparisons on the My School website are biased in favour of private schools. But, his response effectively proves the SOS case and he fails to produce any evidence to support his claims. Indeed, the available evidence supports the SOS case.

The basis of the SOS criticism is that My School uses a methodology to determine so-called “like schools” which is based on the socio-economic status (SES) characteristics of small geographical areas rather than on the SES of families. The problem is that each area contains a mix of families and the higher income families more often choose private schools. For example, 55% of high income families choose private secondary schools compared to 26% of low income families.

This causes My School to systematically over-estimate the level of socio-economic disadvantage in private schools and under-estimate disadvantage in government schools. Consequently, My School compares the test results of supposedly similar private and government schools, but which may have large differences in the SES composition of their enrolments.

Professor McGaw says the SOS argument relies on the geographical areas being heterogeneous [ Australian Financial Review, 17-18 April 2010]. But, “they are not,” he says.

They are fairly small districts of a couple of hundred households and the evidence is that they are fairly homogenous.

Here Professor McGaw contradicts himself and concedes the point by saying that the areas are “fairly homogeneous”. The My School methodology depends on the districts being homogenous, but McGaw admits they are not. Districts that are “fairly homogenous” will include families with different socio-economic characteristics, some with a higher SES and some with a lower SES.

Once it is conceded there are differences in the SES of families within districts, the SOS criticisms of the My School methodology come into play because the higher SES families are more likely to choose private schools. This leakage together with the use of area-based measures of SES to rate the SES of schools leads to bias in the comparison of school results which favours private schools.

The use by My School of an area-based measure of SES to approximate individual family characteristics in the area is based upon an assumption of population homogeneity. That is, it assumes that like people live near like people. The validity of this assumption has been tested by studies comparing individual or family SES values or scores with scores assigned on the basis of the average characteristics of residents living within small areas (called census collection districts). The evidence is that census collection districts are not homogenous.

A study by the Australian Council of Educational Research has shown that the correlation between individual and census collection district measures of SES for a national sample of secondary school students was unacceptably low [Ainley & Long 1995]. The study reported correlations between 0.36 and 0.45 between individual and collection district measures in a sample of secondary school students [73]. They found that the greatest loss in precision occurs in moving from an individual based analysis to the collection district level, and that the additional loss of validity when moving from collection districts to larger geographical areas such as postcodes is not great [81-83].

The Australian Bureau of Statistics has demonstrated that some highly advantaged families live in low SES areas and some disadvantaged families live in high SES areas. In an analysis of census data for Western Australia it found that individual and family relative socio-economic disadvantage was quite diverse within small areas [Baker & Adhikari 2007].

It found that about 20 per cent of people in the most disadvantaged quartile of the individual SES measure lived in census collection districts that were in the highest three deciles of the area-based Index of Relative Socio-economic Disadvantage (IRSD). Over a third of people in the bottom quartile lived in areas in the top five IRSD deciles and six per cent of people in the lowest group in the individual based SES measure lived in collection districts found in the highest IRSD decile.

On the other hand, nearly 20 per cent of people in the most advantaged quartile for individual SES lived in areas that were classified in the bottom three deciles of the IRSD. Over a third of people in the most advantaged quartile lived in areas in the bottom five deciles. Five per cent of people in the highest individual based SES group lived in the collection districts found in the lowest IRSD decile.

The ABS researchers conclude from this study that “there is a large amount of heterogeneity in the socio-economic status of individuals and families within small areas” [1].

This conclusion directly refutes Professor McGaw’s claim that small areas are not heterogeneous. Rather, “there is a large amount of heterogeneity” as SOS has argued.

Heterogeneity in the SES of families in small districts means there will be errors in classifying the SES of individuals, such as students, on the basis of the average SES of the areas. This is emphasised by the ABS:

A relatively disadvantaged area is likely to have a high proportion of relatively disadvantaged people. However, such an area is also likely to contain people who are not disadvantaged, as well as people who are relatively advantaged. When area level indexes are used as proxy measures of individual level socio-economic status, many people are likely to be misclassified. This is known as the ecological fallacy. [ABS 2008: 3; see also Adhikari 2006: 6]

Moreover, the potential for error is quite significant:

These findings indicate that there is a high risk of the ecological fallacy when SEIFA is used as a proxy for the socio-economic status of smaller groups within an area and there is considerable potential for misclassification error. [Baker & Adhikari 2007: 1]

Other researchers have also noted this, for example:

Assigning a value of socioeconomic status to a student on the basis of the area in which they live will introduce a potential error and the magnitude of the error will be greater when the social background of those living in the area is relatively heterogeneous. [Ainley & Long 1995: 53; see also Preston 2010]

The potential for misclassification errors in using area-based measures of SES to approximate individual characteristics was also noted in a report commissioned by the Performance Measurement and Reporting Taskforce of the national education minister’s council [Marks et.al. 2000: 27].

Heterogeneity of family SES in small areas has fatal implications for the reliability of comparisons of so-called “like schools” by My School. It introduces the potential for significant errors in classifying students and measuring school SES. This is clearly demonstrated in an analysis of the Penrith statistical area using 2001 Census data [Preston 2010].

The study shows that a government school drawing students from the ten most disadvantaged collection districts in the region would have 16 students from low income families for every student from a high income family. In contrast, an independent school drawing from the same collection districts would have equal numbers of low and high income families. Yet, using an ABS area-based index of socio-economic status which is similar to that used by My School, the two schools would be classified as “like schools”.

Professor McGaw and the Federal Education Minister claim that “like school” comparisons on My School are robust. The evidence so far available overwhelmingly indicates they are not. Apart from errors in the measurement of school SES and being biased in favour of private schools, they also exclude a range of other factors which have a significant bearing on the classification of so-called “like schools”, thereby distorting comparisons of school results [Cobbold 2010ab].

Trevor Cobbold

References

Adhikari, Pramod 2006. Socio-Economic Indexes for Areas: Introduction, Use and Future Directions. Research Paper, Australian Bureau of Statistics, Canberra.

Ainley, John and Long, Michael 1995. Measuring Student Socioeconomic Status. In John Ainley; Brian Graetz; Michael Long and Margaret Batten, Socioeconomic Status and School Education, Australian Government Publishing Service, Canberra, June.

Australian Bureau of Statistics 2008. Socio-Economic Indexes for Areas (SEIFA) – Technical Paper.

Baker, Joanne and Adhikari, Pramod 2007. Socio-economic Indexes for Individuals and Families. Research Paper, Analytic Services Branch, Australian Bureau of Statistics, Canberra, June.

Cobbold, Trevor 2010a. Like School Comparisons do not Measure Up. Research paper, Save Our Schools, Canberra, February. See Research section.

Cobbold, Trevor 2010b. ‘Like School’ Comparisons on My School are Flawed and Misleading. Save Our Schools, Canberra, February.

Marks, Gary; McMillan, Julie; Jones, Frank L. and Ainley, John 2000. The Measurement of Socioeconomic Status for the Reporting of Nationally Comparable Outcomes of Schooling. Report to the National Education Performance Monitoring Taskforce, MCEETYA, March

Preston, Barbara 2010. Notes on the ecological fallacy when area-based indexes of disadvantage/advantage are applied to schooling in Australia, Canberra, March.

Previous Next

Leave a Reply

Your email address will not be published. Required fields are marked *