The Australian Senate’s Education, Employment and Workplace Relations Committee is currently holding an Inquiry into the effectiveness of the National Assessment Program – Literacy and Numeracy (NAPLAN).
Over 70 submissions have been received of varying quality. In this article, I focus on the submission from the Australian Curriculum, Assessment and Reporting Authority (ACARA). ACARA is the custodian of NAPLAN and how it is used for school transparency and accountability purposes on the MySchool website.
One of the focus questions in the Inquiry’s Terms of Reference refers to the unintended consequences of NAPLAN’s introduction. This is an important question given widespread but mainly anecdotal reports in Australia of: test cheating; schools using test results to screen out ‘undesirable enrolments’; the narrowing of the curriculum; NAPLAN test cramming taking up valuable learning time; Omega 3 supplements being marketed as able to help students perform better at NAPLAN time; and NAPLAN test booklets hitting the best seller lists.
Here is what ACARA’s submission to the Senate inquiry has to say about this issue:
To date there has been no research published that can demonstrate endemic negative impacts in Australian schools due to NAPLAN. While allegations have been made that NAPLAN has had unintended consequences on the quality of education in Australian schools there is little evidence that this is the case. The submission goes on to refer to two independent studies that investigated the unintended consequences of NAPLAN.
ACARA dismisses a Murdoch University research project1 led by Dr Greg Thompson as flawed because its focus is on changes to pedagogical practices as a result of the existence and uses made of NAPLAN. The basis of the dismissal is that if teaching practices change then that is all about teachers and nothing to do with NAPLAN. Yet this report makes clear that teachers feel under pressure to make these changes, changes they don’t agree with, because of the pressures created by the use of NAPLAN as a school accountability measure. In other words, in one clever turn of phrase, ACARA rules out of court any unintended consequences of NAPLAN that relate to changes to teachers’ practice.
ACARA also dismisses a survey undertaken by the Whitlam Institute because it “suffers from technical and methodological limitations, especially in relation to representativeness of the samples used” and dismisses its conclusions, without outlining detailing the findings. Now this survey was completed by nearly 8500 teachers throughout Australia and it was representative in every way (year level taught, sector, gender, years of experience) except for the oversampling in Queensland and Tasmania. In relation to this sampling concern the writers even reported that they weighted the responses to compensate for this sampling differential. This report documents many unintended consequences that ACARA are now saying are not substantiated because of a spurious sampling critique. This is intellectually dishonest at best.
ACARA’s dismissal of all of the findings of these two research projects on spurious grounds while refraining from stating their findings is in stark contrast to its treatment of unsupported statements by non impartial stakeholders about enrolment selection concerns.
In response to the claims that some schools are using NAPLAN results to screen out ‘undesirable students’ ACARA states that it is aware of these claims but appears willing to take at face value comments from stakeholders who represent the very schools accused of unethical enrolment screening.
It is ACARA’s understanding that these results are generally requested as one of a number of reports, including student reports written by teachers, as a means to inform the enrolling school on the strengths and weaknesses of the student. The purpose for doing so is to ensure sufficient support for the student upon enrolment, rather than for use as a selection tool. This understanding is supported by media reporting of comments made by peak bodies on the subject (my emphasis).
ACARAs approach to this whole matter comes across as most unprofessional. But, unfortunately for ACARA, this is not the whole story. There is a history to this issue that began in 2008, almost as soon as the decision to set up MySchool was announced by the then PM Kevin Rudd as part of his school transparency agenda.
Three years ago this month I wrote an article2 about the importance of evaluating the impact of the MySchool Website and the emergence, under FOI, of an agreement in September 2008 by State and Commonwealth Ministers of Education to:
…commence work on a comprehensive evaluation strategy for implementation, at the outset, that will allow early identification and management of any unintended and adverse consequences that result from the introduction of new national reporting arrangements for schools (my emphasis).
It was clear from the outset that this evaluation should have been managed by ACARA as the organisation established to manage the school transparency agenda. In 2010, in response to my inquiry to ACARA on this Ministerial agreement the CEO of ACARA stated that it was not being implemented at that point in time because, early reactions to hot button issues are not useful and because the website did not yet include the full range of data planned.
This was a poor response for three reasons.
Firstly, well-designed evaluations are designed, not as afterthoughts, but as part of the development process. One of the vital elements of any useful evaluation is the collection of baseline data that would enable valid comparisons of any changes over time. For example, information could be collected prior to the MYSCHOOL rollout on matters such as:
• Has time allocated to non-test based subjects reduced over time?
• Has teaching become more fact based?
• Has the parent demographic for different schools changed as a result of NAPLAN data or student demographic data?
• Are more resources allocated to remedial support for students who fail to reach benchmarks?
• Are the impacts different for different types of schools?
Secondly, the commitment to evaluate was driven by concerns about the possibility of schools being shamed through media responses to NAPLAN results, the narrowing of curriculum and teaching, further residualisation of public schools, test cheating and possible negative effects on students and teachers. Identifying these concerns early would allow for revising the design elements of MySchool to mitigate the impacts in a timely fashion. There is no real value in waiting years before deciding corrections are needed.
Thirdly, anyone who seriously believed that all of the data elements agreed as possibly in scope for MySchool was a complete list and able to be developed quickly, was dreaming. Waiting for the full range of data meant, in reality, an indefinite delay. There are still data items in development today.
So now, five years on from the Ministerial directive that there was a need to actively investigate any unintended consequences, there is still no comprehensive evaluation in sight. One suspects that ACARA finds this quite convenient and hopes that its failure to act on this directive stays buried.
However, Ministers of Education still had concerns. In the MCEECDYA Communiqué of April 2012 the following is reported:
Ministers discussed concerns over practices such as excessive test preparation and the potential narrowing of the curriculum as a result of the publication of NAPLAN data on My School. Ministers requested that ACARA provide the Standing Council with an assessment of these matters.
On the basis of this statement, I wrote to ACARA on 27 April 2013 requesting information on action in response to this directive – then over 12 months old. To date I have received no reply.
So what sense can be made of this?
If one takes at face value the statements by ACARA that it knows of no information regarding the extent of unintended consequences one can only conclude that ACARA has twice not undertaken a Ministerial directive.
Here we have a Government body aware of Ministers’ concerns about unintended negative consequences about a program it manages; aware of widespread anecdotal concerns, some of them quite serious; dismissing without any proper argument the few pieces of evidence that do exist; and. refusing to undertake any investigation into this matter despite two Ministerial directives to do so.
Willful ignorance about the potential unintended and harmful impacts of a program an agency has responsibility for whilst all the while professing a strong interest in this matter is highly irresponsible and unprofessional.
It is also quite astonishing given the Government’s commitment to the principle of transparency and the fact that ACARA was established specifically to bring that transparency and reporting to Australia’s schools.
But to then write a submission that almost boasts about the lack of information on this issue, while dismissing with poor arguments the evidence that is growing, is outrageous. It also gives new meaning to a throwaway line in its submission about the negative findings from the Whitlam Institute survey “Further research with parents, students and other stakeholders is needed to confirm the survey results on well-being”.
Further research is indeed needed, and this further research should have been initiated by ACARA quite some time ago– five years to be precise. It is convenient, for ACARA, that such research is not available. It is intellectually dishonest and misleading for ACARA to now state that it “takes seriously reports of unintended adverse consequences in schools. It actively monitors reports and research for indications of new issues or trends.”
Of course, there is another, more alarming possibility, that this work has been undertaken but is not being made public, and that ACARA is misleading the Parliamentary inquiry and the public by denying that any such information exists.
In either case, I am forced to conclude that ACARA does not want any unintended consequences of a program for which it is responsible to be known, in spite of its ‘interest in this issue’ and is persisting in its position of willful ignorance.
In an effort to restore public confidence in its work, ACARA should commit to undertaking this research at arms length using independent researchers and reporting the findings to Parliament, without delay. Perhaps this Inquiry could recommend this.
Margaret Clark
This article is cross-posted from Margaret Clark’s blog Combatting Schooling Injustice with permission.
________________________________________
[1] A preliminary analysis of teacher perceptions of the effects of NAPLAN on pedagogy and curriculum
[2] Margaret Clark, Evaluating MySchool, Professional Educator, Volume 9, Number 2, June 2010, pp 7 – 11