One of the major teacher performance pay schemes in the United States has failed to increase student achievement according to a comprehensive study published this month. Student achievement in reading, mathematics and science was no better in schools participating in the Chicago Teacher Advancement Program (TAP) than in comparable schools that did not implement the program.
However, by changing the working conditions in schools, the program may have contributed to higher teacher retention rates in TAP schools compared to non-TAP schools. The program increased the amount of mentoring, promotion opportunity, and compensation relative to non-TAP schools, and these may have translated into making Chicago TAP schools a more desirable place to continue working.
The Chicago TAP was implemented as a pilot program intended for 40 high-need schools. The program began in 10 schools in 2007 with a rollout adding 10 new schools in each year of a four-year implementation period. The program was introduced with much fanfare by former Chicago Public Schools head Arne Duncan, now US Secretary of Education.
Under the program, teachers can earn extra pay and responsibilities through promotion to mentor or master teacher as well as annual performance bonuses based on a combination of their value added to student achievement and observed performance in the classroom. The idea behind TAP is that giving teachers performance incentives, along with tools to track their performance and improve instruction, will help schools attract and retain talented teachers and help all teachers raise student achievement.
The average payout was about $1,100 in the first year of a school’s implementation, $2,500 in the second and third years, and $1,900 in the fourth year. Principals can earn up to $5,000 each year based on the quality of program implementation and school-wide value added. Other school staff can receive up to $500 in the first year and $1,000 in subsequent years based on school-wide value added.
Teachers who were selected to serve as mentor teachers received an additional salary augmentation of $7,000, and lead (master) teachers received $15,000 with a possibility for $20,000 in the final year for teachers designated as “lead plus.”
The study of the program, carried out by the research group Mathematica Policy Research , found evidence of both positive and negative test score impacts in selected subjects, years, and cohorts of schools, but overall there was no detectable impact on reading, mathematics or science achievement.
It found no evidence to support the hypothesis that the impact on test scores might grow over time as the school district became more familiar and experienced with the program. The evidence did not suggest that impacts grew over the four years in which the program was rolled out in Chicago. The impacts on scores in all three subjects (reading, mathematics and science) were largest in the third year (and statistically significant in reading), but fell in the fourth year to zero or negative and not statistically significant.
The study also tested the hypothesis that impacts on test scores would grow as individual schools became more accustomed to the program. Again, it found no consistent evidence of a learning effect from analysing the results of schools with one, two, three, and four years of experience implementing Chicago TAP. The years of experience in the program had no relationship to size of the impact on student achievement.
The study found some evidence of improved teacher retention in schools participating in the program. Teachers who were working in Chicago TAP schools in 2007 returned in each of the following three years at higher rates than teachers in comparable non-TAP schools. For example, 67% of classroom teachers in cohort 1 in 2007 returned to their same school in 2010 compared to about 56% of teachers in non-TAP schools, an impact of nearly 12 percentage points. In other words, teachers in Chicago TAP schools in 2007 were about 20% more likely than teachers in comparison schools to be in those same schools three years later.
However, the impacts on retention were not uniform or universal across years, cohorts and sub-groups of teachers and some of the estimates of impact on retention were not statistically significant. The study also found some evidence that impacts on retention were stronger for subgroups of teachers, such as those with less experience, but the pattern was not consistent. The overall conclusion was that Chicago TAP most likely had an impact, inducing teachers to stay longer in their schools, but the impacts were not uniform or universal across years, cohorts, and subgroups of teachers. It said that the chief benefit of a positive impact on retention appears to be through reduction in the disruptive effect of turnover and of having to find replacement teachers.
This was the third study of the program. It was based on a hybrid study design that relies on both the random assignment of schools to year of implementation and careful matching of Chicago TAP schools to non-TAP schools. It compared results of participating schools with a group of schools with similar student demographics, school size, teacher-retention rates, accountability status, and levels of student achievement. The two earlier studies found no impact on student achievement and no improvement in teacher retention.
The study adds to the accumulating weight of evidence that teacher bonuses have no effect on student achievement. Studies of large teacher bonus programs operating in Iowa, Texas, Denver, Nashville and New York have found that students in schools participating in teacher performance bonus programs do no better than students in schools not participating in such programs.