Identifying Patterns of Test Preparation, Score Gains, and Score Inflation in Test-Based Accountability Systems

Description

Test-based accountability systems have become a widespread approach to improving educational quality at a large scale. However, studies of these systems have shown that they can have unintended consequences, such as inflating students’ scores on state assessments (Hamilton et al., 2007; National Research Council, 2011). As a result, test users may erroneously infer from widespread score gains that schools are becoming more effective. I examine this issue with three studies focused on different aspects of this problem. The first study analyses elementary math teachers’ self-reported test preparation practices, creating measures of inflationary test preparation and using them to predict student performance on a high-stakes test and a low-stakes test. This study finds that the measures predict greater performance beyond prediction on the high-stakes than on the low-stakes tests for teachers who self-report low use of inflationary test preparation. The second study describes patterns of test performance related to one common test preparation strategy that can cause inflation: focusing on content that is likely to be assessed at the expense of content that is unlikely to be assessed. This study finds that schools that experienced relatively large year-over-year gains may have done so by aligning instruction better toward content with medium-to-high representation on prior tests. The third study explores changes in the predictive relationship between 8th grade test scores and high school graduation to identify any changes that could suggest that scores were inflated. This study finds that changes in these relationships sometimes align with concerns about inflation, with larger changes observed in contexts where prior research suggests there should be more inflationary test preparation. Together, these studies connect test preparation to gains in test scores that may not reflect improvements in students’ life prospects, highlighting the challenges associated with using test-based accountability systems to drive education reform.