How Not to Measure Charter School Quality

In a recent article in the Detroit Free Press, an education policy organization called Education Trust-Midwest expressed concern that taxpayers were supporting too many “failing [charter] school operators.”

A focus on school quality for all schools is certainly needed, but Ed Trust’s analysis of charter public schools is short-sighted, unhelpful and risks creating negative unintended consequences.

The Free Press reported that Ed Trust created a “cut score” for charter school quality based on the state’s “Top-to-Bottom School Ranking.” If a charter school performed better than 33 percent of the schools in the state, it passed; if it fell below this mark, it failed. Further, if a charter school management company had more than half of its schools below this cut score, it was “failing.”

The first problem is that the use of a cut score to measure quality is inherently arbitrary. Labeling one school a failure for having average student tests scores at the 32nd percentile and another a success for being at the 34th percentile is a crude approach to assessing the nuances of what makes a quality school.

Secondly, this analysis does not measure individual learning growth. Many charter schools, especially newer ones, are filled with transfer students from other schools. A snapshot assessment of these students’ test scores may be more a measure of their previous school’s performance rather than their current school.

The state’s top-to-bottom ranking does incorporate an element of test score growth, but these are based on cohort averages, not individual students. What should matter most for the purposes of assessing school quality is the value-added that schools provide students, not a snapshot of their relative ranking to state averages.

To be fair, the state does not currently make available these types of student-level analyses. But that fact does not condone the use of more limited analyses to condemn certain schools or school operators.

Analyses like these also risk a serious negative unintended consequence. For instance, if the state used Ed Trust’s arbitrary standards to limit charter school expansion, it would effectively discourage the creation of new charter schools in the very areas where they are arguably needed most — where students in conventional schools are doing poorly on standardized tests. Yet, these are the very students and families who need choice the most, since they are more likely to be poor and unable to move to a different district or pay private school tuition.

Finally, it is important to keep in mind a fundamental characteristic about public charter schools that make them different from conventional schools. Every single student at a charter school is there as a result of a conscientious decision made by a parent or legal guardian. These decisions are not based solely on average cohort test scores. Rather, parents choose these schools for myriad reasons, including safer environments, the school's mission or emphasized values.

That's why closing a charter school based only on average test scores could still negatively impact many students. For some, a charter school with low test scores might still be the best option currently available. From a statewide policy perspective, some of these decisions about which schools are good enough for students will ultimately have to be left up to parents.

This is not a call for turning a blind eye to charter school performance. They should be assessed just like other public schools, and parents should have easy access to this information. But policy decisions should not be based on heavy-handed and arbitrary analyses of these schools, either.