Are demographic changes behind test score improvements?

DC Mayor Vincent Gray announced record increases in test scores last month, attributing the gains to his education reform policies. But could demographic changes in DC be responsible for the increases? The answer is: we don’t know.

Mayor Gray and DCPS Chancellor Kaya Henderson have claimed that the increases validate their education reform policies and show we must “stay the course”, a subtle jab at Councilmember David Catania’s “reform 2.0” proposals.

But is it possible that the test score increases reflect the growth of middle and upper class families in DC, and not increased school quality?

What does census data say?

DCPS points to the improvements in test scores each year since the 2007 mayoral takeover of DC Public Schools.

But look at the changes in demographics among DC families over a similar time period.

The median income among families in DC has consistently climbed, from $51,411 in 2005 to $75,603 in 2011, according to the census’ annual American Community Survey (ACS). And students from higher-income families tend to do better on standardized tests.

Some important caveats should be made. First, we don’t know whether this demographic shift is reflected in the public school population. Second, the ACS data that is available only goes through 2011, with 2012 data scheduled to be released in September.

DCPS spokesperson Melissa Salmanowitz pointed me to the growth in scores since 2007 for students who receive free or reduced-price lunch, saying these gains had “disproven” the thesis that demographics are behind the overall score gains. Students eligible for free or reduced-price lunch come from families earning under 185% of the poverty line.

It’s true that scores for those students have gone up. But most or all of that gain happened before many of the reforms Gray and Henderson claim are responsible for the increase took effect. After 2009, the scores for students who receive free and reduced price lunches plateaued.

And this spike in scores from 2007 to 2009 wasn’t due to cheating. Scores on the federal NAEP test from students receiving free and reduced-price lunch also spiked from 2007 to 2009 and then leveled off or declined in 2011. The NAEP test is allegedly “uncheatable.”

What does this mean?

Does this mean that demographics, and not school quality initiatives like Common Core, charter expansion, teacher assessments and extended school day, are responsible for the increase in test scores? I don’t believe we can draw that conclusion.

What this means is that we don’t know what is causing the test score gains. We can’t tell what is causing the increase in test scores because we are using static proficiency measures of test scores that have been roundly criticized.

Static measures of growth compare different cohorts of students from year to year. The problem is that when the demographic composition of students in a school changes, test results may go up or down because of that change rather than a change in the quality of instruction.

DC Public Schools and Mayor Gray are currently held to a bar that has been set for them by others, namely OSSE and the federal No Child Left Behind law, and not themselves. And they should be congratulated for moving these static test scores in the right direction.

What needs changing, as the National Academy of Sciences argues, is the bar itself. We need to start holding schools accountable using metrics of growth, such as how many grade levels a school advances its students each year. Isn’t that what matters?

Chancellor Henderson’s response to the suggestion that increased test scores might be the result of an influx of wealthier students was: “Haters are going to hate.” But is it hate to try to follow the data wherever it leads? I don’t think so. I thought that’s what school reform was all about.