Photo by US Department of Education on Flickr.

Student performance in the nation’s capital has increased so dramatically that it has attracted significant attention and prompted many to ask whether gentrification, rather than an improvement in school quality, is behind the higher scores. Demographic change explains some of the increases in test scores, but by no means all of them. We drew this conclusion after analyzing data from the National Assessment of Educational Progress (NAEP), known as the “nation’s report card,” which tests representative samples of fourth- and eighth-grade students in mathematics and reading every two years. NAEP scores reflect not just school quality, but also the characteristics of the students taking the test. For example, the difference in scores between Massachusetts and Mississippi reflects both the impact of the state’s schools and differences in state poverty rates and other demographics. Likewise, changes in NAEP performance over time can result from changes in both school quality and student demographics.

DC’s school demographics have

changed substantially since 2005. The NAEP data show that the proportion of white and Hispanic students in DC has roughly doubled, while the proportion of black students has declined. The question is whether DC’s sizable improvement is the result of changing demographics, as some commentators claim, or improving quality.

Changing demographics are only part of changing test scores

Our analysis indicates that, based on the relationships between demographics and NAEP scores in 2005, demographic changes predict a score increase of four to six points between 2005 and 2013 (the data needed to perform this analysis on the 2015 results are not yet available).

But the actual score increases have generally far outpaced the gains predicted by demographic change alone. For example, in fourth-grade math, demographics predicted a four-point increase, but scores increased 17 points. The figure below shows predicted and actual score increases for all four tests for DC schools overall (including charters) and the traditional school district (DC Public Schools). Only in eighth-grade reading scores at DC Public Schools do demographic shifts explain more than half of the score increase.

Graph from the Urban Institute.

To be sure, our analysis does not account for all potentially important demographic factors. In particular, we do not include any measures of family income. Though researchers often use eligibility for free or reduced-price lunch as a proxy for determining income level, changes to who is eligible make this measure unreliable. With the eighth-grade scores, we did attempt to use parents’ education as a proxy for socioeconomic status, but our results did not appreciably change: changes in demographics still did not account for changes in academic performance.

The bottom line is that gentrification alone cannot explain why student scores improved in Washington, DC, a conclusion that echoes previous analyses using publicly available data. DC education saw many changes over this period, including reform-oriented chancellors, mayoral control, and a rapidly expanding charter sector, but we cannot identify which policy changes, if any, produced these results.

And despite the large gains, DC NAEP scores still reveal substantial achievement gaps—for example, the gap between average scores for black and white students was 56 points in 2015; the gap between Hispanics and whites was 49 points.

In other words, much work remains to be done.

Here’s how we drew our conclusions

Our analysis of student-level NAEP data from DC, including students from charter and traditional public schools, compares the increase in scores from 2005 to 2013 with the increase that might have been expected based on shifts in demographic factors including race/ethnicity, gender, age, and language spoken at home. The methodology is similar to the one used in a new online tool showing state NAEP performance (the tool excludes DC because it is not a state). We used restricted-use, student-level data from NCES to generate these results. (That means that researchers have to get special permission to access files that have scores on a kid-by-kid (but de-identified) basis, whereas most people would have to use data that averaged for entire states and cities (by subject and year). We measured the relationship between DC student scores in 2005 and the student factors of gender, race and ethnicity, age, and frequency of English spoken at home. We then predicted what each student’s score in 2013 would have been if the relationships between demographics and scores were the same in 2013 as they were in 2005. We then compared the average predicted score of the 2013 test-takers (relative to the 2005 average score) to the actual 2013 score (also relative to the 2005 average score).

We tried model variants that included special education status, limited English proficiency status, and eligibility for free- and reduced-price lunch. Including these variables tended to lower the predicted score change (indicating that an even larger portion of the score changes came from non-demographic changes). However, because these variable are also subject to district and school-level definitions (direct certification/community eligibility may have increased the numbers of FRPL-eligible students, for example), we chose not to include these variables in our prediction, and focused only on demographic changes.

Crossposted from the Urban Wire.

Kristin Blagg is a research associate in the Income and Benefits Policy Center at the Urban Institute, focusing on education policy. She holds a BA in government from Harvard University, an MSEd from Hunter College, and an MPP from Georgetown University. Kristin lives in Georgetown.

Matthew Chingos is a senior fellow at the Urban Institute, where he studies education-related topics at both the K-12 and postsecondary levels. His areas of expertise include class-size reduction, standardized testing, teacher quality, student loan debt, and college graduation rates. He lives with his wife and daughter in Logan Circle.