Greater Greater Washington

Education


Flawed study mis-rates potential DC school closings

DC would likely close some successful schools while expanding failing schools if it relies upon a study released last week. The much-anticipated study, which the Deputy Mayor for Education commissioned to help plan school closures and charter school policies, is highly flawed.


Photo by M.V. Jantzen on Flickr.

The goal of the study was to help DCPS balance out near-­empty buildings in some locations with over­crowded ones in others, taking into account the quality of the schools.

For all its colorful charts and maps, the report uses a faulty measure of school quality and does not make any serious attempt to predict how families and schools might react to the changes it proposes. With such important decisions at stake, the Deputy Mayor should insist upon more rigorous analysis.

The report authors crunched a lot of numbers in an admirably short period of time and produced some very interesting descriptive statistics, like the percentage of students below 185 percent of the poverty line in charters (75) versus DCPS (67).

The study counts, within each of 39 neighborhood clusters in the city, the number of "performance," or high quality, seats in schools and compares that to the number of school-age students living in that cluster. The difference is called a service gap.

It recommends schools for closure, or in some cases investment, to reduce these service gaps. But it doesn't justify the type of investment. Is it facilities? More teachers? Better teachers?

The authors define a "performance seat" as a seat in a school in the top tier of a 4-tier rating system they devised. Each school's tier comes from estimated percentages of its students who were judged "proficient" on the state assessment test in recent years, projected 4 years into the future assuming a straight line trend.

This study raises a lot of questions for most researchers and even lay readers. Two big flaws stand out, which are so basic and could do significant damage if city leaders overlook the problems.

It uses a flawed measure of school performance. At the heart of this paper is a 4-tier rating of school quality that relies on the percent of students who are proficient on the state test (called the DC-CAS). Never mind the fact that a proficiency rate throws away information by focusing only on whether a score was above or below a fixed cut point instead of how high or low it was.

Student proficiency rates have long been discredited as a school performance measure because proficiency rates capture student achievement at a point in time, but say little about how much the school or its teachers contributed to its current students' performance.

For example, a middle school could have declining proficiency rates if a feeder school begins sending more at-risk students to it, even if the teachers are especially skilled at working with a challenging population.

At a bare minimum, a sensible measure accounts for what a student knew before enrolling in the school (for example, using the student's score from the prior year). This is why more and more states, including DC, have adopted student achievement growth measures instead of proficiency rates for their teacher and school performance indicators.

Using a trend in proficiency rates doesn't help, and only creates a false sense of "gains" which is more likely to measure demographic change and other differences between successive cohorts of students cycling through a school than the performance of the schools' educators. That's because it compares students in one year to different students, instead of students in one year to the same students in the prior year.

By relying on flawed measures of school performance, policymakers risk closing down schools that are best equipped to work with challenging populations and replacing them with ones that would fail miserably if they started working with a different student body.

It ignores human behavior. There is a big difference between bean-counting and behavioral analysis. The latter recognizes that families make choices (within budget constraints) about where they live and where they send their kids to school.

School leaders make decisions tooover what programs to offer and how to allocate scarce resources to produce successful educational outcomes or whatever else they may value. In the case of charter schools, administrators choose whether to open a charter, where to locate, and what to offer.

In modeling supply and demand, however, the report ignored all of these factors. The report makes no attempt to model the behavior of these actors to predict the effect of different policies on outcomes. It is a bean-counting exercise.

For example, this study would say that a neighborhood has no service gap if it had a successful but highly specialized charter school, such as a Spanish immersion school. Obviously such a school could draw students from all over the city and residents of the immediate neighborhood may either not want to attend such a program or not be able to rely on being admitted because the pool of students in the lottery is so large.

Acting on this flawed study could end up making service gaps worse. For example, an affluent neighborhood may have far too many seats for its own students and yet its schools can be overcrowded because families from far flung neighborhoods want affluent peers or a school in a neighborhood with better housing stock.

Building more schools in the less affluent neighborhoods will not necessarily solve that problem. It might just create more under-utilized space. Yet that's exactly what this study recommends.

A smarter policy would strategically locate new schools partway between the current over-enrolled schools and the under-enrolled ones and design curricular offerings to induce the optimal mixing of students. Or better yet, the policy could rely more on information and transportation than simply construction and demolition.

In other words, knowing that a school is under-enrolled is less important than knowing why it is under-enrolled. It's important to know why parents make the choices that they make, not to just tally up their choices at a moment in time like an accountant.

It is possible to model the supply and demand of schooling without making naïve assumptions about schools and families. For example, there is work in progress by economists at Carnegie Mellon University demonstrating how it can be done.

In my own research I have simulated parental choice outcomes using behavioral parameters estimated from school choice data. This analysis illustrated how family preferences over the racial composition of the student body as well as commute distance and other factors such as school program offerings can influence sorting outcomes.

Planners can also consider trends in demographics, housing construction, and transit. They can simulate the results of a wide range of charter school and DCPS policies including not only facilities siting and improvements but varied attendance zones and expanded access to information about and transportation to schools beyond the immediate neighborhood.

The District needs sophisticated guidance to begin comprehensive, city-wide planning of school closures and investments and to help coordinate land use policy with charter school expansion. Unfortunately, this report doesn't provide enough of this guidance.

Steven Glazerman is an economist who studies education policy and specializes in teacher labor markets. He has lived in the DC area off and on since 1987 and settled in the U Street neighborhood in 2001. He is a co-founder of Washington Yu Ying public charter school and is a Senior Fellow at Mathematica Policy Research, but any of his views expressed here are his own and do not represent Yu Ying or Mathematica. 

Comments

Add a comment »

I agree with your two criticisms of the IIF study. The measure of school performance was incredibly narrow. Projecting a schools future performance based on the past five years of test scores and no other variables is meaningless. I also agree that parents use a complicated set of criteria in choosing a school for their child.

However, I do feel that the IIF report brings up a good point in comparing the location of schools with the location of students. Charter schools are forced to locate in areas they can afford which often pushes them from the core of the city. While all district students have an equal opportunity to apply to these schools, the feasability of attending a school far from home and off of most public transportation is difficult. I don't know that the solution is as simple as incentivizing charters to locate in underserved areas given that the exisiting public schools are generally under-enrolled, but somehow focusing reform in those areas seems important.

by SE on Feb 1, 2012 4:44 pm • linkreport

A smarter policy would strategically locate new schools partway between the current over-enrolled schools and the under-enrolled ones and design curricular offerings to induce the optimal mixing of students.

The premise is flawed here. How would you do "strategically locate" new schools when there is a surplus old school buildings? Or how do you "strategically locate" charters, which can locate where they want? You are implying that some sort of top-down command system should be in place. But since the old DCPS was such a system that had failed and which has been successfully upended by "free-market" charters, I suggest that the odds of that coming to pass are very slim indeed.

In other words, knowing that a school is under-enrolled is less important than knowing why it is under-enrolled.

It is usually glaringly obvious why a school is under-enrolled, and no special study is needed: it sucks. The teaching is to the test and remedial; there are few after-school activities to build student interest; there is high teacher turnover; the building is in terrible condition. Etc.

In my own research I have simulated parental choice outcomes using behavioral parameters estimated from school choice data. This analysis illustrated how family preferences over the racial composition of the student body as well as commute distance and other factors such as school program offerings can influence sorting outcomes.

So your research showed that racial composition and commute distance influences the final school choice...Amazing! Radical! Fascinating!

So how does your simulations compare to the real-world? Can this comparison be relevant enough to learn what is happening in DC?

by goldfish on Feb 1, 2012 11:38 pm • linkreport

@goldfish For better or worse, the city has a great deal of say in the real estate opportunities for charter schools. The city owns a lot of former school buildings, some of which (Stevens ES and Franklin School, for example) have been the subject of intense land use fights.

The disposition of these properties is a political process that rarely places adequate value on the educational benefits of a strategically located school. Educational benefits are difficult to monetize and compare against tax revenues from a hotel or condo, and the benefits of leasing a building to a given school here instead of there are hard to estimate without data.

You listed several reasons why a school might suck, but the appropriate policy response (change the staff, upgrade the facility, or close the facility) depends critically on which of those reasons predominates.

Finally, I would say that I'm optimistic that policymakers would use empirically grounded behavioral parameters to drive decisions and predict the consequences of alternative choices for school quality, social stratification, transportation costs, etc. They don't have to use my old dissertation data from a different city, but that was an example of analysis that helped officials in Minneapolis find comfort that the proposal to shrink school attendance zones in the mid 1990s would not worsen racial segregation, as some had feared.

If a grad student can figure out how to do it, then surely a contracted research firm should be able to provide some sophisticated analysis to simulate attendance and quality outcomes under alternative policies.

by Steven Glazerman on Feb 2, 2012 1:46 am • linkreport

I don't have a lot of time to read the study, but the primary reason schools fail or succeed is parental involvement in their children's lives- and the quality of the lives of these children in general. We don't need any pretty charts or graphs to tell us why schools are successful or not. Very little of it is about teachers or geography or community resources. I live near an apartment complex with a lot of children in it, and the noise in the neighborhood last night was horrible- and coming from the parking lot of the apartment complex. I imagined all those kids being up half the night not getting a good night sleep due to the ruckus outside. I doesn't make for a good learning day today. It has nothing to do with teachers, buildings, or anything else. Even sadder is that the gentlemen making all the noise probably had kids inside trying to sleep.

by Tom A. on Feb 2, 2012 9:43 am • linkreport

@Dr Glazerman: You did not read I provided the link about old school buildings. It describes how one particular DC school building (with 32 acres!) has been offered to the charters and turned down. The building was near Trinidad, one of the areas identified that needed improvements in its schools.

The buildings you mention, Franklin and Stevens, are historic and beautiful in hot real estate areas, so naturally there is a political fight about their disposition. But in any case this does not support your characterization that a new charter school can be forced from on high to locate to a dodgy neighborhood. The charters have their choice of location, and that is part of their strength. Why isn't Yu Ying considering the 32 acre Langdon site?

You listed several reasons why a school might suck, but the appropriate policy response (change the staff, upgrade the facility, or close the facility) depends critically on which of those reasons predominates.
Have you been to one of these schools, like say Eastern a few years back? The school had all of those symptoms and worse, like gang violence. One principal was fired after he was arrested for assault on a student. The "appropriate policy response" was the ultimate gut: take out its educational program, its personnel, and its physical plant, and start new. We will see if it worked.

I'm optimistic that policymakers would use empirically grounded behavioral parameters to drive decisions and predict the consequences of alternative choices for school quality, social stratification, transportation costs, etc.
They do: see for example, the re-opening of Van Ness, which was delayed because there were not quite enough people in the neighborhood yet.

But I think a better thing to advocate for, is that policymakers should think like its customers, the parents, who now have a choice where to send their kids. That means visiting the schools just as the parents do, and learn first hand if a school is working or not.

by goldfish on Feb 2, 2012 9:53 am • linkreport

I also don't know how you can do any legitimate cost-benefit analysis if you don't know how much each school is spending on instruction, teachers, administrators, "consultants," tutors and in the case of some charters, profits. Charters keep that all private. Charters are typically getting more supplemental foundation support than public (although they are getting some too). This cannot be counted on in calculating and projecting true costs into the future. Also: in their "tier" scheme there is no assessment included of attrition rates and graduation rates meaning, how many kids a school starts with, versus how many of the same kids make it out of that school.

by Natalie Hopkinson on Feb 2, 2012 5:52 pm • linkreport

This isn't really related directly to this post, but since you write for greater greater washington how do you think facility planning in general and conversions of local schools to charters where parents no longer have the right to attend is affecting the walkability of the city. I find that now with kids that I need to get to school I now drive more in a day for my kids school commuting than I used to in a week before I had them. There were thousands of families with elementary aged children who couldn't walk to school before Rhee closed 23 schools, the number must be much higher. Current education policy is directly undermining walkability, and neighborhood schools as community institutions. 21st Century schools fund has done some work on walkability and school choice but their last assesment was in 2006.

by Mary Melchior on Feb 3, 2012 10:00 am • linkreport

@Mary Melchior: spot on; check out any drop-off lane in front of a high performing school.
DC has around 30000 charter school students, of which around 50% commuters daily by auto. Fifteen years ago, these kids walked to school. Add to that the number of kids attending DCPS schools as out-of-boundary students, probably another 8-10,000 daily commuters. The newfangled charter-DCPS school market has added a lot of cars to the road. In my family we need two cars to handle the drop-off duties. If my kids were walking, we could probably get by with one.

But the alternative -- where to escape failing city schools, people either move to the suburbs or send their kids to private school -- is worse.

by goldfish on Feb 3, 2012 10:31 am • linkreport

Completely agree about the loss of neighborhood schools affecting not just transportation issues but also the sense of community. I'm very fortunate in that I live two blocks away from a high-performing school that makes our commute a breeze (as much as walking with an easily distracted five year old and his younger brother is a breeze). Part of the reason my local school is so successful is the parent involvement, and the in-boundary parents do more than their share of the heavy lifting. Of course, there are also out-of boundary parents who put forth monumental efforts without any guarantee that younger siblings will even be able to attend the school.

Part of this is due to charters and school choice, but part is also due to DCPS being unable/unwilling to fully fund smaller schools. Anything less than 300 students is unstainable.

by SE on Feb 3, 2012 10:49 am • linkreport

Bigger schools is the result of demands for more money spent on teachers and less money spent on "administrative."

Smaller schools mean more janitors, more principals, more secretaries per student.

Don't dump it on DCPS like somehow they are just against small schools. People constantly complain about the amount of money DCPS spends on the school system and people especially like to differentiate between spending on "teaching" and spending they like to characterize as "lining the pockets of 'administrative' people who do nothing."

by MLD on Feb 3, 2012 11:00 am • linkreport

@MLD: It might not have been clear from my post but I'm not dumping on DCPS. Bigger schools are more able to provide specials, reading specialists, and be more all-around cost-effective. However, I do think an unintended consequence is more difficult transportation issues and a loss of school community.

by SE on Feb 3, 2012 11:03 am • linkreport

Add a Comment

Name: (will be displayed on the comments page)

Email: (must be your real address, but will be kept private)

URL: (optional, will be displayed)

Your comment:

By submitting a comment, you agree to abide by our comment policy.
Notify me of followup comments via email. (You can also subscribe without commenting.)
Save my name and email address on this computer so I don't have to enter it next time, and so I don't have to answer the anti-spam map challenge question in the future.

or

Support Us