Comments on School rankings don't tell you what you need to know - Greater Greater Washington

Comment by David C

Fri, 20 Jul 2012 16:22:31 EDT

It's hard to say how much schools matter. After the Hopwood case made race-based admissions difficult, Texas changed to a Top 10% rule, which admitted any student who graduated in the top 10% of their high school to any state college - and since high schools tend to NOT be diverse, this resulted in a very diverse set of students.

What they found was that the graduation rate went up. Prior to the top 10% rule, kids who graduated in the middle of their high school class but had high SAT scores got in and kids who graduated high but had low scores didn't. The theory is that graduating high in your class shows work ethic and work ethic is far important to graduating from college then basic math and English skills. So, which high school you go to is only important to the extent that it does or does not teach you how to work hard and efficiently.

Comment by LeeinDC

Thu, 19 Jul 2012 09:41:16 EDT

David thanks for the article.

I would agree that a Value Added measure that was believable would be good. I haven't seen one yet but efforts are being made in that direction (impact etc) The DC CAS scores tend to give you an objective feel for the competence of the kids in a school. It doesn't tell you why they score well (teacher, family, peers).

As a parent of 4 DCPS and Charter students I am most concerned about peers, which DC CAS addresses. My kids have had great teachers at "so-so" schools and middling teachers at "great" schools. So that is hard to control for. Peers are what matter most and what you can get a feel for in DC.

Comment by thm

Thu, 19 Jul 2012 02:35:14 EDT

If there were a series of statistical techniques that could do what VAM purports to do, which is to specifically measure a teachers' contribution to student achievement, "controlling" for all the other factors, it could be quite a useful tool. It would be especially useful for those whose approach to education and education reform centered around teacher quality. The potential value of such a tool has led to a significant effort in trying to develop one.

But asserting that VAM can measure what it says to measure does not make it so. Any regression analysis presumes, among many other things, that the dependent variable (e.g. student test scores) can be expressed as a certain linear combination of a certain set of independent variables (e.g. teacher quality, income level, parents education), and 'controlling' for various factors means changing around the parameters that are included and re-running the computations. If the underlying model is wrong, you'll still get numerical results, but they will be meaningless.

This is what the result from Rothstein (cited by, but not written by Darling-Hammond) demonstrates. If the methods of VAM show that fifth-grade teachers predict fourth-grade scores, then there is something very wrong with the model and the methods. I don't see how one can get around that. That's why I call it 'numerology.'

The New Teacher Project 'fact sheet,' and the Brookings report it references to support the claim that there exist academics who believe VAM has "an important role to play" do not even attempt to address this problem. They simply assert that VAM does what it purports to do, which Rothstein has demonstrated that it doesn't.

Comment by SchoolWatcher

Wed, 18 Jul 2012 22:27:19 EDT

Sorry, to be clear, I thought your points were fair ones against value-added at the teacher level. I was (in a single paragraph) pointing out that the study you linked to represented one of several viewpoints from the academic mainstream, but also disputing thm's dismissal of value-added as a valid measurement. I don't want to confound those points -- while value-added is controversial for the very reasons you mentioned, it's also not gibberish.

arf - While you won't find many of the statistics about family or parental income levels, you should be able to get information about a course curriculum from a principal. They may also be willing to share average student growth on widely-used comprehensive tests (GOLD is one example for DCPS in the ECE environment.)

Comment by Gray

Wed, 18 Jul 2012 20:02:57 EDT

@SchoolWatcher:

It's definitely not "numerology," -- it's a regression, and founded in statistics -- even if there are folks who believe the variabilities of a single class are too high to use it in individual teachers' ratings.

I couldn't tell that you were responding to me because I never claimed it's magic or not founded in basic statistical theory.

Yes, it's a regression. And yes, the error term will often swamp any actual effect at the class-year level. It's a useful tool at the aggregate level, but applying it to measure the quality of individual teachers is not a statistically sound practice.

Comment by arf

Wed, 18 Jul 2012 17:43:46 EDT

As a DC parent of a 2 year-old entering next year, I want more than anything a nucleus of middle class parents around me.

The DC-CAS tests reflect information - starting in third grade - on test proficiency, race, economic status, and language.

1. Test proficiency is a limited predictor of parental education, mixed with teacher involvement.

2. Outside of white children(due to known information about incomes of white DC families), race or Hispanic background is a poor predictor of middle class/striver parents.

3. Economic status predicts only part of what you need to know if you're looking for middle class families, but the break isn't set at middle income vs. poor, it's set at poor vs. very poor.

4. The language competence thing is only relevant in DC's Salvadoran-heavy schools, of which there just aren't very many.

5. And I don't see anything in these "report cards" that reflects antisocial behavior or a teacher's added value.

Does anybody know how to get or get proxies for the following?

1. Some kind of proficiency data for preschool or Kindergarten, to see whether the kids are already behind by the time they start school.

2. How many children have at least one live-in parent with an accredited college degree.

3. The number of parents who can contribute significantly to school improvement with either time or money.

4. Parental unemployment rate.

5. Number of security incidents that took place, by year, by problematic student's grade in school.

6. Subjective, non-numerical reviews of the strengths and weaknesses of your kid's potential teachers by more than one qualified outsider.

7. A published curriculum for a given school year.

So - fault all of this as much as you want. These are the data I want as a potentially entering parent.

Comment by SchoolWatcher

Wed, 18 Jul 2012 16:49:37 EDT

Gray -- I was referencing the study you linked to and the post by thm. I see where you're coming from, but wanted to clarify that value-added is an idea entering the mainstream of teaching and, like you said, should be measurement of a school's success. It's definitely not "numerology," -- it's a regression, and founded in statistics -- even if there are folks who believe the variabilities of a single class are too high to use it in individual teachers' ratings.

Comment by Gray

Wed, 18 Jul 2012 15:03:28 EDT

@SchoolWatcher: I'm not sure if you're responding to me, but I'm not arguing that value-added methods are not useful. I'm arguing that they are inappropriate for measuring individual teachers' effectiveness. They are better than the simple methods David Alpert references for measuring school-level or otherwise aggregated effectiveness, but should not be used for yearly evaluations of teachers.

Comment by SchoolWatcher

Wed, 18 Jul 2012 14:44:34 EDT

I find it necessary to point out that not all academic research is dismissive of the use of value-added data. Both reports cited so far in this thread list Linda Darling-Hammond as a co-author. She's an admirable educational researcher but also an outspoken critic of many signatures of the current education reform movement.

The New Teacher Project (Admittedly on the opposite side of the spectrum from D-H) points out several ways that value-added is useful: http://tntp.org/ideas-and-innovations/view/myths-and-facts-about-value-added-analysis

The Gates Foundations' MET project (http://www.metproject.org/) also advances the policy of including value-added as one of several measurements of teacher effectiveness.

Of course, value-added data is only as good as the test it's based on, and small sample sizes can be problematic. But it's simply untrue that it's mere numerology.

Comment by Tom Veil

Wed, 18 Jul 2012 14:29:55 EDT

David, excellent point about drilling into the demographic data to see how a school is performing. I've noticed that most of the schools in Capitol Hill and West End/Dupont exhibit this feature, where once you realize just how many poor students are attending the school, and how well the presumably richer students are doing, all of a sudden its achievement levels are comparable to Arlington and MoCo schools.

Comment by thm

Wed, 18 Jul 2012 13:47:30 EDT

David correctly points out in this article what is, upon reflection, an absolutely obvious and tremendously important point, but one which is routinely ignored: To say that test scores went up, you give a test at one point to a group of students, then you give another test some time later to what is not necessarily the same group of students. There are three ways in which the test scores can go up: low-scoring students from the first test do not take the second test; additional high-scoring students who were not part of the first test do take the second test; or the performance of individual students can improve. I've listed them in order of difficulty and whenever a claim of a test score rise is made, the claimant ought to be required to prove that the rise is not the result of the first two effects.

David downplays his item #3, "The influence of other students," writing mostly about how teachers would be able to allocate time, or if there are problems with bullying. But of course the influence of other students is important, because being in an environment in which one's peers are thinking about the same problems and asking the same questions has synergistic effects. Being surrounded by others who are passionate about any subject or pursuit makes one more passionate about that subject, and the result of the positive feedback loop (in the engineering sense, not the words-of-praise sense) is dramatic. It's the same reason why high-tech companies still have physical offices in which creative and technical folks gather daily.

Finally, the Value Added Modeling, which is a component of IMPACT, is meaningless numerology. Here's an overview of quantitative methods of evaluation teacher performance which references this examination of VAM methods. Quoting

Surprisingly, it found that students’ fifth grade teachers were good predictors of their fourth grade test scores. Inasmuch as a student’s later fifth grade teacher cannot possibly have influenced that student’s fourth grade performance, this curious result can only mean that VAM results are based on factors other than teachers’ actual effectiveness.

And another study they reference found

across five large urban districts, among teachers who were ranked in the top 20% of effectiveness in the first year, fewer than a third were in that top group the next year, and another third moved all the way down to the bottom 40%

All of which I take to mean that no amount of fancy statistical methodology can get around the basic maxim of "Garbage In, Garbage Out."

Comment by Falls Church

Wed, 18 Jul 2012 13:34:26 EDT

For school ratings, I think GreatSchools.org is pretty useful. Both it's overall 10 point rating scale and the underlying data provide a good starting point for evaluating a school.

Don't forget that you should also look at the specific programs a school offers that may be of interest to your specific kid. For example, if your kid is interested in math/science, does the school have a magnet program or partner with organizations such as NIH for special programs? Or if your kid is more linguistically oriented, is there an IB program? Also, extracurricular offerings are important too which isn't something that's captured in test scores. For example, not every school in Arlington even has tennis courts if your kid is a tennis player. H-B Woodlawn only has one sports team for the entire school -- ultimate frisbee.

Comment by KL

Wed, 18 Jul 2012 13:29:12 EDT

At our school, we have amazing teachers and a great school culture, but we feed into a truly terrible middle school. As a result, around 3rd grade many, many of the parents start trying to lottery into an elementary school that feeds into a better middle school. The vacant spaces are taken by out of boundary kids, many of whom come in very behind. So, the students being tested are not the same students who have benefited from our great PK, K, 1st, etc. teachers. It brings our scores way, way down.

Comment by JustMe

Wed, 18 Jul 2012 13:13:25 EDT

The thing about "value added" might be that it is optimized towards adding value for those at a certain level. A school that is best at adding value to students reading well under grade level might not be a good fit for a student who is reading beyond his own grade level.

Comment by DCster

Wed, 18 Jul 2012 13:09:20 EDT

If the interest is comparing teacher quality across schools, then the IMPACT data feeding into the number of 'highly qualified' teachers would be the available data to use. Though changes in the school population (a school closure resulting in an influx of students as Oboe mentions), or stability in school leadership would also be important considerations.

Comment by Gray

Wed, 18 Jul 2012 13:03:09 EDT

@SE: This looks like a good start. It has an extensive bibliography with lots of references that are consistent with what I'm arguing.

http://www.edweek.org/ew/articles/2012/03/01/kappan_hammond.html

Comment by SE

Wed, 18 Jul 2012 12:54:50 EDT

@Gray

Do you know any studies/academic papers backing up what you're saying here? I'd be interested. Thanks.

Comment by SE

Wed, 18 Jul 2012 12:52:42 EDT

An average value-added measure is useful, but still not really enough on its own. I'm most interested in how my particular student is going to improve. Let's assume he's an above average student. If a school is full of below average students, but really great teachers so that the students improvement is substantial, it still might not be a good fit for my own kid. Of course combined with the already availabe school report cards, maybe that would give a good idea.

I think the bottom line is that as much as I love data and realize its value, choosing a school must also include visiting individual schools, considering intangibles such as school atmosphere, thinking of the particular desires for my kid (language immersion), and particular desires of my kid (arts integration, etc.).

Comment by Gray

Wed, 18 Jul 2012 12:50:05 EDT

@MLD

That's why the test metric is only half of the score and not the whole thing. The other half is made up of reviews from principals and multiple classroom observations.

Yep, that's why, but it shouldn't make up half either. It isn't an appropriate metric for evaluating individual teachers.

Comment by oboe

Wed, 18 Jul 2012 12:45:49 EDT

Let's assume that you have a school with a bunch of terrific teachers that is doing a great job educating its kids. One year, a bunch of lower-income kids come into the school. Let's say that the teachers do just as terrific a job educating the existing kids and the new kids. Existing kids don't lose out at all. Yet the school's average test score will go down.

You've just described my local elementary school. Great building, great teachers, motivated and high-performing students in the lower grades. But the students who are in the testing grades are all out-of-boundary, and almost none of the kids in the testing grades has been there for more than a year or two. (Not one of the fifth graders has been there since K).

Of course, the fact that the test scores in the upper grades are so low (coupled with looming middle school) make it less likely that parents will stay which simply reinforces the problem.

Comment by MLD

Wed, 18 Jul 2012 12:39:06 EDT

@Gray

That's why the test metric is only half of the score and not the whole thing. The other half is made up of reviews from principals and multiple classroom observations.

Comment by Gray

Wed, 18 Jul 2012 12:32:05 EDT

@Trulee_Pist:

An average "value-added" score for teachers at a school would help parents make choices, but only if that measuring tool for "value-added" is accurate, well-designed and statistically defensible.

I understand it is not. Is there any evidence that this tool measures what it says it measures?

Why do you think it isn't?

The main issue with the value-added metric has been how DCPS is using it. It works pretty well for analyzing aggregate performance, but should never be used to compare performance over multiple small sample sizes.

We can simplify it this way: each teacher has some quality level, and she is able to increase test scores for students at some rate consistent with that quality level. A value-added metric allows us to control for differences in students so that we can estimate that quality level. However! What we can't control for is the idiosyncratic differences in scores from day to day. If five students in a class of 25 arrive on a late bus and miss their breakfast before the test, we shouldn't use those results to argue that their teacher is worse than she should be. But if we take the value-added metric across many more students, we can allow those negative shocks to be roughly canceled out by positive shocks, and get a better measure of overall quality.

The additional issue with DCPS's use of the metric is that some teachers have cheated on the tests. This worked great for them, but no so great for their students' next teachers, who were told that they actually subtracted value from their students.

Comment by citywalker

Wed, 18 Jul 2012 12:31:36 EDT

The stats also don't tell you whether students at the school are encouraged to ask good questions (the tests only involve answering questions) and invent their own approaches, or even whether or how well they're taught science, social studies, and the arts. They don't tell you how many hours of instructional time the students spend taking practice tests, and whether first graders come home expressing panic about "the big tests" (as mine did), several years before they have to take them. They also don't tell you whether a child who falls behind will be taught at her/his own level or tacitly encouraged to leave.

Comment by Trulee_Pist

Wed, 18 Jul 2012 12:17:17 EDT

Instead of answering the question, what percentage of kids at a school are doing well, which is very dependent on who goes there, this number [IMPACT "value-added" score] would say how much each kid will probably gain from going to the school, which is really what will help parents make choices.

An average "value-added" score for teachers at a school would help parents make choices, but only if that measuring tool for "value-added" is accurate, well-designed and statistically defensible.

I understand it is not. Is there any evidence that this tool measures what it says it measures?

Comment by Megan

Wed, 18 Jul 2012 12:16:27 EDT

David, thanks for running this series. As a parent in a neighborhood with mediocre-to-bad schools, I really struggle with knowing where to draw the line between an acceptable school whose potential (minor?) deficiencies we can make up for with "Component 1", and a school that is actively interfering with my child's ability to learn.