On Thursday, FiveThirtyEight, a national blog popular for its analytical take on everything from Donald Trump to the NBA, published an article claiming that there were 85 fires on the Metrorail system— over four fires per week— between January 1st and last week. That claim is built on inaccurate data, and it overstates the actual risk of fire on Metro.

The graph that FiveThirtyEight @LeahLibresco used in her article.

FiveThirtyEight’s data is wrong

FiveThirtyEight arrived at its count of fire incidents on Metro by tallying up the number of times @IsMetroOnFire has said there was one. The story says @IsMetroOnFire simply tweets every time either the @metrorailinfo or @metroheroalerts accounts mention smoke, fire, or fire department activity, but it looks like @IsMetroFire also tweets anytime anyone using the #wmata hashtag mentions these keywords as well.

The problem with depending on tweets from @IsMetroOnFire, or really from anywhere in the Twitterverse, is that it’s very easy to miscount or classify problems incorrectly— in other words, @IsMetroOnFire sometimes says there was a fire when there wasn’t, and FiveThirtyEight’s story didn’t account for that.

For example a tweet yesterday from @IsMetroOnFire reporting smoke/fire was technically correct: A train offloaded at a Red Line station for a brake problem that caused smoke. But there was no fire in the station, on the tracks, or in the train.

So how many rail fires were there actually?

Metro itself reports official totals of smoke and fire incidents each quarter. Through March 2016, the report was 24 fire incidents, including insulator fires and smoldering rail ties. Another 24 smoke incidents, like burning brakes and brake dust, were reported over that same period.

This is a lot lower than the approximately 75 incidents of fire that FiveThirtyEight told readers occurred through the end of March (the 85 that was reported as the total through May 16th is a bit harder to address those last 10 since Metro won’t release April and May data until the end of the quarter). I monitor Metro regularly, and if I’m at all accurate, there have still not been anywhere near as many incidents of fire on Metro than what FiveThirtyEight suggests.

Also, the Metro data differentiates “smoke” and “fire,” with a smoke incident being something like smoking train brakes while a fire incident could be an arcing insulator or a wooden rail tie fire. Either could lead to the fire department showing up, but only one is a fire.

Also, sometimes the fire department coming has nothing to do with smoke or fire. They might come for a sick passenger on a platform or a false fire alarm, but as long as someone tweets that they came, @IsMetroOnFire reports that there was a fire.

When your graph claiming to show Metro fires depends on tweets related to the fire department, the information it conveys is bound to be off.

Of course, we’d rather there be no fires at all

It may be pretty obvious, but fire in a transit system is bad; in no way do I want to marginalize that fact. Even the 54 incidents I’m aware of this year, which includes both trackside smoke and fire incidents, is way too high. Fire and smoke are what caused a death at L’Enfant last year, what shut down the rail system for a day, and what have caused the FTA to order new Safety Directives on WMATA. The goal, of course, should be zero incidents.

If your’e curious about how many fires another major subway system has, New York City’s MTA reported an average of just over 21 “subway fires” per week from March 2015 to February 2016 (1022 in the year), and 20 per week in the year prior (963).

538 is well-known for their data-driven journalism, and rightly so. But using misleading data to justify calling Metro unsafe is is unfair at best.

Stephen Repetski is a Virginia native and has lived in the Fairfax area for over 20 years. He has a BS in Applied Networking and Systems Administration from Rochester Institute of Technology and works in Information Technology. Learning about, discussing, and analyzing transit (especially planes and trains) is a hobby he enjoys.