Posts about Data Openness

My recently completed master's paper analyzes the factors behind the number of trips at different Capital Bikeshare stations. I created a regression of trips in October 2011 that began at stations in the District. After controlling for 14 variables, the analysis concludes that 5 key factors primarily determine a station's usage:

• The population aged 20-39
• The level of non-white population
• The retail density, using alcohol licenses as a proxy
• Whether Metrorail stations are nearby
• The distance from the center of the CaBi system

I measured each variable based on what's within a ¼-mile walk of each station. With that information, I created a "suitability map," above. For any spot in DC, it projects how much ridership a station would get if DC placed one there. You can also download the KML file to view the analysis in Google Earth.

The map shows that as you get farther from the main activity centers in central DC, there's a dramatic drop-off in station demand. Approximately 13% of Capital Bikeshare stations, as of March 2012, are located in areas where we would expect fewer than 18 trips a day. The actual usage data shows that a significant number of these stations at the edge of the system have even fewer trips.

There are equity reasons to place stations outside the core; policymakers want to make sure that money spent on Capital Bikeshare benefits more than just those who live and work in central areas, and it builds political support from councilmembers representing wards farther away. However, there are multiple areas around the District that are under-served by bikeshare today, yet highly suitable under the analysis.

Planners and policymakers should consider these areas as they build out and tweak the system in the coming years. The figure below shows the coverage gaps by overlaying the existing bikeshare stations and the suitability map.

What can we conclude from this? The Washington region and other cities should consider the following issues when they plan and expand their systems:

Distance from the center matters. This variable accounts for 60% of the variation among station usage, by far the most of any factor. This matches a principle known as the "gravity model" in transportation planning, which predicts more trips between closer locations. Capital Bikeshare's pricing structure also encourages shorter trips by charging for using a bike over 30 minutes, which strengthens this factor.

Carefully weigh goals of equity and coverage against ridership. It's very important to provide active, multi-modal transportation options to low-income and minority communities, and this study does not dispute that. That being said, it is important to carefully assess the tradeoffs among various objectives, especially in light of the relative costs of providing other mobility options for individuals of lower socioeconomic status.

Suburbanization of bikeshare has opportunities and pitfalls. The prospect of a region-wide bicycle sharing system in the nation's capital is an alluring one to advocates. It is easy to imagine a robust, polycentric system around dense nodes like Alexandria, Arlington, Bethesda, College Park, and Silver Spring.

However, some facts could temper that enthusiasm. Even some relatively close-in stations in the District have very low usage. Nearly 40 of the 97 stations in operation during October 2011 experienced 15 or fewer trips a day. Similarly, the densest parts of Arlington, with 18 stations during the same period, had 15% of stations system-wide but just 5% of trips.

To successfully expand bikeshare into the suburbs, planners need to choose station locations wisely, and elected officials need to invest enough to create a critical mass of stations early on. If we rush to build an inadequate suburban system, then it will likley not meet expectations and could act to blunt public support for the program, precluding a more economically sustainable system later on.

Stations can easily move as we learn more. Within a matter of hours, bikeshare operators can load stations on a truck and redistribute them to more suitable locations. While Capital Bikeshare operates year round, colder cities like Montreal and Boston take their stations away each winter. Planners there use the spring launch of the system to refine the location of their stations based on station performance the previous year. Capital Bikeshare should schedule an annual station redistribution.

Promote open bicycle sharing data. Having this data available to graduate students and anyone else promotes transparency, scholarship, and innovation. Bicycle sharing systems are proliferating rapidly, which is very encouraging, but few systems nationwide release trip data.

For instance, despite $4.5 million in grants from public sources ($3 million from the Federal Transit Administration), data from Boston's Hubway remains proprietary because of a private sponsorship agreement with New Balance. New York also hopes to fully fund its system with private dollars, which creates a danger that the same may happen there.

Like other North American cities, DC relied on international practices to plan its original system. Now, with an ample stream of data and more than $13 million in public funding committed to the regional system, it is time to strategically reassess station locations to ensure that bike sharing remains viable for the long term, as a true transportation investment.

Jantzen writes:

The bikes don't have GPS transmitters on them, so all we know is the data for the start and end of each trip. Thus, all dots are shown moving in straight lines. The speed of course doesn't correspond to the rider's actual speed, since we don't know their routes, or whether they paused for stops along the way. ...

The CaBi data includes whether the rider was a casual or registered user, depending on whether the rider's membership was for a month or longer. The movie shows casual users with dots that fade from green to yellow. Registered users are represented with dots that fade from blue to purple. A histogram on the right records the number of bikes in use at each moment.

The movie excludes the 210 trips that began and ended at the same station. 63% of these "round trips" were made by casual users. (Casual users made up 32% of the other trips.) You will still see some dots that appear to be stationary, but they are actually moving very slowly, representing people who go far, far beyond the 30-minute time limit for free trips.

It would be interesting to see one for a weekday as well. We know weekdays look very different from weekends.

Update: Jantzen realized that the original video used data from November 20, 2010, not 2011 as he originally planned. He has now updated the video and the new video is in this post. You can also still see the original video. The new video shows data for Saturday, November 19, 2011.

Mouse over a station to see the trips from that station. The blue circle shows the magnitude of trips starting at that station, and the red circles show the relative number of trips ending at each station.

Rahul Nair, a faculty research assistant at the University of Maryland, has created some great visualizations as well. One shows Capital Bikeshare trips from one selected station. It's displaying the same type of data as the MV Jantzen app, origin-destination pairs for a single origin, though it uses arrows instead of bubbles.

Nair also built a tool showing travel times on the Beltway, using data from INRIX for 2011:

Here's an example service disruption from a report I received from a WMATA insider:

TRAIN GOES TO B4 AT POINT OF POWER, HAVE TO CUT OUT ATP TO MOVE, NOT DISPATCHED, K08, CMD, ATCC, 918

Other reports are a little simpler to understand:

NO ALL DOORS CLOSED CUSTOMER POSSIBLY HOLDING THE DOORS

A lot of this message wouldn't make sense to the vast majority of commuters. WMATA could still post these with a glossary that helps decode even this cryptic report, though there is the possibility that customers would see them and be confused, or call in to customer service about it.

Instead of posting these, WMATA created a "Vital Signs" report, which lists a few high-level metrics like overall rail on-time performance. But one number for rail on-time performance hides a lot of important information. A train can be late up to half the headway and still count as on time, meaning that when trains run every 20 minutes, trains could still be 10 minutes late or early. It doesn't include performance during planned track work, and other factors.

Today, WMATA's approach to public information seems to be to release only a few conclusions, not any deeper information. When the Riders' Advisory Council or others have asked for more, they've been told that it's the job of staff, and nobody else, to analyze data and tell the public and press what to believe about the issues.

But to many riders, this isn't satisfying. WMATA officials say they're aggressively fixing problems, but will those fixes actually lead to better performance, and when? So far, the agency has just cut the on-time performance target from 95% to 90%. It's never met its goal for the frequency equipment breaks down ("mean time before failure") since the data have been reported, and does not appear to be improving.

It's no secret that WMATA's reputation as a reliable transit service is tarnished by frequent service delays and offloads. If Metro begins to publish these reports again, customers could decipher the differences in service disruptions that are the fault of customer behavior like blocking doors, sick passengers, or police activity, and those that are due to maintenance issues like brake, track control circuit, or door problems.

Compare this to San Francisco and Chicago, two transit agencies that have longer histories of reporting service data.

Chicago reports number of rail delays of 10 minutes or more, percentage of track that is affected by a slow zone restriction, miles between rail vehicle defects, percentage of the rail fleet unavailable for service, and percentage of customer complaints not closed out within 14 days.

San Francisco reports how closely they're meeting the schedule (similar to WMATA), how the headways match up against the plan (more useful to customers for frequent routes), the amount of service, late pull-outs, overcrowded vehicles, the number of unexcused absences, mean distance between failures for trains, vacancy rates for service-critical positions, and the complaint resolution rate within 14 days.

San Francisco and Chicago implemented better performance reporting as part of an effort to regain the public trust after a long decline in service. Metro should do the same in a concerted effort to truly move Metro Forward.

O'Brien notes that this is just an estimate. For example, the model assumes that people are very likely to take a cycle track if there is one, which pushes a lot of bike traffic onto the 15th Street lane. He doesn't have data on how many of the rides actually do use 15th versus 14th or another road.

It also doesn't have the Memorial and 14th Street bridges, because the bike paths are discontinuous, so the algorithm assumes everyone going between DC and Crystal City takes the Mt. Vernon Trail and Key Bridge.

Nevertheless, this is an amazing and potentially very useful tool. For example, there are a lot of trips east-west, which the algorithm mainly guesses to take N Street since M Street is such an unfriendly road for cyclists. When DDOT builds a cycle track there, we know there will be a lot of demand and likely very significant usage.

JDLand's Jacqueline Dupree mapped the data showing rides to and from the stations in Near Southeast.

I don't know how much manual work was required to prepare data for her system, but it would be great if the site could let a user view trips for any of the stations in the system in the future.

A different JD, Justin "JDAntos" Antos, dug into the data and devised some very informative graphs.

Lydia DePillis recently noticed that casual members are much more seasonal than longer-term members, leading to big spikes in ridership during the warmer and more tourist-heavy months.

Antos discovered that the percentage of trips by casual users are also far higher on weekends, but the difference is much more pronounced in spring, summer, and fall than in winter:

Confirming what Corey H found, longer-term members are much more likely to return bikes under the 30-minute no-charge threshold than casual members.

Perhaps not surprisingly, Antos also found that trips on weekdays occur much more in rush hours while weekend trips are spread in a bell curve throughout the day.

Read his and the other posts for more great nuggets. What else would you like to know? Have you created any graphs or maps to share?

The average [such] trip is 2 hours, 48 minutes. 76.1% of [these] trips generate usage fees. Breaking that down between casual and members, 86.0% of casual incurred fees on these rides while 18.8% of members incurred fees.

4) Casual vs. members usage fees. 40.7% of casual rides incur fees. 3.3% 3.3% of member rides incur fees.

*Using GPS elevation data so all caveats apply. And it only factors in station-to-station elevation change.

Corey added in an email, "I've already taken the time to clean the data and get it into a usable database. So if there are specific questions you'd like to be answered I can easily put together a query to get those answers (and I'm sure the others can as well)."

What would you like to know about Capital Bikeshare usage? He can't necessarily investigate everyone's questions, but if anyone posts some interesting questions that catch Corey's eye, maybe he will analyze them for us.

What the columns mean

Creating public GTFS feeds (the 1st column) allows someone who's written an app to easily incorporate schedule and route data for a transit agency. GTFS has emerged as a national standard for representing transit feeds, and there's tremendous value in having as many agencies as possible support the same standard. That way, if someone writes an app in Chicago, they can make it work in Denver, Albany, or Miami at the same time.

Most of the transit agencies' feeds including the paths that the vehicles take, but some do not, like DASH. The 2nd column shows this information. Feeds without paths are still usable, but apps that visualize routes, like Transit Near Me, end up showing unsightly diagonal lines cutting across city blocks.

Agencies can also sign a contract with Google to have their routes and schedules on Google Maps. The 3rd column shows agencies which have done this. Some agencies put out their data files, but aren't willing to sign this contract because of indemnification or other clauses which Google unfortunately insists upon. On the flip side, some agencies sign up with Google but then don't publish the GTFS feed publicly.

The agency might provide it to those who ask, or might not, but this dissuades app creators from including this agency, and makes it harder for them to get regular updates. Every agency should strive to host a public and up-to-date GTFS feed on their site so that anyone building apps can easily incorporate that agency's services into the tool.

The other type of open data is real-time locations or predictions. To make this possible, agencies first have to deploy AVL (Automatic Vehicle Location) technology on their buses or trains (the 4th column). The main obstacle is that this is somewhat expensive; a physical device has to go into each vehicle, and those devices then need some amount of maintenance over time.

Once an agency has tracking, it's relatively simple to offer a computer interface for apps to access and tell riders about this information (the 5th column). Most of the agencies with tracking offer such an interface, but while Ride On, MARC, and Loudoun Transit all have public tracking sites that provide some services to riders, but no way for other apps to tap into the information those sites contain.

What agencies can do

Agencies with red X's on this chart can start thinking about how to provide schedule and/or real-time open data. Creating GTFS files isn't extremely difficult, though it does require some staff time to actually do it. For agencies that use scheduling software, the manufacturers of that software often offer modules to export data as GTFS as well.

Some GTFS feeds could benefit from quality fixes. For example, WMATA's Metrorail GTFS file doesn't show the specific paths trains take, and paths are missing for a few bus routes. The "Transparent Metro Data Sets" Application Programming Interface (API), a special interface WMATA created to offer access to much of its data, does include the correct paths. But many people develop apps to access GTFS files for multiple cities. It's much less likely they will put in extra development effort to specifically pull just these route shapes from this unique API.

The Circulator's routes are part of the WMATA GTFS feed, which makes things even easier for apps than having to download a separate feed. One problem is that the route names are all cryptic: there's "DCDGR" for the Dupont-Georgetown-Rosslyn Circulator, or "DC98" for the route which replaced the former 98 bus. Those are fine for internal systems inside the agencies, but they aren't very clear to riders.

Agencies which have provided their data to Google but don't offer the feeds publicly (like DASH, Ride On, and MARC) should post those feeds on their websites and publicly link to the feeds. They are already creating the GTFS files for Google, so it's a trivial step to also let others download the same files.

WMATA also has much of the route data for other local bus systems in the region as well, which it uses in its trip planner. Agencies which don't have GTFS files can give WMATA permission to include their data in its GTFS feed, as the Circulator does.

Agencies with AVL systems already on their vehicles should set up APIs to give apps access to the locations or predictions, and agencies without AVL can work toward getting the budget necessary to deploy AVL.

What others can do

Transit industry associations and vendors which sell technology to transit agencies can all encourage open data to be part of any contract. Vendors can encourage agencies to open their data and provide services to do so, and associations can encourage agencies to ask their vendors for these services.

The industry can also help move toward a clear standard for bus tracking. GTFS has become a standard for schedule and route data because large numbers of agencies went ahead and offered GTFS files. But there is not yet a consensus around what format to use to offer real-time predictions.

WMATA built its own API which provides the data in a certain format. Circulator, The Bus, and CUE all use Nextbus for tracking, which has its own API. ART uses another service, Connexionz. This unfortunately means that anyone building a real-time application and wants to incorporate multiple services has to support at least 3 different APIs.

There are efforts to create such standards, like GTFS-Realtime, but this hasn't realized the same widespread adoption as GTFS, nor has any other standard.

It's still possible to build apps without a standard, and the Mobility Lab's real-time screen project does connect to all 3 different systems in our region. But that requires extra work, not just for the Mobility Lab but for every other app creator who wants to offer predictions for multiple transit agencies.

The easier we make it to build apps, the more we'll get. Ultimately, it would be great for one standard to emerge, and for the various vendors like Nextbus to agree to all offer data to apps in that same standard format.

Update: Commenter intermodal commuter pointed out the real-time status page for VRE. It combines some train positions from GPS and some from manual reports from conductors. There is not an API to access the data. I've corrected the chart.

Update 2: Commenter Adam noted that MARC is actually contained in the MTA Maryland GTFS file, but listed only as routes 300, 301, and 302, which we didn't realize were not commuter buses upon examining the feed. But you can see the MARC lines on Transit Near Me (for example, center around Union Station).

Also, ACCS Web Manager Joe Chapline posted a status update about ART's efforts to get into Google Transit; according to Chapline, this was delayed for a time due to contract issues, and now is awaiting action by the Google legal department, which I know from past personal experience is often understaffed and backlogged.

This data has been available for some time for London, allowing people to create animations of a day's CaBi usage and diagrams of a single bike's path over several days. The folks who built those and other tools can now even adapt their code to work for Capital Bikeshare, if they're so inclined.

Arlington, DC and Alta officials agreed in November to offer the data, after discussions with Tom Fairchild of the Mobility Lab, lab collaborator and advisor Matt Caywood, and CaBi Tracker creator Daniel Gohlke. (I am working on projects for the Mobility Lab as well, but was not involved in this specific discussion.)

To make it even easier to work with the data, Dylan Barlett imported the files into Google Fusion Tables, a tool that lets people easily sort, manipulate and visualize data.

If you put together an interesting analysis or visualization, please send it to us! We'd love to post interesting things you come up with using this or other open data.

One of the main challenges in convincing people to switch to transit is the unpredictability of bus arrivals. If every stop featured a digital screen displaying the number of minutes until each bus arrived, more people would be willing to take the bus.

Outdoor screens, however, are expensive to install, which is why we created this indoor alternative at a fraction of the cost. For the past few months I have been working with Andy Chosak and David Alpert at the Mobility Lab in Arlington to bring this low-cost alternative to fruition.

Every 20 seconds, our web server queries each transit agency for the arrival predictions for the stops near both test sites, then relays the data to the screens. The actual unit inside the shops is just a low-cost, barebones Linux system connected to a standard computer monitor and the business's own Wi-Fi and power. We've configured the box to automatically load up the screen when it starts, so there's no need to log in or launch an app after the unit is plugged in.

We are continuing to build the system so it can be deployed quickly and cheaply throughout the region at participating shops, bars, cafes, and restaurants. Ultimately, a business will be able to sign up, type in their address, and get a screen automatically customized with the nearest bus stops, Metro station, and Capital Bikeshare station. And someone with their own computer connected to a standard computer monitor will be able to set up their own screen for free.

This project is only possible thanks to open data from our transit agencies. We can only pull bus and train predictions as well as the status of each CaBi station because the agencies behind these systems have wisely chosen to provide stop locations, route information, and real-time arrival predictions to outside software developers.

If you run a businesses are interested in finding out more about purchasing one of these screens for your location, let us know at screens@mobilitylab.org.