Article on Open Data For Civic Apps

In the January issue of Communications of the ACM, Melissa Lee, Esteve Almirall, and Jonathan Wareham, have an interesting article on “Open data and civic apps: first-generation failures, second-generation improvements”.

 Reviewing the recent history of US government “open data” efforts, they report that, after an initial splash in 2010, interest waned. Simply making data available accomplished little, and pushes such as the “Apps for Democracy”  created apps, but they were not widely used, did not have high quality, and did not generate social or economic value to extent hoped. Lee and colleagues examine these “first generation efforts”.

As it was apparent that simply dropping gobs of data out there would do little, efforts were undertaken to encourage new uses of the datasets available. We all remember “Apps for Democracy” and the other development contests which have been the predominant strategy to encourage use.

I certainly remember being a little excited by these open contests (let everyone play!), but also how quickly I realized that the open datasets are less than thrilling, and, worse, I have no particularly good ideas what might be done with them. I gave up, though a few folks persisted and created some apps. Good for them, but how well did they pay off for the public, for governments, for businesses?

Lee et al. conclude that these apps had limited value to either government or the public. They argue that the developers in the open contests, like me, had little understanding of the problems of local government, and mainly worked based on their own limited experience, producing multiple apps about restaurants, parks, and public transport. Many of the developments simply incorporated open data into existing apps, which did not offer new services.

There were also problems with funding and promoting the successful apps. A contest is a good way to get recognition, but achieving widespread usage requires external funding and publicity. Contest organizers (typically municipal governments) did not provide much help in these areas.

These contests were not well connected to the operations of the municipal government, which had to provide data, but had little contact with the app development, were not accountable for them, and generally did not expect value from them. Indeed, in some cases it may be illegal for a department to request specific apps or features, because this violates procurement rules requiring open bidding. No wonder when the apps had little impact on government operations.

Lee et al outline the “second generation” of civic apps, which built on these lessons learned.

First of all, they recommend what amounts a “call for solutions” by municipal agencies, to help educate developers on perceived needs. The customer is not always right (sometimes there is a better way that they haven’t thought of), but it’s stupid to ignore people who have daily hands-on experience.

Another idea is to abandon the contest approach, and solicit developers to work directly with municipal agencies and citizens to develop apps in a more deeply informed way. In some cases the contest approach has been replaced by top-down management to develop goal-based solutions. Obviously, this needs to be done intelligently, as in any project.

Another sensible improvement is an effort to create platforms and to encourage software sharing and reuse. For example, “FixMyStreet” originated in the UK, but is available to use anywhere.  (I would like to have this in my local town(s)!) (An earlier incarnation of this idea from downunder was called “It’s Buggered, Mate”.)

Of course, data interoperation is, and will continue to be, a challenge. We have developed significant tools for sharing data in portable ways (e.g., see Linked Data out of the W3C) << link>>. It takes work, but we do actually know good ways to do it, if you have the will to do so.

In the end, the “open data” movement is facing serious political headwinds. Commercial companies make zillions off proprietary data collections which are anything but open. These folks are bringing you the Internet of Too Many Things, which will produce yet more proprietary.

At the same time, governments are clamping down on “leaks”, and classifying all kinds of data for “security” reasons. Worse, there is a huge struggle over videos of police and other public servants. We have all seen the media storms and political fall out from police dash cams and civilian phone videos of violent incidents. Less splashy have been the related analyses of public records of traffic stops, arrests, and shootings.

Simple good government things like “FixMyStreet” are going to face resistance from both the public (do you want your neighbor “reporting” you to the city on a public web site?) and governments (who do not welcome detailed scrutiny or the possibility of selectively manipulated data analyses).

This has never been a purely or even a predominantly technical problem, as Lee et al are well aware. Can we make this work? I thin it will take a lot of trust, which is not an easy thing to create. If open data can help build trust among ourselves, then that will be a huge win, no matter what else the apps might do.


By the way, the same issue of CACM there is a nice article by Neville-Neil about time and clocks in distributed systems. This is a really good summary of a ubiquitous, difficult, and nearly invisible problem in computer systems.

Neville-Neil reminds us about Leslie Lamports classic 1978 article “Time, clocks, and the ordering of events in a distributed system, which should be required reading for all programmers.

By the way, these issues of clocks and timestamps are at the heart and foundation of cryptocurrency technology.  The whole consensus “mining” process is basically a way to agree on the order of events without relying on even a single clock.


  1. Lamport, Leslie, Time, clocks, and the ordering of events in a distributed system. Commun. ACM, 21 (7):558-565, 1978.
  2. Lee, Melissa, Esteve Almirall, and Jonathan Wareham, Open data and civic apps: first-generation failures, second-generation improvements. Commun. ACM, 59 (1):82-89, 2015.
  3. Neville-Neil, George V., Time is an illusion lunchtime doubly so. Commun. ACM, 59 (1):50-55, 2015.


One thought on “Article on Open Data For Civic Apps”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s