OpenData – APIs vs data dumps, it’s economics innit?

  I spotted a tweet from Owen Boswarva a few days ago

“We need to move away from government’s reliance on bulk data sharing and create an economy of APIs.” 100% wrong. #opendata #ODIsummit — Owen Boswarva (@owenboswarva) November 3, 2015

BTW, it wasn’t Owen suggesting we need to move to an economy of APIs, that quote was from Matt Hancock’s address to the Open Data Instate Summit. I replied

It isn’t govt’s job to serve #opendata through APIs. Just release raw data & let market work out what to do with it https://t.co/qlQt04Tn5a — Steven Feldman (@StevenFeldman) November 5, 2015

There was quite a lot of support for my comment but a few people pointed out that real time data needed to be served as feeds not downloads.   Does it matter whether we just dump piles of raw open data out there (wherever ‘there’ is)? Or do we need to serve data through APIs or live feeds creating the much vaunted National Information Infrastructure? For me, this is at the heart of the OpenData economic discussions that are going on in various fora including at the ODI.

Thinking back to the origins of OpenData in the UK (that’s nearly 10 years ago for younger readers, good summary here) there were some basic principles that started with the Free Our Data campaign and underpinned the subsequent adoption of OpenData by the Brown and Cameron governments.

My quick summary is that government departments collect and maintain vast amounts of non-personal data in the course of fulfilling their public tasks and obligations, this data has been collected at the taxpayers’ expense and should be freely available to citizens and businesses to use as they see fit. The benefits were meant to be:

• Transparency – citizens should be able to view the data on which government makes decisions, measures performance or allocates resources.
• Accountability – citizens, journalists and campaigners should be able to hold government to account by using this data
• Innovation – business (particularly a new generation of data smart SME’s) would develop new products and services on the back of OpenData that would generate substantial economic benefits (at one time the figure of £6Bn was mentioned)

I did say ‘my quick summary’

Back in 2010 I wrote:

A snapshot view from a wholly converted sceptic, perhaps even a personal manifesto.

• We have learnt that we cannot always trust our governments, Open Data is the foundation of transparency, transparency will drive accountability, holding our government accountable is the cornerstone of participative democracy.
• There are few facts and many shades of opinions, Open Data allows many analyses  and interpretations, some may be malicious, most won’t be. It isn’t very different to newspapers, we will learn which ones to trust.
• If we really trust the innovation of the crowd opening up data need not be colossally expensive, quick and dirty (but machine readable) is all we need to get started. Most people who talk about the costs of publishing data are probably looking for excuses not to.

In a few years we will look back on Gordon’s Day and the opening of data and wonder what all the fuss was about. It would be nice if by then we have degeeked the whole thing but maybe that’s a bit much to hope for.

How time flies when you are having OpenData fun and what a pleasure it is to look back at something that you have written and discover that you still agree with it, or most of it. I remain convinced about the first two bullets, it’s the third that I am wondering about. It doesn’t seem to me that the we have managed to keep it quick and dirty, government departments are spending significant amounts on making data open through data platforms, services and processes – have a look at the last NAO report on ‘Implementing Transparency’ from 2012 or my post from just after it’s release. Another 3 years on, not a lot has changed, the public sector is very slowly adapting to the principles of OpenData but there is still little evidence of the innovation that will fuel the new OpenData economy. The ODI will cite their recent report that each £ invested in its OpenData challenges will generate £5-10 over the next 3 years, an investment of £1.2m is predicted to create between 75 and 141 jobs. Pretty small beer in my opinion, but presumably this is pump priming. I would like to see some research into private sector investments in open data business and products that are starting to deliver economic benefit and UK jobs.

So back to dumps, APIs and feeds. The evidence for the economic benefits of OpenData is limited, maybe it will come through in time but until it does surely government should be cautious about the level of investment that it makes in OpenData platforms, APIs and Information Infrastructure? I am on the side of the data dumpers, let’s get back to first principles, put the data out there and see what people do with it before we spend further money on APIs and stuff.