Data. First it was Big, then it was Open.

We are producing data at an astonishing rate. As more and more of our tools and toys become ‘smart’, this trend seems set to continue. Even the humble traffic light is now becoming incorporated into systems that make up our new ‘smart cities’.  In virtually every corner of society there is someone either producing or collecting data. ‘Big’ or ‘connected’ data is undoubtedly valuable, providing benefits to business, government ,and for us as stake-holders in our new connected society. What happens to this vast repository of information? It’s an important question that raises issues not only relating to business, but questions how we as a society should benefit from these new insights. In the information age, connected data has become the life blood of our society. For a healthy body blood must be freely available to all the organs that require it. Similarly, in a healthy society data must be freely and easily available to all those that might benefit from it. Any unnecessary barriers in the flow of data act like a blood clot, causing problems and inefficiencies which hurt the whole body. The Open Data Institute (ODI) represents a movement within the field of the web and big data that is working towards making data freely available and crucially useable. The ODI is primarily concerned with the use of government data, but sees itself as an advocate for the advancement of the use of open data within our economy. Historically governments and big corporations have obsessively collected data, creating huge data silos, filled with potential but often without the strategies or skills to utilise these resources. The Open Data movement argues that the value of government data is in its re-use and distribution within the economy. Rather than relying upon bureaucratic systems to make use of this goldmine of data, the ODI and the Open Data movement argue that this resource should feed entrepreneurship within our economy. At Black Swan Data, we have seen time and time again how open data can provide solutions to all sorts of problems: the weather, A&E admissions, geophysical data, mobile network issues, smart meters and the environment. Without easily accessible open data many of these solutions would not have been possible. The idea at the heart of Open Data is that government should publish rather than manage its data, with an eye on improving productivity, openness and stimulating growth. The open spirit of this movement echoes the founding principles of Tim Berners-Lee’s web; free from walled gardens and given as a gift to the world rather than licenced, packaged or sold. In fact, Tim Berners-Lee and colleagues such as Sir Nigel Shadbolt are in the driving seat of the ODI and remain vocal advocates of the open movement. Open Data represents an acknowledgment that there is a symbiotic relationship between business, citizens and government in ensuring that data is best used to serve our society as well as our economy. How data is made open is of course crucial in ensuring that it is of use and not merely adding to the chaos and noise of clumsy data on the web. Tim Berners-Lee has helpfully produced guidelines that suggest how best data should be produced in order to ensure that it is useful. Machine-readable linked data is potentially a stepping stone to a much smarter and sophisticated future web. Public health data, especially anonymised patient data, is an example of how tricky this can be. If freely available, it could provide untold benefits to society. I can scarcely imagine the patterns and signals which are currently sitting in those data sets. For all we know some insight which could lead to a cure for Alzheimer’s is just waiting to be discovered, if only the data were available. Unfortunately however, the technical and ethical challenges to making such a data set available in a useful format are huge. How can the fragmented NHS data silos be combined? And how can the data be anonymised so that it retains its usefulness? These are not easy challenges to solve, but the potential benefit to society makes them too important to ignore. So as users and producers of data we should watch the Open Data movement with great interest. Publishing data that already exists will provide great opportunities for those companies with the capacity to do something with this data. But the real story may lie in the untapped potential benefits to society, whether they be as simple as a live bus timetable app or as revolutionary as life saving health insights. We at Black Swan Data look forward to the day when the Open Data movement is as ubiquitous and powerful as open source is today.