I first moved to London back in 1998 in the middle of the first dot com boom as a developer and to say the streets were paved with gold was a bit of an understatement.
After spending a weekend absorbing “Sam’s teach yourself HTML in 24 hours”, by Monday I was working in a highly charged start up for an hourly rate I’ve only just surpassed as CEO 16 years later. As we know, in 2001 that was all in tatters (and I was back as a junior developer on £18k a year).
What’s this got to do with big data? Well, I’m a big believer that great ideas never fully work the first time around, but surpass on the second. Of course the Internet was a good idea, but we just got a little over excited, then disillusioned, before finally we changed direction and then…. whoa… Who can live without the Internet now?
I call this the ‘pendulum of rightness’.
So, some of us are disillusioned with big data and all the hype, for a time even me.
However, at Black Swan Data we’re now hugely excited and are confident we are back on an upward curve and it came by keeping it simple and just changing our perspective slightly and trying again.
Enter
connected data.
When we first started Black Swan Data, we got very (very) excited when we were able to crunch over 10 million tweets a day. We ran algorithms over them to show amazingly pretty charts. Interesting, but only if you like pretty charts.
The difference came when we started connecting other data to it. Pretty soon we understood it was aggregated datasets connected together rather than the big data sets themselves that were the powerful thought changers.
The beauty of a connected data approach is that you don’t need to spend millions getting it all working to see if it can make a change in your organisation. If you’re not sure whether having a data strategy can help your company, you can find out for tens of thousands rather than millions, sometimes less.
Your first challenge is to get some of those sales figures into spreadsheets. Next you need to work out what other external data to map against it. As our data scientists always say, it’s not about the model it’s about the variables. This process should take no more than a few weeks and before you know it you have the start of a data strategy with proven results from the beginning.
It makes me want to cry when I hear the amount of money some companies have spent building a ‘big data’ or unified data strategy without even knowing which data sources they were trying to connect or why. It reminds me a little of 1998…
