This may sound odd coming from me, as I am the Chief Data Officer for Black Swan Data, and it may be stating the obvious for others, but I’ve always found that even things that are obvious are easy to forget and are worth repeating — this job is not about data. I’m sure you have heard this famous quote before:
“People don’t want quarter-inch drill bits. They want quarter-inch holes.”
Theodore Levitt 
The same thing is true for the field I am in. My job, as well as all the jobs in the fields of data science, data governance, data analytics, data engineering, market research, and insights, none of them are actually about data. Data, algorithms, software, databases these are the equivalents of a quarter-inch drill bit. What people want — the goal of the job, the reason for its’ existence in the first place, is better decisions. The objective for anyone in these fields is to help individual people and ultimately the whole organization to make better decisions. And yet, particularly if you are earlier in your career and reading articles online it can seem like improving decision making is the last thing one should be focusing on. Instead it seems like it’s all about technology, what programming languages you’re using, what are the latest models, what model libraries, what are the data sources, what DB technology to use, whether to use streaming technology or not, etc. It’s certainly important to learn the techniques and the tools to get the job done, and when you first start out that of course is going to be one of the first steps you take. But whenever you start actually working in the field — either at your first job or in a new role at your current company, I always encourage everyone to go and find the human that’s ultimately going to make a decision using the data that you’re providing.
Before you write one line of code, or one row in Excel — find the human who needs to make a decision with your output. Make sure you really understand what they are trying to achieve, the context they are in, and the decisions they need to make. Even if you’re feeding into an automated system, at some point somewhere, there’s a human who’s looking at those decisions that the automated system is making, and making the decision themselves as to whether to continue to invest in that system or not. As you get more senior, the process is the same — from focusing on just one human and their decisions, you start focusing are larger groups of people, bigger systems, until you get to the point where you are trying to help the entire organization to make better decisions. Each part of the data ecosystem helps to make those decisions better:
- Data Governance helps to make sure that there’s clarity in the way that the data is defined, which helps both speed and quality of decisions.
- Data Engineering helps make sure the data is transformed, cleaned, and available where it needs to be so the data scientists and data analysts can spend less time finding and cleaning.
- Data Analysts quickly scan the horizon and help to map out where challenges and opportunities might be, connecting with those human decision makers through stories to communicate and engage.
- Finally, Data Scientists can take advantage of the work the Data Analysts have done to ensure the most specialised resource is used in the most effective way — building models using the power of statistics and machine learning to overcome our human biases’ and predict into the future.
For further reading I highly recommend
a HBR article. As a start up, Black Swan Data doesn't have lots and lots of internal employees using lots of complicated systems for data. However, our main product is a data product that we serve out through our SaaS platform
Trendscope, and so most of my time is spent helping our end clients to make better decisions using our data. My job and my team’s job is to make sure we understand our clients and the decisions they’re trying to make. We can then ensure our data and tools help them make those decisions better. Luckily, I was a client for nine years at PepsiCo where I was in the Global Insights team. This gives me a good base level intuition about what decisions our clients might be faced with and the potential context around those decisions. However, PepsiCo is not the same as all companies. Every company has its own idiosyncrasies, different ways of doing things, different problems to solve. And so it is always important to make sure you’re spending time with those end clients — the humans making decisions — understanding the way they’re making decisions, understanding how the data and the tools and the software and the models that you provide, help them or hurt them make those decisions. At the end of the day, it’s not about you and your ability to write code, or use the latest transformer based deep learning model, and it’s not even about the data itself. It’s about the human making a decision and helping them and their organizations make better decisions each time. To find out more about how we use data and apply Social Prediction to drive business growth and deliver stronger innovation pipelines please
contact our team today.
