How Colgate leverages AI and Big Data

How Colgate leverages AI and Big Data

What are large CPG brands doing to differentiate their consumer intelligence so they can innovate more effectively?

Consumers buying and consumption habits are continuously changing and becoming more fragmented. The problem is, CPGs are no longer equipped with the right tools to analyse, understand and prioritise which changes in consumer behaviour they should act upon. Traditional research methods of asking people direct questions are flawed, time consuming and costly.

So what can CPGs do about this?

Taylor Anderson, Global Director of Insights Capabilities at Colgate-Palmolive and Michael Grogan, Client Strategy Director at Black Swan Data talk about the exciting work Colgate has been undertaking through the use of Social Prediction and leveraging AI and Big Data to drive new trend prediction capabilities.

Framing the challenge: trend identification plus prediction

Colgate have always excelled in identifying the right trends. However, identifying is merely one aspect of successful innovation. In 2019, we estimate the top 10 U.S. CPGs alone waste $2.3bn of capital on launching new product innovations either too early, when the market wasn’t ready, or too late, after the trend had peaked and competitors had stolen the advantage.

Picking up those early signals and predicating the right time to truly capitalise on that trend is integral. Colgate’s focus has been to ensure that they not only identify the correct trends for their category, but also predict the correct time to innovate and launch that new product.

“It’s really incumbent on us to understand when the wave reaches that peak moment, to catch it and to be able to be carried forward by it”


Anderson explains that Colgate’s product development processes weren’t quick or agile enough to keep up with the momentum and changing nature of consumer trends. This lack of speed has been amplified with the acceleration that social media and the digital realm is creating. Misalignment between the speed of the innovation processes and the salient growth in the digital world meant that conversations, consumer needs, desires and trends, could be missed.

The opportunity: going beyond identification to measure growth

Colgate wanted to get to a place where they could actually understand when the right point in time would be to catch that wave.

“For us, recognising our challenges, and the opportunity for partnering with Black Swan, [enabled us] to go beyond identification and also be able to measure growth”


Systematically keeping a pulse on social data was fundamental to the process. Through partnering with Black Swan Data, Colgate was able to become more consistent and purposeful in their insights and market research approach.

Combining Black Swan Data’s advancements in predicative analytics, alongside Colgate’s operational agility, meant that the new product development process enabled true consumer centricity through a digitised insights ecosystem.

Strategic Case Study #1

Understanding the microbiome

Over the last 18 months Colgate has worked with Black Swan Data across their innovation cycle to identify opportunities, optimise concepts and improve outcomes whilst empowering teams with self-serve foresight tools to create faster, more agile innovation processes.

New methodologies enabled Colgate to take a closer look specifically into the microbiome trend. To understand, quantify and prioritise the drivers that fuel this trend and, power any future category growth.

“We not only had a deeper appreciation for what was driving this topic, but who was behind those conversations, for whom this was relevant, and where and how to engage those folk”


These new insights enabled Colgate to apply fresh direction and accurate strategic positioning to the category. Helpful to multiple teams, Colgate applied their findings to brand restaging, communications development and to create long-term strategic drivers for new innovation opportunities and product concepts.

By using a ‘bottom-up’ approach, Colgate removed any disconnect between strategy and application through the understanding of not only the what but also the why.

Strategic Case Study #2

Brazil Haircare

Unearthing an extensive amount of rich content and social conversation from the Brazil Haircare dataset meant the local Palmolive team became exposed to many more opportunities within this category.

Had the organisation approached this market using their usual traditional insights methodology, they would have only discovered the ‘known knowns’, from the place of functional ‘jobs to be done’.

The drivers that emerged from the conversations surfaced that there was a lot of emotional sentiment attached to this market; a cultural nuance from other markets but something that really powered Palmolive’s work.

The Brazil Haircare dataset emphasised that this market struggled with the presented ideal of beauty versus reality; these presented ideals did not reflect or align with consumer’s real lives or needs.

Both Palmolive, and competitors, hadn’t previously been leveraging this emotional space that the dataset unearthed. Palmolive created a new lifestyle platform for haircare innovation very much tied to the ideas of individuality as well as authenticity. Celebrating natural hair in that market, or population, was a new message that resonated exceptionally well.

Integration into the organisation

For any new tool, capability or mindset shift it is important to think ‘bottom-up’ as well as ‘top down’. New insight processes, technology adoptions and partners can only be successful when the whole organisation has buy-in.

For successful adoption, Colgate integrated the work with Black Swan to their own internal efforts to evolve their marketing and insights culture. The organisation ensured that they continued to trust, and rely, on the quantitative data when directing decision and strategy, safeguarding the habit to settle back into the realm of ‘known knowns’.

Finding opportunities to build the work with Black Swan Data into already established processes was fundamental. To succeed, senior leaders would ask the right questions, challenge the status quo and guide their teams through these new ways of working.

A little bit about the data set…

US Oral Care Dataset contains over 4,000 live trends derived from 2.5 million naturally occurring consumer conversations whilst the Brazilian Haircare dataset contained 14 million conversations with a similar number of trends. All conversations were ingested from Twitter, consumer forums, review sites, news publications and blogs over the last two years. Every trend is organised into Black Swan Data’s proprietary Innovation Lens framework. Innovation Lenses allow us to identify and analyse all the trends within a particular area – for example Ingredients, Themes, Benefits or Brands – to understand how they compare and interrelate. We can then analyse them using techniques like network clustering to understand how similar trends ladder-up into macro growth drivers. This unique, consumer defined view of US Oral Care, combined with TPV, enables clients to find micropockets of growth and emerging trends, earlier and more accurately than before.

How Social Prediction can help your business grow

We combine our always-on trend prediction software solution, Trendscope, with Horizon, our programme of consulting that enables businesses to map the future category growth drivers and determine where, how and when to innovate effectively. We work with clients including Colgate, PepsiCo, Danone and McDonald’s to put Social Prediction at the heart of their expanding foresight capability. More specifically, Social Prediction supports: Category strategy We create consumer-defined category landscape segmentation studies built bottom-up’ from every consumer topic that is being talked about by consumers online. This needs-based segmentation is dynamic and can be monitored and tracked over time as consumer tastes and trends evolve. It helps teams connect where to play with how to win and deliver truly consumer centric marketing and innovation strategies.


Our continuous trend identification and prediction tools mean your innovation teams will never miss a trend again. We use this data to build future focussed innovation strategies help brands identify the breakthrough product concepts, and more crucially when is the tipping point to launch.


Our clients have produced Brand Positioning and ATL copy based solely on Social Prediction data that understands how consumers organically talk about the wider category or need-state in their own words. To find out more about how we apply Social Prediction to drive business growth and deliver stronger innovation pipelines please contact our team today.