How Data Science and Behavioural Economics are changing the research world

Stop asking questions and start getting answers

The end of question-based market research

We are living through a revolution in market research. Question-based methods – surveys, interviews, focus groups – are dying out, and will be replaced by observational methods – neuroscience, behavioural economics and data science. I’m lucky to have been in at the beginning of this transformation in my role as Insight Director at PepsiCo. Working for such an iconic brand, which places a high value on consumer insights, I had the opportunity, and the resources, to explore any method of research that might help us make better decisions. This freedom to experiment led me to conclude that there are fundamental flaws with traditional question-based research methods that mean their use is unsustainable in the longer term.



The goal of all research is to give decision makers a glimpse into the future: Should I launch concept A or concept B? Which area of my business should I work on to improve customer satisfaction? Which product features will customers expect my brand to offer? Traditionally this window into the future has been achieved by asking consumers questions about their behaviour, and their underlying needs and feelings, either by asking them to predict what they will do in the future, or to report on what they have done in the past. But there are problems with this approach. Firstly, people don’t always do what they say they will, or report correctly what they have done. Researchers are well aware of this ‘say-do gap’, although it is more widely reported in some categories than in others. For example, the proportion of people who present as ethical consumers in surveys has long been much higher than the proportion who actually buy Fairtrade, recycled or other sustainable goods. Likewise, the influence of Covid is overstating people’s claimed behaviour versus what they are actually doing. We can’t help but take-in the media noise swirling around us and then play that back to researchers as our own behaviour when answering direct response questions. Whilst we can do something to address these over- or under-claims by calibrating survey results; we still can’t know how accurate our survey findings will be and how much we can trust them to guide our decisions. Secondly, and less widely discussed, is the fact that question-based methods of research are rarely replicable. In recent years, the scientific research community has faced up to a crisis of replication and reproducibility, but the issue is still a dirty secret in market research for the simple reason that commercial research projects are rarely repeated and compared. Market research is time consuming and costly enough to do once, let alone twice. However, I have been in a position to conduct repeat research and to observe that the same questionnaire, with the same sample frame, conducted in the same way, by the same agency, will yield different results, and point to different decisions – ‘go with concept A’ versus ‘go with concept B’. This is true of traditional quantitative and qualitative research alike. Despite investment in market research, more than 80% of new FMCG products still fail. We need better decision support. I believe it will come from observational methods, not question-based methods, and many client side Insights leaders like Stephan Gans from PepsiCo are coming to the same conclusions.  

Why question-based research methods fail

There are three reasons that question-based research methods fail:

1. We don’t know what we don’t know: The growing discipline of behavioural economics, and books such asThinking Fast and Slow, Nudge, and Predictably Irrational offer insights that directly address the say-do gap. There are many situations in which unconscious cognitive biases, social norms and heuristics affect our decision making and cause us to behave in seemingly irrational ways. When we ask people about their behaviour, in many cases they don’t really know, and they come up with answers which seem rational based on the information they do have. For example, if we ask someone why they buy a particular cereal they may tell us because it is cheaper, or healthier, but the real reason may just be that they buy it because they have always bought it. Another complication is that the people who write questionnaires and discussion guides are humans too, so the questions that are asked, and the pre-coded lists of answers that are available, can never be free of the researcher’s own assumptions and biases.

 

2. Context is everything: This is really the core of behavioural economics. Context fundamentally shapes all our experiences and, therefore, our decisions. Researcher Brian Wansink conducted a renowned experiment in a local fast food restaurant, which neatly demonstrates the impact of context. Diners ordered their lunch and then were randomly shown to a table either in the existing restaurant, which was brightly lit and had loud music, or in a specially converted room which had dimmed lighting, and was playing mellow jazz. Not only did the busy lunchtime eaters in the room with a more chilled ambiance take longer to eat, they also ate less, and rated the food as better. Although context determines all of human decision making, most market research is not conducted in context. Asking someone to think about which cereal they will buy when they are filling out an online survey is not the same thing as them making a choice as they shop, with a toddler in the trolley seat and a whole shelf to choose from. We know that if you ask the same person the same question but on different days, or in a different mode (online versus telephone versus face-to-face) you will get different answers.

 

3. Sample is unrepresentative: Sample size is an issue – the bigger the sample, the higher the cost, but smaller samples yield more variable results. A further issue is that if you conduct multiple surveys, even using the same sample frame, there will be underlying differences in sample characteristics that lead to different results. Also, what is defined as nationally representative often isn’t. How many people do you know who do online surveys? There are always problems in representing higher income, more educated and older consumers. These problems can be dealt with to an extent by weighting, but this also introduces variability, especially when hard-to-find respondents are weighted up by large factors and the effective base is reduced – information that agencies don’t always reveal and research buyers don’t always know to ask or understand how to evaluate.  The issue is compounded by the prevalence of professional survey takers, and bots, which providers can address to an extent with quality control, but can’t entirely neutralise. The failure of the polls to anticipate the Brexit vote is a case study in the complexities of sample, survey mode and weighting, and just how difficult it is to use surveys to predict voting behaviour; this YouGov blog highlights some of the difficulties that researchers face.

The rise of observational research methods

Over the last two decades there have been many new observational methods developed to try and overcome these issues. I’ve broken them down into those that use neuroscience, behavioural economics and data science.

Neuroscientific research methods

Some neuroscientific methods such as EEG (electroencephalogram) or fMRI measure the brain directly. Others measure physiological proxies for brain activity, such as how our eyes respond, or how our facial expression changes. All of these are becoming more widely used, with the goal of overcoming the say-do gap by observing what is happening in the brain. The  GRIT report shows that 35% of insights buyers and providers are already using eye tracking, 29%, applied neuroscience and 20%, facial analysis. These are high estimates; as discussed, sample composition matters, and GRIT respondents tend to be highly engaged with new methodologies. Based on trends and compared to other emerging methods, the report calls these ‘established niche’ approaches.  This reflects the fact that these methods also struggle with scale, time and cost, which keeps sample sizes small, and the impact for marketers – at least at the moment – negligible. This certainly reflects my experience at PepsiCo, where I was lucky enough to try many of these new techniques, including EEG FMRI testing, Pupil Dilation, Galvanic Skin Response, and Facial Coding. One project involved videoing people while they ate a snack product, and then measuring the brainwaves of other people while they watched the video, to understand what cues would trigger the ‘mirror neuron’ effect – also known as that mouth-watering craving you get when seeing someone else eat an ice cream cone. We got some interesting results that helped us finesse the details of some of our comms but, in general, these techniques gave us little more than that: interesting results, and a shiny new tool to help grab the attention of senior stakeholders. For many of the techniques we conducted parallel testing and, even though the method was very different, nine times out of ten, the technique added no value; the business decision was the same in terms of choosing pack design A or concept B, and there were still issues of credibility around sample sizes and repeatability.  

Behavioural economics research methods

The research world has started to embrace the principles of behavioural economics as a way of dealing with unconscious biases – what we don’t know we don’t know. However, in my view, it is still playing at the margins. Typical approaches include restricting the time respondents get to answer questions, in the hope of accessing System 1 automatic responses, or establishing if respondents are in a hot (such as hungry or excited) or cold state before asking traditional purchase intent questions.  These are all valid improvements, but they miss the mark because the survey is still violating one of the core principles of behavioural economics: to understand behaviour in context the respondent mustn’t know what you are trying to study. For a fully behavioural economics approach to, for example, testing consumer response to a new cereal packaging design, the ideal would be to invite a respondent to shop in a (virtual or real) supermarket, in which the product has been placed on a shelf among existing cereals. Then you just let them shop the shelves and see what happens. This is completely different to traditional research, which starts off by sensitising the respondent to what you are trying to study by showing the respondent a concept, and then asking direct questions about it. The challenge in switching to a purely behavioural way of testing is cost. Doing this at scale, with multiple design options, is prohibitively expensive and time consuming; if you don’t reveal the objective, you will need to put a high volume of respondents through the test to be sure of getting sufficient interaction with your concept. Ways of productizing behavioural approaches will no doubt be developed but, at present, this approach is not economically viable.

 

Data science research methods

With neuroscience approaches adding little value, and behavioural economics not applicable at scale, we can turn to data science as the most effective method of uncovering patterns in human behaviour, without asking questions, by ‘observing’ and mining existing sources of data. Ten years ago there weren’t sufficient numbers of consumers talking online about crisps, or hairspray or multivitamins to drive the approach we use today at Black Swan Data. But developments in data science have come at the same time as a critical mass of online conversations. This means that instead of asking questions of a limited number of consumers, we can analyse the conversations of millions, about a wide range of consumer product categories, on platforms such as Twitter, Reddit and WordPress. It’s like the world’s largest focus group. We focus on a branch of machine learning called NLP (natural language processing), and our approach addresses the issues discussed earlier. When consumers post content online, they aren’t responding to a question or a research stimulus, but acting of their own volition, talking about the things that are relevant and interesting to them. They are posting in their natural context, and not in a contrived market research setting. There are no issues with sample size (our sample is the internet) and no issues with replicability, as we are mining conversations that have already happened. By tuning into what is happening online, we can bridge the say-do gap, and pick up trends before they emerge. It also allows us to do things that no traditional research technique can accurately do – and that is predict. These new techniques enables us to answer questions such as:  

• How do consumers actually talk about my category?
• What’s most important to consumers now? And in the future?
• What are the macro category drivers and headwinds?
• What micro trends are driving the macro trends? How should I translate a macro trend into something actionable for my yoghurt brand?
• What are the biggest opportunities for my portfolio and brand(s)? Which should I prioritise and how should I execute them?

The challenge is mining and processing this data effectively to retain the context. I’ll discuss the details of this complex process in future posts. In the meantime, if you'd like to learn more about how Black Swan uses data science and machine learning to improve decision making contact our team today or check out out my blog on our new COVID Classifier. Dave Soderberg, Chief Data Officer