If you’ve been reading the MR press over the last couple of years, you could be forgiven for thinking that some big bad monsters are coming to eat us all up, and spit us out in forlorn terabytes. But is big data really the nemesis of traditional market research? And will social media analysis really replace direct engagement?

At first glance, big data and social media bear some of the hallmarks of what business theorists term ‘disruptive innovation’; a technological innovation that looks for all the world like a game-changer, whose effect will be to displace incumbents in the industry. But a closer look at both the innovations themselves, and the theory of disruptive innovation, suggests that BD and SM, while clearly disruptive in many respects, are not about to kill off traditional market research.

In 1997 Clayton Christensen wrote a book called the Innovator’s Dilemma, in which he suggested that technological innovation fell broadly into two classes: they were either ‘sustaining’, meaning they enhanced the products and services of incumbents in a market; or they were ‘disruptive’, meaning they posed a threat to first the market position of those incumbents, and second (in some specific circumstances) a threat to their survival as firms.

Within a year of the book’s publication, the term ‘disruptive innovation’ (let’s call it DI from now on) was being used in all sorts of contexts; not always, it has to be said, with any real accuracy. The subtleties of Christensen’s analysis were lost in the rush to turn DI into a scary shibboleth. This panic usage was not confined to the shallow end of the market; some of the academic literature is guilty of using the term without taking on board all that Christensen wrote about it.

First, let’s take a brief look at how the theory evolved. The roots of much of today’s theorising can be found in the work of Joseph Schumpeter (1883 – 1950). In 1942 he published Capitalism, Socialism and Democracy, an analysis of Marxist economic thought. In that book, he coined the term ‘creative destruction’, partly in an attempt to understand the high attrition rate in business. Schumpeter’s analysis is in many ways a warning to capitalism; he says that the constant turnover of market participants indicates a vulnerability in the system itself, one which will eventually lead to its demise.

But some economists chose to deliberately misread him, and instead promote a Darwinian model of the market. High turnover was equivalent to natural selection; not a threat to the system, but a regulator, designed to weed out those who were not fit enough for the environment. This altered the focus of much business theory, and meant that Schumpeter’s basically entropic model was conveniently forgotten, replaced by its Darwinian counterpart. Now, the question of survival took precedence.

Enter Christensen (on the back of a raft of theorising about why and how firms survive innovations that threaten their business); and DI, once freed from the shackles of its actual intellectual underpinning, began to roam the new environment like a super-predator. It’s not the first time a good idea has been transformed into a simplistic behemoth; and this particular monster served the aggressive new business paradigm very nicely.

So do big data and social media constitute a T Rex- sized threat to MR? I would argue no, but to understand why they are, while clearly disruptive, not life-threatening, it is necessary to understand precisely where and how the bullet of DI strikes its target. Christensen locates the direct impact of DI in a firm’s ‘value networks’ – the complex web of relationships between providers and customers – and from this, derives an explanation of how incumbent firms concentrate too much of their attention on satisfying existing customers, and too little on engaging new market entrants.

I suspect that Christensen’s proposition, while useful, still doesn’t show us the bull’s eye. We need to go back to Schumpeter, and his entropic model, to see it. Even Schumpeter doesn’t say it out loud (but then he was before his time; if he had written thirty years later, with the benefit of such theories as commodity aesthetics, his work would have resonated better). It stems from a simple observation about trade, and the impetus for trade.

Societal need (or societal desire, if you prefer) drives trade, and regulates the entropic process. If the requirement for a product or service disappears, the trade built out of that requirement disappears along with it. Stark, but simple; the customer really is king. So if you want to know if an innovation is going to kill you, you have to work out if it is about to make the need for your market disappear. If it is, head for the lifeboats; if it isn’t, make plans for the inevitable disruption, but don’t write your will just yet.

Market research occupies a peculiar niche in this schema. Its business is identifying the very requirements that business depends on for its existence. So in theory it is very well placed to recognise threats, and it should be well equipped to discern whether an innovation is sustaining or disruptive.

Both big data and social media are, at root, about numbers. It is arguable whether they constitute innovations at all; they could just as easily be characterised as economies of scale. The fact that there is more data out there should not be a threat to the people who know most about that data, and what it means. Part of the panic (in addition to the confusion sown by bandwagon commentators using a half-understood theory) stems from the fact that the technology is still opaque; see through it, and it doesn’t look quite so deadly.

MR professionals concerned by the noisy doom-mongers would be well advised to go and read Christensen, and understand him; they would be better advised to go to Schumpeter, and understand him. In an uncertain economic climate, the need for market research increases; and if more tools are available for this task, that can only be good for MR. As for the big bad monsters, they begin to look more like docile herbivores than rampaging carnivores. MR’s only real concern is to make sure it doesn’t eat itself.