Why Big Data Will Never Beat Human Intuition

By Tim Leberecht Tim Leberecht

June 20, 2013 at 8:06am

Big Data is big business. Sensors, GPS tracking, math modeling, and artificial intelligence offer companies real-time market insights at massive scale and open the door to unprecedented ways of monitoring, targeting, and measuring employees and customers. Analyst firm Gartner predicts that enterprises adopting Big Data technologies will “outperform competitors by 20 percent in every available financial metric.”

Big Data might well be “the new oil,” but I would caution us not to worship it as the new religion. I’m afraid that amidst all the data frenzy we are not only losing a more holistic view of business but also an important part of our humanity. How do we appreciate quality if we capture it only in quants? How much (or little) space do we leave for creativity and human expression if we equate better living with better algorithms?

I am not a dataphobe, but I am concerned about relying only on data. I am not against quantitative metrics, but I question their authority as the main indicators of business performance, prosperous societies, and meaningful lives.

Big Data comes with many benefits, but let’s complement it with Big Intuition. Here are six reasons why:

Big Data = Big Brother? The New York Times’ Steve Lohr describes Big Data as a descendant of Taylor’s “scientific management.” Instead of performance in the workplace, which was the focus of Taylorism, we are now measuring happiness and well-being, our consumption preferences, social interactions, physical activities, our attitudes, moods, emotions, behaviors, and bodily functions—in other words, we are measuring our lives.
Sure, to some degree, “quantified self” apps may empower the individual to exert more control over their lives. However, by doing so, we are expanding the dictate of business to once-private terrain, commodifying and colonializing it under the mandate of self-improvement. For many of us, the refusal to measure and quantify these few remaining “sacred spaces” of our lives is the last bastion against the pervasiveness of the commercial.
Big Data is not social. We humans are social animals. Research shows that relationships, especially friendship and marriage, are key factors of happiness and fulfillment. Our brains are wired to care, and our hearts and minds have developed an astounding capacity to empathize and sympathize with fellow humans. We can show compassion, sense mood swings, detect subtle non-verbal cues, tolerate or embrace, accept and reject, love and hurt, experience with all of our senses, act irrationally, and even lose our self-control. These key traits of our humanity are threatened by the “mathematization of subjectivity,” as Leon Wieseltier calls it.
Recent social genomics studies suggest that not only our productivity, but also our evolutionary capacity to connect with others is diminished by digital overload.
Big Data creates small worlds. Morality is gained by way of empathy. Yet, in our age of hyper-connectivity we are increasingly facing the challenge of connecting with the other whose opinions, values, beliefs, faith, and culture may be unlike ours.
As digital technology has become more sophisticated in personalizing and customizing our social experiences, online and offline, based on our preferences, our “Likes,” and online paths, we are increasingly stuck in our own world—the “Filter Bubble,” as Eli Pariser called it, designed by smart algorithms to serve us with content, culture, and company that we are already familiar with and that fall squarely within our comfort zones. We don’t “like” the people and things that are unlike us, and thus feed a vicious cycle of social and cultural narrow-mindedness.
Big Data makes us smarter, not wiser. Our data-driven worlds are not only becoming smaller, they are becoming faster. The real-time flow of information persuades us to react to feedback constantly and instantly. Playing on the title of Alvin Toffler’s 1970 book Future Shock, Douglas Rushkoff calls our current state-of-mind Present Shock, lamenting “a diminishment of everything that isn’t happening right now—and the onslaught of everything that supposedly is.” Real-time feedback loops make it harder for us to step outside the tyranny of the now and see the future—to envision what could be.
Vision is what Big Data can’t deliver. Data might give us information fast, but for quick but profound decisions, intuition is much better suited. Prasad Kaipa and Navi Radjou, in a recent book, urge business leaders to move “from smart to wise.” They have a point. Smart organizations and leaders thrive on constant feedback. Smart is fast. Wise, however, is slow. Wise organizations and leaders need time and take it. Time for pause. Time for reflection. Time for not doing anything. Time to find the signal amid all the noise.
Big Data is (too) obvious. “You can only manage what you measure”—really? The financial crisis has shown that we manage poorly what we measure, and failed mergers, failed product launches, reputational crises, and social media PR disasters, in short, the cultural disconnects within organizations and between brands and their audiences, indicate that we need to get better at managing what we cannot measure.
Leaders now need to have “opposable minds,” as design thinker Roger Martin puts it. Data doesn’t parse ambiguity. The business leader of the 21^st century will no longer be measured by how much uncertainty he or she can eliminate but how much uncertainty he or she can tolerate.
Big Data doesn’t give (or forgive). Data might be able to predict new problems or find new solutions to existing problems, but only human intuition and ingenuity can come up with groundbreaking new ideas. That is a uniquely human gifts—one that creates a “meaning excess,” a sense of wonder and significance that goes beyond reciprocity, beyond merely fixing a problem or meeting a functional need.
By the same token, if we quantify all of our relationships, we will not leave any wiggle room for human discretion and our unique ability to forget, and even if we can’t forget, to forgive. Because we often have mixed feelings about people and their behavior, our judgment can be more than just binary. This means we can assess and respond to ambivalent behaviors with more nuance—we can appreciate the intention over the outcome, if we want, and we can choose to accept failure as a prerequisite of innovation. It is hard to see how we can collaborate with one another, how we can make progress towards any goal, without the ability to forgive.

So how can we become data-savvy, but not data-obsessed? As innovators, marketers, and business leaders, we must constantly defend and push for spaces for Big Intuition. Let’s resist the rush to data and take the time to lean back so we can be fast when it matters. Let’s grant ourselves a data moratorium from time to time that we can use to reflect on what really is important to us and our organizations. Let’s “hack” Big Data with small acts of friction. Let’s de-stigmatize “gut feel”—it is a better lodestar than we might think. And let’s use data to tell our stories, but let’s not allow data be our only story.

We must be giving and listening instead of filtering and targeting and analyzing. We must not reduce people to profiles and graphs, and aim for short-term victories on what Rushkoff terms the “algorithmic battleground,” whilst losing the hearts and souls of the people we serve in the long-term.

Data can give us the illusion of objective truth, yes, but at the end of the day, our employees and customers are not interested in the truth, they seek experiences that feel true. In other words, they seek authenticity. It is this small, but critical, gap between truth and authenticity that gives our brands, our organizations, and our lives meaning. No data can fill it. Only human beings.

You need to register in order to submit a comment.

Sort By: Latest Oldest Likes Replies

May 28, 2014 at 9:14pm

Ghanshyam Sharan Singh

Very interesting thought. I too had similar opinion about "Big Data Syndrome" getting developed in the Knowledge Community. But then I explained to myself. Human Intuition is unique but over the generations this has also evolved. Olden days it was more like a prophecy. The common mass not very knowledgeable used to accept it as a gospel truth. Today our intelligence analyses the intuition before accepting. To me "Big Data" enables us in improving individual intelligence rapidly so that intuition is sharpened. Both may not take each others place but certainly complement each other.

July 27, 2013 at 8:32am

Claudia Lindby

I agree: opportunities with Big Data are enormous - but that our real opportunity to create healthy and meaningful futures lie in human potential, not least the ability and will to dream, envision "new life" and to be bold enough to embrace wisdom. We must be able to combine Big Data and Big Intuition.Thanks for an inspiring post, and for great comments :)

July 8, 2013 at 1:27pm

Pete Ciekurs

Data tells you what "is" or what "was" but you still need intuition to get to what "will be". If you don't ask the right question or frame the problem correctly "big data" will lead you astray.

July 2, 2013 at 4:07pm

Reg Mix

Readers familiar with Kahneman and Twersky's work may disagree.
Impressionistic, fast decision-making based on very limited data (aka Intuition) has its uses, especially when the consequences of not making a decision under uncertainty is a high probability of disaster. But to say that normative analysis of large data sets will "never" beat out intuition -- well, that's just overstatement to get people to read the article. (Certainly got me to read it).

July 27, 2013 at 8:49am

Claudia Lindby

I loved "Thinking, Fast and Slow"! I admit I'm still pondering some of the ideas. In relation to decision making a key question will be, what kind of decision / about what. For me, intuition is closely linked to cell information and body knowledge and therefore cannot be replaced by Big Data for personal decisions that have to do with Integrity and Values (although we can use it to challenge our Values). Great input, Reg Mix - got me thinking twice :)

June 25, 2013 at 11:34am

Clark Dodsworth

I like your summary, Tim.
There are better counter-examples for 'manage what you can measure' than the financial collapse...ones that reflect poor use of real data instead of an illusory house of cards collapsing.

June 23, 2013 at 7:18pm

Howard Johnson

I agree that intuition is not the term I would choose. Like Harrison said; it's really about better data. We can collect data that was previously unavailable and it can sometimes be collected with much greater ease, but more is not necessarily that much better than good sampling practices. Secondly, data is not usable without an interpretive framework. That framework may be unstated, but it is surely there.

June 22, 2013 at 4:08am

Virginia IPeneva

Interesting discussion start point!

What about combining Big data and intuition?
Sometimes stepping out of the black/white point of view gives you better point of view and opens space for intuition.
As for me the Big data modelling, analysis and output is intuition based. It is question of being conscious what is what in the process

June 22, 2013 at 1:58am

Marcus Pietrzak

Great article, agree with your views. Big data is about correlations, that will never beat knowledge, experience and intuition.

However, it became so attractive, since it creates the illusion of getting into control and taking the right decisions if - and only if - we dig deeper and into more data than others.

Look at the recent hype about Big Data in China. What is causing this latest hype there?
- To a degree, it is a political system, that wants to take the right decisions, but has no tradition of asking those who are most affected. So they want to explore their people's behavior, and take the right decisions that will serve their people best, in their own values.

How different is this with profit driven companies? - No difference at all, I'd claim.
At the end of the day, it is about taking the right choices, without actively involving those affected, yet passively drawing information out of their regular behaviours, out of their digital traces.

Still, on the day after, we may wake up and realize, this is all about taking control by means of predicting human behaviour, not by involving the people.

Any freedom minded individual should get critical their, and invoke a lively discussion about ethics in economics and politics. Big Data by itself is not good or bad; the value for society will be determined by what we make of it.

June 21, 2013 at 10:26pm

Sunder M P

It is said that data needs to be chruned to make it information. Information needs to be pored over to make it insight and insight needs to be felt to make it wisdom.

Big Data helps in the data and information stages and just probably enters the domain of insight just that bit. But wisdon it is certainly not. One can crunch all the data that Microsoft has, but it will be deliver the insight that is needed to make an iPad.

June 21, 2013 at 5:49pm

Malcolm Ryder

The virtue of "big" is in having "more" in the same place at the same time under the same processing. I believe that the punchlines worth taking from these many considerations of Big Data are these:
- Data processing is one source of information. Don't make all *decisions* based on the same source
- Showing what we didn't know was "already there" is *not* the same thing as explaining what is "possible"...
- Knowledge is *not* just the results of something; knowledge is a practice and a cause of things as well. Decisions and knowledge are not the same thing.
- Things that have meaning to humans often exceed or lie outside of things that have meaning to mere given roles or processes.

I won't crowd the comments here with the things I've written about this, but I'm happy to share on request...

June 21, 2013 at 4:24pm

Marie Wallace

While l definitely agree that "bigdata" is not the holy grail, l don't believe its a case of data OR intuition. Big data, or more specifically analytics, is simply a tool that we can use to help us make decision. In the right hands it can be gold dust, in the wrong hands its a dangerous weapon. The key is to use it thoughtfully and not blindly.

June 21, 2013 at 3:47pm

Chris Willis

Big Data is not meant to "beat" human intuition. It's a supplementary tool that, when used correctly, helps us test hypotheses, speed intelligent decisions based on fact, and gain insight into the results of our actions for agile correction to meet the demands of a volatile global market. Data, in and of itself, is neither good or evil. Used appropriately, however, it provides the transparency and insight we humans need to work together strategically in large groups.

In Blink, Gladwell exposed the power - and flaws - of snap judgment based solely on intuition. We know that what we think of as "gut instinct" is an instant message sent by the amazing computer that is the human brain. In mere seconds, it has performed countless calculations incorporating true instinct (fight or flight response), tempered by past experience, and education and personal development.

Because we are all unique in our experience and learning, it is very true that the gut instinct of exactly the right person with a high mental acuity and specialized life experience presented with the right situation has the ability to act with a finesse and agility that no computer can achieve - assuming, that is, that this exemplary actor has been presented with a true picture of the problem.

And that brings us to the real value of Big Data. When filtered correctly and placed in the right context, Big Data has the ability to present us a picture of a situation that is outside of our personal experience, and larger than one brain can parse. It can bring us closer to that mythical "one version of the truth" that allows us insight into patterns and trends that are otherwise hidden from us. And it has the ability to paint that picture for many of us at one time - greatly accelerating the time it takes our exemplary intuitive leader to win over his team and get them all rowing in the same direction - and then quickly adjust if necessary. Without the data story, she is left with, "Just trust me on this one!" and picking up the pieces after the results are obvious to everyone.

At the end of the day, regardless of the size or velocity of the data stream we have at our fingertips, it is still on us to ask the right questions, tell compelling data stories, and take the right actions - moral or otherwise. The data itself does not have a moral compass - but we do have the power to use it to help support decisions that to positively affect the ability of many people to support themselves and their loved ones. From that standpoint, it could be argued that it is unethical to shun a powerful tool that could help prevent making a "gut" decision that turned out to be short-sighted. If only we had known!

June 21, 2013 at 2:44pm

Greg Stevenson

Gut intuition had the world flat and the sun revolving around the earth until data corrected our intuition. Gut intuition had being gay as a sickness to be cured and black being inferior to white before data put us on a path of, so far, semi-enlightenment.

"Do unto others as you would have done unto you" is backed up by masses of data and yet "Screw them before they screw you" in a lot of business transactions seems to ge the gut intuition.

June 21, 2013 at 1:52pm

Peter Altschuler

You are ignoring a vital advantage of Big Data analysis, especially graph analysis -- the ability to find relationships that you not only didn't expect but never imagined existed at all. In fields as diverse as medical research, fraud detection, and cybersecurity, graph analysis of Big Data resources can reveal relationships that can accelerate the development for cures for disease, eliminate and prevent financial crime, and discover connections between activities that could affect international and national events.

On a much more mundane level, retailers can gain insights into how specific products sell -- by demographics, geographics, price, and countless unanticipated influences -- and adjust their inventory and sales tactics accordingly.

What graph analysis can do that traditional querying can't is to incorporate hundreds of terabytes of data, regardless of its structure, and process it all at once. Standard queries on commodity hardware need to have homogeneous data resources and can only work effectively on subsets of all the information that's available. That creates siloed results that don't take into account all the facts and their often hidden relationships and do look at things in an unnatural vacuum (which, of course, can be very misleading).

The Big Data insights gleaned with graph analytics -- insights which might not otherwise occur to us -- help us view the world from a revised perspective, and that different view of the world could qualify as a type of wisdom.

June 21, 2013 at 1:34pm

Alfredo Bregni

Data is not information, information is not knowledge, knowledge is not understanding, understanding is not wisdom.

Cfr "A Theory of a System for Educators and Managers", http://www.youtube.com/watch?v=2MJ3lGJ4OFo (since minute 3:37 if you are in a hurry).

June 21, 2013 at 1:01pm

Harrison Withers

The problem with calling it "big" data is that it implies that big means more when really all we wanted all along was "better" data. Having data never forces you to do anything about it, and certainly the prescence of data never forces decisions. Having the right information needs to be part of making the "right" decisions as much as intuition.

I agree HR functions, notice I didn't say departments, needs to practice what you call big intuition, but they also need to be really carefull about appearing to reject the big data trend. There are a lot of things that help us here, it's no so black and white, and it doesn't have to be dehumanizing.

You have a good balance in the article, but the way it's positioned is very us vs. them. It's not data vs. intuition, rather it's how we can augement intuition with data.

Search form