In the fairly recent past, I worked as a part of the leadership team at a company that was built and run almost entirely on the intuition and instincts of a small group of founders. There was plenty of raw data and a good deal of relevant information around too, but when it came to the critical decisions around strategy, key customer relationships and managing enterprise risk, the information generally took second place to the senior leadership’s gut feelings.

Given that this approach had built a highly profitable, $4 billion-revenue public company from scratch in less than a decade, it was hard to argue with — at least for as long as the founders were around and active. However, as the business grew it was clearly going to be harder, and eventually impossible, for every key decision to be vetted by the core set of “intuitions” that had driven success. It was also going to be increasingly difficult to keep regulators and the market (in the form of financial and industry analysts) happy that the business was being run on a sound footing. Even some of the large customers began to question why we were using  incomplete or out-of-date information to manage customer relationships. We never seemed to know as much about them as they expected us to.

Things came to a head when it was time for some of the founders to step aside. New management wanted information-based decisions, not gut feelings — which triggered a cultural upheaval in large part because the founders had generally hired people who operated pretty much the way they did — on intuition and instinct.

As it turns out, I’d seen situations like this before.

John Parkinson 2017 Tech

Although I trained as a mathematician and computer scientist, the most useful classes I took during college were economics and behavioral psychology. In the latter, we spent a lot of time looking at how good human beings are at applying intuition to the world around them and how they generally go about it. Even back then, behavioral psychologists had collected a lot of observational and experimental data on the workings of human intuition. [If you want a generally accessible source on much of this and subsequent work, I highly recommend a 2011 book from economics Nobel laureate Daniel Kahneman, “Thinking Fast and Slow.”]

We generally have good intuition about things that are similar to what we encounter every day, and are able to make “instinctive” decisions (based on comparisons with our experience) that are generally correct.  So we know what a “fair” price is – even for something brand new — or how long a task should take, even though we haven’t done exactly it before. But we have poor intuition about things that are outside of everyday experience (how to identify and react to the risks represented by a tsunami or an earthquake, for example) and very poor intuition about things that are totally alien — things outside of all human experience, like quantum mechanics or nanotechnology.

This is especially true when we are dealing with scale effects (very large or very small numbers, for example), complexity (the technology of the Internet or the behavior of macroeconomic markets) or non-linear processes (such as bank runs or “flash mobs,” where the seemingly rational actions of a few people trigger sudden irrational responses in many others, who may have no clear idea of what is going on but get caught up in the frenzy for fear of missing out or of incurring a loss that could be avoided).

Specific groups can and do use intuition much better than the human population as a whole. Successful scientists, inventors and explorers tend to have pretty good intuition about the “discovery” of new things — and almost every field of human endeavor has examples of people who have a “knack” for getting things right “more often than not.”

Even in business there have been examples where information guided intuition in meaningful ways. The “information based strategy” (IBS) approach adopted by the founders of Capital One Financial in the late 1980s is one of the best known. Capital One used deep analysis of consumer profile data to target (and design) a diverse set of of financial products within what had previously been a “monoline” segment. Other information- (or at least data-) based strategic approaches were adopted by companies such as GE (under Jack Welch). Additionally, many of the data-focused management tools of the past 50 years (activity-based costing, Six Sigma, lean and total quality in operations, and forced-ranking evaluation in human capital management) have been attempts to take the intuition and instinct out of routine business processes.

It’s clear that there have been many successes with information-based approaches, but there have also been some well-documented failures. Automated candidate-assessment software and forced-ranking evaluations alienate employees and may discourage the risk taking that underpins innovation. Six Sigma is a powerful process-improvement tool but a poor occupational religion. And information-based approaches depend on access to comprehensive, accurate and up-to-date information that is actually relevant to the decisions being made. Both too little and too much information often derails these approaches, forcing executives and managers back to using intuition and instincts that may not have been well honed for the task.

So what should we be doing? What place does intuition play in a world awash in “big data” and ever more powerful “analytics”?

As usual, the best answer for organizations seems to be to strike an appropriate (if difficult and dynamic) balance. Just as our intuition is less than perfect, so in general is our information. Using information (remember that it needs to be as complete as possible, accurate and up to date — none of which come free) to enhance and support intuition ought to be the objective. The old adage that “you are entitled to your own opinion but not your own facts” applies even today. But with the right facts to back you up, your opinions are likely to be much more valuable.

John Parkinson is an affiliate partner at Waterstone Management Group in Chicago. He has been a global business and technology executive and a strategist for more than 35 years.