In the fairly recent past, I worked as a part of the leadership team at a company that was built and run almost entirely on the intuition and instincts of a small group of founders. There was plenty of raw data and a good deal of relevant information around too, but when it came to the critical decisions around strategy, key customer relationships and managing enterprise risk, the information generally took second place to the senior leadership’s gut feelings.

Opinion_Bug7Given that this approach had built a highly profitable, $4 billion-revenue public company from scratch in less than a decade, it was hard to argue with — at least for as long as the founders were around and active. However, as the business grew it was clearly going to be harder, and eventually impossible, for every key decision to be vetted by the core set of “intuitions” that had driven success. It was also going to be increasingly difficult to keep regulators and the market (in the form of financial and industry analysts) happy that the business was being run on a sound footing. Even some of the large customers began to question why we were using  incomplete or out-of-date information to manage customer relationships. We never seemed to know as much about them as they expected us to.

Things came to a head when it was time for some of the founders to step aside. New management wanted information-based decisions, not gut feelings — which triggered a cultural upheaval in large part because the founders had generally hired people who operated pretty much the way they did — on intuition and instinct.

As it turns out, I’d seen situations like this before.

John Parkinson

John Parkinson 2017 Tech

Although I trained as a mathematician and computer scientist, the most useful classes I took during college were economics and behavioral psychology. In the latter, we spent a lot of time looking at how good human beings are at applying intuition to the world around them and how they generally go about it. Even back then, behavioral psychologists had collected a lot of observational and experimental data on the workings of human intuition. [If you want a generally accessible source on much of this and subsequent work, I highly recommend a 2011 book from economics Nobel laureate Daniel Kahneman, “Thinking Fast and Slow.”]

We generally have good intuition about things that are similar to what we encounter every day, and are able to make “instinctive” decisions (based on comparisons with our experience) that are generally correct.  So we know what a “fair” price is – even for something brand new — or how long a task should take, even though we haven’t done exactly it before. But we have poor intuition about things that are outside of everyday experience (how to identify and react to the risks represented by a tsunami or an earthquake, for example) and very poor intuition about things that are totally alien — things outside of all human experience, like quantum mechanics or nanotechnology.

This is especially true when we are dealing with scale effects (very large or very small numbers, for example), complexity (the technology of the Internet or the behavior of macroeconomic markets) or non-linear processes (such as bank runs or “flash mobs,” where the seemingly rational actions of a few people trigger sudden irrational responses in many others, who may have no clear idea of what is going on but get caught up in the frenzy for fear of missing out or of incurring a loss that could be avoided).

Specific groups can and do use intuition much better than the human population as a whole. Successful scientists, inventors and explorers tend to have pretty good intuition about the “discovery” of new things — and almost every field of human endeavor has examples of people who have a “knack” for getting things right “more often than not.”

Even in business there have been examples where information guided intuition in meaningful ways. The “information based strategy” (IBS) approach adopted by the founders of Capital One Financial in the late 1980s is one of the best known. Capital One used deep analysis of consumer profile data to target (and design) a diverse set of of financial products within what had previously been a “monoline” segment. Other information- (or at least data-) based strategic approaches were adopted by companies such as GE (under Jack Welch). Additionally, many of the data-focused management tools of the past 50 years (activity-based costing, Six Sigma, lean and total quality in operations, and forced-ranking evaluation in human capital management) have been attempts to take the intuition and instinct out of routine business processes.

It’s clear that there have been many successes with information-based approaches, but there have also been some well-documented failures. Automated candidate-assessment software and forced-ranking evaluations alienate employees and may discourage the risk taking that underpins innovation. Six Sigma is a powerful process-improvement tool but a poor occupational religion. And information-based approaches depend on access to comprehensive, accurate and up-to-date information that is actually relevant to the decisions being made. Both too little and too much information often derails these approaches, forcing executives and managers back to using intuition and instincts that may not have been well honed for the task.

So what should we be doing? What place does intuition play in a world awash in “big data” and ever more powerful “analytics”?

As usual, the best answer for organizations seems to be to strike an appropriate (if difficult and dynamic) balance. Just as our intuition is less than perfect, so in general is our information. Using information (remember that it needs to be as complete as possible, accurate and up to date — none of which come free) to enhance and support intuition ought to be the objective. The old adage that “you are entitled to your own opinion but not your own facts” applies even today. But with the right facts to back you up, your opinions are likely to be much more valuable.

John Parkinson is an affiliate partner at Waterstone Management Group in Chicago. He has been a global business and technology executive and a strategist for more than 35 years.

, , , , , , ,

4 responses to “The Role of Intuition in a World of Big Data”

  1. John,

    Well said. With all of the hype around ‘big data’ it is refreshing to see a ‘quant’ recommending the ‘soft skill’ of intuition. I agree with your notion of ‘AND’ thinking around ‘big data’ and intuition.

    Although I would add a caution of relying on experience too much for that intuition. I recently spent some time with Rita McGrath who wrote ‘The End of Competitive Advatange’ (highly recommend this book) where she makes the point that we now live in a world of transient advantages where competition can come from anywhere and disruption of is rampant.

    This requires us to actually call into question our ‘historical knowledge’ and combine it with a future-focused perspective AND the use of Analytics and data to question and validate those assumptions.

    Thanks for the thought-provoking piece and the book recommendation, I just downloaded it from amazon.

  2. Apologies for the length of this, but I had just written a blog on this very topic when I saw this article. I was thinking about this topic as I watched two movies back-to-back, Moneyball and Trouble With The Curve. It’s a fascinating juxtaposition when you watch each of these recently released movies about strategies to recruit baseball players. On the one hand, you have Billy Beane, General Manager of the Oakland Athletics, who discovers in Moneyball that the traditional system of evaluating baseball player talent is all wrong, and that only through advanced analytical techniques, first proposed by Bill James back in the 1970s, can one cut through the noise and clutter and assemble a competitive baseball team.
    On the other side of the coin, you have the Clint Eastwood character in Trouble with the Curve, a long-tenured baseball scout, who eschews technology in favor of actual in-person player evaluation. His contention is that the numbers never tell you the whole story, that only through years of experience, intuition and personal evaluation can coaches and general managers truly make the right decision on a player.
    As your article points out, this same dichotomy presents itself in business every day. Some leaders “manage from the gut”, to borrow from Jack Welch. Other leaders manage by the numbers. As is often the case in life, there is no one clear cut answer as to who’s right. In my opinion, after 20 years as an analytics leader, the answer is a little bit of both. All too often, the “numbers” people get so enamored with their analytical “toys”, that they lose sight that there are people involved, whether employees or customers, and people are notoriously variable in how they behave over time. The numbers people fall into the trap of relying so heavily on their analytics, that they forget (or ignore) how the raw data came into being in the first place. They often lack the basic knowledge of the business they are analyzing, the processes that generated the data, or the context in which their recommendation must be acted upon. This trap is becoming more pronounced with the advent of the “Data Scientist”, as armies of PhD statisticians crunch through massive quantities of raw data, however by their very nature these “scientists” often deal in the theoretical and are unable to deliver impactful, actionable recommendations because they are often stuck in the theoretical and not in the practical. They lack the fundamental understanding of, or appreciation for, the details in business processes.
    Thinking about Moneyball, the main characters, Peter Brand and Billy Beane, actually had an understanding of the game to complement their analytic techniques, which made them a powerful combination. Once they assembled their team, Billy Beane set out to “coach up” the players and managers on how they needed to implement his theories on the field of play. This is the vital step in making analytics actionable. Their process was not without its missteps. For one thing, the coaching staff, rooted in their traditional view of evaluating player talent, refused to adapt their methods of teaching to the new world that the analytics team was bringing them. Conversely, the analytics team (in this case, Beane and Brand) became so enamored with their statistical methods initially, that they signed a player with a troubled past, despite their scouting staff’s protestations. They forgot that team chemistry and leadership are an important part of the success equation that the numbers don’t reveal. It wasn’t until the coaching staff modified their approach, and they got rid of the troubled player, did the team actually perform.
    On the flip side, in Trouble with the Curve, the general manager of the Atlanta Braves solely uses numbers to guide his decision to draft a “can’t miss” prospective player out of High School, in spite of his talent scout’s (Clint Eastwood) assessment that this young player can’t hit a curveball due to a defect in his swing. In this case, because of the level of competition the player was involved in previously, his raw ability masked this defect, and what appeared superior by the numbers was highly misleading at best. The emphasis here is that there is no substitute for “getting your hands dirty”, that is dropping the numbers and analytics, and actually get out in the field, or the production floor, and really understand what’s going on that drives the numbers. In my experience, the most successful analysts are those who have a firm grasp of the business context in which they are operating. This produces two benefits, the first is that it reduces analytic iterations, or rework. The second is that the results are nearly always actionable and, more importantly, accurate.
    Despite what might on the surface appear to be two diametrically opposed viewpoints in comparing these two movies, in reality is agreement in terms of a symbiotic relationship between analytics, experience and intuition to make fully informed business decisions. It also highlights that success is driven by complete organization buy-in to new analytic approaches, and analytic team commitment to learning the business context in which they are operating.

Leave a Reply

Your email address will not be published. Required fields are marked *