Most of us tend to walk through life relying on intuition for many of our decisions. Gut feeling is often this nagging voice in our head that is telling us to disregard what the data says and do what we feel is the right thing to do. This seems like the best way to make decisions in many areas of our life. Otherwise, we would be paralyzed by indecisiveness and a constant need for data. You can probably try a new brand of yogurt without much analysis, letting your gut feeling tell you that strawberry-flavored will be just fine.

However, it should also be apparent that in many situations, we would be well advised to trust data and algorithms over our gut feeling. And yet, we don’t. The problem is that while we are used to people being wrong, we don’t give the same courtesy to algorithms. To err is human, but seeing the algorithm fail, even once, will lead to a significant drop to our ability to trust it again.

Consider something many of those who drive cars experience. GPS based navigation can pinpoint your place on this planet with incredible accuracy. If you are using a good navigation application with good map data, the algorithm will lead you to your destination in pretty much a hundred percent of cases. However, it is enough for the app to drag you through some narrow backroads only ones, because it doesn’t have the map data on a highway opened last week, and you stop trusting it. You are more likely to ignore its advice next time and go by your gut feeling even though chances are the road you select will be worse than the one proposed by the navigation app.

Berkeley Dietvorst, Joseph P. Simmons, and Cade Massey researched this phenomenon and confirmed that data or evidence-based algorithms predict the future more accurately than human forecasters. However, people would still pick a human forecaster over the algorithm. The researchers call it algorithm aversion. Curiously, this aversion increases when people see the algorithm in action, and even when they see it outperform the human. The human forecaster could make mistakes and still be trusted. If the algorithm made a mistake, the trust in it evaporated.

The explanation most people provide why to bet on humans even when the algorithm is superior was that humans can learn and with practice get better. There might be even an ego at play as if we would admit the algorithm is better than the human forecaster we would also admit that it is better than us, “I know what I’m doing, and some computer won’t tell me what to do.”

Daniel Kahneman and Gary Klein published a study that directly dealt with data-based and intuition-based decision making. Even though intuition can sometimes work, especially for senior professionals, it is far from infallible. The intuition judgment is highly dependent on the predictability of the environment as well as the deep expertise of the individual who knows the ins and outs of that environment.

The same was confirmed by researcher Erik Dane who discovered that intuition works well in areas where we have lots of expertise. It doesn’t work at all in areas where we don’t know what we are doing in the first place. Your intuition only works when you can tap into a database of patterns to compare with and do so in a stable and predictable environment. Your subconscious mind can compare what you see or experience with past events, find patterns, and approximate results. That’s when your intuition can be pretty accurate.

If you don’t have this database of patterns ready to use in your mind, your intuition will fail you. This, of course, means that in a fast-moving and ever-changing world, intuition is less and less reliable. The pattern database is just getting obsolete too fast and can point us easily in the wrong decision. What worked yesterday won’t work tomorrow.

If you question the premise that algorithms do better than humans when it comes to decision making because the environment we live in is so messy and data we collect are inconsistent and lacking, then think again. As Andrew McAfee points out, it is precisely in this messy environment when the algorithms outperform humans the most.

The reason is simple. As stated above, an intuition that is part of human decision-making works best in stable, predictable environments where people have the opportunity to get enough practice with frequent feedback. As McAfee, together with Daniel Kahneman, suggests, this is true, for example, in medicine as human biology doesn’t change that much. It is not true in an environment such as stock trading that is very unpredictable indeed. In these unstable environments, the algorithms win because they are better able to filter out the noise and find the patterns. Less or inconsistent data will lead to a better prediction by the algorithm than a human would make. To be clear, less or insufficient data will, of course, lead to a worse decision regardless of who analyzes them.

Putting it all together

Intuition works. But only in a stable environment, when you have enough expertise and immediate feedback to learn the patterns. If you are a novice in the field, or the environment is constantly changing then you would be well advised to forget what your gut is telling you and go with data and algorithms instead.

---

What are your thoughts on the topic? How important intuition is? What are your strategies to make good decisions? When do you rely on intuition and when don’t?

Photo: nvodicka / Pixabay.com

Follow me on Twitter: @GeekyLeader