Creating change: When your intuition is wrong

Only a third of excellent companies remain excellent over the long term. Even fewer change programs succeed. These are the facts, yet these need not be the odds of success for your organization. Insightful advice (beyond common sense) and pragmatic methods (readily applicable) are available to help almost any leader and organization create successful transformation and sustain excellence.

This post describes two counterintuitive insights about creating lasting change, based on more than a decade of research and client work, including input from more than 600,000 senior executives and employees in 500 organizations, hands-on work with more than 100 organizations, and research drawn from more than 900 books and academic articles. (For more detailed insights, please see our book Beyond Performance: How great organizations build ultimate competitive advantage.) Other posts in this series will describe how any organization can improve its performance and its health simultaneously, and a couple of useful tools for implementing change.

Common sense will often lead you astray

Much of current thinking on change management appeals to common sense. This should raise alarm bells. Leaders who rely on their common sense when influencing change typically spend time and energy on the wrong things, send messages that miss the mark, and create frustrating unintended consequences. Why? Because common sense neglects a basic truth about human nature: that people often act irrationally. Scientific study has shown that we fall victim to subconscious thought processes that influence our behavior even when our rational minds tell us they shouldn't. That's what happens when we spend 10 minutes driving around to find the closest parking space to a store even though we could have saved half that time by parking in the first empty space and walking the few extra yards.

Behavioral scientists have found ways to understand how seemingly irrational decisions like this one are influenced by social, cognitive, and emotional biases and can therefore be predicted. Leaders who want to make change happen should follow their example. The good news is that there's a small number of predictable biases that can be turned from miscues into powerful ways to help transform your organization.

Here's the difference. Leaders using their common sense will want to tell employees about their organization's problems: costs growing faster than revenues, competitors catching up, or whatever the challenge may be. They'll decide what to do about it and tell employees the full story--what, why, and how--everywhere they go. They'll link the results of the transformation program to bonuses so that the effort has teeth. They'll invest in giving employees the technical skills they need to succeed. This all sounds fine--yet chances are they'll fail miserably. We've seen it happen time after time.

So what went wrong? These leaders did some of the right things, but not all. By relying on common sense, they failed to take advantage of less obvious but more potent ways of motivating employees. They missed an opportunity to create broad ownership of the new direction by involving a critical mass of people in setting the aspiration. They focused only on problems to be solved and neglected strengths to build on, breeding fatigue and resistance. They chose the most expensive form of incentives rather than cheaper and more effective ones, and forgot that relational skills are as necessary as technical skills. The result: failure, wasted effort, and frustration all round.

The 'soft stuff' can be managed as rigorously as 'the hard stuff'

We often hear executives say that the health of their organization is important. However, they freely admit they struggle to find concrete, structured ways to measure and manage it. In one survey we conducted, two-thirds of executives wished they had better information to help them determine "what needs to be done to strengthen the company's health for the longer term."

This lack of guidance may be partly to blame for the ill fate of so many change programs. When we analyzed the reasons behind these failures, what we might assume to be the usual suspects--inadequate resources, poor planning, bad ideas, or chance events--turned out to account for less than a third of cases. The vast majority, more than 70 percent, were caused by factors related to organizational health, such as negative employee attitudes and unproductive management behavior.

Yet managers can get access to comprehensive, robust approaches to measure their organization's health. By that, we don't mean employee engagement or customer satisfaction surveys (useful though these may be in certain situations), but tools that yield a deeper insight into what drives an organization's alignment, execution, and renewal. The approach we have developed is the organizational health index or OHI, which assesses performance in terms of 37 management practices that have a proven correlation to financial results. The OHI provides a precise vocabulary and reliable measures for all aspects of the "soft stuff" that matter, making them as tangible and manageable as financial and operational metrics.