As many companies gear up for growth, CFOs are perhaps more optimistic than they have been in a long time. Few, however, are likely to be as optimistic as Wall Street analysts. A recent study by consulting firm McKinsey & Co. finds that on average, analysts’ forecasts of annual earnings have been almost 100% too high, both over the past 25 years and during shorter intervals that were sampled.

Since 1985, “actual earnings growth surpassed forecasts in only two instances, both during the earnings recovery following a recession,” co-authors (and McKinsey consultants) Marc Goedhart, Rishi Raj, and Abhishek Saxena write in the report; the two instances were the periods 1991 to 1996 and 2003 to 2006. In general, analysts “were slow to revise their forecasts to reflect new economic conditions, and prone to making increasingly inaccurate forecasts when economic growth declined.”

Forecast inflation was the subject of much scrutiny back in 2002, when evidence emerged that equity analysts were pumping up certain stocks in order to generate investment-banking business from those companies. Following the 2003 Global Research Analyst Settlement between the Securities and Exchange Commission and 10 Wall Street firms (and two analysts, Jack Grubman and Henry Blodget), some researchers expected forecasts to move closer to reality, since some of the possible perverse incentives for overly glowing reports had been eliminated.

In fact, earnings and analysts’ five-year rolling average estimates do track fairly closely for the years between 2001 and 2006. The authors, however, attribute the coincidence of the two to “strong economic growth” between 2003 and 2006 rather than a more sober view of the markets.  The margin of error widened again in 2007 and 2008, they note, as the economy entered a recessionary period. In general, the accuracy of estimates was higher during periods of growth compared with those of decline.

Rodney Sullivan, who has surveyed many research studies related to analyst behavior as head of publications at the CFA Institute, says the results are no surprise. “One of the things we’ve learned from behavioral finance is humans have a real tendency to be optimistic and overconfident,” he says. And while analysts try to factor in the business cycle, “they tend to miss inflection points in the economy, because it’s very, very difficult” to forecast them, adds Sullivan.  

Indeed, “one reason analysts are slow to update their forecasts is that they do in-depth research and build complex forecasting models…[and] there is a trade-off between accuracy and complexity,” says Kenneth Posner, former managing director and equity analyst at Morgan Stanley and author of a new book, Stalking the Black Swan: Research and Decision-Making in a World of Extreme Volatility (Columbia University Press, 2010).

Sullivan points to one somewhat surprising factor that seems to have an influence on an analyst’s optimism: gender. In a study published last year in the CFA Institute’s Financial Analysts Journal, Emory University professors Clifton Green and Narasimhan Jegadeesh and graduate student Yue Tang found that female analysts consistently forecast earnings to be 0.5% lower, on average, than male analysts. (Interestingly, the forecasts by women were also statistically less accurate than those by men, with men’s errors about 1.5% smaller than women’s).

Posner says that in order to correct the problem, investment and research firms should track the accuracy of analyst estimates, “so that analysts get specific feedback with which to improve their forecasting process,” something “the best shops already do.”

The lesson for executives is a slightly different one. “For executives, many of whom go to great lengths to satisfy Wall Street’s expectations in their financial reporting and long-term strategic moves, this is a cautionary tale worth remembering,” write the McKinsey consultants. Adds Posner: “There is a feedback effect between the markets and [stock] fundamentals, but if a CFO or CEO tries desperately to hit optimistic forecasts and then the stock goes up more, they’re on a path that ultimately leads to a fall.”

For their part, investors seem to be savvy enough to see through whatever window-dressing may happen. “Except during the market bubble of 1999-2001, actual price-to-earnings ratios have been 25 percent lower than implied P/E ratios based on analyst forecasts,” note the McKinsey consultants. As of November 2009, in fact, the actual forward P/E ratio of the S&P 500 was consistent with long-term earnings growth of 5%, a “more reasonable assessment” than the analysts’ vision of the future, in the researchers’ estimation.