Break the Bad Habit of Overreacting to Metrics

Quarter-to-quarter fluctuations may be no reason to pull out your hair, and other tips for evaluating metrics.
Mark GrabanJanuary 6, 2020
Break the Bad Habit of Overreacting to Metrics

CFOs, of course, play a key role in overseeing financial metrics and, increasingly, operational performance measures. They pay a great deal of much-needed attention to the question of what to measure. But an opportunity typically remains for getting better at evaluating metrics.

The methods and mindsets mentioned in this article should save time, improve focus, and thereby drive greater performance this year and beyond.

A common challenge arises when a CFO overreacts to the latest data point. Say, for example, that the “inventory turns ratio” has decreased from 4.1 for the previous quarter to 3.9 for the just-ended one. Is it time to look for a root cause, or to read someone the riot act? Does this decrease indicate a meaningful trend?

4 Powerful Communication Strategies for Your Next Board Meeting

4 Powerful Communication Strategies for Your Next Board Meeting

This whitepaper outlines four powerful strategies to amplify board meeting conversations during a time of economic volatility. 

Not necessarily.

It’s possible that a metric might just be fluctuating around an average, within a typical range. We can call that “routine variation.” It exists in business metrics as well as personal measures like our weight or blood pressure.

Reacting to routine variation (or asking others to investigate or explain it) results in a lot of activity that doesn’t really help improve the business. If inventory turns are just fluctuating, the next reporting period might show an increase to 4.3. Is that change worth celebrating? Is it a new, more positive trend? Again, not necessarily.

Here’s a recommendation: Look at more than two or three historical data points to get a better sense of whether a metric is simply fluctuating or there is a meaningful trend or shift that must be understood.

Since quarterly metrics provide four data points a year, it takes a long time to detect trends. Additionally, sometimes-meaningful variations in monthly or weekly numbers get averaged out and lost in quarterly numbers, which hampers analysis and response.

Recognizing that two (or three) data points are not a trend, some finance organizations utilize a so-called “dashboard” showing a large table of numbers. It actually looks more like a spreadsheet than a dashboard.

Much too often, finance shares financial and operational metrics in overly detailed tables, even though it’s difficult (if not impossible) for people to separate routine variation (a.k.a. “noise”) from exceptional variation (a “signal” in the metric) when it’s displayed this way.

 I recommend visualizing metrics using a simple chart of the last 18 monthly data points, as it’s easier to see trends and shifts over time.

Another recommendation: When displaying time-series data, avoid bar charts (in Excel they’re called “column charts”), even though businesses and financial news publications commonly use them. Research shows that they can be misleading because they draw the human eye to the middle of each bar, while the actual data point that should be taken in resides at the top of the bar (in a vertical bar chart).

It would be better for finance organizations to use “line charts” (as they’re called in Excel), which draw eyes to the data point in question.

Sometimes a line chart shows an undeniable trend or massive change in, for example, accounts receivable or net cash flow. Or, there might be a single data point that looks like a temporary outlier (good or bad) or a shift that, after a few data points, seems permanent. And as suggested above, there might also be data points that look like fluctuations around an average over time.

Outer Limits

If CFOs can break the habit of reacting too often to a metric’s ups and downs, a little bit of simple (but valid) math can help them avoid guessing or using “gut feel” to determine whether there’s a meaningful change (a signal) in the metric.

A proven statistical method called “process behavior charts” (a phrase coined by Donald Wheeler, a leading academic in the field of business analytics) adds calculations to the run chart of a metric’s historical performance: the average, a “lower limit,” and an “upper limit.”

A signal is a message from the metric that says something worth understanding has changed significantly in the business. The signal may suggest it’s time to do root cause analysis, or it may be a confirmation that a change made to the business is creating different results.

There are three rules for finding signals. First, any single data point that’s outside of the lower or upper limit is a signal. Second, eight consecutive data points that are all above or below the average is a signal of a sustained shift in performance.

Third, any three out of four consecutive data points that are closer to a limit than the average signals that the metric does not reflect simply a fluctuation around an average.

As a final opportunity for improvement, display charts that show, on a rolling basis, the last 18 months of data. I’ve seen companies that dutifully track metrics each month (or week) throughout the year, only to start the new year with a blank chart. It doesn’t make sense to go into February with charts that show just one data point; it’s then impossible to separate signal from noise.

I hope this article triggers discussion among executive teams about opportunities to improve the way they look at metrics.

Mark Graban is a consultant, speaker, blogger, and author of management books, most recently “Measures of Success: React Less, Lead Better, Improve More.” He holds an MBA degree from the MIT Sloan School of Management and has worked in manufacturing, software startups, and health care consulting for 25 years.