Testing, Testing: The New Innovation Game

Harrah's Entertainment exemplifies the burgeoning use of broad, IT-enabled experimentation to improve businesses.
David McCannMarch 15, 2010

Business innovation, once the province of teams in white lab coats doing research and development, is moving into the front lines. Increasingly it derives from continual tiny experiments in such areas as business processes and customer relationships rather than a single, company-transforming idea.

Internet-based companies, which can alter their “storefront” at will — “What will happen if we put the shopping cart on the left side of the screen?” — are certainly the most adept at such granular testing. Big players like Google and can sometimes get a statistically significant amount of data to make a decision in a matter of hours.

Offline, the mass-experimentation approach is more difficult but not out of reach. Erik Brynjolfsson, a professor at Massachusetts Institute of Technology’s Sloan School of Management, points to gaming giant Harrah’s Entertainment as a leading-edge experimenter among the brick-and-mortar set.

4 Powerful Communication Strategies for Your Next Board Meeting

4 Powerful Communication Strategies for Your Next Board Meeting

This whitepaper outlines four powerful strategies to amplify board meeting conversations during a time of economic volatility. 

It’s no coincidence that Harrah’s CEO, Gary Loveman, like Brynjolfsson, holds a doctorate in economics from MIT. While information-technology advances allow more precise experiments and measurements, having a culture that encourages their use is equally important, Brynjolfsson says. “[Loveman] brought that mind-set — he likes to crunch the numbers to figure out what could be made more profitable, and it has spread throughout the company,” says the professor, who is also director of MIT’s Digital Center for Business, which researches how IT is transforming companies.

At Harrah’s, the most pervasive use of testing (always with a proper control group) is for marketing, CFO Jonathan Halkyard says. There are literally millions of purchase transactions daily at the company’s 40 gaming properties, 77% of them tied to individual customers, so data-gathering opportunities are almost limitless.

A typical example of a test would be making an attractive hotel offer in Las Vegas for a Tuesday night to a group of customers who live in Southern California and usually visit on weekends. In that instance, “We see whether we can migrate demand from peak to off-peak times and whether we should therefore invest more in those customers,” says Halkyard.

Most companies track the success of marketing initiatives, of course, but Harrah’s stands out from the norm in both the number of experiments and the degree to which they are targeted at narrow slices of customers — sometimes even at individuals.

A customer may, say, come into Caesars Palace and spend $1,000 per hand on blackjack for 15 minutes and leave. “That person’s spend with us might not seem to justify any reinvestment in him,” says Halkyard. “But we can see that he is a 45-year-old male from L.A. who hasn’t stayed with us before and make a guess that we’re getting just a little bit of his gaming budget.”

The customer may then get a marketing offer that’s worth more than the revenue Harrah’s collected from him in those 15 minutes. That kind of predictive analysis is done “hundreds of times a day” at each of the company’s casinos, according to Halkyard.

Harrah’s also is unusually zealous in its efforts to measure customer satisfaction. The company sends out several thousand surveys a week and gets 25% of them back, says Halkyard. That enables a “fairly quick” determination as to the impact of small, experimental changes in such things as front-desk procedures or lounge, valet, or beverage services.

The finance department is heavily involved in making sure the individual casinos perform the expected level of testing, eliminate programs that are not driving profitability, and prop up those that are. And every Tuesday night Halkyard gets a scorecard that compares each property’s labor productivity by department. On Wednesday morning, he calls those that are underperforming. “It’s something that gets a lot of visibility,” he says.

He allows that there’s quite a bit of room for improvement in one area: examining the impact of capital investments. While the company does a good job of that with regard to new slot machines, it hasn’t become very sophisticated yet in measuring how remodeling hotel rooms or restaurants influences the customer experience. “If we were to renovate 500 rooms in a hotel but not the other 500, I suspect we could draw some inferences about whether that was a remunerative investment for the company,” says Halkyard.

The information-technology backbone that enables all of the analysis is a data warehouse that connects the dots among various sources of information — primarily the company’s general ledger, player-rating system, customer-satisfaction scores, and hotel management system.

“IT has created a platform that makes data analysis a lot easier,” says Brynjolfsson. “If something works they can do more of it, and if not there’s a bunch of other experiments running. It’s a rapid-feedback system. That’s the new type of innovation, and it’s very much driven by IT.”

But as far ahead of the curve as Harrah’s may be, its massive-scale business-improvement experimentation is still in its early days. “It’s not like this is a mature set of practices,” says Brynjolfsson. “It’s far from over.”