When Metrics Lie

Our metrics continually lie to us.

Some are so easy to debunk ... "Multichannel Customers Are The Best Customers".

Some are much more complex.

Back in 1993, Lands' End was caught up in the "targeting" movement. In other words, the core monthly catalog was not driving large enough sales increases to please management and shareholders. As a result, management chose to create specialty catalogs, targeted to customers who previously purchased niches of product. What was once a catalog strategy that yielded thirteen catalogs, one every four weeks, became a catalog strategy that offered some customers thirteen monthly catalogs, nine Kids catalogs, nine Home catalogs, six Mens Tailored Merchandise catalogs, four Womens Business Merchandise catalogs ... you get the picture.

The finance department ran profit and loss statements for each business. As the businesses matured, the profit and loss statements indicated that these new, targeted catalogs were working really well, while the monthly catalogs were struggling. Here is an example (numbers dummied-up to protect the innocent):

Financial Profit And Loss Statement
	Core	Startup Line	Total
	Business	Of Business	Business
Demand	$80,000,000	$20,000,000	$100,000,000
Net Sales	$64,000,000	$16,000,000	$80,000,000
Gross Margin	$32,000,000	$8,000,000	$40,000,000
Less Mkt Expense	$16,000,000	$2,720,000	$18,720,000
Less Pick/Pack/Ship	$7,360,000	$1,840,000	$9,200,000
Variable Profit	$8,640,000	$3,440,000	$12,080,000
Less Fixed Costs	$6,400,000	$1,600,000	$8,000,000
EBIT	$2,240,000	$1,840,000	$4,080,000
% of Net Sales	3.5%	11.5%	5.1%

On the surface, the specialty catalog, the one targeted to prior customers of various merchandise assortments, performed well.

This didn't sit well with some of us. So we created year-long tests, where some customers did not receive specialty catalogs.

At the end of the year, we learned that one of the specialty businesses, if terminated, would hand sixty percent of the business back to the core monthly catalogs. In other words, the specialty catalog was cannibalizing the core catalog ... making the core catalog look bad, from a performance standpoint.

We worked with the finance department to re-state the profit and loss statements. Look at the difference a 60% cannibalization rate creates.

Actual Customer Behavior Profit And Loss Statement
	Core	Startup Line	Total
	Business	Of Business	Business
Demand	$92,000,000	$8,000,000	$100,000,000
Net Sales	$73,600,000	$6,400,000	$80,000,000
Gross Margin	$36,800,000	$3,200,000	$40,000,000
Less Mkt Expense	$16,000,000	$2,720,000	$18,720,000
Less Pick/Pack/Ship	$8,464,000	$736,000	$9,200,000
Variable Profit	$12,336,000	($256,000)	$12,080,000
Less Fixed Costs	$6,400,000	$1,600,000	$8,000,000
EBIT	$5,936,000	($1,856,000)	$4,080,000
% of Net Sales	8.1%	-29.0%	5.1%

This profit and loss statement is one that reflects actual customer behavior. The targeted specialty catalog is truly only adding $8,000,000 of incremental volume, and is actually losing $1.9 million a year.

Needless to say, this information was not well received.

But the information represents the truth.

We're so quick to believe the metrics we read every day that we seldom question them. "Oh, this landing page had a 6.3% conversion rate, that's good, that must have worked!!". It is possible that something else caused the landing page to work, that the landing page itself wasn't successful, but that something we aren't measuring caused the success?

We learn that our metrics are wrong when we operate in a "testing culture". A testing culture demands that we ask questions, that we develop hypotheses, and that we test our hypotheses. All of the magic happens when different strategies are tested.

The KPI/Dashboard maven is right to want to create nine important metrics that dictate success --- most employees struggle with multidimensional issues and complexity.

But if you are reading this, you are not like "most employees". You are different. You are curious. You do not accept the version of truth the trade journals, bloggers, vendors, or consultants paint for you.

Truth isn't illustrated by a metric. Truth is multidimensional, and is always available to those who seek it.