Explaining The Matchback Mistake
Now that I've frustrated many of you by not aligning with catalog industry best practices (i.e. the right way to implement results from a "matchback" analysis), allow me to explain the philosophical issues surrounding the methodology.
Catalogers like to look at a "segment" of customers, folks with similar behavior, folks with consistent future performance.
For instance, assume it costs a cataloger one dollar to mail a catalog. Also assume that thirty-five percent of all demand flows-through the p&l, resulting in "contribution" or "variable operating profit".
We mail a catalog to this segment of 10,000 customers, folks who last purchased within the past three months, and have spent $250 - $499 in their lifetime with the company.
By measuring source codes, we learn that this segment spent $2.00 per customer over the telephone. We run a profit and loss statement, and observe the following:
| Households | 10,000 |
| Demand | $20,000 |
| Flow-Through | $7,000 |
| Book Cost | $10,000 |
| Contribution | ($3,000) |
In other words, we lost money mailing this segment of customers.
This is where the matchback analysis comes in. Savvy catalog marketers partnered with list processing and compiled list vendors to "match" all customers who received a catalog, but ordered online instead, "back" to the catalog mailed to the customer. Typically, the most recent catalog mailed gets credit (and we can address all the flaws with that methodology another day).
In this instance, the "matchback" analysis shows that customers mailed this catalog also spent $2.00 online during the life of this catalog. This changes the profit and loss statement, illustrated below:
| Households | 10,000 |
| Demand | $40,000 |
| Flow-Through | $14,000 |
| Book Cost | $10,000 |
| Contribution | $4,000 |
Now all is good in the world! The catalog drove online volume, the profit and loss statement works. Catalog list processing vendors, compiled list vendors, paper vendors, and list rental vendors rejoiced because the catalog becomes a viable marketing vehicle responsible for the majority of the online volume harvested by a business.
This strategy works well when the online channel is incapable of generating its own volume. In 2007, this is often an incorrect and dangerous assumption. This is where mail/holdout testing comes into play.
Simply put, mail/holdout testing shows you how much online volume occurs if a catalog isn't mailed. The methodology points out the fundamental flaw in a matchback analysis.
For many catalogers (certainly not all, maybe not even half), half of the online demand will happen anyway if a catalog is not mailed. In these instances, the mail/holdout testing clearly illustrate this finding (see the last article, business model number three).
In the case of our profit and loss statement, adding in one dollar per customer instead of two dollars per customer changes the profit and loss statement, illustrated below:
| Households | 10,000 |
| Demand | $30,000 |
| Flow-Through | $10,500 |
| Book Cost | $10,000 |
| Contribution | $500 |
In this case, the segment is above break-even, so depending upon your profitability criteria, the segment can be mailed next year.
It is this last profit and loss statement that catalogers need to be evaluating.
Almost all catalogers are mailing too many catalogs due to flaws in the implementation of the matchback analysis. This isn't the fault of your list processing or compiled list vendor. It is our fault, we failed to adequately understand customer behavior.
At Nordstrom, when we killed our catalog division, our online division actually continued to grow sales, year-over-year. Matchback analysis suggested that killing the catalog would create a catastrophe. Our inventory management team nearly fainted, thinking the implosion would be epic!
Mail/holdout testing accurately forecast a subtle sales hit that would largely be offset by organic growth in the online channel. Within a month of killing the catalog, we observed that mail/holdout testing was right, that matchback analyses were highly flawed.
Another flaw in the implementation of matchback analysis is attributing online orders to the original source (which in most cases, is catalog).
In other words, the catalog marketer gives the online channel credit for taking the order, but says that the order could never have happened had catalog marketing not been responsible for originally acquiring the customer. This analytical technique assures that catalogs will always gain too much credit --- in these cases, I've seen orders generated by paid or natural search (i.e. Google) attributed to catalogs, because the customer was acquired via a catalog twelve years earlier. I'd stay away from this popular method of attribution.
I realize what I am saying is utter heresy to most in the catalog industry, as evidenced by the feedback I receive from you! As leaders, we have a responsibility to maximize sales and profit in the business models we support. Let's measure the evolution of our business in a fair manner. We need to take our catalog silo hat off, and put our brand hat on. We'll still find that catalogs are an important part of the marketing mix used to educate customers about our merchandise offering.
Labels: catalog, Database Marketing, Matchback Analysis, Multichannel Forensics, online
posted by Kevin @ 7:05 AM
10 Comments:
Well Kevin, for what it's worth, the same thing happened at HSN... so I join the heresy team.
We had to use control groups (holdouts) for any direct activity because the TV was everywhere, there was no way to get away from it, no way to "hold out" anything. There was always incremental sales in every segment from TV, even if you "did nothing".
Along the way, we realized best practice was not to look at the success of campaigns, but to look at customer profitability over a promotional sequence, knowing TV was always a contributor in some way. This was the only way to know which segments were profitable for catalog and customer retention mailings for TV.
This "matchback offense" is bad in catalog, but you should see what some of the online folks are doing with "allocation models" - attributing a percent of the sale to various online sources based on number of exposures, time sequence (last campaign viewed), etc.
They too will have to move away from "campaign profitability" and towards "customer profitability" as the key metric. As usual, this will only happen when sales growth starts to slow and people begin to ask the right questions.
It will be very hard for my industry to make the move you're talking about.
Very hard.
My industry knows something feels wrong, we just don't know how to articulate our uncomfortable feelings yet.
So I'll keep writing about it until we start to make progress.
As database marketing professionals, we probably haven't done as good a job as we could to help our businesses get through this transition. We have to find a way to bridge campaign-based analysis with customer-based analysis. And even though we've largely done this, our business partners frequently fail to understand what we're communicating.
With regard to measuring what we call multi touch attribution in the online world, at Avenue A | Razorfish we have performed several controlled studies (using holdout groups) to identify the interaction effects that occur across multiple media contacts and to configure consumer-level behavioral models that allocate credit across multiple touchpoints.
It may offend the sensibilities of a direct marketer, but here’s why this move is afoot…
The standard model for crediting transactions to media in online is called last touch attribution. The most recent click on an advertiser’s online media (usually a search result or banner) is credited with the sale. If there is no click, then the last ad viewed is credited.* In either case, 100% credit is given to the last online “touch.”
This, of course, is the matchback analysis problem that you describe, only constrained to the online environment (where we don’t have effort codes to tie a transaction to a specific medium).
We’ve always realized that this was, let’s just say, an imperfect attribution method. So we started using holdout studies to test how much display media (banner ads) drives response through sponsored search. We constructed random groups to receive “control” banners instead of advertiser media – those groups would have the advertiser’s media replaced by “Control” media that contained a message from an unrelated advertiser. We measured response rates relative to the number of consumers (cookies) who were, or would have been, exposed to the advertiser’s display media.
We found that, in most cases, banner ads about doubled search-driven visitation and more than doubled the search conversion rates vs. those who hadn’t been exposed banners. Those conversions, which normally would be credited to search, belong to display media. Attributing those sales to the banners that drove them more than doubles the effectiveness of banner ads… significantly changing the way in smart, numbers-driven marketers would choose to invest their marketing budgets. This will come as no surprise to the catalog folks in your audience.
I spoke on this subject and this research at the Market to the Max conference in Seattle. I made it clear there how this same methodology applies not just to banners and search, but also to catalog and search…or to direct mail and banners, or to all three. The key, of course, is to be able to tie the contact data together.
Others have also weighed in on this matter too. I've seen some great co-presented work by Hanover Direct, Coremetrics, Yahoo, and Atlas Solutions. These, and other great companies, are exploring attribution methodologies that attempt to more accurately credit sales to the causal touchpoints. I support and applaud their efforts.
We cannot get away from that fact that purchasing decisions are made by consumers based on a whole set of experiences and interactions by that consumer and his/her network of friends, colleagues and associates. Each of those interactions plays a different role - sometimes complementary and sometimes competing - in driving awareness, consideration, shopping and, ultimately, the decision to purchase at a given point in time.
Of course, these methodologies are fraught with their own complications. We’ll never be perfect at attributing conversions to the media that drove them. But as database marketers, it’s incumbent upon us to try and understand and convey to our less data-driven colleagues the relative importance of different media in driving transactions. The better we understand this, the higher the marketing returns we will be able to drive (as both a % of investment and in contribution dollars).
Isn’t that the end goal?
* view-through conversion credit is limited to client-specific "conversion window" - usually 30 days or less between media and conversion
I recall the embryonic stages of the testing you're talking about ... back in the day at Avenue A, we put Red Cross banners in the control group!
This post has been removed by the author.
That's the standard, but we also use other charities at the client's request. It's tougher in the days of online branding and rich media...~B
Kevin, I clipped a piece from your original entry below to ask a specific question that it relates to:
It's been proposed that it might be appropriate to matchback sourced paid search demand for branded keywords to a catalog mailing. For example, if you had demand that came through on a general term like 'widgets' then it would be sourced to paid search, but if the term were 'Acme widgets' then it would be matched back to Acme's most recent catalog mailed to that individual. I would be curious to get your thoughts on this new development.
Another flaw in the implementation of matchback analysis is attributing online orders to the original source (which in most cases, is catalog).
In other words, the catalog marketer gives the online channel credit for taking the order, but says that the order could never have happened had catalog marketing not been responsible for originally acquiring the customer. This analytical technique assures that catalogs will always gain too much credit --- in these cases, I've seen orders generated by paid or natural search (i.e. Google) attributed to catalogs, because the customer was acquired via a catalog twelve years earlier. I'd stay away from this popular method of attribution.
I'm not a big fan of the branded / non-branded distinction, when it comes to attribution.
If I get a Lands' End catalog, see a dress I like, then go online and do a search for 'dresses', then click thru to Lands' End, then the catalog deserves to receive partial credit for creating that order.
Branded search verses non-branded is a marketing distinction, in my opinion, not a distinction that is made by the customer.
Yes, I agree with that. My question is more around whether it's ever appropriate if you know that a customer definitively came through paid search, to attribute the sale to the most recent catalog mailing in the last 52 weeks just because their search term had your brand name in it. Some have proposed that if the customer is searching with your brand name, it's because they have a catalog in hand.
I would disagree with what "some" people are suggesting!
Prove to me they have your catalog in their hand.
Prove to me that a catalog mailed 51 weeks ago has any relevance today.
In fact, quickly recite for me the catalogs you received on this date last year?
All kidding aside, the real question is where did prior knowledge of a brand come from? Catalog pundits want to attribute it to the catalog, at times because this increases the relevance of catalog in an age where catalog relevance is ultimately decreasing. There are vendors who depend upon catalogs for their existence --- these vendors want you to attribute the order to a catalog mailed fifty two weeks ago.
Conversely, you have a customer who had a good customer experience. She tells you about the sweater she got at J. Crew. You search for a J. Crew sweater via Google, and buy something online.
These orders have nothing to do with catalog. They have everything to do with a brand providing a consistently good experience for customers. We just don't have a good bucket/categorization to "attribute" this order to.
Post a Comment
Links to this post:
Create a Link
<< Home