Matchback Analysis Challenge
Catalogers --- here's what I'd like for you to do on your next catalog.
Step 1: Randomly select 10,000 customers from your mailing.
Step 2: Divide these customers into two groups.
Step 3: In the first group, code these customers as "mailed". Send these customers a catalog.
Step 4: In the second group, code them in your promotional history as "not mailed". Do not send these customers a catalog.
Step 5: After these catalogs have been mailed and results have been measured, send each group of customers to your favorite matchback vendor, be it Experian or Abacus or anyone else.
Step 6: DO NOT TELL EXPERIAN OR ABACUS THE DIFFERENCE BETWEEN THE TWO GROUPS. Simply perform the matchback analysis on each group.
Step 7: Carefully analyze the results in each group. Remember, the second group DID NOT RECEIVE A CATALOG. Therefore, every order that Experian or Abacus attributes to a catalog in this group represents an "overstatement", a "matchback bias" in web orders that are attributed to catalog mailings. Remember, these customers DID NOT GET A CATALOG, you're just communicating to your matchback vendor that they got a catalog, so that you can measure the "matchback bias" that exists in all of these analyses.
After you've done this little experiment, share the results with my audience. I'll be happy to publish what you've found, regarding "matchback bias". I think you'll be surprised by what you learn from this experiment. I think you'll see that you are over-stating the importance of catalog mailings at driving web sales.
Labels: Abacus, Experian, Matchback Analysis, Matchback Bias
posted by Kevin @ 1:11 AM
10 Comments:
What a great idea, Kevin! I look forward to reading the results. Well done.
And you'll be sure to let me know how Experian does against Abacus, right?
Great idea. You could go further and design some experiments to see how groups that get the catalog, get a personalized catalog, get a small catalog and don't get any catalog perform too. In fact you could make designing such experiments, and measuring the results, part of how you do business so you constantly compare different approaches to see what works best - this is sometimes called adaptive control.
JT
www.edmblog.com
Ron, I'll give you any insider information I get!!
Jeff, we'll see what happens.
James, a lot of bigger companies do this type of testing, and they can afford to. Smaller companies need every penny of revenue they can get, so in my industry, they frequently do not execute the kinds of tests you suggest, so that they can pay the bills. Smaller companies rely upon Abacus or Experian to do matchabacks, and use that data to estimate the value of catalog direct marketing. That's where the overstatement occurs ... if they could afford to do what you propose, I think they would implement your suggestions!
Yes, it will be interesting to see. Programmatically speaking, if the name and address is not on the mail file (+/- a few percent for data errors and matching logic) they can not be allocated to the mailing.
Obviously, there is some variance caused by overlapping mailings and the responder windows used for the mailing.
Also, sometimes a looser matching logic is used (address only) to capture pass-alongs within a business site or physical address (my wife with a different last name gets mailed and I buy).
I look forward to the results.
Jeff,
I apologize for posting to this lead, but I have an important question:
Abacus was bought by Epsilon who in turn is owned by Alliance Data Solutions. Alliance Data Soltuions was recently bought by a private equity firm, The Blackstone Group. The Chinese government then bought a $3 Billion position in The Blackstone Group.
My question is: Who has my data?
Sincerely,
Concerned.
Anonymous, thanks for commenting. You do raise an interesting point, though maybe it could be considered on another blog that is directly addressing that specific topic.
We do have folks from Epsilon/Abacus who read (and comment on) this blog often. These folks work with a high level of integrity.
Anonymous, I can empathize with your concern given the recent news. I am proud to say Abacus has an unblemished record over the past 16 years of protecting our client’s data and upholding our promises in how we use that asset. It’s the foundation of our business and we would never intentionally put it at risk. The quick answer to your concern is nothing changes. We have a state-of-art data center and your data will continue to be treated appropriately regardless of which name is on the building. We would be glad to give you a tour of the facility and review our security, data protection, and business practices, just give your account executive a call.
Ironically, Epsilon manages marketing databases for some of the largest consumer brands and ADS processes many of the credit and payment transactions that occur everyday including owning a bank. Both these companies manage data which is much more sensitive than is used at Abacus and operate with an unyielding commitment to security and data protection. If nothing else, the integrity of your data will only be higher going forward.
Frankly, I would have greater concerns about sharing data with smaller providers who may not have the financial resources to adequately protect it or (heaven forbid) be financialy motivated to provide the data to other companies for use in a manner inconsistent with your agreement with them.
Hope that helps.
Another source of bias, which varies in significance by target market and mailing strategies, is mailing duplicate customers. This primarily occurs between business and home addresses and multiple home addresses.
Most marketing databases contain some portion of buyers at multiple addresses and these are not removed in the merge resulting in mailing to one address and buying from another. If the second address is on the mail file, it will get the credit, if its not, it will be an unallocated order.
Casey --- fortunately, the bugs that you talk about will average out at the same rate, whether in the test group that actually receives the catalog, or the control group that actually does not receive a catalog. Therefore, when subtracting the difference, the biases have a net effect of zero. That's good!
Because matchback algorithms overstate the importance of catalog marketing activities, this test should be able to quantify just how much we overstate catalog marketing activities by.
This has become a big issue --- especially since I'm being asked all the time to quantify the impact of reducing catalog mailings, now that postage is getting out of control. Many catalog marketers are looking to make the transition from catalog marketing to online marketing --- and make that transition in a way that minimizes the hit on top-line sales.
These marketers cannot get away from catalog marketing when matchback algorithms overstate the importance of catalog mailings.
Post a Comment
Links to this post:
Create a Link
<< Home