Mail And Holdout Tests: The Basics

Ok, folks, here are the basics of executing a mail/holdout test ... for both catalog marketers and for e-mail marketers.

Catalog Marketers:

• Identify the audience that is likely to receive catalogs in the next three months.
• You'll need about ten percent of your audience to measure this well.
• Randomly sample 5% of your customer audience. This group can receive catalogs if they are eligible, if they do not meet your selection criteria, don't mail a catalog to this customer.
• Randomly sample 5% of your customer audience. This group is not allowed to receive catalogs, unless they choose to request a catalog.
• Keep customers in each of these two segments for a period of three months.
• At any time during the test period, you are free to analyze the test results. Subtract the difference between the mailed group and the holdout group. This is the incremental value of catalog mailings. Run a profit and loss statement on this incremental difference.
• Analyze the incremental difference in telephone demand, online demand, e-mail demand, paid search, natural search, affiliate marketing, banner ads, portal advertising, and shopping comparison sites. You may find that catalog marketing drives incremental business to some of these channels.

E-Mail Marketing:

• E-Mail marketing results are often over-stated by click-through and conversion rate analyses, except in retail instances, where e-mail marketing results are often under-stated.
• Use the same rules outlined above for catalog marketers. Sample 5% of your e-mail file, and treat these customers normally. Sample 5% of your e-mail file, and do not send e-mail marketing messages to this audience for a period of three months.
• Measure the incremental difference between the mailed group and the holdout group. This is the true value of e-mail marketing. You are likely to be very surprised by the outcome of the test!