Mail And Holdout Tests: Segmentation
If you really want to understand the incremental impact your marketing activities have on your business, be sure to have more than 5% of your file in the mail and holdout groups. Obviously, you cannot have 30% of your file in mail and holdout groups, but you might be able to accommodate 7.5% or 10% of the file in each segment.
Once you have the appropriate number of customers in the mail and holdout group, go ahead and segment each group based on prior customer behavior. Then, run your analysis (incremental results of the mailed group minus the control group) on segments of customers.
Here are some of the segments you might wish to analyze:
- Telephone Shoppers.
- Online Shoppers.
- Your Digital Profiles (often very enlightening!).
- E-Mail Buyers.
- Non E-Mail Buyers.
- Those Who Click On E-Mail Campaigns.
- Those Who Do Not Click On E-Mail Campaigns.
- Recent Buyers (i.e. Last Three Months).
- Lapsed Buyers (i.e. Last Purchase = 13-24 Months Ago).
- Paid Search Buyers.
- Affiliate Marketing Buyers.
- Banner Ad Buyers.
- One Time Buyers.
- Multi-Buyers.
- Multi-Channel Buyers.
- Retail Buyers.
- Those Who Buy From A Merchandise Division.
- Those Who Buy From Multiple Merchandise Divisions.
- Urban vs. Suburban vs. Rural Buyers.
- Discount vs. Full-Price Buyers.
- Those Who Buy Inexpensive Items.
- Those Who Buy Expensive Items.
Oh, there's so many other variables you can create segments from and analyze. Simply put, there's so much value to be had by conducting these type of tests, coupled with very simple measurement techniques.
So, why aren't more e-mail marketers and catalog marketers executing these tests? Your thoughts?
Labels: Mail And Holdout Tests
posted by Kevin @ 8:15 PM
11 Comments:
I think in a tough economy, when most are working hard just to hit a plan, holding out potential sales is tough to push through. Most are more interested in the long term when the short term numbers are above expectations.
I think if the economy keeps improving, more companies will start going in this direction.
As far as email holdout tests go, I wonder why cut back unless you think it will increase sales (less contacts more purchases?). Email is sooo cheap, so the chance of potentially missing a purchase over saving a couple hundred bucks here and there just doesn't move right up my priority list. It's like the long tail of paid search, there is a small cost you could probably save, but the cost is so low all it takes is 1 order and you've got great ROI. I'd rather look for ways to optimize the emails I'm sending than cut costs by not sending them at all.
Derek,
Give this situation a try.
You send 12 catalogs and 52 e-mails a year. In total, you get $45,000,000 from catalog mailing activities and $5,000,000 from e-mail activities.
Now, you test sending 26 e-mails a year instead of 52. In total, you get $46,000,000 from catalog mailing activities and $4,000,000 from e-mail activities.
Finally, you test sending 0 e-mails a year instead of 52. In total, you get $50,000,000 from catalog mailing activities and $0 from e-mail activities.
You think you don't want to know that your e-mail campaigns are completely cannibalizing your catalog activities, causing the company to generate $0 of true incremental volume?
This happens all of the time. Of course, nobody knows this, because nobody tests. If you knew this, you might try harder to figure out the optimal strategy for e-mail marketing among your catlog audience, and among all other customers, right?
Conversely, e-mail is often twice as productive as measured for retail brands, because of what e-mail drives to retail. You cannot see what you drive to retail unless you are willing to execute the holdout test.
Now, if e-mail is twice as valuable as measured via open/click/convert metrics, wouldn't you want to know that? Wouldn't you consider increasing e-mail frequency? Wouldn't you consider adding staff so that you could execute more targeted strategies, with special versions targeted to customers with specific purchase history, a strategy that would increase performance of a strategy that is already working 2x as good as measured via open/clicks/convert? Now you'd have the metrics to prove that you can hire more staff.
As Jay mentions above, most people think you're holding out e-mails and then reducing sales. Most people would actually benefit from testing, learning if e-mail actually works or if it just cannibalizes other channels.
That makes sense, but based on historical information before/after emails, I know I am making more with a catalog and email than I would be with just a catalog. Exactly how much more? I don't know. But I know it is more than enough to offset the low cost of sending out emails.
To judge email frequency I think it is more important to look at direct response (open rates, click-throughs, conversion, and, most importantly, unsubscribe rates).
When it comes to holdout testing, I think it is much more important to be doing holdout testing on catalogs. We all know the limited life print has left, so we should learn how to mail smarter as we taper off catalog mailing and switch to a more e-com heavy business model.
I see validity in doing email holdout testing... Just not for me, right now.
Oh boy!
Jay,
One thing I've found to be effective is calling a project "optimization" instead of "testing"-eg, "Advertising Optimization" instead of "Ad Testing".
When you focus on the intended benefit instead of the process it's much easier for executives to see why the project is worthwhile.
No sure how to read into that response Kevin...
I guess I am confident the increase I have seen from email is large enough to fully offset the low cost of sending. Down the road it might be nice to run some holdouts and see what the real increase is, but that is way down on my priority list compared to things like catalog holdouts. Am I wrong in thinking this?
You're entitled to whatever opinion you think is right for your business, Derek. You're the expert for your business, so go with that!
Thank you, Satvik.
In response to "I think in a tough economy, when most are working hard just to hit a plan, holding out potential sales is tough to push through.", I feel it may be more important to execute these tests in tough economic times.
Bad economy typically means weak sales, which translates to reductions in marketing spend.
Recouping a significant portion of non incremental marketing spend would be huge! That said, it should be done in situations where you expect you're overspending... which can be challenging.
I agree, Mike, thanks!
Post a Comment
Links to this post:
Create a Link
<< Home