A/B Test Design And Incremental Multichannel Campaign Performance

Never before has the traditional "A/B" test been as important as it is in our multichannel ecosystem. Such a simple concept, the "A/B" test is uniquely designed to measure the incremental performance of marketing activities.

As an example, assume a multichannel organization mails a catalog to a housefile list of 1,000,000 names. The database marketer chooses the best 1,100,000 households, and randomly splits them into two groups. The "A" portion of the test are the 1,000,000 households who receive the catalog. The "B" portion of the test are 100,000 households who will not receive the catalog.

Maybe a month after the in-home date, the database marketing analyst is prompted to analyze the results. Within each group, the 1,000,000 who received the catalog, and the 100,000 who didn't receive it, the analyst calculates the average net sales within the catalog/telephone channel, the online channel, and the retail channel.

Here are sample results:

	Quantity	Telephone	Online	Retail	Totals
Received Catalog	1,000,000	$6.00	$8.00	$21.00	$35.00
Did Not Receive Catalog	100,000	$2.50	$7.00	$19.50	$29.00
Incremental Lift		$3.50	$1.00	$1.50	$6.00

In this example, the catalog drove an incremental $3.50 per customer to the catalog/telephone channel, $1.00 per customer to the online channel, and $1.50 per customer to the retail channel, for a total of $6.00 incremental sales per customer.

Because we mailed 1,000,000 households, the total net sales attributed to this mailing is 1,000,000 * $6.00 = $6,000,000.

Some vendors advocate a different methodology --- they advocate allocating any online and retail order generated during the time the catalog was active to the mailing of the catalog. This results in a gross over-estimation of the importance of the catalog. Please don't go down this path.

A similar methodology can be used to test multiple marketing activities at the same time. Assume an e-mail campaign was mailed to the opt-in portion of this audience. Within this audience, you randomly assign customers to one of four test segments. Here are some sample results.

	Quantity	Telephone	Online	Retail	Totals
Catalog + E-Mail	400,000	$5.50	$8.50	$21.25	$35.25
Catalog Only	50,000	$6.00	$8.00	$21.00	$35.00
E-Mail Only	50,000	$3.00	$8.10	$18.65	$29.75
No Catalog, No E-Mail	50,000	$3.50	$7.00	$18.50	$29.00

Tests like these yield interesting and intriguing results. Notice that the best strategy for the catalog/telephone channel was to mail only a catalog. The best strategy for the online channel was to mail a catalog and an e-mail. The best strategy for the retail channel was to mail both a catalog and an e-mail.

Statisticians can assist with significance tests, if you feel that is appropriate. It is more important to simply execute tests of this nature, and learn how all of your marketing activities interact with each other. What you learn about how marketing activities and channels interact with each other within our multichannel ecosystem may surprise you.

What Is More Important, Sales Increases Or Increased Profit?

In a recent comment, Ray S. asked if it is acceptable to have an advertising campaign that increases net sales, but does not increase profit.

Let me pose two scenarios to you.

Campaign Result, Scenario #1: Net Sales increase by $100,000 verses last year's campaign. Profit is flat, compared with last year's campaign.

Campaign Result, Scenario #2: Net Sales decrease by $100,000 verses last year's campaign. Profit increases by $25,000, compared with last year's campaign.

Ok, you are the CEO of this company. Which of the these two scenarios is preferable to you?