Optimizing Direct Mail Testing: Measuring Incrementality

Terence Einhorn, Sr. Solutions Consultant in Sales

Published 03/14/2024

Introduction to Direct Mail Testing

With the continually rising costs of print and production and the multitude of digital alternatives, it’s more important than ever to test and validate the performance of direct mail programs.

Below, we’ll take a quick look at the different types of direct mail experimentation, strategies for maximizing the impact of testing, and how to interpret results to drive performance.

What is the Importance of Testing in Direct Mail Campaigns?

There are many factors that can impact the performance of a direct mail campaign and many levers a brand can pull to optimize its program. 

Everything from the recency and purchasing frequency of your target audience to the frequency and size of catalog drops, book size, and cross-channel support from other media tactics can significantly impact the ROI of direct mail programs.

That’s why it’s so critical to understand how these factors influence performance via in-market testing: failure to do so can lead to millions of dollars left on the table with suboptimal strategies.

Holdout Testing: The Key to Understanding Direct Mail Impact

The first and most important objective of a direct mail testing program is to understand the impact, or “incrementality,” of your program as a whole.

The most simple way to answer this question is with a holdout test, which takes a portion of the target audience (typically 5-10%) and withholds the delivery of direct mail for a set period of time (one catalog drop, one month, etc. depending on details of the program).

The key piece of information revealed in a holdout test is the portion of an audience that would still have converted anyway had they not been served that direct mail drop. This can then be used to adjust the total number of conversions being attributed to direct mail to represent only the true incremental impact of a program.

For example, if the conversion rate in the exposed group is 1.0%, and the conversion rate in the holdout group is 0.5%, that means that half of the conversions being credited to the direct mail drop would have happened anyway, and therefore, half of the sales incurred from that catalog drop should be deducted when calculating the campaign’s ROI(i).

What is Multivariate Testing in Direct Mail?

Once a few holdout tests have been conducted and the overall budget behind direct mail has been balanced against its true Incremental ROI, the next step is to begin optimizing the delivery details of your campaigns.

This entails splitting an audience into multiple groups, similar to a holdout test, but instead of creating treatment vs. holdout groups, we create various treatment groups that represent different delivery strategies.

This process can take many forms, depending on which variables are relevant to a particular program.

Audience Segment Strategy

Within your housefile, you may have several key segments based on either observed conversion data like recency, frequency, and monetary patterns or more qualitative data like demographics and interests.

Either way, you can test which of these segments provides the best return on direct mail Investment by carving out a separate “holdout” cell for each segment and measuring the Incrementality of the segments individually.

This will allow you to calculate the true ROI of each audience segment and adjust the targeting strategy accordingly.

Book Strategy

Perhaps you’ve created a few different formats of your recent catalog you’d like to test.

Splitting an audience into multiple groups and serving either different book formats, book sizes, or even frequencies of drops to the same audience is a great way to dial in the best strategy and balance cost with Incrementality.

Contact Strategy

The most complex but often the most instructive multivariate test for direct mail is a “Contact Strategy” test.

This involves looking at different combinations of list-based Media (e.g., email, SMS, customer social programs, etc.) and evaluating their impact on sales both separately and together.

For example, brands can carve out four different customer cells, one receiving direct mail ONLY, one receiving Meta Advertising ONLY, one receiving Meta Advertising AND direct mail, and one receiving neither.

Such a test would instruct not only the incrementality of direct mail and Meta separately but also the degree to which they work alongside one another.

We may find a synergistic or cannibalistic relationship between these two tactics, which will inform how we fund them alongside one another in the future.

See a graphic example of a more complex Contact Strategy test design below:

Key Considerations for Direct Mail Tests

There are a few key items to consider when running direct mail experiments, all of which can have a significant impact on results.

Universal Holdouts vs. Refreshing Holdouts

A common feature of legacy direct mail testing methods is a “universal holdout”, which is basically a portion of an audience that is continuously held out from receiving direct mail over an extended period of time.

This is a good holdout method if you are trying to gauge the overall, absolute, cumulative impact of a direct mail program, as it takes into account the impact of repeated direct mail exposure over time.

However, if we want to isolate the impact of a given direct mail drop or perhaps test different strategies within the same drop, selecting a new (or refreshed) holdout for the experiment yields a more accurate result.

Time of Year

Another key consideration is the time of year, as seasonal patterns and promotional periods can significantly impact the performance of any media tactic.

A good best practice is to test the total Incrementality of a direct mail program during a “Business as Usual” time of year, as well as your most Seasonal period, to gauge how seasonality impacts performance.

Better yet, you can run a continuous universal holdout to measure how Incrementality changes from drop to drop.

What are the Key Metrics to Measure?

No matter which type of test you deploy, there are a few key metrics to consider when interpreting results:

ROI(i) - Incremental ROI is the ultimate measure of a campaign’s performance and represents how much money was incrementally generated for the business for each dollar spent.

Generally speaking, brands should aim for a total ROI(i) just above their profitable break-even while maximizing spend for a given media tactic.

There are three sub-metrics that ultimately contribute to an ROI(i):

Cost per Piece - Simply put, this is the amount it costs you to reach your audience with direct mail.

Conversion Rate - The percentage of your audience that purchased an item after receiving a direct mail drop.

Incrementality - the portion of these conversions that were truly driven by that media (as opposed to those that would have happened anyway).

There are infinite other factors that contribute to these three variables, but ultimately, these are the core metrics that impact your final ROI(i) and the ones to track most closely.

Interpreting Test Results: Making Data-Driven Decisions

For a simple holdout test, interpreting results is easy: once you’ve calculated your true ROI(i) using incrementality results from the experiment, simply judge whether that ROI(i) is above or below your profitable breakeven as a business.

Due to diminishing returns, ROI(i) will fall as you scale upwards and rise as you remove budget. Therefore, if ROI(i) is lower than breakeven, you should pull back on budget until ROI(i) returns to that breakeven number. If ROI(i) is higher than breakeven, then you should push budget into the program until ROI(i) is close to your profitability target.

For more complicated multivariate experiments, we want to compare the measured ROI(i) between treatment cells (e.g., direct mail + Meta vs. direct mail alone) and identify the most profitable strategy to continue funding moving forward.

No matter the strategy selected, it’s always a good idea to follow up with a basic holdout test to validate performance in the new “Business as Usual” scenario resulting from these learnings.

Want to Learn More About
Incrementality and Attribution?