What is a Holdout Test in Marketing?

• A holdout test measures the causal impact of a marketing channel by withholding it from a representative group and comparing outcomes to a group that continues receiving it as usual.
• The core question a holdout test answers: “How many conversions would I lose if I removed this channel from my mix?” — a counterfactual that platform attribution can never provide.
• Holdout tests come in two main forms: known-audience splits (for CRM channels like email, catalog, SMS) and geographical splits (for broad-targeted channels like CTV, social prospecting, and paid search).
• Holdout tests differ from A/B tests: holdouts measure a channel’s absolute incremental contribution, while A/B tests optimize creative or targeting within a channel.
• Modern measurement programs treat holdout tests as the causal foundation that calibrates ongoing MMM and platform attribution — not as one-time experiments.

In today’s environment of tighter budgets and higher performance expectations, marketers can’t afford to invest in media that isn’t driving causal business impact. The goal isn’t just to prove that a channel is delivering sales, it’s to determine how many of those sales were truly incremental and wouldn’t have happened without that investment.

The reality is that every media platform claims credit for a large share of conversions, many of which would have occurred even without that channel running. The critical question for marketers is: How much of this is real lift, and how much is noise? In 2026, that question has moved from analytical curiosity to boardroom priority: continued signal loss from privacy regulations, the cancellation of Google’s Privacy Sandbox cookie replacement, and the dominance of AI-driven buying algorithms like Performance Max and Advantage+ have made aggregate, experimental measurement essential for defending marketing budgets with confidence.

That’s where incrementality measurement comes in. Holdout experimentation is the gold standard for isolating the true causal contribution of a channel. By deliberately withholding exposure from a representative “holdout” group while running media as usual for the rest of the audience, brands can measure the difference in outcomes and quantify the lift driven by that specific channel.

In other words, a holdout test answers the essential counterfactual: “How many conversions would I lose if I removed this channel from my mix?” This framework is the foundation of modern causal measurement and a core pillar in Measured’s triangulated approach, alongside Media Mix Modeling (MMM) and platform attribution. Together, these methods give brands a continuous, validated view of media effectiveness, without relying solely on platform-reported numbers.

A leading specialty retailer wanted to validate the true incremental value of its catalog program, which had historically been evaluated using last-touch attribution. The brand suspected that a portion of reported sales would have occurred without the catalog, but they had no quantified measure of lift.

Measured designed a known-audience holdout test by randomly splitting the brand’s active customer list into test and control groups, balanced on recency, frequency, monetary value, and catalog eligibility. The control group did not receive the catalog during the test window, while the test group continued receiving it as planned.

After the campaign, Measured analyzed purchase behavior between groups and found that the catalog was delivering a 14% incremental lift in revenue among recipients, far below the 40% contribution claimed by the vendor’s attribution reporting. This insight allowed the brand to optimize catalog circulation toward higher-value customer segments, reducing wasted spend while preserving the majority of incremental revenue.

The test gave the marketing team a defensible, causal read on catalog performance, and the results were fed into the brand’s MMM and triangulated measurement framework to continually calibrate ongoing media investment decisions.

A holdout test removes (or “holds out”) a media from a selected audience and observes their conversion behavior vs. a control audience who continues receiving that media as per usual. 

*Note: the exposed group is referred to as a “control” in this case because they represent the typical Business as Usual scenario of the media being active.

One key feature of incrementality testing (as opposed to on-platform lift studies or brand lift studies) is that results are based solely on a brand’s first-party transaction data. 

This is critical because the core priority in experimentation is to measure a media channel’s net impact on a brand’s sales, including all the interactive effects the media may have on other media channels, organic channels, etc.

Initial Steps

Selecting a Tactic to Test

The first and most obvious step is to select the channel or tactic that you’d like to measure. This has implications for subsequent steps, as different channels are best measured with slightly different techniques (more on that later).

Generally speaking, the best place to start holdout testing is with the tactic receiving the highest spend in your portfolio. This is where informed actions resulting from the holdout test are likely to have the biggest business impact, simply due to the portion of the budget being affected.

Another consideration is the range of hypotheses of performance for a given channel. For example, there is almost no world in which an email program is unprofitable, regardless of its incremental contribution, simply due to how cost-effective it is to run. As a result, email is usually not a priority for testing.

On the other hand, social retargeting campaigns often take up a significant portion of the marketing budget and can perform wildly differently depending on specific targeting details (audience) and campaign execution details (like triggers, timing, frequency, etc.). This makes social retargeting a common priority for experimentation.

For certain platforms with automated targeting algorithms (e.g., Google Performance Max or Meta Advantage+ Shopping), special considerations need to be taken into account with channel/tactic selection. 

Duration and Time of Year

The next key consideration is the time of year and duration of your holdout test. 

Brands instinctively know that media performs differently throughout the year, depending on factors like seasonality, promotional events, etc. This is why it’s important to carefully select time periods that are representative of the business at large or to test the same tactic at multiple points during the year to quantify how these factors impact incrementality.

Additionally, the duration of an experiment should be considered prior to launching. Generally speaking, 30 days is a good standard to realize the full near-term causal impact of a marketing tactic, but for some brands with longer consideration periods, this is not enough.

A good rule of thumb is to run your experiment for at least the length of your average consideration period to ensure you give results time to react to the delayed impact of removing media from an audience.

Minimum duration can also be partially informed by Statistical Significance exercises (more on that later).

There are several methods for “splitting” an audience into representative test and control groups:

Known-Audience Split:

A known-audience split uses a defined list of identifiable users, such as existing customers or subscribers, and withholds a specific media tactic from a randomly assigned subset of that audience. This approach is only feasible in channels with deterministic, user-level targeting capabilities, most often CRM-driven channels like email, direct mail/catalog, SMS, or app push notifications.

When setting up a known-audience split, it’s essential to ensure the test and control groups are balanced on key dimensions that can influence purchase behavior. These typically include recency, frequency, and monetary value (RFM) of prior purchases, as well as eligibility factors such as opt-in/opt-out status for the channel being tested and any overlapping or related campaigns that could contaminate results. Proper stratification ensures the measured lift is truly causal and not driven by pre-existing differences between groups.

Geographical Split:

A geo-split is used when an audience is unaddressable, meaning that individual users cannot be targeted directly from a pre-existing list, and therefore, a known-audience split isn’t feasible. This applies to any channel that employs broad targeting like social prospecting, CTV, paid search, and most retargeting tactics.

The geo-split method identifies specific markets within a broader region (e.g., states within a country) that are statistically representative of that broader region and groups them together. The media in question is then removed from these test markets, and conversion behavior is compared to a group of control markets (business-as-usual).

When designing a geo-split, the main factors to account for and control for are a market’s sales trend and seasonality, population conversion rate (i.e., market penetration), and media relevancy (historical execution of the media in question being representative of the broader region).

Marketers often confuse holdout testing with A/B testing, but these are not the same thing. Holdout tests remove a media to evaluate its “absolute” contribution to the business, while A/B tests serve two different treatments to similar audiences to see which one drives better performance.

What are the Advantages of Holdout Tests Over A/B Testing?

The key advantage of holdout testing is that it measures the absolute incremental contribution of a media to the business. In other words, it tells you if a media provides a profitable return to the business overall, informing whether it should be further funded or not.

A|B Testing does not provide such insight but instead measures the relative performance of two different treatments, all else being equal. This is useful for optimizing messaging and creative within a channel but will not tell you how that channel is performing as a whole.

Scenarios Favoring A/B Testing

As mentioned above, A/B Testing is best used to optimize campaign performance within a given marketing channel. This generally falls into two categories:

Creative Optimization: Serving two different creatives to likewise audiences simultaneously, with the same campaign settings, to observe which creative is more compelling (i.e., which creative drives the higher conversion rate).

Channel Optimization: Varying a certain campaign setting between two likewise audiences while holding all else equal, for example:

• Targeting 
• Bid Types
• Optimization Events

Such “head-to-head” testing informs how to maximize performance or efficiency within a given channel but doesn’t inform total budget allocation between and across channels, which is the end goal for holdout testing.

There are two main methods for calculating the results of a holdout test:

1. Pre-Post Analysis

Also known as a “Difference in Differences” and by various other names, a Pre-Post analysis selects a reference period before the test and then calculates the “difference” in sales between the experiment period and the reference period for both test and control markets. 

These differences are then compared to see if the test (holdout) group “lost sales” compared to the control group.

While theoretically, this works, week-to-week or month-to-month noise and volatility within markets make this an incredibly unstable technique that can yield unreliable results.

2. Baseline Estimation

In causal inference, the factual outcome is the outcome that is observed based on a treatment and the counterfactual is the outcome that is not observed absent the treatment. 

Since the counterfactual is unobserved, we use statistical machine learning to estimate it. The counterfactual prediction model estimates the expected sales in the test markets had media not been removed – essentially baseline sales. The baseline is then compared to the observed sales when the media was removed to arrive at the estimated effect of the media.

There are various statistical techniques that can be used to create the baseline, but the general idea is to employ a regression model that leverages a few years of historical sales data in the control markets to “predict” likely sales in the test markets for any given time period.

The more representative test markets are of the broader region, the more accurate this prediction will be, hence the critical importance of market selection.

Understanding Statistical Significance

When an experiment is completed, the counterfactual prediction model uses the sales of the control markets during the experiment period to estimate the likely baseline sales in the test markets for the same period had media not been removed.

We then attempt to infer that the difference between observed sales during the experiment period and the baseline sales is the incremental contribution of that media to the business, as no other variables were altered during the experiment. This inference is then established with statistical significance.

A hypothesis test on the incremental contribution is employed to check its statistical significance. Essentially, we want to know the chances of the estimated incremental contributions occurring randomly, i.e., without the media intervention/treatment. 

When the chances of random occurrence are low as compared to our risk tolerance to making an error in using the results in decision-making, we conclude that the results are statistically significant, and we have a high confidence in attributing the incremental contribution to the media.  

Making Data-Driven Decisions from your Holdout Test

Ultimately, if a test doesn’t directly inform an investment or optimization decision, it amounts merely to an expensive science project.

Test results must be applied to an attribution framework (systematic, ongoing attribution of sales to media, enabling iROAS calculation and allocation optimization) in order to be valuable.

In practice, that means holdout test results should flow into three places:

• Your Media Mix Model (MMM): Causal holdout results serve as calibration anchors, replacing statistical guesswork with experimental truth. This transforms the MMM from a correlational model into a causal one.
• Your budget allocation framework: Compare each channel’s measured iROAS against your breakeven target. Channels above the threshold earn more investment; channels below get scrutinized or cut.
• Your always-on measurement dashboard: The corrected incremental contribution figure should replace the platform-reported number in ongoing performance tracking, giving your team a single, causally-grounded source of truth.

Check out this FAQ on optimizing ad spend for more on how to leverage test results to drive a broader incrementality capability.

Holdout testing has become a must-have capability for brands looking to prove the value of their marketing budget to the business and make the most out of limited resources. The goal isn’t a one-time test. It’s a testing rhythm that keeps your measurement grounded in real-world cause-and-effect, quarter after quarter, as algorithms, competitors, and audiences shift. For more information on how Measured enables this capability for hundreds of brands, set up a demo today with a Measured expert.