What is Incrementality Testing and How Do You Measure Incrementality?

Nick Stoltz, Expert in Cross-Channel Measurement Strategy and Adoption

Published 07/12/2022

What Is Incrementality Testing?

Media incrementality represents the true incremental contribution of a media channel, campaign, ad set, or tactic to business results.

For example, when a marketer runs an ad on Facebook, Facebook takes credit for 100% of the conversions that ad was in the path of (last-touch attribution). These results can be misleading because they include conversions that still would have happened without being exposed to the ad on Facebook. It is the additional (or incremental) conversions, beyond those that would have converted anyway, that represent the real contribution or impact of the ad. That’s incrementality - the conversions that were truly caused by the media in question. 


AB Test control group 
Why Should Brands Measure Incrementality?

Because they don’t have access to all the data outside their own environment, every digital ad platform uses the same flawed last-touch attribution method. Reports provided by ad platforms will never match up with site-side analytics reports or what’s observed in a brand’s sales data. Taking platform reporting at face value is a risky practice that can lead to bad decisions and lost revenue.   

Only incrementality can reveal which media investments contribute to business metrics and by how much. Measuring for incrementality identifies where to eliminate waste and surfaces opportunities to scale, expand and reallocate media spend for maximum growth.

As access to third-party data and user-level tracking end, the accuracy of platform reporting is eroding even further. Measurement that is independent of platform bias and future-proof against the whims of a constantly changing industry is critical for today’s marketers. Incrementality measurement can deliver where methods like last-touch and multi-touch attribution fail.

Want to dive into details?

See how incrementality testing can help your brand drive business outcomes.

How Do You Calculate Incrementality?

Incrementality is most effectively measured through proven test and control experiment methodology. By withholding the ad or treatment being tested from a statistically significant segment of the intended audience (the control group) marketers can determine the percentage of the target audience that still converts when they are not exposed to the ad. Subtracting that percentage from total conversions by the exposed audience (the test group) results in the actual incremental contribution, or incrementality percentage, of the media in question. 

Incrementality measurement can vary in complexity from a simple holdout test as described above to multivariate experiments so elaborate they require the expertise of a trained data scientist. But, when carefully designed and cleanly executed, controlled experiments can utilize data from an unlimited number of sources to reveal the incremental impact of just about anything marketers want to test – on any outcome that can be measured. 

In general, the control group should represent a minimum of 10 percent of the total test and control reach. The test group will receive the ad; the control group will not receive the ad. There are multiple ways to build a control audience. Some examples are:

  • Present a placebo ad or basic brand “anchor” ad to an exact replica or mirror audience of the exposed audience
  • Suppress a subset of the target audience from the ad exposure
  • Programmatic executions have counterfactual bid loss data for the target audience that did not win the media auction

The Incrementality Calculation Formula

In order to calculate incrementality, you calculate the conversion difference between the test group and the control group.

(Test Conversion Rate – Control Conversion Rate) / (Test Conversion Rate) = Incrementality

Incrementality Test Calculation Example

Your test group saw 1.5% conversion whereas your control group saw a 0.5% conversion. The control data suggests that, without any media exposure, you would have seen a 0.5% conversion. Keep in mind that conversion could mean leads, sales, profit or whatever metric that is important to your business.

So (1.5% – 0.5%) / 1.5% = 66.7% incrementality in conversions

What Channels Make Sense for Incrementality Testing? 

Advertising that generates a lot of impressions but not immediate action like a click still has brand value that ultimately leads to conversions or sales. Channels such as online display, video/YouTube, Facebook, TV, and direct mail all provide impressions, but are not always measured at the click or direct response level. That’s where incrementality testing comes in.

You can read more about the following channels here:

Incrementality Testing with Measured

Using experiments to test the incremental contribution of media to your business is the best way to make informed marketing decisions that fuel growth. 

Only Measured delivers ongoing, reliable insights based on scientifically sound experiment designs that incorporate your source of truth transaction data. Everything is automated – ingestion and management of data from hundreds of sources, experiment design and implementation, and continuous reporting for confident, agile decision-making.

Could your advertising deliver more business value? Check out this post for more on incrementality in marketing.