What’s the difference between A/B testing and incrementality testing?

What is A/B testing?

A/B testing is an experimental approach where a sample of your audience is split into two or more groups to test different variations of a campaign. The goal is to understand which variation performs better based on a predefined metric, like the conversion rate. 

It’s commonly used to test creative designs, landing page setups, and messaging to optimize performance and conversion rates. This testing method is valuable for determining the most effective strategies and designs for engaging your audience and driving desired actions.

What is an incrementality test?

An incrementality test measures the true incremental value or lift associated with a particular marketing or advertising tactic. It aims to understand the percentage of vendor-reported conversions that directly resulted from the specific media tactic being tested rather than other underlying factors such as seasonal demand, general brand awareness, or the influence of outside media. 

At Measured, we use two main types of incrementality tests: geo-testing and known audience testing.

What is geo-testing in incrementality?

Geo-testing measures the incremental contribution of a marketing tactic to advertisers by manipulating media delivery in a small portion of the country.

In its simplest form, a geo-test pauses the delivery of specific media (e.g., Facebook) to the defined geo’s for a period of time, 30 days is common.  If the media is effective at driving advertiser sales, advertiser sales will decline. The magnitude of the decline is determined to be the value of the media. 

However, the process of measuring the impact is more sophisticated. Whereas an A/B test measures the relative difference in performance between two forms of media execution that were actually executed, an incrementality test must be compared to an estimate of what would have happened if the media was not held out.

This estimate of the control scenario may be calculated in different ways. Measured’s use of sophisticated data science techniques allows for a precise estimate of test market sales under a scenario where no media holdout existed. This prediction serves as the baseline upon which the actual change in sales is quantified.

Once quantified, the impact can be projected to a larger, nationally representative group of markets and compared to the media partner’s reported view of sales. The resulting ratio becomes the new estimate of vendor incrementality.

What is known audience testing in incrementality?

With a known audience test, we can deploy a test and control experiment at a deeper level of granularity and precision, having access to the audience deterministically.

Measured conducts this type of testing by first defining a customer or prospect audience segment that Measured or the advertiser determines. Measured commonly uses the RFM segment treatment.  From here, we create a randomly selected control group (i.e., holdout) for each audience segment and apply a unique and specific media treatment to each one. With the media treatments deployed, we compare each segment's exposed vs. control conversion rates to determine incrementality.  

These tests can be designed to measure a single tactic's effectiveness and deliver insights into the optimal multi-tactic contact strategy.

Are A/B and incrementality tests the same?

Not all incrementality tests are A/B tests, and not all A/B tests are incrementality tests. While both methodologies involve splitting sample audiences (or geographic locations) and testing different conditions, their objectives differ. 

A/B testing is a broader methodology that compares responses under different conditions for various purposes like landing page optimization, creative testing, or calculating incrementality. 

On the other hand, incrementality testing is more focused, aiming specifically to determine the direct impact of a particular media tactic on conversions.

Leverage the power of A/B and incrementality tests with Measured

Understanding the effects of your media strategies through A/B and incrementality testing is a key step in optimizing your marketing campaigns. With the Measured dashboard, you can easily analyze and understand the performance of your tests across all ad platforms, track media’s incremental performance using key metrics, and create crucial business reports to optimize your media strategies.

Schedule a demo with us today to learn how you can harness the power of incrementality testing with our software.