The Basics of Incrementality Testing in 2023

Our most recent ebooks and guides break down the steps of geo-testing, the shortcomings of methods such as multi-touch attribution (MTA) and media mix modeling (MMM) alone, and the best ways to implement incrementality. 

So, we decided to take a step back and outline why incrementality is a hot topic in 2023, plus the most important and widely-used terms every marketer should be familiar with when it comes to incremental measurement.

Overview of Incrementality Testing

At its simplest, incrementality is the measure of the true effectiveness of an ad and the value it drives for your brand. By learning the real, causal contribution of an ad campaign, tactic, or media channel, your brand gains indispensable insights that help you allocate your media spend in the most effective way possible. Only incremental measurement reveals what would happen if you stopped advertising on a particular channel, how increasing or decreasing your budget would affect your business results, and what the key contributors are that are driving growth.

Why Is Incrementality Testing Suddenly a Required Capability in 2023?

iOS 17 and Link Tracking Protection 

One of the largest advantages of incrementality testing is that it is built on cohort-level data and unaffected by the changing restrictions on user-level tracking. With the new release of iOS 17, for instance, it continues to become more and more difficult for marketers to rely on tracking data to gauge the effectiveness of their campaigns due to the introduction of a new privacy feature: Link Tracking Protection. 

Link tracking protection greatly hinders any marketers relying on URL parameters to track the customer journey across different platforms and channels to measure their ad campaign effectiveness. As announced by Apple in June,

“Some websites add extra information to their URLs in order to track users across other websites. Now this information will be removed from the links users share in Messages and Mail, and the links will still work as expected. This information will also be removed from links in Safari Private Browsing.”

This means tracking the customer journey just became even more difficult - if you’re still relying on this method of measurement.

The short answer, then, is that incrementality testing is a requirement both now and looking forward. Unaffected by privacy features such as iOS 17, it’s a necessity for every marketer accountable for driving business performance to have a view of media's incremental contribution.

Top Incremental Measurement Terms To Know

To clarify the definitions of incremental measurement terms, let's explore two hypothetical incrementality experiments conducted by a direct-to-consumer retailer to measure the effect of a Google non-brand search as a marketing tactic to drive conversions:

1. Average Effect Experiment: In this experiment, we aim to measure the average effect of existing Google non-brand search investments on conversions. To do this, we designate a set of markets as the "holdout test group." Over a span of several weeks, we withhold any investments in Google non-brand search within this group. Afterward, we compare the observed conversions during this period to the expected conversions based on existing investment levels. Any differences in conversions are attributed to the impact of Google non-brand search.
2. Marginal Effect Experiment: In contrast, the second experiment aims to measure the marginal effect of doubling the investment in Google non-brand search. We identify a group of markets as the "scale test group," and, for several weeks, we double the investments in Google non-brand search within this group. Once again, we compare the observed conversions during this period to the expected conversions based on the original investment levels. Any difference in conversions is attributed to the doubling of Google non-brand search.

In both experiments, we assume that all other factors influencing conversions remain constant within both the test and control groups, maintaining a "business as usual" scenario.

In control markets, those unaffected by the experiment also serve as the reference point to compute expected conversions. They maintain existing levels of investment in the tactic being measured.

The baseline in both experiments reflects the typical investments in all factors influencing conversions.

Here are some key terms related to incremental measurement:

• Test Group: This refers to the markets subject to the experiment where the investment in the tactic being measured deviates from business as usual. It can be the holdout test group for average effect measurement or the scale/reduce test group for marginal effect measurement.
• Control Group: The control group comprises markets used as a reference point for calculating expected conversions under existing or business-as-usual levels of investment in the tactic being measured.
• Incremental Conversions: This represents the number of conversions directly influenced by the presence of the measured tactic. It's calculated as the difference between observed conversions during the experiment and the expected conversions under existing investment levels.
• Confidence Interval: When considering incremental conversions as an average, a confidence interval is employed to account for a level of uncertainty. For instance, a 90% confidence interval provides a range within which the incremental conversions from a repeated experiment are likely to fall with a 90% probability.
• ROAS (Return on Ad Spend): ROAS is calculated by the advertising platform as the ratio of observed revenue to the spend on the tactic.
• Incrementality %: This figure represents the fraction of overall conversions reported by the advertising platform that is genuinely driven by the tactic.
• Incremental Return on Ad Spend (iROAS): iROAS is an adjusted ROAS that unveils the causal, incremental impact of the tactic. It's calculated by applying the incrementality percentage to the platform's ROAS.
• % Conversions (i): This indicates the proportion of overall conversions directly influenced by the tactic, calculated as incremental conversions divided by total conversions.
• % Sales (i): This reveals the share of overall revenue directly attributed to the tactic, calculated as incremental sales divided by total revenue.

For more on incrementality testing and how it can help your brand grow, speak to a Measured team member today.