How Do You Calculate Incremental Sales Driven by a Media Tactic?

Terence Einhorn, Sr. Solutions Consultant in Sales

Published 02/28/2021

Understanding Incremental Sales

What are Incremental Sales?

“Incremental sales” in marketing refers to the causal lift in sales driven by a given marketing activity for a given period of time. Not all sales that occur after a marketing exposure are deemed to be incremental, as often these sales “would have happened anyway” with or without that marketing exposure. 

For example, let’s say a customer is already intending to purchase a certain product after visiting a brand’s website. The brand may subsequently serve a retargeting advertisement in the days following, but this ad did not incrementally drive the purchase as the customer was already planning on purchasing anyway. 

The Importance of Measuring Incremental Sales

There can be a significant difference between the performance one may see on an ad platform or in their site analytics data and the true Incremental performance of that channel. This can lead to a costly misallocation of valuable marketing resources that hold the business back and leave money on the table.

For example, if one were to believe that every conversion happening after a brand search click were incremental (i.e., taking last-touch data at face value), one might consider brand search a wildly profitable marketing tactic.

But as we know, most customers use brand search as a navigational tactic within their browser once they’re already interested in buying a product. This means they will often click an organic (SEO) link for the same brand or product in the absence of a paid link and convert anyway.

In other words, it’s likely that many of the sales being claimed by brand search in a last-touch environment are not incremental to the business.

Methods for Measuring Incremental Sales

The only way to know the true incremental impact of a media tactic is through experimentation (incrementality testing). In academic terms, the clearest definition for the incrementality of a channel is to ask, “How many sales would I lose if I took this channel away?”

That is exactly what incrementality experiments aim to do: split an audience into a test (un-xposed) and control (exposed) group, remove a given media channel from the test group, and observe their conversion behavior compared to the control group.

It is important to note that true incrementality experimentation uses 1st-party transaction data to produce results, as opposed to on-platform conversion tracking or last touch/site analytics tracking.

This is because ad platforms and site analytics tools (such as Google Analytics or Adobe) have limited ability to track conversions back to a user after a given time period, and often none at all if they haven’t physically served an ad to the user.

Even if you’re using a PSA technique for serving a “dummy” ad to a supposed holdout group, this tracking blind spot will not be equivalent across test and control groups, leading to inaccurate results.

Methods for Creating Test vs. Control Groups

There are two main ways to split an audience into representative groups: a “Geo-Split” and an “Audience-Split.”

Geo-Split

A geo-split finds specific markets that are representative of a larger region to use as a test group for experimentation. This could be states, DMAs, or even postcodes within a country or region. 

This process can be as simple as a random 50/50 split or can leverage advanced data science techniques to maximize statistical accuracy while minimizing business risk (more on that later).

The advantage of Geo-Splits is that as long as the given media platform can geo-target its Media delivery, one does not need to rely on specific user-targeting to execute the test.

The main disadvantage is that it isn’t viable for businesses that don’t have broad geographical coverage in terms of distribution of sales (e.g., an insurance company that only operates in two states).

Audience-Split

An Audience-Split carves a specific user list into representative test vs. control groups.

The main advantage of an audience-split is that it is more precise than a geo-split and less prone to “contamination” (e.g., a user in the unexposed group accidentally receiving an ad).

The obvious disadvantage of audience-splitting is that a media platform needs to have an addressable user list in order to execute. This means most prospecting or broad-targeting channels are unable to deploy this technique.

How Do I Calculate Incremental Sales?

Geo-Split

As discussed above, both of these approaches require 1st party transaction data to be used in order to calculate incrementality results.

Geo-experimentation only requires aggregated order data (not user-level) that is broken out by the geography in question (e.g., by state, DMA, etc.).

To calculate results, observed sales in the test markets (what actually happened when we removed Media) are compared with a “counterfactual” prediction (what we would have expected to happen had we not removed media) in the control markets. 

The difference between these two sales figures is said to be the contribution, or incremental sales, driven by that channel. 

The counterfactual prediction is generally calculated by the same model, or set of simulations, that was used to select the representative test markets when designing the experiment. The conversion activity within the control markets during the experimental period will be the main driver of this result.

Alternatively, a counterfactual could simply be the observed sales in the control markets if deploying a random 50/50 geo-split, though this technique is not recommended.

Audience-Split

For audience experimentation, one needs to “match back” user-level orders from 1st Party Transaction data to the user list on the platform in question. (This is done using some sort of common customer identifier, be it an e-mail address, physical address, phone number, or combination of the three).

The conversion rate of the test group is then compared with the conversion rate of the control group to calculate the “incrementality” of the sales coming from the exposed group during the test period.

In other words, what portion of the sales coming from the exposed group during this period was Incrementally driven by the media, as opposed to those that happened anyway?

E.g. : ((Exposed CR - Control CR)/ Exposed CR) * Exposed Sales = Incremental Sales

What are the Challenges in Measuring Incremental Sales?

While the above methods sound simple enough at a high level, there are many challenges with incrementality experimentation that often prevent brands from fully developing this capability.

For one, experimentation can be highly resource-intensive, both in terms of time and financial commitment. For example, dedicated data science resources are required for accurate and reliable market selection and counterfactual prediction for geo-experimentation, while dedicated data management resources are required for the match-back process in audience split testing.

Additionally, incrementality experimentation typically entails some sort of business cost to execute. For example, “holding-out” media from a given set of markets may forfeit hypothetical sales in those markets during the test period. This impact is far greater with a random 50/50 split than with more advanced data science techniques.

Most importantly, it’s often difficult for brands to deploy experimentation in a way that’s consistent, reliable, and seamlessly feeds back to their reporting and optimization framework. This makes it exceedingly difficult for organizations to adopt experimentation as regular practice without a dedicated 3rd party provider.

Common Pitfalls and How to Avoid Them

For brands developing their own incrementality experimentation capability, there are a few common traps to avoid.

The first is that they try to “do too much at once” in an effort to get the most bang for their buck in terms of time and financial resources.

This often leads to highly complicated or overly granular treatment groups (e.g., testing five different combinations of media tactics simultaneously), yielding weak or unreliable reads for each separate treatment. Instead of saving time and money by consolidating multiple experiments into one, this approach often has the opposite effect.

The other common pitfall is that brands will invest in the design and execution of experimentation but do not have a consistent method of applying these results to a reporting, attribution, and optimization framework.

This is an issue because experimentation is episodic in nature and can only be done at a certain level of granularity. In order to make ongoing budget allocation optimizations, you need a formal system that consistently applies experimentation results to a daily, granular attribution framework to enable more tactic decision-making.

Future Trends in Incrementality Measurement

With recent trends in privacy regulation and cookie deprecation, it is becoming increasingly difficult to deploy incremental experimentation that leverages user-level tracking or targeting.

This means that techniques such as audience splits, whether executed by first parties or by vendor platforms, will become less and less feasible.

As such, geo experimentation has become the gold standard for incrementality measurement moving forward and has been endorsed by leading vendor platforms (e.g., Google and Meta).

The increased use of geo experimentation as a basis for incrementality measurement necessitates innovation in the areas of experimental design, test execution, attribution, and optimization to alleviate some of the common challenges mentioned above.

Learn more about the Measured incrementality platform.

Want to Learn More About
Incrementality and Attribution?