How Do I Measure Incrementality in Display Advertising?

Terence Einhorn, Sr. Solutions Consultant in Sales

Published 04/13/2022

Key Metrics for Evaluating Display Advertising Effectiveness

Spend: the amount of money spent on a particular ad or campaign. Spend can include non-working costs (such as agency fees or creative production - more commonly referred to as “Cost”) or be inclusive only of “working” ad spend.

Cost per Impression: CPM represents the average spend required to generate 1000 impressions for a given campaign on average. This is calculated as: Spend/Impressions*1000

Impressions: represents the number of times an ad is served across the course of the campaign. This number does not take into account whether a user actually “viewed” the ad or not.

Viewability: refers to the number of impressions that actually resulted in a user “viewing” the ad. There are a number of ways to define “viewing,” but a general standard is an ad being 50% visible on screen for a minimum of one second (display ads) or two seconds (video ads). This can be calculated as a percentage: Views/Impressions

Reach: represents the total number of users that were served an ad during the course of a campaign. This number is different from impressions because a given user may receive an ad on more than one occasion.

Frequency: the number of times a given user was served the ad over the course of the campaign. Average frequency is calculated as: Impressions/Reach

Clicks: the number of times users clicked on your ad over the campaign period.

Cost per Click: represented as Spend/Clicks, Cost per Click is an important metric in the PPC (pay per click) advertising model. In this instance, vendors charge a set fee for every time a user clicks your ad. This fee, multiplied by the total number of clicks, ends up representing the spend of the campaign.

Conversion Rate: the rate at which users convert after being served, viewing, or clicking an ad. Conversion rates can be calculated based on either of these metrics depending on the use case. You can also convert this metric to Revenue per User, as opposed to Conversions per User, to incorporate the amount of product being purchased by a user on average.

Incrementality: represents the portion of those conversions that were truly driven by the ad exposure (as opposed to those that would have occurred regardless). This is a critical metric in measuring performance as it’s the only metric that vendors cannot calculate directly from campaign data.

ROI(i): Incremental Return on Investment is the most important metric as it quantifies the overall value of a display campaign to the business. It represents how much Incremental Revenue you drove to the business for every dollar you spent, which is the main KPI (Key Performance Indicator) for all media channels.

In cases where purchase revenue is not relevant (e.g., Subscription model, where revenue is realized over a customer lifetime), Cost per Incremental Acquisition, or CAC(i), is the equivalent metric.

Note: Oftentimes, ROI is used interchangeably with ROAS (Return on Ad Spend). Technically speaking, ROAS includes ONLY working media in its “Spend,” while ROI typically also includes non-working fees and “nets out” spend from the numerator, meaning it has a breakeven of $0 as opposed to $1.

How Do I Measure Display Advertising?

Vendor-provided campaign data can help inform performance but generally gives an incomplete picture as vendors can only see user activity on their own platform. In other words, they don’t know what other media or influences in the market may be simultaneously impacting a user's purchase behavior.

This is why Incrementality Measurement is critical: it tells you the true causal impact of a display program by revealing how many of those conversions were likely to have happened regardless of display exposure.

Strategies for Incrementality Measurement

There are three ways to measure Incrementality for a display campaign:

  • PSA Testing
  • Known Audience Experimentation
  • Geo Experimentation

Let’s dive into the strengths and weaknesses of each of these.

PSA Testing

PSA Testing entails carving out a section of a selected audience and serving them a “PSA” Ad (a piece of content that has nothing to do with your business) such that you can track their conversion behavior compared to the audience receiving your real ad.

This is particularly useful for prospecting audiences whose conversion behavior you would not be able to track unless you were serving them some form of media.

Advantages of PSA Testing

  • The first advantage of PSA measurement is that it is often provided by the vendors themselves, usually at little extra cost and minimal extra work for the advertiser.
  • PSA testing is relatively easy to execute and interpret on most display platforms.

Disadvantages of PSA Testing

  • Any incrementality technique reliant on user conversion tracking has a fundamental flaw: it becomes increasingly difficult to link or “match back” user purchases to ad exposure the more time passes between the two.
      • This is especially challenging for PSA testing, where tracked conversion volume for the “Control” group is going to be very minimal (remember, this audience never saw an ad related to your brand). This tends to yield inflated incrementality metrics as the control conversion rate is always close to zero.
  • Any technique that measures incrementality using on-platform conversions cannot identify instances of a vendor under-reporting conversions. There are many cases where upper-funnel display programs are driving more conversions than the vendors themselves are able to track. This blind spot is only solved through testing grounded in a brand’s 1st-party transaction data (see Geo Testing below).

Known Audience Testing (Holdout Testing)

Sometimes used synonymously with “holdout testing,” a known audience test can be used when targeting an addressable audience (i.e., you already know who they are and can track their conversions without serving them an ad).

This entails selecting a portion of the audience and withholding them from media exposure, and subsequently measuring their conversion behavior against the rest of the audience to gauge incrementality.

Advantages of Known Audience Testing

  • Known audience testing can be run in a more granular fashion than other techniques, meaning you can sacrifice fewer potential customers to get the answers you’re looking for.
  • You can also break known audience testing into multiple treatment cells, perhaps to test different creative or messaging approaches against your business-as-usual campaigns.

Disadvantages of Known Audience Testing

  • The obvious disadvantage is that it can only be run on an addressable audience.
      • Some display vendors will be able to run such a test on a large batch of their users but will be reliant on platform-tracked conversions for results, which yields some of the same issues as PSA testing.
  • In cases where a brand has a true known audience list (e.g., a CRM file) against which display can be served, results can be measured with their 1st party transaction data using a purchase “match back,” which is a more ideal technique.

Geo Experimentation (Geo Testing)

Geo-experimentation carves out a selection of test markets (i.e., states and DMAs) within a broader region and withholds advertising for a period to observe the impact on sales compared with the rest of the region.

Advantages

  • The key advantage of geo-experimentation is that measurement is based on a brand's 1st-party transaction data. In other words, it is not reliant on user tracking to produce results.
  • Geo-experimentation is also able to identify when vendors are underreporting their own performance. This can happen with more upper-funnel type display media that take a longer time to drive a return in-market.

Disadvantages

  • The main disadvantage of geo-experimentation is that it can be costly and laborious to perform oneself. This is why it’s generally recommended to use an experienced partner to drive an ongoing testing practice.

How Do I Optimize Display Advertising?

Once incrementality is revealed via experimentation, we can then calculate ROI(i) (Incremental return on Investment).

ROI(i) is the main KPI for any advertising channel. A KPI represents the most holistic and relevant evaluation of a campaign’s performance. It’s also the metric against which you can optimize your media budget.

In other words, your budgetary and strategic decisions should aim to either:

  1. Maximize ROI(i) based on a fixed budget
  2. Maximize spend while maintaining a “profitable” break-even ROI(i) (in cases where budget is fluid)

Note: Breakeven ROI is usually calculated as 1/Gross Margin. For example, if your gross margin is 50%, your “break-even” ROI(i) would be $2.00. 

Many brands will choose to add more costs to get to a “fully baked” net margin number by including operating costs, non-working media, product margin, etc., though this is quite rare.

Understanding Display Advertising ROI

While ROI(i) is the most important metric, all other metrics contribute to this final number, and they all tell you something different about how an ad performed.

In general, there are three metrics that are most important to understanding ROI:

  • Cost Per Impression (CPM)
  • Conversion Rate (CR)
  • Incrementality

For example, if you see your ROI(i) decreased from last month, it could be because:

  • CPM increased (the ads became more expensive)
  • Conversion Rate decreased (e.g., your new creative is less compelling, or perhaps you’ve entered a less seasonal period for demand)
  • Incrementality decreased (you are targeting a higher-propensity audience, meaning more of them are likely to convert regardless of advertising exposure)

It’s important to monitor all of these metrics individually so you know which levers to pull or how to adjust strategy accordingly in order to improve ROI(i).

It should also be mentioned that these factors are not truly independent of one another.

For example, as you scale spend upwards, you will likely experience “diminishing returns” or decreasing ROI(i) on your investment.

This is not only because CR tends to decrease as you scale but also because CPMs tend to increase. As such, these metrics tend to move indirectly proportionally.

Similarly, as CR increases, incrementality tends to decrease (exposing an audience with a higher propensity to purchase will yield more conversions, but it also means more of these conversions were likely to happen regardless).

As such, keeping a close eye on various platform metrics, in addition to routinely testing incrementality, is key to ensuring your brand gets the most out of your display program and drives true impact for your business.