How Incrementality Testing Reveals Hidden Costs of Inaccurate Ad Measurement

Once upon a time, there was an ecommerce store that sold antique furniture. In 2022, as marketers were under increasing pressure to validate every dollar they spent, the brand decided to run incrementality testing with Measured to determine the effectiveness of their ads in creating awareness and driving sales.

Since purchasing furniture is rarely an impulsive buy and usually has a long decision cycle, vendors like Facebook couldn’t track purchases that occurred outside their small attribution windows. The campaigns may have driven purchases that were made weeks after the ad was seen, but were not captured in the reporting.

The results of the first geo-test that the brand conducted with Measured revealed that Meta was underreporting incremental lift of the brand’s Facebook campaigns by 180%. Based on that information, Measured analysts advised the brand to take what Meta reported and multiply it by 1.8 to get the true incremental results.

To confirm their investments were still working, the brand retested the incrementality of Facebook ads a year later. It’s always smart for a business to test periodically because business conditions and industry changes can impact the performance of your media. The 2023 test found that the incrementality of their Facebook ads had dropped by 83% since the test 9 months prior. The media buyer was anxious and sleepless, wondering how to explain the large drop in performance.

When a drastic shift like this happens, Measured analysts first go through diagnostics and quality control scrubbing exercises to make sure that there's no artificial influences or contamination with experiment design or execution. Once we've confirmed the test was clean, how do we explain these surprising results?

To get to the bottom of the significant drop in performance for the furniture brand, we recommended that they look at the campaigns tested in 2022 and 2023 to see what may have changed in regards to campaign set up. After reviewing, the team discovered that the same prospecting campaigns in 2022 and 2023 had different audience targeting selections utilized. In the 2022 campaign, the audience targeting was very broad-focused, including look alike audiences, while the 2023 campaign targeting was focused on customer lists of prior customers and Facebook IDs and names of people who had visited the website, but hadn't made a purchase.

The targeting strategy had shifted from a prospecting/customer acquisition approach to instead remarketing to users who were already aware of the brand. Because the 2023 ads were seen by previous customers and people who had already engaged with the brand, many of their purchases likely would have happened anyway. Because Meta cannot distinguish which conversions would have happened regardless of their ads, it takes credit for all of them. By running incrementality tests that are independent of platform attribution, Measured was able to identify which conversions were actually driven by the ads.

2022 results showed that Meta was underreporting on awareness and prospecting campaigns. 2023 tests showed that Meta was overreporting on lower-funnel retargeting campaigns. These skewed measurement results are an ongoing issue with platform reporting. As a result, when brands rely on those reports for spend allocation, we often find that they are overinvested in retargeting campaigns and underinvested in upper-funnel tactics.

In the dynamic and fiercely competitive environment of media buying, there is constant pressure to tweak campaigns and improve performance, but even small details can have a huge impact on the effectiveness of campaigns. Meta indicated the campaigns were performing better than ever, but Measured proved otherwise.

It’s important to understand that incrementality is not static, and it actually will change based on business conditions, presence of competition in the market, changes in your target strategies, and numerous reasons outside your control. What we learned from this brand’s experience is that platform reporting is imperfect at measuring the true impact of advertising for making critical spend allocation decisions. More often than not it will lead you to wasting dollars and overlooking opportunities to profitably scale and grow.

That’s why Measured gives customers access to a database of thousands of incrementality tests we’ve already run for 100s of brands on 65+ channels and tactics. You can use the Measured platform to view weekly incrementality benchmarks in spend, costs, and ROAS that may be a result of industry-wide trends. With Media Plan Optimizer you can compare multiple spend scenarios in minutes and continuously optimize your full portfolio for incrementality during the periods between running in-market tests.

Are you sure your ad spend is allocated to the channels and campaigns that are going to deliver the best ROAS? Only independent incrementality testing using geo-matched market experiments can reveal the truth.

To learn more about how Measured can reveal the truth in advertising performance, click  here