Hitchhiker’s Guide to Using Experimentation with MMM

Andrew Covato, Ad Tech, Measurement, Growth Consultant/Advisor

Published 04/17/2024

Modern marketers are deeply frustrated with the post-exposure performance metrics reported by ad platforms like Meta and Google, as these figures are biased, based in correlation, and can dangerously overstate performance, leading to an “ad tech tax” incurred by marketers; they pay for conversions that would have occurred anyway.  On the flip side, it could also be missing the value of upper-funnel campaigns, which further confounds things. 

In one example, Google Analytics was found underreporting Meta’s true contribution by 275% while the platform was overstating it! Why are these numbers so unreliable? In today’s privacy-centric world, user-level identifiers are vanishing, rendering click- and view-based measurements like platform post-click or multi-touch attribution simply infeasible. 

While marketing mix modeling (MMM) can provide some meaningful insights, it has significant limitations on its own. To successfully measure and understand their media effectiveness, brands need a combination of incrementality measurement, MMM, and traditional post-exposure attribution to get a more robust view of their performance.

Experimentation 

As we’ve said before, experimentation should be where every marketer begins their measurement journey. It remains the single most accurate method of measuring the causal impact of advertising on sales. Leaning into non-experimental methodologies (such as attribution) should not even be considered until you have started running experiments to understand the impact of your ad spend.

That said, experimentation is just one side of a triangulated measurement methodology that ensures success. 

Geo testing and user-level testing (like Measured’s “Known Audience Testing”) both have their respective limitations: they can be operationally difficult to keep “pure,” result in an “opportunity cost” of lost conversions from withholding ads, are limited in channel coverage, and cannot always provide granular insights. Additionally, some media channels, like influencer, affiliate, and audio, do not provide feasible opportunities for accurate experimentation, as they are not well suited for cohort-based measurement– either by geography or by first-party user data.

Experimentation is still absolutely crucial to solid measurement, and there is a way to overcome many of the limitations outlined above. Advertisers can run a series of tests that are more “macro” in nature (e.g., testing at the tactic or channel level) and then tie the results together with econometric modeling: MMM.

Marketing Mix Modeling

For the last few years, the industry has seen a resurgence of MMM–which was once the go-to measurement methodology in the pre-digital era.  Innovation in MMM continues, with Google recently announcing Meridian, its next-gen open-source MMM tool.

A marketing mix model compares the week-to-week variation in a channel’s spend and impressions to its week-to-week variation in sales. Unlike causal measurement from incrementality experimentation, MMM measures correlation. The more correlated the media is with sales, the stronger the resulting relationship. This leads the model to assign more credit to correlated channels than it does to less correlated channels. 

The benefit of MMM in a measurement solution is that it can measure any and all channels, providing comprehensive coverage. However, traditional, isolated MMM practices face two major challenges that make it just as dangerous to rely upon singularly as post-exposure attribution: 

  1. Most of your activity on most of your channels is likely correlated on a time-series basis (e.g., brands tend to increase spend across the board during seasonal periods).
  2. Demand often drives exposure to lower-funnel channels like brand search and retargeting (e.g., when more people are interested in your brand, brand search clicks will increase, regardless of whether or not they’re driving incremental sales).

Experimentation and MMM are both top-down approaches that work only to a certain level of granularity. To make data-driven business decisions, you also need measurement at the micro-levels. 

That leads us to the third leg of triangulation: platform-reported attribution. 

Attribution

Post-exposure attribution can be considered a generic term that encompasses any methodology where a conversion is “attributed” to an ad if exposure to that ad preceded it. , This could include things like “last-click” multi-touch attribution” (MTA) or platform paradigms like 7-day click/1-day view (“7 & 1”). 

Attribution alone is a terrible way to measure a channel's performance, but it can work with experimentation and MMM as a bottom-up source of data that gives you the granularity they lack. Using platform-reported attribution to monitor day-to-day performance changes on-platform at the campaign and sub-campaign level, providing the granular insights needed for near-real-time optimization.  This can work extremely well so long as attribution is used for relative comparisons only and the magnitude of the contribution is somehow tied back to incrementality.

Triangulation: The All-New Measured Platform

So, how do these three approaches work together?

In the context of triangulation, you would leverage experiment results as inputs–or priors–to calibrate an MMM model, resulting in accurate, week-to-week measurement based on ground truth: a causal reference point of performance. This would be effective for top-line measurement, so you would then apportion credit to more granular campaign elements using attribution information. 


The all-new Measured platform is leading the charge of triangulated measurement, offering brands a full-funnel, causal media measurement and optimization down to the campaign level.

Whether you’re looking for an experimentation partner or have more bespoke needs, you need high-quality experimentation to feed your models. Measured can help - our library of over 25,000 incrementality test results is interoperable with all internal and external MMM platforms. Reach out to an expert today.