Why Incrementality-based Attribution is Better Than MTA or MMM for Optimizing ROAS – A Real World Example

Marketing attribution is a complex, fast moving topic, with a lot of noise, and a lot of companies claiming to have the ideal solution for you.

You’ve likely heard complicated jargon and contradictory statements related to the most common media measurement methods like Multi-Touch Attribution (MTA), Marketing Mix Modeling (MMM), and incrementality measurement (which is what we do at Measured). Each method uses a different approach to revealing insights, meaning some are better than others depending on what you are trying to learn.

When I’m trying to understand a complex domain like this, it helps to relate it back to a simplified real world example, and reason up from there. For an element of nostalgia, for this example, we’re going to go back to a time before most marketing took place online.
Imagine you’re the owner of three coffee shops in similar markets. You have a loyal existing customer base, but you need more new customers to grow the business.

You decide to run two marketing campaigns for customer acquisition:

1. Run an advertisement in local newspapers
2. Pay people to hand out fliers on busy street corners

In both the fliers and the newspaper ad you offer a free cookie. All they need to do is clip out the coupon in the newspaper, or bring in the flier to redeem the offer and get their free cookie. You spend $10k on each of these marketing campaigns, but you need to know how each one performs so you can reallocate the budget to maximize results in the next campaign.

How do you do that?

Multi-Touch Attribution (MTA)

As soon as the fliers hit the streets you start getting hundreds of new customers coming in at each store. Things are going well.

Then the newspaper ad is published and demand skyrockets! You see a huge spike in new customers, and you can measure which campaign is bringing in the most business by counting the coupons coming in. However, in the next few days you notice something is off.

Stores are reporting that new customers are coming in with both the newspaper ad clipping and a flier coupon as well. They’re asking for two free cookies each!

Not so big a deal, you say, you’re happy to allow for multiple touchpoints with potential customers. You had assumed you could assign credit neatly to a single campaign for each new customer, but whenever a customer redeems both offers, you will just divide credit equally between the two campaigns.

By splitting credit for a sale between multiple touch points, you’ve actually just set up a very simple Multi-Touch Attribution (MTA) model. Obviously, things can get much more complicated than that when more channels are in the mix, but even-split across all touchpoints involved is a common approach, and considered an upgrade from the default ‘last touch’ model used by most platforms.

When running a campaign that is meant to bring in new customers (prospecting) cost per acquisition (CPA) is a common metric for measuring performance. Here are your results from your first month of advertising, across all three coffee shop locations:

Overall both campaigns seem to be successful, and you don’t want to be hasty and turn one off just yet. You decide to keep them both running for another month.

Except, in month two you decide to double down on fliers, as they had a slightly better CPA in the first month. You take $2,500 out of the newspaper budget and reallocate it to fliers.

In month 2 you run into a bigger complication. One of your managers catches someone handing out flyers right outside the entrance to one of your locations. They are promptly moved to a location away from the store, but not before they handed out 500 free cookies to people who were already coming into the shop.

That means at least those 500 fliers, and maybe more, were not incremental: those customers were planning to purchase anyway, whether they got the flier or not. In addition, when your baristas chat with new customers, many admit they remember seeing something in the newspaper, but forgot to bring in the coupon.

You realize this is getting too complicated. There are too many other factors that impact whether someone made a purchase or not, so just counting coupons and assigning credit won’t provide accurate insights.

Marketing Mix Modeling (MMM)

To account for the impact of multiple variables you decide to look at historical data to see how much overall sales changed before and after you started advertising. In this instance, you only have a few months' data, but you might uncover some useful correlations.

Here’s what the data looks like so far for all three locations:

Even without a statistical model you can see that sales jumped in June when you started advertising, from 40,000 up to 55,000. Overall sales dipped slightly the month after to 52,000, which is when you cut newspaper spend.

Fast forward 3 months, when you have even more data, you might be able to build a simple linear regression model (a statistical technique for finding correlations in the data), which would do much the same thing as you just did. It would look for patterns related to times you spent more or less money on a channel, and see how they correlate with spikes and dips in topline sales.

This technique is called marketing mix modeling (MMM), and it’s one of the primary ways marketers attribute hard to measure channels like newspaper ads, but also influencer marketing, and TV.

However, we saw some of the limitations here: three months was not enough data, you’d need at least six months or even years worth to account for things like seasonality or changes in consumer trends. MMM also requires an expert in statistics which puts it out of reach for most businesses.

What other options do you have to find the truth about how much your ad campaigns are contributing to sales?

Incrementality Testing

In month three, you decide to try a little experiment. You test a different campaign approach at each of your three locations.

• Location 1: No advertising
• Location 2: Newspaper only
• Location 3: Fliers only

You cut the budget so as to not saturate any one area, and spend $3.3k on newspaper ads in Location 2, as well as $3.3k on people handing out fliers in Location 3.

By separating out your campaigns by location, you can run a rudimentary split-test, with Location 1 as your control group, to check how each method performs in isolation.

Here are the results:

As expected, both advertising strategies increased sales over the baseline. Sales were up by 24% in Location 2 where you advertised in the newspaper, and just over 18% in Location 3 where you ran the fliers.

If we know baseline sales would have been the same in each location without advertising (because they have statistically similar market/audience), we can subtract to find the relative performance of each campaign.

Baseline sales: 12,500 (Location 1)

Now that you have a better understanding of the true performance of each channel, you can make strategic decisions about where to allocate budget.

For example, since newspaper ads performed better, you could test what would happen if you increase your spend in that channel (called scale testing) to find the saturation point (at what spend level performance begins to suffer). Now that you can isolate the performance of different campaigns and channels and more confidently measure effectiveness, you could also explore new channels by running similar tests.

By continuously testing and reallocating budget to the best performing strategies, you can make significant improvements in efficiency and reinvest all the money you save in expanding to new locations.

The solution here was a primitive form of incrementality testing, or RCT (Randomized Controlled Testing, which is the gold standard for establishing causality: proving that one thing causes another, and by how much.

In reality, Incrementality testing tends to be much more sophisticated, and the right techniques need to be used to ensure you’re running a scientifically sound experiment.

Incrementality with Measured

The Measured Incrementality platform enables our clients to run incrementality experiments at the channel, tactic, and campaign level and optimize media using reliable results that are validated by their own sales data. We also have a database of thousands of incrementality tests we’ve already run for 100s of brands, so we can reliably determine which channels and tactics are likely to be more incremental for your business, even before we set up a single experiment.

Unlike MTA and MMM, which are both modeled on correlation, incrementality actually determines what results were caused by the media being measured. It’s the only way to know for certain whether your advertising caused incremental sales.

In the current marketing environment, as increasing data privacy restrictions and regulations limit access to user level data and tracking, MTA solutions are operating at a serious deficit. Connecting all the user touch points on the path to purchase is an impossible task and marketers should be wary about trusting data from any MTA vendor claiming they can do so.

For established brands with years of data, MMM can deliver meaningful analysis at a high level. It’s especially good for understanding how marketing and nonmarketing variables impact overall sales as well as forecasting and budgeting for different potential long term scenarios. But, when marketers need more granularity and ongoing insights at a tactical level, MMM is unable to deliver. It can’t offer insights into campaign-level performance and decision-making.

Only incrementality can provide accurate insights for ongoing tactical decisions about spend allocation and optimization. Used together, incrementality can actually complement MMM’s long-term strategic and planning strengths. Incrementality testing and data can help diminish the correlation/causality issues related to MMM by validating results and training the MMM model.

The rapidly evolving world of marketing attribution can be confusing for marketers at any level of experience, but it helps to have a clear understanding of what each approach can and cannot provide.

If you are still using MTA insights to optimize your spend, there is a very good chance you are missing opportunities to cut waste and drive more sales. With enough data and experienced data scientists, MMM can be a useful strategic planning tool, but the most effective way to maximize sales caused by advertising is to start optimizing your media for incrementality.

If you would like to learn more, book a demo today!