The Rise of AI Campaigns: Part 3
Evaluating the Incremental Impact of AI Campaigns
In previous posts, we’ve defined what we mean by ‘AI campaigns’ and how marketers can set them up to be incremental. Today, we want to talk about what you can do as a marketer to quantify how effective these AI campaigns are.
Let’s dive right into it: The promise of campaigns like ASC (Meta’s Advantage+) and Google’s PMAX is that they allow you to leverage the goldmine of data within these platforms to identify people you would not think to target otherwise.
These campaigns exist to find the “needle in the haystack” and serve as a tool to help you leverage all available signals to most optimally allocate your budget.
However, many advertisers have not set them up this way - which means that when they’re left unchecked, these campaigns will want to target those with high intent (past purchasers, site visitors, cart abandoners, etc.). This is problematic for several reasons:
- These are obvious audiences, and no advertiser needs an algorithm to target them
- A significant portion of your media budgets in separate campaigns is probably already allocated to campaigns specifically targeting them
- They are less incremental than other audiences
How you ultimately approach campaigns like PMAX and ASC boils down to a single question: Are you comfortable giving up control to the platforms to automate decisions otherwise made by yourself or your agency?
Many of our customers choose to let AI and non-AI campaigns coexist, and there’s nothing wrong with that. However, any brand considering a similar stance (no matter how much of your budget you decide to allocate to AI campaigns) should realize that testing is critical to understanding both the overall impact and how these campaigns interact with and affect each other.
How to Evaluate AI Campaigns
There are different ways to approach evaluating these campaigns, which align with common ways of measuring media more broadly:
- In-platform performance
- Media Mix Modeling (MMM)
- Lift tests (i.e., geo tests)
Pros and Cons of Each Evaluation Technique
These campaigns are trained to optimize for conversion rate, and, unfortunately, incrementality and conversion rate are inversely related. This means that taking performance at face value means you’re giving platforms credit for some conversions that would have occurred anyway.
In other words, using in-platform metrics like ROAS to evaluate performance is a surefire way to waste your media budget. Though it may be tempting due to simplicity, it’s also the least accurate approach.
Media Mix Modeling
MMM looks for a correlation between spend and sales, and unless you’re specifically accounting for things like multicollinearity or endogeneity, there can be significant bias with the model mistaking correlation with causation.
Additionally, MMM provides a more aggregated view of performance, typically providing you with average performance over a multi-year period. This is helpful for understanding the strategic value of channels overall - but not very useful when trying to understand the incrementality of specific campaigns that run with different settings at different times of the year.
Given these nuances, MMM alone (i.e., not augmented with causal experimental results) is not well-suited to provide the accuracy needed to evaluate performance or optimize investment level on these types of campaigns.
Geo tests enable you to understand causal relationships at very granular levels both within campaigns and within specific times of the year and are key to evaluating these types of tactics. However, there are specific challenges to be aware of when attempting to test PMAX and ASC.
A geo test works by turning off or scaling up media in a target market and observing the sales impact. However, given the complex algorithms and “black box” nature of these campaigns and audience overlaps with non-AI campaigns, testing can lead to unexpected insights, often revealing previously unknowable things the platforms don’t directly report.
So, how should geo-testing evolve as a result?
First and foremost, it’s important to understand how the presence of AI campaigns impacts testing both AI and non-AI tactics.
Step 1: Understand exactly how your campaigns are set up in order to better understand what might happen when you test.
As discussed in the earlier blog posts, these campaigns have certain levers advertisers can pull to influence the type of inventory. Examples include:
- Brand keyword exclusions (PMAX)
- Prioritizing bidding for new customers (PMAX)
- Defining “existing customer” audiences (ASC)
- The percentage of budget allocated to these existing customer audiences (ASC)
Understanding how the campaign is set up is crucial to predict what might happen when you test it. Your goal is to understand whether the way you’ve set up this campaign means it will behave more like an upper funnel, lower funnel, or full funnel campaign and then use that knowledge to understand how it may compete with other campaigns you’re running.
Traditional, “non-AI” campaigns are more straightforward.
Consider the spend levels across typical campaigns like Facebook Prospecting/Retargeting or Brand Search/Shopping. Ask yourself:
- How do the budget levels on these campaigns compare to your AI campaigns?
- How have you set up these campaigns (i.e., fixed budgets or flexible budgets spending against ROAS or CPO targets)?
What’s most important is to try to assess what percentage of your non-AI campaign budget will be targeting audiences similar to your AI campaigns. The larger the overlap, the more similar these campaigns are to each other, requiring potential modifications to test design.
Step 2: Consider new test designs
Given the potential overlapping nature of these campaigns, you may need to consider test designs you may not have otherwise.
For example, if your campaign setup on an AI campaign is “full-funnel,” and your AI campaign budget is roughly equivalent to your non-AI campaign budgets, in reality, your different campaigns are behaving much more like a single tactic than individual ones.
As a result, it becomes more important to test the entire channel overall and use the findings to optimize your campaign setup.
(For instance, are you curious about what the overall impact is when traditional (non-AI) Facebook Prospecting campaigns interact with (AI) Adavantage+ campaigns? To peek beneath the surface, combine them together into a single test to uncover the incremental impact of Facebook as a channel first, then use the results to optimize the campaign setup and potentially retest.)
Doing this will allow you to validate the overall effectiveness of the channel and help inform performance, arming you with the confidence needed to increase or decrease spending on AI campaigns.
Step 3: Embrace a new test-and-learn mindset
These new AI campaigns require a fundamental change to the way you should think about testing.
Although we have always advocated consistent testing for top spending tactics, 12-18 months ago, it was easier to run a single test on something like FB Prospecting, with the result being a definitive answer.
Now, you will likely need to accept that testing these channels may require multiple iterations. There are a lot of moving parts to consider:
- How has the AI algorithm changed?
- How have my campaign settings changed?
- How has my budget mix between AI and non-AI campaigns changed?
It’s more important to adopt a “test-and-learn” mindset where you run a single test, observe what happens, implement a campaign optimization, and retest until you feel that you have reached a steady state of optimal campaign setup.
We’ve seen many cases of our clients running tests on PMAX/ASC that initially indicated there was no impact, only to discover that there was heavy audience overlap between other campaigns.
This type of initial result may not directly answer the question, “What is the incrementality of campaign X?” but instead may lead to the realization that there has been a significant waste of media budget as a result of audience overlap between campaigns. From there, optimizing campaign settings, targeting, and budget leads to follow-up testing until the optimal setup has been achieved.
With these campaigns, you are less likely to get the exact answer you’re after with a single test. Instead, put on your detective’s hat and be prepared to observe how the platform responds to your test in order to deduce what that means for optimal campaign setup.
AI campaigns came in making big promises. At first glance, they’re the perfect automated, optimized campaign every marketer has been waiting for. They claim to help you target audiences you may not have considered before, but let’s be honest - this isn’t how they behave by default.
So, to ensure these campaigns work for you, you need to confront the challenge of evaluating their effectiveness (and setting up a testing plan to do so). There are three steps you should be taking to do this:
- Before you start testing, you must understand how your campaigns are set up in order to prepare for a testing roadmap
- Based on how these campaigns are set up, prepare to embrace non-traditional test designs, where you may need to first test the entire channel overall before testing specific campaign types
- Embrace an iterative mindset. More than any other type of campaign, these AI campaigns are unpredictable, and insights are often hidden in plain sight. Be prepared to read between the lines and look for any optimization opportunities. It may take 2-3 tests before you can actually identify exactly how incremental these are!
The testing might be more difficult, but at the end of the day, it’s non-negotiable in evaluating the effectiveness of these campaigns.
Is this something you’re struggling with currently? Connect with an expert at Measured today, and we can assist you in thinking through how to best approach this.