What is Geo-Testing?
Geo-testing is one of the best (and, in many cases, only) ways to assess the incrementality of your media across channels without worrying about data fidelity or privacy restrictions. At a high level, geo-testing involves applying a marketing treatment (campaign, promotion, etc.) to some geographies and comparing the impact to untreated geographies. A bunch of math and logistics goes into setting up a geo test – you can dive into details in our guide to geo-testing here.
Measured is committed to helping brands figure out measurement for continuous data-driven growth, which requires adapting to inevitable and frequent changes that affect our environment.
Why is Geo-Testing Beneficial to Marketers?
As user-level tracking and Multi-Touch Attribution (MTA) are laid to rest, experimentation anchored on first party data is the future-proof approach for marketing measurement. Geo-matched market testing is a near-universal approach for measuring incrementality on most media channels, especially for prospecting tactics.
Geo experiments don’t require user-level data, but they can still reveal the incremental contribution of media to any metric that can be collected at the geo level. We’ve heard from some brands that Facebook reps have even recommended running geo-tests to validate results while they work to stabilize reporting after the ripple effects of iOS 14.5 and ATT.
In short, it’s a good time to get to know geo. Watch the video and access the slides from our webinar on geo-testing for an overview of the methodology, a live demo of geo-experiment design, and a case study showing how one brand used geo testing to reveal scale potential of YouTube prospecting. (They doubled their spend while maintaining target ROAS.)
We also had a lively Q&A during the session, which is included below if you’re curious about what your peers are asking.
Geo Testing Q&A
Q: How are conversion attribution windows approached as they vary across platforms? Especially with the iOS 14 impact on Facebook.
A: It essentially represents the denominator in your incrementality percentage calculation. The way we arrive at the number of incremental conversions is independent of in-platform tracking, so you can set the windows however you like.
Q: How does the Measured platform determine the list of markets to be included in a test?
A: We test within a single country, at different geo granularities (State, DMA, City, ZIP). The market selection process is based on a correlation analysis driven by a regression model and a ranking process. This gives us a list of the best test geo markets to use in your design.
Q: What model are you using to predict the conversions?
A: We have three methods: linear regression, lasso, and flat average. We use the model that comes back with the best result in the A/A analysis. Our most used model is Lasso.
Q: What do you use as the source of conversions if not the in-platform tracking?
A: We're looking at the change in your actual sales, not what platforms report. The data comes from your source of truth for transactions, such as your ecommerce platform.
Q: This seems limited to large brands that have a large reach across many markets. Is that correct or can you run with only 3-4 markets? Is that recommended?
A: Smaller brands are actually better suited for tests, since there is a more discernible impact of media on the business. If the brands are limited to certain markets then they have to take a more granular geo grain for testing like DMA or ZIP code.
Q: Does the market selection model account for other channel spends in-market (mix of national and other geo/local) to control for noise?
A: Designs are typically deployed to control for certain channels that are being tested. But markets are selected to make sure they are "matched," meaning the markets being tested are good candidates to understand "national" impact.
Q: So if we heavy-up geographically in-platform, we can ideally see the outcome in our first party data to prove the effect is there?
A: That's correct, you can see the scale in your first party data in those markets where media heavy-up was executed.
Q: Given the focus of this webinar on how marketers can leverage Measured, can you share some insights on how platforms such as Facebook and Google get involved in this process/partnership, if they do?
A: We often have representatives from Facebook and Google join discussions about the client's learning agenda and experiment design considerations. Past that, it's usually an automated execution. Sometimes they also participate in the interpretation of the results. Generally, they are good partners for clients in navigating the testing agenda.
Q: I think you mentioned this is a mid/upper funnel campaign, so it may not drive immediate conversion. Is the ROAS based on sales/conversions or some sort of intermediary metric like site visitors?
A: This would be sales/conversions observed within the observation window. Some channels have more "tail" past the 30 days that is not directly measurable.
Q: In the YouTube example, how do you account for higher media costs (eCPMs) as spend scales?
A: When the spend gets scaled on the platform in specific markets, CPMs do go up, significantly sometimes, so the marginal CPA/ROAS accounts for the CPM environment at higher spend levels. Naturally, auction bidding environments can have more daily fluctuations, but we're looking to understand large scale changes. So the CPM changes can be used directionally.
Q: Do you run market tests on demographic levels other than DMAs? For example ZIP codes?
A: We do. Our product can support State-DMA-City-ZIP designs. DMA is U.S. only.
Q: Clients in certain verticals (such as CPG) don't always have access to first party conversion data. What is your recommendation/solution for them if they want to be able to run geo lift studies?
A: Retailer-based third party transaction data is challenging. At the very least, you need data available with the state/DMA/ZIP geo grain in them to enable testing. Otherwise, you're better off using a Media Mix Modeling approach.
Q: Brands that are CPG focused working with several retailers have weeks of lag time on sales data reporting. How would you design an experiment for them to account for that lag?
A: In theory, this wouldn't hurt our design or read process; it would just take longer for your read to arrive.
Q: Is Measured looking at geo-testing as an alternative to understanding the long-term impact of channels/media, in particular for businesses with an extended sales cycle?
A: Typically, sales that take a really long time to convert are not great candidates for geo testing. Consideration cycles under a month are good candidates for geo.
Q: Regarding your introductory examples: Isn't your design too simple because there are cells missing? What about interaction effects between YT and Paid Search?
A: We will typically design more cells (this was a quick/simple demo). As Nick mentioned, you can get more creative and sophisticated with the design. It all depends on the business questions you are trying to answer. On the reporting side, we will also do an analysis on the source/medium to see if there is an interactive effect in the reporting.
Why Measured for Geo-Testing
Measured’s fully automated geo-testing product leverages cutting-edge data science and machine learning techniques to streamline the test design and deployment process, reducing time to results and operational overhead.
Our platform automatically verifies this accuracy for each experiment by predicting the outcomes in similar markets before the test is even deployed through a technique known as counterfactual prediction. This means shorter tests with industry-leading accuracy, deployable on more channels, and with the ability to run up to eight or more tests simultaneously without contamination.
Our technology allows brands to build scalable, enterprise-grade testing regimens that reveal the true impact of the media investments, all without needing an enterprise-scale data science team. Schedule a demo today to see how it can benefit your brand.