Geo-Testing Series Part 1: Test Design
Andrew Covato, Ad Tech, Measurement, Growth Consultant/Advisor
Published 08/08/2023
Hello Growth Marketers. I’m Andrew Covato, and I’ve spent my career building and deploying incrementality-based measurement and optimization systems for major ad buyers and sellers. I now work with performance marketers, helping them build out scientific growth programs. I also work with measurement platforms, but exclusively with ones doing smart things – namely, helping scale incrementality measurement – just like Measured. Check out a bit more about me at growthbyscience.com.
What is Geo-Testing?
I’m going to kick off a series of mini blog posts that deep dive into geo-testing – one of the best (and, in many cases, only) ways to assess the incrementality of your media across channels without worrying about data fidelity or privacy restrictions. At a high level, geo-testing involves applying a marketing treatment (e.g., campaign, promotion, etc.) to some geographies and comparing the impact to untreated geographies. Over the next few weeks, I’m going to discuss (1) test setup, (2) test management, and (3) analytics & reporting as they relate to geo-testing. I’ll break down what goes into each of those components and how the capabilities of each vary with the level of investment and sophistication an organization may have. This series aims to help you understand the types of investments you may need to make in your testing infrastructure relative to your needs. Today, we’ll kick off with test setup.
How to Set Up a Geo Test
A bunch of math and logistics goes into setting up a geo test – and it can be simplistic or detailed. As with most things, there is a tradeoff between precision/accuracy and complexity. Suppose you’re new to geo-testing and only care about highly directional results. In that case, you can get away with randomly selecting DMAs or states such that ~50% of your business falls into a test geography. Of course, you’ll need to spend a decent amount of time ensuring your test/control geo groups are balanced along many dimensions (e.g., demographics, market trends, product preference, etc.). When you do this, you’ll likely only be able to run one test at a time, and you’ll need to make some massive assumptions about the comparability of test and control cells. 
You could improve upon this base case by conducting a “matched market” test, where you find test/control market pairs that track in terms of relevant KPIs. This may allow you to perform additional tests and get more resolution on results, though they will still be highly directional. However, the most sophisticated way to run a geo test is to use a predicted counterfactual. At a high level, this involves creating a model from trends in several control markets to predict a KPI in the test market. During a test period, you would compare the observed KPI in the test market to the predicted behavior based on the model – and the difference would be your lift. When done correctly, you can drastically reduce the treatment market size and perform more concurrent tests. 
Why Use a Dedicated Measurement Platform for Geo Testing?
Doing this on your own in-house can be a large investment – you need the right kind of data collection, analysis, and deep modeling know-how (check this paper out for an example of what is required). For example, if you are looking to set up a basic test, say, understand the total ROI of all your marketing efforts (which, by the way, is a test everyone should be doing regularly), you would need to pull all your platform spend data by channel, broken out by geo, and do the same for your sales (or other relevant outcome metrics). Your options for doing this are either a) manually (lots of time, very error-prone) or b) programmatically (requires engineering resources, and API knowledge). You would then need to tap a data scientist to analyze this data to help designate test/control markets, and give guidance on spend levels to appropriately power the test. Phew… Platforms like Measured have made test setup easy and scalable. In the vast majority of cases, it simply isn’t worth the internal effort to reinvent the wheel. With that said, if your media investment is still growing, you can probably get away with a simpler, more directional setup. I recommend platforms like Measured to my mid and large-sized performance marketing clients. In the next post, we’ll talk about the most challenging part of incrementality testing: test management. In my view, this is where a SaaS solution can truly benefit a test-oriented marketing organization. TL/DR SUMMARY: Market Selection Strategy
| Level | Beginner (very manual, simplistic design, high business risk) | Intermediate | Advanced (measured) |
| Market Selection Strategy | Randomly holding out DMAs or States | Holding out matched markets | Creating counterfactual prediction based on markets |
| Test Market Size | 50%+ | 10-25% | 0-10% |
| Time Spent on Market Selection | 10 Hours+ (manual selection based on contextual data like Demographics, market trends, product preference etc.) | 5 Hours (data driven analysis using historical Sales data to select markets, done manually by a Data Analyst on an ad hoc basis) | Minutes (end to end automation) |
| Cell Availability | 1 test at a time | 2 Tests at a time | 3+ Tests at a time |
Geo-testing is a powerful tool for revealing the true impact of your media investments, but not all tests are created equal. How mature is your brand’s testing practice? Download the geo-testing scorecard here to find out. Check out part two of this ongoing Geo-Testing Series: Test Management. or download our ebook 'The 3 Essential Steps to Geo-Testing Like a Pro' today.
