Q&A: A Deep Dive into Geo Experimentation


Mahesh Jeswani, SVP, Product

Published 04/24/2024

Andrew Covato, Measurement Consultant (ex-Meta, Google, Netflix, Snap), sits down with Mahesh Jeswani (MJ), SVP of Product at Measured, for a deep dive into automating geo testing. 

AC: Welcome, MJ! Let's start with the basics. Why is geo-testing considered the gold standard for measuring the causal impact of media?

MJ: In today's privacy era, media measurement based on user-level path-building is becoming obsolete, with cohort-based measurement emerging as the new norm. Geo-testing stands out as the most effective method for measuring media impact because it enables marketers to segment audiences by geographic cohorts. At its core, geo-testing involves setting up test and control markets at various levels, be it state, zip code, city, or DMA. Test markets undergo specific treatments, such as withholding media spend on particular tactics for a defined period, while control markets serve as benchmarks for comparison.

AC: That sounds straightforward, but we've heard that implementing and scaling geo-testing can be challenging. Can you elaborate on that?

MJ: Absolutely. Implementing geo-testing can be daunting for several reasons. Firstly, current methods offered by ad platforms or third-party vendors often propose ineffective test designs, leading to wasted resources and unnecessary risks. These methods may demand withholding a significant portion of media—potentially up to half the country—to attain reliable results, hampering potential sales and limiting the number of tests feasible. 

Secondly, deploying geo-testing can be very slow, resulting in increased operational burdens. Coordinating tests across multiple ad platforms simultaneously or ensuring ongoing tests remain uncontaminated demands substantial manual effort, complicating day-to-day operations. 

Lastly, confidently interpreting results and making informed decisions can prove challenging. With numerous variables at play and the dynamic nature of media landscapes, accurately deciphering test outcomes requires deep analytical skills. Additionally, the implications of these decisions on broader marketing strategies must be carefully weighed, adding another layer of complexity to the process.

AC: Given these challenges, should marketers implement geo-testing on their own or outsource it?

MJ: It's a pivotal decision that requires thoughtful deliberation. Internally implementing Geo testing demands significant investments in skilled data science resources, robust data infrastructure, and ongoing operational capacity to effectively manage the tests. 

However, from my experience, a dual approach tends to yield the best results. Firstly, marketers should take the lead in fostering a culture for testing within their organization. This involves ingraining an ethos of experimentation and integrating testing seamlessly into the company's operations.

Secondly, to optimize efficiency and streamline the testing process, marketers should consider partnering with a specialized technology vendor partner like Measured. Such partnerships bring expertise in automating Geo testing and possess a proven track record of deploying testing across numerous brands. By collaborating with external partners, marketers can access additional resources and expertise while retaining control over the strategic direction of their testing initiatives.

AC: How should marketers evaluate an outsourcing vendor?

MJ: Great question. I believe there are three key dimensions to evaluate a vendor: level of automation, comprehensive methodology, and expertise.

Level of automation encompasses the automated acquisition of necessary data; automated test designs with market selection; automated test deployment on ad-platforms; and automated interpretations of confidence in the test results and calibration of models. Just imagine the resources and weeks of effort it would take for a brand to manually perform all these steps without automation, especially when dealing with multiple tests annually.

Comprehensive methodology is equally vital. Some vendors specialize only in testing, while others focus solely on media mix modeling. Dealing with multiple vendors for different methodologies can be a logistical nightmare, involving managing data across various systems and triangulating different methods to gauge the best signals. Opting for a vendor with a comprehensive methodology and built-in automation streamlines this process and ensures consistency and efficiency.

Expertise is paramount, especially for intricate scenarios like conducting large-scale experiments for low-volume channels or running multi-cell experiments to address intertwined tactics and channel overlaps. Additionally, proficiency in deploying testing across various media platforms is indispensable. This expertise allows vendors to leverage insights gained from working with other brands, offering valuable lessons and best practices.

At Measured, we excel in all these areas.

AC: One common question from marketers is how does geo-testing work with marketing or media mix models (MMMs)?

MJ: While Geo testing stands out as the primary method for isolating the causal impact of media investments, it may not always be applicable to every channel. In such instances, the media mix model (MMM) emerges as a valuable alternative. However, the real magic happens when we integrate MMM with Geo testing, thereby enhancing measurement accuracy. 

Here's how it works: a media mix model compares the week-to-week variation in a media channel's spend and impressions to its week-to-week variation in sales, attributing more "credit" to correlated channels. By integrating Geo testing results directly with MMM, we achieve a more precise understanding of causal measurement, offering deeper insights into the effectiveness of our media investments.

AC: Is there a role of ad-platform reported attribution or data in measurement? If yes, how does this play out with Geo testing and MMM?  

MJ: Absolutely. While geo-testing and MMM offer valuable insights, they have limitations in granularity. Geo-testing tends to be episodic and static, while MMM provides a top-down approach that may not capture immediate changes in media execution. To overcome these limitations, ad platform reporting fills the gap by providing granular, dynamic, and deterministic data that allows marketers to break down tactic-level insights to a more detailed level. By incorporating daily signals from ad platforms directly into measurement methodologies, marketers can achieve accurate incrementality measurement across all channels, down to the campaign and ad set level.

Measured seamlessly integrates and triangulates signals from three robust measurement approaches: Experimentation, media mix modeling, and platform-reported attribution. This comprehensive approach ensures that marketers can identify the most accurate and reliable signals at the granular level. 

AC: Can you walk me through an example of a successful Geo test conducted to measure causal media impact at Measured?

MJ: Recently, a client asked about Meta's contribution to customer acquisition. We conducted a Geo test across 5 states, running Meta prospecting in a holdout cell. Initially meeting CPA targets, further analysis revealed audience overlap with active Meta retargeting campaigns, hindering prospecting's effectiveness. The issue stemmed from a lack of audience cap or exclusions in campaign setup. The client launched a Meta retargeting/customer test to confirm the overlap impact. Then, three new experiment prospecting campaigns were initiated with varied strategies. Results informed an optimal account structure and campaign architecture, leading to a 37% increase in channel efficiency and a 7.1% rise in new customer acquisition within 30 days.

AC: You talked about the culture of experimentation. How should brands build a culture of testing?

MJ: Establishing a culture of experimentation hinges on securing leadership buy-in across the organization. This entails involvement from executives, including those in the finance department, who advocate for data-driven decision-making and endorse testing initiatives throughout the company. Once this foundation is laid, the focus shifts to educating and empowering teams. Clear objectives, aligned with the overarching business strategy, are set, and testing is seamlessly integrated into existing processes, such as budgeting on a weekly, monthly, or quarterly basis.

Moreover, collaboration with expert partners and drawing insights from the experiences of other brands play pivotal roles in reinforcing the testing culture. By adhering to a quarterly testing roadmap that encompasses geo-testing and embracing a mindset of continuous learning and adaptation, organizations can nurture a culture where testing becomes ingrained in the DNA. This not only drives sustained growth but also cultivates a competitive advantage in an ever-evolving market landscape.


To see how Measured can help your brand grow, set up a demo today