The practice of geo testing is essential for marketers to understand how much incremental value a tactic adds to their business so they can more confidently optimize future spending. But sometimes hitting the “run test” button can be a challenge – on many levels.
Geo testing can be expensive and time-consuming and many marketers face pressure to deliver quick results. Turning off or reducing media spend in control markets can lead to lost sales, creating conflict between the need for accurate measurement and the desire to maintain or maximize revenue.
Getting stakeholders to understand and trust the results of geo testing can also be challenging, especially if the findings contradict existing assumptions.
There’s a lot at risk. That's why Measured has developed a robust set of guardrails to instill confidence in geo testing results. By prioritizing three critical components in our geo testing process – in addition to our signature “Confidence Score” – marketers can reduce that risk and gain the assurance they need to scale successful strategies and cut losses on underperforming ones.
Geo Testing Guardrails
At Measured, our geo tests are expertly designed, and over 95% of our tests lead to conclusive, actionable results based on three critical parts of the geo-testing process and the meticulous execution of each stage:
- Test Design: We conduct market selection using our leading synthetic control methodology (SCM) and design a holdout/scale test tailored to client objectives, marketing, tactics, etc. To learn more about our SCM approach, please check out this blog.
- Test Execution: There are simple but critical checks in this phase. A flawed process leads to unreliable results. For holdout tests, we verify that tactic spending has been reduced to zero. For scale tests, we confirm that spending levels meet the intended test design – and tests are being continuously monitored through the flight of the test. We also monitor potential non-compliance, ensuring that spending remains business-as-usual in control markets. These checks are enabled through an alert system.
- Test Evaluation: Before measuring the effect, we validate control and test markets using anomaly detection algorithms. These algorithms identify anomalies in both control and test markets. These can arise due to external factors or business conditions which could compromise test results. For example, in a holdout test for a disaster preparedness company, a hurricane-affected test market might experience an unexpected increase in conversions, making it unsuitable for measurement. A similar issue can arise for a control market.
The guardrails above help ensure that our tests remain consistently valid against design assumptions, have minimal bias caused by external disruptions, and are highly reliable for decision-making. Measured also has an alert system in place to support test implementation, covering aspects such as campaign mapping, budgeting, and other key factors. This system provides real-time insights to client services, ensuring smooth execution.
Anatomy of a Measured Confidence Score
Once we've gathered and checked the quality of our test data, we calculate whether the results are real or just random chance. We then give this difference a Confidence Score that tells us how reliable that result or effect is based on a statistical test score.
Measured’s Confidence Score is basically a measure of how sure we are that the changes in your media spending actually caused the results we saw, rather than just random ups and downs in the data. It gives marketers quantifiable reassurance that their media spend is having a real impact and isn't just a statistical fluke, and it complements the overall measurement process.
The Confidence Score is like a grade for your test results. A high grade means you can trust the results and make decisions based on them. A low Confidence Score means you should run the test again with some adjustments to get more reliable data. It's a crucial piece of the puzzle that helps marketers confidently understand what's working and what's not in their media spending.
The Bedrock of Reliable Measurement
By understanding and prioritizing execution and clean up, including anomaly detection, marketers can move beyond guesswork and embrace data-driven strategies that demonstrably drive results. This empowers them to optimize budgets, refine campaigns, and achieve a higher return on investment.
Embracing geo testing signifies a commitment to precision and accountability – it’s the difference between hoping for results and knowing you're driving them. If you’re ready to take your media measurement and optimization practice to the next level, get in touch with a Measured expert today.