Are Bad Marketing Experiments Leading You the Wrong Way?
While data privacy concerns and the regulations designed to address them have been on the rise for a decade, the bold policy moves recently made by platforms like Apple, Google and Facebook have motivated marketers to seek better alternatives to a decomposing system of advertising measurement.
As with any industry problem this ubiquitous, solving the measurement dilemma for marketers could prove quite lucrative - and there is no shortage of vendors claiming to have the answer. As more brands embrace experimentation as a viable solution, many adtech providers have been quick to adapt their language to suggest they have just what marketers need.
At Measured, we are thrilled to see the industry warming up to incrementality testing and experiments. We began developing our methodologies and experiment designs more than five years ago, in anticipation of this very moment. However, marketers should take note: not all experiments are created equal - and very few of them measure the true incremental contribution of media to the business using your source of truth transaction data.
It’s vital that your experiments are rooted in proven scientific methodology and designed to provide reliable answers to the questions you need answered. Anything less could lead you down the wrong path, with serious business implications.
To ensure your ongoing media investment decisions are informed by reliable data insights, here are some critical features to consider as you evaluate potential partners for incrementality testing and experiment support.
Are your experiments designed using proven test and control methodology?
Measuring the true incremental contribution of a channel, campaign or tactic requires experiments rooted in proven test and control methodology. Sending two campaigns to the same audience to see which one “sticks” is not a sound experiment
To be scientifically significant, deploying a clean, uncontaminated control cohort that is an exact replica to the exposed audience (test cohort) is critical - and not easy to achieve. Experiment design involves carefully selecting and controlling all the different variables, then collecting data in very specific ways to unpack reads for each cohort and apply them to a cross-channel reporting framework.
**If you are only making comparisons and not using a credible testing methodology, you aren’t really running experiments. To get meaningful actionable insight, you need experiments carefully designed to measure your desired outcomes, using your preferred metrics/taxonomy.
Are you really measuring the incremental contribution of media?
If you’re only running tests provided by platforms, you’re only looking at relative lift by a publisher who is also selling you advertising inventory. More advanced levels of experiment design and testing that incorporate business transaction data from a source of truth like your ecommerce platform are required to reveal the true incremental contribution of media to a guiding metric like ROAS, CPA, LTV or sales.
**If you are only using self-reported last-click attribution data from the platforms and not reconciling it with source of truth business transaction data, you are getting misguided recommendations that are not based on true incremental contribution.
Is geo-testing part of the experiment strategy?
While in-platform conversion lift testing is still available on some channels, the disintegration of user-level tracking has made them less stable and the reporting less reliable. Anchored on 1st party data, geo-matched-market testing is a near-universal approach for measuring incrementality on almost any platform. For channels that don’t provide control-cell capabilities, geo-testing is the only approach.
If a measurement vendor says they can experiment for incrementality on channels like Google Adwords, Facebook or Pinterest, but they are not using geo-matched market methodology, they are merely using an existing campaign as a baseline, not deploying a true control cohort. This can only tell you if one campaign is more incremental than another, not the overall incrementality of the channel or tactic.
Geo experiments don’t require user-level data, but they can still reveal the incremental contribution of media to any metric that can be collected at the geo level. Brands we work with that have run geo-experiments in tandem with Facebook testing have revealed that, while Facebook attribution has gone down due to shrinking attribution windows and ID resolution challenges, contribution has actually remained steady.
**If you aren’t using geo-matched market testing capabilities you aren’t getting a true read on many platforms. It’s the only future-proof method of measurement that will keep delivering results as ID tracking and in-platform attribution methods decline.
Can you test for scale?
Along with understanding performance and the incrementality of various campaign elements, every marketer wants to know how much further they can take it. How much more can you invest in something before the law of diminishing returns sets in? Scale testing can identify saturation and opportunities for scale without risking budget.
**If you are not testing for scale, you are stuck with running full experiments, at full cost, until diminished returns are observed through wasted budget. Don’t throw money away.
Are your experiments automated and your insights ongoing?
Many experiment providers expect the marketer to manually create and manage campaigns in the chosen platform, and then they simply tabulate the “results.” Not only is this process tedious and ripe for human error, it lacks the ability to execute these experiments at scale and over time.
Continuous testing at scale requires an always-on automated experimental design, campaign creation, and live result dashboards that can be tracked week to week, month to month, and year to year.
**If you are required to set up and manage campaigns and experiments on each channel yourself, you aren’t using experiments designed to reveal valuable actionable insights - you are just buying reports that package your data in a different way.
Experimentation is a must-have for growing DTC Brands
Using experiments to test the incremental contribution of media to your business is the best way to make informed decisions that fuel growth. But, beware of experiment impostors.
Only Measured delivers ongoing, reliable insights based on scientifically sound experiment designs that incorporate your source of truth transaction data. Everything is automated - ingestion and management of data from hundreds of sources, experiment design and implementation, and continuous reporting for confident, agile decision-making.