How are marketing incrementality experiments designed?

How do you design an experiment for marketing campaigns?

Media incrementality experiments are designed to understand the impact of a marketing campaign, channel, or ad on desired marketing objectives. A simplistic design to measure certain marketing stimuli like a TV campaign or a Facebook campaign is a 2-cell experiment, where the marketing campaign is published to a certain group of users and held out to another group of users. The response behaviors of the two user groups are then observed over a period of time. The impact of the marketing campaign is then assessed as the difference in response rates between those two user groups.

The science of experimental design applied to marketing is about carefully selecting and controlling the variables that affect outcomes, designing the approach for sample size sufficiency, and tailoring the overall design to have enough power to read the phenomenon being observed.

What are factors in experimental design for marketing campaigns?

The factors to be controlled depend on the phenomenon being measured. But in general, some of the factors that play a critical role in marketing that are candidates to be controlled are: marketing spend, campaign reach, impression frequency, audience quality, audience type, conversion rates, seasonality, collinearity and interaction effects.

Each marketing channel, like Facebook or TV or Google Search, each have their own unique campaign management levers to control audience reach, spend, frequency, etc. The challenge designing proper experiments is to apply experimental design principles to the specific channels and how they are typically operated by marketers.

Basic principles of experimental design in marketing measurement

Learning objectives: The first and foremost thing is to identify objectives that are meaningful to measure. Typically, these are sales and other business outcomes that marketing campaigns are looking to drive.

Audiences and Platforms: Each marketing platform like Facebook and Google have very specific ways to activate audiences and market to them. Experiments have to be designed around these campaign specific levers to control the factors relevant for the marketing experiments.

Decisions: Marketers make specific decisions around campaigns, like campaign budgets, campaign bids, creative choices, audience choices etc., Experiments have to be designed to inform the specific choices at the level of granularity that is meaningful for marketers.

How do incrementality experiments differ from A/B testing?

A/B tests are a simple form of a two-cell experiment. Typically industrial scale experiments are generally multivariate in nature, maybe 2-cells or more, and designed carefully to control for various factors to enable flighting the experiment and collecting data in very specific ways to enable getting a clean usable read.

What are some examples of marketing incrementality experiments?

Many marketing platforms enable experimentation deliberately or coincidentally. In platforms like Facebook it is possible to select and target audiences in randomized ways but target them differentially. This enables marketers to design experiments and test audiences for different marketing treatments. Similar approaches are taken in tactics like site retargeting where audiences are split into segments and various segments are offered differential treatments, like retargeting some segments, and withholding the retargeting ads from others.

How do MTA (multi-touch attribution) & incrementality experiments work together?

Incrementality testing addresses many of the data and data tracking gaps that serve as severe limitations to MTA’s ability to measure marketing contribution across all addressable marketing channels.

MTA has always had major data gap in the walled gardens (Facebook, AdWords, Instagram, Pinterest, YouTube etc.) in which no customer level data gathering is permitted.  Increasing restrictions on user-level tracking has made MTA even less viable as a singular approach to measurement.  As the market continues to evolve, and new regulations and privacy policies proliferate, MTA will likely be replaced rather than supported by measurement methods like incrementality testing and media mix modeling.

Learn more about marketing experiments and the results brands are seeing using Measured Incrementality at our resources and content hub.