Press    What is A/B Testing for Media?

Original Publisher

A/B testing is a specialized type of incrementality measurement. In A/B testing randomized groups are shown a variant of a single variable (web design, landing page, marketing creative etc.) in order to determine which variant is more effective.

Incrementality measurements use A/B testing in certain media or general marketing channels, such as prospecting, where tracking a media exposure for both the test and control groups is required. In the case of media incrementality the A group (test group) is shown business as usual media exposure while the B group (control group) has exposure withheld or is shown a null media exposure, typically a public service announcement (PSA) for a charity of the marketers choice. The more generic form of A/B testing is called Design of Experiments (DoE).

Easily Run A/B Tests with Measured

We have worked hard to plug-in directly to 100+ media platforms and their APIs. Because of this, Measured provides incrementality measurement and testing with ease and speed. We can then run 100s of audience-level experiments with a quick one-time set up with your publishers. Find out media’s true contribution across all your addressable and non-addressable channels.
Learn more about measured product here.

Read the DTC Marketer’s Guide to Incrementality Measurement.

 

In marketing, A/B testing is a specialized type of incrementality measurement and is very effective when measuring the marginal lift of a media exposure.

A/B testing is a specialized type of incrementality measurement. In A/B testing randomized groups are shown a variant of a single variable (web design, landing page, marketing creative etc.) in order to determine which variant is more effective.

Incrementality measurements use A/B testing in certain media or general marketing channels, such as prospecting, where tracking a media exposure for both the test and control groups is required. In the case of media incrementality the A group (test group) is shown business as usual media exposure while the B group (control group) has exposure withheld or is shown a null media exposure, typically a public service announcement (PSA) for a charity of the marketers choice. The more generic form of A/B testing is called Design of Experiments (DoE).

Easily Run A/B Tests with Measured

We have worked hard to plug-in directly to 100+ media platforms and their APIs. Because of this, Measured provides incrementality measurement and testing with ease and speed. We can then run 100s of audience-level experiments with a quick one-time set up with your publishers. Find out media’s true contribution across all your addressable and non-addressable channels.
Learn more about measured product here.

Read the DTC Marketer’s Guide to Incrementality Measurement.

Original Publisher

 

In marketing, A/B testing is a specialized type of incrementality measurement and is very effective when measuring the marginal lift of a media exposure.

Press    How are Marketing Incrementality Experiments Designed?

Original Publisher

How do you design an experiment for marketing campaigns?

Media incrementality experiments are designed to understand the impact of a marketing campaign, channel, or ad on desired marketing objectives. A simplistic design to measure certain marketing stimuli like a TV campaign or a Facebook campaign is a 2-cell experiment, where the marketing campaign is published to a certain group of users and held out to another group of users. The response behaviors of the two user groups are then observed over a period of time. The impact of the marketing campaign is then assessed as the difference in response rates between those two user groups.

The science of experimental design applied to marketing is about carefully selecting and controlling the variables that affect outcomes, designing the approach for sample size sufficiency, and tailoring the overall design to have enough power to read the phenomenon being observed.

What are factors in experimental design for marketing campaigns?

The factors to be controlled depend on the phenomenon being measured. But in general, some of the factors that play a critical role in marketing that are candidates to be controlled are: marketing spend, campaign reach, impression frequency, audience quality, audience type, conversion rates, seasonality, collinearity and interaction effects.

Each marketing channel, like Facebook or TV or Google Search, each have their own unique campaign management levers to control audience reach, spend, frequency, etc. The challenge designing proper experiments is to apply experimental design principles to the specific channels and how they are typically operated by marketers.

Basic principles of experimental design in marketing measurement

Learning objectives: The first and foremost thing is to identify objectives that are meaningful to measure. Typically, these are sales and other business outcomes that marketing campaigns are looking to drive.

Audiences and Platforms: Each marketing platform like Facebook and Google have very specific ways to activate audiences and market to them. Experiments have to be designed around these campaign specific levers to control the factors relevant for the marketing experiments.

Decisions: Marketers make specific decisions around campaigns, like campaign budgets, campaign bids, creative choices, audience choices etc., Experiments have to be designed to inform the specific choices at the level of granularity that is meaningful for marketers.

How do incrementality experiments differ from A/B testing?

A/B tests are a simple form of a two-cell experiment. Typically industrial scale experiments are generally multivariate in nature, maybe 2-cells or more, and designed carefully to control for various factors to enable flighting the experiment and collecting data in very specific ways to enable getting a clean usable read.

 

What are some examples of marketing incrementality experiments?

Many marketing platforms enable experimentation deliberately or coincidentally. In platforms like Facebook it is possible to select and target audiences in randomized ways but target them differentially. This enables marketers to design experiments and test audiences for different marketing treatments. Similar approaches are taken in tactics like site retargeting where audiences are split into segments and various segments are offered differential treatments, like retargeting some segments, and withholding the retargeting ads from others.

How do MTA (multi-touch attribution) & incrementality experiments work together?

Incrementality testing addresses many of the data and data tracking gaps that serve as severe limitations to MTA’s ability to measure marketing contribution across all addressable marketing channels.

MTA has always had major data gap in the walled gardens (Facebook, AdWords, Instagram, Pinterest, YouTube etc.) in which no customer level data gathering is permitted.  Increasing restrictions on user-level tracking has made MTA even less viable as a singular approach to measurement.  As the market continues to evolve, and new regulations and privacy policies proliferate, MTA will likely be replaced rather than supported by measurement methods like incrementality testing and media mix modeling.

Learn more about marketing experiments and the results brands are seeing using Measured Incrementality at our resources and content hub.

 

Experiments must be designed to inform the specific questions marketers have about their paid media and inform a level of granularity that is meaningful.

How do you design an experiment for marketing campaigns?

Media incrementality experiments are designed to understand the impact of a marketing campaign, channel, or ad on desired marketing objectives. A simplistic design to measure certain marketing stimuli like a TV campaign or a Facebook campaign is a 2-cell experiment, where the marketing campaign is published to a certain group of users and held out to another group of users. The response behaviors of the two user groups are then observed over a period of time. The impact of the marketing campaign is then assessed as the difference in response rates between those two user groups.

The science of experimental design applied to marketing is about carefully selecting and controlling the variables that affect outcomes, designing the approach for sample size sufficiency, and tailoring the overall design to have enough power to read the phenomenon being observed.

What are factors in experimental design for marketing campaigns?

The factors to be controlled depend on the phenomenon being measured. But in general, some of the factors that play a critical role in marketing that are candidates to be controlled are: marketing spend, campaign reach, impression frequency, audience quality, audience type, conversion rates, seasonality, collinearity and interaction effects.

Each marketing channel, like Facebook or TV or Google Search, each have their own unique campaign management levers to control audience reach, spend, frequency, etc. The challenge designing proper experiments is to apply experimental design principles to the specific channels and how they are typically operated by marketers.

Basic principles of experimental design in marketing measurement

Learning objectives: The first and foremost thing is to identify objectives that are meaningful to measure. Typically, these are sales and other business outcomes that marketing campaigns are looking to drive.

Audiences and Platforms: Each marketing platform like Facebook and Google have very specific ways to activate audiences and market to them. Experiments have to be designed around these campaign specific levers to control the factors relevant for the marketing experiments.

Decisions: Marketers make specific decisions around campaigns, like campaign budgets, campaign bids, creative choices, audience choices etc., Experiments have to be designed to inform the specific choices at the level of granularity that is meaningful for marketers.

How do incrementality experiments differ from A/B testing?

A/B tests are a simple form of a two-cell experiment. Typically industrial scale experiments are generally multivariate in nature, maybe 2-cells or more, and designed carefully to control for various factors to enable flighting the experiment and collecting data in very specific ways to enable getting a clean usable read.

 

What are some examples of marketing incrementality experiments?

Many marketing platforms enable experimentation deliberately or coincidentally. In platforms like Facebook it is possible to select and target audiences in randomized ways but target them differentially. This enables marketers to design experiments and test audiences for different marketing treatments. Similar approaches are taken in tactics like site retargeting where audiences are split into segments and various segments are offered differential treatments, like retargeting some segments, and withholding the retargeting ads from others.

How do MTA (multi-touch attribution) & incrementality experiments work together?

Incrementality testing addresses many of the data and data tracking gaps that serve as severe limitations to MTA’s ability to measure marketing contribution across all addressable marketing channels.

MTA has always had major data gap in the walled gardens (Facebook, AdWords, Instagram, Pinterest, YouTube etc.) in which no customer level data gathering is permitted.  Increasing restrictions on user-level tracking has made MTA even less viable as a singular approach to measurement.  As the market continues to evolve, and new regulations and privacy policies proliferate, MTA will likely be replaced rather than supported by measurement methods like incrementality testing and media mix modeling.

Learn more about marketing experiments and the results brands are seeing using Measured Incrementality at our resources and content hub.

Original Publisher

 

Experiments must be designed to inform the specific questions marketers have about their paid media and inform a level of granularity that is meaningful.