What is Incrementality Testing? Guide for Marketers

It's Monday morning and you're reviewing last week's performance of Meta campaigns. You've been trying to find an efficient way to convert new customers for your eCommerce business, but you are struggling. No matter what you try, your prospecting campaigns never seem to get even close to the ROAS levels that Meta is reporting for your retargeting campaigns. You are feeling desperate - how can it be this hard? At the same time, you feel something is off. Are the retargeting campaigns really that effective? 

This is what many performance marketers struggle with on a daily basis.

In this blog, we'll discuss one method for getting closer to the actual incremental revenue driven by advertising: Incrementality Testing.

What is Incrementality Testing?

Incrementality testing is a marketing measurement methodology for estimating the incremental sales impact of advertising. Incrementality tests provide an estimate for the actual revenue driven by a specific marketing activity, in contrast to revenue that would have occurred even without marketing or revenue driven by other marketing activities.

Key output of incrementality tests: 

• Incremental ROAS for the tested marketing activity, and
• A measure of confidence capturing the quality of the test.

Incrementality testing covers various experimentation methods, such as Conversion Lift Studies, Geo Lift studies and natural experiments. All of them operate on the fundamental principle of controlled experimentation. Marketers divide their target audience into two groups: a test group that receives the marketing intervention and a control group that doesn't. By measuring the difference in desired outcomes (typically sales) between these groups, you can estimate the incremental impact of marketing efforts.

The image below illustrates the objective of incrementality testing: Identifying the extent to which each marketing activity is incremental.

How Can Incrementality Testing Help Marketers: Example

Let's get back to the example that started this blog. Let's assume that Meta reports following ROAS to Meta retargeting and Mata prospecting channels:

You are skeptical whether the effectiveness difference can be this huge. To test whether your skepticism is right, you decide to run a Geo Lift test for both channels. After running the test and analyzing it's results you get the following Incremental ROAS estimate for the two channels:

The incrementality tests estimated that Meta Prospecting had Incremental ROAS (iROAS) of 3.8, whereas Meta Retargeting had iROAS of 3.2. It turns out that the prospecting campaigns were actually more effective in driving incremental revenue compared to retargeting campaigns. 

This example illustrates one use-case of Incrementality tests: Getting a one-off snapshot to the Incremental ROAS. Let's next discuss use-cases of Incrementality testing more broadly.

Use-cases of Incrementality Testing

✅ Snapshot of Incremental ROAS. Marketers can use Incrementality testing to get a snapshot of iROAS. This can be helpful for small businesses who are starting their journey towards incrementality-based measurement, as it in getting a sense of how effective certain marketing activities are.

✅ Validate & calibrate Marketing Mix Models. Incrementality tests can be used to in both validating and calibrating Marketing Mix Models. As an example, Incrementality tests are part of Sellforte's MMM calibration workflow.

Key Limitations of Incrementality testing

⚠️ No continuous measurement. Incrementality tests are snapshots in time. For continuous incrementality measurement, marketers should look into Marketing Mix Modeling.

⚠️ Limited to channel-level insights. Incrementality tests are limited to higher level insights, the tests typically high level of spend for the tested activity. Campaign or ad set-level iROAS is not measured with tests.

⚠️ No Marginal Incremental ROAS (miROAS). Incrementality tests do not measures miROAS, i.e. what is the return for the next invested dollar. They focus on measuring iROAS, which average return for the tested activity.

⚠️ Wide confidence intervals: Many Incrementality Tests suffer from wide confidence intervals. Ensure you invest in a proper experiment design.

⚠️ Cost Considerations: Running incrementality tests often means intentionally withholding marketing exposure from potential customers, creating opportunity costs that must be weighed against measurement benefits.

Types of incrementality tests

Marketers currently employ two main types of incrementality tests:

1. Conversion Lift Tests Offered by each Ad Platform

Each Ad Platform offers Conversion Lift Tests for testing incrementality specifically on their platform. Examples of Conversion Lift Tests include:

• Meta Conversion Lift Study
• Google Conversion Lift Study
• TikTok Conversion Lift Study

The main advantage of conversion lift studies within the ad platforms is that their test setup resembles (although with limitations) randomized controlled trials (RCTs), which are typically considered one of the best methods for estimating causal effects. The main idea in conversion lift tests is to divide the target audience into test group and control group, and derive insights about marketing’s incrementality by comparing the two groups. Typically, the test group sees the specific ad to be tested, and the control group sees a blank space.

While on the surface Conversion Lift Tests resemble RCTs, they have received criticism for divergent delivery, meaning that audience in both groups is not selected completely randomly: the algorithm learns to show the test ads to people likely to convert, whereas the blank space is shown to random audience.

2. Geo-Based Incrementality Testing (Geo Lift Tests)

Geo lift tests determine the incremental impact of advertising by comparing geographical regions exposed to a treatment (test areas) with similar regions that are not exposed (control areas). The most common geo-based experiment type is a "hold-out test", where a certain marketing activity is stopped completely, but Geo Lift tests can also include spend increases.

To conduct a geo lift test, regions are first grouped and matched based on various characteristics to ensure comparability. The test runs for a set period, after which performance data from the test and control areas are compared. The difference, or “lift,” seen in the test regions beyond what would be expected in the control regions reflects the true incremental impact of the marketing effort. 

Geo lift testing requires careful planning to ensure statistical power and cost-effectiveness.

Case study: How BrandAlley Used Incrementality Testing to a Validate Marketing Mix Model

To discuss what incrementality tests are and how they work, let's start with a real-world example: BrandAlley's journey to incrementality measurement. BrandAlley is a UK-based online fashion eCommerce business that launches more than 1,000 digital campaigns per year.

To ensure BrandAlley is optimally allocating their media budget across campaigns, they took Sellforte's Marketing Mix Modeling solution into use. After the Marketing Mix Model was implemented, they decided to validate it's findings with an incrementality test. In this case, they implemented Meta's Conversion Lift Study, which is Meta's own tool for running incrementality test on their platform. The study was executed in 4 weeks, and included all campaigns on the Meta platform.

The study found that Meta ROI is 4.00, with a 90% confidence interval between 2.91 and 5.09. The study provided evidence for Marketing Mix Modeling's robustness, because the MMM-estimated ROI of 3.91 was very close to the mean of the study. The findings are summarize in the picture below:

This was an example of one use-case for incrementality tests: validating Marketing Mix Modeling -based measurement with an incrementality test. Another major use-case for incrementality tests is to calibrate Marketing Mix Models with incrementality tests.

Should I use Incrementality tests or Marketing Mix Models for meassring incremental ROAS?

The simple answer is both: the two methodologies work well together. The best practice is to use a Marketing Mix Model for continuous ROI measurement for all channels , and using experiments periodically to validate and calibrate the MMM.

Conclusion

Incrementality testing is an important approach for marketing measurement, shedding light to the advertising effectiveness through rigorous experimentation. The best practice is to use incrementality test and Marketing Mix Modeling together, to achieve accurate and continuous ROI measurement. 

Authors

Lauri Potka is the Chief Operating Officer at Sellforte, with over 15 years of experience in Marketing Mix Modeling, marketing measurement, and media spend optimization. Before joining Sellforte, he worked as a management consultant at the Boston Consulting Group, advising some of the world’s largest advertisers on data-driven marketing optimization. Follow Lauri in LinkedIn, where he is one of the leading voices in MMM and marketing measurement.