What is a Holdout Group?

A holdout group is the part of an audience or market that is deliberately not exposed to a marketing activity during an experiment. It serves as the control group: comparing its outcomes with those of the exposed test group shows what would have happened without the activity, which gives the incremental effect of the marketing.

Why a holdout group is needed

Marketing metrics such as attributed conversions or ROAS show what happened after people saw an ad, not what the ad caused. Many of those people would have bought anyway. A holdout group provides the missing counterfactual: if the groups are comparable, the difference in conversions between the test group and the holdout group is the incremental effect of the activity.

Incremental lift = Conversion rate in test group − Conversion rate in holdout group

For example, if 2.4% of the test group and 2.0% of the holdout group convert, the absolute lift is 0.4 percentage points and the relative lift is 20%. Only the 0.4 points are caused by the campaign. The other 2.0% would have converted without it.

The holdout group estimates the sales that would have happened anyway, so the difference to the exposed group is the media-driven, incremental part:

Randomization

The holdout group only works as a counterfactual if it is comparable with the test group. The reliable way to achieve this is random assignment: each user or region is assigned to the test or holdout group by chance, before the campaign runs. Randomization balances both observed and unobserved differences, such as purchase intent, on average.

A group that is formed after the fact, such as "people who did not see the ad", is not a valid holdout. Ad delivery is targeted at people who are more likely to convert, so exposed and unexposed people differ before any ad is shown. Comparing large Facebook experiments with observational methods, Gordon et al. (2019) found that the observational methods often failed to reproduce the experimental results, even with extensive demographic and behavioral controls.

User-level vs geo-level holdouts

User-level holdouts give more statistical power because there are many units. Geo-level holdouts are independent of user tracking and can measure offline sales, but they have far fewer units. Read more on the geo approach in geo tests.

How the holdout group is kept away from ads

In online advertising, holding out a user is not as simple as not showing an ad. The ad auction continues, and the method used for the holdout affects what the test measures. Johnson, Lewis and Nubbemeyer (2017) compare three approaches:

• Intent-to-treat (ITT). Compare everyone assigned to the test group with everyone assigned to the holdout group, whether or not they were actually reached. The estimate is unbiased, but the effect is diluted by users who would never have seen the ad, which makes it noisy.
• Public service announcements (PSAs). Show the holdout group an unrelated ad, so that exposed users can be identified in both groups. This costs money, and because the PSA campaign is delivered differently from the real campaign, the two exposed groups may not be comparable.
• Ghost ads. The ad system records the users in the holdout group who would have seen the focal ad had they been in the test group, without showing them anything from the advertiser. Comparing these "would-have-been exposed" users with actually exposed users gives a more precise estimate without the cost of PSAs, and it works with performance-optimizing ad delivery.

Holdout size and statistical power

A larger holdout gives a more precise estimate, but every user or region in the holdout is someone the campaign does not reach during the test. Choosing the size is therefore a trade-off between precision and the opportunity cost of not advertising.

Power matters more than marketers often expect. In 25 large field experiments with US retailers and brokerages, Lewis and Rao (2015) showed that the effect of advertising on an individual's purchases is small compared with the normal variation in purchases, so even experiments with millions of users can have too little power to measure returns precisely.

For a fixed total sample, the standard error of the difference between two groups is proportional to √(1/p + 1/(1 − p)), where p is the holdout share. The table shows how a smaller holdout reduces precision compared with an even 50/50 split:

A 10% holdout keeps 90% of the audience exposed, but it needs about 2.8 times the total sample of a 50/50 split to reach the same precision. Small holdouts are therefore suitable for large audiences and large expected effects. For smaller brands or channels, a larger holdout or a longer test is usually needed. The Sellforte post What is the right holdout-group size for an incrementality test? discusses the trade-off in practice.

Conversion lift studies on Meta and Google

Ad platforms offer built-in holdout experiments called conversion lift studies:

• Meta. In a Meta lift study, the audience is randomized into test and control groups, and the control group is withheld from the ads (Meta, n.d.-c). Holdout tests in Meta's Test and Learn tool use a 10% control group (Meta, n.d.-b), and Meta treats a confidence of 90% or more as statistically reliable for lift tests (Meta, n.d.-a).
• Google. Google Ads Conversion Lift can be run with user-based or geo-based methodology. The user-based version reports the difference in conversions between an experiment group that saw the ads and a control group that did not; the geo-based version uses regions as units and can include offline data (Google, n.d.).

Eligibility rules, minimum budgets and settings change over time, so check the current help pages before planning a study. Platform lift studies measure one platform with its own data, and advertisers cannot audit the randomization themselves. They are still among the most useful inputs for calibrating a marketing mix model; see Can you trust Meta Conversion Lift tests for MMM calibration?

Key takeaways

• A holdout group is a randomized control group that is not exposed to the marketing activity, so that the difference in outcomes measures incrementality.
• Holdouts can be at the user level (platform lift studies) or the geo level (geo lift tests). Ghost ads give more precise user-level estimates than intent-to-treat or PSA designs.
• Holdout size is a trade-off between precision and lost reach. Smaller holdouts need much larger samples to reach the same precision.

Related terms

• Incrementality
• Geo lift test
• Causality
• Causal inference

Further reading on Sellforte

• What is incrementality testing? A practical guide
• What is the right holdout-group size for an incrementality test?
• Should we run a full marketing blackout or a partial holdout test?

References

1. Google. (n.d.). About Conversion Lift. Google Ads Help. Retrieved 2026-10-30, from https://support.google.com/google-ads/answer/12003020
2. Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019). A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Science, 38(2), 193–225. https://doi.org/10.1287/mksc.2018.1135
3. Johnson, G. A., Lewis, R. A., & Nubbemeyer, E. I. (2017). Ghost ads: Improving the economics of measuring online ad effectiveness. Journal of Marketing Research, 54(6), 867–884. https://doi.org/10.1509/jmr.15.0297
4. Lewis, R. A., & Rao, J. M. (2015). The unfavorable economics of measuring the returns to advertising. The Quarterly Journal of Economics, 130(4), 1941–1973. https://doi.org/10.1093/qje/qjv023
5. Meta. (n.d.-a). About confidence in your tests and experiments. Meta Business Help Center. Retrieved 2026-10-30, from https://www.facebook.com/business/help/239549606692303
6. Meta. (n.d.-b). About holdout tests in Test and Learn. Meta Business Help Center. Retrieved 2026-10-30, from https://www.facebook.com/business/help/1575448755848995
7. Meta. (n.d.-c). Lift Study. Marketing API documentation, Meta for Developers. Retrieved 2026-11-02, from https://developers.facebook.com/docs/marketing-api/guides/lift-studies

Authors

Dr. Paavo Niskala is a Principal Engineer at Sellforte. With PhD in the field of computational plasma physics, he has over 10 years of experience in designing and building complex data-intensive systems. Paavo has especially focused on using data science in critical business applications, such as Marketing Mix Modeling, which helps businesses make better marketing decisions. Follow Paavo in LinkedIn.