Holdout Testing

A holdout test keeps a randomly chosen share of an audience out of your advertising and compares their behavior with the people who were exposed. It is the core design under most lift tests. Google's lift studies page describes the groups as people who are shown your ads and a control group of people who are not, with the difference in outcomes giving the lift. Meta's Brand Lift documentation uses the term holdout group, randomly divided from the sample, and warns that people in it should not see your ad. Advertisers also run their own holdouts on email lists and retargeting audiences. Platform lift tools can run the split for you, while a customer-list holdout lets you control it directly and apply it to email as well.

The mechanism depends on random assignment. Before the campaign starts, a fixed percentage of the audience, often 5 to 20 percent, is suppressed. The rest is eligible for ads. After the test period you compare the conversion rate of the exposed group with the held-out group. Because both groups were drawn from the same audience at the same time, seasonality, pricing, and promotions affect them equally. The gap is the effect of the ads. A holdout can be set up inside a platform lift tool, or by excluding a random slice through a customer list or audience exclusion. A permanent small holdout can serve as an always-on check, so you do not need a new experiment every time you question a result.

Holdouts matter most where reported results are likely to include people who would have converted anyway. Retargeting is the classic case: you show ads to people who already visited your site, and many of them were about to buy. CRM and email audiences behave similarly. Holdouts also fit always-on programs, since a small permanent holdout, say 5 percent, keeps a running baseline without pausing everything. The trade-off is the cost of the held-back group, who miss any benefit the ads would have given. Keep it as small as the test power allows, and plan the duration before starting. Check that the holdout looks like the exposed group in size, recency, and past spend before the campaign starts.

Leakage is the main failure. If the held-out group can still be reached by other campaigns, such as a broad prospecting ad, a different platform, or a lookalike audience that overlaps, then the baseline is no longer clean and the measured lift shrinks. Meta's Brand Lift page makes the same point: running more than one campaign may result in people in the holdout group seeing your ad. Other mistakes include a holdout too small to detect a realistic lift, changing targeting midstream, and counting conversions from people who were not in either group. Check that the exposed and held-out groups look similar before reading results. Be careful with very small audiences, since a 10 percent holdout of 2,000 people has only 200 users and will give a wide range.

To run one this week on a retargeting campaign, take your audience list, randomly assign 10 percent to a holdout, and exclude that segment from every campaign you control. Run for two to four weeks, longer if your sales cycle is slow. Count conversions in each group from the same source, such as your own order system. Compute the conversion rate of each, take the difference, and multiply by the size of the exposed group to estimate incremental conversions. Divide spend by that figure to get cost per incremental conversion. Share the plan with anyone who runs other campaigns, so no one reaches the held-out group by accident.