Testing & Measurement

Geo Testing vs Holdout Testing: Which Incrementality Test to Run

In short: Both are live experiments that produce a real causal incrementality number instead of a modeled or observed one. Geo testing randomizes or matches whole regions, changing spend in some markets while holding others steady as a control. Holdout testing randomizes at the audience level, permanently blocking a slice of a specific list from ads while everyone else keeps seeing them. Geo testing tells you whether a channel or market move creates incremental revenue; holdout testing tells you whether a specific audience - usually retargeting or CRM - was ever incremental to begin with. Pick geo testing to validate a channel or new market, and standing holdout testing to police whether your warmest, already-converting audiences are worth the spend.

By the AdFlint research team · Fact-checked against current Google and Meta platform behavior · Last reviewed July 2026

Geo Testing

Turns spend up or down in selected regions and compares outcomes against matched control regions to estimate causal effect.

You pair similar markets on historical sales, change spend in the test set, and model what the test regions would have done without the change. Because it works on aggregate regional revenue, it survives cookie loss entirely and can measure channels no pixel reaches. It suits national advertisers with enough geographic spread. The pitfall is contamination, whether media spillover across market boundaries or a promotion that ran in one region.

Full definition

Holdout Testing

Permanently excludes a randomly selected share of your audience from seeing ads, so their behavior serves as an ongoing baseline.

A fixed percentage of users or accounts is suppressed from targeting for the duration of the test, and the gap between held-out and exposed groups gives you the incremental effect. Advertisers keep a standing holdout on retargeting and CRM audiences, where reported returns are most likely to be double-counted demand. Watch for leakage: if the held-out group can still be reached by other campaigns, the baseline is contaminated.

Full definition

Side by side.

The differences that actually change what happens in your account.

	Geo Testing	Holdout Testing
What gets randomized	Geographic markets - DMAs, states, or postal regions matched on historical sales trajectory.	Individual users or accounts within one targetable audience, chosen at random.
What you actually change	Spend level in the test markets goes up, down, or to zero versus the control markets.	A fixed percent of the audience is suppressed from ad delivery for the whole test window.
Best question it answers	Does this channel or this new market cause incremental revenue at all.	Would this specific audience - usually retargeting or CRM - have converted without the ad.
Identity requirement	None. Works on aggregate regional sales, immune to cookie or device loss.	Needs a stable identifier (device ID, email, CRM ID) to keep the same people excluded for the whole run.
Typical duration	A defined project window, usually several weeks to a full quarter.	Can run as a time-boxed test or stay on indefinitely as a standing baseline.
Main contamination risk	Spillover - media or shoppers crossing from control markets into test markets.	Leakage - the excluded group still gets reached by a channel or campaign you don't control.
Grain of the answer	Whole channel or whole market, not individual campaigns or keywords.	Whole audience or campaign the holdout was built against, same coarse grain.
Who can run it	Advertisers with enough distinct, comparable geographic markets to build a control group.	Advertisers with a large enough named audience list to reach statistical power.

What actually separates them.

Geo testing changes a dial (spend level) across markets, while holdout testing flips a switch (shown or not shown) across people, so the two experiments manipulate different variables entirely.

A geo test ends when the project window closes and reverts to normal delivery, while a holdout is often left running indefinitely as a permanent check on one audience.

Geo testing can measure channels with no pixel or login at all, such as TV or out-of-home bought at the market level, while holdout testing only works on channels where you can identify and suppress specific people.

Contamination in a geo test comes from geography (people or media crossing market lines); contamination in a holdout comes from channel coverage (another campaign still reaching the excluded people).

Geo testing needs a fresh matched-market design for every question you ask, while a single holdout can keep answering the same question - is this audience incremental - month after month without rebuilding anything.

Which one should you use?

Use Geo Testing when

You want to know if a new channel (say, a national TV or OOH buy) is generating incremental revenue and there's no pixel or login to track it.
You are deciding whether to expand into new markets and want a real read before committing full budget.
You have enough distinct regions with comparable sales histories to build a credible matched control.
You need one clean answer about a channel-level or market-level spend change, not a read on one narrow audience.

Use Holdout Testing when

You suspect your retargeting or CRM campaigns are mostly reaching people who were going to buy anyway.
You have a large enough named audience (retargeting pool, email list, lookalike segment) to split with statistical power.
You want an ongoing baseline that keeps checking incrementality every month, not a one-time project.
You can actually enforce suppression - the platform or your own infra can reliably exclude the same people from delivery for the whole run.

Common questions.

Can I run a geo test and a holdout test at the same time?

Yes, and they answer different questions so they don't conflict. Just make sure the geo test's test markets and the holdout's excluded audience don't overlap in a way that muddies either read - for example, don't holdout retargeting in the same regions you're testing paid social spend increases in.

Why did my holdout group still convert almost as much as the exposed group?

That's usually the finding, not a bug - it means the audience was largely going to convert anyway and the ads bought little incremental lift. It's most common on retargeting and branded search-adjacent audiences, where the ad often lands on someone already mid-purchase.

How many markets do I need for a credible geo test?

There's no fixed number, but you need enough comparable regions to build a matched control with a similar sales trend to your test markets, and enough spend change to produce a signal above normal market noise. Advertisers with only a handful of markets usually don't have enough spread to run this reliably.

Is a holdout the same as an A/B test?

It's the audience-level version of the same idea - random assignment, one group exposed and one not - but it's usually run as a standing, ongoing check rather than a one-time test, and it measures incrementality (would they have converted anyway) rather than which creative or landing page wins.

Or stop choosing between them.

AdFlint picks the setting, writes the ads, and keeps optimizing inside the Google and Meta accounts you already own.

Try AdFlint free See how it works

Related comparisons

All comparisons