Testing & Measurement

A/B Testing vs Incrementality Testing: Which Question Are You Answering

In short: Both are randomized experiments you run in an ad account, but they compare different things. A/B testing splits an audience between two ad treatments to see which wins - both groups see something. Incrementality testing splits an audience between a treatment and a true no-ad control, to see whether the advertising caused anything at all. A/B testing can only ever crown the better of two options you already believe in; incrementality testing can tell you the whole idea is worthless. If you already know the ads work and want the better version, A/B test; if you suspect the ads might not be adding anything, run an incrementality test.

By the AdFlint research team · Fact-checked against current Google and Meta platform behavior · Last reviewed July 2026

A/B Testing

Splits a randomized audience between two variants that differ in one element, then compares outcomes to decide which performs better.

Both platforms provide built-in split tests that divide the audience so people see only one cell, avoiding the overlap you get from simply running two ad sets side by side. Use it for creative, landing pages, audiences, or bid strategy, one variable at a time. The usual failure is calling a winner on a handful of conversions, where the observed gap is well inside normal variance.

Full definition

Incrementality Testing

Measures how many conversions the advertising actually caused by comparing an exposed group against a randomized control that saw nothing.

You withhold ads from a randomly chosen slice of the eligible audience or market, then read the difference in outcomes between exposed and held-out groups. It answers the question attribution cannot: how much of this would have happened anyway. Advertisers run it on brand search, retargeting, and any channel with suspiciously good reported returns. The mistake is running it too small or too briefly to detect a realistic lift.

Full definition

Side by side.

The differences that actually change what happens in your account.

	A/B Testing	Incrementality Testing
What the control group sees	The other variant, not nothing - both groups in an A/B test are exposed to some ad.	No ad at all, which becomes the counterfactual baseline for the comparison.
Question it answers	Which of these two treatments performs better on the metric you picked.	Whether the ad caused any of the outcome, or whether it would have happened anyway.
Where you'd run it	Google Ads experiments or Meta's A/B test tool, inside a single campaign.	Platform lift studies, geo holdouts, or a dedicated method, usually spanning a whole channel.
Typical duration	Days to a couple weeks, until one variant reaches significance.	Weeks to a full quarter, long enough to cover a realistic conversion window with enough control volume.
What a win tells you	Variant B beat variant A, but both groups were exposed, so neither number is a baseline.	The exposed group outperformed a group that saw nothing, so spend is producing conversions that would not exist otherwise.
Failure mode	Declaring a winner off too few conversions, where the gap sits inside normal variance.	Sizing the control group or window too small to detect a realistic lift, so a real effect reads as noise.
What it costs you when wrong	You ship the losing creative or landing page and keep testing minor variants forever.	You keep funding a channel that never added conversions, or kill one that actually was working.

What actually separates them.

A/B testing always exposes both groups to some ad; incrementality testing exposes one group to none, so only incrementality testing has a true zero-spend baseline.

A/B testing answers a relative question, which variant, while incrementality testing answers an absolute one, did the spend cause anything.

A/B test results live and resolve inside a single campaign's settings, while incrementality tests usually span a whole channel or audience segment and get read outside the campaign UI.

A/B testing can run constantly on small creative decisions; incrementality testing is resource-heavy enough that most advertisers run it a few times a year on a channel, not per creative.

Winning an A/B test never proves the campaign itself is incremental, since you can pick the best of two ineffective options and never know it, which is exactly what incrementality testing catches.

Which one should you use?

Use A/B Testing when

You already believe the channel works and want to know which creative, landing page, or audience performs better.
You are choosing between two bid strategies or two ad copy angles and need a fast, cheap read.
Your conversion volume is high enough to reach significance within a couple of weeks.
You want an answer that lives inside the platform UI without coordinating a longer study.

Use Incrementality Testing when

Your attribution numbers on a channel look implausibly good, especially on brand search or retargeting.
You are deciding whether to fund or cut an entire channel, not pick between two versions of one campaign.
You suspect a large share of reported conversions would have happened without any ad exposure.
You have enough scale to hold back a meaningful control group without starving the test cell.

Run the numbers yourself

A/B Test Significance Calculator

Enter visitors and conversions for two variants to get conversion rates, uplift, z-score, p-value, and whether the result is significant.

Open the free calculator

Common questions.

Can an A/B test tell me if my campaign is incremental?

No. Both cells in an A/B test see ads, so even a clear winner only tells you which version performs better relative to the other - neither result tells you what would have happened with zero ad exposure. To answer that you need a design with a true no-ad control, which is what incrementality testing provides.

Which one should I run first?

Incrementality first if you have never validated the channel, because there is no point optimizing creative on a channel that might not be adding conversions. A/B testing is the tool you reach for once you know the channel earns its budget and you are refining execution within it.

Do I still need incrementality testing if my A/B tests keep showing improvement?

Yes, they answer different questions. A campaign can show a steady stream of A/B winners while the whole channel adds zero incremental conversions, because every winner was only ever compared to another ad, never to no ad at all.

Why does incrementality testing take so much longer than an A/B test?

You need enough held-out volume and enough time inside a realistic conversion window to detect a lift that is usually smaller and noisier than the gap between two creative variants. A/B tests compare two similar treatments, which tends to produce a cleaner, faster read than comparing exposure to no exposure.

Or stop choosing between them.

AdFlint picks the setting, writes the ads, and keeps optimizing inside the Google and Meta accounts you already own.

Try AdFlint free See how it works

Related comparisons

All comparisons