What is incrementality testing, and what does it answer that attribution cannot?
Incrementality testing measures what advertising actually caused, by withholding it from a comparable group and comparing outcomes. It answers the one question attribution cannot: how many of these conversions would have happened anyway. Geo holdouts and budget-split tests are the usual formats, and the result is often uncomfortable for whichever channel the attribution model had been flattering.
Attribution and incrementality look similar and ask opposite questions. Attribution takes the conversions that happened and divides credit between the touchpoints that preceded them, so by construction every conversion gets assigned to something. Incrementality asks whether those conversions needed the advertising at all, which means it needs a group that did not receive it. The two most common designs are a geographic holdout, where matched regions run with the channel switched off, and a budget split, where spend is deliberately varied and the outcome difference measured.
The channels where the two disagree most are the predictable ones. Branded search and retargeting both sit closest to a purchase that was frequently already going to happen, so they intercept credit under any last-touch style model while contributing least incrementally. This is why a test result can be genuinely unwelcome, and why agreeing in advance what you will do with each possible outcome matters more than the test design.
Three things spoil most tests. Changing several things at once makes the effect unattributable to any of them. Running for too short a period means delayed conversions are still arriving when you read the result, so the treated group looks worse than it is. And running a test too small to detect the effect size you care about produces a confident-looking null result that means nothing. Decide the minimum difference worth acting on before you start, and size the test to see it.