Blog
Turning ads on and off is not a test
Meta does not recommend testing by turning ad sets or campaigns on and off manually. Our reading: a test is decided before it starts, not read off whichever ad looks best afterwards, and that holds in Google Ads too.
Melvin Salas16 November 20263 min readVerified 16 November 2026
Why the on and off test fails
Our reading: the simplest ad test is also the least reliable one. Pause the old ad, turn on the new one, and compare this fortnight with the last.
Our reading: that compares two different fortnights as much as two ads. Demand, competition and the season all move between them, and nothing holds them still.
Meta makes the same point in its own terms: it does not recommend testing informally, such as by turning ad sets or campaigns on and off manually, and says informal testing can lead to inefficient ad delivery and unreliable test results.
On Meta: use an A/B test
A Meta A/B test shows each version to a segment of your audience, ensuring that nobody sees both, then determines which version performs best. Meta says it helps ensure that your audiences will be evenly split and statistically comparable, while informal testing can lead to overlapping audiences.
Meta advises using the same budget for both versions in a test to ensure a fair comparison. If you create the test by duplicating an existing campaign, your ad budget is also duplicated.
Meta says A/B testing can measure the performance of each strategy on a cost per result basis or cost per conversion lift basis, and suggests choosing a hypothesis before you select a variable to test. Our reading: write both down before launch, so the result is judged against the question you asked and not the one the numbers make easiest to answer.
On Google: judge the ad group, not the single ad
Google says responsive search ads test combinations of ads automatically, and that because they help you qualify for more auctions, ad-level performance and metrics like clickthrough rate and conversion rate may not paint a full picture of your performance.
Its advice is to focus on incremental impressions, clicks, and conversions that are garnered for entire ad groups and campaigns. It adds that if the system is showing an ad more often than another, it can mean that it's better at winning impressions in the auction itself, which is often a strong reason to prefer that creative over others.
To test one change across many ads, Google points to ad variations, which allow you to easily create and test variations of your ads across multiple campaigns or your entire account, with details like an end date and the percent of traffic to show the modified ads. Google says ad variations are best used when you want to test one change across multiple campaigns or your entire account, while custom experiments are best to test multiple simultaneous changes on a smaller scale.
Decide the test before it starts
This is our practice, not Google's or Meta's. Decide the question, the measure and the stopping point before the test starts. Change one thing. Run both versions at the same time rather than one after the other. Then read the result for the whole ad group or ad set, not only the single ad with the best click-through rate.
Google's own testing tips point the same way: establish a testing threshold and stick to it, wait until you have served enough impressions to be confident in your results, and limit how many elements you test within your ads.
Questions about testing ads
Can I test two ads by pausing one and running the other?
Meta does not recommend testing informally, such as by turning ad sets or campaigns on and off manually, and says informal testing can lead to unreliable test results. Our reading: the same caution applies in Google Ads, because the two periods differ in more than the ad.
Is the best Google ad the one with the highest CTR?
Not on its own. Google says looking at clickthrough rate or conversion rate alone can be misleading, and suggests focusing on incremental impressions, clicks, and conversions for entire ad groups and campaigns.
Should both versions of a Meta A/B test get the same budget?
Yes. Meta says to use the same budget for both versions in a test to ensure a fair comparison.
This is how we work, not only what we write about
Riibon's AI surfaces what it finds with the evidence attached, and a person decides. Recommendations are stress-tested before anyone acts on them, and every change is recorded with what happened afterwards.
Ask the Riibon app what changed in your account, and what happened next. Access starts with a short call: we look at your ads together, then run them for you or set the app up with you.