Creative testing: the budget a verdict needs

Creative testing is traffic bought to learn something by a date. Its budget is not a daily spend picked by feel. It is the purchases a verdict needs, times what each purchase costs.

Most creative tests skip that sum. Four creatives get a few hundred euros each, and someone reads the dashboard after a week. One creative shows 26 purchases and another 15, and the first one is crowned. At that size, the gap is mostly chance.

The sum below runs on the store this blog uses throughout. It ends on the number to compute before the first euro of a test is spent: what the verdict costs, and whether the budget can buy it.

Creative testing compares ad creatives on purchases per euro, so its budget is the purchases a verdict needs times the cost per purchase. At €25 a purchase, spotting a creative that brings 1.5 times the purchases takes 79 per creative: €1,975 each. A €600 creative only spots one that doubles them.

Key takeaways

• Creative testing is traffic bought for an answer: its budget is the purchases a verdict needs times the cost per purchase.
• Purchases per creative ≈ 7.84 × (2 + lift) ÷ lift²: 24 to spot twice the purchases, 79 for 1.5 times, 432 for 1.2 times.
• At €25 a purchase, four creatives at €600 each can only spot one that halves the cost per purchase.
• Meta says ad sets usually leave learning after about 50 results in a week; results read before that are learning-phase results.
• Price the verdict before the test: purchases needed × last month's cost per purchase × creatives, and cut creatives if it does not fit.

What creative testing actually buys

A creative test does not buy reach. It buys an answer to one question: does this creative bring more purchases per euro than that one? The answer has a price, and the price depends on how small a difference the test has to see.

The ad platform adds a second condition. Meta's help page on the learning phase says ad sets leave it “as soon as they can deliver stably. This usually occurs after about 50 results in the week after the ad set's last significant edit.” Before that, ad sets “usually have a higher CPA”, and “your results aren't necessarily indicative of future performance.”

So a creative that runs in its own ad set and stops at 20 purchases was read while the system was still learning. Its cost per purchase describes the learning phase, not the creative.

The purchases a verdict needs

The number of purchases a test needs comes from the same two-proportion formula behind A/B testing with low traffic, written in purchases instead of sessions. With equal spend on each creative, 95 percent confidence and 80 percent power:

Purchases per creative ≈ 7.84 × (2 + lift) ÷ lift²

lift = how many more purchases per euro the better creative brings
       (0.5 = one and a half times as many)

Three differences, three prices in purchases:

Better creative brings     lift   purchases per creative
Twice the purchases        1.0    7.84 × 3.0 ÷ 1.00 ≈ 24
1.5 times the purchases    0.5    7.84 × 2.5 ÷ 0.25 ≈ 79
1.2 times the purchases    0.2    7.84 × 2.2 ÷ 0.04 ≈ 432

The last line matches the low-traffic post: 432 purchases is about its 42,700 sessions at 1 order in every 100. Going from twice to 1.5 times triples the purchases. Going from 1.5 to 1.2 times multiplies them by more than five.

The creative testing example: €25 a purchase

Take the store this blog uses throughout: 300,000 sessions a year, 1 order in every 100, €68 average order. That is 3,000 orders a year, about 250 a month. Say paid social buys a purchase at €25. That figure is declared for the example, not a client's number. It sits under the €30.60 of margin a €68 order keeps at 45 percent, the figure in what AOV means.

Budget per creative = purchases needed × cost per purchase

Twice the purchases:      24 × €25 = €600
1.5 times the purchases:  79 × €25 = €1,975
1.2 times the purchases: 432 × €25 = €10,800

Now the usual test: €2,400 split across four creatives. Each gets €600, which buys 24 purchases. That can only see a creative that doubles the purchases per euro, one that brings the cost per purchase down from €25 to €12.50. Anything smaller reads as noise, and the dashboard still shows a winner.

And 24 purchases is under half of the 50 results Meta says an ad set usually needs to leave learning. At €25, those 50 cost €1,250 a week for each ad set.

Which tests a store's budget can actually read, and where its paid traffic loses orders after the click, is what The Conversion Audit maps in five business days, for $500.

Ad creative testing that can finish

When the verdict costs more than the budget, there are three honest moves. Splitting the same money across more creatives is not one of them.

• Fewer creatives. The same €4,000 buys two creatives at €1,975 each instead of six at under €700. Two readable answers beat six unreadable ones.
• Bigger differences. A new color on the same photo needs the 1.2 line: €10,800 per creative. A different promise, a different product or a person instead of a packshot is a bet on the 1.5 or twice line, which the budget can reach.
• A fixed size, decided first. Evan Miller's advice in How Not To Run an A/B Test applies to ads as much as pages: “Decide on a sample size in advance and wait until the experiment is over.” He also warns that “repeated significance testing always increases the rate of false positives.” Checking the dashboard every morning and stopping on a good day is exactly that.

Meta adds one more rule for the length of the test: “By editing an ad, ad set or campaign during the learning phase, you reset learning.” A test that gets a new budget halfway starts again.

Creative testing against the calendar

On the example store, the 1.2 line needs 864 purchases across two creatives. That is three and a half months of every order the store takes, paid or not. Small differences between creatives are real. A store this size cannot afford to prove them.

The spend is not lost, though. At €25 a purchase, each order keeps €5.60 of the €30.60 margin before fulfillment. The verdict is paid for by the orders it brings, as long as the cost per purchase stays under €30.60. That is also why test spend belongs in customer acquisition cost, not in a separate line called learning.

What changes is the question asked of the budget. A test should name its hypothesis, its one variable, the difference it needs to see and the date it closes, the same frame as marketing experiments. If the date falls after the launch it was meant to inform, the test is a report, not a decision.

The creative testing number before spending

Purchases per creative = 7.84 × (2 + lift) ÷ lift²
Budget per creative    = purchases per creative × cost per purchase
Test budget            = budget per creative × number of creatives

On the example store, at €25 a purchase: 79 purchases per creative, €1,975 each, €3,950 for a two-way test that can see a creative bringing 1.5 times the purchases. Four creatives at €600 can only see one that doubles them.

What to do by Monday: take last month's cost per purchase. Decide the smallest difference worth acting on: twice, 1.5 times or 1.2 times. Multiply its purchases by the cost per purchase, then by the number of creatives. If the total is more than the test budget, cut creatives or make them more different before the first euro goes out.

Work through this with your own numbers

You are planning a creative test for an online store's paid social ads. Inputs: [cost per purchase last month], [test budget], [number of creatives planned] and [average order value and gross margin per order]. For three target differences (the better creative brings 2, 1.5 and 1.2 times the purchases per euro), calculate purchases needed per creative as 7.84 × (2 + lift) ÷ lift², where lift is 1.0, 0.5 and 0.2. Multiply by the cost per purchase to get the budget per creative, then by the number of creatives. Say which difference the planned budget can actually detect. If it can only detect a doubling, propose fewer or more different creatives. Also check that the cost per purchase stays under the margin per order, and fix the number of purchases per creative before the test starts.

FAQ

What is creative testing?

Creative testing runs two or more ad creatives with equal spend to learn which brings more purchases per euro. It is traffic bought for an answer, so it needs a size and a closing date decided before it starts, not a winner picked from the dashboard.

How much budget do you need for creative testing?

Purchases needed per creative times the cost per purchase. At €25 a purchase, spotting a creative that brings 1.5 times the purchases needs 79 per creative: €1,975 each, €3,950 for two. Spotting one 1.2 times better needs 432: €10,800 each.

How many purchases do you need to test ad creatives?

It depends on the difference the test has to see. At 95 percent confidence and 80 percent power: about 24 per creative to spot one that doubles purchases, 79 for 1.5 times, and 432 for 1.2 times. Smaller differences cost far more.

How long should a creative test run?

Until each creative reaches the purchases fixed in advance, without edits. Meta says an ad set usually leaves the learning phase after about 50 results in a week, and editing it during learning resets it. Stopping early on a good day raises false positives.

How many creatives should you test at once?

As many as the budget can read. Divide the test budget by the cost of one verdict. With €4,000 and €1,975 per creative, that is two. Six creatives on the same money only spot differences close to twice the purchases.

Before the next test goes live, know what its verdict costs and where the store loses orders after the click. The Conversion Audit maps it in five business days, for $500, and the map stays with you whether or not you hire anyone.