Marketing experiments: when you launch in drops

Marketing experiments are how a brand finds out which of its ideas actually move customers, and for a brand that launches in drops, seasons or collections, most of them should not be A/B tests. An experiment is a habit with five parts: a hypothesis, one variable, one measure, a decision and a written record.

If your team ships campaigns every month and nobody can say which ones worked, effort is not the problem. Nothing was set up to be read. A launch without a hypothesis ends with a feeling; the same launch with one ends with an answer you can use on the next.

Marketing experiments are tests with five parts: a hypothesis, one variable, one measure, a decision written in advance and a record afterwards. For a brand that launches in drops or seasons, most of them are not A/B tests: presales, limited first runs, prototype pages and customer conversations answer faster than split traffic can.

Key takeaways

• A marketing experiment has five parts: hypothesis, one variable, one measure, a decision set in advance and a written record.
• On a store with 25,000 sessions a month, one A/B test takes about 3.4 months.
• Presales, limited first runs, prototype pages and customer conversations answer in one to three weeks, without split traffic.
• Learning rate equals decisions recorded divided by launches in the year.

What a marketing experiment is, in five parts

The hypothesis comes first because intuition is a weak judge of new ideas. Harvard Business Review's account of online experiments opens with a case at Bing: in 2012 an employee proposed a change to how ad headlines were shown, and it was judged a low priority and shelved, because even experts have a hard time assessing new ideas. The five parts are what separate an experiment from an activity:

• A hypothesis. One sentence that could turn out to be wrong: "shoppers leave the jackets page because the fit is unclear".
• One variable. The single thing that changes. Change the photos, the price and the copy at once and the result belongs to none of them.
• One measure. The number that will settle it, chosen before the launch: the category's conversion rate, the share of first-time buyers who order again within 60 days, the deposits on a presale.
• A decision. Written in advance: if the measure moves this much, the change stays; if not, it goes.
• A record. One paragraph after the window closes: what was expected, what happened, what changes next.

The record is the part most teams skip, and it is the part that compounds. In growth marketing for creative businesses, every cycle starts from what the previous one proved; a team with no records starts every cycle from opinion.

Why most marketing experiments here are not split tests

A split test needs a fixed number of sessions, not a fixed number of weeks. On the blog's worked store (300,000 sessions a year, about 25,000 a month, a 1.0 % ecommerce conversion rate), detecting a lift from 1.0 % to 1.2 % with 95 % confidence and 80 % power takes about 42,700 sessions per variant, roughly 85,400 in total. A standard sample size calculator gives the same order of magnitude. At 25,000 sessions a month, one test takes about 3.4 months.

A brand that launches every six weeks would close one test every two launches, and each launch would contaminate the test running under it. That is the traffic floor that makes a testing retainer expensive below a certain volume. The answer is not to stop experimenting. It is to choose experiments that read in days, on the scale the brand already has.

Four marketing experiments that fit a launch calendar

Each of these answers a different question, and none needs split traffic:

• A presale. Open orders or deposits for a piece before producing it. It answers "will people pay for this, at this price?" in one to two weeks, and the measure is deposits against the run size.
• A limited first run. Produce part of the planned quantity and count the restock alerts when it sells through. It answers "how much demand did the drop leave unserved?", which sizes the next run.
• A prototype page. Publish a product page or a collection that does not exist yet, with a waitlist instead of a cart. It answers "is there interest in this direction?" before a single unit is made.
• Ten customer conversations. Call ten people who bought twice and ten who bought once. It answers "what makes someone come back?" in a week, and the measure is how many give the same reason without being prompted.

These are the experiments that fit the way creative brands sell. They are also cheaper to read than most dashboards: the measure is a count the brand already has in its admin or its inbox. If you want to know which of your moments should get the first experiment, The Conversion Audit maps it from your own admin in five business days.

One quarter on the worked store, three answers

Here is how a quarter could look on the worked store, as an example. The figures are arithmetic on that store, not results anyone has promised.

Weeks 1 to 2: a presale. Hypothesis: a new colorway sells at the full €68. Variable: the colorway, offered to the email list before production. Measure: deposits against a 200-piece run. Decision written in advance: under 60 deposits, the colorway waits; above it, the run goes ahead.

Weeks 3 to 6: a product page change. Hypothesis: the bestselling category loses shoppers on fit. Variable: fit notes and photos on that category's size guide. Measure: the category's conversion rate against the previous four weeks. If the change were worth 0.1 points across the store, it would be 300 orders and €20,400 a year.

Weeks 7 to 8: ten conversations. Hypothesis: repeat buyers come back for the packaging, not the product. Measure: how many of ten mention it unprompted.

Three records, three decisions, one quarter. None of the three needed split traffic, and each one ends with something the brand did not know at the start.

Weeks 9 to 12 go to reading the three records and choosing what the next quarter tests. A UX design audit can feed that list, but it does not replace the record of what was actually tried.

Your learning rate, in one number

The number that tells you whether a brand is experimenting is not how many tests it ran. It is how many decisions it can point to.

Learning rate = decisions recorded ÷ launches in the year

A brand with eight launches and no written hypothesis has a learning rate of zero, however busy the year felt. The same eight launches with one variable each give eight answers. If one of those eight moves the conversion rate on a store like the worked one by 0.1 points, it is worth €20,400 a year, and the other seven still told the brand what not to repeat. Count your launches from the last twelve months, count the decisions you can find written down, and divide.

The rate is not a benchmark to compare with other brands. It is a trend to compare with your own last year. A brand that moves from zero recorded decisions to four in twelve months has built the thing a testing tool cannot sell it: a written memory of what its own customers respond to. That memory is what makes the second year cheaper than the first, because every new idea starts from the four answers instead of from a meeting.

Work through this with your own numbers

You run marketing for a brand that sells through [drops / a catalog / a membership / seasons] and launches about [number] times a year, with [monthly sessions] sessions a month and a [conversion rate] conversion rate. List the last [number] launches and, for each, write the hypothesis it tested, the one variable, the measure and the decision, or 'none'. Calculate the learning rate as decisions recorded divided by launches. Then propose one experiment for the next launch that does not need split traffic (a presale, a limited first run, a prototype page or customer conversations), with a hypothesis, one variable, one measure, a decision threshold written in advance and the date the window closes.

FAQ

What is a marketing experiment?

A marketing experiment is a change with a hypothesis, one variable, one measure, a decision set in advance and a written record of what happened. Without the record, it is an activity: the brand did something, but it cannot reuse what it learned.

Do marketing experiments have to be A/B tests?

No. A split test needs a fixed number of sessions, and on a store with 25,000 sessions a month one test takes about 3.4 months. Presales, limited first runs, prototype pages and customer conversations answer in days or weeks.

How many marketing experiments should a brand run?

One per cycle is enough if each one is recorded. The useful count is decisions recorded divided by launches in the year: eight launches with one written decision each beat twenty tests nobody can explain.

What should the first marketing experiment test?

The moment with no number today and the highest value per point. On a store that is often the product page of the bestselling category or the second order; the arithmetic for each is in the growth marketing post on this blog.

Finding which of your moments deserves the first experiment, in your own admin, is what The Conversion Audit maps. Five business days, $500, and the map stays with you whether or not you hire anyone next.