How to Run a Shopify A/B Test That Tells You Something

A/B testing is the most respected and most misused practice in ecommerce. The problem is arithmetic: most stores do not have the traffic to detect the size of effect they are hoping for, so they run tests that cannot conclude and then act on the noise.

The sample size question comes first

Before designing a test, work out whether it can finish.

The inputs: your current conversion rate, the smallest lift worth detecting, and standard confidence levels (95% significance, 80% power).

Roughly, for a 2% baseline conversion rate:

Lift you want to detect	Conversions needed per variant	At 2% CVR, sessions per variant
20% (2% → 2.4%)	~1,700	~85,000
10% (2% → 2.2%)	~6,700	~335,000
5% (2% → 2.1%)	~26,000	~1,300,000

Those are per variant. Double them for the test total.

A store doing 30,000 sessions a month can detect a 20% lift in about six weeks. It cannot detect a 10% lift in any reasonable timeframe, and it will never detect 5%.

That is not a reason to give up. It is a reason to be honest about which tool fits.

If you do not have the traffic

Most stores do not, and there are better options than pretending:

Test bigger changes. A radical product page redesign might produce a 25% effect, which is detectable. Moving a button 20 pixels is not.

Test higher in the funnel. Add-to-cart rate is 5-10x more frequent than purchase, so it needs far less traffic to reach significance. If the change is meant to affect the product page, measure the product page metric.

Use qualitative research instead. Ten session recordings and five user interviews will tell you more about a confusing checkout than an underpowered test ever will.

Just ship the obvious. If a change is clearly better — faster load, an answered objection, a fixed bug — ship it. Testing self-evident improvements wastes traffic you could spend on a genuine question.

The rules for tests you do run

Run full weeks. Traffic composition varies by day. A test running Tuesday to Friday samples a different audience than one running a full week. Always end on the same weekday you started.

Do not stop early because it looks good. Peeking at a running test and stopping when significance first appears is the most common way to get a false positive. Decide the sample size up front, and look when you reach it.

One change at a time, unless you are deliberately testing a whole redesign and accept that you will not know which element did the work.

Check the segments after, not during. If the overall result is flat but mobile is up 15% and desktop down 15%, that is worth knowing. But look at segments as a hypothesis generator for the next test, not as a way to rescue a null result.

What is worth testing

In rough order of expected effect:

1. Product page layout and section order
2. Pricing presentation — bundles, tiers, per-unit framing
3. Shipping messaging and threshold placement
4. Product photography style
5. Headline and value proposition
6. Cart drawer structure

And what is not worth it for most stores: button colours, minor copy edits, icon changes. The effect size is smaller than your test can detect, so the result is a coin flip.

Recording what happened

Keep a simple log: hypothesis, what changed, dates, sample size, result, decision. Two things come from it — you stop re-running tests someone tried eighteen months ago, and you build a picture of what your customers respond to that is worth more than any single result.

Include the losers. A test that failed is information about your customers, and it is the half of the record most teams throw away.