All articles Comparisons

Converters vs Non-Converters: Find What Differs

Developer at a dark desk between a gray and an electric-blue column of light, comparing two groups

Most teams compare converters vs non-converters the same way: split a report into two columns, scan for the biggest gap, and call it an insight. The gap is real in the data and often noise in the world. A segment with nine conversions out of forty visitors can show a rate twice the site average and mean nothing at all.

Add cuts and it gets worse. Pages, events, traffic source and device multiply into dozens of two-way comparisons, each with its own small denominator. Sorting by rate then rewards whichever segment has the fewest visitors, because a single person moves that number the most.

Three habits fix this: a minimum sample you check before reading any rate, a two-proportion z-test for the difference, and Wilson intervals so you know how far to trust each number. All of it runs on first-party events — the same data Revenue AI reads when it compares converters and non-converters.

Cut the data along four axes only

Good comparisons start with segments that share a definition. Pick four axes and stop there:

  • Pages: the landing page, plus the second page viewed — not “pages visited” as a count.
  • Events: the one action that plausibly precedes conversion (pricing view, demo click, docs read).
  • Source: campaign or referrer, grouped until each bucket actually carries traffic.
  • Device: mobile versus desktop at minimum. Add browser only when you have a hypothesis for it.

Merge the long tail. A page that receives a dozen visitors a month is not a segment; it is a rounding error waiting to be reported as a finding. Fewer, stable segments beat a wide table you cannot defend.

Set a minimum sample before you look

The z-test needs the normal approximation to hold: at least five converters and five non-converters in each group. Below that, no test tells you anything useful, no matter how wide the gap looks.

Four translucent glass panels lit with blue edges representing four data segments

Above that floor, the sample you need depends on the difference you care about, not on the size of your site. The standard approximation for the required size per group is:

n ≈ (z₁₋α/₂ + z₁₋β)² × [p₁(1−p₁) + p₂(1−p₂)] / (p₁−p₂)²

Small expected effects need large samples; a difference you would actually act on needs fewer. If you cannot reach the number, widen the date range or merge similar segments. Do not lower the bar.

Test the difference with a two-proportion z-test

Pool both groups to get the null-hypothesis proportion, then compare:

p1, p2 = x1 / n1, x2 / n2
p = (x1 + x2) / (n1 + n2)              # pooled proportion
se = sqrt(p * (1 - p) * (1/n1 + 1/n2))
z = (p1 - p2) / se

At 95% two-sided, act only when |z| > 1.96. Use a one-sided test only if you committed to a direction before opening the data; picking the side afterwards doubles your false positives. Report the difference with its sign and both denominators, not “converts twice as well”.

Report Wilson intervals, not bare rates

Rates near 0 or 1, and small groups, break the simple interval. Wilson stays inside 0–1 and behaves at low counts. For 3 conversions in 20 visitors the point estimate is 15%, and the Wilson interval at 95% is roughly 5% to 36% — a range that should stop you from rewriting a page on the spot. Two segments whose intervals overlap heavily will usually fail the z-test anyway. Show the interval next to the rate and a reader can judge the claim without rerunning your math.

Glowing blue distribution curve on black glass with an offset marker line

Separate a real difference from many comparisons

Test twenty segments at 95% and roughly one will look significant by construction. Two habits keep that in check. First, pre-register the segments you will test before you run the query; anything you found by scanning is a hypothesis, not a result. Second, if you insist on scanning, tighten the threshold by the number of tests (α divided by the number of comparisons).

Then confirm the measurement is comparable: bots and AI agents excluded from both groups, no duplicate events, no internal traffic, same event definition on mobile and desktop.

Turn surviving differences into a decision

A significant difference is not yet a decision. Attach money to it: revenue per converter, value lost on the segment that lags. Check which channel brought each group with first-touch or last-touch attribution, because a device gap is often a source gap wearing a costume. Only then write the recommendation — with the interval, the sample and the evidence beside it.

Two overlapping translucent light bands, gray and blue, showing overlapping confidence ranges

Revenue AI does this inside your account: it splits converters from non-converters by page, event, source and device, runs the significance test, and puts the value at stake next to the reason.

FAQ

How many conversions do I need before comparing two segments?

Enough to satisfy the normal approximation in both groups — at least five converters and five non-converters each — and enough for the difference you care about to clear the noise. Size it from your expected rates with the formula above instead of picking a round number.

Is a chi-square test different from a two-proportion z-test?

For a 2×2 table they are equivalent: the chi-square statistic equals z², and both carry the same assumptions. Use whichever your team reads more easily.

Can I test a segment I noticed while browsing a dashboard?

Run it as a new test on a fresh date range or a holdout period. A segment chosen because it looked good is selected for noise, and testing it on the same window mostly confirms the noise.

If you would rather not maintain the math, Revenue AI runs these comparisons on your first-party events; the 14-day trial starts at https://pryntid.com/products/revenue-ai. See https://pryntid.com/platform for how events are collected, or https://pryntid.com/docs if you prefer to wire your own.

Try it free

Prynt is device intelligence with a free plan: 5,000 identifications a month, visitor IDs with confidence and bot detection — no credit card. Upgrade to Pro for VPN, proxy, tampering and behavioral signals.

Keep reading