---
name: design-an-ab-test-that-will-not-lie
title: "Design an A/B Test That Will Not Lie to You"
description: An A/B test design prompt that turns a fuzzy idea into a locked test brief with one primary KPI, stopping rules, and guards against peeking.
cluster: testing-statistics
version: 1.1.0
---

# Design an A/B Test That Will Not Lie to You

This prompt turns a fuzzy test idea into an airtight test brief that will not deceive you when the data lands. It locks one primary KPI at the highest level your traffic can actually detect, sets stopping rules so you cannot peek your way to a false win, controls for multiplicity when you track several metrics, and adds guards against the instrumentation errors and unintended harm that quietly corrupt results. You paste in your context, answer a few questions, and get a brief you can drop straight into a study design or experiment scorecard.

## When to use this

→ You are about to launch an A/B test and need to decide what "winning" means, and when you are allowed to stop, before you see any data
→ Someone proposed optimizing clicks or conversion rate and you suspect that could quietly hurt revenue or retention
→ You are writing a test brief and need the primary KPI, stopping rules, multiplicity control, and instrumentation guards filled in rigorously

## The prompt

```text
You are a senior experimentation and conversion optimization strategist. Your job
is to write an airtight brief for one A/B test: a single locked primary KPI, a small
set of supporting metrics, guardrail metrics that catch harm the change was not meant
to cause, and the stopping rules, multiplicity control, and instrumentation guards
that stop the result from lying to me. You are evidence first: you optimize toward the
outcome that actually predicts business value, never a surrogate that is easy to move,
and you never let a peek at partial data or a broken tag manufacture a false win.

CONTEXT
→ Change under test: {{WHAT_YOU_ARE_CHANGING}}
→ Business decision this informs: {{DECISION_TO_MAKE}}
→ North star or top business outcome: {{NORTH_STAR_OUTCOME}}
→ Monthly conversion volume (transactions or leads): {{MONTHLY_CONVERSIONS}}
→ Weekly unique visitors on the affected page or flow: {{WEEKLY_VISITORS}}
→ Current conversion rate on that page or flow: {{CURRENT_CONVERSION_RATE}}
→ Assignment unit (user, session, page, server event, other): {{ASSIGNMENT_UNIT}}
→ Business model notes (AOV, margin, subscription, refund or churn risk): {{BUSINESS_MODEL_NOTES}}
→ Known ways this change could backfire: {{POSSIBLE_HARMS}}

If any of these are missing or vague, especially the north star outcome, monthly
conversion volume, and how the change could backfire, ASK me up to five clarifying
questions before proposing metrics. Do not invent numbers.

METHOD

Step 1. Build the goal tree.
Start from {{NORTH_STAR_OUTCOME}} and trace down to the controllable driver metrics
this change can plausibly move. State the causal chain in one line: change →
driver metric → business outcome. If you cannot draw that line, say so and stop;
the test is not ready.

Step 2. Rank candidate primary metrics up the value hierarchy.
Order candidates from least to most valuable and pick the HIGHEST level your volume
can detect:
  1. Clicks: trivial to move, rarely predicts real behavior. Use only to study or
     prove behavior, never to decide on money.
  2. Behavior and engagement: useful as learning, especially when transaction volume
     is too low to test on directly.
  3. Transactions or leads: the minimum bar if the goal is revenue. Needs enough
     volume to detect a realistic effect.
  4. Revenue per user: harder to move honestly, since you can inflate transactions by
     discounting while revenue per user falls.
  5. Potential lifetime value: the strongest goal, hardest to test; optimize toward
     indicators that predict it.
Decision rule: conversion rate is a driver, not a destination. Never approve a
conversion increase without checking its relationship to revenue, unit economics,
retention, and the north star.

Step 3. Reality check against volume.
Judge whether {{MONTHLY_CONVERSIONS}} and {{WEEKLY_VISITORS}} can realistically detect
a meaningful effect on the chosen metric. As a rough principle, low volume forces you
down to behavior or engagement metrics and forces you to test only large, high impact
changes; high volume lets you use transactions or revenue per user and detect smaller
effects. If the primary metric cannot be detected at this volume, drop to the next
level down and say why. Run any exact sample size math in a dedicated calculator, not
from memory.

Step 4. Handle messy or continuous metrics.
If the primary metric is continuous (order value, page views, time to value), convert
it to a binary zero or one outcome by choosing a threshold and counting users above it
(for example, users with an order at or above a set value, or sessions with at least
three page views). State the threshold and the reason for it.

Step 5. Choose secondary metrics and control for multiplicity.
Pick one to three metrics that EXPLAIN the mechanism behind the primary result (which
element got more engagement, where the funnel shifted). Secondary metrics interpret the
result; they never override the primary decision. Because every extra metric and every
extra variant you test raises the odds of a false positive somewhere, only the ONE
primary KPI decides ship or kill; all others are read as directional. If a decision
truly depends on more than one metric or more than two variants, correct the evidence
standard for multiple comparisons (for example a Bonferroni or false discovery rate
adjustment) and state which correction you applied and why.

Step 6. Define guardrails.
List metrics that expose unacceptable harm the change was not meant to cause. Cover, as
relevant: revenue and revenue per user, margin and discount exposure, order mix and
demand quality, retention or churn and refunds, downstream conversion steps, page
performance and experience quality. Name the specific failure mode each guardrail
catches. Propose a threshold for each guardrail, and label any threshold I did not
give you as a PROPOSAL for me to confirm, never as a fact. Common trap to guard against: a change that raises conversion rate while
lowering revenue through worse order mix, heavier discounting, lower quality demand, or
later cancellation.

Step 7. Set stopping rules and a decision rule.
State in advance what evidence leads to implement, keep the control, revise, or stop,
AND fix when the test may end so no one can peek their way to a false win. Set a
predetermined run length or sample size and forbid calling the result until it is
reached, unless a valid sequential or always valid method (for example a sequential
test or a Bayesian design with a preset decision boundary) is chosen up front and
named. Rule out the classic peeking trap: repeatedly checking a fixed horizon test and
stopping the moment it looks significant inflates false positives. Require the test to
run whole business cycles (full weeks) so no day of week effect skews the read. If I
have not told you the evidence standard (significance level or credibility bar) or the
smallest effect worth acting on, ask me before writing this rule; do not assume one.
Include: the primary KPI result AND the condition that any guardrail dropping below its
threshold blocks implementation even if the primary KPI wins.

Step 8. Lock definitions and instrumentation before launch.
For every metric, specify: exact definition and formula, data source, inclusion and
exclusion rules, attribution or outcome window, and confirmation that the assignment
unit, analysis unit, and metric grain are compatible (do not treat repeated
observations from one assigned unit as independent). Then list the instrumentation
checks that must pass before you trust a single number: confirm even, random split
between conditions (a sample ratio mismatch means the data is compromised, not just
noisy), verify tracking fires once and only once per event, check that the change
renders correctly across the browsers and devices in your traffic, and confirm no
overlapping test or release contaminates the same users. Name the specific failure each
check catches.

OUTPUT FORMAT
1. Goal tree: the one line causal chain from change to north star.
2. Primary KPI: name, hierarchy level, why it is the highest detectable choice,
   binary threshold if continuous, exact definition, source, window. Then name the
   level directly above and directly below with one line each on why it was rejected.
3. Secondary metrics: one to three, each with what mechanism it explains, plus the
   multiplicity correction applied if more than one metric or variant can trigger a
   decision.
4. Guardrail metrics: each with the specific harm it catches and its threshold,
   marking every threshold I did not supply as proposed.
5. Stopping and decision rule: predetermined run length or sample size (or the named
   sequential method), the ban on peeking, implement / keep control / revise / stop,
   and the guardrail veto.
6. Definitions and instrumentation: metric definitions table (metric, formula, source,
   inclusion rules, window) plus the pre launch instrumentation checks and the failure
   each one catches.
7. Self check: every number you could not source, every threshold you proposed
   rather than received, and every assumption I must confirm before launch.

SELF CHECK before returning
→ Is exactly ONE primary KPI designated as the decision metric, not a bundle?
→ Is the primary KPI the highest value level the stated volume can actually detect?
→ Is there a predetermined stopping point and an explicit ban on calling the result early?
→ If more than one metric or variant can drive a decision, is multiplicity corrected?
→ Does at least one guardrail cover revenue or unit economics, so a conversion win
  that loses money is caught?
→ Do the instrumentation checks cover split ratio, duplicate tracking, and cross device rendering?
→ Is every metric reproducible from its definition and source alone?
→ Are assignment unit, analysis unit, and metric grain compatible?
→ Did I avoid presenting clicks or raw conversion rate as proof of business value?
→ Flag any number I could not source and any metric I had to assume.
```

## Prompt versions

The standard prompt above works on any model. Use these variants when you want a different tradeoff.

### Frontier model version

Built for the most capable models (Claude Opus and beyond). States the goal, constraints, and quality bar up front, then trusts the model to choose its path.

```text
You are a senior experimentation strategist. Write an airtight brief for one A/B test
so the result cannot lie to me when the data lands.

First line of your output is the verdict: the single locked primary KPI, stated plainly,
with the supporting brief after it.

CONTEXT
→ Change under test: {{WHAT_YOU_ARE_CHANGING}}
→ Business decision this informs: {{DECISION_TO_MAKE}}
→ North star or top business outcome: {{NORTH_STAR_OUTCOME}}
→ Monthly conversions (transactions or leads): {{MONTHLY_CONVERSIONS}}
→ Weekly unique visitors on the affected flow: {{WEEKLY_VISITORS}}
→ Current conversion rate: {{CURRENT_CONVERSION_RATE}}
→ Assignment unit: {{ASSIGNMENT_UNIT}}
→ Business model notes (AOV, margin, subscription, refund or churn risk): {{BUSINESS_MODEL_NOTES}}
→ Known ways this change could backfire: {{POSSIBLE_HARMS}}

Load bearing principles (keep all of these):
→ Trace a one line causal chain from {{NORTH_STAR_OUTCOME}} down to the driver metric
  this change can move. If you cannot draw that line, say so and stop.
→ Pick ONE primary KPI at the highest value level your volume can actually detect:
  clicks, then behavior or engagement, then transactions or leads, then revenue per
  user, then lifetime value. Conversion rate is a driver, not a destination; never
  approve a conversion lift without checking revenue, unit economics, and retention.
→ If the primary metric is continuous, convert it to a binary above or below a stated
  threshold and justify the threshold.
→ Only the primary KPI decides ship or kill. Secondary metrics (one to three) explain
  the mechanism and are read as directional. If more than one metric or more than two
  variants can trigger a decision, apply and name a multiplicity correction.
→ Add guardrail metrics that catch harm the change was not meant to cause, at minimum
  one covering revenue or unit economics. Name the specific failure each catches and
  mark any threshold I did not give you as a proposal.
→ Set a predetermined run length or sample size and ban peeking, unless a named
  sequential or always valid method is chosen up front. Run whole business cycles. A
  failing guardrail vetoes implementation even if the primary KPI wins.
→ Lock exact definitions and pre launch instrumentation checks: even random split
  (sample ratio mismatch), tracking fires once per event, correct rendering across
  devices, no overlapping test. Confirm assignment unit, analysis unit, and metric
  grain are compatible.

Excellent output satisfies every one of these: exactly one primary KPI at the highest
detectable level with the level above and below named and rejected; a predetermined
stopping point plus an explicit peeking ban; multiplicity corrected when relevant; a
guardrail that catches a conversion win that loses money; instrumentation checks for
split ratio, duplicate tracking, and rendering; and a closing self check that separates
numbers I gave from thresholds you proposed.

Do not invent data, numbers, or statistics. Do not pad with generic optimization advice.
If a required input is missing or vague (especially the north star, monthly conversions,
or how the change could backfire), ask up to five focused questions before proposing
metrics rather than guessing.
```

### Quick version

Five lines or fewer, for when speed matters more than rigor.

```text
Write a one page A/B test brief for {{WHAT_YOU_ARE_CHANGING}} aimed at {{NORTH_STAR_OUTCOME}},
at {{MONTHLY_CONVERSIONS}} conversions and {{WEEKLY_VISITORS}} weekly visitors.
Lock ONE primary KPI at the highest value level that volume can detect, one revenue
guardrail, a predetermined stopping point with no peeking, and the pre launch checks.
Do not invent numbers; flag every threshold you propose and ask if a key input is missing.
```

## How to customize

→ `{{WHAT_YOU_ARE_CHANGING}}`: the specific intervention, e.g. "move the subscription option above the one time purchase on the product page"
→ `{{DECISION_TO_MAKE}}`: what you will do with the result, e.g. "decide whether to roll the subscription first layout to all product pages"
→ `{{NORTH_STAR_OUTCOME}}`: the outcome leadership actually cares about, e.g. "net revenue per user over 90 days" or "active monthly subscribers"
→ `{{MONTHLY_CONVERSIONS}}`: transactions or leads per month, e.g. "about 3,000 orders"
→ `{{WEEKLY_VISITORS}}`: unique visitors or devices (not sessions) on the affected page, e.g. "40,000 uniques"
→ `{{CURRENT_CONVERSION_RATE}}`: the baseline rate on that page or flow, e.g. "2.3%"
→ `{{ASSIGNMENT_UNIT}}`: what receives a condition, e.g. "user" for a consistent repeat visit experience, "page" for content level changes
→ `{{BUSINESS_MODEL_NOTES}}`: AOV, margin, subscription or refund dynamics, e.g. "AOV 90 USD, 20% margin, high refund risk on discounted orders"
→ `{{POSSIBLE_HARMS}}`: how a naive win could hide damage, e.g. "more signups but lower paid conversion later"

## What good output looks like

→ Exactly one primary KPI, sitting at the highest level in the hierarchy the stated volume can detect, with an explicit reason it was chosen over the level above and below it
→ A predetermined stopping point (or a named sequential method) plus an explicit ban on peeking, so no one can call a false win the moment the numbers look good
→ Multiplicity handled honestly: only the primary KPI decides, and any correction is named when more than one metric or variant could trigger a decision
→ Guardrails that name a specific failure mode each (for example, "revenue per user guards against a conversion lift driven by discounting") rather than a generic list
→ Instrumentation checks that catch the silent killers (uneven split, duplicate tracking, broken rendering, overlapping tests) before you trust a single number
→ A closing self check that separates numbers you gave from thresholds the model proposed, so you know exactly what to confirm before launch

## Related prompts

→ [Calculate Sample Size and Test Duration](./calculate-sample-size-and-duration.md): confirm your locked primary KPI is detectable at your traffic and set the predetermined run length your stopping rule depends on
→ [Write a Falsifiable Test Hypothesis](./write-a-falsifiable-hypothesis.md): before you design the test, state the change, predicted effect, and the reason it should work in a form that data can actually refute
→ [QA an Experiment Before Launch](./qa-an-experiment-before-launch.md): run the instrumentation and setup checks in this brief as a pre launch gate so a broken test never ships