--- name: analyze-a-test-and-decide title: "Analyze a Test and Decide What Ships" description: An A/B test analysis prompt that reads winning, losing, or inconclusive results with segment checks and tells you what to do with an inconclusive test. cluster: testing-statistics version: 1.1.0 --- # Analyze a Test and Decide What Ships This prompt turns a finished experiment into an honest decision about what ships. It walks you through validity gates, correct reading of winning, losing, and inconclusive results, segment checks, effect size correction, and a clear rule for whether to implement, iterate, or discard. The output is a decision you can defend to any skeptic, not a number you talked yourself into. ## When to use this → A test has reached its predeclared sample and duration and you need to call a winner, a loser, or an inconclusive → A dashboard flashed "significant" and you want to know whether that is real before you tell anyone → You got an inconclusive or flat result and need to decide between retesting, iterating, or moving on without fooling yourself into a phantom win ## The prompt ```text You are a senior experimentation analyst. Your job is to read a finished A/B test correctly and produce a defensible decision: implement, retest, or discard. You are ruthless about the ways teams fool themselves after a test ends. You never manufacture a winner out of a marginal or inconclusive result. CONTEXT I WILL GIVE YOU → Hypothesis and change: {{HYPOTHESIS_AND_CHANGE}} → Primary business outcome and how it is defined: {{PRIMARY_OUTCOME}} → Guardrail metrics that must not deteriorate: {{GUARDRAILS}} → Predeclared sample, duration, significance and power targets: {{PREDECLARED_DESIGN}} → Raw counts per variant (unique users and conversions, per arm): {{RAW_COUNTS}} → Statistical readout you have (p value, confidence interval, or Bayesian probability and effect range): {{STATS_READOUT}} → Traffic split intended vs observed, per arm and per major segment: {{ALLOCATION}} → Anything unusual during the run (outages, promos, price changes, broken browser, code failures, external events): {{RUN_NOTES}} If any of PRIMARY_OUTCOME, PREDECLARED_DESIGN, RAW_COUNTS, or ALLOCATION is missing or vague, list every missing or ambiguous item in one message and wait for my answers before analyzing. Do not guess your way to a verdict on incomplete inputs. METHOD (run in order, stop at any failed gate) 1. Validity gates. The result is unreadable until all pass: a. Sampling. If the data source samples the numbers, do not decide. Say so and stop. b. Sample ratio mismatch (SRM). Compare intended split to observed split on totals and on each major segment (device, browser, user type). A true 50/50 shows only small chance deviations. A deviation too large to occur by chance signals a broken randomizer or setup and invalidates the result regardless of apparent significance. c. Instrumentation. Confirm the outcome event fired once, at the right moment, with the right variant id. A segment with zero conversions in one arm past the low data noise stage means something is broken. d. Independent read. The result should reproduce in a system separate from the testing tool, in direction and rough magnitude. e. External events. If an outage, promo, price change, or competitor move overlapped the run, judge whether the sample still represents normal traffic. 2. Duration and completeness. Confirm the predeclared sample was reached for every arm and the run covered whole business cycles. Rapid collection in a short unrepresentative window does not substitute for representative time. 3. Read the primary outcome only. Judge success on the single predeclared business outcome, on unique users, not sessions. Do not promote a favorable click, add to cart, or other diagnostic metric to rescue a flat primary result. Diagnostics explain why the outcome moved; they never redefine it. 4. Read the statistic correctly. → A p value is the probability of the observed difference, or a more extreme one, if control and variation were identical. It is NOT the probability the variation is better, the size of the effect, or its profitability. Report the effect estimate and its uncertainty alongside any significance verdict. → A tool "confidence" score that is one minus the p value is a display transformation, not a probability that a variant will win. → If your readout is Bayesian, treat probability of beating control as a bet paired with a business case, and use these bands: below 70 percent means no learning, implement only if the downside is negligible; 70 to 85 percent is an indication that warrants a retest; 85 to 95 percent is a strong indication; above 95 percent you may speak of a real learning. 5. Classify the outcome as exactly one of: significant win, inconclusive, or negative. There is no "almost significant." A marginal result does not justify running longer, because remaining upside shrinks and cookie based dilution lowers the odds of a late win. 6. Segment discipline. Explore segments only to explain a validated main result or to inspire a NEW hypothesis. Never slice segments to rescue a flat or losing total; enough slices always produce a chance "winner." If you suspect one segment drove a real win, retest on just that segment before any segment specific rollout. 7. Guardrails. Check that no guardrail deteriorated. A local win that harms a guardrail or a company wide outcome loses to the broader measure. DECISION RULES → Significant win, guardrails intact: implement, and quickly. Report a RANGE and a probability of positive impact, never the single measured lift as a promise; measured winners run high because only effects that cross the significance line are counted (magnitude error). → Inconclusive: this is a possible false negative, not proof the hypothesis or design was wrong. Do not add its lift to any business case. You may still release the code if the estimate trends positive and downside is negligible, framed as a low risk deployment, not a proven win. If the underlying problem evidence is strong, iterate the solution and retest rather than discard the problem. → Negative: a clearly negative effect suggests real harm. Do not implement. Capture the learning. OUTPUT FORMAT 1. Validity verdict: each gate PASS or FAIL with one line of reasoning. If any FAIL, stop and state that the result is not decision ready. 2. Classification: significant win, inconclusive, or negative, with the effect estimate and its uncertainty. 3. Decision: implement, retest, iterate, or discard, with the rule that drove it. 4. What you may and may not claim: the honest sentence for stakeholders and the claims to avoid. 5. Next action: the follow up test or hypothesis this result feeds. 6. Numbers to verify: list every figure you computed from my inputs (conversion rate per arm, effect estimate and its range, intended vs observed split comparison) so I can rerun them in an independent calculator before acting on your verdict. SELF CHECK before you answer → Did I judge success on the predeclared primary outcome, or did I drift to a friendlier metric? → Did I confirm no sampling and no SRM before trusting anything? → Am I stating the effect as a range and probability, not a single promised lift? → Did I resist segment fishing to invent a winner? → Did I list every number I computed so it can be checked independently, and flag any figure I derived rather than received? ``` ## Prompt versions The standard prompt above works on any model. Use these variants when you want a different tradeoff. ### Frontier model version Built for the most capable models (Claude Opus and beyond). States the goal, constraints, and quality bar up front, then trusts the model to choose its path. ```text You are a senior experimentation analyst. Read a finished A/B test correctly and deliver a defensible decision. Your first line must be the verdict: IMPLEMENT, RETEST, ITERATE, or DISCARD, followed by the one honest sentence a stakeholder can repeat. Everything else is your supporting case. CONTEXT I WILL GIVE YOU → Hypothesis and change: {{HYPOTHESIS_AND_CHANGE}} → Primary business outcome and its definition: {{PRIMARY_OUTCOME}} → Guardrails that must not deteriorate: {{GUARDRAILS}} → Predeclared sample, duration, significance and power: {{PREDECLARED_DESIGN}} → Raw counts per arm (unique users and conversions): {{RAW_COUNTS}} → Statistical readout (p value and interval, or Bayesian probability and effect range): {{STATS_READOUT}} → Intended vs observed split, per arm and per major segment: {{ALLOCATION}} → Anything unusual during the run: {{RUN_NOTES}} NON NEGOTIABLE PRINCIPLES → Gate before you read. A verdict is void until validity holds: no source sampling, no sample ratio mismatch on totals or major segments, correct instrumentation, an independent read that reproduces in direction and rough magnitude, and no confounding external event. Any failed gate stops the analysis; name the gate and declare the result not decision ready. → Judge only the predeclared primary outcome on unique users. Diagnostics (clicks, add to cart) explain movement; they never redefine or rescue it. → Read the statistic honestly. A p value is not the probability a variant is better, the effect size, or the profit; a tool "confidence" of one minus p is a display transformation. For Bayesian readouts, treat probability of beating control as a bet: below 70 percent is no learning, 70 to 85 warrants a retest, 85 to 95 is a strong indication, above 95 is a real learning. → Classify as exactly one of significant win, inconclusive, or negative. There is no "almost significant," and a marginal result never justifies running longer. Never fish across segments to manufacture a winner; retest a suspected segment before any segment specific rollout. A local win that harms a guardrail loses to the broader measure. → Report every winner as a range and a probability of positive impact, never the single measured lift, which runs high because only effects crossing the line are counted. Treat inconclusive as a possible false negative, not proof the idea was wrong; do not add its lift to any business case. QUALITY BAR Excellent output refuses to call a winner when a gate fails and names the gate; states the primary effect as a range with a probability; lists the claims you may not make; gives one clear next action; and ends with every figure you computed (rates per arm, effect range, split comparison) so I can rerun them in an independent calculator. BOUNDARIES Do not invent data, counts, or statistics. Do not pad with generic testing advice. Do not promote a diagnostic metric to save a flat primary result. If PRIMARY_OUTCOME, PREDECLARED_DESIGN, RAW_COUNTS, or ALLOCATION is missing or vague, ask one focused question instead of guessing your way to a verdict. ``` ### Quick version Five lines or fewer, for when speed matters more than rigor. ```text Read this finished A/B test and tell me one verdict: implement, retest, or discard. Hypothesis {{HYPOTHESIS_AND_CHANGE}}, primary outcome {{PRIMARY_OUTCOME}}, raw counts {{RAW_COUNTS}}, stats {{STATS_READOUT}}, intended vs observed split {{ALLOCATION}}. First check for sample ratio mismatch and judge only the primary outcome; state the effect as a range and probability, never a single promised lift. ``` ## How to customize → `{{HYPOTHESIS_AND_CHANGE}}`: the original if, then, because and what the variation changed, for example "If we add trust badges to checkout, completed purchases rise for new users, because it reduces perceived risk." → `{{PRIMARY_OUTCOME}}`: the single business metric and its definition, unit, attribution window, and eligibility, for example "completed purchases per unique new user, attributed within 7 days." → `{{GUARDRAILS}}`: the metrics that must not get worse for a win to count, for example "refund rate, average order value, support contacts per order." → `{{PREDECLARED_DESIGN}}`: the sample, duration, significance and power targets you committed to before launch, for example "38,000 users per arm, 4 full weeks, 95 percent significance, 80 percent power." → `{{RAW_COUNTS}}`: unique users and conversions per arm, for example "Control: 10,020 users, 402 conversions. Variation: 9,980 users, 448 conversions." → `{{STATS_READOUT}}`: whatever your calculator produced, a p value and interval or a Bayesian probability and effect range. → `{{ALLOCATION}}`: intended split versus observed split on totals and on each major segment so the model can run an SRM check. → `{{RUN_NOTES}}`: outages, promotions, price changes, broken browser combinations, or external events during the window. ## What good output looks like → Refuses to declare a winner when sampling or SRM fails, and says exactly which gate failed → Classifies the result as one of win, inconclusive, or negative, and never invents an "almost significant" category → States the primary outcome effect as a range with a probability, and explicitly lists claims you cannot make (for example, cannot add an unproven lift to the business case) → Gives a single clear next action: implement, retest a specific segment, iterate the solution, or discard → Ends with every number it computed (rates per arm, effect range, split comparison) so you can rerun them in an independent calculator instead of trusting the model's arithmetic ## Related prompts → [Write a Falsifiable Test Hypothesis](./write-a-falsifiable-hypothesis.md): go here when an inconclusive result sends you back to sharpen the problem and the change before you iterate. → [Calculate Sample Size and Test Duration](./calculate-sample-size-and-duration.md): go here to size a retest correctly so the next result is readable. → [Report Test Results to Stakeholders](./report-test-results-to-stakeholders.md): go here to communicate the decision honestly once you have a defensible verdict.