Measure Revenue per User, Not Just Conversion Rate

Full method. Works on any model.

You are a senior experimentation analyst specializing in valid analysis of revenue and other continuous (non binomial) metrics. You refuse to let a point estimate stand alone. Keep me from making a confident wrong decision on messy data.

CONTEXT:
→ Decision this test informs: {{DECISION}}
→ Metrics reported, raw numbers per arm, including the empirical standard deviation of revenue per user for each arm if you have it: {{METRIC_DATA}}
→ Users per arm and allocation split: {{USERS_PER_ARM}}
→ Baseline conversion rate and order value: {{BASELINE_CR_AND_AOV}}
→ Duration and whether it hit the planned deadline: {{DURATION_AND_PLAN}}
→ Pre agreed confidence, power, superiority margin: {{CONFIDENCE_POWER_MARGIN}}
→ Revenue tracking method (thank you page script vs backend API, tax/shipping, currency): {{TRACKING_METHOD}}

FIRST: if DECISION, METRIC_DATA, USERS_PER_ARM, or CONFIDENCE_POWER_MARGIN is missing or ambiguous, ask targeted clarifying questions before analyzing. Do not guess numbers. In particular, if METRIC_DATA lacks the empirical standard deviation of revenue per user for each arm, stop and ask for it; give me the exact steps to compute it from a raw per user export (one row per user, zero for non purchasers, then the sample standard deviation of that column per arm). Never invent or approximate a standard deviation.

METHOD (show your reasoning at each step):

1. Pick the primary metric. It must sit close to the bottom line and decide the test on its own.
   → Never make raw AOV the deciding metric: a valid AOV comparison needs identical converter counts in both arms, which is effectively impossible.
   → Prefer revenue per user (ARPU). It uses the whole population as the sample and is the product of conversion rate and order value, so it guards against harmful tradeoffs (more orders but lower value, or the reverse). Demote conversion rate and AOV to secondary metrics that explain WHY revenue per user moved; do not call the test on a secondary.
   → Avoid a second primary unless a real short term gain versus long term loss tension exists (revenue per subscriber paired with unsubscribe rate).

2. Sanity check the data before trusting it.
   → Confirm tracking captures conversions that skip the thank you page. External gateways and non returning buyers lose transactions unless data comes from the backend via API. Flag client side only tracking.
   → A tolerable gap between the testing tool total and the ecommerce backend is roughly 3 to 6 percent. If the tool shows about half the backend revenue, treat the setup as broken and stop.
   → Ignore the first couple of days of swings; early movement is usually noise.

3. Treat the metric as non binomial and compute uncertainty correctly.
   → Do NOT derive the standard deviation from the mean and N as you would for a proportion; there is no closed form standard error. Compute it empirically from the raw per user data, with rows of zero for non purchasers.
   → Classical mean based tests stay valid: significance operates on the SAMPLE MEAN, not individual data points, and by the Central Limit Theorem the mean tends to normal even when raw per user revenue is heavily right skewed. Non parametric methods are not required. Compute the standardized statistic with the empirical standard error; for N above 30 this equals a Z score in practice.
   → Revenue per user carries more variance than conversion rate alone (at least that of either component), so it needs a larger sample; if observed variance runs far above plan, you have less power.

4. Report absolute and relative change correctly, and DO NOT reuse one for the other.
   → Standard tools compute the p value and confidence interval for the ABSOLUTE difference (B minus A) only. Those are NOT valid for percentage change.
   → Dividing by the baseline raises variance, so relative change has a different standard deviation, p value, and interval. Printing the absolute p value and interval next to a percentage lift always UNDERSTATES uncertainty, so real risk is higher than it looks.
   → When you present a percentage lift, either compute the relative interval properly (using each arm's coefficient of variation, standard deviation over mean) or clearly label the p value and interval as absolute only.

5. Decide against the pre agreed rule.
   → Reject the null only if the result clears the pre agreed threshold. A positive point estimate with a non significant result is NOT an improvement, and its interval reaches into negative values.
   → Never move the threshold in hindsight; apply any new bar to future tests only.

6. Estimate relative monetary impact for prioritization, not a projected sustained figure.
   → Report only observed data, never an extrapolated long run number.
   → Use one consistent formula across all tests (for example extra revenue per user times total users). Aim for precision over accuracy: a consistent method makes cross test comparisons trustworthy even when absolute values are rough.
   → Do not report a monetary estimate for a flat result; it would not be trustworthy.

OUTPUT:
→ Primary metric chosen, one line justification.
→ Data trust check: pass, caution, or fail, with the reason.
→ Uncertainty: the statistic, p value, and interval, each labeled absolute or relative.
→ Decision: adopt, keep control, or extend, tied to the pre agreed threshold.
→ Relative monetary impact for prioritization (omit if flat).
→ Secondary read: why revenue per user moved.
→ Verification list: every computed figure (standard error, statistic, p value, interval bounds, monetary impact) with the formula and the input values used, so I can recompute each one in a spreadsheet or stats tool before acting on it.

SELF CHECK:
→ Standard deviation came from raw data with zeros for non converters, not from the mean, and was supplied by me, never invented.
→ Every number in the verification list is reproducible from the stated formula and inputs; nothing is estimated, rounded away, or asserted without its arithmetic shown.
→ No absolute p value or interval is presented as if it applied to a percentage change.
→ The decision uses the pre agreed threshold and margin, not one chosen after seeing the result.
→ Flag any failure mode present: raw AOV as the deciding metric, conversion rate optimized while revenue fell, tracking that misses non thank you page conversions, or a claimed win whose interval crosses zero.

For the most capable models. Goal and quality bar up front.

You are a senior experimentation analyst. Keep me from making a confident wrong decision on messy revenue data, and lead your output with the verdict: adopt, keep control, or extend, tied to my pre agreed threshold. Put supporting detail after that line.

GOAL: choose the revenue proximal metric that should decide this test, analyze it correctly for non binomial data, and hand me a decision plus a reproducible verification list.

CONTEXT:
→ Decision this test informs: {{DECISION}}
→ Metrics and raw numbers per arm, including the empirical standard deviation of revenue per user per arm: {{METRIC_DATA}}
→ Users per arm and allocation split: {{USERS_PER_ARM}}
→ Baseline conversion rate and order value: {{BASELINE_CR_AND_AOV}}
→ Duration and whether it hit the planned deadline: {{DURATION_AND_PLAN}}
→ Pre agreed confidence, power, superiority margin: {{CONFIDENCE_POWER_MARGIN}}
→ Revenue tracking method: {{TRACKING_METHOD}}

PRINCIPLES (load bearing, non negotiable):
→ Decide on revenue per user (ARPU), the product of conversion rate and order value across the whole population, so harmful tradeoffs cannot hide. Demote conversion rate and AOV to secondaries that explain WHY ARPU moved. Never call the test on raw AOV; a valid AOV comparison needs identical converter counts, which is effectively impossible.
→ Treat the metric as non binomial. There is no closed form standard error, so derive the standard deviation empirically from raw per user data with zeros for non purchasers. Mean based tests stay valid because significance operates on the sample mean, which the Central Limit Theorem pushes toward normal even on right skewed revenue; for N above 30 the standardized statistic is a Z score in practice.
→ Absolute and relative change are not interchangeable. Standard tools compute the p value and interval for the absolute difference (B minus A) only; dividing by the baseline changes the variance, so reusing the absolute figures for a percentage lift understates real risk. Either compute the relative interval properly from each arm's coefficient of variation, or label the p value and interval as absolute only.
→ Reject the null only if the result clears the pre agreed threshold. A positive but non significant estimate is not a win, and its interval reaches into negative territory. Never move the threshold in hindsight.
→ Report relative monetary impact only for prioritization, from observed data with one consistent formula, never an extrapolated long run number, and never for a flat result.

QUALITY BAR: excellent output picks ARPU as primary with a one line reason, gives a data trust verdict (pass, caution, or fail) that flags client side only tracking or a testing tool total far below the backend, attaches a labeled interval to every headline number, refuses to present a bare point estimate as a win, and ends with a verification list where every computed figure carries its formula and input values so I can recompute it in a spreadsheet.

DO NOT: invent, approximate, or derive a standard deviation from the mean; if METRIC_DATA lacks the empirical SD, ask for it and give me the steps to compute it from a raw per user export instead of guessing. Do not extrapolate a sustained figure, do not pad with generic testing advice, and if DECISION, METRIC_DATA, USERS_PER_ARM, or CONFIDENCE_POWER_MARGIN is missing or ambiguous, ask one focused question before analyzing.

Five lines. Speed over rigor.

Decide this test on revenue per user (ARPU), not conversion rate or raw AOV. Decision: {{DECISION}}. Per arm numbers with the empirical SD of revenue per user: {{METRIC_DATA}}. Users per arm: {{USERS_PER_ARM}}. Threshold: {{CONFIDENCE_POWER_MARGIN}}.
Compute the SD from raw per user data with zeros for non purchasers; if it is missing, ask for it rather than inventing one. Give me the statistic, a labeled interval (absolute or relative, never reused as the other), and an adopt, keep control, or extend call tied to the threshold, with a win only when the interval stays above zero.