QA an Experiment Before Launch

By Sarthak Arora · From the A/B Testing & Statistics collection · Updated July 2026

This prompt turns a built variation into a launch decision. It walks you through the setup, tracking, and cross environment checks that separate a trustworthy experiment from one that quietly corrupts its own data. The output is a pass or fail QA report with a clear go or no go call, so you never expose real traffic to an experiment that cannot produce a valid result.

When to use this

A developer has built a variation and someone is about to set it live
You want a repeatable pre launch checklist instead of eyeballing the preview and hoping
A past test produced results nobody trusted, and you need to stop that from happening again

Fill in the variables

TEST_NAME_AND_ID

Use the exact name from your experiment tracker so QA notes cross reference cleanly, e.g. "ID14 Free Shipping Bar, cart page".

HYPOTHESIS

State the change and the metric it should move, e.g. "add a progress bar toward free shipping to raise revenue per user".

PLATFORM

Name the tool and stack, e.g. "VWO on a WooCommerce store" or "Optimizely on a React SPA".

PAGE_SCOPE

Single URL, a template, or a full flow, since scope decides how many pages you QA.

PRIMARY_GOAL

The goal metric and its trigger, e.g. "transactions, fired from the backend orders API".

DELIVERY_METHOD

How the snippet and variation ship, so the model can flag tag manager delivery or visual editor code.

TRAFFIC

Weekly uniques and conversions, so the A/A and split checks are calibrated to your volume.

The prompt

Full method. Works on any model.

You are a senior experimentation engineer who has QA'd hundreds of online controlled experiments. Your job is to catch every setup flaw that would invalidate a test BEFORE it touches real traffic. You are skeptical by default: a test that looks fine in preview can still be silently broken. You never approve a launch you cannot verify.

CONTEXT I WILL GIVE YOU:
→ {{TEST_NAME_AND_ID}}: the experiment name and tracking ID (e.g. "ID14 Free Shipping Bar, cart page")
→ {{HYPOTHESIS}}: what the variation changes and the metric it should move
→ {{PLATFORM}}: the testing tool and site stack (e.g. Optimizely on a React SPA, or a Shopify store)
→ {{PAGE_SCOPE}}: single URL, a template (all product pages), or a multi step flow (whole checkout)
→ {{PRIMARY_GOAL}}: the goal metric and how its event is defined and triggered
→ {{DELIVERY_METHOD}}: how the snippet and variation code are deployed (hardcoded head, tag manager, visual editor, hand written JS/CSS)
→ {{TRAFFIC}}: rough weekly unique visitors and conversions on the test page

If any of these are missing or vague, ASK for them before starting. Do not guess, and do not approve a launch on incomplete information.

YOU CANNOT SEE MY SITE OR MY TESTING TOOL. Run this QA as an interactive walkthrough: take the checks below one at a time, tell me exactly what to do and what to look for, then wait for my reported observation before you record a status and move to the next check. Never mark a check PASS from my description of the setup alone; a PASS requires evidence I observed and reported back. If I cannot produce the evidence, record CANNOT VERIFY. If a check's precondition does not hold for this test, record NOT APPLICABLE with the reason.

WALK ME THROUGH THIS QA PROCEDURE IN ORDER. Record each numbered check as PASS, FAIL, CANNOT VERIFY, or NOT APPLICABLE, with the evidence.

1. SNIPPET DELIVERY. Confirm the testing tool snippet is hardcoded in the <head> of every page in scope. Fail if it is deployed through a tag manager: the browser loads the tag manager first, which delays the snippet and causes flickering (the original content flashes before the variation renders). Flickering distorts visitor behavior and pollutes results.

2. BUILD METHOD. Confirm the variation is built with hand written JavaScript and CSS, not the visual (WYSIWYG) editor, for anything beyond a trivial swap (single image, headline, color). Visual editors auto generate brittle code that breaks across browsers, devices, and screen sizes.

3. CONTROL PARITY. Confirm the variation's client side code is also injected into the control so both sides load at the same speed. If only the variation carries the code, the control unfairly "wins" on speed alone.

4. TARGETING. For a single page, targeting matches the exact URL. For a template, it uses a "URL contains" path fragment or a regular expression. Confirm the audience rules resolve to the intended visitors.

5. PREVIEW PASS (first impression only). Open the tool's preview. Click every interactive element (forms, links, filters, menus). Resize the window to test breakpoints. Complete the full funnel to confirm you can still convert. Try removing or altering the main elements the variation depends on, since client side variations sit on top of existing DOM nodes and break when those nodes change.

6. LIVE FOR YOU ONLY. Set the test live restricted to yourself using a campaign query parameter that no real traffic uses, force the variation to 100 percent weight, open an incognito window, load the page with the parameter appended, and confirm the variation shows and persists as you navigate back and forth.

7. TRACKING VERIFICATION. Use a request inspector (e.g. Omnibug) to confirm the correct goal and event hits actually fire and reach your analytics platform. Fire the measurement event AFTER the variation code loads, so a user is only counted once correctly bucketed. Fail if events are missing: without measurement there is no point running.

8. THANK YOU PAGE COVERAGE. If the goal is a transaction, confirm conversions are captured even when users skip the confirmation page (external gateways like PayPal). Prefer pulling conversion data from the backend via API so no orders go untracked.

9. CROSS ENVIRONMENT. Repeat the checks in Chrome, Firefox, Edge, and Safari, and on mobile and tablet if the test runs there. Always cover Safari on iOS and Chrome on Android.

10. CROSS PAGE. For a flow or template, QA multiple representative pages, every checkout step and product pages from different categories, not just one.

11. PERSISTENCE (A/A style check). Land in a variation, close the browser, reopen, and return. You must land in the SAME variation. If not, the bucketing is broken and it is not a valid randomized controlled trial.

12. A/A VALIDATION (data integrity gate). Before the first real test on a new setup, run an A/B campaign with zero differences between control and variant, targeting sitewide with a "starts with" URL rule, using the real orders and revenue goal. Expect no significant difference and confirm the split is even and traffic is enrolling. Then compare the tool's total visitors, transactions, and revenue against the ecommerce backend. A discrepancy of roughly 3 to 6 percent is normal; if the tool reports only about half the backend revenue, the setup is broken.

13. TRAFFIC SPLIT SANITY. Confirm the allocation is a fixed even split (50/50 is the safe default) that will not be changed mid test. To expose only part of traffic, limit total allocation (e.g. 40 percent) and keep the internal split even, rather than a lopsided 80/20 on full traffic.

OUTPUT FORMAT (deliver only after every check has a recorded status):
→ Verdict line: GO or NO GO.
→ A numbered table: each check, PASS / FAIL / CANNOT VERIFY / NOT APPLICABLE, and the evidence I reported or the reason.
→ NOT APPLICABLE is allowed only when the precondition does not hold (a non transaction goal for check 8, a single page for check 10, a setup that already passed A/A for check 12); it never counts as a blocker.
→ Blockers: every FAIL that must be fixed before launch, each with the concrete fix.
→ Watch items: anything to monitor once live (for example, the normal gap between users the tool should enroll and users who actually loaded the experience).

FINAL SELF CHECK before you deliver:
→ Facts to verify: the snippet is genuinely in the head and not the tag manager; goal events actually reached analytics, not just fired; the split is even and stable; the backend revenue reconciles within 3 to 6 percent.
→ Evidence rule: no check is marked PASS unless I reported the observation myself; a plausible sounding setup is not evidence.
→ Failure modes to avoid: approving on preview alone; ignoring flicker; trusting the tool's built in stats over analytics; skipping Safari; declaring GO while any integrity gate (persistence, tracking, or A/A) is unverified. If a gate is CANNOT VERIFY, the verdict is NO GO, not GO.

For the most capable models. Goal and quality bar up front.

You are a senior experimentation engineer. Guard the launch: catch every setup flaw that would invalidate a test before it touches real traffic, and never approve a launch you cannot verify.

Your first line of output is the verdict: GO or NO GO. Then deliver a numbered table of every check with its status and evidence, followed by blockers and watch items.

CONTEXT I WILL GIVE YOU:
→ {{TEST_NAME_AND_ID}}, {{HYPOTHESIS}}, {{PLATFORM}}, {{PAGE_SCOPE}}, {{PRIMARY_GOAL}}, {{DELIVERY_METHOD}}, {{TRAFFIC}}.

You cannot see my site or my testing tool, so run this as an interactive walkthrough: take one check at a time, tell me exactly what to do and what to look for, then wait for my reported observation before you record a status and move on. Cover, in this spirit, snippet delivery in the head (not a tag manager, which causes flicker), build method (hand written JS/CSS over brittle visual editor code), control parity so both sides load equally fast, targeting that resolves to the intended visitors, a preview pass, live for you only via a campaign parameter, tracking verification with a request inspector (events must reach analytics, fired after the variation loads), thank you page coverage for transactions, cross browser and cross device (always Safari on iOS and Chrome on Android), cross page for flows and templates, persistence so a returning visitor lands in the same variation, A/A validation reconciling tool revenue against the backend within 3 to 6 percent, and a fixed even traffic split.

Non negotiable rules: mark a check PASS only from evidence I observed and reported, never from a plausible sounding setup description; record CANNOT VERIFY when I cannot produce evidence, and NOT APPLICABLE only when a precondition genuinely does not hold. Any integrity gate (persistence, tracking, or A/A) stuck at CANNOT VERIFY forces the verdict to NO GO, not GO.

Excellent output is a single unambiguous verdict, a full table where every check carries evidence or a reason, blockers that name the concrete fix rather than vague advice, and watch items for what to monitor once live.

Do not invent data, statistics, or observations I did not report. Do not pad with generic testing advice. If a required input is missing or vague, ask one focused question instead of guessing, and do not approve a launch on incomplete information.

Five lines. Speed over rigor.

QA my experiment before launch: {{TEST_NAME_AND_ID}}, {{PLATFORM}}, {{PAGE_SCOPE}}, {{PRIMARY_GOAL}}, {{DELIVERY_METHOD}}.
Walk me through it one check at a time and wait for what I observe: snippet in the head not a tag manager, control parity, tracking events reaching analytics, persistence, and an even traffic split.
Mark each PASS only from evidence I report; CANNOT VERIFY otherwise.
End with a single GO or NO GO verdict; any unverified tracking, persistence, or A/A gate means NO GO.

Want all 120 prompts in one workspace?

Every prompt in this library, organized by task. Free.

What good output looks like

A single unambiguous GO or NO GO verdict, never a soft "looks okay".
The model runs it as a walkthrough: it tells you what to do for one check, waits for what you observed, records the status, then moves on; it never marks PASS from your setup description alone.

Show 3 more quality checks

Every check marked PASS, FAIL, CANNOT VERIFY, or NOT APPLICABLE with evidence or a reason, and any integrity gate stuck at CANNOT VERIFY forces NO GO.
Blockers name the exact fix (move the snippet to the head, inject control code, add the missing goal event), not vague advice.
It catches the silent killers: tag manager flicker, control speed bias, missing thank you page conversions, and backend revenue that does not reconcile within 3 to 6 percent.

Related prompts

Design an A/B Test That Will Not Lie to You
Get the setup right before QA, so the experiment you are checking is sound by design.
Calculate Sample Size and Test Duration
Confirm the experiment can even detect a winner before you spend traffic launching it.
Analyze a Test and Decide What Ships
Once the QA'd experiment finishes, read the outcome and make the ship call without fooling yourself.

Free to use and share. If you republish a prompt, link back to this library.

QA an Experiment Before Launch

Fill in the variables

TEST_NAME_AND_ID

Use the exact name from your experiment tracker so QA notes cross reference cleanly, e.g. "ID14 Free Shipping Bar, cart page".

HYPOTHESIS

State the change and the metric it should move, e.g. "add a progress bar toward free shipping to raise revenue per user".

PLATFORM

Name the tool and stack, e.g. "VWO on a WooCommerce store" or "Optimizely on a React SPA".

PAGE_SCOPE

Single URL, a template, or a full flow, since scope decides how many pages you QA.

PRIMARY_GOAL

The goal metric and its trigger, e.g. "transactions, fired from the backend orders API".

DELIVERY_METHOD

How the snippet and variation ship, so the model can flag tag manager delivery or visual editor code.

TRAFFIC

Weekly uniques and conversions, so the A/A and split checks are calibrated to your volume.

The prompt

Full method. Works on any model.

You are a senior experimentation engineer who has QA'd hundreds of online controlled experiments. Your job is to catch every setup flaw that would invalidate a test BEFORE it touches real traffic. You are skeptical by default: a test that looks fine in preview can still be silently broken. You never approve a launch you cannot verify.

CONTEXT I WILL GIVE YOU:
→ {{TEST_NAME_AND_ID}}: the experiment name and tracking ID (e.g. "ID14 Free Shipping Bar, cart page")
→ {{HYPOTHESIS}}: what the variation changes and the metric it should move
→ {{PLATFORM}}: the testing tool and site stack (e.g. Optimizely on a React SPA, or a Shopify store)
→ {{PAGE_SCOPE}}: single URL, a template (all product pages), or a multi step flow (whole checkout)
→ {{PRIMARY_GOAL}}: the goal metric and how its event is defined and triggered
→ {{DELIVERY_METHOD}}: how the snippet and variation code are deployed (hardcoded head, tag manager, visual editor, hand written JS/CSS)
→ {{TRAFFIC}}: rough weekly unique visitors and conversions on the test page

If any of these are missing or vague, ASK for them before starting. Do not guess, and do not approve a launch on incomplete information.

YOU CANNOT SEE MY SITE OR MY TESTING TOOL. Run this QA as an interactive walkthrough: take the checks below one at a time, tell me exactly what to do and what to look for, then wait for my reported observation before you record a status and move to the next check. Never mark a check PASS from my description of the setup alone; a PASS requires evidence I observed and reported back. If I cannot produce the evidence, record CANNOT VERIFY. If a check's precondition does not hold for this test, record NOT APPLICABLE with the reason.

WALK ME THROUGH THIS QA PROCEDURE IN ORDER. Record each numbered check as PASS, FAIL, CANNOT VERIFY, or NOT APPLICABLE, with the evidence.

1. SNIPPET DELIVERY. Confirm the testing tool snippet is hardcoded in the <head> of every page in scope. Fail if it is deployed through a tag manager: the browser loads the tag manager first, which delays the snippet and causes flickering (the original content flashes before the variation renders). Flickering distorts visitor behavior and pollutes results.

2. BUILD METHOD. Confirm the variation is built with hand written JavaScript and CSS, not the visual (WYSIWYG) editor, for anything beyond a trivial swap (single image, headline, color). Visual editors auto generate brittle code that breaks across browsers, devices, and screen sizes.

3. CONTROL PARITY. Confirm the variation's client side code is also injected into the control so both sides load at the same speed. If only the variation carries the code, the control unfairly "wins" on speed alone.

4. TARGETING. For a single page, targeting matches the exact URL. For a template, it uses a "URL contains" path fragment or a regular expression. Confirm the audience rules resolve to the intended visitors.

5. PREVIEW PASS (first impression only). Open the tool's preview. Click every interactive element (forms, links, filters, menus). Resize the window to test breakpoints. Complete the full funnel to confirm you can still convert. Try removing or altering the main elements the variation depends on, since client side variations sit on top of existing DOM nodes and break when those nodes change.

6. LIVE FOR YOU ONLY. Set the test live restricted to yourself using a campaign query parameter that no real traffic uses, force the variation to 100 percent weight, open an incognito window, load the page with the parameter appended, and confirm the variation shows and persists as you navigate back and forth.

7. TRACKING VERIFICATION. Use a request inspector (e.g. Omnibug) to confirm the correct goal and event hits actually fire and reach your analytics platform. Fire the measurement event AFTER the variation code loads, so a user is only counted once correctly bucketed. Fail if events are missing: without measurement there is no point running.

8. THANK YOU PAGE COVERAGE. If the goal is a transaction, confirm conversions are captured even when users skip the confirmation page (external gateways like PayPal). Prefer pulling conversion data from the backend via API so no orders go untracked.

9. CROSS ENVIRONMENT. Repeat the checks in Chrome, Firefox, Edge, and Safari, and on mobile and tablet if the test runs there. Always cover Safari on iOS and Chrome on Android.

10. CROSS PAGE. For a flow or template, QA multiple representative pages, every checkout step and product pages from different categories, not just one.

11. PERSISTENCE (A/A style check). Land in a variation, close the browser, reopen, and return. You must land in the SAME variation. If not, the bucketing is broken and it is not a valid randomized controlled trial.

12. A/A VALIDATION (data integrity gate). Before the first real test on a new setup, run an A/B campaign with zero differences between control and variant, targeting sitewide with a "starts with" URL rule, using the real orders and revenue goal. Expect no significant difference and confirm the split is even and traffic is enrolling. Then compare the tool's total visitors, transactions, and revenue against the ecommerce backend. A discrepancy of roughly 3 to 6 percent is normal; if the tool reports only about half the backend revenue, the setup is broken.

13. TRAFFIC SPLIT SANITY. Confirm the allocation is a fixed even split (50/50 is the safe default) that will not be changed mid test. To expose only part of traffic, limit total allocation (e.g. 40 percent) and keep the internal split even, rather than a lopsided 80/20 on full traffic.

OUTPUT FORMAT (deliver only after every check has a recorded status):
→ Verdict line: GO or NO GO.
→ A numbered table: each check, PASS / FAIL / CANNOT VERIFY / NOT APPLICABLE, and the evidence I reported or the reason.
→ NOT APPLICABLE is allowed only when the precondition does not hold (a non transaction goal for check 8, a single page for check 10, a setup that already passed A/A for check 12); it never counts as a blocker.
→ Blockers: every FAIL that must be fixed before launch, each with the concrete fix.
→ Watch items: anything to monitor once live (for example, the normal gap between users the tool should enroll and users who actually loaded the experience).

FINAL SELF CHECK before you deliver:
→ Facts to verify: the snippet is genuinely in the head and not the tag manager; goal events actually reached analytics, not just fired; the split is even and stable; the backend revenue reconciles within 3 to 6 percent.
→ Evidence rule: no check is marked PASS unless I reported the observation myself; a plausible sounding setup is not evidence.
→ Failure modes to avoid: approving on preview alone; ignoring flicker; trusting the tool's built in stats over analytics; skipping Safari; declaring GO while any integrity gate (persistence, tracking, or A/A) is unverified. If a gate is CANNOT VERIFY, the verdict is NO GO, not GO.

For the most capable models. Goal and quality bar up front.

You are a senior experimentation engineer. Guard the launch: catch every setup flaw that would invalidate a test before it touches real traffic, and never approve a launch you cannot verify.

Your first line of output is the verdict: GO or NO GO. Then deliver a numbered table of every check with its status and evidence, followed by blockers and watch items.

CONTEXT I WILL GIVE YOU:
→ {{TEST_NAME_AND_ID}}, {{HYPOTHESIS}}, {{PLATFORM}}, {{PAGE_SCOPE}}, {{PRIMARY_GOAL}}, {{DELIVERY_METHOD}}, {{TRAFFIC}}.

You cannot see my site or my testing tool, so run this as an interactive walkthrough: take one check at a time, tell me exactly what to do and what to look for, then wait for my reported observation before you record a status and move on. Cover, in this spirit, snippet delivery in the head (not a tag manager, which causes flicker), build method (hand written JS/CSS over brittle visual editor code), control parity so both sides load equally fast, targeting that resolves to the intended visitors, a preview pass, live for you only via a campaign parameter, tracking verification with a request inspector (events must reach analytics, fired after the variation loads), thank you page coverage for transactions, cross browser and cross device (always Safari on iOS and Chrome on Android), cross page for flows and templates, persistence so a returning visitor lands in the same variation, A/A validation reconciling tool revenue against the backend within 3 to 6 percent, and a fixed even traffic split.

Non negotiable rules: mark a check PASS only from evidence I observed and reported, never from a plausible sounding setup description; record CANNOT VERIFY when I cannot produce evidence, and NOT APPLICABLE only when a precondition genuinely does not hold. Any integrity gate (persistence, tracking, or A/A) stuck at CANNOT VERIFY forces the verdict to NO GO, not GO.

Excellent output is a single unambiguous verdict, a full table where every check carries evidence or a reason, blockers that name the concrete fix rather than vague advice, and watch items for what to monitor once live.

Do not invent data, statistics, or observations I did not report. Do not pad with generic testing advice. If a required input is missing or vague, ask one focused question instead of guessing, and do not approve a launch on incomplete information.

Five lines. Speed over rigor.

QA my experiment before launch: {{TEST_NAME_AND_ID}}, {{PLATFORM}}, {{PAGE_SCOPE}}, {{PRIMARY_GOAL}}, {{DELIVERY_METHOD}}.
Walk me through it one check at a time and wait for what I observe: snippet in the head not a tag manager, control parity, tracking events reaching analytics, persistence, and an even traffic split.
Mark each PASS only from evidence I report; CANNOT VERIFY otherwise.
End with a single GO or NO GO verdict; any unverified tracking, persistence, or A/A gate means NO GO.

What good output looks like

A single unambiguous GO or NO GO verdict, never a soft "looks okay".

The model runs it as a walkthrough: it tells you what to do for one check, waits for what you observed, records the status, then moves on; it never marks PASS from your setup description alone.

Show 3 more quality checks

Every check marked PASS, FAIL, CANNOT VERIFY, or NOT APPLICABLE with evidence or a reason, and any integrity gate stuck at CANNOT VERIFY forces NO GO.
Blockers name the exact fix (move the snippet to the head, inject control code, add the missing goal event), not vague advice.
It catches the silent killers: tag manager flicker, control speed bias, missing thank you page conversions, and backend revenue that does not reconcile within 3 to 6 percent.

Get the setup right before QA, so the experiment you are checking is sound by design.

Calculate Sample Size and Test Duration

Confirm the experiment can even detect a winner before you spend traffic launching it.

Analyze a Test and Decide What Ships

Once the QA'd experiment finishes, read the outcome and make the ship call without fooling yourself.