EXP-0341RUNNING
Hero headline test
Running 4d 12h
A — "Built to last."
2.31%
B — "Yours forever, basically."
▲2.74%
Confidence86%
B leading. Holding for one more cohort before calling.
Agentic A/B Testing
Two ideas walk in.One ships.
Your AI lab partner runs the experiment, watches the data, and quietly replaces the loser. You see the winner in the changelog.
RUNNING·Cart-page CTA·64% confidenceWINNER·Hero headline B·+18% convKILLED·Free-shipping banner C·-3.2% convSTARTED·Pricing page·0/2000 visitorsPROMOTED·Checkout button shape·+6.4% convRUNNING·Bundle thumbnail·52% confidenceWINNER·Email subject A·+24% openKILLED·Yellow nav theme·-1.1% convSTARTED·PDP photo order·0/3500 visitorsPROMOTED·Sticky add-to-cart·+9.7% convRUNNING·Cart-page CTA·64% confidenceWINNER·Hero headline B·+18% convKILLED·Free-shipping banner C·-3.2% convSTARTED·Pricing page·0/2000 visitorsPROMOTED·Checkout button shape·+6.4% convRUNNING·Bundle thumbnail·52% confidenceWINNER·Email subject A·+24% openKILLED·Yellow nav theme·-1.1% convSTARTED·PDP photo order·0/3500 visitorsPROMOTED·Sticky add-to-cart·+9.7% conv
Lab notebook · Live
What's on the bench this week.
EXP-0342WINNER
Add-to-cart button colour
Running 2d 03h
A — Orange
3.42%
B — Teal
▲3.95%
Confidence97%
B promoted to default. Variant A retired.
EXP-0343CALLED
Free-shipping threshold
Running 6d 18h
A — $50 threshold
4.10%
B — $75 threshold
▲4.42%
Confidence95%
B calls — lifts AOV without pushing checkout drop-off.
How it knows when to call
It calls the test when it's actually called.
Naive 50/50 splits run forever and burn revenue on the loser. Sequential testing with Bayesian updating ships winners faster and kills losers the moment the data is decisive.
Confidence ladderBayesian
Posterior50%
Posterior80%
Posterior95%Ship it
Posterior99%Ship it
Tests stop the moment the posterior crosses 95%. No fixed sample size. No "wait two weeks because the calendar said so."
01
Draft
AI proposes variants, predicts traffic needed, simulates risk.
02
Run
Sequential split. Bayesian posterior updates after every cohort.
03
Promote
Winner ships at 95% confidence. Loser is rolled back automatically.
What we test out of the box
Every variable. Already wired up.
Headlines
Tone, length, claim
CTAs
Verb, colour, placement
Images
Lifestyle vs. studio
Pricing
Anchors, rounding, tiers
Bundles
Mix and discount depth
Free-shipping thresholds
$50 vs $75 vs always free
Email subjects
Curiosity vs benefit
Send times
Window, timezone, frequency
What it actually does
What better tests actually buy you.
Median across Amboras stores · Q3
+0%
Incremental revenue / quarter
Baseline · 2.1%
Lifted · 2.6%
Time-to-decision
3.2 days
vs. 14 days for fixed-horizon tests
Win-rate
38%
of variants beat control by ≥ 95% confidence
False-positive rate
< 5%
across 4,200 shipped experiments
Ship today
Stop guessing. Start measuring.
Agentic A/B testing ships with every Amboras storefront. No third-party app. No statistician on retainer.
- Bayesian sequential
- Variant generation by AI
- Auto-promote winners
- Roll back losers