Our A/B test crashed the checkout page on a busy Monday

We ran a headline test on the product page in Denver and somehow pushed a bad script that killed the buy button for two hours. Traffic was up 30% that day, so we lost real sales, maybe around $4,000 based on our usual conversion rate. I killed the variant within ten minutes once I saw the error logs, then we rolled back to the old version. The lesson stuck with me, so now we stage every test on a clone server before we touch the live site. Has anyone else had a simple test blow up production like this, and what did you change in your review process?

1 comments

Log in to join the discussion

Log In

1 Comment

dianab682mo ago

Oh man, the part about the buy button dying for two hours gave me flashbacks. We had a similar thing happen last year where a simple font change on our cart page somehow broke the whole checkout flow on mobile. Took us longer than ten minutes to catch it because the error logs were buried under a bunch of other noise, so we lost a whole afternoon of sales. After that we made it a rule that any test that touches payment or checkout gets a full manual test on a staging site first, even if it feels like overkill. The scary thing is how fast a tiny script change can ripple through everything. Now we also keep a screenshot of the live page before we push anything so we can compare right away if something looks off.

5