Our A/B test went sideways when the junior dev pushed the wrong button

I was sitting at my desk in Austin last Thursday, watching the real-time dashboard for a big campaign we were running. Suddenly my boss runs over screaming that the landing page was showing the old broken design with the giant red font. Turns out the junior dev accidentally pushed the production toggle instead of staging, and 14,000 visitors saw that mess before we killed it. We spent the next 2 hours rolling back and writing a manual apology email to the client. Has anyone else dealt with a deploy gone wrong like that?

2 comments

2 Comments

paige8702d ago

Oh boy, another classic case of "oops wrong button" syndrome. I bet that junior dev felt like they were starring in their own personal horror movie while watching 14,000 people see that mess. At least you didn't have to explain to the client that the giant red font was actually a "deliberate design choice" - silver linings, right?

susan_wells2d ago

Right? Total nightmare fuel. @paige870 what finally saved my skin was having a pre-written apology macro ready for stuff like that. Just one click and it sends a polite "our bad" message before you even have time to panic.