My A/B test sample size was TOO small for 6 months - here's what I learned

I ran a landing page test for a client in Austin and got a 12% lift, but when I checked the stats it was basically noise because I only had 300 visitors per variant. Now I use a sample size calculator BEFORE starting, and it saved me from making wrong calls twice already.

2 comments

2 Comments

nathan_foster6021d ago

Learned this one the hard way myself. Ran a headline test for a month, got a 7% winner, client was thrilled. Three months later the data flipped completely. Sample size calculators feel annoying when you just want to start testing, but they're worth the wait. That 300 visitor per variant mistake is brutal. Been there.

tylerw9221d ago

That "300 visitors per variant is brutal" line from @nathan_foster60 really hits home for me. I used to be one of those people who thought sample size calculators were just overthinking things, you know? Like, just run the test and see what happens. But after I spent two months on a button color test (don't laugh) and got a 5% lift that completely disappeared when I scaled it up, I finally learned my lesson the hard way. It stings to wait, but I'd rather wait a few extra weeks than make a bad call that costs way more in the long run. Now I'm that annoying person who triple checks my numbers before starting anything.