Had a chat with my team's analytics guy yesterday that totally flipped my thinking on A/B testing

I've always treated A/B tests as these big, formal things with huge sample sizes and waiting weeks for results. But our data lead pointed out that a quick 3 day test on a small landing page change taught him more than a month long study on a major feature. He said most of us treat these tests like academic papers when really we just need a directional signal, not certainty. It hit me different because I've been sitting on a test idea for 6 weeks now waiting for enough traffic. He said run it anyway with a 80% confidence threshold and move on. So I kicked off a test on a button color change today with just 500 visitors targeted. Has anyone else found success running smaller, faster tests instead of waiting for that perfect sample size?

2 comments

2 Comments

taylorcarr7d ago

People overcomplicate everything these days, like cooking a gourmet meal when a grilled cheese would do the job fine. Speed beats perfect more often than not.

mila_mitchell7d ago

Speed buys you time to learn what actually matters before you waste effort perfecting the wrong thing.