Found a weird trick for A/B testing on a shoestring budget

I was running split tests for a client's landing page with like 50 visitors a day, and the results were all over the place. Then I accidentally left a Monday morning-only discount code running for a whole week and saw a 30% jump in conversions just from that one day's traffic. Turns out timing your tests to specific days or even hours can filter out noise way better than chasing a bigger sample size. Has anyone else played around with time-based splits instead of just random traffic splits?

2 comments

2 Comments

karenw902d ago

My last job had the same problem with low traffic and wild results. We were testing headlines on a local service site and Monday mornings were always dead but Thursdays blew up. It took me three months of garbage data before I realized our weekday vs weekend traffic acted like two different audiences. Once I started splitting tests by day of the week instead of random assignment the patterns got way clearer. It's kind of wild how much noise you can cut out just by paying attention to when people actually show up.

troyjackson1d ago

Wait, is it really that deep @karenw90? Seems like overthinking it.