Hot take: 90% of A/B tests are useless if you don't segment your audience first

Last month I ran a landing page test for a client selling dog accessories in Austin. Tested a blue button versus a green button and saw no change. Then I split the data by new visitors vs returning visitors. The green button got 23% more clicks from returnees but 12% fewer from new folks. If I hadn't looked closer I would have called it a wash. Has anyone else found that segmenting totally changes your test results?

2 comments

2 Comments

james67210d ago

Yeah man that's the exact kind of thing that drives me nuts about A/B testing. You gotta slice the data by traffic source or device type too. I had a similar thing with a signup flow where overall conversion was flat but desktop users went up 18% and mobile dropped 15%. If you're just looking at the average you'll miss the real story every time.

sean_foster527d ago

Look, I think there's another layer to this that nobody's mentioned yet. The time of day people hit your test can completely flip results, especially if you're running a global audience or testing during a holiday week when traffic patterns change. I started tracking the hour-by-hour data on my tests and found that what worked for morning visitors actually hurt conversion for evening users in a few cases.