Has anyone seen drastic changes to their variants’ performance over time?
In one test w/ 4 variants, I had a top performing challenger (~3k impressions each for test & control) where the top performer reached 30% performance & 97% confidence after 1 day. After reaching 10k per treatment, it’s only 17% confident that it’s 1% stronger.
While I’m more inclined to trust the version with more impressions generally, i’m bewildered by what’s going on:
- My GA numbers have flatlined to almost 0 but conversions are still coming in on unbounce steadily.
- this is the second test where a strong performer does an about face as it gains more traffic.
The 2016 unbounce fall lookbook also shows a lot of winning results with 2k-5k visitors so while I lean towards more traffic, i’m curious how much folks are allowing in their tests.
My main KPI is a purchase (ie full sign up for subscription). thanks!