Question

A/B test confidence level after less than 24h

GregorySR
Userlevel 2
Badge +1

Hi!

I launched a new version of a page yesterday as a 50-50 split to see if it performs better than our current version.

Checked in this morning, less than 24 hours after launching it, to see that the platform is already fully confident that the other version is better… This seems like a hasty decision to me! Is it normal that it would give an answer so quickly? Especially since it mostly ran at night, not our best conversion times anyway...

Should I reset the overall stats every time we launch a new version to force a better comparison? I enjoy seeing the historical data, so I hope that’s not the only solution.

 

Need a mortgage in Canada? Come see us at nesto.ca!

2 replies

julien_level
Userlevel 5
Badge +4

Hi @GregorySR 

 

On most of our campaigns, the form usually include the telephone number and when we A/B test we usually wait to have a 100 leads to compare. So if you are in the same condition i would say you can be pretty confident too.

BUT, it also depends on you trafic at the time of the testing so you should also have a look at the graph (even if it’s on a 7 rolling days) as your CR for the champion is calculated on a surely older version with a lot more leads.

Julien
Taryn_Lomas
Rank
Badge

Hey @GregorySR 

 

Taryn from Unbounce support here - I just wanted to confirm that what @julien_level said about A/B testing confidence is calculated based on the number of visitors to a page is correct. Do you happen to know how much traffic/leads your page received since turning the test on? 

 

You shouldn’t have to reset your stats every time you run a new A/B test. If you think you’re experiencing issues with it, please feel free to reach out to us at support@unbounce.com and we can definitely look investigate this further for you :) 

 

Cheers,

 

Taryn

Reply


Terms & ConditionsCookie settings