A/B Variants Not Showing Equally Despite 50%/50%

  • 24 April 2017
  • 4 replies
  • 24 views
Adam_G_Katz

Hi, guys:

I have two variants and they’re both set to show 50%. One is showing 6 visitors, the other is showing only one.

Shouldn’t it be 3 and 3?

4 replies

Alex_Scovell

I expect that the 50% is calculated for each visitor rather than for overall.

So like flipping a coin six times, 5 and 1 is very doable but over time it will even out.

Will_Prout

I’m having the same issue but with more like 500 visits vs 300 visits, i think it has to do with returning visitors seeing the same variant they had seen previously. If that’s the case, is there any way to turn that function off so that each variant has a 50% chance of showing even for returning visitors?

GregorySR
Userlevel 2
Badge +1

@Will_Prout from an A/B testing perspective, letting a user see different variants when they come back can lead to data issues.

Think of it this way: you want to know if A works better than B, so people who see A should only see A, and same for B.

If each time they come back to the site they see a different version, then you’re no longer testing if A is better than B. You have to take into account “does seeing A and then B work better than B and then A”. What about “seeing A 3 times and then B twice”, etc.

Need a mortgage in Canada? Come see us at nesto.ca!
Will_Prout

@GregorySR, I agree letting users see different variants would be imperfect but in our case returning users behavior is distinctly different from users who are visiting for the first time. If we let returning users see different variants, their anomalous behaviour would still be split evenly between the 2 variants 50/50 and any results would even out. right now we’ve got one variant with far more visits due to those returning users and so we cant accurately measure the performance of the new variant against our incumbent since the incumbent gets all the returning user traffic.

I guess the real solution is to duplicate the incumbent and turn off the existing one when launching a new test but its annoying that unbounce doesn’t have a setting for this.

Reply


Terms & ConditionsCookie settings