I noticed this morning that when I reload the page another variant loads for the user. This really isn’t the best way of doing A/B testing. Ideally the user should be uniquely identified and the variant that was shown to them should be re-shown every time they visit until they complete the ‘goal’ of the page.
Hi Peter, are you using a cookie blocker of any sort, or testing from different browsers? We *do* remember the variant that was displayed to a visitor, and show them the one they saw previously (using a cookie). The only time we show a different variant is when you discard a variant, in which case we’ll make a new random selection. Definitely interested in tracking this down if it’s not working properly for you!
No. I’m not testing from different browsers. Cookies are enabled. No variants were discarded.
I was using Safari 4 ( specifically - 4.0.5 (6531.22.7)) for the test on Mac OSX 10.6.2