Hi Lars,

Good question and I have wondered this myself!  I have not reached Z yet so let me know what you find 🙂

In regards to the A/B testing of these work-in-progress landing pages, do you have lots of traffic coming to them so that your tests are still statistically significant despite the rapid changes?  Traffic can be a hugely limiting factor for A/B testing and I see many leaning on user testing to jump the curb these days.  What methodology do you go by and it will be interesting to hear from the community what ways everyone overcomes traffic limitations to achieve statistical significance or ultimately increase conversion rates despite it?

Warmest regards,

Joe

Hi Lars!  Once a variant code hits ‘Z’ we start at ‘AA’, ‘AB’, etc.  There’s a practical limit of 100 concurrent variants in a test because traffic weights are assigned in increments of whole percentage points, meaning the largest test you could run concurrently is with 100 variants with 1% of traffic each.  

I echo Joe’s question regarding traffic and achieving statistically significant results.

Hope that helps!

Cheers,

   Carl

Thanks a lot both of you. Actually my question was by curiosity because I’m preparing a complex page for test and to do that I’m working on many variants of the page before starting tests.

That’s a good point with a max of 100 - but I guess no one would ever get to that point because you would get very few visits per variant.

When you work on new variants you may delete some, copy to new ones, edit and delete others, so even though the number of variants may be low the letter keeps adding up.

However I found a solution for this:

• While preparing and editing you can make a lot of variants and you can change the url away from the planned one


• When ready you just copy your master to a new url and then it becomes variant A with no statistics on for a fresh start.
  
  Finally we are in the middle of revising our testing rules. But some of the ingredients are these:


• Use Chi-square method


• At least 1 week, but rather at least 3 weeks


• Numbers exceeding 1000 visits


• Significans level above 95%


• If you’re far from significans after +1000 visits, several weeks you should stop


• If you _re close you could decide to go 1-2 weeks more


• Everything is statistics - theoretically your numbers could change next week… but, on the other hand a lot of other things in life aren’t that precise


• It’s interesting to see if you can find patterns splitting your data by country, traffic source etc to use the chi-square method to find subseries of data that doesn’t follow the whole group


• It’s like-wise interesting to add dataseries from different base-pages where you’re testing similar changes to see if there is a meta-pattern that can be confirmed


• The only true conclusion you can make is when you can say that within a certain confidens level, e.g. 98% you can’t conclude anything. There is no true confirmation to make in tests - only that you haven’t confirmed your hypothesis yet.


• Preparation is important: Identify and analyze problems, come up with a hypothesis to mitigate the problem and test it to see if you have a confident base to believe in your hypothesis


• Don’t use the simple Unbounce figures. Understand the statistic methods and set up your additional Google Analytics and Excel monitoring …
  
  Well, I hope some of this can be of use… 
  
  Best regards
  
  Lars

Lars, 

I would love, love,  love _to see an insiders glimpse into how you dig into the statistics. In fact, a walk through of how you set things up and track in Google Analytics and Excel would be extremely useful for those who want to take a/b testing to the next level. Would you be interested in hosting some kind of Google Hangout or Workshop and walk us through it?