Confidence formula

Hmm yes that makes sense. I had thought it could be something like this and I’ll look into the chi-square test to understand it a bit better.

However I’m not sure the chi-square is such a good instrument. Here is some sample data for some recent new variants:

21 1 4.76% -27% 24%  
4 0 0.00% -100% 40%  
 
Vis = Visitors  
Con = Conversions  
Rate = Conversion rate  
Del = Delta  
Conf = Confidence ```  
 
So obviously the rates here aren't very telling, especially the lower variant. It's just to demonstrate that the confidence feels like a random made-up number (obviously it's not -- but it does feel like it).

Hi Carl - better guidelines on how use the test results, specifically the confidence %, would be helpful.

Coming from a project where I was using Google Site Optimizer, I was expecting a “winner” to be declared. During my very first Unbounce test, I had a variant at 98% and was patiently waiting for it to get to 100% so it could be declared the “winner,” before I realized/learned/looked at the manual and realized that’s not how Unbounce’s “confidence %” is supposed to work.

Hey guys, this is all great feedback!

Jerome, further to your comments about weighting different aspects of the numbers, what you’re actually talking about is coming up with new statistical methods. We worked with a local university to ensure we were using a technique that was valid, statistically speaking. We’ve discussed ways in which we can present this information better, and definitely hope to have some improvements there soon, but we’re not likely to go about inventing new methods. 😉

Incidentally, we chose not to use the same approach as Google. Google’s tool provides a “confidence interval” for the test metric, a range of values that represents a prediction of the “real” value. However, that approach typically requires larger sample sizes. We chose an approach that will let customers who aren’t driving larger volumes still achieve valid results.

Matthew, really appreciate your comments as well, especially with regard to comparisons to Google’s tool. We’re hoping to provide something similar (a clear winner/loser indication) relatively soon.

Thanks guys!

There’s a rough guideline for the chi-square test that most of the cells in the input should have a minimum value of five (paraphrasing there, the guidelines vary and are a little more specific). However, there’s also the guideline that the sample size must be “sufficient”. This would include things like allowing your experiment to run for at least a week to correct for typical periodic fluctuation, and it of course depends on the nature of your audience and traffic sources.

Anyway, we’re definitely going to improve the presentation of the test results soon. Likely we’ll de-emphasize the numeric confidence score, and place more emphasis on an indicator that displays whether or not you’ve potentially achieved a significant result.

Though I can’t answer the actual question, I can suggest that you simply set its traffic to 0% – then it is effectively disabled but still in there for statistics.

Hey guys,

Each challenger variant is an independent test against the champion, so the only thing discarding another challenger variant will do is change the rate of future traffic reaching the remaining challengers. It won’t change the confidence of remaining challengers directly. Think about it this way, if you’re testing three pages, and you discard one of them, does that give you *more* information about the other two?

Once you reach sufficient confidence (95%+) on a particular variant, you’re good to either promote it or discard it as appropriate. Remember too what the test is actually telling you. With a 95% confidence, there’s a 1 in 20 chance of being wrong about your hypothesis, whereas with 99% confidence there’s only a 1 in 100 chance.

I’ll see what I can do about providing some sample calculations. We’re using a stats library that can compute a confidence value, which is something you wouldn’t necessarily want to do by hand. Typically when computing a chi-square test by hand you’ll produce a p-value, and then lookup the corresponding confidence from a pre-computed table. This will tell you whether or not you’ve achieved a particular pre-computed confidence level, but won’t give you the exact confidence level itself.

Good questions!