How much traffic are you going to send to this page with the 60 variants?

Don’t know ! I m gonna use it for our next social campaign … Why ?

Without knowing too much about your traffic volume or differences between the variants, have you considered testing fewer than 60 at once?

For example, testing 20 simultaneously would be an easy 5% traffic allocation per variant.

Have you considered an iterative test plan that allows you to test fewer pages at first, then learn and iterate new variants based on winning variants from earlier rounds?

In order to reach statistical significance, you need to have enough traffic to determine if your experiment is outside of the standard deviation. Without knowing what the conversion goal is i.e. click-through vs form submit, you would need between 720k 1m visitors. Never mind about how many conversions you would need on each variant.

As @Andrew suggested, have you considered an iterative testing plan?

Hey @gadboukhris,

@Joe_Savitch & @Andrew make some excellent points .

If you are running more than 2 to 3 variants at a time you are:

a) Wasting time - If your changes between variants are too small, you won’t be able to detect a significant lift. If the changes are bigger, you are essentially wasting time putting together 60 variants. Solution: Go back to the drawing board. Revisit your hypothesis and make sure the variations you are creating are based on a solid understanding of what your are trying to measure and improve.

b) In need of a refresher on Statistics - I’m not going to get into the details here but how are you going to account for type I errors with that many variants? At 95% significance you are going to get at least 3 variants showing an improvement that in reality doesn’t exist. Just trying to calculate your alpha error for that many variations should give you pause (1-(1-a)^m)

To sum it up… instead of trying to get equal weight, I would seriously rethink your approach and hypothesis.

Just because Google tried testing a gazzilion shades of blue on their buttons, it doesn’t mean it’s a good idea.

Best,

Hristian

Thank you for your help 🙂