A/B testing : change the name of the variants, is it a Best Practice?

  • 15 October 2021
  • 2 replies
  • 3 views
M

Hello everybody,
Soon I will launch my first A/B test with Unbounce and I planned to do some changes during the next days based on the variants performance. Periodically I want to change the variants’ names to remind me what changes I did on that version. I’m wondering if changing the variants’ name during the A/B testing may be a good or a bad practice.
May this affect the conversion statistics? May you help me to do some evaluation to define the best practice about it? I do not see any

Here a practical example:
first 4 days - I will publish the versions: V1, V2, V3
then after 4 days - I see that the V2 and V3 are performing better and I selected them to add other changes, so I want to change the variants’ name with: V2.1 and V3.1
after 4 days - I see that the V3.1 is better than I want to add 3 different changes, then I will change the variants’ names like this: V3.1.1 , V3.1.2, V3.1.3

thank you in advance for your help
cheers

2 replies

digibomb
Userlevel 6
Badge +1

In terms of naming conventions, yes this could work. You could also use a hypothesis spreadsheet to track the changes made along the way: why are we making these, what we hope to achieve, etc.

But, like @Hristian stated 4 days is too short, no matter how much traffic you have. We typically wait 2 weeks before we make any changes. We need data to analyze before we can even know what to change.

In very rare circumstances to some major shift early on, and even then that could just a be a blip.

Hristian
Userlevel 7
Badge +3

Hey @Mica,

I would suggest reading up a bit on A/B testing and more specifically A/B testing statistics.

You need to calculate your required sample size and stick to it. Stopping and changing tests after 4 days would results in bad data and/or even negative effects on your performance.

Start with just your original page and 1 variant. Calculate the necessary sample size. Let the test run for at least a week (possibly more depending on a few other factors). Make sure you get at least 100 conversions or more per variant.

Best,
Hristian

Hristian K. → experimentation-as-a-service → https://www.revise.agency/

Reply


Terms & ConditionsCookie settings