A/B test gone wrong!? #HALP


#1

1: What challenge are you currently trying to solve?
I’m working on refreshing and optimising a series of existing landing pages for a client for legal services. There are various different categories e.g family law, civil law, social security etc… that are the same page, but just a different heading and form.

The forms are embedded via JotForm as they require more extensive branching and a number of steps. Using the native forms was not an option.

I tried an a/b test with a refreshed page design and form design that was more cohesive to the overall style. Clean, easy to read – in theory should have improved the conversions (or at least matched). Instead, it’s lowered the conversions on almost all the pages. :roll_eyes: :tired_face:

So I’ve scratched that for now – and going right back to basics. I’m doing a very simple a/b test of the original page with just a simpler form with a green button. Will see how this runs.

But in the meantime, I’d love some outside perspective on why the original page is still showing the best conversions – to me, the original page doesn’t feel as trustworthy, easy to understand or credible as the new design I did.


2: How are you driving traffic to your page?
95% of the traffic is from Adwords and is routing thru call extensions (even before it hits the website). 90% is on mobile. All US based.


3: What is your conversion goal?
Form submission and phone calls. Equal priorities.


4: Provide a link to your published landing page / convertable:
See live page for the original design – https://socialsecuritydisability.requestlegalhelp.com/

Screenshots of failed design attached as well (since it’s not live now).


#2

90% from mobile!

Do you have a screenshot of the mobile design? I think that’s probably going to be the most relevant piece to look at, considering it accounts for the vast majority of the traffic.

Do you have a breakdown of the devices (traffic from iPhone vs Android, iPad vs iPhone 6, etc.)? Have you thoroughly tested the form from all of the most popular devices? I recommend a tool like BrowserStack for this if you don’t have physical devices at your disposal, as it connects to and emulates real devices.

In my experience, when common sense fails, there’s usually a technical problem. :slight_smile: Given the fact that you’ve had to use a third party form, there’s all the more possibility for something to not play nicely.


#3

Very good points!

Mobile screenshot of the new design below.

The forms were definitely visible on the mobile view. I adjusted each page manually and left plenty of space to accommodate the varying form lengths. The client’s budget really didn’t allow for full testing via BrowserStack unfortunately.

But in theory (not that theory is going well for me lately :joy:) the forms/call buttons should all function fine – as that’s what has been already used on the original landing page design, and converting up to 18%.


#4

Totally missed the second half of this sentence when I read it originally! Do you mean that 95% of your traffic never even hits the landing page and converts via call extenions?

If so, did you make any changes to your ad copy/targeting at all?

Comparing the two landing pages, and barring any potential technical issues, it definitely doesn’t seem like the page design would be the issue. I completely agree that the re-design is more cohesive and conveys more trust than the original, and the verbiage is nearly the same between the two pages.

However, if the vast majority of your leads are from call extensions, the landing page itself would have little bearing on that (aside from maybe affecting quality score, which I would think would go up if anything).

Are you looking at an overall view of your conversions, or segmented by variant? Can you segment by conversion type e.g. conversions from call extensions vs. form submissions vs. landing page phone calls? This might give you a better idea of where your conversion rate is falling off. If call extension CVR dropped off, for example, the issue is on the Adwords end of things, but if form submission CVR dropped off, you might want to take a closer look at your form UX.


#5

Yes, I believe that’s what he meant – the pages just catch extra leads. Ad targeting/copy has been the same throughout.

These were just the call conversions that dropped (however, most of the form submissions did too) … but it’s a bit redundant now as my client got me to just roll out the green button variation now after it increased the conversions on one of the pages. :woman_shrugging: Client is happy! :wink:

Thanks for your advice @leah.ann, such good questions around this.