Mistakes to avoid while doing A/B Testing?

Now that we have learned about what is A/B testing its basics, here are a few things that you need to avoid while doing A/B testing.  

1. CRO might not be the main problem: 

One of the main reasons to do A/B testing is to increase your conversion rate with the help of optimization. A/B testing is one of the best tools to utilize for optimization, however, A/B testing only works if the product fits the market.  

For example, you start a SaaS business that provides services to B2B on CRM. You build your product, you release it to the public and wait for the sales to flood in. Two months later, the floodgates are still closed, and you’re left wondering what’s going on. You turn to the internet to learn about what can help you grow your business, and you learn about CRO.  

You start to read about CRO and think it will help you solve the problem. Surely, CRO worked for you and you get right back in the market. But in most cases, this might not work because the bigger problem lies underneath the surface. If that’s the case then we need to gaze more into detail than just A/B Testing. The bigger problem might be that you are trying to convince people to buy something completely different than what the market is actually asking for.  

2. Stopping tests too early or running it for a long time: 

Patience is a virtue. When it comes to AB testing this proverb, in particular, holds true because you want to test based on the audience size, a number of variants, and a number of users per variant. Usually, it’s three to six weeks a safe bet, but this also depends upon the amount of traffic flowing in your website. I hurried, the data we obtain can give us false information which can result in severe consequences.  

Running a test for too long or too short can result in the test failing or giving inaccurate results. Even if one variant is winning in the first few days compared to the other ones, it does not necessarily mean that we stop the test and declare the winning variant. Sometimes it takes time to produce significant results. Therefore, setting your test timeframe should be carefully calculated based on the factors aforementioned.  

3. Split testing a wrong page  

Marketers who have been doing split tests already know the time value money of A/B testing. Split tests are only good if they are done on the right page. How to know which page to do a split test in? Well, the answer is simple: the best pages to do A/B test are the pages that have conversion value, meaning the pages that make a difference to your conversion. 

The most visited page that extends to your sales funnel is the pages that you need to test. Some of the examples are: 

• Homepage 
• About Us 
• Products 
• Contact  

Depending upon what type of industry you are and what type of e-commerce you do, these pages will vary, but you get the idea of it. Don’t waste your time and money testing something that won’t be working. 

4. Control group and experimentation: 

Experiments need a control group, without properly setting control group you won’t be able to tell the difference if the experiment that you just did is justifiable or not. The independent variable being tested (hypothesis) should not be mixed with the rest of the group in order to get the results that you want.  

An experiment to test your hypothesis should be based on a control group. The effect of a change that is unknown and comparisons between the control group and the experimental group are used to measure the effect of the changes made to your content.  

If you are running an experiment to determine the optimal button placement for your call to action, for instance, one of the pages of your test should include the original button placement.  

In this way, when conducting your test, you will be able to gain an accurate answer if whether the change that was made to the original button placement gives any value to the experiment. 

Marketers divide their control group based on the number of variants they are testing. If you are doing one variant test then the traffic should be divided 50/50. If you’re testing two variants against the control, traffic should 33/33/33.  

However, be wary of testing too many variants at one time.  

5. Running too many split tests for one experiment 

While A/B testing can give you valuable insight with experiments that are being performed. However, running too many tests on one experiment at one time will lead to failure. It’s ok to run multiple A/B tests. For example, you can test three different versions of your call-to-action button at one time. (Testing the same variant is different than doing a multivariant test for each item).  

Experienced optimizers know that running more than four tests at a time can give you results that are going to be tough to interpret and hard to analyze. One of the biggest reasons is that the more variation you run the bigger the sample size you need. This is because you need an ample amount of traffic to test each version to get reliable results.  

6. Making changes in the middle of the test 

Changing your test in the middle of the test will invalidate your test and the results obtained from the tests are skewed. Therefore, making changes in the mid-test will result in inconclusive answers.  

If there is an absolute necessity to make changes to your test, you need to start from scratch again. If you have a grammatical error or the design that you made has some flaws, it’s better to scrap the running experiment and start it from scratch.  

Like any other form of marketing, a solid A/B testing strategy requires you to think carefully about your customers, your product and your market.  

Pro tip: Establish your goal, metrics, and set the time before even starting the test.  

7. Using the wrong tool 

Almost every CMS tool or website management system offers A/B testing functions. However, to properly perform the A/B test seriously, you need a purpose-built testing tool.  

There are a number of A/B testing tools in the market and the list keeps growing. To decide which one to go for your experiment, ask yourself questions like: 

• Skill level of my team 
• Technical resources required for the job 
• The support system from the company that you are trying to get the tool from 
• The volume needed to run a test 
• Other tools required or not (before and after the test) 

You can get more detail about Ptenging and it’s product feature over here. 

8. Mobile responsive A/B test 

Mobile responsive design has been part of the design strategy for many marketers as technology has evolved the way that we consume our content. Keeping mobile devices in mind, content should be created to test both mobile and desktop traffic. Depending upon what type of device your customers mostly visit your site or consume your content from, based on the data you need to make sure to look into numbers in the results from your test to match your objective.  

It is important to establish basic segmentation within the A/B test. Your analytics will help you give you an accurate number of different conversion rates, through Mobile and Desktop, so calibrate your A/B tests to take into account different parameters such as this.   

Conclusion 

Lastly, A/B testing is an amazing tool to optimize your website/content and to increase your conversion, provided that you have: Time and Traffic. 

These are the two most crucial resources to have before you even think about starting to A/B testing. However, don’t get disappointed if you feel like those are the two resources that you don’t have. There are other ways to optimize websites and content. Check out more on our blog section to learn about other ways to optimize your website using Heatmap tools. 

If you are considering A/B testing for your site, explore our A/B testing in the 2020 guide.