Page Loading...

A/B Test Manager - Always Be Testing!

Vera Mayer
Vera Mayer
VP Professional Services

Don't underestimate the power of testing!


Always Be Testing!

Whenever you create a digital advisor, you do so based on what you think will work best for your users. Our best practices can help you to start off on the right foot, but you never know what's really working for your audience until you put your advisors to the test.

Humans are unpredictable; we act differently in different environments. That's why a digital advisor that generates awesome results in one environment might underperform in another.

We believe that A/B Testing should actually stand for, "Always Be Testing." If you want to create successful advisors that perform well and have a positive impact on your bottom line, frequent testing is not just recommended: it is essential.

A/B testing (also known as split testing or bucket testing) is the process of running a series of randomized experiments that involve different variants of your advisor, then subjecting these variants to performance comparison in order to ascertain which will be the most effective.

This allows you to learn how certain elements of your advisors impact user behavior. Since your advisors should aid human decision making, you should (and must) take into account your users and their opinions.

The SMARTASSISTANT A/B Test Manager allows you to test and optimize your digital advisors without hassle.


These are the steps to successful testing

  1. Identify areas for improvement: Analytics is your friend. It will provide insight into the areas that can be improved. Inspect the bounce rates, the position in your advisor where users drop off, time spent, answers selected etc.
  2. Define your goals: What do you want to improve and which metrics determine whether a variant was more successful than the original version?
  3. Create hypotheses: Hypotheses are assumptions that identify the elements that might impact the metrics you would like to improve

    Some examples include:

    • adding images vs. no images
    • changing the wording of questions and answers
    • changing the decision paths and Q&A flow
    • changing the design and layout
    • changing button colors and positions
    • changing the interaction and effects
    • adjusting the number of questions and steps
  4. Setup variants: Make a copy of your advisor with the appropriate changes to effectively test your hypotheses.
  5. Kick off the experiment: Run your tests using the SMARTASSISTANT A/B Test Manager. Depending on the number of visitors, you might want to run tests for anywhere from a few days to a couple of weeks. Make sure that the sample size is large enough for your results to be statistically significant. Based on your setup, SMARTASSISTANT will randomly display different versions of your advisor to different visitors. The default distribution is 50:50, however you’re free to adjust this ratio via the A/B Test Manager dashboard.


  1. Analyze the results: After completing of the experiment, go back to Analytics to understand which advisor performed better and whether the difference is statistically significant.


Some tips

  1. Cultivate a culture of experimentation in your company. A/B Testing is also a good method for dealing with HiPPO's (Highest Paid Person's Opinion). Test out your HiPPO’s suggestion with a subset of your customers and take an objective look at the results.


  1. Start soon. Based on multiple A/B Tests, we know that even small changes can have profound impacts on the advisor’s performance.
  1. Remember that some test results will show negative outcomes. Don’t give up when a test fails! Don't fall in love with your hypothesis. A negative outcome is no reason to despair. In fact, scientists embrace negative results because they provide you with valuable insight for optimization.
  1. Keep the team small to avoid delays. Involving too many people in your experiment setup will lead to long decision cycles that and will slow down your efforts. Keep your experimentation team small in order to move quickly and reap the benefits of your testing without dealy.
  1. Look beyond your core metrics. Your core metrics are significant, but they may not tell the whole story. Looking at different metrics allows you to understand what is really happening. Analyze your core metrics for different customer segments, geo, languages, locations, and so on. Make sure that you stay abreast of the external environment that may influence your test results, such as launched marketing campaigns, dispatched newsletters, etc.
  1. Test small changes. It's best to test one change at a time and give each test sufficient time to run.


Can I Test More Than One Thing At a Time?

Yes. The SMARTASSISTANT A/B Test Manager supports running multivariate tests that involve multiple variants.

Happy testing!

PS: Your success manager is happy to help you define a suitable strategy that allows you to get the maximum out of your advisors. Just reach out if you need support!

Recent Posts

  • Experience
  • Automation
Create meaningful, need-based facets and filter labels There are few (if any) transactional websites that don’t offer faceted search or filters. When used correctly, these tools are straightforward and easy-to-use. They minimize cognitive friction...
  • Experience
  • Automation
Define contextual decision paths with ease SMARTASSISTANT allows you to create individual advisor flows based on a user’s context. This means that you can dynamically display only those question and answer options that are relevant to the user's...
  • Design
An easy way to customize the look and feel of your digital advisors With SMARTASSISTANT you can quickly create digital advisors that match your style.   Design themes With SMARTASSISTANT themes, you can change the design of your digital advisors in...
  • Data Management
  • Automation
Create Rich, Consistent, and Meaningful Product Details with Computed Attributes   Added-value services such as site search, filters, and digital advice can only be as good as the underlying data. Meaningful product information is nothing short of...
  • Optimization
  • Automation
Ensure high quality with automated functional testing of your digital advisors   Smart Tester is an automated testing feature that executes tests for your advisors and reports outcomes. This feature is included with all SMARTASSISTANT digital...