14.03.2016 |

Episode #9 of the course “The Fundamentals of User Experience” by Cassius Kiani


You’ll never have good UX without testing and ongoing improvements.

Clever planning and implementation will create good UX on paper, but you’ll never truly know the impact unless it’s being measured.

By testing and measuring these results, you’ll learn what does and doesn’t work while improving a user’s overall experience with a product.

There are two common types of testing you’ll encounter and need to be aware of:

A/B Testing
A/B tests work using a control variant (A) and a challenger variant (B). Typically, you’ll test these variants side by side, with a certain set of users interacting with A and the other with B.

Users will only ever see one variant in A/B testing—either A or B—with only one notable difference between both variants.

You’ll then track the metrics that matter to you, comparing which variant was most successful. Due to the nature of A/B testing, you’ll need fairly sizeable user groups to increase the accuracy of the tests. Otherwise, it’s difficult to make an informed decision.

Examples include changing the secret ingredient in a chocolate bar and asking 100 users to test A and another 100 users to test B. Users can then rate the chocolate bar on taste, texture, and satisfaction.

A further example could be changing the imagery on a landing page. You’ll then measure which variant caused the most users to interact with the page, sign up, or buy something.

Multivariate Testing
Not quite as common as A/B testing, multivariate testing is still a staple for user optimization. Like A/B testing, users are only shown either the control variant or the challenger variant.

Unlike A/B tests, multivariate tests change several elements and create an entirely different challenger variant. Though the goals are the same, the challenger variant in a multivariate test explores an entirely new approach.

With the chocolate example, the goal would still be to create the most delicious chocolate treat, but this may be achieved by reshaping the bar and adding additional ingredients (raisins, nuts, cookies, etc).

With the landing page example, this could include redesigning the entire look and feel, changing the copy, or reconsidering the art direction.

There are pros and cons to both A/B testing and multivariate testing. Some situations will call for one, and some for the other.

To learn more about when to use either A/B or multivariate testing, check out this article below:
Comparing a Multivariate Test to an A/B Test

Recommended book

“Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics” by William Albert


Share with friends