A/B Testing Overview

Whether you're new to Userpilot or you're an experienced user, you must be curious about the best way to measure your goals conversions or how to measure the impact of a certain change to your Userpilot flows.

Userpilot now makes it easy to test flows against a control group (Userpilot vs nothing). With the help of our new A/B testing experimentation tool, your team can now make more informed decisions and measure the impact that flows are having on your growth goals with higher accuracy.

Example use cases

• Understand the impact of showing a flow on a given goal to a specific segment versus showing nothing at all.
• Study, analyze, and decide the best version of the flow that would result in better engagement and usage of a certain event in your application. 
• Measure the effectiveness of flows in increasing your adoption goals and retention rates 

How it works

Userpilot will equally divide your users who are eligible to enter the experiment and assign them automatically into 2 different groups:1 Group (A) and 2 Group (B).

The sample group (A) is the holdback/control group – not eligible to see the flow. The sample group (B) is the variant group – eligible to see the flow. 

Note: Any user who doesn't match the selected targeting and triggering Settings of the flow, will be considered ineligible to enter the experiment.

After the experiment has been completed, the test results will favor one group over the other or return a neutral result when there is no significant change between groups A and B. 

• Group A (out / control) is favored – the flow has a minor effect on the goal conversion
• Group B (in / variant) is favored – the flow has a noticeable effect on the goal conversion

General Notes

• You will not be able to run an experiment without setting a goal first.
• You will not be able to run an experiment if the frequency setting of the flow is set to 'show every time' or 'trigger manually'.
• Users who have achieved the given goal prior to the start of the experiment will automatically be excluded from the experiment.
• Allow experiments to run until there are enough participants for the result to be considered 'statistically significant'.
• Avoid running experiments on segments with small audience sizes. 

If you have any questions or concerns please reach out to us at support@userpilot.co