Setting up an A/B experiment

After building your experience content and previewing it to make sure the look & feel are as expected, there are now two ways to deliver a Userpilot experience:

Rolling out the experience to all users in the specified segment without testing its effect on the goal first (Default).
Rolling out the experience to a subset of your users to determine if showing the experience has a significant effect on driving up the specific goal.

If you're looking to run an A/B experiment, you'll first have to set your desired Targeting options – users that don't match any of the Targeting settings will be considered ineligible to enter the experiment.

Page Targeting

Here you can define what pages (URLs) in your app the experiment will run on. Make sure to exclude any locations in your app that would cause any significant outliers.

Priority

If the user is eligible to see the experiment, the experiment will have the highest priority amongst other experiences on the same page.

Frequency

There are several frequency options to deliver Userpilot experiences and A/B testing here is relevant for experiences that are set to be shown only once.

Choose an audience

Select a previously created Segment or use the "Users that match specific conditions" option to create a condition (or multiple conditions) that specifies the subset of your user base that you would like to run the experiment on.

We don't recommend running an experiment on a small user sample group. The larger the sample size, the better.

Tip: Make sure to exclude any internal testing accounts that would result in skewing the data and in turn leading to incorrect test results.

Define a goal

To be able to run an experiment, you have to set up a conversion goal so you can track how effective the experience was in achieving your target. You can select a previously defined goal or use the "Define new goal" to create a condition (or multiple conditions) that defines how success would look like to your team.

Experimentation Time Frame

Options:

Once a result has been determined (default): the experiment will run for one week and for a minimum of 100 sample size
Run for a specific period of time: the experiment will run for the specified time frame, then the test results will be shown disregarding the sample size.

General notes:

Once an experiment is published and while it's running, the content & steps of the experience will not be editable and the selected targeting settings will be unchangeable.
After an experiment is completed, the experience will automatically become unpublished and a notification will be sent to your team via email.
To maintain the experiment's integrity, restarting the experiment will overwrite the old results and start it from scratch.