Setting up an A/B experiment
- Rolling out the experience to all users in the specified segment without testing its effect on the goal first (Default).
- Rolling out the experience to a subset of your users to determine if showing the experience has a significant effect on driving up the specific goal.
If you're looking to run an A/B experiment, you'll first have to set your desired Targeting options – users that don't match any of the Targeting settings will be considered ineligible to enter the experiment.
Here you can define what pages (URLs) in your app the experiment will run on. Make sure to exclude any locations in your app that would cause any significant outliers.
If the user is eligible to see the experiment, the experiment will have the highest priority amongst other experiences on the same page.
There are several frequency options to deliver Userpilot experiences and A/B testing here is relevant for experiences that are set to be shown only once.
Choose an audience
Define a goal
To be able to run an experiment, you have to set up a conversion goal so you can track how effective the experience was in achieving your target. You can select a previously defined goal or use the "Define new goal" to create a condition (or multiple conditions) that defines how success would look like to your team.
Experimentation Time Frame
- Once a result has been determined (default): the experiment will run for one week and for a minimum of 100 sample size
- Run for a specific period of time: the experiment will run for the specified time frame, then the test results will be shown disregarding the sample size.
- Once an experiment is published and while it's running, the content & steps of the experience will not be editable and the selected targeting settings will be unchangeable.
- After an experiment is completed, the experience will automatically become unpublished and a notification will be sent to your team via email.
- To maintain the experiment's integrity, restarting the experiment will overwrite the old results and start it from scratch.