Creating and running an A/B-test

This page covers the classic A/B test workflow.

For the new UI walkthrough, see Using the new A/B tests UI.

Create a new A/B test

1. Open More settings… in the left navigation.

2. Open A/B tests.

1. Select the channel in the top bar.

2. Make the channel editable.

You can create A/B tests in editable child channels and editable non-hierarchical channels.

You cannot create them in parent channels.

1. Click CREATE NEW TEST +.

Configure the A/B test

1. Give the test a clear name.

2. Choose UNTIL TARGET EFFECT REACHED when you need statistically reliable results.

Avoid a fixed end date unless statistical significance is not important for this test.

1. Set the start date and time.

Set the start far enough in the future to finish your variant B changes.

After the configured start time passes, you can no longer change variant B.

1. Leave the other values at their defaults unless you need a specific setup.

2. Click CREATE TEST.

3. Save the transaction.

Use to save and close the transaction.

A backup comment is requested so you can identify the backup later.

Use to discard the changes and close the transaction.

1. Update your shop data.

Edit variant B

1. Select the B variant of your A/B test.

2. Make it editable.

3. Change only the settings you want to measure.

These changes apply only to variant B.

They do not affect the base configuration.

Keep the scope small.

That makes the results easier to interpret.

For the supported feature list, see A/B-Test.

To return to the base configuration, select Base configuration in the A/B test selector.

Run the A/B Test

1. Open the test actions with .

2. Select begin test.

3. Save the configuration.

It is safe to start the test before the scheduled time.

The test still waits for the configured start time.

1. Update your data.

The A/B test starts at the configured start time.

When it ends, the status shows completed and Ended ... ago.

Avoid running several A/B tests in the same channel at the same time.

Analyze A/B test results

1. Select the channel that contains the completed A/B test.

2. Make the channel editable.

3. Review the results.

Key metrics:

• Search Count

• Click Through Rate %

• Conversion %

• Sales per Search ∅

Open a metric header for more detail.

Compare the baseline and variant B.

Usually, higher values indicate better performance.

In the example above, variant B performs better in click-through rate and conversion.

Variant A performs better in sales per search.

Select Start analysis for a more detailed result.

Finalize the A/B Test

1. Select the channel that contains the completed A/B test.

2. Make the channel editable if needed.

3. Choose the final action:

• Click apply variant B to roll out the tested configuration.

• Click retain variant A to keep the baseline.

If reliability is unclear, keep the baseline and repeat the test.

Use UNTIL TARGET EFFECT REACHED instead of a fixed end date when statistical validity matters.

1. Save the changes to the channel.