A/B-Tests

A/B tests compare your base configuration against one test variant in live traffic.

They help you validate changes before rollout.

Resource usage and number of tests

• Thesaurus
• Stopwords
• Preprocessor Entries
• Trash Can Entries
• Campaigns
• Search Algorithm Settings
• Result listing Settings
• Personalization (except for (de)activating the module)
• Semantic Enhancer
• Facets & ASN: Filters
• Single Word Search
• AtlasAI
• Ranking Rules, but with the following exceptions:

  • Ranking Rules are only A/B testable for the scopes 'Search' and 'Navigation', not for the scope 'Suggest'.
  • The ranking for the suggest module will always use the Ranking Rules from the base variant and never apply changes from the B variant.

Regular A/B tests require one additional in-memory copy of the configuration per test.

Ranking rule tests require one additional ranked product list.

You can create multiple A/B tests.

More parallel tests need more traffic.

That means each test usually needs longer to reach reliable results.

Support overview

Fully supported

These features support A/B testing without known module-specific limits:

• Thesaurus
• Stop Words
• Preprocessor entries
• Trash entries
• Campaigns
• Result Listing
• Semantic Enhancer
• Single Word Search
• Atlas AI
• Vector Search
• Filters

Limited support

These features support A/B testing with restrictions:

• Personalization

  • Personalization settings are supported
  • Activating or deactivating the module is not supported
• Ranking rules

  • Supported for Search and Navigation
  • Not supported for Suggest
  • Suggest always uses the base variant
• GPT Synonyms

  • You can activate or deactivate the module
  • You can choose whether to apply pending synonyms
  • You can add, modify, and delete synonyms
  • You can access synonym generation in the Cockpit
  • Other general settings on that page stay disabled
• Recommendations

  • Manual recommendations are supported
  • Other recommendation types are not supported
  • Matrix edits are not supported
• Search Algorithm

  • Standard search settings are supported
  • Exact Search support is still being verified

Not supported

These features do not support A/B testing:

• ASO
• Field settings
• Geo
• CSI
• Suggest
• Import and database settings

Important note

Send the sid parameter with every request.

sid contains the session ID that assigns each request to variant A or B.

Without sid, requests cannot be assigned reliably.