Simultaneous A/B Tests

Since it is possible to run multiple A/B tests at once, and a user can be in more than one concurrent A/B test, you should verify that your code does not allow collisions between multiple tests by checking that tweaks are not applied to the same variables and elements.

If Tests Don’t Target the Same Users

The Mixpanel UI allows you to run multiple A/B tests at once, but it is best practice to only run one A/B test at a time, even if your tests target different users. Creating individual, independent testing environments allows you to accurately isolate and gauge the effects of your variables. In other words, running multiple tests concurrently makes it difficult to pinpoint the driving cause for user behavior.

Instead of multiple A/B tests, consider one A/B test with many variants, which will allow you to better analyze the results of your testing.

How do I measure the effectiveness of my A/B tests?

Which Test will Each User Get?

Concurrent non-conflicting A/B tests are displayed simultaneously, but those acting on the same element/variable are not.

If one user is targeted by multiple A/B tests from different projects, the A/B test they see depends on several factors, including whether they’re on iOS or Android. 

Is this article helpful?
2 out of 4 found this helpful



Article is closed for comments.