Experiments

The Experiments report analyzes how A/B test variants impact your metrics.  Experiments does this by calculating the difference between variant groups and the effects of the variants on selected events.

Experiments requires an A/B test, its variant, and a dashboard that contains the metrics you are measuring. An experiment query calculates the variants’ effects on the dashboard metrics by calculating the delta and the lift between the two variants. 

Access the Report

To access Experiments, click on Analysis in the top navigation, then select Experiments.

Screen_Shot_2020-01-14_at_5.35.19_PM.png

Build a Query

To use Experiments you must have a dashboard, and you can either use an existing experiment or you must build one in the query builder.

Select an Experiment

Custom Experiment - This option allows you to define the control and variant groups of the experiment. These groups can be defined by cohort, user profile property, or event property filters.

Messages with Variants - This option is available if you have created a message with multiple variants in Mixpanel’s Messages tool. This allows you to build an experiment using those variants.

Mobile A/B Tests - This option is available if you have created a mobile A/B test in Mixpanel’s A/B Testing tool. This allows you to build an experiment using those variants.

Tracked Experiments - This option is available if you have experiments in your implementation. Mixpanel automatically detects any experiments that began in the last 30 days, and the report detects and displays them in the dropdown.

Choose a Control Group

Select a group of users that are not exposed to the variant as your control group. For example, in onboarding flow testing, users exposed to the original, not new, onboarding flow should be the variant.

In a Custom Experiment, the control group can be a cohort or any other users filtered by events and properties.

In Messages or Mobile A/B test experiments, you can choose the control group from the list of variants created as part of the Message or Mobile A/B Test.

It is important to ensure that this group is a true control. Introducing two new variants may abstract the report results.

Select a Variant Group

Select the group of users exposed to the new experience as your variant group. For example, in onboarding flow testing, users exposed to the new onboarding flow should be the variant.

In a Custom Experiment, the variant group can be a cohort of any other users filtered by events and properties.

In Messages or Mobile A/B test experiments, you’ll be able to choose the variant group from the list of variants you created as part of the Message or Mobile A/B Test.

Select a Date Range

Select the date range of the experiment. In most cases you should choose the date your experiment began as the start date. 

Supported Metrics

Experiments will run calculations on the following supported metrics:

  • Insights - line charts with “Total” count, including charts with breakdowns.
  • Insights - line charts with “Unique” count, including charts with breakdowns.
  • Funnels - funnels with any number of steps.

Report Calculation Details

The following section describes the equations used in the Experiments report.

Control and Variant Group Rate

The group rate is calculated for both control and variant groups. It is calculated differently depending on the selected metric type.

If calculating using totals in Insights, then the group rate is calculated as:

\[Group\,Rate= {{ (\# \,of\,events) \over (\# \,of\,users)} \over (time)}\]

If calculating using uniques in Insights, then the group rate is calculated as:

\[Group\,Rate= { (\# \,of\,users\,who\,performed\,metric\,event)  \over (\# of\,users\,in\,group)}\]

This value is  a percentage, because the maximum possible value is 1.  We therefor display the percentage of users in the control group who performed the metric event.

If calculating using funnels, then the rate is the overall conversion rate of the funnel for users in the group.

Lift and Lift Trend

Lift is the percentage difference between the control group and variant group rates. Lift is calculated as (variant rate - control rate) / control rate.

\[Lift= { (variant \,group\,rate - control \,group\,rate) \over (control \,group\,rate)}\]

You can also switch between lift and the delta, which is the absolute difference in rates, variant rate minus control rate.

Confidence

Confidence is the probability that the lift or delta between your control and variant groups is significant.

For conversions we calculate a standard confidence score for binomial outcomes, and for event counts we calculate a standard confidence score for poisson outcomes.

The trend line in the column displays how confidence has changed over the selected date range.

Interpret the Results

The Experiments report locates significant differences between the Control and Variant groups.   Metric rows in the table are highlighted when any difference is calculated with 95% or greater confidence.

  • Positive differences, where the variant rate is higher than the control rate, are highlighted green.
  • Negative differences, where the variant rate is lower than the control rate, are highlighted red.
  • Statistically insignificant results remain gray.

Screen_Shot_2020-01-14_at_5.38.54_PM.png

Add Experiments to an Implementation

This guide explains how to track users in each variant of an A/B test by integrating experiment data into your implementation.

Mixpanel will automatically populate the Experiment, Control, and Variant dropdowns within the report if sent in the proper format.  

Mixpanel scans for experiments that began in the date range you’ve selected for the report.  If any are found, then they will appear under the “Tracked Experiments” sub-header. To do this  you must send data in the following format:

Event Name: “$experiment_started”

Event Properties:

  • “Experiment name” - the name of the experiment to which the user has been exposed
  • “Variant name” - the name of the variant into which the user was bucketed, for that experiment

An example track call would appear like this:

mixpanel.track('$experiment_started', {'Experiment name': 'Test', 'Variant name:' 'v1'})
Did this answer your question?

Comments

0 comments

Please sign in to leave a comment.