Mixpanel’s Impact report measures the effects of product or marketing launches on your key metrics. Impact calculates the user adoption of the launch, the impact of the launch on an important event, and the differences between users that adopt the launch and those that do not.

Impact is currently in open BETA and is subject to change.

## Access Report

To access **Impact**, click on **Analysis **in the top navigation, then select **Impact**.

## Build a Query

To build an Impact query, first select a launch event. This is the event that you are measuring as the cause of change.

Add any additional filters to narrow the launch event parameters by selecting the filter button.

Select a metric event. You are measuring the impact of the launch event on this metric event. Add additional filters to narrow the metric event parameters.

Select the time range. The time range is a fixed period of time that determines the adopters and non-adopters of the launch event. For example, if the time range is set at 30 days, then that user has 30 days to perform both the launch event and metric event to be considered an adopter.

The chart will not necessarily change if you change the time range.

You can also apply a global filter from the same location.

## Report Calculation Details

The report presents the results in plain english, in an Impact table that includes the Impact values, and in a chart that displays the impact of the launch over time.

Adopters are users that have done the launch event. Non-adopters are users that have not done the launch event. Both adopters and non-adopters must perform the metric event to be included in the report.

The Adoption Rate is the number of users that perform the launch event divided by the total user count:

\[\%\,adoption = average\left( (adopters) \over (adopters + non\,adopters) \right)\]

### Impact Chart

The Impact Chart shows how the rate of metric event occurence changes over time. The y-axis of the chart is the average number of the metric event count and the x-axis is time spanning 30 days.

Unlike other Mixpanel charts, the Impact Chart displays time in relative time, not calendar time. The chart centers around the first day that the launch event is available, or “day zero”. The chart displays the 15 days before and after day zero.

Every user in the report can have a different day zero. For users in the adopter group, day zero is the day that they perform their first launch event. For users in the non-adopter group, day zero is the day the first adopter performed the launch event (which is most likely the launch day of the feature).

Each data point on the chart is calculated as follows:

\[metric\,event\,rate = { (metric\,event\,count) \over (total\,user\,count)}\]

Each point on the line is the average number of times users in the group did the metric event on that relative day.

For example, if 10 adopters did the launch event for the first time five days ago, and they then did the metric event a total of 30 times today. The +5 days point on the adopter line would read 3. The math would be as follows:

- 30 = The number of metric events performed by the adopters on day 5
- 10 = The number of users who did the launch event 5 days before the first day
- 3 = 30/10

You can see how frequently users in each group perform the metric event, both before and after the launch.

### Impact Table

The Impact Table summarizes the results of the Impact chart. It displays the average rate at which users in the adopter and non-adopter groups performed the metric event over the course of the 15 days before and after launch.

Each of these four averages is also reflected on the chart as a horizontal dotted line.

The Impact Table compares users that performed the launch event and those that did not. For both groups of users, Mixpanel calculates the average number of times per day that users performed the metric event before the launch, after the launch, and the difference between the two (the delta).

The average number of times that users performed the metric event before the launch is calculated by:

\[group\,average\,value = average\left( (group\,metric\,event\,count \div group\,user\,count) \over (day\,count) \right)\]

The average rate is calculated before the first day of the launch event, after the launch event, and the report also displays the difference between those rates.

Lastly, the Impact is calculated by comparing the results of the adopters and non-adopters. These results are presented as an overall delta with a corresponding confidence score.

The overall delta is calculated by subtracting the pre-launch delta from the post-launch delta.

The confidence score is a measurement of statistical significance, and how likely it is that that the overall delta is statistically significant.

## Interpret the Results

In general, when the overall delta is positive, and the confidence score is 95% or more, it indicates a successful launch. The most successful launches will see the rate at which adopters perform the metric event increase post-launch, while the same rate for the non-adopters remains relatively constant.

In the Impact chart, look for the average gap in usage between the adopters and non-adopters to increase after the launch. Typically, it is preferable for the increase in the gap to be mostly driven by an increase in the rate at which adopters perform the metric event, rather than a decrease in the rate at which non-adopters perform it.

## Confidence Calculation

Impact includes confidence to indicate the statistical significance of report calculations. Interpret the confidence as the probability that the final delta is primarily caused by the launch event, as opposed to existing by chance.

The confidence is calculated as a confidence interval using a cumulative distribution function while assuming a normal distribution.

The confidence variables are as follow:

**X**= The number of events per day performed by adopters in the pre-launch period.**Y**= The number of events per day performed by adopters in the post-launch period.

**A**= The number of events per day performed by non-adopters in their pre-launch period.**B**= The number of events per day performed by non-adopters in their post-launch period.

**Y - X**= The difference in number of events per day performed by adopters between post-launch and pre-launch periods.**B - A**= The difference in number of events per day performed by non-adopters between the post-launch and pre-launch periods.

- The mean of
**Y - X**over time is denoted by:

\[\mu_{Y-X}\]

- The mean of of
**B - A**over time is denoted by:

\[\mu_{B-A}\]

- The standard deviation of
**Y - X**over time is denoted by:

\[\sigma_{Y-X}\]

- The standard deviation of
**B - A**over time is denoted by:

\[\sigma_{B-A}\]

With the variables assigned, the calculations for confidence are as follow:

\[\mu_{final} = \mu_{Y-X} - \mu_{B-A}\]

\[\sigma_{final} =\sqrt{{\sigma_{Y-X}^2 \over N_{Y-X}} - {\sigma_{B-A}^2 \over N_{B-A}}} \]

\[z_{score} = \left|\mu_{final} \over \sigma_{final}\right|\]

\[p_{value} = 1-cdf(z_{score})\]

\[confidence = 1 - p_{value}\]

## Comments

Please sign in to leave a comment.