Why do I get differing numbers when comparing Funnels and Insights?
Funnels and Insights have different default counting methods. Insights by default counts events using "Totals", whereas Funnels are by default calculated with the Conversion Criteria in "Uniques".
Funnels uses "Uniques" as its default as it is intended to help you gauge your product's effectiveness in urging your users to complete certain flows (e.g. Registration, Purchase). If the numbers were in Totals, a few select power users could skew your data. Each user will only be counted once for a funnel within any given time period you set. The Funnels report with 'Unique' conversion criteria will show only the conversion rates/steps the first time a user enters the funnel, even if they complete or drop out of the funnel multiple times, hence giving a 'Unique' count.
You can toggle the conversion criteria to 'Total' or 'Sessions' counting method. Please note that not every Step 1 event will enter a 'Total' conversion funnel. Learn more about counting methods here.
Can multiple events count towards the same Funnel step?
Custom events were designed to specifically target this exact issue. With custom events, you can essentially "combine" events into a single event so that you can create Funnels that measure the following:
Step 1: Event A OR Event B
Step 2: Event C
This allows users to make multiple paths to the same end goal. Learn more about creating custom events here.
How does Mixpanel handle simultaneous events?
Mixpanel Funnels include a built-in two second grace period. This means any consecutive steps that have timestamps within two seconds of one another are interchangeable. This window allows for inconsistencies that arise with cross-platform tracking, batch event sending, mobile time tracking, or variations in connection speed.
What does this mean in terms of your funnel? In a funnel with step A -> B, if a user completes event B first and then performs event A within two seconds, Mixpanel counts this as a conversion from A -> B or B -> A.
What is the difference between Per-Step Filtering and Global Filtering?
You can use both per-step and global filters to decide which users and events to include in a funnel. While per-step filters only apply to a single step and global filters apply to all steps in a funnel, there is more to the distinction than just that.
Next, let's expand on the differences between per-step (pre-query) and global (post-query) filtering and how it can affect your results.
Per-Step Filtering (Pre-Query)
Per-step filters allow you to apply event or user property filters to each step of your funnel. This adds granularity and specificity to your conversion criteria. In selecting a per-step filter, you are tightening the parameters of events that qualify for conversion.
Per-step filters are performed pre-query. Thus, these filters will impact your funnel’s conversion by removing events that don’t qualify from consideration of being included in a funnel calculation.
Let's illustrate this with some examples.
Consider a four-step funnel where a user must Search > Item Detail Page > Add to Cart > Complete Purchase, all within a one-day conversion window.
As you can see in the below activity feed, this user's first Search event happens more than one day before any Item Detail Pages, and as such, they would not convert in this funnel.
However, if I put a per step filter on the Search event and require a user to use a Brower = Safari, then I will restrict entry into the funnel to only Search events on Safari browsers.
Below we see the same user's activity stream where their first Search event is with the Chrome browser, and thus this user’s first Search event is not included in a funnels calculation. However they have a second Search that is using Safari, and this event qualifies them to enter into the funnel. Unlike before when we did not filter, now each of their subsequent steps happens within the conversion window, and this user reaches full conversion to the end of the funnel.
If we were to take it further and place a per-step filter on each and every step in this funnel, so that all steps must be completed with events where Browser = Safari, then this user would only convert to the second step of the Item Detail.
This is because, as we can see below, the first Search/Safari step is followed by two steps that aren’t tracked in the funnel: Item Detail Page/Chrome that is filtered out of this funnel because it does not fulfill the Browser= Safari criteria, and Add to Cart/Safari that is not considered because it is not preceded by an Item Detail Page/Safari. The next step that is tracked in the funnel (Item Detail Page/Safari) converts the user to Step 2, but then Add to Cart/Chrome is filtered out of the funnel. Since there is not another Add to Cart/Safari before the Purchase event or before the 1 day conversion window runs out, this user times out of the funnel after Step 2.
Global Filtering (Post-Query)
Global filters apply your selected filters to the entire report after the Funnels query has been run. Since Funnels will calculate conversions first and then apply your global filters, entire conversions with any part that does not meet these filter requirements will be excluded from the report's results.
For example, let's use the same four-step funnel where a user must Seach > Item Detail Page > Add to Cart > Complete Purchase, all within a one-day conversion window.
In the below example, the Funnels query will calculate a conversion because the user moves from Search, then Item Detail Page, then Add to Cart and lastly to Purchase within the 1 day conversion window. However, after the query is calculated, the global filter of Browser = Safari is applied. Since there are steps of this calculated funnel that have Browser = Chrome, the entire funnel will be filtered out of the aggregate results.
How does Mixpanel calculate statistical significance?
Note
This section is intended for users who want to understand the mathematics involved in statistical significance in depth.
To determine whether a particular segment’s conversion rate is significantly different from the overall conversion rate, we use a hypergeometric distribution to calculate statistical significance. The hypergeometric distribution is used to model the probability of picking k items of a particular type in n attempts without replacement from a population of size N having K items of the same type.
For example, let’s say we have a sock drawer with 20 socks, 10 blue and 10 red. If we randomly picked 10 socks one at time from the drawer without putting them back between picks, and we wanted to know the probability of 9 of those socks being red and 1 of them being blue, we would use a hypergeometric distribution to calculate that.
This is applied to funnels by considering the total number of users who enter the funnel to be a finite population of size N, out of which a subset of users convert (K). We then estimate the probability of getting k conversions in a particular segment (given that there were n users who entered the funnel in that segment) if users in that particular segment were picked at random from the overall user set. The higher the probability, the higher likelihood that variations we see in conversion rate are due to random chance.
To calculate the actual p-value, we estimate the hypergeometric cumulative distribution function (CDF) for N, K, n.
In the CDF, the value of any point (X) represents the probability that a random draw would result in fewer conversions P(k < X). 1 - P(k < X) represents the probability that a random draw would result in more conversions P(k >= X).
These two probabilities are used to represent the probability that the selected segment will either outperform (P(k < X)) or underperform (P(k > X)) the overall set of users. Mixpanel takes the higher probability of the two, and calculate the p-value as 1 - max(P(outperform), P(underperform)).
Comments
Please sign in to leave a comment.