Impact of an incorrect retention metric

Most common mistakes to avoid:

  1. Not aligning with natural frequency:

    1. Too much - Spam: e.g., Express sending 1-2 emails a day about buying a suit, I only buy a suit much less frequently than that.
    2. Too little - forgettable.

  2. Combining Actions: teams can be confused or choose the metrics that are easier to influence (e.g., clicks by having content with click-bait).

    At one point we combined the actions of pins and clicks into a metric called WARCS (Weekly Active Repinners and Clickers). This led the team in the wrong direction. Casey Winters, Former Growth at Pinterest, Grubhub

  3. Optimizing for revenue: revenue is the output, the input is product usage. Set retention metrics based on inputs (usage) not outputs.

Defining your metric

Frequency

What is the natural frequency in which the user experiences the problem? Validate your qualitative hypothesis with a histogram.

  1. Select a use case:

    Take all users who are more than 28 days old, activated, and completed the core action at least once in the last 28 days. Weed out those that haven’t activated.

  2. Create a frequency histogram: e.g., this shows that most users were active 23 of 28 days.

    Untitled

  3. Analyze the distribution: if most users congregate at 20+ then your natural frequency is daily, for weekly ~4-7 days, monthly about 1-2 per 28 days.

    1. If you have a longer time horizon, plot it out for a longer period of time.
    2. If there’s a mismatch between histogram and the frequency hypothesis, there’s a gap to address - either you improve the product to solve the more frequent use case properly, or update your natural frequency hypothesis.

Action hypothesis

Core action hypothesis:

  1. What action indicates that you’re delivering value to the user? Our core action hypothesis must match to this.
  2. Create a hypothesis of which actions correspond with value that solves the user’s problem.
  3. Validate with quantitative data by looking at a histogram (again) 🙂

Core behavior

Confirm the core action hypothesis

  1. Form groups that successfully did that action for successive periods
  2. Create a cohort chart (or retention curves) for different action hypotheses
  3. Analyze what’s happening - for example, pinning leads to better retention per the data in the chart. Just viewing the feed is much less effective at retaining users.

Untitled

Who?

Who is included in the metric? Who is a user?

  1. Instead of “weekly active users” you can define at “weekly active repinners” since it is specific to those that pinned and will re-pin.

Metric examples