Curated Data Science by Rahul

Product Analytics in PM Interviews

Author: Rahul Chaudhary

product-management analytics A/B-testing funnel-analysis

I recently watched a video titled How to Think Product Analytics in PM Interviews by Amazon Sr PM, Vivek Pandey.

The Essence of Product Analytics

Product analytics isn’t just about crunching numbers—it’s about understanding the game you’re playing. The speaker emphasized that product managers essentially do four things daily:

  1. Ideating
  2. Prioritizing
  3. Persuading
  4. Executing

Analytics serves as the backbone for all these activities, bringing order to the chaos of constant product changes. When interviewers probe candidates about analytics, they’re really assessing:

  1. Can you identify relevant data?
  2. Can you interpret data correctly?
  3. Can you use data to persuade others?
  4. Are you realistic about what data can be collected?
  5. Do you understand which metrics you can control as a PM?

The Framework: Four Types of Product Analysis

The speaker categorized product analysis into four buckets:

  1. Sizing the opportunity
  2. Prioritization
  3. Success definition
  4. Issue diagnosis

Let’s break these down.

Sizing the Opportunity

This involves five key steps:

  1. Clarify the scope
  2. Identify the base population
  3. Determine addressable segments
  4. Estimate purchase frequency
  5. Calculate unit price

The speaker walked through an example of launching a bicycle-based food delivery service in Seattle. He estimated a potential annual revenue of $5.4 million by following these steps:

320,000 * (30,000/320,000) * 3 * 5 * 12 = $5.4 million annually

This framework can be applied to various product scenarios, not just e-commerce. The key is to make logical assumptions and clearly articulate your reasoning.

Prioritization, Success Metrics, and Issue Diagnosis

For these three types of analysis, the speaker introduced a powerful tool: the funnel.

The Funnel: A Powerful Framework for Product Analysis

The funnel concept is central to product analytics, serving as a versatile tool for prioritization, success definition, and issue diagnosis. Let’s dive deeper into how the speaker elaborated on this framework.

Understanding the Funnel

The speaker presented a basic e-commerce funnel:

100% -> Site visitors 70% -> Item detail page views 20% -> Add to cart 7% -> Checkout 3% -> Purchase completion

However, he emphasized that understanding the funnel is about more than just knowing these percentages. It’s about comprehending the entire user journey and the factors influencing each step.

Inputs and Outputs

For each step of the funnel, there are inputs (factors you can influence) and outputs (results you measure). The speaker provided several examples:

Inputs for Site Visitors:

Inputs for Item Detail Page Views:

Inputs for Add to Cart:

Inputs for Checkout:

Outputs:

Levers of Control

A critical point the speaker made is that not all inputs are within a product manager’s control. For instance, you might not be able to directly influence product ratings or reviews, but you can control the UX of how reviews are displayed.

Using the Funnel for Prioritization

When prioritizing features, the speaker suggested considering:

  1. What part of the funnel does each feature impact?
  2. How much impact will it have on that step?
  3. How does that translate to the overall goal (e.g., revenue, user growth)?

He gave an example:

While Feature B seems to have a smaller percentage increase, it might have a more significant impact on the bottom line because it’s further down the funnel.

Success Definition with Funnels

The funnel helps define success metrics for new features or products. The speaker emphasized choosing metrics that are:

  1. Relevant to the part of the funnel you’re targeting
  2. Have low variance (to get statistically significant results faster)
  3. Align with overall business goals

For instance, “orders per user” might be a better success metric than “revenue per user” due to lower variance.

Issue Diagnosis Using Funnels

When a key metric drops, the funnel provides a structured approach to diagnosis:

  1. Start at the top of the funnel and work your way down
  2. Identify at which step the drop-off occurs
  3. Analyze the inputs for that specific step
  4. Consider segmenting data (e.g., by platform, user type, geography) to isolate the issue

The speaker shared an interesting anecdote about a issue at Groupon where the problem was ultimately traced to screen sizes on different mobile devices - highlighting how granular this analysis can get.

Funnel Variations

The speaker noted that funnels can vary by:

  1. Platform (e.g., mobile vs. desktop)
  2. User segment (e.g., new vs. returning users)
  3. Product category

Understanding these variations is crucial for accurate analysis and targeted improvements.

Limitations of the Funnel Model

While powerful, the speaker acknowledged that the funnel model has limitations:

  1. It doesn’t capture non-linear user journeys (e.g., users jumping back to earlier steps)
  2. It may oversimplify complex user behaviors

He mentioned more advanced techniques like Markov chain analysis to model more complex user flows, but emphasized that the basic funnel model is still invaluable for its simplicity and broad applicability.

Practical Application in Interviews

For product management interviews, the speaker stressed the importance of:

  1. Quickly sketching out a relevant funnel for the product in question
  2. Identifying key inputs and outputs at each step
  3. Demonstrating an understanding of which factors are controllable
  4. Using the funnel to structure your approach to prioritization, success metrics, or problem diagnosis

By mastering the funnel framework, product managers can bring structure to their thinking and communicate their ideas more effectively, both in interviews and on the job.

This framework helps in:

The Power and Pitfalls of A/B Testing

The speaker devoted significant time to A/B testing, highlighting its importance in data-driven decision making. Here are the key points:

  1. Metric selection is crucial. Choose metrics with low variance to get faster, more reliable results. For example, “orders per user” is better than “profit per user” due to lower variance.

  2. Understand statistical significance. Don’t run tests until they become significant; set a sample size beforehand and stick to it.

  3. Avoid peeking at results before reaching the predetermined sample size. The speaker shared a startling statistic: in a set of 100 random experiments, 60% showed statistical significance at some point before reaching the full sample size, but only 1% was truly significant at the end.

  4. Consider seasonality. Ensure your test runs long enough to account for different days of the week, times of day, etc.

  5. Be aware of situations where A/B tests aren’t feasible, such as with content-based services like Netflix or Prime Video.

The Myth of “If You Can’t Measure It, You Can’t Manage It”

The speaker concluded with an interesting tidbit: the often-quoted phrase “If you can’t measure it, you can’t manage it” is actually a misquotation. W. Edwards Deming, to whom this is often attributed, actually said the opposite: “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.”

This serves as a reminder that while data is crucial, it’s not everything. Product management requires a balance of data-driven decision making and intuition. You can’t always wait for perfect data; sometimes, you need to take calculated risks based on your judgment and experience.