A/B Testing

• Product Analytics in PM Interviews

  July 23, 2024

  I recently watched a video titled How to Think Product Analytics in PM Interviews by Amazon Sr PM, Vivek Pandey. The Essence of Product Analytics Product analytics isn’t just about crunching numbers—it’s about understanding the game you’re playing. The speaker emphasized that product managers essentially do four things daily: Ideating Prioritizing Persuading Executing Analytics serves as the backbone for all these activities, bringing order to the chaos of constant product changes.

• Decoding A/B Testing: The Consequences of Continuous Monitoring

  July 9, 2024

  I recently watched a detailed seminar by Ramesh Johari of Stanford University (here) that delved into the practical implications of A/B testing, particularly in the context of continually monitored experiments. The focus was on how standard statistical methods may fail when applied in this dynamic scenario and how his team at Optimizely has adapted methodologies accordingly. A/B testing, or A/B experimentation, is essentially a modern incarnation of randomized controlled trials (RCTs).

• The Ultimate Guide to A/B Testing: Lessons from Ronny Kohavi

  May 23, 2024

  I recently watched a video titled The Ultimate Guide to A/B Testing | Ronny Kohavi. Kohavi, a seasoned expert on A/B testing, lays out critical lessons surrounding experimentation culture that resonate well with data-driven organizations. Kohavi emphasizes an essential axiom: “Test everything.” Every code change or feature introduced should undergo an experiment because even small tweaks can yield surprising results. For instance, Kohavi cites examples from Bing where a minor change in ad display led to a stunning 12% revenue increase, equating to around $100 million at the time.