Decoding Financial Times' Next Best Action Recommender System

Author: Rahul Chaudhary

In a video, Adam Gajtkowski, a data scientist at the Financial Times, presents the workings of their Next Best Action (NBA) recommender system, which aims to enhance user engagement on their platform.

Need for a Recommender System

The core rationale behind building a recommender system for a newspaper like the Financial Times is rooted in their business model. With 80% of their customers engaging through digital subscriptions, there’s an abundance of data that can be analyzed to preemptively address user disengagement. Using this data, the NBA system assists in predicting user preferences and crafting personalized engagements.

A critical engagement metric used by the Financial Times is ARBV, which stands for Recency, Frequency, and Volume. This metric allows them to quantify user interaction and thus serves as a basis for guiding their recommendation engine.

Components of the Recommender System

Gajtkowski divides the NBA system into two main components:

1. Action Values: These represent the predicted increase in user engagement from performing a particular action.
2. Likelihood Models: These predict the probability that a user will take a specific action in the future.

Workflow Overview

The workflow of the recommender system consists of both static and dynamic elements:

• The static elements are hosted in Google Cloud Storage, consisting of trained likelihood models and action values.
• The dynamic elements are updated daily and make use of the most recent user behavioral data to provide refined recommendations.

The dynamic component fetches the latest data, processes it, and outputs a ranked list of next best actions tailored for each user. Outputs are stored in BigQuery for ongoing analysis.

Understanding Action Relative Value Calculations

To compute the action relative value for given actions, Gajtkowski’s model performs the following calculation:

$$ \text{Action Relative Value} = \sum \left( \text{Action Likelihood}{user} \times \text{Weight} \right) + \sum \left( \text{Action Value}{user} \times \text{Weight} \right) $$

Where:

• Action Likelihood is the predicted probability of the user taking that action.
• Action Value indicates the expected increase in engagement from performing that action.

This calculation leads to a ranking of actions for each user, and those with the highest relative value are prioritized.

Ranking and Filtering Actions

After calculating action relative values, these values are ranked. However, they go through a resolution filter to ensure actions suggested haven’t already been performed by the user within the last three months. This means that if a user has already signed up for the newsletter in the past quarter, that suggestion gets filtered out, allowing for fresh recommendations.

Importance of the Likelihood Model

Gajtkowski points out, through correlation and principal component analysis, that the likelihood model is significantly more indicative of user engagement than action values. The analysis shows a stark distinction:

• Disengaged users cluster around a central point, indicating low interactions.
• Engaged users are spread across multiple components, suggesting that any form of engagement, regardless of action type, can be beneficial.

Action Inputs and Selections

Action selection relies on users’ engagement behaviors over a three-month window. Input components include daily engagement metrics and recent user activities. The model evaluates users’ actions based on whether they flag as true (performed) or false (not performed), passing the data to the likelihood and action value models for further processing.

For example:

• If a user frequently uses the Financial Times app, the model may flag related actions as true, building a case for elevated engagement predictions.

Statistical Evaluation

The recommender system has undergone multiple rounds of optimization. Initially, the system assumed equal importance of both components but evolved to allow fine-tuning of weights based on performance analysis.

In a split A/B testing of recommendations, the NBA system demonstrated significant improvements:

• A 63% boost in newsletter signups.
• A 4% increase in newsletter open rates.
• An uptick in the click-through rate for newly supplied newsletter promotions by 4.4%.

Conversely, some actions did not yield positive engagement, suggesting that not all recommendations are universally effective.

Iterative Model Development

The development of the NBA system hasn’t been static. Aspects like adjustable importance of model components and dynamic action value rankings were introduced to cater to changes in user behavior. The team continuously evaluates potential new features or actions to enhance predictive capabilities.

Monitoring System Performance

Performance assessments of the recommender system occur daily through automated dashboards and monitoring tools. Metrics like Root Mean Squared Error (RMSE) are closely monitored to ensure consistent performance. Alerts are set for team members if the model runs unexpectedly fail or if statistical outputs deviate from norms.

Future Enhancements

Plans for future iterations include:

• Exploring new features for gradient-boosted machine models.
• Tailoring action value rankings based on user cohorts to drive higher personalization levels.

By adapting the model architecture and assessing the connections between user engagement and churn rates, the Financial Times continues to refine their approach to personalized recommendations.

If you’re interested in seeing the presentation yourself, you can watch it here.

This ongoing focus on data-driven user engagement not only highlights the Financial Times’ commitment to enhancing reader experience but also underscores the dynamic nature of data science applications in media.