Jun 15, 2023

Recommender systems explained: the main models and when to use them

Most recommendation projects go wrong because the team picks a model before it decides what the system should do. Choosing between collaborative filtering, content-based filtering and a deep learning model is the easy part. Knowing which one suits your data, your catalogue and your users is harder.

This guide is for product, data and engineering leads who need to choose an approach or review one. It covers the main families of recommender models, what each needs from your data, where each one fails and how to measure whether any of them is working.

What a recommender system does

A recommender system predicts how much a user will value an item they have not yet seen. It then ranks the items so the most valuable ones appear first. The item can be a product, an article, a film, a job advert, a supplier or a next step in a workflow.

Every recommender starts from the same raw material: a record of users, items and the interactions between them. It helps to picture this as a large table, the user-item matrix. Each row is a user, each column is an item and each cell holds an interaction, such as a purchase, a rating, a click or a view.

Two facts about that table shape every design decision:

It is almost empty. A typical shopper interacts with a tiny fraction of a catalogue. In most real systems more than 99 per cent of cells are blank.
Most signals are implicit. Few people leave star ratings. Most of what you know comes from behaviour, such as clicks, dwell time, add-to-basket and repeat purchase. A blank cell does not mean "not interested". It usually means "never seen".

The job of the model is to fill in the blanks well enough to rank them.

The main families of models

1. Popularity and rules

The simplest recommender shows the most popular items, the newest items or items picked by a merchandiser. It needs no history about the individual user, so it works on day one and for anonymous visitors.

It is also a baseline that more advanced models must beat. If a machine learning model cannot outperform "best sellers this week" in a fair test, it is not ready.

Use it for: new users, new sites, homepage slots and as a fallback when other models have no signal.

Limits: everyone sees the same thing, and popular items become more popular whether or not they are a good fit.

2. Content-based filtering

Content-based filtering recommends items that are similar to items the user already liked. Similarity comes from the items' own attributes: category, brand, price band, colour, ingredients, text descriptions or images.

The system builds a profile for each item, often as a vector of features, and a profile for each user from the items they engaged with. It then recommends items whose profiles are close to the user's.

Use it for: catalogues with rich, reliable product data, and new items that have no interaction history yet. A new product can be recommended as soon as it is described.

Limits: it only recommends more of the same. A user who bought one hiking boot sees more hiking boots. It is also only as good as your product data. Missing or inconsistent attributes give poor recommendations.

3. Collaborative filtering

Collaborative filtering ignores item attributes. It uses the pattern of interactions across all users. The idea is simple: people who agreed in the past tend to agree in the future.

There are two classic forms:

User-based: find users whose history looks like yours, then recommend what they liked that you have not seen.
Item-based: find items that tend to be liked by the same people, then recommend items similar to the ones you engaged with. "Customers who bought this also bought" is item-based collaborative filtering.

Item-based methods are usually more stable and easier to run at scale, because item relationships change more slowly than user tastes.

Use it for: sites with plenty of interaction data, where the value lies in connections that product attributes do not show. Collaborative filtering can learn that people who buy a certain tent also buy a certain stove, even if nothing in the product data links them.

Limits: it cannot recommend an item nobody has interacted with, and it cannot serve a user with no history. This is the cold start problem. It also struggles when interactions are very sparse.

4. Matrix factorisation

Matrix factorisation is the model-based form of collaborative filtering, and for many years it was the standard approach. It became widely known through the Netflix Prize competition, which ran from 2006 to 2009.

The method compresses the sparse user-item matrix into two smaller matrices. One describes each user as a short list of numbers. The other describes each item the same way. These numbers are called latent factors, or embeddings. The model learns them so that the dot product of a user vector and an item vector predicts the interaction in that cell.

Nobody defines the factors in advance. In a film catalogue one factor might end up tracking "serious drama versus light comedy" and another "mainstream versus niche", but the model finds these patterns by itself.

Common training methods include alternating least squares (ALS), which suits implicit feedback and runs well in parallel, and stochastic gradient descent.

Use it for: a strong, well-understood baseline when you have a reasonable volume of interaction data. It is fast to serve, because a recommendation is a nearest-neighbour search over item vectors.

Limits: the basic form has the same cold start problem as other collaborative methods, and it does not use context such as time, device or what the user is doing right now.

5. Hybrid models

A hybrid combines collaborative and content-based signals so that each covers the other's weak points. Content features handle new items and new users. Interaction patterns find the connections that attributes miss.

Hybrids can be built in several ways:

Blend the scores of separate models with weights.
Switch between models depending on how much data is available, for example content-based for a new item and collaborative once it has history.
Feed both kinds of signal into one model. Factorisation machines, for example, learn latent factors for users, items and any other features you supply, such as category or price band.

Use it for: most production systems. In practice, few mature recommenders use one technique alone.

6. Deep learning models

Neural networks extend the ideas above. They can take in many more signals and learn more complex patterns. Three designs are common:

Two-tower models. One neural network turns the user and their context into a vector. A second turns each item into a vector. The score is the similarity between the two. Because item vectors can be calculated in advance, this design can search millions of items quickly. Many large platforms use it for the first stage of retrieval.
Wide and deep models. These combine a simple linear model, which memorises specific feature combinations, with a deep network, which generalises to combinations it has not seen. They are often used for ranking.
Sequence models. These treat a user's history as an ordered sequence, much as a language model treats words. Transformer-based models such as SASRec and BERT4Rec predict the next item from the items before it. They work well when order matters, such as in a browsing session or a playlist.

Deep models can also use unstructured data directly. Text embeddings from language models and image embeddings let the system understand that two products are similar even when their metadata says nothing.

Use it for: large catalogues, high traffic and teams that can run and monitor the infrastructure.

Limits: more data, more compute and more engineering. A deep model that is poorly tuned often loses to a well-tuned matrix factorisation baseline. Start simple and earn the complexity.

How production systems fit together

Large recommenders rarely run one model over the whole catalogue. They work in stages:

Candidate generation. Fast methods, such as item-to-item similarity, matrix factorisation or a two-tower model, reduce millions of items to a few hundred possible candidates.
Ranking. A richer model scores each candidate using many features, including user history, item attributes, price, stock and context.
Re-ranking and business rules. The final list is adjusted for diversity, freshness, stock levels, margin, compliance and items the user has already bought.

This structure lets you improve one stage at a time and keeps response times low.

Problems every recommender must handle

Cold start. New users and new items have no history. Use popularity, content features and simple onboarding questions until behaviour builds up.
Popularity bias. Models trained on past behaviour favour items that were already shown often. Without correction, the long tail of your catalogue stays invisible.
Feedback loops. The system learns from clicks on items it chose to show. Over time it can narrow what users see. Some controlled exploration, such as showing a small share of less certain items, keeps the data honest.
Changing tastes. Seasons, trends and life events change what people want. Retrain on a regular schedule and give recent behaviour more weight.
Explainability. Users and internal teams trust recommendations more when they can see why an item appeared. "Because you viewed X" is a simple and effective explanation.

How to measure whether it works

Use two kinds of evaluation and do not rely on only one.

Offline evaluation tests a model on historical data that it did not see during training. Common metrics include:

Precision at K and recall at K: how many of the top K recommendations were relevant, and how many of the relevant items appeared in the top K.
NDCG (normalised discounted cumulative gain): rewards relevant items that appear near the top of the list.
Coverage and diversity: how much of the catalogue the system recommends, and how different the items in one list are.

Split the data by time, not at random. Training on next month's behaviour to predict last month's gives results that look much better than reality.

Online evaluation tests the model with real users, usually in an A/B test. Measure the outcome the business cares about, such as conversion, revenue per visitor, repeat purchase or content completion, not only click-through rate. A model can raise clicks while lowering basket value.

Offline metrics tell you which models are worth testing. Only online tests tell you which one to keep.

Choosing an approach

Use your data, not fashion, to choose:

Little interaction data, good product data: start with content-based filtering and popularity.
Plenty of interaction data: start with item-based collaborative filtering or matrix factorisation, then add content features to handle cold start.
Large catalogue, high traffic, strong engineering team: move to a staged system with a two-tower retrieval model and a separate ranking model.
Order and session matter: consider a sequence model.

Whatever you choose, build the baseline first, measure it fairly and change one thing at a time.

Before you build

The model is usually the smallest part of the work. Most of the effort goes into the data underneath it: clean interaction logs, consistent product attributes, a reliable link between sessions and customers, and a way to feed results back into training.

If you are planning a recommender and want a second view on the data, the model choice or the evaluation plan, talk to us.