BM25 (Best Match 25)

What is BM25?

BM25 (often called Okapi BM25) is a classic information-retrieval ranking function that scores documents based on term frequency, inverse document frequency, and document length normalization. It’s lexical (keyword-based), fast, explainable, and still a state-of-the-art baseline for search.

How BM25 works (quick)

• Token match: Break query and documents into tokens; match shared tokens.
• TF & IDF: More occurrences of a rare term boost the score; common words matter less.
• Length normalization: Prevents very long documents from winning just by having more words (controlled by b).
• Saturation: Extra repetitions of a word help but with diminishing returns (controlled by k1).
• Per-field control (BM25F): Apply BM25 per field (title, description, attributes) with different boosts, then combine.

Typical parameters: k1 ≈ 1.2–2.0, b ≈ 0.6–0.9. Tune per language and corpus.

Why it matters in e-commerce

• Great first pass: BM25 is ideal for initial recall from large catalogs before re-ranking.
• Explainable: Easy to debug—scores come from visible tokens, not black boxes.
• Low latency: Works well under storefront SLAs and peak traffic.
• Hybrid ready: Pair BM25 recall with vector retrieval and re-ranking (LTR/BERT) for best relevance.

Best practices

• Use BM25F: Heavily boost title, moderately attributes, lightly description.
• Language-aware analysis: Proper tokenization, stopwords, stemming/lemmatization by locale.
• Synonym expansion: Apply query-time synonyms (and spelling tolerance) to reduce vocabulary mismatch.
• Field hygiene: Keep titles concise; expose critical attributes as separate fields.
• Tune k1/b by index: Different product verticals and locales benefit from different settings.
• Hybrid pipeline: BM25 recall → vector retrieval → ML/transformer re-rank of top-K.

Challenges & trade-offs

• Vocabulary mismatch: BM25 can miss relevant items if words differ (sofa vs couch).
• Typos & morphology: Needs spell-tolerance and proper analyzers to cope with misspellings and inflections.
• Short queries/short fields: Sparse signals reduce separation; field boosts help.
• Non-text attributes: BM25 doesn’t “understand” numeric/range facets (price, size) without filters.

Examples (storefront)

• Query “gtx trail running shoes men 45”: BM25 boosts titles with GTX and trail; filters handle men and 45.
• Query “wireless anc headphones”: High weight from title/attributes; description adds minor support.
• Query “red midi dress floral”: Title + attribute tags dominate; synonyms (crimson, rose) at query time widen recall.

Summary

BM25 is a fast, dependable ranking baseline for product search. Use it to retrieve a strong candidate set, then layer filters, business rules, and semantic re-ranking to deliver highly relevant results under tight latency.