Stemming

What is Stemming?

Stemming trims words to a root form using heuristic rules (e.g., Porter stemmer). Unlike lemmatization (which uses vocabulary/grammar to find the dictionary lemma), stemming is faster but rougher.

How It Works (quick)

• Tokenize → stem: Apply language-specific suffix rules (e.g., “shoes” → “shoe”, “running” → “run”).
• Index & query: Stem at both index and query time to align forms.
• Field strategy: Apply stemming on long text fields; keep exact fields (brand/SKU) unstemmed.

Why It Matters in E-commerce

• Recall: Matches morphological variants (“jacket” ↔ “jackets”).
• Localization: Helps across plural/case systems (EN, DE, ES), if tuned per locale.

Best Practices

• Mix with lemmatization: Prefer lemma where available; fall back to stem for speed.
• Per-locale analyzers: Use the right stemmer (or none) per language.
• Protect precision: Never stem SKU/MPN/brand exact fields; be cautious in titles.
• Phrase integrity: Combine with phrase/proximity so stems don’t break units.
• Evaluate: Track Precision/Recall trade-offs and zero-result changes.

Challenges

• Over-stemming (“organization” → “organ”), under-stemming, compounds (e.g., German), and brand collisions.

Examples

• Query “jackets” finds items titled “jacket.”
• Stem in descriptions, but keep exact brand unstemmed to avoid drift.

Summary

Use stemming to lift recall on general text, but protect exact fields and high-precision phrases. Favor lemmatization where quality matters, and measure the impact.

‍