Linguistic Indexing

What is Linguistic Indexing?

Linguistic indexing augments basic token storage with linguistic signals: lemmas/stems, phrases (bigrams/trigrams), part-of-speech cues, entities/attributes, and locale rules. The goal is higher recall and cleaner precision using language-aware fields.

How It Works (quick)

• Pre-processing: Tokenize, normalize case/diacritics, split compounds.
• Morphology: Store lemma fields alongside surface forms; optional stems.
• Phrases: Build bigram/phrase fields for common multi-word units.
• Entities & attributes: Extract brand, model, material, size into structured fields.
• Per-field analyzers: Titles vs descriptions vs attributes use different analyzers.
• Hybrid ready: Works with lexical scoring and feeds features to re-rankers/LTR.

Why It Matters in E-commerce

• Intent capture: Matches natural phrases (“air max 270”) and inflections.
• Quality facets: Reliable attributes enable filters and clean URLs.
• Explainability: Highlights come from linguistic fields, aiding trust and CTR.

Best Practices

• Design a field schema: exact (SKU/brand), lemma, phrase, attributes.
• Use locale-specific analyzers and vocabularies.
• Keep synonyms late-bound to avoid index bloat; store safe normals (e.g., hyphen variants).
• Log field contributions for debugging; rebuild after analyzer updates.
• Pair with vector recall but keep lexical as the fast baseline.

Challenges

• Maintenance across languages; OOV brands/models; over-indexing increases size and merges.

Examples

• Title bigram field lifts “air max”; lemma field connects “jackets/jacket”.
• Entity extraction writes GORE-TEX, EU 45, trail into attributes for facets.

Summary

Linguistic indexing layers lemmas, phrases, and entities onto your index so lexical retrieval becomes robust, explainable, and multilingual—perfect for product search.