Multilingual RAG: Techniques and Best Practices for Retrieval Across Languages

Most RAG systems are built and tested in a single language, then quietly break the moment a user asks a question in a language different from the documents — or the documents themselves are a mix of English, Spanish, Portuguese, and whatever else an organization has accumulated over the years. For enterprises and government platforms, this isn't an edge case. It's the default.

Multilingual RAG adds a layer of complexity on top of standard RAG: the query language, the document language, and the answer language can all be different, and each stage of the pipeline needs to account for that.

Why this breaks naive RAG

A typical RAG setup embeds documents and queries with the same model, then does a nearest-neighbor search. That works fine when everything is in one language. But:

• Monolingual embedding models cluster text by language before they cluster it by meaning, so a Spanish query barely retrieves relevant English documents even if the content matches.
• Chunking strategies tuned for English (fixed token counts) behave differently on languages with different tokenization density — Chinese or Japanese pack far more meaning per token than English or Spanish.
• Sparse retrieval (BM25) depends on tokenizers and stemmers that are language-specific, so a single sparse index tuned for one language underperforms badly on others.

Common techniques

1. Cross-lingual embedding models. Models like multilingual E5, LaBSE, Cohere's multilingual embeddings, or OpenAI's text-embedding-3 family are trained so that semantically equivalent sentences in different languages land close together in vector space. This is the foundation: it lets you embed a Spanish query and retrieve English documents without translating anything first.

2. Retrieve-then-translate, not translate-then-retrieve. It's tempting to machine-translate everything into one pivot language before indexing. In practice this adds latency, cost, and translation-induced errors at index time — for every document, regardless of whether it's ever queried. It's usually better to keep documents in their original language, retrieve cross-lingually with a multilingual embedding model, and only translate the final passages you're about to show or feed to the LLM.

3. Hybrid search with per-language sparse indexes. Dense retrieval alone can miss exact terms — product names, legal references, error codes. Keeping a BM25 or SPLADE index per detected language, and merging its results with dense retrieval, recovers keyword-level precision without losing cross-lingual recall.

4. Language-aware chunking. Chunk size should be set relative to how information-dense a language is, not as a fixed token count applied everywhere. CJK languages typically need smaller chunk sizes (in tokens) than Latin-script languages to hold a comparable amount of content.

5. Multilingual reranking. After retrieval, a cross-encoder reranker trained on multilingual pairs (e.g., BGE-reranker-v2-m3, Cohere rerank-multilingual) reorders candidates by relevance regardless of language. This step tends to give the largest quality jump for the least engineering effort.

6. Metadata-based language filtering and boosting. Tagging every chunk with its detected language lets you filter, boost, or exclude by language when it matters — for example, preferring the original-language version of a legal document over a translated copy, even if both are in the index.

Best practices

• Detect language early, and keep it as metadata everywhere — on the document, the chunk, and the query. It's the cheapest signal you have, and most of the pipeline decisions above depend on it.
• Don't force full-corpus translation. Translate only what's about to be shown to a user or passed to the LLM, and cache those translations — not the whole index.
• Evaluate per language, not just in aggregate. A system that looks strong overall can be quietly failing for one language pair. Benchmarks like MIRACL and MKQA are built exactly for this and are worth running as a regression test.
• Normalize text consistently — Unicode normalization (NFC), consistent handling of diacritics and accents, and locale-aware casing avoid subtle retrieval misses that are hard to debug later.
• Preserve original-language citations. For legal, government, or compliance use cases, showing a translated quote instead of the source text can undermine trust in the answer — keep the original text available alongside any translation.
• Watch tokenizer mismatches between embedding and reranking models. Mixing models trained on different tokenizers can silently degrade multilingual performance in ways that are easy to miss in English-only testing.

Putting it together

At indexing time, documents get tagged by language, chunked accordingly, embedded with a multilingual model, and stored with their language as metadata:

At query time, the query is embedded as-is — never translated first — and matched cross-lingually against the same vector store, combined with per-language hybrid search, reranked, and only then handed to the LLM to answer in the language the user asked in:

None of these techniques are exotic on their own. The difference between a multilingual RAG system that works and one that quietly underperforms for half its users usually comes down to whether language was treated as a first-class signal throughout the pipeline, or bolted on as an afterthought.

If you're running RAG in production today: do you actually know how it performs for every language your users query in, or only for the one you tested in?