Hybrid RAG: BM25 + Vector + Rerank

## Full reference: hybrid retrieval stack

### 1. Why hybrid

Pure vector search fails on three query classes:

- **Exact tokens** — SKUs, error codes, product names, typos. Dense embeddings smear these into neighbourhood.
- **Negation and numbers** — "not 2FA", "v18", "80%". Embeddings ignore these routinely.
- **Rare terms** — proper nouns appearing <5 times in the corpus. No semantic gradient to lean on.

Pure BM25 fails on paraphrase and conceptual questions. Combining both with RRF fixes both failure modes at near-zero code cost. Cross-encoder reranking then pushes precision on the merged shortlist.

This is the stack Microsoft's "Azure AI Search" docs, Pinecone's hybrid-search guide, Weaviate's hybrid module, and Anthropic's 2024 contextual-retrieval post all converge on.

### 2. Pipeline

```
query ──► BM25 top-50 ──┐
                        ├──► RRF merge top-20 ──► cross-encoder rerank ──► top-5 ──► LLM
query ──► Dense top-50 ──┘
```

Knobs (with sane defaults):

| Knob | Default | Notes |
|---|---|---|
| Sparse candidates | 50 | Raise to 100 for technical corpora (code, errors). |
| Dense candidates | 50 | Match sparse. |
| RRF k | 60 | The original paper's value. Rarely move. |
| Merged pool | 20 | Input to reranker; 20–40 is the sweet spot. |
| Final top-k | 5 | 3–8 depending on generator token budget. |

### 3. BM25 index

Preprocessing matters more than the scoring formula:

- Lowercase, Unicode-normalise.
- Tokenise on whitespace + punctuation — but keep identifiers (`foo_bar`, `Foo.Bar`) intact for code corpora.
- Remove stopwords only for prose; keep them for code.
- Stem optionally (Porter or Snowball) for English prose.
- Store per-doc length; BM25 needs it.

Libraries: `rank_bm25` (Python, in-memory, fine to 1M docs), `Elasticsearch`/`OpenSearch` (scales), `tantivy` (Rust, fast).

### 4. Dense index

- **Embedding model**: `BAAI/bge-large-en-v1.5` is the 2025 default open-source model. `text-embedding-3-large` (OpenAI) or `voyage-3` (Voyage) if you use closed APIs. All three are cosine-similarity-normalised.
- **Store**: Qdrant, pgvector, Weaviate, Pinecone. Use HNSW with `M=32, ef_construction=200` as a reasonable default for <10M vectors.
- **Chunking**: 300–500 tokens per chunk with 20% overlap. Split on semantic boundaries (Markdown headings, paragraphs) not raw char count.

### 5. Reciprocal Rank Fusion (RRF)

```python
def rrf(rankings: list[list[str]], k: int = 60) -> dict[str, float]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return scores
```

RRF is score-scale-free — BM25 scores (0–50) and cosine similarities (0–1) don't need normalisation. This is why it beats weighted linear combinations in practice.

### 6. Cross-encoder rerank

Bi-encoder retrievers (dense) embed query and doc independently. Cross-encoders concatenate `[CLS] query [SEP] doc` and run a single forward pass per pair — far more accurate, much slower. Only use them on the merged shortlist (~20 pairs).

Options:

- **BGE-reranker-large**: open source, ~560M params, ~80ms/pair on GPU, free.
- **Cohere Rerank v3**: hosted API, ~40ms/pair p95, $1/1k searches.
- **Voyage rerank-2**: hosted, competitive with Cohere.
- **ColBERT / ColBERTv2**: late-interaction, faster than cross-encoder at scale but indexing is heavier.

Budget: a 20-pair rerank on BGE-large-fp16 is ~1.6s on CPU, ~160ms on a T4 GPU. Batch the pairs.

### 7. Citations contract

Every retrieved doc must carry:

```json
{
  "id": "doc_42#chunk_3",
  "text": "…",
  "source": "https://docs.example.com/auth/oauth",
  "rerank_score": 0.91,
  "bm25_rank": 4,
  "dense_rank": 2
}
```

The generator renders citations as footnote-style superscripts. Never let the generator invent URLs — citations come only from `retrieved_docs[].source`.

### 8. Evaluation

You need a golden set of (query, ideal_doc_ids) pairs. Track:

- **Recall@50** of the merged pool — did the right doc make it to rerank?
- **NDCG@10** after rerank — is rank quality good?
- **Faithfulness** (LLM-as-judge) — does the generated answer actually use the retrieved docs?
- **Latency p95** at each stage.

Budget 50–200 labelled queries minimum. Reuse the `golden-eval-harness` pack to wire this into CI.

### 9. Common pitfalls

1. **Tokenising code like prose** → camelCase and snake_case split into pieces BM25 can't match. Use a code-aware tokeniser.
2. **Normalising RRF scores to top=1** → breaks the score-scale-free property. Leave them.
3. **Reranking the full corpus** → cost explodes. Rerank only the merged shortlist.
4. **Chunking too small** (<150 tokens) → dense retrieval loses context; reranker starves.
5. **Returning raw chunk text to the LLM** → no citation possible. Always carry `source` through.
6. **Reranker on an old query/passage pair format** → BGE expects `[query, passage]`; Cohere expects `documents[]`. Read the model card.

### 10. Cost envelope (per query, indicative)

| Stage | Cost | Latency |
|---|---|---|
| BM25 (50 hits, in-memory) | ~$0 | 2–10 ms |
| Dense (50 hits, Qdrant) | ~$0 | 15–40 ms |
| RRF merge | ~$0 | <1 ms |
| Rerank 20 pairs (Cohere) | ~$0.0010 | 40–80 ms |
| **Total** | **~$0.001** | **~100 ms** |

### 11. When to stop

If your eval shows Recall@50 >0.98 *and* post-rerank NDCG@10 plateaus, you are retrieval-bound no more — spend the next cycle on the generator (prompting, grounding checks) or on chunking quality, not on more fusion.