Your reranker didn't improve results. Here's why

Troubleshooting · ~9 min read · Updated 25 Aug 2026

You added a rerank stage, the latency went up, and the answers are the same. That is a common outcome, and it is almost never because the reranker is bad. In nearly every case the model is doing its job on inputs that make its job pointless. Work through these in order — the first two explain most failures.

1. Retrieval never returned the right document

This is the single most common cause. Reranking is a precision stage; it cannot fix recall. If your vector search misses the relevant chunk entirely, the reranker faithfully ranks a set of documents none of which answer the question.

Test it directly, before touching the reranker. Take 20 queries where you know the correct document, retrieve your normal top-k, and just check membership:

hits = 0
for q, gold_id in labelled:
    candidates = retriever.search(q, k=50)
    if gold_id in [c.id for c in candidates]:
        hits += 1

print(f"recall@50 = {hits / len(labelled):.2f}")

If that number is below roughly 0.8, stop tuning the reranker — you are working on the wrong stage. Fix retrieval first: widen k, improve chunking, or add keyword matching. A hybrid retriever that combines BM25 with vectors usually lifts recall more than any reranker choice will, because the two fail on different queries.

2. Your candidate pool is too small

Reranking the top 5 and keeping 5 does nothing — there is no reordering to do that matters. The reranker needs room to work: retrieve wide, rank, then cut.

If you retrieve 50 and keep 5, the reranker has 45 candidates it can promote into the answer. That is where the lift comes from. Our cost calculator shows what widening the pool does to your bill — usually less than people fear, and a step rather than a slope on per-search pricing.

3. Passages are being silently truncated

Every cross-encoder has a maximum input length, and the query plus the passage share it. Exceed it and the tail of your passage is cut off — often the part that actually answers the question. Nothing errors; the score just comes back meaningless.

Classic MiniLM-style rerankers cap at 512 tokens for the pair. Modern hosted models are far roomier — Cohere Rerank 4 and Voyage rerank-2.5 both take 32,000 — so a model swap can fix this outright. If you are self-hosting a 512-token model, check where your passages actually land:

lengths = [len(tok.encode(d)) for d in passages]
over = sum(1 for n in lengths if n > 480)   # leave room for the query
print(f"{over}/{len(lengths)} passages will be truncated")

You can watch this happen in the demo: it warns when a passage exceeds the selected max_length, and you can see the score for a long passage change as you shorten it.

4. Chunks are too big to rank

Even inside the length limit, a large chunk dilutes the signal. A 2,000-word page that mentions your topic once looks, to the model, mostly like text about something else. Its relevance score lands in the middle, below a short passage that is entirely on-topic.

This is the opposite failure from truncation and it has the opposite fix: smaller chunks. 200–400 tokens with a little overlap is a reasonable starting point for reranking. If you need the surrounding context for generation, retrieve and rank the small chunk, then expand to its parent section before putting it in the prompt.

5. Language or domain mismatch

An English-only reranker on Chinese, German or mixed-language content will produce scores, and they will be close to noise. Check that the model you picked actually covers your languages — bge-reranker-v2-m3 and Qwen3-Reranker cover 100+, while several strong English models cover exactly one.

Domain is subtler. Code, legal clauses and clinical notes all break models trained on web prose, because the vocabulary that signals relevance is different. If your content is one of these, test against a domain-appropriate option before concluding that reranking does not help. The code search scenario in the demo shows how differently a general model treats an exact function name versus a paraphrase.

6. Misreading the scores

Two mistakes here, both common:

• Treating scores as probabilities. Most cross-encoders emit an unbounded logit, not a calibrated 0–1 relevance. A score of 3.2 does not mean "76% relevant". Only the ordering is meaningful.
• Comparing scores across models or queries. A score of 0.4 from one model and 0.4 from another mean nothing to each other, and the same model's scores are not comparable between two different queries. If you want an absolute cutoff, calibrate a threshold on your own labelled data — do not import one from a blog post.

If you are filtering with something like score > 0.5 and getting empty result sets, this is why.

7. You cannot see the improvement

Sometimes reranking is working and the measurement is not sensitive enough to show it. Two things to check:

• Are you measuring ranking, or the final answer? If your LLM was already getting the right answer from a mediocre ordering, better ordering will not change the output — it will change your token bill instead. That is a real win, just not the one you were watching.
• Is your eval set big enough? Ten queries cannot distinguish a 5% lift from noise. Thirty to a hundred labelled queries is the usual minimum — see how to evaluate rerankers for NDCG@k and MRR, which detect ordering changes that an exact-match check will not.

Checklist

If every box is ticked and reranking still shows no lift, that is a legitimate finding: your retrieval is already good enough for your queries, and you can drop the stage and keep the latency. That is a better outcome than a reranker that quietly does nothing.

Keep reading