Embedding Batch Size: How One Oversized Request Silently Disabled Our RAG Retrieval

When a citation-verification job returned "no supporting passage" for a claim we knew was on page 14 of the source PDF, the obvious suspect was the LLM judge. The actual culprit was three layers deeper, in the embeddings pipeline, and it had been quietly degrading retrieval on every large document for weeks. The lesson — embedding batch size matters more than the standard cookbook example leads you to believe — landed during a retrieval-quality sprint last week on Acurio, our citation-verification product. The bug was small. The blast radius wasn't. This is the writeup.

The Symptom: A 19-Point Verdict Gap on Big Sources

We ship a hybrid retrieval pipeline — semantic embeddings on top, BM25 underneath, a reranker in the middle — to power AI citation verification. On most documents, the three layers agree and the judge sees the strongest passage for any cited claim. On a specific subset of documents, agreement collapsed. The judge would deliver "unknown" or "unsupported" verdicts on citations a human reviewer could find in fifteen seconds.

The pattern was the giveaway. Failures clustered on documents above roughly 600 chunks — long guidelines, multi-chapter PDFs, thesis-length sources. Anything that fit comfortably in a single small embeddings call worked. Anything that didn't, didn't.

We ran a recall-only harness across the ten hardest production citations (no LLM judge, just "did the right chunk make the top-K?") and the gap was stark. On the affected documents, the right passage was nowhere in the candidate pool. Retrieval wasn't ranking it badly — it had no semantic representation of those chunks at all. The verdict-accuracy gap measured from production logs across the affected sources came in at about 19 percentage points lower than the rest of the corpus.

That is a silent, partial degradation: the product still ran, the UI still returned answers, and no error ever surfaced to the user. The worst class of bug in any retrieval system.

The Cause: One Oversized Request, One All-or-Nothing Guard

Our embedTexts function passed an entire document's worth of chunks to an OpenAI-compatible embeddings endpoint in a single request, with dimensions pinned so the database column width couldn't be silently violated by a provider default change.

That worked fine until a document arrived with 752 chunks. The endpoint, faced with an input array that large, returned an HTTP 200 with data: []. No error, no rate-limit signal, no "too many inputs" message — just an empty list where we expected 752 vectors.

The next layer made the problem worse. The caller, reasonably, refuses to insert a partial embeddings batch — if the vector array length doesn't match the chunk array length, something has gone wrong and you don't want to commit a half-written document to pgvector where downstream queries will misalign chunk indices to embeddings. The guard fired, the entire document's embeddings were dropped, and the chunks landed in the database with empty embedding columns.

Now the silent part. Our retrieval path is hybrid: if a chunk has a vector, semantic similarity contributes; if not, BM25 carries the chunk on lexical overlap alone. Big documents kept appearing in search results — just with the semantic half of the engine switched off for them. BM25 is good at "find me a chunk that uses these words." It is not good at "find me a chunk that supports this claim." Verdict quality cratered without a single error in the logs.

Six production documents were sitting at zero embeddings by the time we caught it.

The Fix: Batch, Stay Length-Preserving, Backfill

The repair is a 60-line change in the embeddings module, and the shape generalises to any pipeline calling a remote embeddings API.

Batch the request. A new EMBEDDING_BATCH_SIZE (default 96) caps the request size below any provider's per-call ceiling. The 752-chunk document now goes out as eight requests of 96 instead of one of 752. Cost: one extra round-trip per batch; on documents that fit in a single call, no change.

Make the function length-preserving. This is the part the textbook examples skip. The internal result array is pre-allocated to input.length and filled by absolute position. A batch that fails leaves its slots as empty arrays — it does not shift the rest of the response left, and it does not abort the whole call. The caller's "vector count must equal chunk count" guard still holds, because the lengths still match, and the insert path skips the empty rows. One bad batch loses 96 embeddings instead of 752, and the other 656 chunks land correctly.

Distinguish hard failure from partial success. The function returns null only when no batch at all succeeded. Any successful batch is enough to flip the result from "fall back to BM25 only" to "use what we have." The all-or-nothing semantics of the old code — one failure equalled total failure — are gone.

Write a durable backfill. A backfill-source-embeddings script scans the database for chunks with zero-length embeddings, re-embeds them, and writes the vectors in place. Dry-run is the default; --write commits. The same script repaired the six damaged production documents the morning the fix landed.

The retrieval recall test that had shown zero hits on affected documents now reported top-1 / top-3 / top-5 numbers in line with the rest of the corpus. The verdict-accuracy gap closed.

What We Measured and Dropped

A retrieval sprint is also a chance to test the techniques you have been meaning to try. Most of them did not help on our data, and the discipline of saying so out loud matters more than shipping every paper you have read.

We measured reciprocal-rank fusion between the semantic and BM25 candidate pools — no significant lift over the existing weighted blend. We measured pool-widening (top-50 to top-100 before reranking) — no measurable accuracy gain, just longer rerank latency. We tested a parent-child embed-small strategy — the smaller embedding model lost ground on the hardest semantic-only citations and broke even at best. All three are now noted as "tried, dropped" in the engineering log, so the next person who reaches for them sees the prior result.

One technique did land, behind a flag: neighbor expansion. On a hit, splice the ±1 adjacent chunks into the judge's context. On the ten hardest production citations, this lifted verdict agreement from 5/9 to 7/9 with no majority regressions. We shipped it off by default — the validation sample is biased toward false negatives, so we cannot yet measure the opposite risk (extra context credentialing a partially supported claim as fully supported). When we have a balanced eval set, the flag flips.

The same instinct — measure the quiet failure modes before scaling the loud features — runs through how we built the AI citation verification workflow itself and through the agent approval gates we wrote about last month. Production retrieval lives or dies on the edges nobody wrote a test for.

What This Means for Anyone Running a RAG Pipeline

Three takeaways, in the order they will save you the most time.

Cap your embedding batch size below any provider ceiling, and treat a length mismatch as a fail-this-batch event, not a fail-this-document event. Providers tend to fail oversized embeddings requests as empty arrays rather than as 4xx errors, and most reference implementations propagate that emptiness all the way to the database. Pre-allocate the result, fill by index, and let one bad batch be one bad batch.

Add a "did we silently fall back?" detector to your retrieval logs. Our real oversight was not the bug itself — it was the absence of a metric for "documents indexed with zero semantic vectors." Any chunk landing in the store with an empty embedding column should be a counter you can graph. The same shape — silent partial degradation under a try/catch — showed up earlier this month in our FastAPI rate-limit-header incident, and the lesson is the same: a library that swaps behavior based on the shape of its inputs will eventually find a shape you did not test.

Keep a backfill script next to every "this column can be empty" pipeline. When the bug ships — and they all ship occasionally — the difference between "we re-ran one script and the production data healed" and "we wrote a recovery tool under pressure" is whether you put that script in the repo on day one. Default to dry-run, gate the writes, ship it next to the function it backfills.

If you are running retrieval in production and want a second pair of eyes on the batching boundary, the length-preserving guard, and the backfill story before you ship, book a free AI Potenzial-Check — or read how we think about AI agents for Swiss SMEs for the broader architecture context.