The retrieval questions nobody asks until it is too late

Retrieval projects have a characteristic shape. Weeks go into choosing a vector database, benchmarking embedding models, and arguing about similarity metrics. Then the system ships and the complaints are that it cannot find things people know are in there, that it answers confidently from a policy that was superseded in March, and that legal has a question about a deletion request.

None of those complaints are about the vector database. They come from four decisions that were made quickly, early, and usually by whoever was writing the ingestion script: how documents get split, how queries match, how the index learns that a source changed, and whether anything recorded where each vector came from.

This is the list we would work through before touching the infrastructure question.

Chunking shapes retrieval quality more than model choice does, and in the systems we have looked at it is a recurring root cause of disappointing results.

The naive approach splits on a character count. It is fast, it is what every tutorial does, and it cuts through the middle of things that only mean anything whole. Split a contract mid-clause and you can retrieve an obligation whose negation lives in the next chunk. Split a table and you get numbers with no column headers, which is worse than retrieving nothing because it looks like an answer.

The questions worth answering deliberately: what is the smallest unit of this corpus that still means something on its own, does a chunk carry enough surrounding context to be interpreted correctly, and does it carry the metadata that tells you what it is, which version it belongs to, and who may see it.

The last one matters more than it looks. Metadata attached at chunk time is nearly free. Metadata reconstructed later is an archaeology project, and every control downstream, entitlement and lineage included, depends on it existing.

Semantic search finds things that mean something similar. That is the point of it and it is genuinely powerful for questions phrased in natural language.

It is also unreliable for the queries people actually type into internal systems, which are frequently identifiers: an account number, an error code, a product SKU, a case reference, a surname. Vector similarity has no special respect for an exact string. A query for invoice 4471 can happily return invoice 4417, which is textually adjacent, semantically near-identical, and completely wrong.

Hybrid retrieval, running keyword and semantic search together and merging the results, is the standard answer and it is not exotic. The reason to decide it early is that discovering the need late usually means reindexing.

The diagnostic: look at a sample of real queries before launch, not the ones you imagined. If more than a small fraction contain identifiers, semantic-only retrieval will feel broken in a way users cannot articulate. They will report that it is bad rather than that it cannot match strings, and you will spend a quarter chasing the wrong problem.

Most retrieval systems are built with a one-time ingestion and a plan to sort out refreshing later. Later arrives as a superseded policy being quoted back to a customer.

There are three distinct events and they need different handling. A document is updated, so the old chunks are wrong and must be replaced rather than supplemented, or the index will hold both versions and retrieve whichever is more similar. A document is deleted, which is the hard case below. And a document's permissions change, which usually breaks first because it is the event nobody instrumented at all.

The uncomfortable property of staleness is that it is invisible from inside the system. A stale index returns confident, well-formed, correctly cited answers from superseded material. There is no error, no empty result, nothing that looks like a failure. It reads exactly like working.

So the question is not whether you refresh. It is what you can prove about how far behind you are. A system that can state its lag is manageable; one that cannot has to be trusted, and trust is not a control.

This is the one that becomes urgent without warning, and it is the least recoverable.

When a document is deleted, whether from a retention schedule or a person exercising a right to erasure, the deletion has to reach everything derived from it. The chunks. The embeddings. Any cached answer that quoted it. Anything fine-tuned on it. Possibly logs, if the retrieved content was logged, which it usually is.

Doing that requires a mapping from source document to every derived artifact, maintained at ingest. It cannot be built afterwards. Nobody can look at a vector and determine which document produced it. If that mapping was never written down, the honest answer to can you prove the deletion was complete is no, and the honest remediation is a full rebuild.

Recording lineage at ingest costs almost nothing: an identifier on each chunk and a table. The asymmetry between that cost and the cost of not having it is as large as any in this field, and it is invisible until the day someone with standing asks.

Once retrieval respects entitlements, a new case appears that most systems handle badly. Someone asks a question whose answer lives entirely in material they cannot see.

The honest response is that nothing is available to them, and ideally who to ask. The tempting response, and the default in most implementations, is to answer from whatever partial material survived the filter. That produces a confident, well-cited, materially incomplete answer, with nothing to indicate that the substantive material was withheld.

That failure is harder to catch than a refusal, because nothing about it looks wrong. It is the same shape as a capped output that reads as complete: the system's silence about what it did not include is the defect.

Design the empty case deliberately. It is a small amount of work and it is the difference between a system that respects a boundary and one that quietly routes around it.

If you are starting, decide lineage and chunk metadata first, because both are nearly free at ingest and effectively unrecoverable later. Then decide hybrid retrieval, because changing your mind means reindexing. Then the refresh model, because staleness is invisible and will be believed. Then the empty case.

The vector database can be chosen last, and it will matter less than any of the above.

None of this is exotic and none of it requires research. It requires deciding four things on purpose that are usually decided by default, in a script, by whoever had the afternoon.