Retrieval and grounding

The first question is what the corpus actually is and who maintains it. Retrieval over an unmaintained shared drive returns the contents of an unmaintained shared drive, now phrased fluently and delivered with no hedging, which is strictly worse than the drive itself because the fluency reads as authority.

Superseded material is the specific failure mode. A policy replaced in March and a copy of the version it replaced sitting in a subfolder are equally retrievable, and nothing in the model can prefer the current one unless the index carries that distinction explicitly. Recency of file modification is not that distinction, and it is what most systems substitute for it.

Answers carry citations, and a citation is a link to the passage, not a filename. Without one there is no way to check an answer, and an unverifiable answer from a system that never sounds uncertain is more dangerous than no answer at all — it will be believed at exactly the rate it is wrong.

Permissions are applied at query time against the person asking, not baked into the index once when it is built. A system able to quote a document to somebody who could not have opened it has created a data leak with a conversational interface, and it will do so without logging anything that looks like an incident.

Freshness is engineered, not assumed. When the underlying material changes the index changes with it, because index staleness is invisible from the outside: the system keeps answering confidently from content that was withdrawn a month ago, and nothing about the response indicates it.

Chunking and document structure carry more of the answer quality than the choice of model does. A table split across two chunks answers wrongly rather than not at all, headings that establish which product a section refers to are frequently discarded during ingestion, and a great deal of apparent model failure is structure that was destroyed before the model saw it.

The system is built to say when it does not know. A retrieval pipeline that always returns something will return the nearest available passage regardless of whether it bears on the question, and that failure is indistinguishable from success at a glance, which is what makes it expensive.

Where the corpus cannot be made trustworthy inside the engagement, the recommendation is to fix the corpus first, and we say so plainly. A documentation problem wearing a technology costume is still a documentation problem, and it gets more expensive once a system is answering from it.