Ron Yellin
Search Engineer, KMS Lighthouse
KMS LIGHTHOUSE · PERSPECTIVE
Why single-shot retrieval quietly fails on real enterprise questions, and what changes when the assistant is allowed to plan, search, judge, and admit what it does not know.
Retrieval-augmented generation fixed the most visible problem with large language models. Rather than asking a model to answer from whatever it absorbed during training, you retrieve relevant material from your own knowledge base and ask it to answer from that instead. Hallucination rates fall, answers start reflecting your actual policies, and the whole thing can be stood up in an afternoon with an embedding model and a vector database. For a couple of years that felt like most of the problem solved, and a lot of enterprise deployments stopped there.
The trouble is that the standard pipeline is a chain of single-shot decisions. Embed the question, run one similarity search, take the top few chunks, hand them to the model, print whatever comes back. Each of those steps happens exactly once, with no mechanism for noticing that it went badly, and in the aggregate that produces a system with no opinion whatsoever about the quality of its own retrieval.
Agentic RAG is the fix, and the name is more intimidating than the idea. It means the assistant is allowed to decide what to do next — run another search from a different angle, throw out a result that only looked relevant, ask a clarifying question, or admit it could not find the answer — instead of following the same fixed script every time. That is the whole concept. Everything below is what falls out of it.
It helps to look at a real question rather than a benchmark one. A contact center agent types: "Customer moved states mid-policy and wants to add a driver, what happens to the premium?" That is three questions wearing a trenchcoat. It spans mid-term endorsement rules, interstate address changes, and premium recalculation, and depending on the states involved it may pull in a jurisdiction-specific compliance requirement as well. A single embedding of that sentence produces a single point in vector space, and the nearest neighbours to that point will skew toward whichever sub-topic dominated the phrasing. The retriever comes back with five reasonable-looking passages about adding drivers, nothing at all about interstate moves, and the model, which is nothing if not accommodating, writes a fluent and completely confident paragraph on the basis of two thirds of the evidence. Nobody in the pipeline was tasked with noticing the gap, so nobody noticed it.
One search and done, versus a loop that judges its own evidence and searches again when the answer is not there yet.
What changes when retrieval gets agency
The shift that agentic RAG makes is to stop treating retrieval as a fixed pipeline and start treating it as a loop the model itself controls. The model decides how many searches to run and from which angles, judges whether what came back is actually sufficient, runs again if it is not, and retains the authority to decline to answer. That sounds like a small architectural change and it is not, because it moves the quality judgement inside the system rather than leaving it with the person reading the output.
Query planning comes first. Before anything is retrieved, the question is decomposed into its constituent parts, so the example above becomes separate searches for mid-term driver addition, interstate address change, and premium recalculation rules. Alongside decomposition sits reformulation, generating alternative phrasings of each sub-question, and this matters more than it sounds like it should. Your knowledge base was written by policy authors and compliance teams; your questions arrive in the vocabulary of agents and customers. That vocabulary mismatch is one of the largest and quietest causes of retrieval failure in production systems, and generating several phrasings of the same intent is the cheapest available mitigation.
Those queries then run in parallel rather than in sequence, and typically across more than one retrieval strategy. Dense vector search handles semantic similarity, keyword search catches the exact product codes and policy names that embeddings are notoriously bad at, and metadata filters constrain by jurisdiction, effective date, or document status. Hybrid retrieval reliably beats either approach used alone, and running the angles concurrently means the coverage improves without the latency cost that a sequential chain of the same searches would incur.
Then comes the step that first-generation systems almost universally skip, and the one that does the most work. Before anything is generated, each retrieved passage is graded against the original question: does this answer it, partly answer it, or merely look like it might because it shares vocabulary? Material that fails is discarded rather than swept into the context window on the theory that more context cannot hurt. It can. Irrelevant passages measurably degrade answer quality, and models remain unreliable at surfacing information buried in the middle of a long context, so a smaller curated set consistently outperforms a larger padded one. Grading is also what makes version conflicts tractable, because when a legacy procedure and its current replacement both surface, something has to prefer the current one on grounds other than cosine similarity.
If the graded evidence comes back thin, the loop runs again with reformulated queries. If it is still thin after that, the system does the thing that actually separates a trustworthy assistant from a merely plausible one, which is to say so. It asks a clarifying question, or it states plainly what it could not find, rather than assembling a well-written answer with nothing underneath it. This is worth being blunt about, because it tends to get discounted in evaluations that only score accuracy: an assistant that is right eighty-five percent of the time and candid about the remainder is operationally more valuable than one that is right ninety-two percent of the time and confident always. The first can be acted on without independent verification. The second cannot, which means every answer it gives has to be checked, which means it has saved nobody any time.
Why the enterprise case is different from the demo case
Agentic retrieval demos nicely, but that is not the reason it matters. It matters because enterprise knowledge has structural properties that naive retrieval handles badly, and those properties do not show up in a proof of concept built on a clean document set.
Enterprise knowledge bases are large, heavily overlapping, and versioned, so the same topic exists as a legacy procedure, a current procedure, and two or three regional variants, all of which are semantically near-identical to the question being asked. Enterprise answers also carry consequences that a consumer chatbot does not: a wrong answer about a benefits entitlement, a claims decision, or a regulated disclosure is not a poor user experience, it is exposure. That is why citation stops being a nice touch and becomes structural. When an answer points back to the specific source passage it came from, the agent receiving it can verify it in seconds, the knowledge team governing it can audit it, and the organisation can defend it if it is ever challenged. Grounding without attribution asks people to trust the system; grounding with attribution lets them check it, and at enterprise scale only the second one holds up.
Access is not uniform either. Two employees asking an identical question should sometimes get different answers, because they are entitled to see different content, and that constraint has to be enforced at the retrieval layer by filtering the candidate set before generation ever happens. Any architecture that retrieves broadly and then filters the output has already exposed the content it was supposed to protect, whatever the final response looks like. And enterprise conversations are rarely single-turn. A follow-up such as "and what if they are a new customer?" is a modification of the previous question, not a fresh one, so a system that holds on to its curated evidence set can answer immediately and coherently, while one that treats every turn as an independent lookup pays the full retrieval cost again and risks contradicting itself in the process.
The trade-offs worth being honest about
None of this is free, and it is better to price it openly than to discover it in month three. Running several searches, grading the results, and sometimes retrieving again means more model calls and a meaningfully higher token spend per question than a single-shot pipeline. Parallelism keeps the wall-clock latency respectable, but the cost line is real. Debugging is harder too, since a deterministic pipeline fails in predictable places while a loop that writes its own queries can fail in ways that require tracing the reasoning path, which raises the bar on observability and on evaluation tooling considerably.
The most important caveat is that better retrieval does not repair bad content. If the underlying knowledge is stale, contradictory, or simply absent, an agentic system will find that out faster and more visibly than a basic one, which is genuinely useful but is not the same as solving it. Organisations that invest in the assistant while leaving the knowledge behind it untended tend to learn this expensively. It is also fair to say that the full loop is not always warranted; for simple high-volume factual lookups, classic retrieval is cheaper and entirely adequate, and several mature deployments route on question complexity, sending the easy traffic down a fast path and reserving the deliberative loop for the questions that actually need it.
How KMS Lighthouse implements the pattern
The KMS Lighthouse intelligent knowledge assistant is built on exactly this model. It plans its search by breaking a complex question into several focused queries covering different wordings and angles, runs those searches in parallel across the approved knowledge base rather than settling for the first phrasing it was handed, and then weighs what came back, keeping only the material that genuinely answers the question and setting the rest aside. It answers in plain language with citations pointing to the exact source passages, applies the user's permissions on every single query so that security travels with the request rather than being bolted on afterwards, and asks a clarifying question instead of guessing when something is genuinely unknowable.
What that produces day to day is broader coverage than any single query can reach, follow-up questions answered instantly because the assistant reuses evidence it has already curated, consistent answers across every agent and every channel because everyone is drawing on the same governed knowledge, and a verifiable source sitting behind every claim. For the teams running large and fast-changing knowledge bases, where the cost of a confidently wrong answer is measured in escalations, repeat contacts, and occasionally in regulatory attention, that combination of thoroughness and accountability is the whole point.
