Agentic RAG: from retrieval to reliable reasoning

Retrieval-augmented generation is often introduced as three boxes: retrieve documents, put them into a prompt, and generate an answer. That is a useful starting point. It becomes less useful when a question needs several sources, the first search fails, or the available documents disagree.

Imagine asking a university assistant: “Can a student from the old curriculum take the new machine-learning course, and which prerequisite exceptions apply?” One document describes the new course. Another defines equivalence between curricula. A third contains a newer exception. A plausible paragraph assembled from only the first document could be confidently wrong.

Agentic RAG adds a decision process around retrieval. The system can identify missing evidence, choose a retrieval action, inspect the result, and decide whether another step is justified. The engineering challenge is making those decisions bounded, observable, and accountable to evidence.

The basic model: knowledge and evidence

A language model assigns probabilities to token sequences. RAG gives that model an external collection of evidence at answer time. In the original RAG formulation, generation combines a parametric model with a retriever over non-parametric memory. Modern applications often implement a simpler retrieve-then-prompt pipeline; these implementations should not be confused with reproducing that paper’s training procedure.

A useful conceptual decomposition is:

Here, $q$ is the question, $d$ is retrieved evidence, and $y$ is the answer. The retriever determines which evidence is available. The generator determines how that evidence is used. This expression is a mental model, not a claim that an ordinary chat application explicitly computes the sum.

A strong generator cannot recover a prerequisite exception that never entered its context. Conversely, retrieving the correct exception does not guarantee that the generator interprets it correctly. Evaluate these as separate failure modes.

What makes retrieval agentic?

A fixed workflow might always retrieve ten passages and rerank five. An agentic workflow makes some choices conditional on the question and intermediate evidence: search the handbook, inspect an effective date, look up a referenced policy, or stop because the question is already answered.

Represent the process as state:

$E_t$ is accumulated evidence, $U_t$ is the set of unresolved requirements, $B_t$ is the remaining budget, and $a_t$ is the next action. The policy $\pi$ may combine deterministic rules with model-selected actions. There is no requirement that every transition be delegated to a model.

flowchart TD
  Q["Question q"] --> Plan["Identify unresolved requirements U"]
  Plan --> Act["Choose action: search, look up, or stop"]
  Act --> Retrieve["Retrieve and rerank evidence"]
  Retrieve --> Update["Add evidence to E, update U, spend budget B"]
  Update --> Done{"U empty or budget spent?"}
  Done -- no --> Act
  Done -- yes --> Answer["Grounded answer with citations, or abstain"]

For the curriculum question, unresolved requirements might be “identify the student’s curriculum,” “find prerequisite equivalence,” and “check whether an exception is still valid.” After each lookup, update those requirements. The result is a task with explicit completion criteria, rather than an endless conversation with a search engine.

Research such as Self-RAG studies learned retrieval and critique mechanisms using reflection tokens. A production loop that asks an ordinary model to critique a passage is a related design idea, not an implementation of Self-RAG’s trained mechanism. Keeping that distinction clear makes evaluation claims more meaningful.

Prepare a corpus that can answer the question

Start with documents, not embeddings. Preserve a stable document identifier, source URL, version, owner, effective dates, section headings, and access rules. The passage “this requirement does not apply” is nearly useless if its heading and scope were discarded during chunking.

Split documents around meaningful sections. A chunk should contain enough context to interpret a claim without becoming an entire handbook. Overlap can protect sentences near boundaries, but excessive overlap creates duplicate evidence and wastes the context budget. There is no universally correct chunk length: test whether the retrieved unit can support your actual questions.

Store an immutable chunk identifier alongside each text span. A citation should resolve to the passage actually used, not merely to a homepage that happens to mention the topic. If a document changes, preserve the relationship between the answer, the old version, and the new version.

Authorization belongs in the retrieval path. Filter candidates using the requesting user’s permissions before disclosing text to a model or returning snippets. An instruction saying “do not reveal private documents” is not a substitute for enforcing access at the data boundary.

Retrieve broadly, then rerank deliberately

Lexical retrieval is good at exact terms: course codes, product identifiers, names, and uncommon phrases. Dense retrieval can help when the question and document use different wording. Hybrid retrieval combines these signals, but their raw scores usually have different scales.

One simple rank-fusion rule is:

Each retriever $r$ contributes according to a document’s rank; $c$ controls how sharply the top ranks dominate. A missing document contributes nothing from that retriever. The advantage is operational simplicity: fusion does not require pretending a cosine score and a lexical score are directly comparable. The constant is a parameter to evaluate, not a universal optimum.

A reranker can then judge a smaller candidate set against the full question. Keep diversity: five nearly identical passages from one policy version may be less useful than three passages covering different requirements. Deduplicate, respect version filters, and inspect whether reranking improves evidence coverage rather than merely making individual passages look relevant.

Work through one question

Suppose the first search returns a new course outline and an old handbook. The outline lists a prerequisite, but it does not establish equivalence for the old curriculum.

1. Decompose the question. Record the course, curriculum version, and specific exception being checked. Ask the user for missing identifying information instead of guessing.
2. Retrieve candidate policies. Search exact course codes and semantic descriptions. Restrict documents to the relevant institution and authorized collection.
3. Check evidence coverage. Mark the prerequisite as supported, equivalence as missing, and exception validity as unresolved.
4. Choose a targeted lookup. Search for the equivalence rule and follow a referenced policy section. Avoid simply repeating the original broad query.
5. Resolve time and scope. Prefer a currently effective rule for the relevant curriculum; do not assume that a newer document automatically supersedes every older rule.
6. Answer with boundaries. Cite the prerequisite and equivalence rule separately. If the exception cannot be established, say which part remains unresolved.

This process can produce a partial answer. “The prerequisite is documented, but the exception is not established by the available sources” is a useful result. It gives the reader a next action without inventing a policy.

Build a bounded controller

The following is architectural pseudocode. retrieve, assess_coverage, and compose stand for components with their own contracts; this is not a complete model integration.

def answer(question, user, retrieve, assess_coverage, compose):
    evidence = {}                 # stable chunk ID -> authorized passage
    attempted_queries = set()
    unresolved = requirements_for(question)

    for step in range(3):
        query = next_query(question, unresolved, evidence)
        if query in attempted_queries:
            break
        attempted_queries.add(query)

        passages = retrieve(query=query, user=user, limit=12)
        for passage in passages:
            evidence[passage.chunk_id] = passage

        unresolved = assess_coverage(question, list(evidence.values()))
        if not unresolved:
            break

    return compose(
        question=question,
        evidence=list(evidence.values()),
        unresolved=unresolved,
        require_citations=True,
    )

The iteration cap alone is insufficient. Bound elapsed time, output tokens, retrieved bytes, and tool calls as well. Propagate a deadline into each dependency so one slow search does not consume the entire request budget. Detect repeated queries and repeated evidence. Log why each action was chosen without storing sensitive raw content unnecessarily.

A useful action-level question is whether the expected improvement is worth its cost:

This is a design objective, not a calibrated confidence score. In practice, a simple rule such as “one missing factual requirement permits one targeted lookup” may be easier to evaluate than a model estimating abstract information gain.

Measure retrieval, answers, and trajectories

Build a small evaluation set with questions, acceptable evidence, required facts, permissions, and expected abstentions. Include questions that should not trigger retrieval, questions with no answer in the corpus, and questions where older and newer documents conflict.

For a question with a known relevant evidence set $G$ and retrieved top-$k$ set $R_k$, evidence recall is:

This only means what your relevance judgments mean. If several passages can independently support the same fact, evaluating coverage of required facts can be more useful than counting every relevant passage. Maintain that annotation policy consistently.

For the answer, check whether each factual claim is supported, whether its citation points to the supporting span, and whether required facts are omitted. A citation that mentions the same topic is not necessarily an entailing citation. For the trajectory, measure tool calls, repeated searches, latency, cost, and unauthorized retrieval attempts.

Compare against a non-agentic baseline using the same corpus and model. If the loop makes easy questions slower without improving difficult questions, route only the difficult subset through it. Average quality alone can hide a damaging increase in tail latency or confident unsupported answers.

Design for the failures you will actually see

Retrieved text is data, including when it contains sentences that resemble instructions. Keep the tool contract and system policy separate from document content. A malicious document should not gain permission to call a new tool, change access filters, or send data elsewhere.

Model-written critiques can also be wrong. If the same model generates and verifies an answer, agreement between the two passes is not independent evidence of correctness. Use source-span checks, deterministic validation for dates and identifiers, and human review for consequential unresolved cases.

An agent can amplify a poor index by repeatedly finding the same irrelevant evidence. It can also conceal missing information behind fluent explanations. Make missing requirements visible in the trace, support abstention, and preserve a direct way for the user to inspect the source.

Practice: add agency only where it helps

Build a corpus of twenty documents with two deliberately conflicting versions. Write fifteen questions, including three with no supported answer. Implement a fixed hybrid-retrieval baseline first. Then add exactly one adaptive action: reformulate a query when a required fact is missing.

Compare fact coverage, unsupported claims, citation correctness, and response time. Inspect the worst five trajectories manually. Your goal is not to maximize the number of steps. It is to learn which decision improves the answer enough to justify another dependency and another failure mode.

A useful Agentic RAG system knows what it has established, what it has not established, and when another search is unlikely to help. That discipline matters more than the number of agents in the diagram.