Building a Grounded AI Pipeline: The Seven Stops to a Working Architecture

I wanted one guarantee: every factual sentence the agent produces should be traceable to evidence it actually retrieved, not something the model made up because it sounded plausible. The requirement seemed straightforward, until I tried to make a model with fewer than 3 billion parameters do both jobs at once: answer the question, and correctly cite the evidence behind it. It couldn’t. That failure is the through-line of everything that follows: seven stops and four major architectural designs shipped and torn out before I found one that a small model could actually sustain.

This is the idea that shaped Mythrix, a symbolic knowledge retrieval system I’m building that connects a structured symbolic model with a corpus of primary reference sources: every answer should be an auditable evidence chain. The constraint throughout was small, local models only, no frontier cloud model to lean on when they got it wrong. The architecture that finally worked was simple to describe: one model generates the answer, a second model verifies it against retrieved evidence. The architecture was simple. Getting it to work was not. Here’s how.

The constraint that shaped everything

I drew the line early: the agent’s LLM is allowed to orchestrate the interaction (hold the conversation, decide which tool to call) but never to retrieve or interpret. It never reads the corpus directly; every factual claim has to come from a tool result. That boundary is what makes grounding checkable at all: if the model could freely narrate over the corpus, there would be nothing to validate a citation against. The rest of this article is about the mechanism that checks the model kept its side of that bargain.

flowchart LR
    U[User question] --> A["agent (model)\nchooses a tool"]
    A -->|"tool call"| T["tools\n(read-only)"]
    T -->|"evidence"| A
    A -->|"final answer"| G["grounding check\n(this article)"]
    G --> R[Reply to user]

Every redesign that follows made the checking side of that mechanism less intelligent and more mechanical. That was exactly the point: a validator shouldn’t be creative.

Stop 1: Let the model cite itself

The first design was the obvious one: tell the model, in the system prompt, to wrap every grounded claim in a citation marker. [G1] meant a graph fact, a structured piece of knowledge from Mythrix’s symbolic model, while [S1] meant a source passage, a verbatim excerpt retrieved from the indexed corpus. The model was also told to number markers sequentially as it wrote the answer, based on the evidence it used.

On the code side, I derived the expected marker range from that turn’s tool results. The first graph result was G1, the second G2, and so on. The same applied to source passages. I then checked that every marker in the model’s reply fell within those ranges.

It mostly worked, until I realized the citations proved nothing. The model didn’t have to have seen the evidence it was citing; it only had to guess a valid number. Closing that hole only addressed the symptom. The actual problem sat underneath the whole idea of self-citation: asking a small local model to compose an answer and correctly cite it inline, in the same pass, is asking it to do two jobs at once. The second job degrades whenever it competes with the first for the model’s limited attention. Guessable ids were the first crack. The next two stops exposed how deep it went.

Stop 2: Closing the guessing hole wasn’t enough

I replaced sequential ids with opaque ids generated by the tools themselves. Guessing became impossible. Citation reliability still got worse.

In my tests, roughly 2 out of 10 requests came back with no citation at all. After I closed off guessing, that rose to roughly 3 in 10. When the model did cite something, it often used a real id but in the wrong format: plain text instead of [Sabc123], or an invented notation of its own.

Closing the guessing hole removed one failure mode, but it also exposed the underlying limitation: the model was still trying to write prose and track opaque ids at the same time.

Stop 3: Validate, then retry in the graph

Both failures looked recoverable rather than fundamental architecture failures, so I added a graph node to catch them: malformed or fabricated markers, and replies that left some available evidence uncited. The node explained exactly what had gone wrong and routed the turn back for a bounded retry.

This is the stop I ultimately discarded, not because retrying is a bad idea in the abstract, but because of what retrying does to a small model under pressure:

• A pushback that just said “cite the id” got read as “give me the id”: the model would drop its prose entirely and reply with a bare comma-separated list, which then failed validation again for having nothing left to attach a citation to.
• A pushback that named the missing ids let the model paste them onto existing prose without actually re-grounding the claim, defeating the entire point of the check.
• Retries share the same step budget as tool calls, so a turn that needed three honest correction rounds could hit the graph’s recursion limit and lose the whole answer.

Every fix bought a small reliability gain and cost a prompt more complexity on a model that was already stretched thin. Slower, and not meaningfully better. The problem wasn’t that the model needed another chance. Retries treated the symptoms. The architecture was still wrong.

Stop 4: Remove citation responsibility from generation, add a fact-checker

The idea that stuck: stop asking the generation model to cite anything at all. Let it write the best answer it can. Then hand that answer, plus the turn’s evidence, to a second model call whose only job is to check it.

For the verification call, I tested three small local models head to head: qwen3:1.7b (with thinking mode disabled), qwen2.5:3b, and phi4-mini. They performed similarly, with qwen3:1.7b performing slightly better.

For this iteration, I intentionally kept phi4-mini for validation despite qwen3:1.7b performing slightly better, because the goal was to test the architecture with independent model roles, not to maximize validation accuracy with a single model.

The first version of this asked the fact-checker to echo the answer back with grounding tags inserted inline, verified by confirming that stripping the tags reproduced the original text byte-for-byte. A representative failure: given a passage it had just read as evidence, and an answer that only quoted part of it, the fact-checker would “complete” the quote by copying in the rest of that same passage: real text, straight from the source in front of it, just not text the original answer had actually written.

I spent a significant part of the project hardening that design and kept finding new ways it failed, regardless of prompt wording. Each new failure shape needed its own prompt adjustment or normalization step, and each change uncovered the next one.

Stepping back, the design itself was the problem: reproduction had a failure mode I couldn’t eliminate on a small local model, even with careful instruction design, because reproducing text you’re also editing is a harder task than it looks, and relying on model output comparison to catch it is a maintenance nightmare, not a validation strategy. I was so close to making it hold, one more normalization step away every time, but close was never going to be enough.

Stop 5: Stop asking for prose, ask for a verdict

Instead of asking the verification model to reproduce the answer with citations, I split the answer into a list of independent sentences, deterministically, in code, before the model ever saw it. That turned one large, fuzzy comparison into many small, bounded ones.

The question stopped being “is this whole answer grounded?” It became “is this one sentence supported by this evidence?”

The fix was structural, not another prompt tweak: stop giving the fact-checker anything to reproduce. Every version through Stop 4 treated grounding as a single long-form task: read the answer, compare it against all the evidence, and return a single verdict. That framing, not any particular prompt wording, was the actual bottleneck: it left the model free to touch, rephrase, or reorder text it was only ever supposed to inspect.

The first version handed the fact-checker that pre-split, numbered list of sentences plus the evidence, and asked it to generate a small JSON verdict per sentence: supported or not, and which evidence id(s) if so. The specific reproduction failures from Stop 4 were no longer possible by construction. The model was told to skip sentences with no factual claim to check, like greetings or transitions.

Making even this simpler task reliable took a few more rounds against real local-model output. Across the candidate models I tested, including phi4-mini from Stop 4, the same failure patterns kept appearing:

• Sentence splitting broke on abbreviations and list markers, "p. 143" read as a sentence boundary. Fixed with a smarter boundary regex.
• Paraphrase got punished as hallucination. Reframed the task around detecting real hallucination, not exact wording.
• “Skip if no claim” became an escape hatch for sentences the model found awkward to classify.
• Some sentences were silently ignored. The model sometimes returned verdicts for only part of the list, leaving gaps in the validation output. I tried enforcing a one-result-per-sentence contract, but the model still occasionally skipped sentences.

Then a bigger setback: no matter how much further I iterated the prompt, none of the candidate models consistently hit the mark on this task.

Stop 6: Complete a document, don’t generate one

Deterministic code creates the verdict document before the model sees it, with every sentence already present. Each sentence gets its own slot, with index and text already filled in, and the fact-checker’s only job is to complete it: fill in supported and citations.

The model receives a document like this:

{
  "results": [
    {
      "index": 0,
      "text": "...",
      "supported": null,
      "citations": []
    },
    ...
  ]
}

paired with an evidence block, keyed by the same normalized ids available for citation:

[Sabc123]
The Magician: mastery of the four elements, willpower...

The model fills in supported and citations on each entry and returns the same shape; index and text are never under the model’s control.

flowchart LR
    A[Generated <br>answer]
    A --> B[Split into<br> sentences]
    B --> C[Build validation<br> JSON]
    C --> D[Fact-check <br>model]
    D --> E[Completed <br>JSON]
    E --> F[Parse <br>verdicts]
    F --> G[Grounding <br>score]

This same pass exposed one more interface mismatch: the model returned citation ids with brackets still attached ([Sabc123] instead of Sabc123), causing otherwise-correct citations to fail matching. Fixed in the parser, not the prompt.

The model no longer generated a validation document. It completed one. This is the design I ultimately settled on: two model calls, one writing freely, one completing a skeleton whose structure is enforced outside the model.

Stop 7: One last issue

Stop 6 held up until real-model testing exposed one last gap. The validator already rejected unknown citations and lowered the score accordingly. The remaining gap was subtler: the model could mutate a valid id into a different-looking id that no longer matched the evidence.

A test run surfaced it: the model received a real citation_id and reproduced it incorrectly: Sef60ed → Sse60ed.

The fix was surprisingly small: drop the leading type letter (S for source passages, G for graph facts). Shorter and purely hex, the id reads less like a word and more like an opaque token, something to copy rather than rewrite.

None of these fixes, across Stops 5 through 7, are retries. None of them reject an answer. Every failure mode (an unreachable model, an unparseable response, a sentence with nothing to classify) falls back to the original, untouched answer with no score footer.

Grounding stopped being a gate that blocked answers. It became a signal appended to the answer.

By the end of this process, qwen3:1.7b was capable of doing both roles in separate calls, generation and fact-checking, and became the model every other candidate was measured against.

A complete pass

Suppose the user asks:

List emotions in the current passages.

The generation model produces a normal answer, with no citations or grounding markers:

The current passages evoke the following emotions:

1. Joy (Sara's declaration: "God hath made a laughter for me")
2. Disbelief (Abraham's reaction to Isaac's birth at old age)

Deterministic code then splits the answer into checkable sentences and prepares the validation document before the fact-checker ever sees it:

{
  "results": [
    {
      "index": 0,
      "text": "The current passages evoke the following emotions:",
      "supported": null,
      "citations": []
    },    
    {
      "index": 1,
      "text": "Joy (Sara's declaration: \"God hath made a laughter for me\")",
      "supported": null,
      "citations": []
    },
    {
      "index": 2,
      "text": "Disbelief (Abraham's reaction to Isaac's birth at old age)",
      "supported": null,
      "citations": []
    }
  ]
}

The validator receives that document together with the retrieved evidence. It is not asked to find evidence; it only decides whether the provided evidence supports each sentence and returns the matching ids:

[83a8f2]
And Sara said: God hath made a laughter for me: whosoever shall hear of it will laugh with me.

[851a93]
Who would believe that Abraham should hear that Sara gave suck to a son, whom she bore to him in his old age.

It then completes only the missing fields:

{
  "results": [
    {
      "index": 0,
      "text": "The current passages evoke the following emotions:",
      "supported": null,
      "citations": []
    },        
    {
      "index": 1,
      "text": "Joy (Sara's declaration: \"God hath made a laughter for me\")",
      "supported": true,
      "citations": ["83a8f2"]
    },
    {
      "index": 2,
      "text": "Disbelief (Abraham's reaction to Isaac's birth at old age)",
      "supported": true,
      "citations": ["851a93"]
    }
  ]
}

Finally, the parser validates every returned id, computes the grounding score, and appends it to the original answer:

Facts checked: 100%

The complete validation exchange is larger, but mechanically identical. Deterministic code prepares the document, the validator fills only supported and citations, and the parser computes the final score. At no point does the validator rewrite, reproduce, or edit the original answer.

Putting it all together

The goal never changed: every factual claim should be traceable to real evidence, checked by something other than the model’s own word for it, using only small, local models. Seven stops to get there:

Stop	Key idea	Outcome
1	Self-citation	Guessable ids
2	Opaque ids	Citation reliability still poor
3	Retry loop	Made small models less reliable
4	Separate fact-checker	Reproducing prose proved brittle
5	Structured verdicts	Models could silently omit sentences
6	Pre-filled document	Shipped
7	Citation id mutation	Closed the last validation gap

Implementation note. For this validation task, disabling qwen3’s thinking mode consistently improved reliability: real-model integration runs went from 3 of 7 passing with thinking on to 7 of 7 with it off. The model wasn’t being asked to solve a hard reasoning problem; it was classifying one sentence against a small evidence set.

Looking back, every successful redesign had the same direction: remove one responsibility from the model and give it to deterministic code. Sentence splitting, document construction, parsing, and scoring all migrated into software until the model was left with a single narrow task: deciding whether one sentence was supported by the evidence. That turned out to be small enough for a 1.7B model to do reliably.

Small models are cost-effective for validation work, once the scaffolding around them is deterministic. What’s left for the model, after Stop 6’s redesign, is a single, narrow classification task, inexpensive to run reliably: no frontier API dependency, no GPU cluster, just a 1.7B-parameter model on a laptop checking sentences against evidence.

Building grounded AI turned out not to be a prompting problem. It was a software architecture problem.

---

Further reading. The full mechanism is documented in docs/architecture/agent-graph.md. The repository also preserves the discarded architectures through its spec-driven development history.

Copyright © 2026 Guido Marelli. This article is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) License. To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0/.