Bedrock Guardrails Contextual Grounding: How We Wired It Into an Auto Insurance RAG Agent

We built a customer support agent for an auto insurance carrier. Policyholders ask it things like "is my windshield covered?", "what's my rental reimbursement limit?", "do I have roadside assistance?" It answers by retrieving the relevant sections of that policyholder's declarations page and coverage documents, then summarizing them with Claude on Bedrock.

It worked well in testing. Then during a review pass we found a transcript where a customer asked about glass coverage and the agent answered, confidently and in a friendly tone, that they had full glass coverage with no deductible. They didn't. Their policy had glass coverage subject to the comprehensive deductible. The retrieved chunk said so plainly. The model just smoothed it over.

In most apps that's an annoying bug. In insurance it's an errors-and-omissions problem. A wrong coverage answer that a customer relies on is the kind of thing that ends up in front of a regulator or a lawyer. So "the model is mostly right" wasn't good enough. We needed the system to know when it was off-script and refuse rather than guess.

That's what Amazon Bedrock Guardrails contextual grounding check does. This post is how we implemented it.

The code below is a simplified pattern, not client code. Numbers are representative of what we saw, not exact figures from the engagement.

What contextual grounding actually checks

It's not a content filter and it's not a denied-topics list. It's a second model that scores the response against two things you give it:

1. The grounding source. The reference text the answer is supposed to be based on. In a RAG app, that's your retrieved chunks.
2. The user query. The question being answered.

It produces two independent scores in [0, 1]:

• Grounding score. How well the response is supported by the source. Low score means the model said something that isn't in the source, or contradicts it. Our glass-coverage answer would tank here.
• Relevance score. How well the response actually addresses the query. Low score means technically grounded, but off-topic. The model answers "your collision deductible is $500" when the question was about rental reimbursement.

You set a threshold per dimension. If a response scores below either one, the guardrail intervenes. It blocks the response and hands back your configured blocked-message, exactly like the other guardrail policies. Your app then decides what to do with that: return a safe fallback, escalate to a human, re-retrieve, whatever.

The key mental model: grounding is not correctness. It checks whether the answer is faithful to the text you gave it. If your retrieval pulls the wrong chunk, a high grounding score just means the model faithfully repeated the wrong thing. More on that later. It bit us.

Wiring it in: the qualifiers are the whole trick

Here's the part that trips people up. The contextual grounding check needs to know which part of the payload is the grounding source and which part is the query. You tell it with content qualifiers.

If you're using Bedrock Knowledge Bases end-to-end, RetrieveAndGenerate can attach a guardrail and wire the qualifiers for you. We weren't. We had our own retrieval because policy documents live in the carrier's system, not in a Bedrock KB. So we ran our own generation and then called the standalone ApplyGuardrail API on the output. That's the more flexible path and the one worth understanding, because it works regardless of how you retrieve or generate.

The shape of it:

import boto3

bedrock = boto3.client("bedrock-runtime")

def check_grounding(query: str, retrieved_context: str, model_answer: str):
    return bedrock.apply_guardrail(
        guardrailIdentifier=GUARDRAIL_ID,
        guardrailVersion=GUARDRAIL_VERSION,
        source="OUTPUT",  # we're evaluating the model's answer
        content=[
            {"text": {"text": retrieved_context,
                      "qualifiers": ["grounding_source"]}},
            {"text": {"text": query,
                      "qualifiers": ["query"]}},
            {"text": {"text": model_answer}},  # the thing being scored
        ],
    )

Three content blocks: the source (tagged grounding_source), the question (tagged query), and the answer (untagged, that's what gets evaluated). Miss the qualifiers and the check has nothing to ground against. It'll either no-op or score nonsense. This is the single most common implementation mistake we see.

The response tells you whether the guardrail intervened and gives you the per-dimension scores in the assessment, so you can log them and tune later. We log every score on every request from day one. You can't tune thresholds you didn't measure.

Where it sits in the pipeline

user question
   |
   v
retrieve policy chunks --> retrieved_context
   |
   v
generate answer with Claude --> draft_answer
   |
   v
apply_guardrail(OUTPUT, grounding_source=retrieved_context, query=question, draft_answer)
   |
   +-- passed --------------> return draft_answer
   +-- intervened ----------> return safe fallback / escalate to human / re-retrieve

A few decisions that mattered:

• It runs after generation, not instead of it. You pay for the generation call whether or not the answer survives the guardrail. That's the cost of the safety net.
• Streaming gets awkward. If you stream tokens to the user, you've already shown them half a hallucinated sentence before the guardrail has anything to score. We buffer the full answer, run the check, then release it. Slightly slower perceived response, but for coverage questions that's the right trade. Bedrock does support guardrail checks over streaming with chunked evaluation, but for high-stakes answers we'd rather not show anything we might retract.
• The fallback matters as much as the check. When the guardrail blocks, we don't say "I can't help." We say "I want to make sure I get your coverage details exactly right, let me connect you with a representative," and we attach the retrieved sections for the human. A blocked response should fail useful, not just fail closed.

Tuning the thresholds is the actual work

The default instinct is "set both thresholds high, like 0.9, so nothing slips through." That over-blocks badly. A perfectly good answer that paraphrases the policy ("your windshield is covered under comprehensive, so the $500 comprehensive deductible applies") scores lower than a robotic verbatim quote, because paraphrase introduces words that aren't literally in the source. Crank grounding to 0.9 and you'll block half your good answers and dump everyone to a human queue.

The other instinct, "set it low so the agent stays helpful," leaks exactly the hallucinations you added the guardrail to catch.

So we tuned it like any other classifier:

1. Built a labeled eval set. A few hundred real (anonymized) question + retrieved-context + answer triples, each hand-labeled "faithful" or "not faithful," with a deliberate slice of the subtle cases: right coverage, wrong deductible; right deductible, wrong limit; "I don't know" answers; questions the context genuinely doesn't cover.
2. Ran the check across all of them and looked at the score distributions for the faithful vs. unfaithful sets. There's a band where they overlap. Your threshold lives at the edge of that band, biased toward whichever error is more expensive. For us, a wrongly-released coverage answer was far worse than a wrongly-blocked one, so we biased toward blocking.
3. Tuned grounding and relevance separately, because they fail differently. Grounding needed to be fairly strict. Relevance we could keep looser. An off-topic-but-accurate answer is annoying, not dangerous, and a too-strict relevance threshold blocked legitimate "here's the related thing you probably also need to know" answers.

Without naming exact numbers: grounding ended up meaningfully stricter than relevance, and both ended up nowhere near 0.9. The number isn't transferable anyway. It depends on your chunk sizes, your model, and how much your answers paraphrase. Measure your own distributions.

One more tuning lesson: re-check thresholds when you change chunk size. Big grounding sources dilute the signal. Bury the relevant sentence in three pages of unrelated policy boilerplate and the grounding score drifts down even for good answers. Tighten your retrieval and the same threshold suddenly behaves differently.

The gotchas

Grounding is not correctness. Said it above, repeating it because it's the one that actually hurt us. Early on, retrieval occasionally pulled the prior policy term version instead of the current one. The model faithfully summarized it. Grounding score: high. Answer: wrong. The contextual grounding check did its job perfectly. It confirmed the answer matched the source, and the answer was still wrong because the source was wrong. The fix was upstream, in retrieval and document versioning. The guardrail is the last line, not the only line.

"I don't know" answers score strangely. When the model correctly says "your documents don't specify a rental reimbursement limit," that's the right answer, but it's not "grounded" in the source in the usual sense and it's arguably low-relevance to the literal question. Decide explicitly how you want those handled. We treat a graceful no-answer as a pass and route it to a human if the customer pushes.

Latency and cost are real. It's a full extra inference pass on every answer. For a support agent that's fine. A few hundred extra milliseconds, a small per-request cost. For a high-QPS, low-stakes feature it might not be worth it. Budget for it. Don't discover it in the bill.

Multi-turn conversations need you to decide what the grounding source is. For a one-shot Q&A it's obvious: the chunks retrieved for this question. In a back-and-forth, do you ground against just this turn's retrieval, or the accumulated context? We scope it to the current turn's retrieved chunks plus anything explicitly carried forward, and we re-retrieve aggressively rather than letting stale context ride along.

When we'd skip it

If the answer isn't something a customer will act on (a "summarize these three articles for me" feature, internal search, anything where a small infidelity costs nothing) the extra latency and cost probably aren't worth it, and looser KB relevance scoring is enough. We reach for contextual grounding when a wrong answer has consequences: coverage, eligibility, money, compliance. Insurance is squarely in that bucket. So is anything in fintech, healthcare, or legal.

The bottom line

Contextual grounding turned our auto insurance agent from "usually right" into "right, or it tells you it isn't." That's the bar for anything customer-facing in a regulated domain.

If you take three things from this:

1. The qualifiers (grounding_source, query) are the implementation. Get those right or the check does nothing. The standalone ApplyGuardrail call gives you the most control if you're not all-in on Bedrock Knowledge Bases.
2. Tune the thresholds against a labeled eval set, separately per dimension, biased toward whichever error costs more. 0.9-everything is not a strategy.
3. It checks faithfulness to the source, not truth. Fix retrieval and document versioning too, or you'll faithfully serve wrong answers.

---

Written by Ramesh Nori. If this was useful or you have feedback, reach me at cloudbuckle@gmail.com.