What does "knowledge infrastructure" actually mean for AI agents?

Knowledge infrastructure is the system that manages what your agents know. It includes the knowledge sources (docs, Confluence, Notion, etc.), the mechanisms that keep knowledge current and consistent, the APIs agents use to retrieve knowledge, and the observability that lets you see what each agent knows and where it came from. Think of it as the plumbing between agent queries and authoritative information.

Why do agents give inconsistent answers if they're using the same model and prompts?

Because they're querying different knowledge. Agent A might be pulling from a doc updated three days ago. Agent B might be pulling from a cached version from last week. Agent C might be pulling from a different source entirely. Same model, same prompts, different knowledge = different answers. This is a knowledge infrastructure problem, not a model problem.

How do I handle cross-agent knowledge consistency without rebuilding everything?

Start by identifying your single source of truth for product knowledge - usually your primary product documentation. Make sure all agents query that source, not multiple copies or cached versions. Then implement automatic propagation so that when the source updates, all agents see the change.

What's the difference between a vector database and a knowledge layer?

A vector database stores and retrieves embeddings based on semantic similarity. A knowledge layer manages knowledge: keeping it current, detecting contradictions, understanding document structure, synchronizing across multiple agents, providing tracing and observability. A vector database is a component that might sit inside a knowledge layer - but a knowledge layer is something larger and more structured.

How does a knowledge layer fit into an existing agent architecture?

Agents replace their direct knowledge source connections with API calls to the knowledge layer. Instead of Agent A querying Confluence directly and Agent B querying Notion, both agents query the knowledge layer API. The knowledge layer abstracts the underlying sources, handles updates, detects conflicts, and serves consistent answers. You don't have to rewrite your agents - just change where they get knowledge from.

How do I know if knowledge staleness is causing my accuracy degradation?

Run a retrieval audit on your 20-30 most recent failure cases. For each failure, check whether the retrieved context was outdated, incomplete, or simply not found. If more than 60% of failures trace back to knowledge-layer problems rather than model reasoning errors, staleness is likely your primary blocker. Most teams find this is true. Almost no teams check.

What's the difference between output monitoring and retrieval observability?

Output monitoring tracks what your model says. Retrieval observability tracks what your retriever finds. Output monitoring catches hallucinations after they happen. Retrieval observability lets you diagnose why they happened, whether the right context was retrieved, whether it was current, and whether the retriever understood the document structure. You need both, but most teams only have the first.

How does hierarchical retrieval reasoning (HRR) actually work?

HRR maps relationships between documents before retrieval happens. Instead of treating every chunk as independent, it understands that section 3.2 belongs to chapter 3, which belongs to a specific product version. When a query comes in, it retrieves at the right level of the hierarchy, not just the closest embedding match. This means if you ask about a process, it retrieves the full process context, not just the paragraph that happened to match your query vector.

How long does it take to integrate a knowledge layer into an existing RAG pipeline?

For most teams, 2-6 weeks depending on how many data sources you're connecting and how clean your existing infrastructure is. The integration is usually straightforward: swap your vector database calls for knowledge layer API calls, connect your data sources, and configure your sync frequency. Teams using native connectors for Confluence, Notion, Salesforce, SharePoint, and Google Drive report no custom pipeline work required.

What if we're already using LangChain or LlamaIndex?

A knowledge layer is retrieval-agnostic. It sits upstream of your retriever and changes what your retriever searches over. You can use it with LangChain, LlamaIndex, agent frameworks, or internal architectures. Most teams add the knowledge layer by adding a single API call in their retrieval chain. Your existing orchestration stays intact.

Retrieval Observability: Seeing the Full Chain Behind Every AI Answer

TL;DR: Retrieval observability means seeing the entire chain of how an AI system retrieved and ranked information before generating an answer ,not just the final output. Without it, debugging a wrong AI answer is like debugging a SQL query by only looking at the UI result. The solution: trace every retrieval decision, make answers deterministically reproducible, and expose confidence scores so you're asking the right question, what did the model retrieve? not just what did it say?

The Debug Session Nobody Wants

A customer reports that your AI assistant gave them information about a feature that was deprecated three versions ago. The output looks confident. Reasonable. Wrong.

What happened?

You pull up the chat logs. You look at the answer. It reads fine. You ask the model the same question again. This time it answers correctly — pulling from the current version. You run it a third time. Back to the old answer.

This is the moment every engineer working on RAG systems hits a wall. The visibility stops at the output. What got retrieved? What documents were ranked highest? Why was that old version considered more relevant than the current one? None of these answers are visible. The answer is wrong, but the path to wrongness is invisible.

This is the core problem that retrieval observability solves.

"I can see what the AI outputs. I can't see what it retrieved. That's the problem."

And the follow-up is worse:

"Same question. Different answer on replay. That's not an accuracy problem — that's a non-determinism problem."

Without visibility into the retrieval chain, debugging becomes guesswork. Is the model hallucinating? Is the retrieval system broken? Is the knowledge base stale? Or is the query being rewritten in unpredictable ways? Without a trace, you're blind.

What Is Retrieval Observability?

Retrieval observability means seeing the complete chain of decisions made during information retrieval — before the language model ever generates a response.

Specifically: visibility into what query variations were tried, which documents or knowledge nodes were ranked highest, why they were ranked that way, and what confidence the retrieval system assigned to each choice.

In traditional database debugging, you don't just see the output of a query — you see the query plan, the indexes hit, the execution time, the rows scanned. That's why database engineers can diagnose problems efficiently. The information retrieval layer of an AI system deserves the same level of inspection.

Without retrieval observability, the system is a black box: query goes in, answer comes out. You have no visibility into the middle — and that's where most AI errors live.

Why Output Monitoring Isn't Enough

Consider the standard approach to AI quality: test the final answer. Is it correct? Does it match expected output? Does it sound coherent?

This is observability of the last step only. It's equivalent to debugging a complex SQL query by only reading the UI result set. You can see that the result is wrong, but without seeing the query that ran, the indexes used, or the filter conditions, you can't fix it.

Most output-only monitoring tells you that something failed. Retrieval observability tells you where it failed.

Consider this failure scenario:

An AI system is supposed to answer questions about API documentation. The latest version is in the knowledge base. A user asks: "Does your API support OAuth 2.0?"

The system answers: "No, we only support API key authentication." This is wrong.

Output monitoring shows: Answer is factually incorrect.

Retrieval observability shows: The retrieval system ranked a document from version 2.1 (which predates OAuth support) higher than the current version 3.2. The confidence score was 0.92, which is why the model trusted it.

These are completely different diagnoses requiring completely different fixes:

Output monitoring: "The answer is wrong. Ask the model to be more careful."
Retrieval observability: "The ranking function is prioritizing older documents. Adjust recency weighting or implement version-aware chunking."

Without the trace, the former is all you get.

The Non-Determinism Problem

Ask an RAG system the same question twice. Get a different answer.

This happens more often than most engineers realize. Same query, same knowledge base, different results. The model might be drawing from different retrieved documents. The retrieval system might be returning results in a different order due to vector similarity tie-breaking. The prompt might have been rewritten differently on the second run.

Non-determinism makes debugging impossible. If a user reports a wrong answer and the team tries to reproduce it, they might get the correct answer on the first retry. Is it fixed? No. Run it again. Wrong answer reappears.

This is a nightmare for production systems. Compliance requires audit trails. Customers require consistency. Support tickets require reproducibility.

The reasons most RAG systems are non-deterministic:

Floating-point similarity scores don't sort identically across runs if scores are close
Lazy query rewriting that varies based on model temperature or sampling
Uncontrolled retrieval ranking that treats tied relevance scores as equivalent

Retrieval observability solves this by making every step explicit and reproducible. If the system can show the reasoning trace for a given query, it can be replayed. Same question, same trace, same answer, every time.

This is deterministic replay — and it's essential for enterprise AI.

The 4-Phase Reasoning Trace

A complete retrieval trace captures four critical phases:

Phase 1: Query Rewrite

The original user query often isn't the best question to ask the knowledge base. "does the api support webhooks" gets rewritten to "API webhook integration support." The trace should show the original query, the rewritten query, and the reason for the rewrite.

If answers are inconsistent, sometimes the issue is inconsistent query rewriting.

Phase 2: Sub-Queries

Complex questions often need multiple retrieval passes. "What does your API support and what are the limits?" spawns two sub-queries: "API supported features" and "API rate limits and quotas."

The trace should show which sub-queries were generated and why. If a sub-query is missing, that's a knowledge gap. If it returns no results, that's a coverage problem.

Phase 3: Node Selection + Reason

For each sub-query, which knowledge nodes were retrieved and why?

If the wrong node is ranked first, this is where you spot it.

Phase 4: Confidence Score

After all nodes are retrieved and ranked, the system should assign a confidence score: how confident is this retrieval result going to produce a correct answer?

Signal	Meaning
High confidence + wrong answer	Problem is in the model layer
Low confidence + any answer	Problem is in the retrieval layer

Without this score, there's no way to distinguish retrieval failures from model failures.

Deterministic Replay: Auditing AI Answers

Suppose a compliance officer asks: "Show me how your AI system arrived at this customer answer."

With retrieval observability and deterministic replay, the answer is: "Run this exact query. Here's the trace. Here are the retrieved documents. Here's the reasoning chain. Run it. Same answer. Every time."

This is auditable AI.

Non-deterministic systems can't do this. They can show what happened but can't prove it would happen again. That's unacceptable for regulated industries.

Deterministic replay requires: seeding random number generators, freezing query rewriting logic, locking retrieval ranking, and recording the full reasoning trace.

The payoff: a complete audit trail. Same query, same trace, same answer, same documents retrieved, same confidence score.

What Good Retrieval Traces Reveal

Scenario 1: Stale document retrieved

The trace shows: the older doc scored 0.89, the current one scored 0.84. Recency weighting isn't implemented. Fix: add version-aware retrieval.

Scenario 2: Wrong section of the right document

The trace shows: the right document was retrieved, but the wrong chunk was ranked first. Fix: improve chunk boundaries or add section-level metadata.

Scenario 3: Low confidence + hallucination

The trace shows: confidence 0.31, minimal match, no direct answer found. The model answered anyway. Fix: add confidence thresholds — if confidence < 0.5, return "I don't know" instead of letting the model guess.

Scenario 4: Conflicting knowledge nodes

The trace shows: on run 1, doc A (score 0.92) was retrieved first. On run 2, doc B (score 0.91) was retrieved first. The docs contradict each other and the retrieval order is determined by a float tie-break. Fix: break ties deterministically and flag conflicting sources.

Each of these scenarios is invisible without a retrieval trace. Output monitoring shows only that something went wrong. The trace shows what went wrong and where.

Adding Retrieval Observability to a LangChain Pipeline

Most LangChain RAG pipelines look like this:

query → retriever.invoke() → documents → llm.invoke() → answer

What's missing: visibility into what retriever.invoke() actually returned and why.

To add retrieval observability:

Wrap the retriever with logging — capture retrieved documents, relevance scores, and metadata.
Make query rewriting explicit — log both the original and rewritten queries with the rewrite reason.
Add a ranking step — assign confidence scores and capture why documents were ranked in a given order.
Record the trace — serialize the full retrieval trace (phases 1–4) before passing documents to the LLM.
Implement deterministic seeding — set random seeds at the start of each trace so decisions are reproducible.‍

The trace object is what enables debugging and replay. For a production-ready implementation with full retrieval tracing, see how Brainfish's knowledge layer provides built-in observability →

What We Learned from Analyzing 1M Support Interactions — Real-world retrieval patterns from 1 million support interactions, including what causes wrong answers
Answering the Tough Questions About Brainfish — How Brainfish approaches retrieval tracing and observability under the hood
Compliance-Grade AI: How High-Governance Teams Pilot Without Risk — Why deterministic replay and audit trails matter in regulated industries
Why Brainfish — The case for AI-native knowledge infrastructure with full retrieval observability built in

‍

"AI that can't explain its retrieval chain can't be trusted in production."

See a full retrieval trace in action → Book a demo

‍

import time
import requests
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader

# --- 1. OpenTelemetry Setup for Observability ---
# Configure exporters to print telemetry data to the console.
# In a production system, these would export to a backend like Prometheus or Jaeger.
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
span_processor = SimpleSpanProcessor(ConsoleSpanExporter())
trace.get_tracer_provider().add_span_processor(span_processor)

metric_reader = PeriodicExportingMetricReader(ConsoleMetricExporter())
metrics.set_meter_provider(MeterProvider(metric_readers=[metric_reader]))
meter = metrics.get_meter(__name__)

# Create custom OpenTelemetry metrics
agent_latency_histogram = meter.create_histogram("agent.latency", unit="ms", description="Agent response time")
agent_invocations_counter = meter.create_counter("agent.invocations", description="Number of times the agent is invoked")
hallucination_rate_gauge = meter.create_gauge("agent.hallucination_rate", unit="percentage", description="Rate of hallucinated responses")
pii_exposure_counter = meter.create_counter("agent.pii_exposure.count", description="Count of responses with PII exposure")

# --- 2. Define the Agent using NeMo Agent Toolkit concepts ---
# The NeMo Agent Toolkit orchestrates agents, tools, and workflows, often via configuration.
# This class simulates an agent that would be managed by the toolkit.
class MultimodalSupportAgent:
    def __init__(self, model_endpoint):
        self.model_endpoint = model_endpoint

    # The toolkit would route incoming requests to this method.
    def process_query(self, query, context_data):
        # Start an OpenTelemetry span to trace this specific execution.
        with tracer.start_as_current_span("agent.process_query") as span:
            start_time = time.time()
            span.set_attribute("query.text", query)
            span.set_attribute("context.data_types", [type(d).__name__ for d in context_data])

            # In a real scenario, this would involve complex logic and tool calls.
            print(f"\nAgent processing query: '{query}'...")
            time.sleep(0.5) # Simulate work (e.g., tool calls, model inference)
            agent_response = f"Generated answer for '{query}' based on provided context."
            
            latency = (time.time() - start_time) * 1000
            
            # Record metrics
            agent_latency_histogram.record(latency)
            agent_invocations_counter.add(1)
            span.set_attribute("agent.response", agent_response)
            span.set_attribute("agent.latency_ms", latency)
            
            return {"response": agent_response, "latency_ms": latency}

# --- 3. Define the Evaluation Logic using NeMo Evaluator ---
# This function simulates calling the NeMo Evaluator microservice API.
def run_nemo_evaluation(agent_response, ground_truth_data):
    with tracer.start_as_current_span("evaluator.run") as span:
        print("Submitting response to NeMo Evaluator...")
        # In a real system, you would make an HTTP request to the NeMo Evaluator service.
        # eval_endpoint = "http://nemo-evaluator-service/v1/evaluate"
        # payload = {"response": agent_response, "ground_truth": ground_truth_data}
        # response = requests.post(eval_endpoint, json=payload)
        # evaluation_results = response.json()
        
        # Mocking the evaluator's response for this example.
        time.sleep(0.2) # Simulate network and evaluation latency
        mock_results = {
            "answer_accuracy": 0.95,
            "hallucination_rate": 0.05,
            "pii_exposure": False,
            "toxicity_score": 0.01,
            "latency": 25.5
        }
        span.set_attribute("eval.results", str(mock_results))
        print(f"Evaluation complete: {mock_results}")
        return mock_results

# --- 4. The Main Agent Evaluation Loop ---
def agent_evaluation_loop(agent, query, context, ground_truth):
    with tracer.start_as_current_span("agent_evaluation_loop") as parent_span:
        # Step 1: Agent processes the query
        output = agent.process_query(query, context)

        # Step 2: Response is evaluated by NeMo Evaluator
        eval_metrics = run_nemo_evaluation(output["response"], ground_truth)

        # Step 3: Log evaluation results using OpenTelemetry metrics
        hallucination_rate_gauge.set(eval_metrics.get("hallucination_rate", 0.0))
        if eval_metrics.get("pii_exposure", False):
            pii_exposure_counter.add(1)
        
        # Add evaluation metrics as events to the parent span for rich, contextual traces.
        parent_span.add_event("EvaluationComplete", attributes=eval_metrics)

        # Step 4: (Optional) Trigger retraining or alerts based on metrics
        if eval_metrics["answer_accuracy"] < 0.8:
            print("[ALERT] Accuracy has dropped below threshold! Triggering retraining workflow.")
            parent_span.set_status(trace.Status(trace.StatusCode.ERROR, "Low Accuracy Detected"))

# --- Run the Example ---
if __name__ == "__main__":
    support_agent = MultimodalSupportAgent(model_endpoint="http://model-server/invoke")
    
    # Simulate an incoming user request with multimodal context
    user_query = "What is the status of my recent order?"
    context_documents = ["order_invoice.pdf", "customer_history.csv"]
    ground_truth = {"expected_answer": "Your order #1234 has shipped."}

    # Execute the loop
    agent_evaluation_loop(support_agent, user_query, context_documents, ground_truth)
    
    # In a real application, the metric reader would run in the background.
    # We call it explicitly here to see the output.
    metric_reader.collect()

Retrieval observability means seeing the complete chain of retrieval decisions before the model generates an answer. Without it, debugging wrong AI answers is like debugging SQL by only reading the UI result. Learn how to add full retrieval tracing, deterministic replay, and confidence scoring to your LangChain pipeline.

Frequently Asked Questions

How do I add retrieval tracing to an existing LangChain pipeline?

Wrap your retriever with explicit logging that captures original and rewritten queries, retrieved documents with relevance scores, and metadata. Add a ranking step that assigns confidence scores. Record the full trace before passing documents to the LLM. For deterministic replay, implement random seeding so the same seed produces the same query rewriting and ranking decisions.

What does non-determinism mean in a RAG context, and why is it a problem?

Non-determinism means asking the same question twice produces different answers or different retrieval chains. It's a problem because it makes debugging impossible — if an answer is wrong and you try to reproduce it, you might get the right answer instead. It also breaks compliance and audit trails. Deterministic replay means the same query always produces the same trace, the same retrieved documents, and the same answer.

How is retrieval observability different from standard RAG monitoring?

Standard RAG monitoring observes the final answer quality — Is it correct? Does it match user feedback? Retrieval observability observes the *retrieval layer itself* — What was retrieved? Why? With what confidence? You can have 95% accuracy and still have terrible retrieval observability, meaning you can't diagnose *why* the other 5% failed.

What's the difference between retrieval observability and just logging what documents were retrieved?

Logging tells you what was retrieved. Observability tells you why it was retrieved and how confident the system is in that retrieval. Observability includes the full reasoning trace: query rewrite decisions, relevance scores, ranking logic, and a confidence score indicating whether the retrieved context is likely sufficient for a correct answer. Logging is data; observability is diagnosis.

Share this post

Justin Gabriel

March 31, 2026

Head of Product Marketing, Brainfish