An AI agent feedback loop with a human knowledge owner

Some people use “AI agent feedback loop” for an agent that checks its own output and tries again. Others mean training a model on human ratings. Here it means a narrower loop that is easy to skip. When the agent doesn’t have the information to answer, the gap should reach the person who owns that information, and that person decides what changes.

Who does what

The knowledge owner is a person or a team, and often there are several. HR owns the leave policy and IT owns the VPN instructions.

Why the agent shouldn’t fix its own knowledge

You could let an agent write what it learns back into its knowledge base. That causes two problems.

First, the agent can’t check what it’s told. If a user says “the refund window is 90 days now”, the agent has no way to know whether that’s true.

Second, anything a user types is untrusted input. If user messages can change the knowledge base, any user can change the answers everyone else gets, including someone doing it on purpose with prompt injection.

So the agent reports and a person decides. If a user offers a correction, it can travel with the report as an unverified proposal that the owner checks against a real source.

Two ways to report

The agent can report by calling a tool, or your application can report from code. They catch different cases.

Use REST for anything you need to record every time. Add the tool to catch the gaps only the model can see. Then read what arrives and adjust the agent’s instructions.

Setting it up with Kebivo

Kebivo is an open-source service you host yourself. It accepts these reports over MCP and REST, groups repeats and routes them to owners.

Give the agent a gap-only token

Create a project for the agent and issue a project token with the feedback:gap scope. Then point the agent’s MCP client at Kebivo’s Streamable HTTP endpoint, /mcp. In the mcpServers format that Claude Code and similar clients use:

{
  "mcpServers": {
    "kebivo": {
      "type": "http",
      "url": "http://127.0.0.1:8000/mcp",
      "headers": {
        "Authorization": "Bearer ${KEBIVO_TOKEN}"
      }
    }
  }
}

This is examples/mcp-client-config.json from the repository. It reads the token from an environment variable, so the file holds no secret. Kebivo’s MCP server accepts a bearer token and doesn’t support OAuth, so clients that require OAuth can’t connect.

Kebivo picks which tools to show from the token’s scopes and the project’s corrections policy. On a local instance running Kebivo 0.1.1, tools/list with a gap-only token returned one tool, report_knowledge_gap, whose only required input is question. When the agent calls it with:

{"question": "Which regions does the EU data residency option cover?", "tags": ["compliance"], "agent_id": "support-agent"}

the tool returns:

{"event_id": "d19f7922-882e-46fd-87ab-2c1c4e74d786", "status": "accepted"}

accepted means Kebivo has stored the report. Grouping and routing happen afterwards. Optional inputs include context, previous_answer, tags, conversation_id and agent_id, and the MCP guide lists their limits.

Tell the agent when to report

The tool’s description tells the model when to call it. For agents that report over REST, the repository’s examples/agent-instructions.md has a block of instructions to adapt. The file says MCP agents don’t need it, because the tool descriptions carry the same guidance. Its rules are still a useful starting point. When the information is missing, out of date, contradictory or too vague, the agent should tell the user it couldn’t find a reliable answer and report the gap, not invent an answer. It shouldn’t report small talk or questions it answered, and it should leave out personal data.

Keep corrections off until you need them

New projects don’t accept corrections. To accept them, an administrator turns on the project’s corrections policy and issues a token with the feedback:correction scope. Only then does the submit_knowledge_correction tool appear. Kebivo stores every correction as unverified, shows it in the inbox under “Submitted correction (unverified)”, and never changes your knowledge base. Keep correction tokens away from customer-facing agents. Give them only to a workflow you trust to handle corrections that users supply explicitly.

Route reports to the owner

Tag reports by topic, or give each project default tags; one option is a project per agent or team. Routing rules send each tag to its owner by email or webhook. Every report also stays in the inbox, grouped with its repeats.

The owner updates the source, checks that the agent now answers the question, and marks the group resolved. If the same question arrives again, worded the same way, Kebivo reopens the group. A RAG feedback loop for unanswered questions covers that fix-and-recheck step in more detail.

What this setup doesn’t do

Kebivo can’t make the agent call the tool, judge whether an answer was right, or apply fixes. A model’s tool calls aren’t guaranteed, which is why the REST path exists: your code can send the report itself whenever it detects a miss. The REST guide shows the request. The loop closes when a person updates the source and marks the group resolved.

To wire the tool into a real agent, follow the LangGraph guide or the OpenAI Agents SDK guide. Both use runnable examples from the repository. The quickstart starts Kebivo and creates the token.