AI Agent Incident Response: What to Do When an Agent Misbehaves
When an agent misbehaves, stop its next action at a point it cannot route around and revoke every credential it holds. Then freeze the evidence before anyone fixes anything: the full trace, every tool call and its arguments, and the model, prompt, tool and memory versions. Trace the chain from input to action, reverse what it did, clean its memory and turn the incident into a test. NIST SP 800-61r3 maps each step.
On this page
Most incident response plans assume the thing that went wrong is a compromised account, a piece of malware or an outage. A misbehaving AI agent is none of those. It acted with valid credentials, through tools it was allowed to use, and it may have been steered by a sentence in a document that nobody would call malware.
That changes the order of operations more than the principles. The principles are already written down.
What counts as an AI agent incident
Write the definition before you need it. NIST's incident response guidance expects incidents to be declared when adverse events meet defined incident criteria (DE.AE-08), and an agent needs criteria of its own. Four are a reasonable start:
- An action outside its boundary. A tool it should not have used, a target outside its remit, an amount above its limit.
- Data where it should not be. Customer records in an outbound email, a secret in a tool argument, an internal document in a public reply.
- An instruction that did not come from its principal. It followed text from a web page, a tool result or another agent as though the user had said it.
- Runaway behaviour. Loops, fan-out or spend with no matching outcome.
A wrong answer in a draft is a quality problem. A wrong action in a system of record is an incident.
How NIST SP 800-61 Rev. 3 maps to an AI agent incident
NIST SP 800-61 Rev. 3, published in April 2025, superseded the 2012 Rev. 2. It reframes the old four-phase life cycle around a model built on the six CSF 2.0 Functions: Govern, Identify and Protect as preparation; Detect, Respond and Recover as the response; and an Improvement category that takes in lessons at any point. It also tells organisations to use whichever model suits them. Here is a six-step version for agents, with the CSF 2.0 outcomes each step serves.
| Step | What it means for an agent | CSF 2.0 outcomes in SP 800-61r3 |
|---|---|---|
| 1. Detect | Spot the action that broke the criteria, and declare | DE.AE-02, DE.AE-03, DE.AE-08 |
| 2. Contain | Stop its next action; revoke what it holds | RS.MI-01 |
| 3. Preserve | Freeze traces, versions and memory before changing anything | RS.AN-06, RS.AN-07 |
| 4. Investigate | Rebuild the chain from input to action; size the damage | RS.AN-03, RS.AN-08 |
| 5. Recover | Reverse the effects, clean memory, reinstate with less access | RS.MI-02, RC.RP-01 to RC.RP-06 |
| 6. Learn | Turn the incident into a test and a control | ID.IM-03, ID.IM-04 |
Step 1: Detect it from what the agent did, not what it said
Agent incidents rarely surface as errors. The tool calls succeeded; they should not have. Tracing shows what happened, not whether it was allowed, so detection needs signals about actions: an unusual tool pattern, a new destination, spend rising without results, an independent check returning fail, a customer complaint.
SP 800-61r3 asks for information to be correlated from multiple sources (DE.AE-03). For an agent, that means joining its trace to the logs of the systems it touched. Suppose the trace shows one routine "update record" call; the database log shows it was the fortieth that minute.
Step 2: Contain it — stop the next action, then revoke its credentials
NIST defines containment as preventing the expansion of an incident. For an agent, expansion is the next tool call. Two moves, in this order:
- Stop its actions at a point it cannot route around. A line in the system prompt saying "stop" is a request. The stop has to sit between the agent and its targets, which is the argument in rogue agent containment.
- Revoke or rotate every credential it held. API keys, OAuth tokens, delegated sessions, database roles. If the agent was steered, assume anything it could read, its own secrets included, may have been copied out. The OWASP Top 10 for Agentic Applications lists kill switches and credential revocation for rogue agents (ASI10), with quarantine for forensic review.
Then follow its output downstream. Scheduled jobs, queued messages and other agents that consumed its output can carry the incident on after the agent has stopped.
If the agent shares a service account, revoking it takes the others down too: the best argument for giving every agent its own identity before the first incident.
Step 3: What evidence to preserve after an AI agent incident
The instinct is to fix: patch the prompt, clear the memory, redeploy. Each destroys evidence. SP 800-61r3 asks that incident data be collected with its integrity and provenance preserved (RS.AN-07), and that the investigators' own actions be recorded too (RS.AN-06). Preserve:
- The full trace: inputs, retrieved documents, tool results, and every tool call with its arguments and response.
- The versions in play: model, system prompt, configuration, and the tool definitions as they were at the time. An MCP server can change a tool's description after you approved it, so today's definition proves nothing about last night's.
- Memory: a snapshot of conversation memory and any vector store the agent writes to, taken before anyone purges it.
- Authority: the credentials and scopes it held, and on whose behalf it acted.
- Checks: what evaluated each action, its verdict and its version, and any human approvals.
The agent's own logs are a start, not the record. A system that has just failed is a poor witness to its own failure, which is why the audit trail belongs outside the agent.
Step 4: Investigate the chain from input to action
SP 800-61r3 asks you to establish the sequence of events and the root cause (RS.AN-03). For an agent, the sequence is a causal chain with three links:
- Where the instruction entered: the user, a document, a web page, a tool result, memory or another agent.
- Which call carried it out: the tool, the arguments, the target.
- Which control should have stopped it, and why it did not: absent, misconfigured, skipped or overridden.
Don't ask the agent what happened. Its explanation is newly generated text, not a record. Read the trace.
Then estimate the magnitude (RS.AN-08). Suppose a support agent followed an instruction hidden in one customer's email and refunded the wrong account. Which other sessions read that email, which agents share that memory, and where else does the instruction sit?
Step 5: Recover the data, the memory and the agent's access
Reverse the effects. Void the payments, revert the writes, correct the messages, using backups whose integrity you have verified first (RC.RP-03).
Clean the memory. OWASP's guidance on memory and context poisoning (ASI06) calls for snapshots, rollback and quarantine of suspect entries. Poisoned memory left in place replays the incident next session.
Reinstate the agent deliberately. New credentials, narrower scopes, and OWASP's steps for bringing a rogue agent back: a trusted baseline, fresh attestation and human approval before it returns to production.
Declare the end of recovery against criteria (RC.RP-06), not when the alerts go quiet.
Step 6: Turn the incident into a test
SP 800-61r3 says lessons should often be shared as soon as they are identified, not held until recovery ends. The most useful lesson is the trace itself. Remove the personal data and it becomes a regression case that every future version must pass, which is how red-team findings should be used too.
An AI agent incident response checklist
Answer these before the first incident:
- Does each agent have written incident criteria that separate quality problems from incidents?
- Can you stop one agent's actions without relying on the agent, and without stopping the rest?
- Can you list, and revoke, every credential each agent holds?
- Are prompts, tool definitions and memory versioned, so you can show what was in play?
- Does a record exist outside the agent's own logs?
- Who decides when an agent is reinstated, and what must they see first?
Where Syntrox fits
Syntrox sits outside the agent and records, for every trace it evaluates, the score, confidence and judge version behind the verdict: the "checks" evidence in the list above. Nucleus runs at temperature 0 with a fixed seed, so re-running a preserved trace during an investigation returns the same verdict, stamped with the judge version that produced it. It treats any text inside the material it evaluates as evidence, never as instructions, so an injected instruction in that trace is weighed, not obeyed, and one addressed to the evaluator counts against the output.