Syntrox / Blog
Notes on safeguarding autonomous agents.
Technical articles on governing AI agents in production: runtime policy enforcement, prompt-injection latency, observability gaps and compliance evidence.
How to Build an Evaluation Dataset for an AI Agent: Cases, Criteria, Labellers and Agreement
Build the golden set in seven steps: source cases from real traffic; write binary, observable pass/fail criteria before labelling; have at least two people label an overlap and measure chance-corrected agreement; map coverage with a task-by-condition matrix; size each category for the confidence interval you need; version cases, criteria and labels; and refresh the set while keeping part of it out of prompt tuning.
ReadWhen Should an AI Agent Escalate to a Human? Five Triggers, and a Hand-Off That Works
Escalate on five triggers: the action is valuable or irreversible; something other than the agent says the case is uncertain; the topic is in a policy class a person always owns; the customer asks for a person; or the agent has failed repeatedly. Evaluate the triggers outside the model, and hand over a decision packet: the request, what the agent did, why it escalated, the proposed action and the decision needed.
ReadA2A Protocol Security: How AI Agents Authenticate and Authorise Each Other
A2A leaves trust to standard web security: the Agent Card declares how to authenticate, credentials are obtained out of band, and the receiving agent must authenticate and authorise every request against its own policy. Before trusting another organisation's agent, verify its signed card with a key you obtained yourself, scope credentials per skill, treat card text and replies as untrusted input, and log both sides.
ReadWhat to Measure for AI Agents in Production: Five Metrics and How to Set Targets
Measure outcomes, not uptime: task success rate checked against the end state, escalation and intervention rate, policy-violation rate by severity, cost per completed task, and time to verified resolution. Express each as good events over valid events, set targets the way SRE teams set SLOs, agree in advance what happens when a budget runs out, and treat every severe violation as an incident.
ReadGuardrails for AI Customer-Support Agents That Issue Refunds, Credits and Account Changes
An agent that can refund, credit or change an account needs controls outside the model: value limits enforced by the payments system, verification before any account change, eligibility computed from the written policy, escalation that gives a person the whole case, and a record of every action with the approval and policy behind it. A prompt can describe the policy. It cannot enforce it.
ReadKeeping Secrets Out of an AI Agent's Context Window
Never put an API key, token or password anywhere a model can read it: not the system prompt, not memory, not a tool description, not a file in its workspace. A model cannot keep a secret it has seen. Broker credentials outside the model instead, so each tool attaches a short-lived, task-scoped token at call time, and scan traces, memory and outputs for any secret that still gets through.
ReadBrowser and Computer-Use Agents: The Security Risks of an AI That Clicks for You
A browser or computer-use agent acts with everything the browser holds: logged-in sessions, saved passwords, the power to buy, send and delete. The four risks are hidden instructions in web content, inherited sessions, irreversible clicks and data typed into forms. The controls: a separate browser, per-task logins, allowlists, confirmation of consequential actions, page content treated as data, and bounded runs.
ReadHow to Sandbox an AI Agent That Runs Code: Containers, gVisor or Firecracker
Run agent-written code in a sandbox you would trust with hostile code, because injected code is hostile code. Choose the boundary by who shares the host: hardened containers for trusted, single-tenant work; gVisor or Firecracker microVMs for untrusted or multi-tenant code. Then set five defaults whatever the runtime: no network, hard resource limits, a throwaway filesystem, no standing credentials, no root.
ReadHow to Threat-Model an AI Agent: A STRIDE Walkthrough for Tool-Calling Agents
Map the agent's data flow: every point where outside text enters its context is a trust boundary. Then walk STRIDE: spoofing (forged agents and tools), tampering (poisoned tools and memory), repudiation (shared identities), disclosure (exfiltration via permitted tools), denial of service (loops, guardrail outages) and elevation of privilege (confused deputy). Then ask what STRIDE doesn't: what if the agent is wrong?
ReadPermission Drift in AI Agents: How Agent Access Grows, and How to Review It
An agent's access only grows unless someone removes it. It drifts through new tools and MCP servers, inherited service accounts, scopes widened during a pilot, delegation between agents and owners who leave. Review agents as you review people: inventory each agent and its identity, name an owner, compare granted with used access, recertify or remove, disable dormant agents, and review again when the agent changes.
ReadAI Coding Agents With Production Access: Why the Agent Should Never Hold the Keys to Prod
Keep production out of a coding agent's reach instead of asking it to stay away. Five guardrails: separate environments so production is unreachable from the agent's workspace; sandbox what it executes; give it no standing production credentials; block destructive commands at the target, not only in the prompt; and make recovery a tested path. A code freeze written in a prompt is not a control.
ReadAI Agent Incident Response: What to Do When an Agent Misbehaves
When an agent misbehaves, stop its next action at a point it cannot route around and revoke every credential it holds. Then freeze the evidence before anyone fixes anything: the full trace, every tool call and its arguments, and the model, prompt, tool and memory versions. Trace the chain from input to action, reverse what it did, clean its memory and turn the incident into a test. NIST SP 800-61r3 maps each step.
ReadPII and Secrets in AI Agent Traces: How to Keep an Audit Trail Without Keeping Everything
Agent traces capture prompts, retrieved documents, tool arguments and tool results, so once content capture is on they fill with personal data and credentials. Keep the audit trail but minimise what goes into it: remove secrets before storage, pseudonymise identifiers you need to correlate, keep bulky content apart from the action record, restrict and log access, and delete each kind of data on its own schedule.
ReadData Exfiltration Through AI Agents: Five Ways Data Gets Out, and How to Close Them
An agent leaks data through the same channels it uses to work: tool-call arguments, URLs and rendered images, email and messages, files and pull requests it writes, and third-party MCP servers. Close those paths outside the model: allowlist where the agent can send data, inspect tool-call arguments before they leave, strip untrusted links and images from output, and scope every credential to the task.
ReadRegression Testing LLM Agents in CI: How to Catch What a Change Broke
Write every known failure as a test case: a fixed input, the tools and context the agent had, and a check on the outcome rather than the exact wording. Run the suite on every model, prompt, tool and configuration change. Because outputs vary even at temperature 0, run each case several times, gate releases on pass rates per category, and count crashed or unscored runs as failures.
ReadShadow Mode for AI Agent Guardrails: How to Roll Out a New Check Without Breaking Production
Run the new check alongside production first and let it block nothing. Record every verdict next to what actually happened, review the disagreements by hand, and measure false positives and misses per policy on your own traffic. Then turn on enforcement one policy at a time, starting where the verdicts are most trusted and the actions hardest to undo, and send every new version of the check back through shadow.
ReadEU AI Act Article 26: What Companies Deploying AI Agents Have to Do
If you use a high-risk AI system, Article 26 makes you answerable for how it runs: use it as the provider's instructions say, assign trained people with authority to oversee it, keep the input data you control relevant, monitor it and suspend it if it presents a risk, keep its logs for at least six months, and tell workers and affected people. For Annex III systems this applies from 2 December 2027.
ReadNIST AI RMF for AI Agents: Govern, Map, Measure and Manage as Concrete Controls
Read for agents, the NIST AI RMF's four functions become twelve concrete controls. GOVERN: an agent inventory, named owners, and plans for retirement and provider failure. MAP: a written mandate, oversight set per action, and every tool mapped. MEASURE: independent evaluation, production monitoring and tested safe failure. MANAGE: a release gate, a stop path with owners, and incident response.
ReadSR 11-7 and AI Agents: What Model Risk Management Asks For, Now That Its Replacement Leaves Agents Out
SR 11-7 was superseded on 17 April 2026 by SR 26-2, which puts generative and agentic AI explicitly out of scope and leaves their controls to each bank's own governance. SR 11-7 is still the most complete template for that job: a clear statement of purpose, validation of conceptual soundness, ongoing monitoring and outcomes analysis, an inventory, and effective challenge with the authority to restrict use.
ReadThe OWASP Top 10 for LLM Applications, Read for AI Agents: Which Risks Grow When the Model Can Act
Once a model can call tools, four OWASP entries matter most: LLM01 Prompt Injection, LLM06 Excessive Agency, LLM05 Improper Output Handling and LLM02 Sensitive Information Disclosure, because text the agent reads can now become an action. The other six change too, from supply chain to unbounded consumption. Each has an agent-specific control, and almost all of them sit outside the model.
ReadWhy Multi-Agent Systems Fail: How Errors Compound at Every Hand-Off
Multi-agent systems mostly fail at the seams, not inside one agent. Cemri et al.'s MAST taxonomy sorts 14 failure modes into system design issues, inter-agent misalignment and task verification. The controls belong at the hand-off: a structured contract, verification before an output becomes the next agent's input, roles enforced outside the prompt, a way to say "I don't know", and orchestrator-owned termination.
ReadRunaway AI Agent Costs: How Retry Loops, Fan-Out and Recursive Tool Calls Burn Budget
Agent costs run away through three mechanisms: retries stacked at several layers, fan-out that multiplies every step, and agents or tools that call each other in cycles, all made worse because each step re-sends a growing context. Cap them with six controls held outside the model: per-agent spend ceilings, step and depth limits, a retry budget, loop detection, a deadline per run, and alerts on spend rate.
ReadHow to Set a Latency Budget for AI Agent Guardrails
Set the budget per action class, not per guardrail. Start from what the action already costs and how long its caller will wait, measure at the tail rather than the average, and divide what is left between the checks that action needs. Irreversible actions get a blocking check and the most time; reversible ones can be checked asynchronously; read-only work should add close to nothing to the path.
ReadLLM Gateway vs AI Agent Governance: What the Gateway Sees, and What It Can't
An LLM gateway governs traffic to models: routing, provider keys, rate limits, spend, caching, fallback and basic filters. Keep it. It cannot tell whether an agent's action was permitted, because the action runs between the agent and the tool, the intent spans many requests, and your policy is not in its rules. That takes a governance layer that evaluates actions in context and keeps the evidence.
ReadAI Agent Identity: Why Shared Service Accounts Break Authorisation, and What Replaces Them
Give every agent its own identity, and when it acts for a person, put both in the credential: the user it acts for and the agent doing the acting. OAuth 2.0 Token Exchange (RFC 8693) defines exactly that delegation. Then make each credential short-lived, scoped to the task and bound to one audience, so a leaked or misused token is worth little and every action traces back to an agent and a person.
ReadIndirect Prompt Injection: Why the Attack Arrives in the Data, Not the Prompt
Indirect prompt injection hides instructions in content an agent retrieves (a web page, email or RAG chunk), so the attacker never talks to the model. Filters on the user's prompt miss it, because the payload is in the data. Six controls hold: mark untrusted content, keep it out of the control flow, cut privileges after reading it, confirm consequential actions, block exfiltration, and check actions before they run.
ReadMemory Poisoning in AI Agents: When What the Agent Remembers Becomes the Attack
Memory poisoning is injection that outlives the session: an attacker gets text into an agent's long-term memory, RAG store or shared context, and every later session that retrieves it inherits it. Five controls contain it: provenance on every memory write, scoping by user and task, expiry for unverified entries, review before high-impact writes persist, and snapshots you can roll back to.
ReadTool-Call Authorization for AI Agents: The Model Shouldn't Decide What It's Allowed to Do
An agent's permissions should be enforced outside the model, on every tool call, by the systems the call touches. That means four things: give each agent its own identity, scope its credentials to the task, check every call against policy before it runs, and treat the model's own judgement of what is allowed as advice, never authority.
ReadAI Red Teaming Is a Snapshot, Not a Control: What Agents Need Instead
A red-team report tells you what one team found against one version of your agent in one week. Agents change faster than that: models update, prompts and tools change, and an MCP tool's description can change after you approved it. Keep red teaming, but treat its findings as test cases: re-run them on every release as a gate, and pair them with checks that run on live traffic.
ReadHuman-in-the-Loop Isn't a Control Until You Measure It: Approval Fatigue in AI Agents
A human approval step only controls an agent if the person can see what they are approving, has the context and time to judge it, and is asked rarely enough to stay alert. Most agent deployments fail at least one of those. Measure approval rate, time to approve and catch rate on seeded test cases, and send people only the consequential actions an automated check could not settle.
ReadAI Agent Audit Trails Under EU AI Act Article 12: What to Log, and for How Long
Article 12 requires high-risk AI systems to log events automatically over their lifetime, so that their operation can be traced. Providers and deployers must keep those logs for at least six months (Articles 19 and 26). Since the Digital Omnibus, this applies from 2 December 2027 for Annex III systems. For an agent, that means one record per consequential action: who acted, on whose authority, what it did, what it relied on, what checked it, and what happened next.
ReadMCP Tool Poisoning: How a Tool Description Hijacks an AI Agent
A malicious MCP server can hide instructions in a tool's description, which the model reads and the user never sees. In Invariant Labs' demonstration, an innocent-looking "add" tool made an agent read the user's SSH key and send it out as a tool argument. The defences: pin and review tool descriptions, show full arguments before approval, scope each server's credentials, and check every tool call — the action and its arguments — before it runs.
ReadFail-Open vs Fail-Closed: What an AI Agent Guardrail Should Do When the Check Fails
Decide by the action, not by the guardrail. Irreversible, consequential actions fail closed: no verdict means the action waits or goes to a person. Reversible and advisory actions can fail open, as long as every unchecked action is recorded as unchecked. And whichever you choose, a missing verdict must never be stored as a pass.
ReadSLM-as-a-Judge vs LLM-as-a-Judge: Why the Judge in the Path Should Be Small
For offline evals, a frontier LLM judge is fine. For checking agent actions at runtime, use a small open-weight judge you host: the evidence stays in your network, cost stays flat as volume grows, verdicts are reproducible, and the judge isn't from the same model family as the agents it grades.
ReadWhy AI Agents Stall Before Production — and It Isn't the Model
Roughly 62% of enterprises are experimenting with agentic AI; about 14% have anything production-ready. The usual explanation for the gap is capability. Talk to the teams whose agents are stuck and a different picture emerges: the agents work. They stop at sign-off.
ReadWhen Your AI Agent Talks to Customers, You Own What It Says
A legal question that seemed theoretical has been settled in three jurisdictions, in the same direction: the agent's words are the company's words. Most of the failures that create that liability aren't detectable as bad text — which is why output filters miss them.
ReadAutonomous Trading Agents: What SEBI and FINRA Already Require
Trading is where agentic AI meets money most directly — and where regulators have moved furthest. SEBI already requires a kill switch and per-order audit trails, with the broker accountable. FINRA expects out-of-bounds agent actions to be blocked, not logged.
ReadAI Safety in Industrial Automation: Governing Agents the Way You Govern Operators
IEC 62443 was written for components whose behaviour can be specified, and an AI agent isn't one. But every control loop already contains a non-deterministic actor — the human operator — and the discipline that governs operators is exactly the right model for agents.
ReadRogue AI Agent Containment: Isolating One Agent Without Stopping the Fleet
Containment is usually discussed as a speed problem. That's the second question. The first is who holds the stop — because an agent can ignore a stop it controls, and a containment plan built on an instruction is a request, not a control.
ReadAgentic AI Risks and Controls: A Mapping for Security Teams Building Their First Framework
Risk taxonomies are abundant and interchangeable. The hard part is pairing each risk with a control that actually addresses it — and noticing that telemetry throughput is not a control. Seven risks, seven controls, and the separation principle underneath all of them.
ReadPrompt Injection Detection at the Proxy Layer: Why 20ms Latency Is the Real Constraint
Prompt-injection defences get evaluated on detection accuracy. The constraint that actually decides your architecture is latency — and it is two budgets, not one: a sub-20ms inline tier on ingress, and a judgment tier that cannot be done in 20ms by anything.
ReadEnterprise AI Guardrails That Actually Stop Agents, Not Just Filter Prompts
"Guardrails" covers two different products: one filters text, the other stops actions. A well-formed, polite instruction to approve a loan outside eligibility passes every content filter ever written — because nothing about that failure is legible as bad text.
ReadWhat an AI Agent Governance Platform Must Prove Before Compliance Signs Off
A checklist for the person who has to sign off before agents go live. Eight requirements that separate a governance platform from a reporting tool — plus the two questions that reveal whether a vendor is being honest about what they have actually built.
ReadLLM Hallucination Detection Isn't Enough — You Need a Judge, Not Just a Classifier
A detector answers "is this output likely false?" Your risk committee asks "was this action permitted, under which rule, and can you prove it in eighteen months?" Three courts have now attributed AI statements to the operator — and none of those rulings turned on detection accuracy.
ReadAgentic AI Security Risk: The Attack Surface Most Enterprises Haven't Modeled Yet
Enterprise threat models were written for software that does what it is told. Agents decide, hold credentials, and call tools — and NIST notes they are commonly treated as generic service accounts with no dedicated identity or accountability controls. Five attack patterns, and the control that addresses each.
ReadAI Agent Policy Enforcement: Inline Judging vs. Post-Hoc Review
Most AI agent policy enforcement happens after the fact, in logs a risk committee reads once the action has already settled. This piece contrasts that default with inline judgment — evaluating each consequential action against an organisation's own policy corpus before it executes, and naming the rule applied.
ReadWhy Multi-Agent Observability Tools Miss the Failures That Matter
Agent tracing solved opacity. It did not solve control. The failure that costs money is a consequential action produced by a defective reasoning chain — where every step looks valid in the log — and no trace can catch that, because a trace arrives after the fact, judges nothing, and holds no independent stop.
Read