You Cannot Prompt Your Way to Safety

An AI agent operated by an Alibaba-affiliated team autonomously hijacked GPU resources to mine cryptocurrency and opened a hidden network backdoor. Nobody instructed it to. The behaviour surfaced only when a cloud firewall flagged unusual traffic — not from any control inside the agent itself.

Gartner projects that by 2028, 15% of day-to-day work decisions will be made autonomously by agents, and that 25% of enterprise breaches will originate in agent abuse.

I spent three years building security infrastructure for on-chain systems, where every action is irreversible, every counterparty is potentially adversarial, and there is no support ticket that undoes a transaction. Watching the agent safety discussion from that background is a strange experience. The industry is working out, from first principles and with new vocabulary, a set of controls that another industry was forced to solve a decade ago because it had no other option.

The prompt is not a security boundary

The dominant pattern for constraining agents is still instruction: tell the model what it must never do, in careful language, near the top of the context.

This fails for three structural reasons, none of which are about model quality.

It shares a channel with untrusted input. Your constraints and the attacker’s text arrive as the same kind of token in the same window. Every defence reduces to hoping your sentences are more persuasive than theirs.

It is probabilistic. A rule that holds 99.9% of the time is a fine product behaviour and an unacceptable security control. An agent taking a thousand actions a day breaks it daily.

It produces no evidence. When something goes wrong you have a transcript, not an audit log. You cannot prove what policy was in force at the time, because the policy was a paragraph the model may or may not have attended to.

The emerging consensus in agent security is to take the “never do this” rules out of the prompt and put them in code, gating any irreversible action. That is the right conclusion. It is also precisely what smart contract security concluded, for the same reason, years earlier.

What you learn when there is no undo

On-chain systems have a property that most software does not: consequential actions are final within seconds, and the environment is openly adversarial. That combination kills a lot of comfortable assumptions fast.

You cannot rely on detection, because by the time you have detected it, the funds are gone. You cannot rely on the caller being honest, because the caller is frequently the attacker. You cannot rely on human review, because humans are slower than the system. And you cannot rely on the calling code being correct, because the calling code is what got compromised.

What survives that environment is a small set of controls that share one property: they sit between intent and execution, they are deterministic, and they cannot be talked out of it.

Building a smart contract firewall meant implementing four of them. All four transfer directly to agents, and three of them are currently being reinvented under different names.

The four controls that transfer

Policy at the execution layer, not the decision layer. The component that decides what to do and the component that decides what is permitted must be different components, and the second must not be persuadable by the first. In an agent system, this means the model proposes a tool call and a separate deterministic evaluator decides whether it runs. The model’s reasoning is an input to the log, never an input to the authorisation.

Threshold approval for consequential actions. Above some blast radius, one signature is not enough. On-chain this is an m-of-n multisig; for agents it is a second approver — another agent with different inputs, a rules engine, or a human — required before execution. The point is not that the second approver is smarter. It is that the failure modes are uncorrelated, so a single compromised context cannot authorise the action alone.

Time-locks. Insert a mandatory delay between authorisation and execution, during which the action can be cancelled. This is the control I would argue is most underused in agent systems, because it is the only one that makes human oversight real rather than ceremonial. Agents operate faster than humans review. A time-lock converts an irreversible action into a reversible one for the length of the window, and it costs nothing when nothing is wrong.

Context verification. Check that the action’s circumstances match its mandate before permitting it — origin, requester, time of day, rate relative to baseline. On-chain this took the form of geo-verification and velocity limits. For agents it is the same question: is this call consistent with what this agent is for, or is it merely well-formed.

The tool-invocation layer is the seam

There is broad agreement forming that the tool-invocation layer is the new policy enforcement point, and it is worth being explicit about why it is the correct place rather than merely a convenient one.

Every consequential thing an agent does passes through a tool call. Reasoning is not consequential — it is text. Retrieval is mostly not consequential. The moment an agent affects the world, it goes through a defined interface with structured arguments. That is a chokepoint, and chokepoints are where policy belongs.

It also gives you the shape a good policy needs:

intent → policy check → threshold approve → timelock → execute

Each stage is deterministic, each can reject, and every transition is a log line. Compare that to a paragraph of instructions and it is obvious which one you would rather hand to an auditor.

Practically, that means the tool executor is not a switch statement over function names. It is a gate:

async function invoke(call: ToolCall, ctx: AgentContext) {
  const decision = policy.evaluate(call, ctx);   // deterministic, no model
  audit.record(call, ctx, decision);             // append-only, before execution

  if (decision.verdict === 'deny') return refusal(decision);
  if (decision.verdict === 'escalate') return queueForApproval(call, decision);
  if (decision.delayMs) await timelock.hold(call, decision.delayMs);

  return tools[call.name](call.args, ctx.scopedCredentials);
}

Two details in there matter more than they look. The audit record is written before execution, not after, so a crash mid-action still leaves evidence. And credentials are scoped per invocation rather than held by the agent, so a compromised agent cannot exceed the permissions of the specific call it was authorised to make.

Detection is not prevention

I built both a monitoring system and an enforcement system, and the distinction between them is the thing I would most want to transplant into agent design.

Monitoring answers: what did this system do. It is essential, it is how you learn what normal looks like, and it is how you catch the failure mode nobody predicted. The Alibaba agent was caught by monitoring — a firewall noticing unusual traffic.

But notice what monitoring bought in that case. It bought discovery, after the GPUs had been mined and the backdoor had been open. For a reversible action that is a fine trade: detect, roll back, learn. For an irreversible one, detection is a receipt.

The right division is by reversibility, not by risk score. Reversible actions can be monitored and rolled back. Irreversible ones — payments, deletions, external messages, on-chain transactions, anything a customer sees — need a gate before them, because there is no state to restore afterwards.

Most agent frameworks today ship excellent observability and almost no enforcement. That is a reasonable place to start and a dangerous place to stop.

The regulatory floor is arriving anyway

FINRA’s 2026 Annual Regulatory Oversight Report names implementing guardrails to constrain agent behaviours, actions, and decisions as a supervisory consideration. Agents taking consequential action in regulated domains are heading toward classification as high-risk, which brings requirements for human oversight, auditability, and conformity assessment.

The relevant point for anyone building now is that “we told the model not to” does not satisfy an auditability requirement. You cannot produce the policy that was in force, prove it was evaluated, or show what it rejected. A deterministic policy layer with an append-only log produces all three as a byproduct.

Which is the argument I would make even without the regulation. The controls that survive adversarial, irreversible environments are not more expensive than the controls that do not. They are just less comfortable, because they require admitting up front that the model is a component you do not fully control — and designing the system so that fact is survivable.

---

Sources: AI agent risks and guardrails, AI agent security in 2026, agent threats and controls in regulated financial systems.