← Back to posts

Agent Security Has to Protect the Whole Trajectory

Black Hat's AI program and new research converge on the same warning: a sequence of individually permitted actions can still violate a system's safety rules.

Black Hat's AI Summit centered what agents do, what they can access, and which tools they invoke. A paper released the same week gives the problem a useful name: moving from per-action checks to trajectory assurance. Traditional authorization can judge whether one action is allowed. It may miss what a sequence of allowed actions produces together.

Consider a procurement agent. It is allowed to read vendor files, create a comparison, and email a manager. Each action is legitimate. If it extracts a confidential reserve price and later includes that number in an email to a vendor, the system has leaked data without violating any isolated tool permission.

Long-horizon planning expands that gap. An attacker does not need to trigger an obviously dangerous call immediately. Malicious instructions hidden in a page, email, retrieved document, memory, or tool result can first alter the plan, collect useful context, and wait for a later opportunity. A parameter scan at the moment of execution may arrive too late.

Trajectory assurance treats business rules as invariants that persist through the task: one customer's data must not shape another customer's output; an untrusted page cannot expand authority; an irreversible payment requires finance approval. The system checks these constraints as state changes, not only when a button is pressed.

That requires provenance. Important data needs a structured record of where it came from, how it was transformed, and which destination received it. A chat transcript is not data lineage, and a final action log cannot recover decisions that disappeared between a plan and a tool result.

Risk must accumulate as well. Reading a public page is low risk. Reading a private contract is higher risk. Combining the two and sending the output to an external domain changes the risk abruptly. Authorization has to consider the data and actions already held by the task, not treat every call as a fresh event.

Recovery is part of security. A binary allow-or-deny gate often pushes an agent into repeated retries or forces the user to restart. A better control can suggest a safe continuation: remove sensitive fields, switch to a read-only tool, request a named approval, or re-plan from the most recent valid checkpoint.

Approval interfaces should translate technical access into business consequences. 'Allow send_email' is weak. 'Send a summary containing customer contract prices to an external domain' gives a person the object, data, irreversibility, and risk needed for a real decision.

Multi-agent systems stretch the trajectory further. A researcher gathers sources, a writer creates content, and a browser agent publishes it. Scope and data labels must survive every delegation. Local compliance by each agent does not guarantee global compliance if context loses its constraints at the handoff.

The practical test is to select five high-risk outcomes and replay them backward. Which data entered, which agents handled it, where authority changed, which evidence justified continuation, and what could be rolled back? If the team must guess, it does not yet have observability. Production agent security has to understand the complete task, not merely guard each click.

— End —