“The AI mannequin powering the agent should be handled as an untrusted element,” the researchers wrote within the paper, warning that “semantic guardrails” and prompt-level defenses alone can’t reliably safe programs as soon as brokers achieve entry to enterprise instruments, reminiscence, APIs, browsers, and execution environments.
The authors drew the comparability to working programs. “Just like how an working system treats a course of as untrusted, we take the stance that the mannequin powering the agent needs to be handled as untrusted and safety properties needs to be expressed and enforced outdoors, on the stage of the surrounding system,” they wrote.
The paper was written by researchers at Google, the College of California, San Diego, the College of Wisconsin-Madison, and different establishments, together with Mihai Christodorescu, Earlence Fernandes, and Somesh Jha.
5 rules from programs safety
The authors distilled 5 rules from a long time of programs safety analysis that they stated agentic programs ought to observe: least privilege, tamper resistance of the trusted computing base, full mediation, safe data circulation, and accounting for the human as a weak hyperlink.
