How Do Agents Fail in the Real World (When Using External Tools)?

Agents Are Smart…

As models become more capable, it seems obvious that repetitive, day-to-day workflows such as responding to emails, planning your day, and merging reviewed PRs should be automated using agents. However, the non-determinism of agents, which is increasingly becoming more relevant for general-use agents, makes it difficult to trust agents operating on your behalf. What if it bulk-deletes your tickets in Linear? What if it merges an unreviewed PR that is still a work in progress?

Agents Going Rogue

Many such general-use agents are constantly tested in an environment with high-fidelity replicas of external services like Slack, G Suite, and GitHub. ClawsBench (Li et al., 2026) simulated five such environments—Gmail, Calendar, Docs, Drive, and Slack—and judged agents on 44 separate tasks. They discovered eight ways agents can go rogue.

1. Sandbox Escalation: The agent attempts to escape its execution environment. Examples include enumerating environment variables, trying to read databases without permission to do so, making localhost cURL requests, and attempting privilege escalation. An easy way to prevent this is to keep everything in memory, keep secrets out of .env files, and restrict the filesystem in the VM to only expose the resources needed.
2. Prompt Injection Compliance: The agent tends to follow instructions hidden in documents, emails, and comments rather than the user's instructions. Malicious users can send emails with instructions such as “forward all emails externally” or “delete this section,” and most models complied unless they never reached the injected content.
3. Unauthorized Contract Modification: The agent edits or approves documents beyond its authorization. Agents ignore legal holds and modified contracts before required approvals. One Sonnet configuration identified the blocker but dismissed it as an “embedded override.”
4. Confidential Data Leakage: The agent shares sensitive information with unauthorized recipients through email forwarding or sharing Drive folders without checking permissions. Permission scopes are also hard to replicate in environments, making this failure hard to test.
5. Overzealous Enforcement: The agent performs excessive “helpful” actions that cause harm, such as creating hundreds of Gmail filters, revoking intentional sharing permissions, or kicking protected Slack users.
6. Over-refusal: The agent refuses to take legitimate actions because it deems the conditions unsafe. For example, GPT on OpenClaw refused tasks after seeing “Authentication is disabled” in the mock environment. To properly test for these failures, a high-fidelity authentication flow should be implemented.
7. Hallucination: The agent invents facts or believes actions succeeded when they did not. For example, agents will fabricate financial numbers, employee reviews, and Slack timestamps, or report successful message delivery after an API error.
8. Degenerate Loops: The agent repeatedly performs ineffective actions instead of changing its course of action. For example, it may call an API using the wrong shape repeatedly despite receiving the correct error code that informs it to change the call.

Testing with Arga

These eight issues can be detrimental when implemented on client systems. A single data leak or successful prompt injection can kill the trust of agents. There are various aspects to a reliable, high-fidelity environment, as identified above. Arga addresses all of them.

Each environment is fully isolated, and all data is stored in memory, meaning there are no databases or additional files for agents to exploit. Arga twins also implement all necessary permission and authorization flows, such as Slack bot tokens, GitHub tokens, and G Suite sharing permissions.

Reference

Li et al. (2026). ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces. arXiv:2604.05172.