AI Agents Can Code 10,000 Lines of Hacking Tools In Seconds - Dr. Ilia Shumailov (ex-GDM)

What It Covers

Dr. Ilia Shumailov, former Google DeepMind ML security researcher, examines why AI agents represent an unprecedented security threat, how prompt injection attacks defeat all current defenses, why supply chain vulnerabilities in ML libraries expose millions of devices, and how a new system called CAML enforces data flow policies to protect sensitive information from agentic systems.

Key Questions Answered

• •AI Agents as Threat Actors: Agents operate 24/7, touch every network endpoint simultaneously, and can generate 10,000 lines of functional hacking tools in seconds by reconstructing them from training data. No human insider threat operates this way. Existing enterprise security models assume human-speed, human-rational adversaries — assumptions that completely break down when agents are introduced into corporate infrastructure.
• •CAML System for Data Protection: Shumailov's CAML framework rewrites user queries into Python programs with explicit data flow graphs, then enforces policies via an interpreter — not a model. Example policy: passport numbers only flow to tools whose domain contains ".gov.uk." This approach separates sensitive data from model inference entirely, preventing prompt injection from ever accessing private variables.
• •Prompt Injection Defeats All Current Defenses: Research on Gemini showed that malicious content embedded in emails reliably redirected agent behavior away from user tasks. Every academic defense technique tested, including those implemented by security startups, failed. Shumailov found near-universal methods to produce adversarial emails that override agent instructions regardless of which defensive prompt engineering approach was applied.
• •ML Supply Chain Vulnerabilities: Hugging Face's `trust_remote_code` flag loads and executes external code at model load time — structurally identical to the Log4j vulnerability that compromised hundreds of millions of devices. PyTorch's nightly build was compromised via a malicious unregistered package that received thousands of downloads. Running local models outside sandboxed environments exposes the host machine to full remote code execution.
• •Security vs. Safety Distinction: Security addresses worst-case performance against active adversaries; safety addresses average-case system behavior. This distinction matters practically: jailbreaks require adversarial search, making them a security problem. Shumailov argues AI safety researchers conflated the two fields prematurely, importing adversarial threat modeling into domains — like standard model reliability — where it does not belong and creates analytical confusion.
• •ML Models as Trusted Third Parties: Shumailov proposes that verified ML inference can replace expensive cryptographic protocols for certain private computations. Using the Yao Millionaire Problem as an example, two parties agree on a model, a prompt, and constrained outputs, then run inference on a platform providing integrity verification. This creates a new trust primitive distinct from zero-knowledge proofs, MPC protocols, or trusted execution environments.