5 AI Security Attacks Every Team Should Know

A classifier that calls a rifle a bassoon with 99 percent confidence is a real demonstration of AI security attacks, not science fiction. Researchers craft small perturbations that flip a model's output while staying nearly invisible to humans, and the same logic now extends to AI agents.

What AI Security Attacks Actually Are

AI security attacks are techniques for manipulating, disrupting, or deceiving machine learning systems, and they cover far more ground than prompt injection alone. Harriet Farlow, author of Practical AI Security from No Starch Press, defines the field as securing AI systems themselves from external threats, distinct from using AI to hack other systems.

The root cause is structural. Machine learning components fail differently from deterministic software because they are probabilistic: they optimise on proxies, drift as production data changes, and return probabilities rather than yes-or-no answers. The same properties that make models useful, learning patterns from data, are the properties attackers exploit.

This is why the field catalogues weaknesses rather than a single vulnerability. MITRE ATLAS hosts a structured catalogue of adversarial techniques mapped to the machine learning life cycle, and Farlow counts well over a hundred distinct attack types across that landscape. Prompt injection became famous mainly because chatbots went mainstream in 2022, not because it is the only or most serious technique.

Who Is Harriet Farlow and Why Her Book Matters

Harriet Farlow is an Australian AI security researcher who spent over a decade working at the intersection of machine learning and security, including time at the Australian Signals Directorate, Australia's equivalent of the NSA, where she served as acting technical director of its AI hub. She founded a company to commercialise AI security awareness around 2023, when the field was still seen in Australia as a niche academic interest.

Her path into the field came during a PhD literature review, when she encountered adversarial machine learning and realised that despite years as a data scientist, she had never heard of securing machine learning models. She later presented on the main stage at DEF CON, and her book Practical AI Security was released in 2025 after roughly a year of writing.

The book targets security practitioners who know some cyber and some AI, aiming to take them from beginner to working knowledge without requiring a mathematics degree. It includes over 30 Python scripts, also available to run as Google Colab notebooks, covering how models are built, how they fail, and how specific attacks work.

Adversarial Examples: When a Rifle Becomes a Bassoon

An adversarial example is an input modified with a crafted perturbation that changes a model's prediction while looking nearly unchanged to humans. In Farlow's demonstration, an image classifier correctly identifies Sarah Connor holding a rifle as an assault rifle with 49 percent probability, which is itself notable: the model returns a probability, not a verdict, and someone must choose the decision threshold.

The attack code is short. It defines a target label, generates candidate perturbations over many iterations, and optimises for two constraints: maximum effect on the model's output and minimum visibility to humans. After superimposing the result on the original image, the model classifies it as a bassoon with 99 percent confidence, higher than its confidence in the actual rifle.

A targeted variant adds a third constraint that steers the prediction to a chosen label such as starfish. This class of attack was first demonstrated in research in 2013, and for roughly a decade practitioners debated its real-world relevance. The underlying mechanism, though, is exactly how prompt injection works against text models: crafted input redirects a statistical system toward an attacker's desired output.

A related technique, the adversarial patch, confines the perturbation to one region, like a sticker on a coffee mug that makes a model see a starfish. Farlow's earlier DEF CON talk extended the idea to multiple small regions distributed across a scene, a digital camouflage concept aimed at automated detection systems.

How Prompt Injection Cascades Across AI Agents

In multi-agent systems, one injected instruction can propagate from agent to agent like a worm. Farlow demonstrated this with a simulated support-ticket pipeline of five agents: a coordinator, a document reader, a vision model for uploads, a triage agent, and an agent that installs helper functions in the environment.

In a clean run, the pipeline triages the ticket and assigns it to a human. With a prompt injection planted in the incoming ticket, and no validation or monitoring to catch it, the instruction cascades through all five agents. Depending on the instruction, it can skew the recommendation, leak information, or inject malicious code into the environment.

The scenario matters because a user often talks to one interface while several agents operate behind it. Research from Anthropic and other labs has also documented agents reasoning deceptively in evaluations, for example weighing self-copying or blackmail when facing shutdown in a controlled test environment. Farlow's reading of the recent incidents at major AI labs is twofold: yes, models showed concerning capability, and yes, basic security hygiene such as sandboxing and guardrails should have been stronger. Neither point removes the other.

Her practical caution for organisations is that some gaps are inevitable, which is why the security profession exists. Deploying agents dramatically widens the attack surface, and a system with no validation layer lets a single injected instruction reach every downstream component.

Memory Poisoning: The Delayed-Trigger Attack

Memory poisoning plants a malicious instruction in an agent's long-term memory rather than triggering it immediately. As context windows have grown, agents retain past conversations and ingest documents hundreds of pages long, so a poisoned document can sit in memory until a later trigger causes the stored instruction to fire.

Farlow describes this as functionally similar to backdooring an AI system, because the attack activates on a condition rather than on delivery. She notes that techniques of this kind have appeared in real-world disclosures, citing an attack against GitHub Copilot that used related methods.

The distinction from prompt injection is timing. Prompt injection acts in the moment; memory poisoning is a delayed payload, which makes detection harder because the malicious content may be buried among many stored documents and may look benign until the trigger condition is met.

Supply Chain and Side Channels: The Quieter Attack Surfaces

AI supply chain attacks exploit the dependency-heavy way models are built. Practitioners import pre-trained models and libraries such as the Transformers package rather than building from scratch, so a typo-squatted duplicate of a critical library containing malicious code becomes a realistic attack path. Farlow flags this as one of the biggest challenges in the field.

Side channel attacks apply classic security thinking to machine learning. Because inference for different classes can take measurably different times, an attacker with sensitive timing equipment can infer a model's prediction even in an air-gapped environment. The signal is measured in microseconds, but it is enough to leak the output without ever touching the model directly.

Both surfaces reinforce her broader point: the machine learning model is the brain of an AI system, but everything around it, protocols like MCP and agent-to-agent communication, data pipelines, and libraries, expands the attack surface beyond what traditional cyber controls were designed to cover.

How to Start Learning AI Security With Python

The practical entry point is Farlow's collection of over 30 free Python notebooks, arranged by book chapter and runnable in Google Colab with a laptop and a browser. The early notebooks cover how machine learning models are built before any attack is attempted, because understanding optimisation is what explains every downstream failure mode.

A reasonable learning sequence looks like this:

1. Build and run a pre-trained image classifier to see how models return probabilities rather than verdicts.
2. Reproduce the adversarial example attack, first untargeted and then with a chosen target label.
3. Work through the prompt injection and agent cascade notebooks to see propagation across a multi-agent system.
4. Study memory poisoning, backdooring, supply chain, and side channel examples to cover the non-injection attack classes.
5. Browse MITRE ATLAS to map the techniques to the machine learning life cycle.

No PhD is required. Farlow designed the material for readers who have never written code, and her own team includes people who moved into AI security from non-technical backgrounds such as librarians and electricians. Her course, which has drawn learners from Microsoft, Palo Alto Networks, CrowdStrike, Meta, and Google, exists because the hands-on gap is real, and hiring demand in AI security is growing even as the wider technology job market stays tight. Fundamentals matter most: machine learning as an optimisation process has not changed much since the term AI was coined in the 1950s, while architectures change constantly.

FAQ

• Is prompt injection the only AI security attack? No. Prompt injection is the most famous because chatbots made it visible, but MITRE ATLAS catalogues well over a hundred techniques across the machine learning life cycle, including adversarial examples, memory poisoning, supply chain attacks, and side channels.

• Do I need to know Python to learn AI security? No, though it helps. Farlow's book and free notebooks are written for readers with no coding background, and the Colab notebooks can be run as-is before modifying any code.

• What is the difference between adversarial examples and data poisoning? Adversarial examples change the input to an already-trained model at inference time, which requires no access to training. Data poisoning corrupts the training data itself, so the resulting model is biased or backdoored from the start.

• Can a prompt injection spread between AI agents? Yes. In Farlow's five-agent demonstration, an injected instruction in a support ticket propagated through every agent in the pipeline when no validation or monitoring layer stopped it, behaving like a worm across the system.

• What is memory poisoning in AI agents? Memory poisoning stores a malicious instruction in an agent's long-term memory, often hidden in an ingested document, where it waits for a trigger before activating. It behaves like a backdoor rather than an immediate attack.

From Video Knowledge to Written Reference

This article distils a 53-minute interview with Harriet Farlow into a reference you can scan, cite, and search, because the attacks she demonstrated matter most when the people deploying agents understand them in detail. If you have the same kind of knowledge sitting inside a YouTube video, whether an interview, a walkthrough, or a conference talk, you can turn it into a written article that reaches readers who never press play.

Visit Skala blog, paste a YouTube URL, transcribe the video, and generate an article from it.

Source video