Of course your AI can be prompt injected #
There’s a common slogan among security experts that states “People are the weakest link”. If even a single person can be tricked into believing a falsehood - such as having won a raffle or urgently needing to confirm their password - an entire network is at risk. While there’s an ongoing debate about the truth in this slogan, it translates well into AI systems.
With enough attempts, someone will find a prompt injection for your AI system and get it to believe in a falsehood and act accordingly. Screenshots are shared, replies pile up and pressure builds up to make sure the AI products cannot be prompt injected anymore.
While the request is understandable, it’s not helpful. Prompt injections are as real and as unavoidable as phishing and fraud. You can easily end up spending a quarter of your budget defending against the wrong threat.
This article supports a single argument from the offensive-security side of the fence: Prompt injection is real, it is largely unfixable, and it is still not the thing you should be losing sleep over. The thing you should be losing sleep over is what your model is wired to do when
You will not “fix” prompt injection #
The past decades have taught us how to deal with injection attacks in traditional systems. Be it SQL injections or HTML injections, modern framework have found ways to deal with this and the general principles are straightforward: Treat all user input as untrusted data streams which will not be evaluated. There is a strong separation of data and logic.
LLM based AI does not have this distinction. All inputs are part of a single token stream. System prompts, tool output, user input, everything is evaluated as one, unified token stream. And as a consequence, any part of the token stream could be read as instructions to follow. There is no templating framework which will solve this for you, it’s a hard restriction imposed by the way LLMs work.
Companies can try to introduce “prompt firewalls”, guardrails, AI based message classifiers, and delimiters to prevent prompt injections. They certainly help to raise the bar, but each of them has its imperfections and limitations. Countermeasures for risks need to be in proportion and while some protection are worthwhile investments, aiming for perfect protection is a costly mistake.
Motivated attackers will always find a way to bypass countermeasures and for attacks against LLMs, these approaches are well documented in academic papers and applied in open source, offensive tooling.
If the security of your AI system depends on the model never being tricked, you’re building security on hope.
What actually matters: Blast Radius #
So if the perimeter can be breached, how then should one secure the system? For this, let’s consider two different AI systems.
System 1 is a chatbot. If it’s jailbroken, it might say embarrasing phrases and it can be embarrassing, but it’s limited.
System 2 is an agent capable of performing actions autonomously. It has access to critical data, systems and secrets. If this is broken, attackers can act on its behalf.
Even with the same underlying vulnerability of prompt injections, the damage differs dramatically.
This class of risks is old, Norman Hardy coined the attack “Confused Deputy” back in 1988. It’s excessive privilege and inference-based security, it’s shipping with administrative privileges for now and never getting around to fixing it, it’s not a new problem. The good news is that this is therefore a well understood problem with tried and tested countermeasures. While prompt injections are AI specific vulnerabilities, excessive permissions are a well understood challenge - and it’s where the money is. Exactly this is the change in discussions we need to build safer AI systems.
The shift we need #
Once we shift from trying to create un-jailbreak-able models towards hardened systems with a controlled blast radius, we’re on the right track. The core risks introduced by AI agents are manageable. They are boring. And this might be its flaw! AI is exciting, managing permissions is not. But it’s effective and you’ve already got all the tools you need in your security team.
While every company and AI setup is different, I do have a few guidelines I would recommend to everyone responsible for an AI system with agentic capabilities.
- Treat every model output as untrusted input to the next system. The moment model output selects or parameterizes an action to be executed, a deterministic layer (not AI) has to authorize it.
- Scope tools to least privilege, per user. A tool call should run with the permissions of the human on the other end, not a god-mode service account. This must be deterministic and not decided on the fly by another AI system.
- Design systems where prompt injection incidents become boring. Don’t model your threats around jailbreak-resistance but instead by its blast radius. If not much can happen when your AI is jailbroken, you’ve built the right thing.
The simple truth to take away #
Let’s start asking what happens when an AI system is compromised instead of asking if it will be. It will happen eventually.
Guardrails are great but if you spend more time architecting the next layer of exfiltration protection, you’re wasting time and money on the wrong thing. Get basic protections in place and then start probing about the blast radius of that AI system.
For the forseeable future, prompt injections will remain an unsolved issue. Once it is solved, modern framework will adopt these solutions just like we have done with SQL injection and HTML injection. We’re not quite there yet.
But what we can do today is shift our focus on how our existing management systems and requirements can include agentic AI systems. While the models are new, the security challenges are not.
Let’s chat! #
If you would like to discuss opinions or need your system to be evaluated, let’s get in contact and have a chat! You will most easily reach me via LinkedIn.