AI Security: The Permissions Nobody Audited

Almost every conversation I have about AI security opens at the model, and I have come to think that is the wrong end of the system to start from.

The question usually arrives phrased as trust. Can it be tricked? Is one model safer than another? Which vendor should we be on?

Those are answerable, and the answer is unhelpful in a very specific way: yes, it can be tricked, all of them can, and the one you migrate to next year will be too. Meta said it plainly when it published its Agents Rule of Two in October 2025 — prompt injection is "a fundamental, unsolved weakness in all LLMs."

If being fooled is a given rather than a risk, then the interesting question moves. It stops being will this happen and becomes what is the system holding when it does.

That second question has nothing to do with the model. It is a question about permissions — and permissions are the part a software company decided entirely by itself.

The defence you are counting on has probably been broken

The reason I do not think this gets solved by waiting is that the defences keep failing in the same embarrassing way.

A team of researchers including people from Google DeepMind, OpenAI and Anthropic took twelve recently published defences against jailbreaks and prompt injection and attacked them properly — tuning and scaling gradient descent, reinforcement learning, random search and human-guided exploration against each one. They got attack success rates above 90% for most of them. The line in the abstract I keep rereading is the next one: the majority of those defences had originally reported near-zero attack success rates.

Nobody was lying. They were measuring against a static attacker. The paper's title is the whole lesson — The Attacker Moves Second. A defence evaluated against yesterday's attack reports a number that describes yesterday.

I take two things from that. The first is that a filter, a classifier or a clever system prompt is worth having and worth nothing to rely on. The second is that any security posture whose load-bearing assumption is "the model will refuse" is holding weight it cannot carry.

The failures being catalogued are not model failures

What convinces me more than the research is the shape of what has actually gone wrong in production.

OWASP's GenAI Exploit Round-up for Q1 2026 catalogues eight major GenAI and agentic exploits between January and April. Three were excessive agency or autonomy failures. Three were supply chain. One was prompt injection. One combined excessive agency with data disclosure.

Read the tally rather than the incidents and something stands out. The category everyone worries about accounted for one of eight. The category nobody puts on a roadmap — an agent permitted to do more than the task required — accounted for three, and contributed to a fourth.

The report is direct about where these come from: most of these events "are not yet mapped to traditional CVE identifiers," arising instead from misconfiguration, design flaws, supply-chain weaknesses and prompt injection. Only one of the eight got a CVE at all.

That is worth sitting with if you run a security programme. A CVE is how vulnerabilities enter the pipeline — scanners read advisories, tickets get filed, the process works. A design flaw in what a feature is allowed to reach produces no advisory. Nothing arrives. The existing machinery is not failing to catch these; it was never pointed at them.

OWASP's own Top 10 for LLM Applications already names the two that matter most here — LLM01 Prompt Injection and LLM06 Excessive Agency. The first is the one that gets discussed. The second is the one that decides the damage.

Your retriever is an access-control system nobody designed as one

Here is where I think SaaS has a harder problem than most, and a quieter one.

A retriever ranks by relevance. That is the entire job — find the chunks closest to the question. Relevance is not authorisation, and a vector index has no opinion about who is asking. A paper presented at the ACM Conference on AI and Agentic Systems in May 2026 gives this a name I find useful: the relevance-authorization gap. Its evaluation is blunt about the consequence — with retrieval left ungated, cross-tenant probes came back holding unauthorised data between 62% and 80% of the time, depending on how the system was orchestrated.

Nothing was hacked in those runs. The system worked as designed. It was asked for the most relevant chunk and it returned the most relevant chunk, which happened to belong to somebody else.

The structural reason this bites software companies specifically is that the permissions were never in the document to begin with. They lived in the source system — the CRM, the ticket queue, the file store, each with its own rules about who sees what. Then an ingestion job chunked the text and wrote embeddings, and the rules did not travel with the chunk. One index now serves every tenant, and the boundary that used to be enforced by the source system is enforced by whatever filter somebody remembered to add at query time.

Tool access has the same shape and worse consequences. AgentRedBench, a benchmark published in mid-2026, tested exactly this: 215 underspecified-authorization scenarios across 24 enterprise integrations, run against an eight-model panel from Anthropic, OpenAI and Google. Attack success ran from 32% to 81% with no guard in place. Underspecified authorisation is a precise phrase for a familiar situation — nobody wrote down what the agent was not allowed to do, so the answer got decided at runtime by a model reading an attacker's text.

What a real one looks like

The clearest illustration I know is EchoLeak, catalogued as CVE-2025-32711 in Microsoft 365 Copilot and written up in detail by Pavan Reddy and Aditya Sanjay Gujral.

An attacker sent an email. That was the whole user interaction — the recipient did not have to open it, click anything or ask Copilot about it. Hidden instructions in the message evaded Microsoft's cross-prompt-injection classifier, link redaction was circumvented with reference-style Markdown, an auto-fetched image carried the payload out, and a Teams proxy that the content security policy already allowed provided the exit. Internal data left the tenant.

What I find instructive is that no individual piece of that is a serious flaw. A classifier with a gap. A Markdown renderer doing its job. Images that load. A proxy on an allow-list for good reasons. The authors describe the result as privilege escalation across LLM trust boundaries — and every step in the chain was a permission somebody granted deliberately, for a sensible reason, without knowing what it would sit next to.

That is the failure mode I would design against. Not a single catastrophic hole, but a sequence of individually reasonable grants that only becomes an exfiltration path once a model is placed in the middle of them and something it reads turns hostile.

The five questions I want answered in writing

When I want to know whether an AI feature has been secured rather than reviewed, these are the five things I look for:

• What does it read, and which of those sources can a stranger write to? Every input the system ingests, sorted by who controls the text. Email, uploaded files, web pages, ticket bodies and CRM notes are all attacker-writable in most products.
• What can it reach, and what does the task actually need? The gap between the access granted and the access required is the blast radius, and it is usually nobody's job to notice it widening.
• Does authorisation happen at retrieval, or was it assumed at ingestion? Permissions change after the chunk is written. A filter applied at query time is the only one that reflects today.
• Which actions cannot happen without a person? Named in advance, for the actions with consequences — not added after an incident makes the list obvious.
• What does each finding actually reach? A prompt injection into a read-only summariser and one into an agent that can send mail are not the same severity, however similarly they are labelled.

None of those five require agreement about which model is safest. All five are answerable from a company's own architecture — which also means nobody outside it can answer them on its behalf.

The variable you control

The uncomfortable part of AI security is that the headline risk is not yours to fix. Prompt injection is being worked on by every frontier lab, and it is still, as Meta put it, unsolved.

The part I find genuinely encouraging is that it was never the variable that decided the outcome. What decides the outcome is reach — what the system can read, what it can retrieve, what it can act on, and where a person still stands between an instruction and its consequence. All of that is a set of decisions a SaaS team makes, and can therefore make on purpose.

The model being fooled is the weather. What it is holding at that moment is architecture.