What changes with AI
The biggest upside. The biggest risk.
They are the same thing: autonomy. Software that acts on its own inside your systems is exactly why AI is worth doing, and exactly what makes it dangerous. So we name the failure modes plainly, because a vendor who cannot describe the risk cannot design against it.
What we design against
Five failure modes, following the OWASP Top 10 for LLM Applications, which keeps prompt injection at the top of the list.
Prompt injection
Anything the system reads can try to give it orders.
Instructions and data share one channel, and there is no parameterised query for language models. A supplier email or a PDF can carry instructions, so the defence has to be architectural: retrieved content never widens what the system may do.
Excessive agency
Most agents are given far more power than their job needs.
A system that can read what it needs, but also write, delete and send, has a surface far larger than its purpose. We scope authority per operation.
Sensitive information disclosure
Data escapes through the side doors, not the front one.
Responses, logs, error messages and downstream integrations. The controls are what reaches the context, who may ask, and what the system may send outward.
Unreliable output treated as fact
A confident wrong answer is only a risk once something acts on it.
So consequential output carries its sources, and a person reviews it before anything hard to reverse is committed.
Identity and authorisation drift
When a system acts for everyone, whose permissions apply?
Authorisation is enforced at the system of record for the acting identity, never approximated inside a prompt.
The thing to understand
The capability and the exposure are the same feature.
Autonomy is why AI is worth doing and why it is dangerous. You cannot keep one and remove the other. You can only decide how far it reaches, and who checks it.
Scope beats restriction
Useful systems need real access. The question is not whether to grant it, but how narrowly and to which records.
Assume the input is hostile
Anything the system reads may try to redirect it. Retrieved content can inform a decision; it must never widen authority.
Blast radius, not probability
Design for what a mistake costs, not for how unlikely it seems. The unlikely case is the one that reaches an auditor.
Why this is urgent now
Autonomy raises the cost of a mistake
Drafting text is low stakes. Releasing an order is not.
Once software acts in a system of record, errors carry commercial and regulatory consequences.
Adoption is outrunning governance
AI is being deployed faster than anyone is overseeing it.
That gap is where incidents happen, and why the security review is where AI initiatives most often stop.
Regulation already assumes AI is in use
The transparency obligations are not coming. They are here.
Oversight, documentation and evidence are far cheaper to design in than to add once a reviewer asks for them.
Bring us your security review.
Your questionnaire, your data-protection requirements and the systems in scope. We would rather work through them now than at the end.