Threat modelling an LLM feature without boiling the ocean

Most requests to “review the AI risk” arrive with the system already half-built and a go-live date already promised. The assessment that helps is not a taxonomy of everything that could go wrong with large language models. It is a short, specific answer to what this feature, in this architecture, can be made to do that it should not.

That answer takes about half a day if you work it in a fixed order.

1. Draw what crosses the boundary, not what the model is

Start with one diagram, and put on it only the flows that cross a trust boundary:

• what reaches the prompt, and who controls each part of it: system instructions, retrieved documents, user input, tool output, conversation history
• what the model can cause to happen: tools, function calls, downstream APIs, database writes, outbound email, code execution
• what leaves: responses, logs, telemetry, anything sent to a model provider
• whose identity each of those actions runs as

The last point is the one most often left off, and it is usually where the finding is. A retrieval assistant that queries a document store using a service account with tenant-wide read access has an authorisation problem that no amount of prompt hardening will fix.

2. Work four classes, in this order

The published catalogues, the OWASP Top 10 for LLM applications and MITRE ATLAS, are useful as a checklist at the end. They are a poor place to start, because they invite a sweep through categories that do not apply. Four classes cover most real systems.

Untrusted content reaching the prompt. Every source that is not authored by you is attacker-influenceable: a web page the agent fetches, a PDF a user uploads, a ticket description, the body of an email, the output of another model. Indirect prompt injection is not an exotic attack, it is the default condition of any system that summarises content it did not write. Ask what the worst instruction in that content could achieve given the model’s tools, and you have your severity.

Actions with more authority than the request. The blast radius of an injected instruction is bounded entirely by what the model is permitted to do. A summariser that can only write text has a content problem. A summariser that can call a payments API has a very different one. This is an authorisation design question, not an AI question, which is why it is often nobody’s job.

Data crossing a boundary it should not. Context windows are copy operations. Retrieval that ignores the requesting user’s entitlements reproduces the underlying document store’s access-control failures at conversational speed and with a helpful summary attached. Also trace the second hop: prompts and responses in logs, in vendor telemetry, in an evaluation dataset, in a support tool.

Reliance the design does not earn. What happens when the output is wrong, and who notices? An assistant that drafts for a human reviewer fails softly. An agent that files, approves or sends fails silently, and at volume.

3. Decide what is actually enforced

For each finding, name the control point, and be honest about which of these it is:

• Prevented. The action is impossible. Scoped credentials, allowlisted tools, an egress policy, a schema that rejects anything unexpected, a human approval gate on anything irreversible.
• Detected. The action is possible but produces an alert or an auditable record, with someone accountable for reading it.
• Accepted. Nothing stops it and nothing sees it, and the organisation has agreed that this is acceptable.

Any control described as “instruct the model not to” belongs in the third bucket. System prompts are product behaviour, not a security boundary, and writing them down as one is how a threat model becomes a comfort blanket. The same applies to output classifiers used alone: useful for reducing noise, insufficient as the only thing between untrusted input and a privileged action.

4. Write it so a delivery team can build from it

The deliverable is not a risk register. It is a page or two per finding: what can happen, the path it takes through the diagram, the control that closes it, and who owns that control. Rated by what an attacker gets, not by how novel the technique sounds.

Two habits make the output durable. Record the assumptions behind each rating, such as “the retrieval index contains only documents the requesting user may already read”, so that when one of them changes, the finding it supported gets revisited rather than quietly invalidated. And keep the diagram in the repository next to the code, because the version in the slide deck stops being true in about three sprints.

What this deliberately leaves out

Model-level concerns like training data poisoning, extraction and fine-tuning integrity are real, and for an organisation consuming a third-party model through an API they are mostly the provider’s controls to operate and yours to verify contractually. Assess them in vendor onboarding, where you can do something about them, rather than in a feature review where you cannot.

The point of holding the scope this tight is that the review finishes. An assessment that takes six weeks arrives after the system is live, and the finding you most needed to raise, that the agent runs as an account with more rights than any of its users, was visible in the first hour.