Prompt is not permission

“Only answer using the current user's information. Use only data they are entitled to see.”

That is a reasonable instruction to put in an AI assistant's system prompt. It tells the model how we want it to behave. The mistake is treating those words as the mechanism that makes unauthorized access impossible.

Teams of all sizes are building assistants to answer customer questions and handle requests. In an enterprise, a banking assistant may be part of a wider push to adopt generative AI and automate service. Under pressure to deliver, a rule written in the prompt can look like a control already implemented.

It is an easy mistake to make. The assistant follows the rule in a few tests, refuses a request for someone else's information, and gives sensible answers to ordinary questions. A newer model may follow the instruction more reliably. We start to feel that the boundary is in place. But the system may still give the assistant tools that can retrieve any customer's records. The prompt asks it to exercise restraint while the application leaves the door open.

This was the question behind my banking-assistant study: what changes when we move a restriction from a prompt into the code that retrieves the data?

Consider a hypothetical customer who signs in and asks for their balance or recent transactions. The application knows who is authenticated. The assistant interprets the question and chooses a tool. In one design, the tool accepts an account number or customer identifier and retrieves whichever record that identifier names. The system prompt tells the assistant to use only the signed-in customer's information.

If the assistant requests another customer's account, the query can still succeed. The model may have misunderstood a request, followed a misleading instruction, or simply made the wrong choice. Whatever caused that choice, the application has allowed it to cross the customer's data boundary. A refusal in yesterday's test does not change what today's tool call is permitted to do.

In a stronger design, the application supplies the authenticated customer's identity to the data-access code. A balance or transaction query is constrained to accounts owned by that customer. An account identifier can select among accounts the customer is entitled to use; it cannot expand that entitlement. Asking for someone else's identifier does not make their records accessible.

The important difference is where the decision happens. The model still helps choose what information would answer the question. The application decides whether it can retrieve that information. Permission comes from the authenticated session and the access policy, rather than from the identifier the model happens to request.

This does not require the assistant to write SQL, the language used to request records from a relational database. In the banking study, the application defined the queries in advance and allowed the tools to supply values, such as an account number. These were parameterized queries: supplied values were kept separate from the instructions in the query. That helps protect against SQL injection, where input is used to alter the query itself. It does not decide whether the customer is allowed to see the requested record. A query can safely use the supplied account number and still return someone else’s account. The application must also check ownership.

The same distinction applies to read-only access. It is useful protection against changes to the database. It does not stop a connection with broad read access from retrieving information the current user should not see. Preventing unauthorized reads and preventing writes are different responsibilities, and both need concrete controls.

For an assistant that can change data, those controls must cover the allowed operations too. Its database or service credentials should grant only the capabilities its workflow needs. If the assistant has no legitimate reason to delete records, its execution path should not carry that power. Where changes are needed, the application should check the operation and its scope before executing it. A prompt saying “never delete data” is still worth having, but it cannot revoke a capability granted elsewhere.

Instructions can also arrive inside the records being read. A transaction remark might tell the assistant to look up another account. The remark is data from the banking workflow; it is not authority to widen access. Session-constrained queries help even if the model follows that instruction. They do not settle every disclosure question, though. A customer's own transaction may legitimately contain a counterparty's account reference. We still need to decide which fields the customer may see, rather than assuming that every foreign identifier is a leak or that every field in an owned record is safe to expose.

Better models and clearer prompts can reduce mistakes. They remain valuable. They should work alongside access controls, with checks in the application, query or database layer that hold regardless of the model's request. Those checks need to cover every available retrieval and action path; a constrained balance tool does little good if another tool offers unrestricted access to the same data.

A useful test is to deliberately submit a request outside the permitted scope and inspect what actually happens. Does the tool return foreign records? Does a forbidden change reach the database? Then check that ordinary authorized work still succeeds. An assistant that denies everything is not a useful banking service, and a fluent refusal does not prove that the underlying access path is secure.

When evaluating an assistant, I would ask what prevents a wrong tool request from succeeding. The answer should point to an enforced boundary we can inspect and test. “We told the model not to” describes an instruction. It does not describe a permission check.

Browse all 147 expected cases, 30 planted payloads and the companion workbook →

Further reading

OWASP: Prompt Injection

OWASP: Excessive Agency

For the worked example and study limits, read the banking companion.