← All posts
AI security

Before you launch an LLM app: a security checklist

Prompt injection, data leaks, and over-powered agents are the risks that reach production. Here is what to test before your LLM feature goes live, and how.

Most LLM features pass their demo and fail their first week in production. Not because the model is wrong, but because nobody asked what happens when a user, a document, or a web page tells the model to do something else.

Traditional security reviews don't catch this. A penetration test checks your API, your auth, and your infrastructure. It rarely checks whether a customer support bot can be talked into revealing another customer's order history. That gap is where most LLM incidents happen.

This checklist covers what to test before launch. It follows the OWASP Top 10 for LLM Applications, focused on the risks that most often reach production.

1. Assume the prompt will be injected

Prompt injection is the top risk on the OWASP list for a reason: a language model cannot reliably tell the difference between your instructions and text it is asked to read.

There are two kinds to test:

Indirect injection is the one teams miss. If your app runs retrieval (RAG) over documents that anyone outside your company can write, those documents are part of your attack surface.

What to do before launch:

2. Control what the model can see

A model can only leak what is in its context window. The most effective data protection is not giving it the data in the first place.

If you serve users in Japan, personal data in prompts and logs falls under the Act on the Protection of Personal Information (APPI). Know where your model provider processes and stores that data before you ship.

3. Limit what an agent can do

The risk grows sharply once a model can take actions: send email, update records, call APIs, run code. OWASP calls this excessive agency.

The question to ask for each tool an agent can use: if an attacker controlled the model's next output, what is the worst this tool could do?

4. Never trust model output

Model output is user input by another name. If it flows into another system, validate it the way you would validate anything a stranger typed.

5. Red-team it before your users do

Automated scanners help, but an LLM feature needs someone actively trying to break it in the context of your product. A useful pre-launch red-team exercise covers:

  1. Your threat model — who would attack this, and what would they want: data, free usage, reputational damage, or access to connected systems?
  2. Every input channel — chat, uploads, retrieved content, tool responses, and any multilingual paths. Test in every language you support; filters tuned on English often miss Japanese.
  3. Every tool and permission — try to make the agent misuse each one.
  4. Findings with fixes — each issue ranked by impact, with a concrete change in code or configuration, not just "improve the prompt."

Then keep the test cases. Every model upgrade, prompt change, or new tool should run against them again, the same way you run regression tests.

The short version

None of this makes an LLM app perfectly safe. It makes the failures small, visible, and recoverable, which is what you need before you put one in front of customers.

Have a project like this?

Tell us what you need to build, secure, migrate, or audit. We will come back with a clear next step.

Book a consultation