We built a chatbot or RAG assistant on our own documents. Is it safe to launch?

The situation

Your team has built an assistant. It answers questions from your policies, contracts, tickets or product documentation using retrieval-augmented generation, or RAG: the system finds relevant passages in your documents and lets a language model write an answer from them. The demo is impressive and the business wants it live, for staff or for customers.

Someone asks the question you have been avoiding: what happens if a user asks for something they are not allowed to see? Or pastes in text designed to make the assistant misbehave? Your developers believe the system prompt and the model’s safety features will handle it. You are not sure that belief has been tested.

Why this is hard

• RAG retrieval ranks documents by similarity, not by who is allowed to read them. Unless authorization is designed into retrieval, a user can receive passages from documents they should never see.
• Documents can carry instructions. A document uploaded by a user, or retrieved from a shared source, can contain text that redirects the assistant. This is indirect prompt injection, and conventional application testing does not look for it.
• The vector database, where document fragments are stored as searchable vectors, is often reached with a single shared credential, so the database itself cannot tell one user from another.
• Caches and conversation memory can leak between users and tenants even when retrieval itself is correctly restricted.
• Model safety features are not an authorization system. They reduce certain harmful outputs; they do not know your access rules.
• Standard penetration testing covers the web application around the assistant. It rarely covers the retrieval pipeline and the model’s behavior.

What goes wrong when it is ignored

• An employee or customer receives confidential content from another department, another customer or another tenant, and the exposure is discovered through a complaint.
• A planted document causes the assistant to give wrong, manipulated or harmful answers under your brand.
• A government or regulated customer’s security review asks how RAG authorization works, and the answer is not acceptable.
• The assistant is withdrawn after launch, which is far more visible than delaying it.

How I approach it

1. Review the architecture first. Data sources, ingestion pipeline, chunking and embedding, vector store, retrieval logic, prompt assembly, model endpoint, caches, memory, tools the assistant can call, identities and logging. I draw the trust boundaries and mark where authorization is actually enforced and where it is only assumed.
2. Test the retrieval-specific failure modes. Structured test cases for data exfiltration through retrieval, cross-user and cross-tenant leakage, vector-database authorization gaps, poisoned or malicious documents, indirect prompt injection and system prompt or context leakage. These are the failure modes I reproduce in the Shah AI Security Lab; the linked labs show the method and the evidence.
3. Check the hidden layers. Caches, conversation memory, shared service credentials to the vector store, and any tools or APIs the assistant can invoke on a user’s behalf.
4. Rate what was found in business terms. Which findings block launch, which can be fixed in the first release, and which are accepted risks with a documented owner.
5. Design the hardened version. Authorization enforced at retrieval and at the vector store, tenant-scoped caches and memory, trusted-source labeling and content inspection for ingested documents, output validation, and logging of identity, prompt, retrieved context and the authorization decision for every request.
6. Retest and prepare for launch. The same test cases are run against the hardened system, and the results are written up so you can show a customer, an auditor or your own board how the assistant was assessed.

What you receive

• An architecture and trust-boundary review of the assistant.
• Reproducible findings for retrieval authorization, leakage, injection and poisoning scenarios, with evidence.
• A severity and business-impact rating per finding and a launch recommendation.
• A hardened reference design for retrieval authorization, state isolation, ingestion trust and logging.
• Retest results and a launch-readiness summary suitable for customer or audit review.

The assessment method is being built and published openly in the Shah AI Security Lab, lab by lab. The labs linked on this page show the current evidence base rather than a claimed track record.

Where to start

A free 30-minute Architecture Decision Review at /free-architecture-review/ is enough to walk through the assistant’s architecture and tell you which of these failure modes are likely in your design. Or describe the system through the contact form and choose the AI security assessment topic.