A RAG chatbot is an AI assistant that searches your own documents before it answers. It finds the passages that match the question, hands them to a language model, and the model writes a reply based only on those passages, with links to the sources. RAG stands for retrieval-augmented generation.
That is the textbook answer. This guide is about what happens after the definition: when RAG is the right tool and when it isn’t, where these bots still get things wrong, how to test one before your customers do, and what it really costs to build and run.
We’re writing this from hands-on work. Our team built the RAG pipeline in Abee, an enterprise assistant that answers staff questions from company documents and connects to CRM and HR systems. We also built Sakkar AI, a research assistant that answers health questions from published medical research instead of the model’s memory. The lessons below come from those builds and from scoping chatbot projects with founders.
The short version
- RAG changes what a model knows; fine-tuning changes how it behaves. For answering questions from your documents, you almost always want RAG.
- You may not need RAG at all. If your knowledge base is under about 500 pages and rarely changes, putting all of it in the prompt is often simpler.
- Live data is not a RAG problem. “Where is my order?” needs a connection to your order system, not a document search.
- RAG reduces wrong answers but does not remove them. Specialist legal AI tools built on RAG still gave wrong answers to 17% to 33% of test questions in a Stanford study.
- Most quality problems are retrieval problems. If the right passage never reaches the model, no prompt can save the answer.
How a RAG chatbot works, step by step
Every RAG system has the same five steps. The difference between a demo and a product is how carefully you build each step.
1. Collect and clean the source content
Help articles, PDFs, policies, SOPs, product specs, past support tickets, Notion or Google Drive pages. Whatever goes in sets the ceiling for what comes out. If two versions of your refund policy are in the library, the bot will sometimes quote the old one, and it will cite it confidently.
Worth doing first: list your top 30 customer or staff questions and check that each answer exists in exactly one current document. It takes about two hours and often prevents more bad answers than switching to a better model.
2. Split the content into chunks and index them
Documents are cut into chunks of a few paragraphs. Each chunk is turned into an embedding, a list of numbers that captures meaning, and stored in a vector database such as Supabase pgvector or Pinecone.
Chunking is where many projects quietly lose accuracy. A chunk that says “the company’s revenue grew by 3%” is useless on its own, because it no longer says which company or which quarter. Anthropic tested a fix it calls contextual retrieval: add a short sentence of context to every chunk before indexing it. Combined with keyword search, that cut failed retrievals by 49% in their tests, and by 67% with a reranking step on top.
3. Search by meaning and by keyword
The system also turns the user’s question into an embedding and pulls the chunks closest in meaning. That is why “can I get my money back?” finds a section titled “Refund policy”.
Meaning-based search alone misses exact terms such as product codes, error numbers, and names. Production systems run a keyword search (usually BM25) alongside it, merge the results, then use a reranker to put the best passages on top.
4. Filter by permissions before the model sees anything
This step is missing from most explainers, and it matters most for internal assistants. If a sales rep asks about salaries, the HR documents must be removed from the search results before they reach the model. Telling the model “don’t reveal HR data” in the prompt is not access control; a cleverly worded question can get around an instruction.
From our builds: in Abee, every user’s role decides which records and documents they can search, alongside AES-256 encryption and multi-factor login. The permission check happens in the data layer, not in the prompt.
5. Write a grounded answer, cite it, or hand off
The model receives the question plus the top passages, with instructions to answer only from them, cite them, and say it doesn’t know when the passages don’t contain the answer. When it can’t answer, or the topic is sensitive (billing disputes, legal or medical questions), it should hand the full conversation to a person so the customer doesn’t repeat themselves.
Do you even need RAG? A quick decision guide
A good vendor should sometimes tell you not to build RAG. Here is how we decide on a first call.
| Your situation | What to build |
|---|---|
| Under about 500 pages of content that rarely changes, one audience | Put all of it in the prompt and use prompt caching. No vector database needed. |
| Hundreds of documents, frequent updates, or answers that must cite sources | RAG |
| Different people may see different documents | RAG with permission filtering before retrieval |
| The answer lives in a live system: order status, account balance, leave days, a CRM record | A direct connection to that system (API or tool calling), often next to RAG |
| You need a fixed tone, format or labelling task | Fine-tuning, rarely on day one |
The 500-page line comes from Anthropic’s own guidance: below roughly 200,000 tokens, you can often skip retrieval and include the whole knowledge base in the prompt. Above that, bigger isn’t better. Chroma’s context rot research tested 18 leading models and found that accuracy drops as inputs get longer, even on simple tasks. Retrieval keeps the model’s input short and on topic.
From our builds: Abee’s client had a different core problem than “we need a chatbot”. Staff were switching between separate CRM and HR systems to find answers. So the project combined RAG over documents with real-time API connections to those systems. A bot that only searches documents can’t answer “what’s the status of my request?”
RAG vs fine-tuning vs long context
| RAG | Fine-tuning | Everything in the prompt | |
|---|---|---|---|
| What it changes | What the model can look up | How the model writes and behaves | What the model reads on every question |
| Updating knowledge | Edit a document, live in minutes | Retrain the model | Edit the prompt |
| Shows sources | Yes | No | Possible, less precise |
| Per-user access control | Yes, by filtering results | No | Only by building separate prompts |
| Scales to large libraries | Yes | Not for facts | No, accuracy and cost suffer |
| Best for | Answers from changing documents | Tone, format, narrow tasks | Small, stable knowledge bases |
Where RAG chatbots still go wrong
RAG is often sold as the cure for AI making things up. It isn’t. A Stanford RegLab and HAI study tested legal research tools from LexisNexis and Thomson Reuters, which use RAG and were marketed as avoiding hallucinations. They still gave false or unsupported answers to between 17% and 33% of test questions. Better than a general chatbot, but far from zero.
And the business carries the risk. In Moffatt v. Air Canada (2024), the airline’s chatbot told a customer he could claim a bereavement discount after booking. The real policy said otherwise. Air Canada argued the chatbot was a separate entity responsible for its own words; the tribunal disagreed and ordered the airline to pay. Whatever your bot says, you said.
The failures we watch for most:
- Stale or conflicting documents. The bot finds the old policy and cites it. Fix: one owner per document and a review date.
- Chunks without context. The right sentence is found but no longer says what it refers to. Fix: contextual chunking and sensible chunk sizes.
- Tables, scans and slides. Scanned PDFs and complex tables often index badly. Fix: convert them properly before indexing and test questions that depend on them.
- Questions that need a calculation or live data. “How much would 3 seats cost on the annual plan?” needs logic or a pricing system, not a paragraph. Fix: tool calls for numbers and records.
- Prompt injection. A user or a document contains instructions that try to override the bot. It is the first risk in the OWASP Top 10 for LLM Applications. Fix: permission filtering outside the model, limited tool access, and logging.
How to test a RAG chatbot before your customers do
You can run this test on any vendor’s demo, including ours.
- Write 30 to 50 real questions from support tickets, sales calls, and staff chats, with the correct answer and source document for each.
- Add 10 questions the bot should refuse: answers that aren’t in your documents, off-topic requests, and one attempt to make it ignore its instructions.
- Score three things for every answer: is it correct, does it cite the right source, and did it say “I don’t know” when it should have.
- Re-run the full set after every change to documents, prompts or models. A fix in one place often breaks an answer somewhere else.
When a test fails, check retrieval first: did the right passage reach the model? In our experience, the fix is more often a missing or messy document than the code.
Rules to know if you serve UK or EU users
Since 2 August 2026, Article 50 of the EU AI Act requires that people are told when they are talking to an AI system, unless it is obvious from the context. In practice, put a clear “AI assistant” label on the chat window and offer a route to a human. GDPR still applies to everything users type, so you need a data processing agreement with your model provider and a clear answer on where the data is stored and whether it is used for training. This is general information, not legal advice.
What a RAG chatbot costs to build and run
Build cost depends on the number of data sources, integrations, channels, and how strict access control needs to be. Our current ranges:
| Project type | Typical build cost | Typical timeline |
|---|---|---|
| Website or support chatbot | $1,500 to $3,000 | 1 to 2 weeks |
| RAG knowledge assistant | $3,000 to $6,000 | 3 to 5 weeks |
| Enterprise AI assistant (roles, integrations, audit logs) | $6,000 to $12,000 | 6 to 10 weeks |
Running costs work differently than most people expect. Indexing your documents is cheap: at OpenAI’s listed price of $0.02 per million tokens for its small embedding model, embedding 1,000 pages of text costs about one cent. The real monthly cost is the language model reading passages and writing answers to every question.
The biggest lever is how much text you send it: passing 20 chunks per question instead of 5 roughly quadruples the input cost of each answer. We size this against your expected question volume before the build. Our AI chatbot development cost guide has monthly ranges by traffic level.
For AI features beyond a chatbot, see our guide to LLM integration cost and timelines.
7 questions to ask before you hire anyone to build one
- “Show me what happens when the answer isn’t in our documents.” You want a clear “I don’t know” and a hand-off, not a guess.
- “How do you stop users seeing documents they shouldn’t?” The answer should be filtering before retrieval, not a line in the prompt.
- “How will we measure accuracy?” Look for a written test set with scores; re-run after every change.
- “Do we need RAG, or would a long prompt do?” An honest builder will sometimes say you don’t need it.
- “Which of our questions need live system access?” Orders, accounts, and bookings need integrations, and they change the scope.
- “How do we update content, and how fast does it go live?” Your team should be able to update a document without a developer.
- “Where does our data go?” Ask which model provider, which region, whether data is used for training, and whether they sign a DPA.
Frequently asked questions
A RAG chatbot is an AI assistant that looks up answers in your own documents before it replies. It searches your knowledge base, finds the relevant passages, and writes an answer based on them, usually with links to the sources, instead of relying on what the model memorised in training.
RAG stands for retrieval-augmented generation. Retrieval is the search step that finds relevant passages in your data. Generation is the language model writing the answer. Augmented means the answer is built on the retrieved passages.
For answering questions from business documents, usually yes. RAG is easier to update, can cite its sources, and can respect user permissions. Fine-tuning is better for teaching a model a fixed tone, format, or narrow task. Some advanced systems use both.
Yes. RAG reduces wrong answers but does not remove them. A Stanford study found specialist legal tools built on RAG were wrong on 17% to 33% of test questions. Testing on real questions, citations, permission controls, and a clear “I don’t know” response keep the risk low.
Often not. If your content is under about 500 pages and rarely changes, you can usually put all of it in the prompt with prompt caching. RAG becomes worth it with large or changing libraries, when you need citations, or when different users may see different documents.
A website or support chatbot typically takes 1 to 2 weeks. A knowledge assistant connected to several internal sources takes 3 to 5 weeks. An enterprise assistant with role-based access, integrations, and audit logs takes 6 to 10 weeks.
Want a straight answer on whether RAG fits your business?
Send us your top questions and tell us where the answers live. On a 15-minute call, we’ll tell you whether you need RAG, a simpler prompt-based bot, or a system integration, and what it would cost. See our AI chatbot development services or book a free call.
