Technology Insights

How to Build an Enterprise RAG Knowledge Base: Data, Permissions, Accuracy and Cost

An enterprise RAG knowledge base lets AI first find the relevant passages in your company's own documents, then answer from those passages and cite them. Whether it works depends less on the model than on four things: clean documents, permission filtering, accuracy testing with real questions, and a plan for maintenance after launch. This guide covers suitable use cases, data preparation, permissions and personal data, acceptance testing, and cost and timeline by project stage.

By Founder & CEO, JoinXPublished 9 min read
How to Build an Enterprise RAG Knowledge Base: Data, Permissions, Accuracy and Cost

Key takeaways

  • RAG (retrieval-augmented generation) has AI search company documents first and answer from the retrieved passages, so answers can cite sources and documents can be updated without retraining the model.
  • How accurate a RAG knowledge base is depends mainly on document quality: with outdated versions, duplicates and scanned files left uncleaned, even a stronger model will answer poorly.
  • Permissions must be filtered at retrieval time; documents a user cannot see must never reach the AI's answer, or the knowledge base becomes a data leak.
  • Acceptance testing should measure accuracy, citation accuracy and appropriate refusals on the same set of real questions, not on questions hand-picked for a demo.
  • Based on JoinX's custom project rates in Taiwan, a single-use-case PoC costs about NT$300,000 to 800,000 and a production launch about NT$800,000 to 2.5 million, with model API and hosting billed by usage.
Contents

Whether an enterprise RAG knowledge base is usable depends mostly on documents and permissions, not on the model. RAG (retrieval-augmented generation) has AI look up your company's own documents before answering, then respond from what it found and cite the source. Below, in the order decision makers usually ask, we cover where a RAG knowledge base fits, what to prepare, how to set permissions, how to test it and what it costs.

What is an enterprise RAG knowledge base, and how is it different from just using ChatGPT?

A RAG knowledge base has AI find the relevant passages in company documents first, then answer from those passages with citations. Asking ChatGPT directly relies on the model's training data, which does not know your internal rules and cannot show where an answer came from.

The approach was first proposed in a 2020 paper by Lewis et al. at Facebook AI Research, so that models could draw on external knowledge when answering and that knowledge could be updated without retraining. In a company, the external knowledge is operating manuals, product specifications, regulatory notices, customer service FAQs and similar documents.

ApproachWhere answers come fromAfter documents changeCan it cite sources?Good for
Public AI service as isThe model's training dataUnaware of new company rulesNoGeneral writing, translation, brainstorming
Fine-tuning a modelData trained into the modelNeeds retrainingNoFixed tone, format or classification tasks
RAG knowledge baseCompany documents retrieved for each answerJust update the documentsYesInternal rules, product information, support Q&A

Most companies want answers based on their latest rules with a clear basis, so RAG is usually a better first step than fine-tuning. For how generative AI, RAG and AI agents differ, see Enterprise AI adoption: GenAI, RAG and AI agents compared.

Which use cases suit a RAG knowledge base?

Good use cases share three traits: the answer is in documents, many people ask, and someone can judge a wrong answer. The most common are internal procedure lookups, customer service FAQs and product information for sales, where documents already exist and benefits are easy to measure.

Use caseUsersMain documentsCost of errorsRecommendation
Internal procedures and policiesEmployeesSOPs, manuals, internal rulesLow: staff can judge for themselvesBest first use case
Product and terms lookup for salesSales and store staffSpecifications, price lists, contract templatesMedium: wrong answers affect customersCite sources; a person confirms key details
Customer service FAQsAgents or customersFAQs, procedures, announcementsMedium: wrong answers raise complaintsGive it to agents first, then customers once stable
Compliance and contract pre-reviewLegal and complianceLaws, notices, contractsHigh: errors carry legal liabilityAI only compares and summarizes; people decide

Rule out unsuitable cases too. Questions that need live calculations or transaction data (such as "how much is left on my policy") require system API integration, not just documents. Areas where documents contradict each other and no one maintains them will not be answered accurately either. For customer-facing AI support, see AI customer service.

What data do you need to prepare for a RAG knowledge base?

You need a document list with owners: which documents go in, which version is current, who updates them and who may see them. Document cleanup is usually the largest early workload and the key to accuracy.

In practice, document problems are far more common than model problems:

  • Duplicate versions: The same policy exists as a 2023 version, a 2024 version and a "draft amendment". If the AI retrieves the old one, it answers wrongly. Before launch, decide to include only current versions or mark effective dates on documents.
  • Hard-to-parse formats: PDFs scanned as images, Excel sheets full of merged cells and notes hidden in slide comments all need to be converted into readable text and tables first.
  • No owner: If no one updates the knowledge base when documents change, answers go stale within months. Assign an owning team to every document category.
  • Unwritten knowledge: Rules that senior staff "just know" but never wrote down are invisible to AI. Projects often use the opportunity to document them.

Start with one department and tens to a few hundred documents for a PoC, confirm accuracy, then expand.

How should a RAG knowledge base handle permissions and personal data?

Filter permissions at retrieval time: decide which documents a user may search based on who they are, then let the AI answer only from those documents. If you retrieve everything and mask afterwards, sensitive content can still end up in the answer.

OWASP's 2025 Top 10 for LLM applications lists both sensitive information disclosure and vector and embedding weaknesses as risks; the latter includes weak access control in vector databases that lets users retrieve content they should not see. In a project, that translates into the following:

RiskWhat happensWhat to do
Unauthorized retrievalA regular employee gets answers from manager-only pay or HR rulesTag documents with permissions and filter by the logged-in user at retrieval
Personal data leaksCustomer names, ID numbers or account numbers appear in answers or are sent to an external modelMask or exclude personal data before indexing; intercept it before questions are sent
Data leaving approved environmentsDocuments go to a cloud service that violates company policyCheck the model service's data terms, or use an enterprise plan or private deployment
No audit trailWhen something goes wrong, no one knows what the AI based its answer onLog questions, retrieved passages and answers

Taiwan's Personal Data Protection Act requires that personal data be collected, processed and used within a specific purpose. Before putting documents containing personal data into a knowledge base, confirm the use falls within the original purpose; if unsure, leave them out.

How do you test the accuracy of a RAG knowledge base?

Use the same set of real questions to test three things: answer accuracy, whether citations are correct, and whether it honestly says it does not know when documents lack the answer. Set the test questions before development, not from questions picked for a demo.

NIST's 2024 generative AI risk profile (AI 600-1) notes that generative AI can produce content that sounds plausible but is wrong. RAG reduces this risk but does not remove it, so acceptance testing must cover what happens when there is no answer. Recommended steps:

  1. Collect real questions. Gather 100 to 300 questions from support logs, internal inquiry emails or staff interviews, and record the correct answer and the document that should be cited.
  2. Mix question types. Include questions with clear answers, questions that need several documents, and questions the documents do not cover, which should be refused.
  3. Set thresholds. Decide target levels for accuracy, citation accuracy and correct refusals, agreed jointly by business teams and IT.
  4. Retest after every change. After changing how documents are chunked, retrieval settings or prompts, rerun the same questions and compare.
  5. Keep collecting after launch. Add questions users report as wrong to the test set and retest regularly.

How much does an enterprise RAG knowledge base cost, and how long does it take?

Based on JoinX's custom project rates in Taiwan, the requirements review is free, a single-use-case PoC costs about NT$300,000 to 800,000, and a production launch about NT$800,000 to 2.5 million, with cross-department work quoted by scope. Model API and hosting are billed by usage.

StageEstimated costTypical scopeTypical timeline
Requirements reviewFreeUnderstand your workflow and data, judge whether AI fits, suggest the first use case1–2 meetings
Single-use-case PoCNT$300,000–800,000One department's documents, basic retrieval and Q&A interface, testing against the question set4–8 weeks
Production launch (single use case)NT$800,000–2.5 millionAccount and permission integration, document update workflow, activity logs, monitoring and deployment2–4 months
Cross-department, multiple systemsQuoted by scopeMultiple knowledge bases, business system APIs, AI agent workflows4+ months

Cost is driven mainly by three things: the number of documents and how hard they are to clean up, how complex the permission rules are, and which existing systems must be connected (for example employee accounts, document management or support systems). These figures are a starting point for budgeting, not a fixed quote.

How does JoinX build enterprise RAG knowledge bases?

JoinX sets the test questions and permission rules before development starts: first prove that real questions are answered accurately and that restricted documents cannot be retrieved, then expand. Development follows SDD plus TDD, so the question set is part of the tests and can be rerun after every change.

For an insurance company's AI assistant platform, several requirements were built in from the start: every answer cites the document passages it relied on so staff can click back to the original; documents are filtered by user permissions at retrieval; and personal data is intercepted before a question is sent to the model. For the same insurance company, the AI agent interface was designed around the order in which frontline sales staff work, with AI helping to look up and organize information and a person confirming at the end. See the insurance company AI agent case.

Before delivery, JoinX runs its own source code scans and vulnerability scans; black-box testing is performed by a third party, which JoinX can help arrange, and vulnerabilities are fixed before handover. The document update workflow, question-set retesting and model version changes after launch are also handed over at delivery.

FAQ

Is an enterprise knowledge base chatbot the same thing as RAG?

In most cases, yes. What people now call an enterprise knowledge base chatbot is usually built with RAG: users ask in a chat, the system retrieves company documents, and AI answers from the results. Older FAQ bots that matched keywords and returned fixed replies are not RAG; they can only answer questions set up in advance.

We already use generative AI. What skills does the team need to add RAG next?

You need three roles: people who know the documents and the business to clean up documents and write test questions, IT to handle accounts, permissions and system integration, and an owner who can decide thresholds and launch scope. Development and model tuning can be outsourced, but document ownership and acceptance criteria must stay in-house, or no one can maintain it after launch.

Can a RAG knowledge base be error-free?

No. RAG grounds answers in company documents and cites sources, which greatly reduces made-up answers, but it can still be wrong when it fails to retrieve the right passage or the document itself is wrong. That is why answers should cite sources so people can check, high-risk questions should go to a person, and wrong answers should keep being added to the test set.

Will documents put into a RAG knowledge base be used to train the model?

It depends on the model service and plan. Most cloud models' enterprise plans and API terms state that customer data is not used for training by default, but terms differ between providers, so check data location, retention and usage scope line by line before adopting. For highly sensitive documents, consider private deployment or leave them out of the knowledge base.

What are the ongoing costs after a RAG knowledge base launches?

Mainly model API usage, hosting and vector database fees, and maintenance staff. Usage fees grow with the number of questions; maintenance covers document updates, retesting, model version changes and system fixes. Estimate monthly usage from the question volume during the PoC before going live.

Sources

  • Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020)Patrick Lewis et al., Facebook AI Research
  • OWASP Top 10 for LLM Applications 2025OWASP Foundation
  • Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1, 2024)National Institute of Standards and Technology (NIST)
  • Personal Data Protection Act (Taiwan)Ministry of Justice (Laws & Regulations Database of the Republic of China)

These are the regulations and sources cited at the time of writing. Rules may be revised; always check the latest official announcement.

About the author

Jack Lee

Founder & CEO, JoinX

Jack founded JoinX in 2016 and leads a team of 80+ people across Taiwan and Japan. JoinX has delivered 520+ projects for 220+ clients, including 56 mid-to-large systems.

Related articles