Your enterprise knowledge base is connected to an LLM — so why are the answers still inaccurate?

Take the same set of policy documents, product material and historical tickets, give them to two teams, and let each build a Q&A system. The results often differ enormously: one handles most everyday questions, while the other needs three passes through the documents to answer "how is annual leave calculated" — and still ends up asking a person.

The difference is usually not the model.

A more reliable rule: when answers are wrong, check the data and the retrieval first, and only then consider the model. If errors cluster around "this rule is not in the knowledge base at all", "three documents describe the same thing differently" or "the answer cites last year's version", a bigger model will not help. If the data is correct and the citations are accurate, and only the wording or the reasoning is off, then it is time to look at the model and the prompting strategy. Getting this order wrong usually means paying to upgrade the model and ending up with exactly the same errors.

Below I go from "how to judge" to "the four layers" to "how to implement", focusing on the parts readers can verify themselves.

1. Split "inaccurate" into four kinds before deciding where to look

"Inaccurate" is a vague label. On concrete questions it shows up in at least four ways, and each points to a completely different layer.

Symptom	Layer most likely at fault	Would a bigger model help	How you can verify
It says "there is no such rule" although the document clearly states it	Retrieval: chunking, terminology, permission filtering	Largely no	Search the knowledge base directly with the wording from the document and see whether it is found
The answer cites a repealed policy or an outdated price	Data: versions, validity, single source of truth	No	Sample the cited passage and check its effective date against the current version
Two documents contradict each other and the answer picks one at random	Data: unresolved conflicts	No	Find the conflicting documents and confirm which one is the authoritative version
The right documents were found, but the answer drops a key condition, miscalculates or overgeneralizes	Generation: composition and reasoning	Possibly	Walk through the answer item by item against the source text
The question is outside the knowledge scope, yet the system still produces a plausible answer	Generation: missing refusal design	Usually worse	Ask questions the knowledge base cannot answer and see whether it admits it does not know
Something that worked last week no longer works this week	Operations: index refresh, permission changes	No	Check the document update time and the index rebuild log

How to use the table: collect 20 to 30 real wrong answers first, classify each one, and see which layer they cluster in. If 80% of the errors sit in the data and retrieval layers, the bottleneck is knowledge governance, not model capability; comparing parameter counts or leaderboard scores at that point is spending money in the wrong place.

One easily overlooked detail: many teams test only "questions the knowledge base can answer" during the demo, and the resulting accuracy looks great. What really determines the production experience is "questions the knowledge base cannot answer" and boundary questions — precisely the two categories that demos tend to skip.

2. The data layer: a knowledge base is not a filing cabinet moved into a system

The most common problem with enterprise knowledge bases is not too little material, but several versions of the same thing living side by side.

Policy documents are the clearest case: a procedure from three years ago, last year's revision and this year's supplementary notice all sit in the system, with similar titles and overlapping text. People know the newest one governs; the retrieval system does not. It simply returns all three by similarity and lets the model "synthesise" an answer — one that very likely blends repealed clauses with current ones.

The fix is not a smarter model but three things at ingestion:

First, record the validity status of every knowledge item. Effective dates, repeal flags and supersession relationships belong in metadata, not just in the first line of the body. Retrieval should return only the current version by default, surfacing historical versions only when someone explicitly asks how something used to be regulated.

Second, assign an owner to each knowledge category. Whoever drafts a policy is responsible for updating the knowledge item after a revision; product material is maintained by the product owner. A knowledge base without owners starts to drift within six months. The vendor cannot do this on the client's behalf, because deciding "which version is authoritative" is a business authority, not a technical one.

Third, write down the exceptions and the unwritten rules. Many wrong answers stem from "the policy says this, but in practice there is an exception that only long-tenured staff know". The travel policy says "accommodation is reimbursed per the standard", while in practice there is an internal peak-season uplift that was never written down. No model can invent knowledge that is missing. It is worth spending dedicated time in the scoping phase asking business staff: "Are there exceptions to this rule? Who approves them?"

Two further categories are often missed: information inside tables and attachments, and scanned documents. The former is easily flattened into plain text during chunking, losing row and column relationships; the latter effectively does not exist in the knowledge base unless OCR has been run. A spot check before ingestion — "open 10 random files and see whether the content is complete and readable" — is far more effective than tuning the model afterwards.

Permissions belong here too. An enterprise knowledge base almost always contains tiered information: compensation policy open only to HR and managers, customer contracts only to the relevant team. The right approach is to filter candidate documents by user identity before retrieval, not to retrieve everything and then ask the model to "be careful not to say it". Once content enters the model context it can leak through phrasing, examples or summarisation, and that is very hard to catch by testing. The test is simple: for a user without permission, the system should behave as if it cannot find the information, not as if it knows but will not say.

3. The retrieval layer: the material is in the knowledge base, so why isn't it found

Once the data is governed, the second layer of problems appears: the content is correct but never retrieved.

Chunking decides whether something can be found at all. Splitting by a fixed character count is the easiest approach and the most error-prone: a policy cut in the middle separates "approval authority" from "amount thresholds" into two chunks, retrieval matches only one, and the answer loses a key condition. A more robust approach is semantic chunking — one policy, one process step, one set of Q&A pairs per chunk — keeping the section context on each chunk. There is no universally optimal granularity, but there is a test: can this chunk be understood on its own, without the surrounding context?

Internal names must be recognised. Employees ask using internal shorthand: project codes, "that system", a product's former name, a department's habitual wording. These words often never appear in formal documents. The answer is to maintain aliases and synonyms on the knowledge item rather than expecting the model to guess. Abbreviations, typos and colloquial phrasing need the same treatment.

Process knowledge deserves its own page. For a question like "how is a customer refund handled", the answer is scattered across the customer-service standard, the finance policy and the system manual. Relying on retrieval luck to gather all three has a low success rate. The reliable approach is to compile a single process page: trigger conditions, what to do at each step, who approves, how exceptions are handled. Cross-document answers should be assembled once by a person, not stitched together on the fly every time.

Questions about metric definitions should not go to document Q&A. "What was last month's return rate in East China" is answered by a database, not by documents. Forcing document retrieval on it produces a document that defines the metric and then an outdated number. The correct split: definitions, calculation rules and business logic live in the knowledge base; actual figures come from structured queries. Mixing them is a direct reason many projects are abandoned by the business side after launch.

Index refresh needs a time commitment. If a policy changes today, when will the Q&A reflect it? If the index is rebuilt weekly, say so inside the system and give administrators an "update now" entry point. Otherwise users hit "I saw the new notice, but it is still quoting the old rule" — and trust is gone after one occurrence.

4. The generation layer: let it dare to say "I don't know"

With the first three layers solid, the generation layer has a clear job: faithfully restate what was found, and admit it plainly when nothing was found.

Three design points:

Cite down to the passage. Attach the cited document and section to each answer so users can verify in one click. This is not only about trust but also about operations — when something goes wrong you can tell immediately whether retrieval found the wrong passage or the data itself is wrong. A Q&A system without citations cannot attribute errors, and therefore cannot improve.

A refusal needs an exit. "I don't know" is an acceptable answer in enterprise settings, but stopping there makes users abandon the tool. A sound refusal also offers: links to possibly relevant documents, the right person to ask (which department, which role), and an entry point to a human agent. Design the refusal as a path, not a wall.

High-risk questions stay out of automated answers. Anything involving monetary calculations, contract terms, legal liability, medical or safety advice, or external commitments should go straight to a human, or provide the source link without a generated conclusion. That boundary belongs in system configuration and the acceptance checklist, not only in a prompt — a prompt can silently change behaviour with one model upgrade; configuration and rules do not.

One more practical design: when a question is ambiguous, clarify before guessing. "How do I get reimbursed" might mean travel, entertainment or procurement. Asking "is this a travel expense or something else" is more useful than a generic answer, and it exposes structural gaps in the knowledge base: if the same vague question keeps appearing, that is a signal to add an overview page.

5. The operations layer: no evaluation set, no acceptance

The biggest difference between knowledge-base Q&A and ordinary feature development is that there is no "click once and you know whether it is right" way to accept it. That is why you need your own evaluation baseline early in the project.

How to build the baseline. Take 30 to 50 questions from real usage: customer-service logs, internal group chats, ticket titles, the things business staff ask on the phone. Cover three categories — high-frequency questions, error-prone questions, and questions that clearly should not be answered. For each, note which points the standard answer must contain, the acceptable range of phrasing, and the situations that must be refused.

Which metrics to watch. Four are enough, and none is complicated:

• Answer usability: can a business user act on it directly without asking a colleague;
• Citation accuracy: do the cited documents and passages actually support the conclusion;
• False refusal rate: the share of questions that do have an answer but are answered with "I don't know";
• Out-of-scope answering rate: the share of answers that state firm conclusions without supporting evidence in the knowledge base.

The last two say the most about whether the system can be trusted, yet they are most often skipped — because they do not affect how clever it looks.

Classify bad cases by layer. Log every wrong answer against one of the four layers (data, retrieval, generation, operations) and review the distribution weekly. If errors cluster in the data layer for several weeks in a row, it is time to push the business side to clean up documents instead of continuing to tune prompts. This distribution is also the best progress material for client conversations: it turns "it feels inaccurate" into "11 of this week's 18 wrong answers came from outdated files that were never removed".

Re-run the baseline after knowledge updates. Every bulk update should be followed by a baseline rerun to confirm that nothing which used to be right became wrong. This step is often skipped, and the cost is a burst of puzzling errors during the transition between versions.

6. Three kinds of teams start from different places

Building knowledge-base Q&A is not the same job everywhere; the first move should differ by company.

Policy and compliance documents (HR, administration, legal). Start with version cleanup and permission tiers: merge multiple versions of one topic into "current" and "historical" layers, then define visibility by role. In these projects the data-governance workload typically exceeds the system development workload.

Business data (orders, inventory, customers). Start by deciding which questions should go through structured queries. Document Q&A should cover only "how is the rule defined, how does the process run"; numeric questions belong to reports or APIs. Projects that mix the two usually end up with neither working well.

External customer service. Start with the high-frequency question set and refusal rules: settle the 50 most common questions and their standard replies, and specify which topics are never answered automatically (price commitments, liability determinations, after-sales exceptions). Tolerance for a wrong answer is far lower externally than internally, so the boundary must come first.

If the knowledge volume is small and the questions are highly concentrated (say, a few dozen FAQs about a single product), a mature product or a manually maintained FAQ is the better deal. Custom development pays off when material is scattered, permissions are complex, integration with business systems is required, and the company needs to build its own evaluation and operations mechanism — in those cases a general-purpose product struggles to cover the last mile.

7. A sequence you can follow

1. Scope it. Decide which categories of questions the first release answers and which it explicitly will not. Keep it narrow and finish one category properly.
2. Clean the data. Merge versions by topic, mark validity status, and name an owner for each knowledge category.
3. Build the baseline. Collect 30 to 50 real questions and write down the required points of each standard answer and the refusal conditions.
4. Retrieval first, generation second. Make "can it find the right passage" verifiable before tuning how answers are composed.
5. Agree acceptance and review. Define the four metrics, the sampling frequency and the post-update regression method, and write them into the delivery documents.

Steps 2 and 3 are mainly done by client-side business staff; the vendor supplies the structure, tools and judgement criteria. Treating them as "the technology side's job" is the most reliable way for a project to get stuck after launch in "the system is fine, it just answers inaccurately".

8. What we usually take on in such projects

We build enterprise AI software; knowledge-base Q&A is rarely a standalone product but one link inside a business process. Based on our published company introduction and project process, our work falls into four stages:

Requirement discovery. Turn "we want an AI Q&A" into an actionable question list: who will ask what, who owns the answer, and what must never be answered automatically. The output is a question taxonomy and a boundary statement, not a feature list.

Solution and demo. We first deliver an interactive demo and an initial implementation path (stage breakdown, rough timeline, work scope) so the client can see how the main flow runs before committing to development. Knowledge projects need this step in particular: the real difficulty of data governance usually only surfaces during the demo.

Development and delivery. The system side covers retrieval, permission filtering, citations, refusal and human handoff, evaluation tooling and the admin console; the business side confirms data definitions and version decisions. This division should be written into the contract and the acceptance documents, so that business judgement is not silently transferred to the vendor after go-live.

After go-live. We help deploy baseline runtime monitoring and alerting, and periodically provide performance reports and optimisation advice. Performance for knowledge-base Q&A is not only response time; it is the distribution of wrong answers and the freshness of material — and that depends on someone on the client side continuing to maintain the knowledge items. Tooling can only make that maintenance visible.

One clarification: the intelligence-library admin console case published on our site is an information collection, retrieval and review system. Its practices — multi-dimensional search, traceable information, role separation and process logging — are relevant to this topic, but it is not an LLM Q&A product and cannot be presented as a delivery case for Q&A projects.

9. Questions we hear often

Should we just switch to a bigger model?
Look at the error distribution first. If most errors are "not found" or "cited an old version", switching models will not help. If citations are correct but the answer drops a condition, miscalculates or overgeneralises, upgrading the model or adjusting the generation strategy is worth trying — but still measure the improvement against the baseline first.

Can we stop it from refusing and always give an answer?
For internal lookup scenarios you can loosen up (provide citations and let people judge for themselves). For customer-facing service, money, legal and safety topics, we do not recommend it. A refusal with a human entry point is better than a confidently wrong answer.

How many people are needed after go-live?
It depends on how often knowledge changes. Companies with frequent policy changes usually need a part-time knowledge administrator spending a few hours a week on new documents and bad cases; where changes are rare, monthly batches are enough. Budget for that effort before go-live.

10. Sources

The company's service scope, project process and after-sales arrangements described in this article come from our public company introduction page and are the company's own statements; the description of the intelligence-library admin console case likewise comes from a public case page. The methodology sections on data governance, retrieval design, refusal and evaluation metrics reflect our judgement and recommendations from project work and are not a commitment to specific outcomes; readers should adapt them to their own data situation and compliance requirements. This article cites no product prices, accuracy figures or third-party benchmark conclusions.