Why does an application using a large-model API still answer inaccurately or invent facts?
Do not replace the model first. Separate the system into source, retrieval, context, generation, and business-tool layers, then replay the same failed cases through each layer. That is how you find where an unsupported answer was introduced.
“Inaccurate” can describe very different failures: the approved material never contained the answer; the system failed to retrieve it; the right evidence was present but the model did not follow it; or the prose was correct while the application queried the wrong customer or order. Prompt tuning or a larger model may improve a demo while leaving versioning, permissions, and retrieval defects untouched.
When defining model, data, and production boundaries, also compare What happens if an AI application misses its accuracy target after launch? and How can AI improve a document-intensive professional workflow?; the linked guidance adds context that should be considered in the same decision.
| Symptom | Inspect first | Repeatable check | Typical remedy |
|---|---|---|---|
| Old price or policy | Source ownership and version | Compare against the effective approved answer | Retire old content; add owner and expiry |
| Existing answer is missed | Parsing, chunking, indexing, retrieval | Did the correct passage enter top-k? | Repair OCR/metadata; use hybrid retrieval or reranking |
| Evidence is found but extra facts appear | Context and generation instruction | Is each claim supported by the supplied passages? | Reduce noise, require citations, refuse without evidence |
| Entities are mixed after several turns | Session state and identity | Replay turns and user switches | Isolate sessions and reconfirm the target |
| Wrong order or duplicate action | Tool arguments, authorization, idempotency | Inspect sandbox inputs and receipts | Server-side checks, confirmation, idempotency, rollback |
Build a replayable incident set from production failures. Preserve the user's question, knowledge version, retrieved passages, full context, model and prompt version, tool arguments, and final output. Sample across common categories, failure patterns, and high-consequence cases; keep adding newly discovered defects. A business owner should specify the acceptable answer, mandatory facts, and severe-error definition. If no authoritative answer exists, classify it as a knowledge-governance gap rather than a model failure.
For each case, run three tests. First, can a person find the answer in the approved repository? Second, given the correct evidence, can the model produce a supported answer? Third, does the full retrieval-and-tool pipeline succeed? This separates retrieval recall, generation faithfulness, and integration defects. The NeurIPS 2020 RAG paper describes retrieval-augmented generation, but retrieval cannot correct a false source or guarantee that a model will use the evidence faithfully.
More context is not necessarily better. Whole manuals may contain obsolete versions and irrelevant text. The Lost in the Middle study found that a model's use of relevant information changes with its position in a long context. Test chunk boundaries, headings and tables, hybrid keyword/semantic recall, metadata filters, and reranking instead of merely increasing top-k. Permission filters must run server-side before retrieval; showing sensitive text to a model and asking it not to repeat it is not access control.
Report retrieval recall@k, context precision, evidence-supported accuracy, citation consistency, correct refusal, and severe-error rate separately. The RAGAS paper likewise treats context, faithfulness, and answer quality as distinct dimensions. Automated judges can triage volume, but a business reviewer must decide high-risk errors.
Every source, parser, retrieval, prompt, or model change should rerun the frozen cases and add the new regression. Release gates must describe sampling and risk categories, not just claim “95% accuracy”; latency, handoff, and full task cost also matter. Wavesteam should deliver the failure taxonomy, replay set, layer-specific metrics, and before/after evidence. If the client cannot identify authoritative material and an answer owner, the correct diagnosis is a knowledge-governance gap, and expansion should pause.