What happens if an AI application misses its accuracy target after launch?
When performance misses its target, first identify whether the cause is knowledge, retrieval, model behaviour, a business interface, or the evaluation scope. The supplier should fix implementation defects, the client owns authoritative business content, and third-party model changes are handled under the agreed supply-chain terms.
A promise to move an AI system “from 80 to 90” is meaningless without a task-specific definition of accuracy. Before contracting, freeze a representative set of cases and define correct, partially correct, required refusal, and severe error. Set separate pilot and production gates. Otherwise, a post-launch dispute becomes impossible to distinguish from a changed expectation.
When defining model, data, and production boundaries, also compare Why does an application using a large-model API still answer inaccurately or invent facts? and Does my AI agent need a model trained specifically for my business?; the linked guidance adds context that should be considered in the same decision.
| Cause | Diagnostic evidence | Primary owner | Response |
|---|---|---|---|
| Missing, stale, or conflicting authority | No approved source contains the answer | Client business owner | Resolve and approve content; supplier ingests it |
| Parsing, chunking, retrieval, or access defect | Correct source exists but never reaches allowed context | Supplier | Repair to defect severity and regress |
| Model departs from supplied evidence | Correct passage is present but output contradicts it | Supplier leads; model vendor supplies underlying service | Adjust context, prompt, model, or human gate |
| Third-party upgrade or retirement | External behaviour changes under comparable inputs | Shared under contract | Pin, switch, regress, or degrade |
| Business API returns wrong data | Trace points to source field, authorization, or mapping | Interface owner | Correct system/data/mapping, not the prompt |
| New request exceeds agreed scope | Task and source are absent from evaluation | Joint change decision | Add cases, cost, and a new threshold |
The supplier cannot dismiss every defect as model probability; the client cannot add an industry, language, or autonomous action after acceptance and assume the original metric still applies. The contract should cover evaluation-set ownership and sampling, severe-error categories, model and knowledge versions, warranty, included optimization capacity, third-party charges, response times, and termination conditions.
Keep a before-and-after record for every failed case: question, expected answer, knowledge version, retrieved evidence, prompt and model version, tool calls, and actual output. First test whether the evidence can be found, then hold correct evidence constant to test generation, and finally run the whole system. Change one major variable at a time. After a repair, rerun the full frozen set and add the new incident as a regression case.
Metrics depend on the task. Knowledge Q&A needs retrieval recall@k, evidence-supported accuracy, citation consistency, correct refusal, and severe errors. Extraction needs field accuracy, omissions, and human review minutes. Agents need task completion, tool-argument correctness, duplicate or unauthorized actions, and rollback success. ISO/IEC 25059:2023 supplies common concepts for evaluating AI-system quality, but does not generate a universal passing score.
After launch, monitor degradation caused by changing knowledge, user mix, models, and APIs. Set review frequency according to volume, update rate, and consequence. If a severe-error threshold is crossed, disable that capability or return it to human handling. Google Research's ML Test Score supports testing and monitoring across data, models, and infrastructure rather than treating acceptance as a one-off event.
If Wavesteam provides ongoing optimization, each service report should show new failures, attribution, completed repairs, regression results, unresolved risks, and time used. The client must provide a business owner empowered to decide the right answer. When either prerequisite is unavailable, narrow the live scope instead of allowing uncontrolled responses to continue.