METRIS™ Discuss an AI decision →
← All use cases

Use case 4 · Bayesian accuracy monitoring

Watching an AI tool through a vendor update

The pattern

AI tools change when vendors update them. A fixed set of questions with known answers, asked every month, shows when accuracy drops.

Question
Would we notice if an AI tool's answers got worse?
Lane
Advice relied on
Basis
Reporting and standard practice

What METRIS does

  • Ask a fixed bank of questions with known correct answers every month
  • Estimate accuracy with an interval and alert when it is credibly below target
  • Hold decisions that rely on the tool while an alert is open

One run: accuracy through a vendor update

Simulation study · synthetic data
Estimated accuracy by month with 95% interval60%70%80%90%100%1357912MonthVendor updateTarget 90%Alert

Accuracy drops from 95% to 85% at the month-5 update. With 40 questions a month, the alert fires in month 6.

Method

accuracy θ | data ~ Beta(1 + correct, 1 + wrong) (2-month window) alert when P(θ < 0.90 | data) > 0.95
Accuracy before update
95%
After update
90%, 85% or 80%
Questions per month
40 or 100
Runs
1,000 per setting

One wrong answer proves little. A posterior probability says how sure we are that accuracy fell below what the business needs, and it updates every month.

Detection by size of drop and question bank

Monte Carlo · synthetic data
Probability of detection within the year40 questions · accuracy to 90%25% detected40 questions · accuracy to 85%93% detected40 questions · accuracy to 80%99.5% detected100 questions · accuracy to 90%33% detected100 questions · accuracy to 85%99.8% detected100 questions · accuracy to 80%100% detected1,000 runs each. False alarms before the update: at most 0.5%.

A drop to 85% is detected in 93% of years with 40 questions a month and 99.8% with 100. A drop to exactly the 90% target is still flagged in 25% and 33% of years, because each monthly check adds a chance of a false alarm.

Reproducibility

Correct answers are binomial; posteriors are Beta with a uniform prior. Random seed 20261013; Python with NumPy 2.5.3 and SciPy.

Sources

This pattern is not based on any single record. Its figures come from the model stated above.

Study design and analysis: Suneeta Modekurty, SANJEEVANI AI

A statistical study of a pattern, not of any event. It identifies no organization or person. Simulated figures use synthetic data; published figures are industry-wide.