METRIS™
Use case 1 · Bayesian audit monitoring
When review becomes a rubber stamp
The pattern
A plan watches how often reviewers agree with the AI. Rising agreement looks like success. It can also mean reviewers have stopped checking.
- Question
- Is clinician review still catching AI errors?
- Lane
- Human review
- Basis
- State law and standard practice
What METRIS does
- Blind re-review of a random monthly sample of AI-assisted denials
- Estimate the catch rate: of the AI's wrong recommendations, how many reviewers overturned
- Alert when the catch rate is credibly below target, with an interval, not a guess
The metric plans watch vs. the one that matters
Agreement with the AI rose from 91.2% to 96.9%, which looks like improvement. Over the same months the true catch rate (dashed) fell from 70% to 15%. Auditing 150 cases a month flagged it in month 8.
Method
- Decisions
- 2,000 a month
- AI error rate
- 10%
- Catch rate
- 70% → 15% from month 5 to 9
- Audit
- Random blind re-review
- Runs
- 1,000 per audit size
Agreement mixes cases where the AI was right with cases where it was wrong, so it rises when reviewers stop catching errors. Only an independent audit separates the two.
How fast an audit finds it
Across 1,000 simulated years, auditing 150 cases a month flagged the decline by month 9 in 95% of runs; auditing 30 flagged it in 48%. Larger audits flag the decline sooner.
Reproducibility
Monthly decisions, AI errors, reviewer catches and audit draws are binomial; posteriors are Beta with a uniform prior. Random seed 20261013; Python with NumPy 2.5.3 and SciPy.
Sources
This pattern is not based on any single record. Its figures come from the model stated above.
A statistical study of a pattern, not of any event. It identifies no organization or person. Simulated figures use synthetic data; published figures are industry-wide.