METRIS™
METRIS · Use cases · US health plans
Patterns behind denial problems at US health plans, and what measuring them would show
Six recurring risks in AI-assisted prior authorization, each treated as a measurement problem. Every card is a study: a model of the risk, a method that detects it, and the figures that show how fast, at what cost, and with what error.
Is clinician review still catching AI errors?
Agreement rose from 91% to 97% while the true catch rate fell from 70% to 15%. Human review Use case 2Acceptance samplingHow many vendor notices should the plan read each week?
For a 95% chance of catching a 3% error rate, read 99 notices a week; for 1%, 299. Delegated vendor Use case 3CUSUM change detectionHow soon would we see a wrong authorization rule?
A 2.0% → 2.6% rise is detected in a median of 15 days; a quarterly review waits for the quarter to close. Rules and configuration Use case 4Sampling error analysisHow much can a sample of files tell an examiner?
With 60 files, a true 20% violation rate is estimated anywhere from 10% to 30%. Human review Use case 5Sensitivity analysis on national dataHow many denials would fail if challenged?
3.6 million denials were never appealed; if even 10% would fail, that is 0.36 million. All lanes Use case 6Shift-share decompositionWhy did our posted overturn rate move?
The rate rose 5.7 points; 4.8 of them trace to one service. All lanesEach use case is a statistical study of a recurring risk pattern, not a report on any event. No organization, person, place or date from any real case appears here. Figures marked as simulations use synthetic data with a fixed random seed, so every number can be reproduced. Figures marked as published data use industry-wide public sources cited on each page. METRIS has no relationship with any organization in the underlying public record. "What METRIS does" describes what a measurement program would track; it does not claim any outcome would have been prevented.
METRIS · Use cases · Small and midsize businesses
When a small business runs on AI, what does measuring it show?
Six recurring risks for any business that puts AI into its daily work, in any industry. Every card is a study: a model of the risk, a method that detects it, and the figures that show how much checking is enough.
How often does an AI-drafted document carry a bad citation?
At a 3% per-citation error rate, 26% of 10-citation documents contain at least one bad citation. Human review Use case 2Power and false-alarm analysisCan a small business trust the four-fifths rule?
With 40 applicants per group and no bias at all, the four-fifths rule still flags 31% of pools. Hiring Use case 3Zero-failure test boundsHow many tests before a chatbot can be trusted with customers?
After 50 passing tests the true failure rate could still be 5.8%; after 300, 1.0%. Customer-facing AI Use case 4Bayesian accuracy monitoringWould we notice if an AI tool's answers got worse?
A drop from 95% to 85% accuracy is caught in 93% of runs with 40 questions a month, a median of 2 months after the update. Advice relied on Use case 5Exact confidence boundsHow much testing backs an accuracy claim?
At 97% observed accuracy, it takes about 340 test cases before 'at least 95%' is supported. Claims about AI Use case 6Exposure mappingWhere is our business data going into AI tools?
In a simulated month of prompts, 3 of 21 tool and data pairs carry 62.4% of weighted exposure. Data lossEach use case is a statistical study of a recurring risk pattern, not a report on any event. No organization, person, place or date from any real case appears here. Figures marked as simulations use synthetic data with a fixed random seed, so every number can be reproduced. Figures marked as published data use industry-wide public sources cited on each page. METRIS has no relationship with any organization in the underlying public record. "What METRIS does" describes what a measurement program would track; it does not claim any outcome would have been prevented.