METRIS™ Discuss an AI decision →
← All use cases

Use case 1 · Error propagation analysis

Small error rates, many bad documents

The pattern

An AI tool that gets 97% of citations right still produces documents with errors, because every document carries many citations.

Question
How often does an AI-drafted document carry a bad citation?
Lane
Human review
Basis
Court records and standard practice

What METRIS does

  • Check every citation, figure and name in AI-drafted work against its source before it leaves
  • Estimate the tool's per-item error rate from audits, with an interval
  • Report the share of documents sent with an unchecked item

Chance a document has at least one bad citation

Exact calculation · assumed parameters
Probability of at least one bad citation by citations per document1% per citation3% per citation5% per citation0%25%50%75%100%151015202530Citations per document26% at 10 citations

At a 3% error rate per citation, 26% of documents with 10 citations contain at least one bad one; at 5%, 40%. Per-item accuracy hides per-document risk, assuming errors occur independently.

Method

P(document has ≥ 1 bad item) = 1 − (1 − e)^k e estimated from audits with a 95% Wilson interval
Per-citation error (e)
1%, 3%, 5%
Citations per document (k)
1 to 30
Audit sizes
25 to 800 citations

Vendors report accuracy per answer; the business is exposed per document. The conversion is simple, and it changes the decision about what to check.

How many items to audit to know the error rate

Exact calculation · assumed parameters
95% interval for the error rate by audit sizeObserved rate, 3% true rate95% Wilson interval0%5%10%15%20%2550100200400800Citations audited (log scale)Estimated error rate, 95% intervalTrue rate 3%

Finding 1 bad citation in 25 audited puts the rate anywhere from 0.7% to 19.5%. Auditing 800 (24 bad) narrows it to 2.0% to 4.4%.

Reproducibility

Closed-form calculations; no random draws. Python with NumPy 2.5.3 and SciPy.

Sources

This pattern is not based on any single record. Its figures come from the model stated above.

Study design and analysis: Suneeta Modekurty, SANJEEVANI AI

A statistical study of a pattern, not of any event. It identifies no organization or person. Simulated figures use synthetic data; published figures are industry-wide.