What it takes to say 'at least 95% accurate'

The pattern

An accuracy claim is a statistical claim. Observing 97% on a small test set does not support saying 'at least 95%'.

Question: How much testing backs an accuracy claim?
Lane: Claims about AI
Basis: Federal enforcement and standard practice

What METRIS does

Keep the test set, results and date behind every AI claim
Compute the exact lower confidence bound for each claim
Retire or reword claims the bound no longer supports

Lower bound when 97% of answers are right

Exact calculation · assumed parameters

With 97% of answers right, the exact 95% lower bound first reaches 95% at about 340 test cases. With 100 cases the honest claim is closer to 92%.

Method

one-sided Clopper–Pearson lower bound: L = Beta⁻¹(0.05; k, n − k + 1) claim 'at least c' is supported when L ≥ c

Observed accuracy: 97%
True accuracy (power): 97%
Claims: At least 90%, at least 95%
Test cases (n): 20 to 1,000

The bound is what a careful reviewer would compute. Publishing the number with its test size turns a marketing claim into a defensible one.

Chance a test supports the claim

Exact calculation · assumed parameters

If true accuracy is 97%, the chance a test supports 'at least 95%' stays low until several hundred cases; 'at least 90%' is supportable far sooner.

Reproducibility

Exact binomial and Beta quantiles; no random draws. Python with NumPy 2.5.3 and SciPy.

Sources

This pattern is not based on any single record. Its figures come from the model stated above.

Study design and analysis: Suneeta Modekurty, SANJEEVANI AI

A statistical study of a pattern, not of any event. It identifies no organization or person. Simulated figures use synthetic data; published figures are industry-wide.