Four of the Seven Steps in Measuring AI Belong to the Business

When a health plan asks whether its prior authorization AI is working, the question usually lands on the data science team's desk. I am not convinced that is where most of the answer lives. Seven steps turn a business goal into a number someone can act on. The business owns four of them, engineering owns one, and data science owns two.

The vice president of utilization management at a regional health plan has a problem. Prior authorization requests wait days in a queue. Physicians' offices call to ask where a request stands, and patients wait for care that will almost certainly be approved. The plan puts an AI model at the front of the queue. When a request clearly meets the plan's clinical criteria, the model approves it. Every other request goes to a clinician, exactly as before. The vice president states what they need in one sentence: decide requests faster without approving care that does not meet clinical criteria.

That sentence is the requirement. It is not yet a measurement. These are the seven steps between the two:

1. Requirement (business owner). Decide prior authorization requests faster without approving care that does not meet clinical criteria.
2. Measurable condition (business owner, with the clinical quality lead). Of the auto-approvals that receive a clinical audit, no more than 2 percent fail the criteria, in each window of 300 audited requests.
3. Observable signal (engineering). Each auto-approval records the model version that made it, and each audit records whether the request met the criteria.
4. Measurement function (data science). Audited auto-approvals that failed the criteria, divided by all audited auto-approvals in the window.
5. Statistical model (data science). The probability that a window stays at or below 2 percent, with an interval around it.
6. Decision threshold (business owner). Pass, watch or fail.
7. Action (business owner). Expand auto-approval to more services, keep it under review, or roll it back.

The business owns the beginning and the end

The four business steps are the ones that carry judgment. A data scientist can calculate a failure rate to three decimal places. A data scientist cannot decide whether 2 percent is acceptable for imaging when the plan might accept only 1 percent for specialty drugs. When the vice president leaves steps 2 and 6 to the analysts, the analysts pick a number to get the work moving, and that number quietly becomes the plan's tolerance without anyone with authority ever choosing it.

The quiet break is at step 3

A clinician audits an auto-approval days or weeks after the model made it. Unless engineering links each audit result back to the request and to the model version that approved it, the failure rate cannot be calculated for that model at all. The cost shows up the first time the vendor updates the model in the middle of a quarter. Without the link, nobody at the plan can say whether the update made auto-approvals better or worse, and the plan keeps running a model it can no longer evaluate.

Without all seven, the next decision is an opinion

Six months after launch, the medical director will ask to extend auto-approval from imaging to physical therapy. With all seven steps defined, the answer is a failure rate with an interval, read against a threshold the vice president set in advance. With any step missing, the answer comes from whoever argues most confidently in the meeting.

What to do this month

Pick one AI system your organization runs and write down the business problem it was meant to solve. Then fill in the other six steps and name the owner of each. Wherever a step has no owner, the measurement cannot exist yet, and you now know who to ask.