Catching a configuration drift in days, not quarters

When an authorization list in a system drifts from policy, denials for a missing authorization tick up slightly. A 30% rise hides inside daily noise.

Question

How soon would we see a wrong authorization rule?

Lane

Rules and configuration

Basis

Standard practice

What METRIS does

• Track daily denials for a missing authorization by service
• Run a CUSUM detector tuned to the rise that matters
• Check each flagged day against the authorization list in force on the date of service

Daily rate rises from 2.0% to 2.6% on day 120. The CUSUM statistic accumulates the evidence and crosses its threshold on day 132.

Daily requests (n)

300

Baseline rate (p0)

2.0%

Shifted rate (p1)

2.6%, a 30% rise

Threshold (h)

4.4, about one false alarm every two years

Runs

1,000; detector restarted at the change

A quarterly average dilutes a small shift with weeks of normal days. A CUSUM keeps a running likelihood ratio, so evidence builds daily from the first day of drift.

Over 1,000 runs, CUSUM detected the change in a median of 15 days (10th to 90th percentile 7 to 29). A quarterly review waits until the quarter closes: 60 days here, about 45 on average. False alarms differ too: about one every two years for CUSUM, one every five years for a quarterly test.