Drift detection

Distribution drift is one of the primary risks to adaptive medical AI systems in production. AEGIS monitors for drift at every iteration using statistical tests applied to the feature distributions of the golden and drifting datasets. This page covers the methodology, configuration, and calibration of drift detection.

Covariate drift vs concept drift

AEGIS distinguishes between two forms of distribution shift:

Covariate drift is detected directly by comparing D_G and D_D,k feature distributions. Concept drift is detected indirectly through its effect on model performance -- when P(Y|X) changes, performance on the stable golden dataset D_G will degrade.

**Sepsis example:** The sepsis model used 202 features total, but only K=34 key clinical features were monitored for drift. Drift was not introduced uniformly across all iterations. Iterations 1--5 used real Hospital A data with no artificial drift (drift scores 0.000). Iteration 6 was a cross-site scenario (50% Hospital B data, drift score 0.412). Iteration 7 was clean Hospital A data. Iteration 8 used extreme feature scaling (drift score 1.000). Iteration 9 was a clean recovery. Iteration 10 was catastrophic corruption (drift score 1.000, all 34 monitored features significantly shifted). Concept drift was detected indirectly through performance degradation, with sensitivity degrading from 0.723 to 0.428.

Bonferroni-corrected Kolmogorov-Smirnov test

AEGIS recommends the Kolmogorov-Smirnov (KS) test with Bonferroni correction as the default drift detection method. This is Equation 2 from the manuscript.

Method

For each of K features, perform a two-sample KS test comparing the distribution in D_G against D_D,k:

For feature j (j = 1, ..., K):
    H_0: F_G,j = F_D,j    (distributions are identical)
    H_1: F_G,j ≠ F_D,j    (distributions differ)

    Compute p_j = KS_test(D_G[:, j], D_D,k[:, j])

    Feature j has significant drift if p_j < alpha'

Bonferroni correction

To control the family-wise error rate when testing K features simultaneously:

alpha' = alpha / K

This is a conservative correction that reduces the probability of false drift detections at the cost of reduced sensitivity to true drift.

Drift score computation

The drift score is the proportion of features with statistically significant drift:

drift_score = |{j : p_j < alpha'}| / K

The drift score ranges from 0.0 (no features drifted) to 1.0 (all features drifted).

Implementation

from scipy import stats
import numpy as np

def compute_drift_score(golden_data: np.ndarray,
                        drifting_data: np.ndarray,
                        alpha: float = 0.05) -> dict:
    """
    Compute Bonferroni-corrected KS drift score.

    Parameters
    ----------
    golden_data : np.ndarray   Shape (n_golden, K)
    drifting_data : np.ndarray Shape (n_drifting, K)
    alpha : float              Family-wise significance level

    Returns
    -------
    dict  {drift_score, n_drifted, K, alpha_prime, p_values}
    """
    K = golden_data.shape[1]
    alpha_prime = alpha / K

    p_values = []
    n_drifted = 0

    for j in range(K):
        stat, p_val = stats.ks_2samp(golden_data[:, j], drifting_data[:, j])
        p_values.append(p_val)
        if p_val < alpha_prime:
            n_drifted += 1

    return {
        'drift_score': n_drifted / K,
        'n_drifted': n_drifted,
        'K': K,
        'alpha_prime': alpha_prime,
        'p_values': p_values
    }

Feature selection

Not all features need to be monitored for drift. The choice of features affects both the drift score and the Bonferroni correction severity (since K determines alpha').

Selection criteria

**Sepsis example:** The sepsis model used 202 total features, but only 34 key clinical features were selected for drift monitoring: HR, O2Sat, Temp, SBP, MAP, DBP, Resp, EtCO2, BaseExcess, HCO3, FiO2, pH, PaCO2, SaO2, AST, BUN, Alkalinephos, Calcium, Chloride, Creatinine, Bilirubin_direct, Glucose, Lactate, Magnesium, Phosphate, Potassium, Bilirubin_total, TroponinI, Hct, Hgb, PTT, WBC, Fibrinogen, Platelets. With K = 34 and alpha = 0.05, the Bonferroni-corrected threshold was alpha' = 0.05 / 34 = 0.00147. This conservative correction ensured that drift detections were genuine distribution shifts rather than multiple testing artefacts.

Minor and major drift thresholds

The drift score is classified into three severity levels using the theta thresholds:

drift_score < theta_minor,low          → No significant drift
theta_minor,low <= drift_score
    < theta_minor,high                 → Minor drift (informational)
theta_minor,high <= drift_score
    < theta_major                      → Moderate drift (heightened monitoring)
drift_score >= theta_major             → Major drift (triggers A3 alarm)

CDM integration

Drift severity maps to CDM behaviour as follows:

Calibration from baseline variability

To set meaningful drift thresholds, establish a baseline by measuring drift between temporal or random subsets of your own data:

Calibration procedure

1. Split D_T,0 into two random halves (A and B)
2. Compute drift_score(A, B) -- this represents natural variability
3. Repeat 100 times with different random splits
4. Build the baseline distribution of drift scores

def calibrate_drift_thresholds(training_data: np.ndarray,
                                n_bootstrap: int = 100,
                                alpha: float = 0.05) -> dict:
    """
    Calibrate drift thresholds from baseline variability.

    Returns suggested theta_minor_low, theta_minor_high, theta_major.
    """
    n = training_data.shape[0]
    baseline_scores = []

    for _ in range(n_bootstrap):
        idx = np.random.permutation(n)
        half = n // 2
        A = training_data[idx[:half]]
        B = training_data[idx[half:2*half]]
        result = compute_drift_score(A, B, alpha)
        baseline_scores.append(result['drift_score'])

    baseline_scores = np.array(baseline_scores)

    return {
        'theta_minor_low': float(np.percentile(baseline_scores, 75)),
        'theta_minor_high': float(np.percentile(baseline_scores, 95)),
        'theta_major': max(float(np.percentile(baseline_scores, 99)), 0.90),
        'baseline_mean': float(baseline_scores.mean()),
        'baseline_std': float(baseline_scores.std()),
        'baseline_max': float(baseline_scores.max())
    }

5. Set thresholds based on percentiles:

Alternative drift detection methods

While KS + Bonferroni is the default, AEGIS supports other methods:

Document your chosen method and justification in the S2 configuration form.

Concept drift proxy at P2

AEGIS detects concept drift indirectly through the P2 condition. P2 is CLINICAL REVIEW, triggered when a metric regresses beyond tolerance tau from the fixed performance reference P^ref. This regression may indicate concept drift indirectly:

IF any_metric degraded > tau from P^ref (fixed performance reference):
    → CLINICAL REVIEW (P2): regression from reference, concept drift may be occurring

This proxy approach avoids the need for labelled data in the drifting dataset (which may not be immediately available in production). Note that major drift is detected separately via A3 ALARM, which fires independently of the P2 condition.

Source

Manuscript Section II-D (drift detection configuration within instantiation parameters) and Equation 2 (Bonferroni-corrected KS test). Drift threshold calibration methodology is described in Supplementary S2.5. Feature selection guidance and the concept drift proxy are discussed in Section II-B-2 (MMM specification). The sepsis feature list and drift progression across iterations are detailed in Section IV-C.