Cross-validation you can defend¶
trustcv chooses a cross-validation split that matches how your data was collected, checks the folds you actually scored for leakage, and reports what it could and could not verify. Built for medical and regulated machine learning; works with scikit-learn, PyTorch, TensorFlow, MONAI and JAX.
The right split¶
32 methods (the 29 in the paper plus later additions) for independent, grouped, temporal, spatial and multilabel data. recommend_cv picks one from how your data is structured.
Leakage checks¶
Group overlap, duplicates, target proxies, clustered rows and identifier columns, checked on the folds that were scored.
Honest intervals¶
Confidence intervals that account for overlapping training sets (corrected_t) or resample whole patients (oof_bootstrap).
A status you can read¶
Every result ends in one overall status, and each check says what it established and what it could not see.
A first validation¶
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from trustcv import TrustCV, recommend_cv
X, y = load_breast_cancer(return_X_y=True)
plan = recommend_cv(X, y) # independent rows here, so stratified K-fold
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
result = TrustCV(method=plan.method, declare_independent_samples=True).validate(
model=model, X=X, y=y, cv=plan.splitter
)
print(result.overall_status) # PASSED
print(result.summary())
Preprocessing sits inside the Pipeline, so it is refit in every fold. Rows that share a patient need groups=: see the quickstart.
What the result tells you¶
| Overall status | Meaning |
|---|---|
| PASSED | Every applicable check passed. Not proof that no leakage exists: some kinds cannot be seen from the data. |
| NEEDS_REVIEW | Signs of possible leakage that cannot be proven automatically. A person should confirm them. |
| NOT_FULLY_VERIFIED | Nothing suspicious, but something could not be checked (for example, no patient IDs were given). |
| FAILED | A check found leakage, such as the same patient in training and test. |
Learn more¶
Choosing a split¶
Match the split to repeated patients, time order, sites and geography.
Data leakage¶
What each check catches, and the leaks no check can see.
Deep learning¶
Runners for PyTorch, TensorFlow, MONAI and JAX, and early stopping without touching the test fold.
Documenting validation¶
Record split choices and uncertainty for clinical and regulatory review.
Podcast¶
The reproducibility problem in medical AI · 21:50 · AI-generated audio about data leakage and validation in clinical settings.
Developed at SMAILE, Karolinska Institutet's core facility for AI. trustcv supports documentation of validation choices; it does not make a model or device compliant with any regulation.