Interactive visualisations

A cross-validation estimate is only as good as the split behind it. This page shows how each trustcv splitter divides rows into training and test folds, and what goes wrong when the split does not match how the data were collected.

How splitters divide data

Pick a data structure and a method. Each row of the diagram is one fold (or one fit); each square is one row of data. Training rows are solid, test rows are hatched, and the legend under the diagram names every other state.

Data structure Independent rows Grouped (patients, sites) Time-ordered Spatial

Method

Folds:

5

The diagrams are schematics on small made-up datasets (30 rows, ten patients, or a 12 by 12 map). The fold patterns follow the split logic in trustcv/splitters/ with the parameters named under each diagram; rows are drawn in fold order where that is easier to read. Run the splitter on your own data to see its exact folds, as in the example below.

What to look for:

Data structure	The rule the split must keep	Typical splitter
Independent rows	Test rows are new rows; keep the class balance	StratifiedKFoldMedical
Grouped	No patient (or site) on both sides	StratifiedGroupKFold, LeaveOneGroupOut
Time-ordered	Never train on rows recorded after the test rows	TimeSeriesSplit, PurgedGroupTimeSeriesSplit
Spatial	Test locations are not surrounded by training neighbours	SpatialBlockCV, BufferedSpatialCV

Two methods in the time-ordered list train on data from after the test block: PurgedKFoldCV and CombinatorialPurgedCV. Purging removes the rows right next to the test block, but the estimate is not forward-only. When the question is "how will this model do on future patients", use a forward-only splitter. EnvironmentalHealthCV, which adds environmental variables to spatial and temporal blocks, is not drawn here; see the API reference.

Print the folds of a real splitter

The same pictures, as text, from the actual splitters. # is test, t is training and . is a row the fold does not use.

import numpy as np
from trustcv import GroupKFoldMedical, PurgedKFoldCV, TimeSeriesSplit

def show(cv, X, **split_args):
    for train, test in cv.split(X, **split_args):
        print("".join("#" if i in test else "t" if i in train else "." for i in range(len(X))))

X = np.zeros((30, 1))
show(TimeSeriesSplit(n_splits=4), X)                 # forward-only, growing training set
print()
show(PurgedKFoldCV(n_splits=4, purge_gap=2), X)      # trains on both sides of the test block
print()
patients = np.repeat(np.arange(10), 3)               # ten patients, three rows each
show(GroupKFoldMedical(n_splits=5, random_state=0), X, groups=patients)

Simulated scenarios

Simulated These scenarios are illustrations, not experiments. Pressing Run simulation draws five fold scores at random around values chosen to show the usual direction of the effect: a split that leaks scores higher than one that matches the question. No model is trained and no dataset is used, and the size of a real gap depends on your data. For real numbers, run the notebooks or the tutorial example.

Scenario

Validation approach

Run simulation

Some approaches in the scenarios are common in their fields but are not trustcv classes: ancestry-stratified, batch-aware, scaffold and target-based splits. They are labelled "concept". You implement them by building a group label (batch, scaffold or target ID) and passing it to a grouped splitter, for example StratifiedGroupKFold(n_splits=5).split(X, y, groups=scaffold_ids).

The related code links point to scripts in the repository: small ones ship with the package in trustcv/examples/, heavier ones (deep learning, spatial models) live in examples/.

Next steps

• Choosing a split turns these rules into a decision for your data.
• Tutorials work through each case with code that runs.
• Quiz tests the ideas on twenty clinical scenarios.