Skip to content

Notebooks

The repository has fifteen Jupyter notebooks in notebooks/. They range from a tour of the independent-sample splitters to full studies on public clinical data. Most use scikit-learn's breast cancer dataset or synthetic data; notebooks 09 and 12 download real public datasets.

In Colab, install trustcv first

Colab does not come with trustcv. Before running a notebook, add a cell at the top with

!pip install trustcv

and run it. Several notebooks have no install cell of their own (the "Installs trustcv" column below), and they fail at the first import trustcv without it. Notebooks that need extra libraries say so near the top.

Some notebooks were written for earlier releases (from 1.0.5 to 1.1.0) and their text mentions those versions. If a cell fails with the current release, please open an issue.

All notebooks

# Notebook What it covers Data Installs trustcv
01 IID methods showcase Each independent-sample splitter on its own: hold-out, K-fold, stratified, repeated, LOOCV, LPOCV, Monte Carlo, bootstrap and nested CV Breast cancer Yes
02 Advanced workflow: UniversalCVRunner The lower-level pieces: UniversalCVRunner, ClinicalMetrics and reporting Breast cancer From GitHub, if missing
03 TrustCVValidator showcase Hold-out across random seeds, stratified against non-stratified; scores, intervals and checks Breast cancer No
04 TrustCVValidator: IID comparison Every independent-sample method through TrustCVValidator, with summaries side by side Breast cancer No
05 Cross-validation comparison trustcv next to plain scikit-learn and other libraries (XGBoost, LightGBM, H2O, PyCaret, Keras, PyTorch) Breast cancer, small synthetic set No; also needs the other libraries
06 Deep learning showcase with glossary Keras models through UniversalCVRunner and KerasSkWrap, with a glossary of terms. TensorFlow only MNIST No (the install line is commented out); needs TensorFlow
07 Grouped CV for medical AI Every grouped splitter on a multi-site clinical dataset, with leakage checks Synthetic multi-site Yes, from GitHub
08 Structured scenarios: temporal, spatial, hierarchical How the right split changes when data are time-ordered, spatially correlated or nested Synthetic From GitHub, if missing
09 UCI HAR: grouped leakage and nested CV Sitting against standing: record-wise K-fold compared with subject-aware and nested validation UCI HAR (downloaded) Optional setting in the first cell
10 Advanced workflow: clinical metrics A custom run with UniversalCVRunner, clinical metrics (likelihood ratios, NNT) and an HTML report Wine, breast cancer No
11 Spatial methods with split_kwargs The four spatial splitters through UniversalCVRunner.run(split_kwargs=...) Synthetic environmental health Optional setting in the first cell
12 PhysioNet 2019 sepsis Grouped, temporal and cross-hospital validation on real ICU data from two hospital systems. Long-running; start with its smoke-test mode PhysioNet Challenge 2019 (downloaded) Optional setting in the first cell
13 Leakage checker: injection tests Controlled injections of patient overlap, duplicates and temporal violations, to test where the checker's rules fire Synthetic Optional setting in the first cell
14 LOGO metric feasibility warning Why fold-wise AUC can be undefined in leave-one-group-out, and pooled out-of-fold metrics instead Synthetic, six patients No
15 Trust claims Feature selection before CV reported honestly, the Pipeline fix, and recommend_cv Breast cancer No

"Optional setting" means the notebook has an install switch in its first cells; set it to install from PyPI or GitHub, or run !pip install trustcv yourself.

A suggested order

  1. Start with 01 and 15: the splitters, and why honest checks matter.
  2. Then 03 and 04 for the high-level validator, 07 for grouped data and 08 for temporal, spatial and hierarchical data.
  3. For real data, 09 (wearable sensors) and 12 (ICU).
  4. 02, 10, 11 and 06 cover the lower-level runner and deep learning; 05, 13 and 14 go into specific questions.

Run them locally

git clone https://github.com/ki-smile/trustcv.git
cd trustcv
pip install -e ".[all]"
pip install jupyterlab
jupyter lab notebooks/

Notebooks run from inside the repository pick up the checked-out source. See Install for the optional extras.