API reference¶

Generated from the docstrings of the installed source. Import everything from the top-level package, e.g. from trustcv import TrustCV, StratifiedGroupKFold; deep-learning runners live in trustcv.frameworks.* and need their framework installed.

Validation¶

TrustCVValidator ¶

TrustCVValidator(method: str = 'stratified_kfold', n_splits: int = 5, random_state: int = 42, shuffle: bool = True, check_leakage: bool = True, check_balance: bool = True, declare_independent_samples: bool = False, target_auc_threshold: float = 0.99, target_correlation_threshold: float = 0.99, max_target_features: int = 10000, permutation_check: bool = False, n_permutations: int = 20, compliance: Optional[str] = None, *, metrics: Optional[List[str]] = None, return_confidence_intervals: bool = True, ci_level: float = 0.95, ci_method: str = 'corrected_t', n_bootstrap: int = 1000, holdout_test_size: Union[float, int] = 0.2, holdout_stratify: bool = False, repeated_kfold_repeats: int = 1, repeated_kfold_stratify: bool = False, n_repeats: Optional[int] = None, lpocv_p: int = 2, p: Optional[int] = None, monte_carlo_iterations: int = 50, n_iterations: Optional[int] = None, iterations: Optional[int] = None, monte_carlo_test_size: Union[float, int] = 0.2, mc_test_size: Optional[Union[float, int]] = None, bootstrap_validation_iterations: int = 200, bootstrap_iterations: Optional[int] = None, bootstrap_validation_estimator: str = 'standard', bootstrap_estimator: Optional[str] = None, test_size: Optional[Union[float, int]] = None, stratify: Optional[bool] = None, hierarchy_level: str = 'patient', warn_metric_feasibility: bool = True, check_independence: bool = True, target_strong_threshold: float = 0.9, max_audit_folds: int = 10)

Main validator class for medical machine learning

Features: - Automatic patient-level splitting - Data leakage detection - Clinical metrics calculation - Regulatory compliance reporting

Initialize TrustCV Validator

Parameters:

method : str Cross-validation method ('kfold', 'stratified_kfold', 'patient_grouped_kfold', 'temporal', 'holdout', 'repeated_kfold', 'repeated_group_kfold', 'hierarchical_group_kfold', 'loocv', 'lpocv', 'monte_carlo', 'bootstrap') n_splits : int Number of CV folds random_state : int Random seed for reproducibility shuffle : bool Whether to shuffle data when using k-fold style splitters (default: True) check_leakage : bool Whether to check for data leakage check_balance : bool Whether to check class balance declare_independent_samples : bool Declare that rows are independent when no group identifiers exist. This makes group leakage explicitly not applicable. target_auc_threshold : float Binary univariate ROC-AUC threshold for target-leakage warnings. target_correlation_threshold : float Absolute Spearman threshold for regression target-leakage warnings. max_target_features : int Maximum number of feature columns scanned for target leakage. permutation_check : bool Whether to rerun CV with shuffled labels as a CV-loop sanity check. n_permutations : int Number of shuffled-label CV runs. compliance : str Report label used by fit_validate ('FDA' writes a JSON documentation file, 'CE' prints a placeholder, None does nothing). It does not check or establish regulatory compliance, and validate() ignores it. holdout_test_size : float or int Fraction or absolute count for hold-out validation when method='holdout' holdout_stratify : bool If True, enables stratified hold-out splitting (uses y labels) repeated_kfold_repeats : int Number of repetitions when method='repeated_kfold' or method='repeated_group_kfold' (alias: n_repeats) repeated_kfold_stratify : bool Whether to use stratified repeats (alias: stratify when method='repeated_kfold') lpocv_p : int Number of samples to leave out for method='lpocv' (alias: p) monte_carlo_iterations : int Random split iterations for method='monte_carlo' (aliases: n_iterations, iterations) monte_carlo_test_size : float or int Fraction or count for Monte Carlo test splits (alias: mc_test_size; test_size also handled) bootstrap_validation_iterations : int Number of bootstrap resamples when method='bootstrap' (alias: bootstrap_iterations) bootstrap_validation_estimator : str Bootstrap estimator variant ('standard', '.632', '.632+') (alias: bootstrap_estimator) test_size : float or int, optional Alias for holdout_test_size (and, when using Monte Carlo splits, monte_carlo_test_size) stratify : bool, optional Alias for holdout_stratify to toggle stratified hold-out splitting hierarchy_level : str, default="patient" Group hierarchy level used when method='hierarchical_group_kfold'.

validate ¶

validate(*, model, X: Union[ndarray, DataFrame], y: Union[ndarray, Series], patient_ids: Optional[Union[ndarray, Series]] = None, groups: Optional[Union[ndarray, Series]] = None, cv: Optional[BaseCrossValidator] = None, leakage_checker: Optional[Any] = None, sample_weight: Optional[ndarray] = None, metrics: Optional[List[str]] = None, scoring: Optional[Dict[str, Any]] = None, ci_cluster: bool = True) -> ValidationResult

Run cross-validation with the requested metrics and return a ValidationResult.

Parameters:

Name	Type	Description	Default
`model`	`estimator implementing fit/predict`		required
`X`	`array - like`		required
`y`	`array - like`		required
`patient_ids`	`array - like`	Alias for groups; convenient when using patient-level splitters	`None`
`groups`	`array - like`	Explicit group labels for the cross-validator.	`None`
`cv`	`cross - validator`	If None, uses the validator's configured splitter	`None`
`leakage_checker`	`DataLeakageChecker`	If provided, runs leakage checks on the dataset	`None`
`sample_weight`	`array - like`	Per-sample weights passed to estimator.fit when supported	`None`
`scoring`	`dict`	sklearn-style scorers mapping name -> scorer (string for get_scorer, callable scorer, or make_scorer output). When provided, overrides the built-in metrics list for this validation call.	`None`
`metrics`	`list of str`	Per-call metric override (e.g., ["accuracy", "f1"]). Ignored if `scoring` is provided.	`None`

fit_validate ¶

fit_validate(model, X: Union[ndarray, DataFrame], y: Union[ndarray, Series], patient_ids: Optional[Union[ndarray, Series]] = None, timestamps: Optional[Union[ndarray, Series]] = None, scoring: Optional[Dict[str, Any]] = None) -> ValidationResult

Perform medical cross-validation

Parameters:

model : sklearn estimator Model to validate X : array-like Features y : array-like Labels patient_ids : array-like, optional Patient identifiers for grouped splitting timestamps : array-like, optional Timestamps for temporal validation scoring : dict, optional Scoring metrics

Returns:

ValidationResult object with comprehensive metrics

suggest_best_method ¶

suggest_best_method(X: Union[ndarray, DataFrame], y: Union[ndarray, Series], patient_ids: Optional[Union[ndarray, Series]] = None, timestamps: Optional[Union[ndarray, Series]] = None) -> str

Return a compatible method string; prefer recommend_cv for details.

This legacy wrapper cannot return the recommended splitter, warnings, rationale, or copyable code. Use trustcv.recommend_cv for those fields.

ValidationResult `dataclass` ¶

ValidationResult(scores: Dict[str, ndarray], mean_scores: Dict[str, float], std_scores: Dict[str, float], confidence_intervals: Dict[str, Tuple[float, float]], fold_details: List[Dict], leakage_check: Dict[str, Any], recommendations: List[str], diagnostics: Dict[str, Any] = dict(), checks: Dict[str, CheckResult] = default_checks(), overall_status: str = '', permutation: Dict[str, Any] = dict(), ci_method: str = '', ci_level: float = 0.95)

Results from medical cross-validation.

Notes

leakage_check['has_leakage'] is a deprecated inverted legacy key: True means the external detector did not find leakage. Use checks or leakage_check['external_leakage_detected'] instead.

metric_feasibility_warnings `property` ¶

metric_feasibility_warnings: List[str]

Warnings from fold-level metric feasibility diagnostics, if available.

summary ¶

summary() -> str

Generate summary report

dashboard ¶

dashboard(title: str = 'TrustCV Results', save_path: Optional[str] = None) -> None

Display an interactive multi-panel Plotly dashboard.

Opens a rich visual summary equivalent to summary() but rendered as interactive charts.

Parameters:

Name	Type	Description	Default
`title`	`str`	Title shown at the top of every figure.	`'TrustCV Results'`
`save_path`	`str or None`	If given, the dashboard is written to this path as a self-contained HTML file (e.g. `save_path="report.html"`). When None (default) nothing is written to disk.	`None`

to_dict ¶

to_dict() -> Dict

Convert to dictionary for JSON export

CheckResult `dataclass` ¶

CheckResult(name: str, status: str, message: str, details: dict = dict())

Result of one TrustCV integrity check.

Parameters:

Name	Type	Description	Default
`name`	`str`	Stable machine-readable check name.	required
`status`	`str`	One of `PASSED`, `FAILED`, `WARNING`, `NOT_CHECKED`, `NOT_APPLICABLE`, `INFO`, or `ERROR`.	required
`message`	`str`	Honest one-line description of what was or was not established.	required
`details`	`dict`	Supporting values for programmatic inspection.	`dict()`

Notes

A PASSED result covers only the condition named by this check. It cannot establish that no unmeasured source of leakage exists.

recommend_cv ¶

recommend_cv(X: Any, y: Any, *, groups: Any = None, timestamps: Any = None, coordinates: Any = None, n_splits: int = 5, random_state: int = 42) -> CVRecommendation

Recommend a runnable CV splitter from supplied dependence metadata.

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Feature matrix and target.	required
`y`	`array - like`	Feature matrix and target.	required
`groups`	`array - like`	Subject, patient, site, or other repeated-unit identifiers.	`None`
`timestamps`	`array - like`	Observation times. Their presence takes precedence over spatial and grouped recommendations.	`None`
`coordinates`	`array - like`	Two-dimensional spatial coordinates.	`None`
`n_splits`	`int`	Requested fold count.	`5`
`random_state`	`int`	Seed for splitters that shuffle.	`42`

Returns:

Type	Description
`CVRecommendation`	Category, method, runnable splitter, rationale, warnings, and code.

Notes

The advisor can respond only to metadata supplied here. It cannot discover hidden repeated subjects, temporal ordering, spatial autocorrelation, or leakage from preprocessing performed before the call.

CVRecommendation `dataclass` ¶

CVRecommendation(category: str, method: Optional[str], splitter: Any, rationale: str, warnings: List[str] = list(), code: str = '')

Actionable recommendation for a cross-validation design.

Parameters:

Name	Type	Description	Default
`category`	`(iid, grouped, temporal, spatial)`	Dependence structure driving the recommendation.	`"iid"`
`method`	`str or None`	TrustCV method string when one maps directly to the design.	required
`splitter`	`object`	Instantiated splitter accepted by `TrustCV.validate(cv=...)`.	required
`rationale`	`str`	Plain-language reason for the choice.	required
`warnings`	`list of str`	Data limitations or fold-feasibility concerns.	`list()`
`code`	`str`	Valid Python example that constructs the splitter.	`''`

Notes

The recommendation uses only supplied arrays and simple target/group diagnostics. It cannot infer unprovided patient IDs, timestamps, spatial dependence, causal structure, or preprocessing leakage.

Splitters: independent samples¶

HoldOut ¶

HoldOut(test_size: Union[float, int] = 0.2, random_state: Optional[int] = None, stratify: Optional[ndarray] = None)

Hold-Out (Train-Test Split) Validation

Splits data into single training and test sets. Computational complexity: O(T(n))

Parameters:

Name	Type	Description	Default
`test_size`	`float or int`	If float, represents proportion of dataset for test set (0.0, 1.0) If int, represents absolute number of test samples	`0.2`
`random_state`	`int or None`	Random state for reproducibility	`None`
`stratify`	`array - like or None`	Data to use for stratified splitting	`None`

split ¶

split(X, y=None, groups=None)

Generate train/test indices

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns number of splits (always 1 for hold-out)

KFoldMedical ¶

KFoldMedical(n_splits: int = 5, shuffle: bool = False, random_state: Optional[int] = None)

Bases: BaseCrossValidator

Standard k-Fold Cross-Validation with medical considerations

Computational complexity: O(k·T(n(k-1)/k))

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of folds	`5`
`shuffle`	`bool`	Whether to shuffle data before splitting	`False`
`random_state`	`int or None`	Random state for reproducibility	`None`

split ¶

split(X, y=None, groups=None)

Generate train/test indices for each fold

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

StratifiedKFoldMedical ¶

StratifiedKFoldMedical(n_splits: int = 5, shuffle: bool = False, random_state: Optional[int] = None)

Bases: BaseCrossValidator

Stratified k-Fold Cross-Validation

Maintains class distribution in each fold. Particularly important for imbalanced medical datasets.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of folds	`5`
`shuffle`	`bool`	Whether to shuffle data before splitting	`False`
`random_state`	`int or None`	Random state for reproducibility	`None`

split ¶

split(X, y, groups=None)

Generate stratified train/test indices

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

RepeatedKFold ¶

RepeatedKFold(n_splits: int = 5, n_repeats: int = 10, random_state: Optional[int] = None, stratify: bool = False)

Bases: BaseCrossValidator

Repeated k-Fold Cross-Validation

Performs k-fold CV multiple times with different randomization. Reduces variance in performance estimates.

Computational complexity: O(r·k·T(n(k-1)/k))

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of folds	`5`
`n_repeats`	`int`	Number of times to repeat the k-fold CV	`10`
`random_state`	`int or None`	Random state for reproducibility	`None`
`stratify`	`bool`	If True, uses RepeatedStratifiedKFold (requires labels)	`False`

split ¶

split(X, y=None, groups=None)

Generate train/test indices for repeated k-fold

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

LOOCV ¶

LOOCV()

Bases: BaseCrossValidator

Leave-One-Out Cross-Validation

Each sample is used once as test set. Provides nearly unbiased estimate but computationally expensive.

Computational complexity: O(n·T(n-1))

Warning: Not recommended for large datasets (n > 1000)

split ¶

split(X, y=None, groups=None)

Generate train/test indices for LOOCV

get_n_splits ¶

get_n_splits(X, y=None, groups=None)

Returns the number of splitting iterations

LPOCV ¶

LPOCV(p: int)

Bases: BaseCrossValidator

Leave-p-Out Cross-Validation

Generalization of LOOCV where p samples are left out. Number of iterations: C(n, p) = n!/(p!(n-p)!)

Parameters:

Name	Type	Description	Default
`p`	`int`	Number of samples to leave out in each iteration	required

Warnings

The number of splits grows combinatorially with p and the sample size.

split ¶

split(X, y=None, groups=None)

Generate train/test indices for LPOCV

get_n_splits ¶

get_n_splits(X, y=None, groups=None)

Returns the number of splitting iterations

BootstrapValidation ¶

BootstrapValidation(n_iterations: int = 100, estimator: str = 'standard', random_state: Optional[int] = None, **kwargs)

Bootstrap Validation with .632 and .632+ estimators

Samples with replacement for training, uses out-of-bag samples for testing.

Parameters:

Name	Type	Description	Default
`n_iterations`	`int`	Number of bootstrap iterations	`100`
`estimator`	`str`	Type of bootstrap estimator: 'standard', '.632', or '.632+'	`'standard'`
`random_state`	`int or None`	Random state for reproducibility	`None`

split ¶

split(X, y=None, groups=None)

Generate bootstrap train/test indices

compute_error ¶

compute_error(train_errors, test_errors, y_true=None, y_pred_train=None)

Compute bootstrap error estimate

Parameters:

Name	Type	Description	Default
`train_errors`	`array - like`	Training errors for each iteration	required
`test_errors`	`array - like`	Test (out-of-bag) errors for each iteration	required
`y_true`	`array - like`	True labels (needed for .632+ estimator)	`None`
`y_pred_train`	`array - like`	Training predictions (needed for .632+ estimator)	`None`

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

MonteCarloCV ¶

MonteCarloCV(n_iterations: int = 100, test_size: Union[float, int] = 0.2, random_state: Optional[int] = None)

Bases: BaseCrossValidator

Monte Carlo Cross-Validation (Random Sub-Sampling)

Randomly splits data multiple times into train/test sets. Unlike k-fold, test sets may overlap between iterations.

Parameters:

Name	Type	Description	Default
`n_iterations`	`int`	Number of random splits	`100`
`test_size`	`float or int`	If float, proportion of dataset for test set (0.0, 1.0) If int, absolute number of test samples	`0.2`
`random_state`	`int or None`	Random state for reproducibility	`None`

split ¶

split(X, y=None, groups=None)

Generate random train/test splits

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

NestedCV ¶

NestedCV(outer_cv=None, inner_cv=None, n_jobs=-1)

Nested Cross-Validation for hyperparameter tuning

Uses outer CV for model evaluation and inner CV for hyperparameter selection. Provides unbiased performance estimate when tuning hyperparameters.

Parameters:

Name	Type	Description	Default
`outer_cv`	`cross-validator object`	Outer cross-validation for model evaluation	`None`
`inner_cv`	`cross-validator object`	Inner cross-validation for hyperparameter tuning	`None`
`n_jobs`	`int or None`	Parallel jobs for the inner grid search (`-1` = all cores, `1` = no worker processes; `None` follows the active joblib backend)	`-1`

fit_predict ¶

fit_predict(estimator, X, y, param_grid)

Perform nested cross-validation

Parameters:

Name	Type	Description	Default
`estimator`	`estimator object`	The model to evaluate	required
`X`	`array - like`	Feature matrix	required
`y`	`array - like`	Target values	required
`param_grid`	`dict`	Hyperparameter grid for tuning	required

Returns:

Name	Type	Description
`scores`	`list`	Outer CV scores for each fold
`best_params`	`list`	Best parameters found for each outer fold

get_n_splits ¶

get_n_splits()

Returns the number of outer CV splits

Splitters: grouped¶

GroupKFoldMedical ¶

GroupKFoldMedical(n_splits=5, shuffle=True, random_state=None)

Bases: _BaseKFold

Patient-aware K-Fold cross-validator

Ensures all records from a patient stay in the same fold. Critical for preventing data leakage in ML.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of folds	`5`
`shuffle`	`bool`	Whether to shuffle patients before splitting	`True`
`random_state`	`int`	Random seed for reproducibility	`None`

Examples:

>>> from trustcv.splitters import GroupKFoldMedical
>>> cv = GroupKFoldMedical(n_splits=5)
>>> for train, test in cv.split(X, y, patient_ids):
...     # All records from same patient in same fold

split ¶

split(X, y=None, groups=None)

Generate indices to split data into training and test set.

Parameters:

Name	Type	Description	Default
`X`	`(array - like, shape(n_samples, n_features))`	Training data	required
`y`	`(array - like, shape(n_samples))`	Target variable	`None`
`groups`	`(array - like, shape(n_samples))`	Patient identifiers	`None`

Yields:

Name	Type	Description
`train`	`ndarray`	Training set indices for fold
`test`	`ndarray`	Test set indices for fold

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

StratifiedGroupKFold ¶

StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=None)

Bases: _BaseKFold

Stratified Group K-Fold cross-validator

Combines stratification (preserving class balance) with grouping (keeping patient records together). Essential for imbalanced medical datasets with multiple records per patient.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of folds	`5`
`shuffle`	`bool`	Whether to shuffle data before splitting	`True`
`random_state`	`int`	Random seed	`None`

Examples:

>>> cv = StratifiedGroupKFold(n_splits=5)
>>> # Preserves both class distribution AND patient grouping
>>> for train, test in cv.split(X, y, patient_ids):
...     train_model(X[train], y[train])

split ¶

split(X, y, groups)

Generate stratified grouped splits

Parameters:

Name	Type	Description	Default
`X`	`(array - like, shape(n_samples, n_features))`	Training data	required
`y`	`(array - like, shape(n_samples))`	Target variable (required for stratification)	required
`groups`	`(array - like, shape(n_samples))`	Patient identifiers	required

Yields:

Name	Type	Description
`train`	`ndarray`	Training indices
`test`	`ndarray`	Test indices

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

LeaveOneGroupOut ¶

LeaveOneGroupOut()

Leave-One-Group-Out Cross-Validation (LOGOCV)

Each group (e.g., patient, hospital) is used once as test set, which estimates performance on a group the model has never seen.

Takes no parameters; pass groups to :meth:split.

Examples:

>>> logo = LeaveOneGroupOut()
>>> for train, test in logo.split(X, y, groups=patient_ids):
...     # Each patient used once as test set

split ¶

split(X, y=None, groups=None)

Generate leave-one-group-out splits

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Features	required
`y`	`array - like`	Labels	`None`
`groups`	`array - like`	Group identifiers	`None`

Yields:

Type	Description
`train, test indices`

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

LeavePGroupsOut ¶

LeavePGroupsOut(n_groups)

Leave-P-Groups-Out cross-validator

Provides train/test indices where p groups are left out for testing.

Parameters:

Name	Type	Description	Default
`n_groups`	`int`	Number of groups to leave out in each iteration	required

split ¶

split(X, y=None, groups=None)

Generate indices to split data where p groups are left out

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

RepeatedGroupKFold ¶

RepeatedGroupKFold(n_splits=5, n_repeats=10, random_state=None)

Repeated Group K-Fold Cross-Validation

Performs group k-fold CV multiple times with different randomization. Reduces variance in performance estimates for grouped data.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of folds	`5`
`n_repeats`	`int`	Number of times to repeat the CV	`10`
`random_state`	`int or None`	Random state for reproducibility	`None`

split ¶

split(X, y=None, groups=None)

Generate repeated group k-fold splits

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Features	required
`y`	`array - like`	Labels	`None`
`groups`	`array - like`	Group identifiers	`None`

Yields:

Type	Description
`train, test indices`

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

HierarchicalGroupKFold ¶

HierarchicalGroupKFold(n_splits=5, hierarchy_level='patient', shuffle=True, random_state=None)

Bases: _BaseKFold

Hierarchical Group K-Fold for multi-level medical data

Handles nested grouping structures like: - Hospital → Department → Patient - Study Site → Patient → Visit

When splitting at a higher level (e.g., hospital), all nested entities (departments, patients) within that group stay together in train or test.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of folds	`5`
`hierarchy_level`	`str`	Level to split on. Must be a key in the hierarchy dict.	`'patient'`
`shuffle`	`bool`	Whether to shuffle groups before splitting	`True`
`random_state`	`int or None`	Random state for reproducibility	`None`

Examples:

>>> from trustcv.splitters import HierarchicalGroupKFold
>>> hierarchy = {
...     'hospital': [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3],
...     'department': [0, 0, 1, 2, 2, 3, 4, 4, 5, 6, 6, 7],
...     'patient': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
... }
>>> cv = HierarchicalGroupKFold(n_splits=2, hierarchy_level='hospital')
>>> for train, test in cv.split(X, y, hierarchy=hierarchy):
...     # All patients from same hospital stay together

split ¶

split(X, y=None, groups=None, hierarchy=None)

Generate hierarchical splits

Parameters:

Name	Type	Description	Default
`X`	`array-like of shape (n_samples, n_features)`	Training data	required
`y`	`array-like of shape (n_samples,)`	Target values	`None`
`groups`	`array-like of shape (n_samples,)`	Group labels. If provided without hierarchy, used directly.	`None`
`hierarchy`	`dict`	Dictionary mapping level names to group arrays. Each array has shape (n_samples,) with group assignments. Example: {'hospital': [...], 'patient': [...]}	`None`

Yields:

Name	Type	Description
`train`	`ndarray`	Training set indices for this split
`test`	`ndarray`	Test set indices for this split

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations.

MultilevelCV ¶

MultilevelCV(n_splits=5, validation_level='level_2')

Multi-level Cross-Validation for hierarchical data

Handles data with multiple hierarchical levels (e.g., Hospital → Department → Patient). Can perform validation at any level of the hierarchy.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of folds	`5`
`validation_level`	`str`	Which hierarchy level to use for splitting	`'level_2'`

split ¶

split(X, y=None, groups=None)

Generate indices for multi-level validation

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

NestedGroupedCV ¶

NestedGroupedCV(outer_cv=None, inner_cv=None, n_jobs=-1)

Nested Cross-Validation for Grouped Data

Performs nested CV while preserving group structure. Outer loop for evaluation, inner loop for hyperparameter tuning.

Parameters:

Name	Type	Description	Default
`outer_cv`	`grouped CV object`	Outer CV for model evaluation	`None`
`inner_cv`	`grouped CV object`	Inner CV for hyperparameter tuning	`None`
`n_jobs`	`int or None`	Parallel jobs for the inner grid search (`-1` = all cores, `1` = no worker processes; `None` follows the active joblib backend)	`-1`

fit_predict ¶

fit_predict(estimator, X, y, groups, param_grid)

Perform nested grouped cross-validation

Parameters:

Name	Type	Description	Default
`estimator`	`estimator object`	The model to evaluate	required
`X`	`array - like`	Feature matrix	required
`y`	`array - like`	Target values	required
`groups`	`array - like`	Group identifiers	required
`param_grid`	`dict`	Hyperparameter grid for tuning	required

Returns:

Name	Type	Description
`scores`	`list`	Outer CV scores
`best_params`	`list`	Best parameters for each outer fold

get_n_splits ¶

get_n_splits()

Returns the number of outer CV splits

Splitters: temporal¶

TimeSeriesSplit ¶

TimeSeriesSplit(n_splits=5, gap=0, test_size=None, max_train_size=None)

Bases: _BaseKFold

Time-aware cross-validation for clinical data

Ensures training data always precedes test data temporally. Critical for predictive models in clinical settings.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of splits	`5`
`gap`	`int`	Gap between train and test (e.g., for prediction horizon)	`0`
`test_size`	`int or float`	Size of test set in each split	`None`

Examples:

>>> from trustcv.splitters import TemporalClinical
>>> tscv = TemporalClinical(n_splits=5, gap=7)  # 7-day gap
>>> for train, test in tscv.split(X, timestamps=dates):
...     # Train on past, test on future

split ¶

split(X, y=None, groups=None, timestamps=None)

Generate indices for temporal splits

Parameters:

Name	Type	Description	Default
`X`	`(array - like, shape(n_samples, n_features))`	Training data	required
`y`	`(array - like, shape(n_samples))`	Target variable	`None`
`groups`	`(array - like, shape(n_samples))`	Group labels (e.g., patient IDs)	`None`
`timestamps`	`(array - like, shape(n_samples))`	Temporal information for each sample	`None`

Yields:

Name	Type	Description
`train`	`ndarray`	Training set indices
`test`	`ndarray`	Test set indices

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

BlockedTimeSeries ¶

BlockedTimeSeries(n_splits=5, block_size='day')

Bases: _BaseKFold

Blocked time series cross-validation

Preserves temporal dependencies by keeping time blocks together. Useful for seasonal medical data or clustered events.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of splits	`5`
`block_size`	`int or str`	Size of temporal blocks ('day', 'week', 'month', or integer)	`'day'`

Examples:

>>> btscv = BlockedTimeSeries(n_splits=5, block_size='week')
>>> for train, test in btscv.split(X, timestamps=dates):
...     # Blocks of weeks stay together

split ¶

split(X, y=None, groups=None, timestamps=None)

Generate blocked time series splits

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Features	required
`y`	`array - like`	Labels	`None`
`groups`	`array - like`	Group labels	`None`
`timestamps`	`array - like`	Temporal information	`None`

Yields:

Type	Description
`train, test indices`

RollingWindowCV ¶

RollingWindowCV(window_size, step_size=1, forecast_horizon=1, gap=0)

Rolling Window Cross-Validation (Walk-Forward Validation)

Fixed-size training window that slides through time. Maintains constant training set size, important for stability.

Parameters:

Name	Type	Description	Default
`window_size`	`int`	Size of training window	required
`step_size`	`int`	Step size for sliding window	`1`
`forecast_horizon`	`int`	Number of periods ahead to predict	`1`
`gap`	`int`	Gap between training and test sets	`0`

split ¶

split(X, y=None, groups=None)

Generate rolling window splits

Yields:

Type	Description
`train, test indices`

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Get number of splits

ExpandingWindowCV ¶

ExpandingWindowCV(initial_train_size=None, step_size=1, forecast_horizon=1, gap=0, **kwargs)

Expanding Window Cross-Validation

Training set grows over time, always starting from the beginning. Useful when all historical data is relevant.

Parameters:

Name	Type	Description	Default
`initial_train_size`	`int`	Number of samples in the first training window (the deprecated names `min_train_size` and `initial_size` are still accepted)	`10`
`step_size`	`int`	Step size for expanding window	`1`
`forecast_horizon`	`int`	Number of periods ahead to predict	`1`
`gap`	`int`	Gap between training and test sets	`0`

split ¶

split(X, y=None, groups=None)

Generate expanding window splits

Yields:

Type	Description
`train, test indices`

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Get number of splits

PurgedKFoldCV ¶

PurgedKFoldCV(n_splits=5, purge_gap=0, embargo_size=0.0, **kwargs)

Purged K-Fold with Embargo for Financial Time Series

Implements purging and embargo to prevent data leakage in financial/medical cost prediction scenarios.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of folds	`5`
`purge_gap`	`int`	Number of samples to purge between train and test	`0`
`embargo_size`	`float`	Percentage of test set size to embargo after test set (renamed from embargo_pct)	`0.0`

split ¶

split(X, y=None, groups=None, timestamps=None)

Generate purged k-fold splits

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Features	required
`y`	`array - like`	Labels	`None`
`groups`	`array - like`	Group labels	`None`
`timestamps`	`array - like`	Timestamps for ordering	`None`

Yields:

Type	Description
`train, test indices`

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

CombinatorialPurgedCV ¶

CombinatorialPurgedCV(n_splits=5, n_test_splits=None, purge_gap=0, embargo_size=0.0, strict_order=True, **kwargs)

Combinatorial Purged Cross-Validation (CPCV)

Advanced method for financial time series that generates multiple train/test combinations with purging.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of groups to split data into	`5`
`n_test_splits`	`int`	Number of groups to use as test set	`2`
`purge_gap`	`int`	Purge gap between train and test	`0`
`embargo_size`	`float`	Embargo percentage	`0.0`

split ¶

split(X, y=None, groups=None)

Generate combinatorial purged splits

Yields:

Type	Description
`train, test indices`

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

PurgedGroupTimeSeriesSplit ¶

PurgedGroupTimeSeriesSplit(n_splits=5, purge_gap=0, embargo_size=0.0, group_exclusive=False, **kwargs)

Bases: _BaseKFold

Purged Group Time Series Split for medical panel data

Combines: - Temporal ordering (time series) - Group preservation (patient data) - Purging (gap to prevent leakage) - Embargo (no trading period simulation)

Essential for financial-medical hybrid applications (e.g., healthcare cost prediction, insurance claims)

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of splits	`5`
`purge_gap`	`int`	Temporal gap between train and test	`0`
`embargo_size`	`float`	Fraction of data to embargo after test	`0.0`

Examples:

>>> pgts = PurgedGroupTimeSeriesSplit(n_splits=5, purge_gap=30)
>>> for train, test in pgts.split(X, groups=patients, timestamps=dates):
...     # Patient-aware temporal splitting with purging

split ¶

split(X, y=None, groups=None, timestamps=None)

Generate purged group time series splits

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Features	required
`y`	`array - like`	Labels	`None`
`groups`	`array - like`	Patient/group identifiers	`None`
`timestamps`	`array - like`	Temporal information	`None`

Yields:

Type	Description
`train, test indices`

NestedTemporalCV ¶

NestedTemporalCV(outer_cv, inner_cv)

High-level nested CV for temporal (and optionally grouped) data.

outer_cv: trustcv splitter for unbiased evaluation (e.g., TimeSeriesSplit, PurgedGroupTimeSeriesSplit) inner_cv: trustcv splitter for hyperparameter tuning (same family as outer)

Splitters: spatial¶

SpatialBlockCV ¶

SpatialBlockCV(n_splits: int = 5, block_shape: Literal['grid', 'kmeans'] = 'grid', block_size: Optional[float] = None, random_state: Optional[int] = None, coordinates: Optional[ArrayLike] = None, **kwargs)

Bases: BaseCrossValidator

Spatial Block Cross-Validation

Divides spatial data into blocks to handle spatial autocorrelation. Essential for geographic medical data analysis.

Rows are grouped into spatial blocks, and whole blocks are assigned to n_splits folds, so every row is tested exactly once and no block is split between training and test.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of folds	`5`
`block_shape`	`(grid, kmeans)`	`'grid'`: rectangular cells; `'kmeans'`: one k-means cluster of the coordinates per fold	`'grid'`
`block_size`	`float`	Side length of square grid cells, in coordinate units, on a fixed grid anchored at 0. When omitted, the bounding box is cut into about `n_splits` cells. Only used with `block_shape='grid'`.	`None`
`random_state`	`int or None`	Seed for assigning blocks to folds (and for k-means)	`None`
`coordinates`	`array-like of shape (n_samples, 2)`	Default coordinates, used when `split` gets none	`None`

split ¶

split(X, y=None, groups=None, coordinates=None)

Generate spatial block splits

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Features	required
`y`	`array - like`	Labels	`None`
`groups`	`array - like`	Group labels	`None`
`coordinates`	`array-like of shape (n_samples, 2)`	Spatial coordinates for each sample	`None`

Yields:

Type	Description
`train, test indices`

BufferedSpatialCV ¶

BufferedSpatialCV(n_splits: int = 5, buffer_size: float = 0.1, distance_metric: str = 'euclidean', block_shape: str = 'grid', random_state: Optional[int] = None)

Bases: BaseCrossValidator

Buffered Spatial Cross-Validation

Creates buffer zones around test blocks to reduce spatial autocorrelation. Samples in buffer zones are excluded from training.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of spatial blocks	`5`
`buffer_size`	`float`	Size of buffer zone around test blocks	`0.1`
`distance_metric`	`str`	Distance metric for buffer calculation	`'euclidean'`
`block_shape`	`str`	Shape of spatial blocks	`'grid'`
`random_state`	`int or None`	Random state for reproducibility	`None`

split ¶

split(X, y=None, groups=None, coordinates=None)

Generate buffered spatial splits

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Features	required
`y`	`array - like`	Labels	`None`
`groups`	`array - like`	Group labels	`None`
`coordinates`	`array-like of shape (n_samples, 2)`	Spatial coordinates	`None`

Yields:

Type	Description
`train, test indices`

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

SpatiotemporalBlockCV ¶

SpatiotemporalBlockCV(n_spatial_blocks: int = 3, n_temporal_blocks: int = 3, buffer_space: float = 0, buffer_time: int = 0, block_shape: str = 'grid', random_state: Optional[int] = None, **kwargs)

Bases: BaseCrossValidator

Spatiotemporal Block Cross-Validation

Handles data with both spatial and temporal dimensions. Essential for disease spread modeling and environmental health studies.

Parameters:

Name	Type	Description	Default
`n_spatial_blocks`	`int`	Number of spatial blocks	`3`
`n_temporal_blocks`	`int`	Number of temporal blocks	`3`
`buffer_space`	`float`	Spatial buffer size	`0`
`buffer_time`	`int`	Temporal buffer size (in time units)	`0`
`block_shape`	`str`	Shape of spatial blocks	`'grid'`
`random_state`	`int or None`	Random state for reproducibility	`None`

split ¶

split(X, y=None, groups=None, coordinates=None, timestamps=None)

Generate spatiotemporal block splits

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Features	required
`y`	`array - like`	Labels	`None`
`groups`	`array - like`	Group labels	`None`
`coordinates`	`array-like of shape (n_samples, 2)`	Spatial coordinates	`None`
`timestamps`	`array-like of shape (n_samples,)`	Temporal information	`None`

Yields:

Type	Description
`train, test indices`

get_n_splits ¶

get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations

EnvironmentalHealthCV ¶

EnvironmentalHealthCV(spatial_blocks: int = 4, temporal_strategy: str = 'seasonal', environmental_vars: Optional[List[str]] = None, buffer_config: Optional[dict] = None, **kwargs)

Specialized CV for environmental health studies

Combines spatial, temporal, and environmental factors. Handles pollution studies, climate health impacts, etc.

Parameters:

Name	Type	Description	Default
`spatial_blocks`	`int`	Number of spatial regions	`4`
`temporal_strategy`	`str`	Temporal splitting strategy: 'seasonal', 'yearly', 'custom'	`'seasonal'`
`environmental_vars`	`list`	Environmental variables to consider	`None`
`buffer_config`	`dict`	Buffer configuration for different factors	`None`

split ¶

split(X, y=None, coordinates=None, timestamps=None, environmental_data=None, **kwargs)

Generate environmental health CV splits

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Features	required
`y`	`array - like`	Health outcomes	`None`
`coordinates`	`array - like`	Spatial locations	`None`
`timestamps`	`array - like`	Time points	`None`
`environmental_data`	`dict`	Environmental covariates	`None`

Yields:

Type	Description
`train, test indices`

get_n_splits ¶

get_n_splits()

Returns estimated number of splits

Splitters: multilabel¶

MultilabelStratifiedKFold ¶

MultilabelStratifiedKFold(n_splits: int = 5, shuffle: bool = False, random_state: Any = None)

Thin wrapper around iterative-stratification's MultilabelStratifiedKFold.

Parameters mirror sklearn's StratifiedKFold for familiarity.

MultilabelStratifiedGroupKFold ¶

MultilabelStratifiedGroupKFold(n_splits: int = 5, shuffle: bool = True, random_state: Any = None, alpha: float = 1.0, beta: float = 1.0, eps: float = 1e-09)

Bases: _BaseKFold

Group-aware multilabel stratified K-Fold cross-validator.

Parameters mirror sklearn's KFold plus weighting terms for the greedy assignment heuristic.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of folds.	`5`
`shuffle`	`bool`	Whether to shuffle groups before sorting for tie-breaking.	`True`
`random_state`	`int or None`	Random seed used when `shuffle=True`.	`None`
`alpha`	`float`	Weight for label balance cost.	`1.0`
`beta`	`float`	Weight for fold size balance cost.	`1.0`
`eps`	`float`	Small constant to avoid division by zero in weight computation.	`1e-9`

MultiLabelGroupSplitter ¶

MultiLabelGroupSplitter(n_splits: int = 5, shuffle: bool = True, random_state: Any = None, agg: str = 'max')

Iterative-stratified multilabel splitter that first aggregates labels per group, then applies iterative stratification across groups, and finally maps groups back to sample indices.

This gives you the two key properties needed for patient-level CV on multilabel data:

No patient appears in both train and validation (group hold-out).
Per-label prevalence is balanced across folds as well as possible via iterative stratification.

Parameters:

Name	Type	Description	Default
`n_splits`	`int`	Number of folds.	`5`
`shuffle`	`bool`	Whether to shuffle groups before splitting (passed to iterstrat).	`True`
`random_state`	`int or None`	Random seed for reproducibility.	`None`
`agg`	`(max, any)`	How to aggregate multiple samples from the same group into a single multilabel row. `"max"`/`"any"` mark a label present if it appears in any sample of the group.	`"max"`

Leakage and balance checks¶

DataLeakageChecker ¶

DataLeakageChecker(verbose: bool = True, *, target_auc_threshold: float = 0.99, target_correlation_threshold: float = 0.99, target_strong_threshold: float = 0.9, max_target_features: int = 10000, near_duplicate_distance_ratio: float = 0.01, near_duplicate_absolute_threshold: float = 0.001, covariate_shift_alpha: float = 0.05)

Comprehensive data leakage detection for ML

Checks for: - Patient-level leakage - Temporal leakage - Feature leakage - Preprocessing leakage - Duplicate samples - Near-duplicate samples (cosine similarity) - Hierarchical group leakage

Examples:

>>> checker = DataLeakageChecker()
>>> report = checker.check_cv_splits(X_train, X_test, y_train, y_test, patient_ids)
>>> if report.has_leakage:
>>>     print(report)

check ¶

check(X: Union[ndarray, DataFrame], y: Optional[Union[ndarray, Series]] = None, groups: Optional[Union[ndarray, Series]] = None, timestamps: Optional[Union[ndarray, Series]] = None, coordinates: Optional[Union[ndarray, DataFrame]] = None, n_splits: int = 5, random_state: Optional[int] = 42, folds: Optional[List[Tuple[ndarray, ndarray]]] = None) -> LeakageReport

Convenience wrapper to check leakage via CV-style splits.

If folds (a list of (train_idx, test_idx)) is given, exactly those splits are audited; pass the folds your model was scored on. Otherwise, if groups are provided, uses GroupKFold; else StratifiedKFold when labels are available, else KFold. Those default splits describe what these splitters would do, not your own CV.

When coordinates are provided but no explicit spatial threshold is given, the threshold is auto-computed as the 10th percentile of pairwise distances from a random sample.

Returns a LeakageReport aggregated over folds.

check_cv_splits ¶

check_cv_splits(X_train: Union[ndarray, DataFrame], X_test: Union[ndarray, DataFrame], y_train: Optional[Union[ndarray, Series]] = None, y_test: Optional[Union[ndarray, Series]] = None, patient_ids_train: Optional[Union[ndarray, Series]] = None, patient_ids_test: Optional[Union[ndarray, Series]] = None, timestamps_train: Optional[Union[ndarray, Series]] = None, timestamps_test: Optional[Union[ndarray, Series]] = None, coordinates_train: Optional[Union[ndarray, DataFrame]] = None, coordinates_test: Optional[Union[ndarray, DataFrame]] = None, spatial_threshold: Optional[float] = None) -> LeakageReport

Check for data leakage between train and test sets

Parameters:

Name	Type	Description	Default
`X_train`	`array - like`	Training and test features	required
`X_test`	`array - like`	Training and test features	required
`y_train`	`array - like`	Training and test labels	`None`
`y_test`	`array - like`	Training and test labels	`None`
`patient_ids_train`	`array - like`	Patient identifiers	`None`
`patient_ids_test`	`array - like`	Patient identifiers	`None`
`timestamps_train`	`array - like`	Temporal information	`None`
`timestamps_test`	`array - like`	Temporal information	`None`
`coordinates_train`	`array - like`	Spatial coordinates	`None`
`coordinates_test`	`array - like`	Spatial coordinates	`None`
`spatial_threshold`	`float`	Distance threshold for spatial proximity check	`None`

Returns:

Type	Description
`LeakageReport`	Detailed leakage analysis

spatial_check ¶

spatial_check(coordinates_train: Union[ndarray, DataFrame], coordinates_test: Union[ndarray, DataFrame], threshold: float) -> Dict[str, Any]

Simple spatial proximity flag between train and test sets.

Computes the minimum distance from each test point to any train point and reports the fraction within the given threshold, along with mean/min of those minimum distances.

Parameters:

Name	Type	Description	Default
`coordinates_train`	`array - like(n_train, 2)`		required
`coordinates_test`	`array - like(n_test, 2)`		required
`threshold`	`float`	Distance threshold under which a test point is considered "near" a train point.	required

Returns:

Type	Description
`Dict`	{ 'near_fraction': float, 'mean_min_distance': float, 'min_min_distance': float, 'threshold': float }

check_near_duplicates ¶

check_near_duplicates(X_train: Union[ndarray, DataFrame], X_test: Union[ndarray, DataFrame], similarity_threshold: Optional[float] = None, *, distance_ratio: Optional[float] = None, absolute_threshold: Optional[float] = None) -> Dict[str, Any]

Detect unusually close test rows using calibrated Euclidean distance.

Features are standardized using training statistics. A test row is flagged only when its nearest training-row distance is below a small fraction of the median within-training nearest-neighbour distance.

Parameters:

Name	Type	Description	Default
`X_train`	`array - like`	Numeric feature matrices.	required
`X_test`	`array - like`	Numeric feature matrices.	required
`similarity_threshold`	`float`	Deprecated compatibility parameter. It maps to a distance ratio of 1 - similarity_threshold.	`None`
`distance_ratio`	`float`	Fraction of typical within-training nearest-neighbour spacing used as the flagging threshold. Defaults to the checker configuration.	`None`
`absolute_threshold`	`float`	Fallback distance in standardized units when de-duplicated training rows still have zero nearest-neighbour spacing. Defaults to the checker configuration.	`None`

Returns:

Type	Description
`dict`	Counts and calibrated distance thresholds.

Notes

This check finds extremely close rows after numeric standardization. It cannot identify semantically duplicated records with different feature encodings.

check_hierarchical_leakage ¶

check_hierarchical_leakage(groups_train: Union[ndarray, Series, List], groups_test: Union[ndarray, Series, List], parent_groups_train: Union[ndarray, Series, List], parent_groups_test: Union[ndarray, Series, List]) -> Dict[str, Any]

Check for hierarchical group leakage (e.g. hospital -> patient).

If any parent group appears in both training and test sets this constitutes hierarchical leakage, even when child groups are properly separated.

Parameters:

Name	Type	Description	Default
`groups_train`	`array - like`	Child-level group identifiers.	required
`groups_test`	`array - like`	Child-level group identifiers.	required
`parent_groups_train`	`array - like`	Parent-level group identifiers (same length as child groups).	required
`parent_groups_test`	`array - like`	Parent-level group identifiers (same length as child groups).	required

Returns:

Type	Description
`dict`	`has_leakage`, `overlapping_parents`, `overlap_count`.

check_preprocessing_leakage ¶

check_preprocessing_leakage(X_original: Union[ndarray, DataFrame], X_processed: Union[ndarray, DataFrame], split_indices: Tuple[ndarray, ndarray]) -> bool

Check if preprocessing was done before or after splitting

Parameters:

Name	Type	Description	Default
`X_original`	`array - like`	Original data before preprocessing	required
`X_processed`	`array - like`	Data after preprocessing	required
`split_indices`	`tuple`	(train_indices, test_indices)	required

Returns:

Type	Description
`bool`	True if leakage detected

check_feature_target_leakage ¶

check_feature_target_leakage(X: Union[ndarray, DataFrame], y: Union[ndarray, Series], threshold: Optional[float] = None, *, auc_threshold: Optional[float] = None, correlation_threshold: Optional[float] = None, max_features: Optional[int] = None, strong_threshold: Optional[float] = None) -> Dict[str, Any]

Scan individual features for near-deterministic target encoding.

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Feature matrix. Only the first max_features columns are scanned.	required
`y`	`array - like`	Binary labels or a one-dimensional regression target.	required
`threshold`	`float`	Backward-compatible alias for correlation_threshold.	`None`
`auc_threshold`	`float`	Binary ROC-AUC threshold. A feature is flagged at or above this value, or at or below 1 - auc_threshold.	`None`
`correlation_threshold`	`float`	Absolute Spearman-correlation threshold for regression.	`None`
`max_features`	`int`	Maximum number of columns to inspect.	`None`
`strong_threshold`	`float`	Lower tier: features at or above this ROC-AUC (or `1 - value`), or absolute Spearman correlation, are listed in `strong_features` without counting as leakage.	`None`

Returns:

Type	Description
`dict`	Scan result with suspicious feature records and truncation details.

Notes

This univariate scan can find features that almost directly encode a target. It cannot prove that a feature is causally valid, detect multivariate encodings, or determine when an upstream transformation was fitted.

comprehensive_check ¶

comprehensive_check(X: Union[ndarray, DataFrame], y: Union[ndarray, Series], groups: Optional[ndarray] = None, timestamps: Optional[ndarray] = None, coordinates: Optional[ndarray] = None, feature_threshold: float = 0.95) -> Dict[str, Any]

Run all available leakage checks

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Feature matrix	required
`y`	`array - like`	Target variable	required
`groups`	`array - like`	Group identifiers (e.g., patient IDs)	`None`
`timestamps`	`array - like`	Temporal information	`None`
`coordinates`	`array - like`	Spatial coordinates	`None`
`feature_threshold`	`float`	Threshold for feature-target correlation	`0.95`

Returns:

Type	Description
`dict`	Comprehensive leakage report with recommendations

LeakageReport `dataclass` ¶

LeakageReport(has_leakage: bool, leakage_types: List[str], severity: str, details: Dict[str, Any], recommendations: List[str])

Detailed leakage detection report

summary `property` ¶

summary: str

Human-readable one-line/paragraph summary.

to_dict ¶

to_dict() -> Dict[str, Any]

Dictionary form for programmatic use/serialization.

BalanceChecker ¶

BalanceChecker(threshold: float = 0.1)

Checks for class imbalance and data distribution issues

Critical for medical datasets where disease prevalence can be extremely low (< 1% positive cases)

Parameters:

Name	Type	Description	Default
`threshold`	`float`	Imbalance threshold for warnings (default 10% difference)	`0.1`

check_class_balance ¶

check_class_balance(y: ndarray, groups: Optional[ndarray] = None) -> Dict[str, Any]

Check class distribution in dataset

Parameters:

Name	Type	Description	Default
`y`	`array - like`	Target labels	required
`groups`	`array - like`	Group identifiers (e.g., patient IDs)	`None`

Returns:

Type	Description
`dict`	Balance report with statistics and warnings

check_feature_distribution ¶

check_feature_distribution(X: ndarray, feature_names: Optional[List[str]] = None) -> Dict[str, Any]

Check feature distributions for potential issues

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Feature matrix	required
`feature_names`	`list`	Names of features	`None`

Returns:

Type	Description
`dict`	Feature distribution analysis

check_cv_balance ¶

check_cv_balance(X, y, cv_splitter, groups=None)

Check balance across CV folds

Parameters:

Name	Type	Description	Default
`X`	`array - like`	Features	required
`y`	`array - like`	Labels	required
`cv_splitter`	`cross-validation object`	CV splitter to test	required
`groups`	`array - like`	Group labels	`None`

Returns:

Type	Description
`dict`	CV balance analysis

generate_report ¶

generate_report() -> str

Generate human-readable balance report

Returns:

Type	Description
`str`	Formatted report

Metrics¶

ClinicalMetrics ¶

ClinicalMetrics(confidence_level=0.95, prevalence=None)

Comprehensive clinical metrics calculator

Calculates: - Sensitivity (TPR, Recall) - Specificity (TNR) - PPV (Positive Predictive Value, Precision) - NPV (Negative Predictive Value) - NNT (Number Needed to Treat) - NNS (Number Needed to Screen) - Likelihood Ratios - Diagnostic Odds Ratio - Youden's Index - All with confidence intervals

Examples:

>>> metrics = ClinicalMetrics()
>>> results = metrics.calculate_all(y_true, y_pred, y_proba)
>>> print(results.summary())

Parameters:

Name	Type	Description	Default
`confidence_level`	`float`	Confidence level for intervals (default 0.95)	`0.95`
`prevalence`	`float`	Disease prevalence if different from dataset	`None`

calculate_all ¶

calculate_all(y_true: ndarray, y_pred: ndarray, y_proba: Optional[ndarray] = None, sample_weight: Optional[ndarray] = None, groups: Optional[ndarray] = None) -> Dict

Calculate all clinical metrics

Parameters:

Name	Type	Description	Default
`y_true`	`array - like`	True labels	required
`y_pred`	`array - like`	Predicted labels	required
`y_proba`	`array - like`	Predicted probabilities	`None`
`sample_weight`	`array - like`	Sample weights	`None`
`groups`	`array - like`	Patient (cluster) ids. When given, the AUC confidence interval resamples whole patients instead of rows, which keeps repeated measures from one patient together and avoids too-narrow intervals.	`None`

Returns:

Type	Description
`dict`	Dictionary containing all metrics with confidence intervals

format_report ¶

format_report(metrics: Dict) -> str

Format metrics into clinical report

Parameters:

Name	Type	Description	Default
`metrics`	`dict`	Calculated metrics	required

Returns:

Type	Description
`str`	Formatted report

check_fold_metric_feasibility ¶

check_fold_metric_feasibility(y_true: Sequence, test_indices_by_fold: Iterable[Sequence[int]], metric_names: Optional[Iterable[str]] = None, min_test_samples: int = 5, require_both_classes_for_auc: bool = True) -> Dict[str, Any]

Diagnose whether fold-wise classification metrics are feasible.

Parameters:

Name	Type	Description	Default
`y_true`	`array - like`	Full target vector aligned with the original dataset.	required
`test_indices_by_fold`	`iterable of index arrays`	Validation/test indices for each fold.	required
`metric_names`	`iterable of str`	Metrics requested by the user or produced by the workflow.	`None`
`min_test_samples`	`int`	Test folds smaller than this threshold are flagged as unstable.	`5`
`require_both_classes_for_auc`	`bool`	If True, ROC-AUC is marked infeasible for single-class validation folds.	`True`

Returns:

Type	Description
`dict`	Structured diagnostics with per-fold feasibility rows and warning text.

Deep-learning runners¶

UniversalCVRunner ¶

UniversalCVRunner(cv_splitter: Any, framework: str = 'auto', adapter: Optional[FrameworkAdapter] = None, verbose: int = 1)

Framework-agnostic cross-validation runner

Automatically detects the framework and runs appropriate cross-validation.

Notes

This path does not run the full TrustCV integrity suite by default. Returned CVResults therefore contain eight NOT_CHECKED entries and overall_status="NOT_FULLY_VERIFIED" unless an explicit supported integrity callback supplies partial evidence. Performance results from this runner are not a leakage PASSED claim.

Initialize universal CV runner

Parameters: cv_splitter: Cross-validation splitter from trustcv framework: Framework name or 'auto' for auto-detection adapter: Custom adapter (overrides framework parameter) verbose: Verbosity level (0=silent, 1=progress, 2=detailed)

detect_framework ¶

detect_framework(model: Any) -> str

Auto-detect framework from model type

Parameters: model: Model object or class

Returns: Framework name

get_adapter ¶

get_adapter(framework: str) -> FrameworkAdapter

Get or create adapter for specified framework

Parameters: framework: Framework name

Returns: Framework adapter instance

run ¶

run(model: Union[Any, Callable], data: Any, epochs: Optional[int] = None, optimizer: Optional[Any] = None, loss_fn: Optional[Any] = None, metrics: Optional[List[str]] = None, callbacks: Optional[List[CVCallback]] = None, groups: Optional[ndarray] = None, split_kwargs: Optional[Dict[str, Any]] = None, fit_kwargs: Optional[Dict[str, Any]] = None, evaluate_kwargs: Optional[Dict[str, Any]] = None, warn_metric_feasibility: bool = True, recommend_pooled_oof: bool = True, min_test_samples: int = 5, inner_validation: float = 0.0, inner_validation_strategy: str = 'auto', inner_random_state: Optional[int] = 0, report_heldout_per_epoch: bool = False, **kwargs) -> CVResults

Run cross-validation with automatic framework detection

Parameters: model: Model instance or function that returns model data: Data in framework-appropriate format epochs: Number of epochs (for neural networks) optimizer: Optimizer (framework-specific). For PyTorch and MONAI pass a function model -> optimizer so each fold's new model gets its own; the default is Adam. loss_fn: Loss function (framework-specific) metrics: List of metrics to compute callbacks: List of callbacks groups: Group labels for grouped CV split_kwargs: Metadata passed only to the CV splitter, such as coordinates, timestamps, or environmental_data fit_kwargs: Arguments passed only to model fitting/training, such as sample_weight evaluate_kwargs: Optional evaluation-only arguments. These are passed when the selected adapter's evaluate method supports the provided keywords; otherwise they are reserved for adapter support and ignored. warn_metric_feasibility: If True, warn when group-based validation folds are too small or single-class for fold-wise AUC, sensitivity, or specificity. recommend_pooled_oof: If True, include pooled out-of-fold metric aggregation guidance in diagnostics. min_test_samples: Minimum held-out fold size considered stable for fold-wise classification metrics. inner_validation: Neural networks only. Fraction of each training fold set aside for model selection: per-epoch val_* metrics (for EarlyStopping and ModelCheckpoint) are computed on it, and the held-out fold is scored once, after training. 0 trains for the full epochs with no val_* metrics. inner_validation_strategy: "auto" (group, then stratified, then random) or "last" (the last part of each training fold, for time-ordered data). inner_random_state: Seed for the inner split (offset by the fold index). report_heldout_per_epoch: Also record held-out metrics per epoch in metadata["heldout_curves"], for plotting only. Callbacks never see them. **kwargs: Backward-compatible legacy arguments. Known splitter metadata is routed only to splitting; other legacy arguments are routed to splitting and fitting when explicit dictionaries are not provided.

Example: >>> from sklearn.ensemble import RandomForestClassifier >>> from trustcv.splitters import SpatialBlockCV >>> runner = UniversalCVRunner(cv_splitter=SpatialBlockCV(n_splits=5)) >>> results = runner.run( ... model=RandomForestClassifier(), ... data=(X, y), ... metrics=["roc_auc", "accuracy"], ... split_kwargs={"coordinates": coordinates}, ... )

Returns: CVResults object with scores, models, and explicit structured integrity checks. Integrity checks are NOT_CHECKED by default.

run_with_hyperparameter_tuning ¶

run_with_hyperparameter_tuning(model_fn: Callable, param_grid: Dict[str, List], data: Any, scoring: str = 'accuracy', n_trials: int = 10, **kwargs) -> Dict[str, Any]

Run cross-validation with hyperparameter tuning

Parameters: model_fn: Function that takes params and returns model param_grid: Dictionary of parameter names and values to try data: Training data scoring: Scoring metric to optimize n_trials: Number of trials for random search **kwargs: Additional arguments for run()

Returns: Dictionary with best parameters and results

CVResults `dataclass` ¶

CVResults(scores: List[Dict[str, Any]], models: Optional[List[Any]] = None, predictions: Optional[List[ndarray]] = None, probabilities: Optional[List[ndarray]] = None, indices: Optional[List[Tuple[ndarray, ndarray]]] = None, metadata: Dict[str, Any] = dict(), diagnostics: Dict[str, Any] = dict(), checks: Dict[str, CheckResult] = default_checks(), overall_status: str = 'NOT_FULLY_VERIFIED')

Standardized results container for cross-validation

Attributes: scores: Per-fold metric dicts, e.g., [{"score": 0.91, ...}, ...] models: Trained models from each fold (optional) predictions: Discrete predictions per fold (if available) probabilities: Prediction probabilities per fold (classification) indices: Train/test indices for each fold metadata: Additional info (framework, n_splits, cv_method, ...) checks: Structured integrity results. UniversalCVRunner leaves all checks NOT_CHECKED unless an explicit supported integrity callback is used. overall_status: Conservative integrity status. It is NOT_FULLY_VERIFIED when integrity checks were not run.

Notes

UniversalCVRunner evaluates model performance across frameworks but does not run the TrustCV integrity suite by default. A CVResults object therefore must not be interpreted as a leakage PASSED result.

metric_feasibility_warnings `property` ¶

metric_feasibility_warnings: List[str]

Warnings from fold-level metric feasibility diagnostics, if available.

fold_details `property` ¶

fold_details: List[Dict[str, Any]]

Backward-compatible per-fold details list.

Each entry contains at least: - fold: 1-based fold index - metrics: dict of scalar metrics for that fold - n_train / n_val: when indices are available

metrics `property` ¶

metrics: Dict[str, Dict[str, float]]

Aggregated metrics per name: {name: {"mean": float, "std": float}}.

mean_score `property` ¶

mean_score: Dict[str, float]

Mean value per metric: {name: mean}. Backward-compat for callers expecting a mapping.

mean_scores `property` ¶

mean_scores: Dict[str, float]

Backward-compatible alias for mean_score.

std_scores `property` ¶

std_scores: Dict[str, float]

Std value per metric: {name: std}.

KerasCVRunner ¶

KerasCVRunner(model_fn: Callable, cv_splitter: Any, adapter: Optional[TensorFlowAdapter] = None, compile_kwargs: Optional[Dict] = None, store_models: bool = False)

High-level cross-validation runner for Keras models

Runs cross-validation with Keras models: a fresh model per fold, the held-out fold used only for scoring, and per-fold results that can support documentation.

Initialize Keras CV runner

Parameters: model_fn: Function that returns a new model instance cv_splitter: Cross-validation splitter from trustcv adapter: TensorFlow adapter (creates default if None) compile_kwargs: Arguments for model.compile() store_models: Whether to store trained models (can use significant memory)

run ¶

run(X: ndarray, y: ndarray, epochs: int = 10, validation_split: float = 0.0, callbacks: Optional[List[Any]] = None, groups: Optional[ndarray] = None, inner_validation: float = 0.0, inner_validation_strategy: str = 'auto', inner_random_state: Optional[int] = 0, **fit_kwargs) -> CVResults

Run cross-validation with Keras model

The held-out fold of each split is used only to score the trained model. It is never passed as validation_data, because early stopping or checkpointing on it would select the model on the data that then scores it.

Parameters: X: Feature array y: Target array epochs: Number of training epochs per fold validation_split: Legacy Keras split: the last fraction of each training fold (not group-aware). Cannot be combined with inner_validation. callbacks: List of callbacks (both Keras and trustcv). Keras callbacks are copied per fold so their state (e.g. the best value seen by EarlyStopping) does not carry over. groups: Group labels for grouped CV inner_validation: Fraction of each training fold held out as validation data for early stopping and checkpoints (0 = none: train for epochs). Group-aware when groups is given. When > 0 and no EarlyStopping is passed, one is added that restores the best weights on the inner split. inner_validation_strategy: "auto" (group, then stratified, then random) or "last" (the last fraction of each training fold, for chronologically ordered data) inner_random_state: Seed for the inner split (offset by fold) **fit_kwargs: Additional arguments for model.fit()

Returns: CVResults object with scores and models

TorchCVRunner ¶

TorchCVRunner(model_fn: Callable, cv_splitter: Any, adapter: Optional[PyTorchAdapter] = None, store_models: bool = False)

High-level cross-validation runner for PyTorch models

Runs cross-validation with PyTorch models: a fresh model per fold, group-aware splitting and per-fold results that can support documentation.

Initialize PyTorch CV runner

Parameters: model_fn: Function that returns a new model instance cv_splitter: Cross-validation splitter from trustcv adapter: PyTorch adapter (creates default if None) store_models: Whether to store trained models (can use significant memory)

run ¶

run(dataset: Any, epochs: int = 10, optimizer_fn: Optional[Callable] = None, loss_fn: Optional[Any] = None, scheduler_fn: Optional[Callable] = None, callbacks: Optional[List[Any]] = None, groups: Optional[ndarray] = None, inner_validation: float = 0.0, inner_validation_strategy: str = 'auto', inner_random_state: Optional[int] = 0, report_heldout_per_epoch: bool = False) -> CVResults

Run cross-validation with PyTorch model

Parameters: dataset: PyTorch Dataset or tuple of (X, y) epochs: Number of training epochs per fold optimizer_fn: Function that takes model and returns optimizer loss_fn: Loss function scheduler_fn: Function that takes optimizer and returns scheduler callbacks: List of callbacks groups: Group labels for grouped CV inner_validation: Fraction of each training fold set aside for model selection. Per-epoch val_* metrics (for EarlyStopping and ModelCheckpoint) are computed on it; the held-out fold is only scored once, after training. 0 trains for the full epochs with no val_* metrics. inner_validation_strategy: "auto" (group, then stratified, then random) or "last" (the last part of each training fold, for time-ordered data). inner_random_state: Seed for the inner split (offset by the fold index). report_heldout_per_epoch: Also record held-out loss per epoch in metadata["heldout_curves"], for plotting only. Callbacks never see it.

Returns: CVResults object with scores and models

MONAICVRunner ¶

MONAICVRunner(model_fn: Callable, cv_splitter: Any, adapter: Optional[MONAIAdapter] = None, store_models: bool = False)

High-level cross-validation runner for MONAI medical imaging models

Simplifies running cross-validation with MONAI models while ensuring best practices for medical imaging validation.

Initialize MONAI CV runner

Parameters: model_fn: Function that returns a new model instance cv_splitter: Cross-validation splitter from trustcv adapter: MONAI adapter (creates default if None) store_models: Whether to store trained models (can use significant memory for 3D models)

run ¶

run(data_dicts: List[Dict], epochs: int = 100, train_transforms: Optional[Any] = None, val_transforms: Optional[Any] = None, optimizer_fn: Optional[Callable] = None, loss_fn: Optional[Any] = None, metrics: Optional[List[Any]] = None, scheduler_fn: Optional[Callable] = None, callbacks: Optional[List[Any]] = None, groups: Optional[ndarray] = None, inferer: Optional[Any] = None, inner_validation: float = 0.0, inner_validation_strategy: str = 'auto', inner_random_state: Optional[int] = 0, eval_every: int = 5, report_heldout_per_epoch: bool = False) -> CVResults

Run cross-validation with MONAI model

Parameters: data_dicts: List of dictionaries with 'image' and 'label' keys epochs: Number of training epochs per fold train_transforms: MONAI transforms for training val_transforms: MONAI transforms for validation optimizer_fn: Function that takes model and returns optimizer loss_fn: MONAI loss function metrics: List of MONAI metrics scheduler_fn: Function that takes optimizer and returns scheduler callbacks: List of callbacks groups: Group labels for grouped CV (e.g., patient IDs) inferer: MONAI inferer for sliding window inference inner_validation: Fraction of each training fold set aside for model selection. val_* metrics for EarlyStopping and ModelCheckpoint are computed on it; the held-out fold is scored once, after training. 0 trains for the full epochs with no val_* metrics. inner_validation_strategy: "auto" (group, then stratified, then random) or "last" (the last part of each training fold, for time-ordered data). inner_random_state: Seed for the inner split (offset by the fold index). eval_every: Callbacks run every eval_every epochs and on the last epoch, after the inner validation split is evaluated. report_heldout_per_epoch: Also record held-out metrics at those epochs in metadata["heldout_curves"], for plotting only. Callbacks never see them.

Returns: CVResults object with scores and models

JAXCVRunner ¶

JAXCVRunner(model_fn: Callable, cv_splitter: Any, adapter: Optional[JAXAdapter] = None, store_models: bool = False)

High-level cross-validation runner for JAX/Flax models

Simplifies running cross-validation with JAX/Flax models while ensuring best practices.

Example:

import flax.linen as nn
from trustcv.frameworks.jax import JAXCVRunner, JAXAdapter
from trustcv.splitters import StratifiedKFold

class MLP(nn.Module):
    @nn.compact
    def __call__(self, x):
        x = nn.Dense(64)(x)
        x = nn.relu(x)
        x = nn.Dense(2)(x)
        return x

runner = JAXCVRunner(
    model_fn=MLP,
    cv_splitter=StratifiedKFold(n_splits=5)
)

results = runner.run(X, y, epochs=10)

Initialize JAX CV runner

Parameters: model_fn: Function/class that returns a new Flax module cv_splitter: Cross-validation splitter from trustcv adapter: JAX adapter (creates default if None) store_models: Whether to store trained states (can use memory)

run ¶

run(X: ndarray, y: ndarray, epochs: int = 10, optimizer: Optional[Any] = None, loss_fn: Optional[Callable] = None, callbacks: Optional[List[Any]] = None, groups: Optional[ndarray] = None, inner_validation: float = 0.0, inner_validation_strategy: str = 'auto', inner_random_state: Optional[int] = 0, report_heldout_per_epoch: bool = False) -> CVResults

Run cross-validation with JAX/Flax model

Parameters: X: Feature array y: Target array epochs: Number of training epochs per fold optimizer: Optax optimizer (default: Adam with lr=1e-3) loss_fn: Custom loss function callbacks: List of trustcv callbacks groups: Group labels for grouped CV inner_validation: Fraction of each training fold set aside for model selection. val_* metrics for EarlyStopping and ModelCheckpoint are computed on it; the held-out fold is scored once, after training. 0 trains for the full epochs with no val_* metrics. Restoring best weights restores the TrainState; non-parameter collections (BatchNorm statistics) are not handled. inner_validation_strategy: "auto" (group, then stratified, then random) or "last" (the last part of each training fold, for time-ordered data). inner_random_state: Seed for the inner split (offset by the fold index). report_heldout_per_epoch: Also record held-out loss per epoch in metadata["heldout_curves"], for plotting only. Callbacks never see it.

Returns: CVResults object with scores and optionally states

KerasBaseWrap ¶

KerasBaseWrap(build_fn: Callable[..., Any], epochs: int = 10, batch_size: int = 32, verbose: int = 0, build_kwargs: Optional[Dict[str, Any]] = None, compile_kwargs: Optional[Dict[str, Any]] = None, task: Optional[str] = 'auto', threshold: float = 0.5, proba_mode: str = 'auto', multilabel: Optional[bool] = None, output_transform: Optional[Callable[[Any], Any]] = None, output_key: Optional[str] = None, output_index: Optional[int] = None, allow_dataset: bool = True, accept_dict_inputs: bool = True, squeeze_regression: bool = True, fit_kwargs: Optional[Dict[str, Any]] = None, random_state: Optional[int] = 42)

Bases: BaseEstimator

Base wrapper that handles model creation, compilation, and raw prediction.

KerasClassifierWrap ¶

KerasClassifierWrap(build_fn: Callable[..., Any], epochs: int = 10, batch_size: int = 32, verbose: int = 0, build_kwargs: Optional[Dict[str, Any]] = None, compile_kwargs: Optional[Dict[str, Any]] = None, task: Optional[str] = 'auto', threshold: float = 0.5, proba_mode: str = 'auto', multilabel: Optional[bool] = None, output_transform: Optional[Callable[[Any], Any]] = None, output_key: Optional[str] = None, output_index: Optional[int] = None, allow_dataset: bool = True, accept_dict_inputs: bool = True, squeeze_regression: bool = True, fit_kwargs: Optional[Dict[str, Any]] = None, random_state: Optional[int] = 42)

Bases: KerasBaseWrap, ClassifierMixin

Classification-friendly wrapper with probability + label handling.

KerasRegressorWrap ¶

KerasRegressorWrap(build_fn: Callable[..., Any], epochs: int = 10, batch_size: int = 32, verbose: int = 0, build_kwargs: Optional[Dict[str, Any]] = None, compile_kwargs: Optional[Dict[str, Any]] = None, task: Optional[str] = 'auto', threshold: float = 0.5, proba_mode: str = 'auto', multilabel: Optional[bool] = None, output_transform: Optional[Callable[[Any], Any]] = None, output_key: Optional[str] = None, output_index: Optional[int] = None, allow_dataset: bool = True, accept_dict_inputs: bool = True, squeeze_regression: bool = True, fit_kwargs: Optional[Dict[str, Any]] = None, random_state: Optional[int] = 42)

Bases: KerasBaseWrap, RegressorMixin

Regression wrapper.

inner_validation_split ¶

inner_validation_split(train_idx: Any, y: Any = None, groups: Any = None, *, fraction: float = 0.1, strategy: str = 'auto', random_state: Optional[int] = None) -> Tuple[Optional[Tuple[ndarray, ndarray]], str]

Split a training fold into inner-train and inner-validation indices.

Parameters:

Name	Type	Description	Default
`train_idx`	`array-like of int`	Indices of the outer training fold. May contain repeats (bootstrap); a repeated row always stays on one side, and multiplicity and order are preserved.	required
`y`	`array - like`	Full target array (indexed by `train_idx`); enables stratification for classification-like targets.	`None`
`groups`	`array - like`	Full group array; when given, no group is on both sides.	`None`
`fraction`	`float`	Share of the training fold (by unique rows, or by groups) used for inner validation; must be in (0, 1).	`0.1`
`strategy`	`(auto, last)`	`"auto"`: group split if `groups`, else stratified when feasible, else random. `"last"`: the last `fraction` of `train_idx` in its given order, for chronologically ordered data; with `groups`, earlier rows of a group that reaches the validation tail are left out of inner training (purged), so validation never precedes training.	`"auto"`
`random_state`	`int`	Seed for the random strategies.	`None`

Returns:

Type	Description
`(split, reason)`	`split` is `(inner_train_idx, inner_val_idx)` or `None` when no valid split exists; `reason` names the strategy used or why it failed.

Callbacks¶

CVCallback ¶

Bases: ABC

Base callback class for cross-validation events

Callbacks allow for monitoring, early stopping, checkpointing, and custom logic during cross-validation.

on_cv_start ¶

on_cv_start(n_splits: int) -> None

Called at the start of cross-validation

on_fold_start ¶

on_fold_start(fold_idx: int, train_idx: ndarray, val_idx: ndarray) -> None

Called at the start of each fold

on_epoch_start ¶

on_epoch_start(epoch: int, fold_idx: int) -> None

Called at the start of each epoch (for iterative training)

on_epoch_end ¶

on_epoch_end(epoch: int, fold_idx: int, logs: Optional[Dict[str, Any]] = None) -> Optional[str]

Called at the end of each epoch

Returns: Optional string 'stop' to trigger early stopping

on_fold_end ¶

on_fold_end(fold_idx: int, results: Dict[str, Any]) -> None

Called at the end of each fold

on_cv_end ¶

on_cv_end(all_results: List[Dict[str, Any]]) -> None

Called at the end of cross-validation

EarlyStopping ¶

EarlyStopping(monitor: str = 'val_loss', patience: int = 5, mode: str = 'min', restore_best: bool = True, min_delta: float = 0.0001, verbose: bool = True)

Bases: CVCallback

Early stopping callback to prevent overfitting

Implements best practices for early stopping with patience and optional restoration of best model weights.

Initialize early stopping callback

Parameters: monitor: Metric to monitor patience: Number of epochs with no improvement to wait mode: 'min' or 'max' - whether lower or higher is better restore_best: Keep a copy of the best epoch's weights; the runner restores them when training for the fold ends (early stop or last epoch) min_delta: Minimum change to qualify as improvement verbose: Whether to print messages

on_fold_start ¶

on_fold_start(fold_idx: int, train_idx: ndarray, val_idx: ndarray) -> None

Reset early stopping for new fold

on_epoch_end ¶

on_epoch_end(epoch: int, fold_idx: int, logs: Optional[Dict[str, Any]] = None) -> Optional[str]

Check if training should stop

ModelCheckpoint ¶

ModelCheckpoint(filepath: str, monitor: str = 'val_loss', mode: str = 'min', save_best_only: bool = True, save_freq: int = 1, verbose: bool = True)

Bases: CVCallback

Save model checkpoints during training

Implements best practices for model checkpointing including saving best models and periodic snapshots.

Initialize model checkpoint callback

Parameters: filepath: Path pattern for saving models (can include {fold} and {epoch}) monitor: Metric to monitor for best model mode: 'min' or 'max' - whether lower or higher is better save_best_only: Only save when monitored metric improves save_freq: Without save_best_only, save every save_freq-th epoch (epochs 5, 10, ... for 5, counting from 1) verbose: Whether to print messages

on_fold_start ¶

on_fold_start(fold_idx: int, train_idx: ndarray, val_idx: ndarray) -> None

Reset best score for new fold

on_epoch_end ¶

on_epoch_end(epoch: int, fold_idx: int, logs: Optional[Dict[str, Any]] = None) -> Optional[str]

Save model checkpoint if conditions are met

ProgressLogger ¶

ProgressLogger(log_file: Optional[str] = None, metrics: Optional[List[str]] = None, verbose: int = 1, groups: Optional[ndarray] = None)

Bases: CVCallback

Log progress during cross-validation

Provides detailed logging for documentation and debugging.

Initialize progress logger

Parameters: log_file: Optional file to write logs to metrics: Specific metrics to log verbose: Verbosity level (0=silent, 1=progress, 2=detailed)

set_groups ¶

set_groups(groups: Optional[ndarray]) -> None

Attach group labels so we can report group counts per fold.

on_cv_start ¶

on_cv_start(n_splits: int) -> None

Log CV start

on_fold_start ¶

on_fold_start(fold_idx: int, train_idx: ndarray, val_idx: ndarray) -> None

Log fold start

on_epoch_end ¶

on_epoch_end(epoch: int, fold_idx: int, logs: Optional[Dict[str, Any]] = None) -> Optional[str]

Log epoch metrics

on_fold_end ¶

on_fold_end(fold_idx: int, results: Dict[str, Any]) -> None

Log fold results

on_cv_end ¶

on_cv_end(all_results: List[Dict[str, Any]]) -> None

Log CV completion and save logs if requested

ClassDistributionLogger ¶

ClassDistributionLogger(labels, label_names: Optional[Dict[Any, str]] = None, verbose: int = 1, decimals: int = 1, output_key: Optional[str] = None, output_index: Optional[int] = None)

Bases: CVCallback

Display per-fold class composition for train/validation splits.

Useful to verify that each fold preserves label balance.

Parameters: labels: Array-like target variable used for splitting label_names: Optional mapping {raw_label: display_name} verbose: 0=silent, 1=summary per fold decimals: Number of decimal places for percentages output_key/index: For multi-output labels (dict/list), choose which head to log

LeakageDetectionCallback ¶

LeakageDetectionCallback(data, groups=None, timestamps=None, coordinates=None, verbose: int = 1)

Bases: CVCallback

Automatic data leakage detection callback for UniversalCVRunner.

Checks each fold for data leakage issues (duplicate samples, group overlap, near-duplicates) and reports a summary at the end of cross-validation.

Parameters:

Name	Type	Description	Default
`data`	`tuple`	Full dataset as (X, y) or (X, y, groups)	required
`groups`	`array - like`	Group identifiers (e.g., patient IDs)	`None`
`timestamps`	`array - like`	Temporal information	`None`
`coordinates`	`array - like`	Spatial coordinates	`None`
`verbose`	`int`	Verbosity level (0=silent, 1=summary, 2=per-fold)	`1`

on_fold_start ¶

on_fold_start(fold_idx: int, train_idx: ndarray, val_idx: ndarray) -> None

Run leakage checks on the current fold split.

on_cv_end ¶

on_cv_end(all_results: List[Dict[str, Any]]) -> None

Aggregate fold reports and print a summary.

RegulatoryComplianceLogger ¶

RegulatoryComplianceLogger(output_dir: str, study_name: str, include_data_characteristics: bool = True, include_model_details: bool = True)

Bases: CVCallback

Log run metadata and per-fold results to a JSON file

The log can support regulatory documentation; it records what was run, not whether any regulation's requirements are met.

Initialize the run logger

Parameters: output_dir: Directory for the JSON logs study_name: Name of the study/experiment include_data_characteristics: Log data distribution info include_model_details: Log model architecture details

on_cv_start ¶

on_cv_start(n_splits: int) -> None

Document CV methodology

on_fold_start ¶

on_fold_start(fold_idx: int, train_idx: ndarray, val_idx: ndarray) -> None

Document data split for each fold

on_fold_end ¶

on_fold_end(fold_idx: int, results: Dict[str, Any]) -> None

Document fold results

on_cv_end ¶

on_cv_end(all_results: List[Dict[str, Any]]) -> None

Write the run log (metadata and per-fold results) to a JSON file

Datasets¶

load_heart_disease ¶

load_heart_disease(return_X_y: bool = False, as_frame: bool = True) -> Union[Bunch, Tuple]

Load heart disease dataset (synthetic version for demo)

Parameters:

Name	Type	Description	Default
`return_X_y`	`bool`	If True, returns (X, y) tuple only.	`False`
`as_frame`	`bool`	If True, returns pandas DataFrame/Series. If False, returns numpy arrays.	`True`

Returns:

Type	Description
`Bunch or tuple`	By default, returns sklearn-style Bunch with attributes: - data: Features (age, blood pressure, cholesterol, etc.) - target: Binary target (0: no disease, 1: disease) - feature_names: List of feature names - target_names: List of target class names - patient_ids: Patient identifiers - DESCR: Dataset description If return_X_y=True, returns (X, y) tuple instead.

Examples:

>>> # sklearn-style API (recommended)
>>> dataset = load_heart_disease()
>>> X, y = dataset.data, dataset.target
>>> print(f"Dataset shape: {X.shape}")

>>> # Direct tuple unpacking
>>> X, y = load_heart_disease(return_X_y=True)

load_diabetic_readmission ¶

load_diabetic_readmission(return_X_y: bool = False, as_frame: bool = True) -> Union[Bunch, Tuple]

Load diabetic patient readmission dataset (synthetic)

Parameters:

Name	Type	Description	Default
`return_X_y`	`bool`	If True, returns (X, y) tuple only.	`False`
`as_frame`	`bool`	If True, returns pandas DataFrame/Series. If False, returns numpy arrays.	`True`

Returns:

Type	Description
`Bunch or tuple`	By default, returns sklearn-style Bunch with attributes: - data: Features - target: Readmission within 30 days (0: no, 1: yes) - feature_names: List of feature names - target_names: List of target class names - patient_ids: Patient identifiers - admission_dates: Hospital admission dates - DESCR: Dataset description If return_X_y=True, returns (X, y) tuple instead.

Examples:

>>> # sklearn-style API (recommended)
>>> dataset = load_diabetic_readmission()
>>> X, y = dataset.data, dataset.target
>>> print(f"Readmission rate: {y.mean():.1%}")

>>> # Direct tuple unpacking
>>> X, y = load_diabetic_readmission(return_X_y=True)

load_cancer_imaging ¶

load_cancer_imaging(return_X_y: bool = False) -> Union[Bunch, Tuple]

Load simulated cancer imaging features dataset

Parameters:

Name	Type	Description	Default
`return_X_y`	`bool`	If True, returns (X, y) tuple only.	`False`

Returns:

Type	Description
`Bunch or tuple`	By default, returns sklearn-style Bunch with attributes: - data: Imaging features (texture, intensity, shape metrics) - target: Cancer presence (0: benign, 1: malignant) - feature_names: List of feature names - target_names: List of target class names - patient_ids: Patient identifiers - DESCR: Dataset description If return_X_y=True, returns (X, y) tuple instead.

Examples:

>>> # sklearn-style API (recommended)
>>> dataset = load_cancer_imaging()
>>> X, y = dataset.data, dataset.target
>>> print(f"Feature dimensions: {X.shape[1]}")

>>> # Direct tuple unpacking
>>> X, y = load_cancer_imaging(return_X_y=True)

generate_synthetic_ehr ¶

generate_synthetic_ehr(n_samples: int = 1000, n_features: int = 20, n_patients: Optional[int] = None, temporal: bool = False, prevalence: float = 0.3, random_state: int = 42, return_X_y: bool = False) -> Dict

Generate synthetic Electronic Health Record data

Parameters:

Name	Type	Description	Default
`n_samples`	`int`	Number of records	`1000`
`n_features`	`int`	Number of features	`20`
`n_patients`	`int`	Number of unique patients (for grouped data)	`None`
`temporal`	`bool`	Whether to include temporal information	`False`
`prevalence`	`float`	Disease prevalence	`0.3`
`random_state`	`int`	Random seed	`42`
`return_X_y`	`bool`	If True, returns (X, y) tuple instead of dict. This provides sklearn-compatible API.	`False`

Returns:

Type	Description
`dict or tuple`	Dictionary containing X, y, patient_ids, and optionally timestamps. If return_X_y=True, returns (X, y) tuple instead.

Examples:

>>> data = generate_synthetic_ehr(n_samples=1000, temporal=True)
>>> X, y = data['X'], data['y']
>>> print(f"Generated {len(X)} records with {X.shape[1]} features")

>>> # sklearn-compatible API
>>> X, y = generate_synthetic_ehr(return_X_y=True)

generate_temporal_patient_data ¶

generate_temporal_patient_data(n_patients: int = 100, n_timepoints: int = 12, n_features: int = 10, outcome_type: str = 'binary', missing_rate: float = 0.1, random_state: int = 42, return_X_y: bool = False) -> Dict

Generate temporal patient data with realistic patterns

Parameters:

Name	Type	Description	Default
`n_patients`	`int`	Number of patients	`100`
`n_timepoints`	`int`	Number of time points per patient	`12`
`n_features`	`int`	Number of features	`10`
`outcome_type`	`str`	'binary', 'continuous', or 'survival'	`'binary'`
`missing_rate`	`float`	Proportion of missing values	`0.1`
`random_state`	`int`	Random seed	`42`
`return_X_y`	`bool`	If True, returns (X, y) tuple instead of dict. This provides sklearn-compatible API.	`False`

Returns:

Type	Description
`dict or tuple`	Dictionary with temporal patient data. If return_X_y=True, returns (X, y) tuple instead.

Examples:

>>> data = generate_temporal_patient_data(n_patients=50, n_timepoints=24)
>>> print(f"Shape: {data['X'].shape}")
>>> print(f"Unique patients: {data['patient_ids'].nunique()}")

>>> # sklearn-compatible API
>>> X, y = generate_temporal_patient_data(return_X_y=True)

Aliases¶

Alternative names kept for compatibility. They refer to the same objects.

Alias	Same as
`TrustCV`	`TrustCVValidator`
`MedicalValidator`	`TrustCVValidator (deprecated)`
`LeakageChecker`	`DataLeakageChecker`
`ClassBalanceChecker`	`BalanceChecker`
`KFold`	`KFoldMedical`
`StratifiedKFold`	`StratifiedKFoldMedical`
`LeaveOneOut`	`LOOCV`
`LeavePOut`	`LPOCV`
`GroupKFold`	`GroupKFoldMedical`
`PatientGroupKFold`	`GroupKFoldMedical`
`TemporalClinical`	`TimeSeriesSplit`
`BlockedTimeSeriesSplit`	`BlockedTimeSeries`
`RollingWindowSplit`	`RollingWindowCV`
`ExpandingWindowSplit`	`ExpandingWindowCV`
`PurgedKFold`	`PurgedKFoldCV`
`CombinatorialPurgedKFold`	`CombinatorialPurgedCV`
`SpatialBlockSplit`	`SpatialBlockCV`
`BufferedSpatialSplit`	`BufferedSpatialCV`
`SpatiotemporalBlockSplit`	`SpatiotemporalBlockCV`
`EnvironmentalHealthSplit`	`EnvironmentalHealthCV`
`KerasSkWrap`	`KerasClassifierWrap`

API reference¶

Validation¶

TrustCVValidator ¶

validate ¶

fit_validate ¶

suggest_best_method ¶

ValidationResult dataclass ¶

metric_feasibility_warnings property ¶

summary ¶

dashboard ¶

to_dict ¶

CheckResult dataclass ¶

recommend_cv ¶

CVRecommendation dataclass ¶

Splitters: independent samples¶

HoldOut ¶

split ¶

get_n_splits ¶

KFoldMedical ¶

split ¶

get_n_splits ¶

StratifiedKFoldMedical ¶

split ¶

get_n_splits ¶

RepeatedKFold ¶

split ¶

get_n_splits ¶

LOOCV ¶

split ¶

get_n_splits ¶

LPOCV ¶

split ¶

get_n_splits ¶

BootstrapValidation ¶

split ¶

compute_error ¶

get_n_splits ¶

MonteCarloCV ¶

split ¶

get_n_splits ¶

NestedCV ¶

fit_predict ¶

get_n_splits ¶

Splitters: grouped¶

GroupKFoldMedical ¶

split ¶

get_n_splits ¶

StratifiedGroupKFold ¶

split ¶

get_n_splits ¶

LeaveOneGroupOut ¶

split ¶

get_n_splits ¶

LeavePGroupsOut ¶

split ¶

get_n_splits ¶

RepeatedGroupKFold ¶

split ¶

get_n_splits ¶

HierarchicalGroupKFold ¶

split ¶

get_n_splits ¶

MultilevelCV ¶

split ¶

get_n_splits ¶

NestedGroupedCV ¶

fit_predict ¶

get_n_splits ¶

Splitters: temporal¶

TimeSeriesSplit ¶

split ¶

get_n_splits ¶

BlockedTimeSeries ¶

split ¶

RollingWindowCV ¶

split ¶

get_n_splits ¶

ExpandingWindowCV ¶

split ¶

get_n_splits ¶

ValidationResult `dataclass` ¶

metric_feasibility_warnings `property` ¶

CheckResult `dataclass` ¶

CVRecommendation `dataclass` ¶

LeakageReport `dataclass` ¶

summary `property` ¶

CVResults `dataclass` ¶

metric_feasibility_warnings `property` ¶

fold_details `property` ¶

metrics `property` ¶

mean_score `property` ¶

mean_scores `property` ¶

std_scores `property` ¶