KD

KD transfers a trained teacher model's learned behavioral distribution into a smaller student model, most commonly by training the student to match the teacher's soft probability output distribution, enabling compression of capability without commensurate compression of performance ^{[72, 84]}. Its Cross-Model Transfer classification captures the defining structural property: a complete, separately trained model is the proximate information source for the student's training—not a labeled dataset, not a teacher's parameters directly, but the teacher's output distribution. This cross-model information flow distinguishes KD from all other compression-adjacent techniques: Model Compression operates on a single model's own parameters while KD requires a functioning teacher model to generate training signal, making Teacher-Derived. The Computational Efficiency goal drives the student's architectural reduction (smaller, faster inference). From a validation standpoint, a KD event establishes a strict behavioral dependency chain: verifying the student model requires comprehensive technical documentation of both the teacher model's original training data provenance and the exact algorithmic distillation protocol used to generate the soft targets, ensuring full auditability of the behavioral alignment downstream. The student artifact inherits the teacher's behavioral envelope but introduces a compression-induced approximation gap; this gap must be characterized quantitatively—via behavioral regression testing across the teacher's validated operational envelope.