Training Strategies
Self-Play
In brief — site editorial, not from the paper
Improves a model by having it compete against or critique itself, generating its own training signal without external labels.
Definition
Self-play generates training data by having a model compete against or evaluate itself, producing an auto-curriculum of increasing difficulty without requiring externally provided labeled data. The Environment-Derived coordinate is the defining structural property: the training signal is generated internally through the model's own interactions with copies or historical checkpoints of itself—no human annotation, no static corpus, and no external oracle are involved in data generation. This self-referential data source produces the compound Alignment, Reasoning, Task Specialization profile—the three operationally distinct goals served by self-play at different D6 (Model Type) levels: domain mastery through competitive co-evolution in DL game agents (Task Specialization) [182], instruction-following improvement through self-generated preference pairs in LLM alignment (Alignment) [42], and chain-of-thought reasoning cultivation through iterative self-verification in reasoning models (Reasoning) [226]. This multi-goal profile would be irresolvable in a single-axis taxonomy; the taxonomy's value is precisely in making explicit that the same D1 (Mechanism) and D3 (Data Requirements) regime serve fundamentally different D2 (Goal) depending on the deployment context—and that each dictates entirely distinct evaluation metrics, regression test suites, and operational baselines. Compared to Curriculum Learning, which also generates structured training sequences, self-play mainly differs on D1 (Mechanism) (Parametric Update vs. Pipeline-Mediated) and D3 (Data Requirements) (Environment-Derived vs. Pipeline-Dependent)—Curriculum Learning sequences existing data, while self-play generates new data through model-environment interaction. The DL scope (skipping FM) reflects that standard FM FT pipelines do not typically employ self-play, which re-emerges only as a specialized adaptation mechanism for alignment and reasoning at the LLM tier. Because the specific D2 (Goal) being pursued dictates the validation protocol, any self-play deployment must explicitly document which goal dimension applies, providing a clear audit trail for the intended-purpose and risk-rationale in compliance documentation.
Verbatim from the paper — Training Strategies
Notes from the table
Self-play skips the FM tier because standard FM pre-training and FT pipelines do not employ self-play; it re-emerges at the LLM level through RLHF-adjacent techniques (e.g., self-play FT, SPIN ) that exploit autoregressive generation as an opponent model.
Nearest profiles
FT (full) (0.36), MTL (0.36), Adversarial Training (0.39), Retraining (0.43), Training (0.43)
Computed from the taxonomy data (Gower distance over all six dimensions)
References
- [42] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu 2024. Self-play fine-tuning converts weak language models to strong language models. Proceedings of the 41st International Conference on Machine Learning
- [182] David Silver et al. 2017. Mastering the game of Go without human knowledge. Nature
- [226] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman 2022. STaR: self-taught reasoner bootstrapping reasoning with reasoning. Proceedings of the 36th International Conference on Neural Information Processing Systems
Numbered as in the paper
Cite this row
Self-Play — six-dimensional profile D1: parametric-update D2: alignment, reasoning, task-specialization D3: environment-derived D4: ad-hoc-permanent, scheduled-permanent D5: whole-model D6: dl, llm Source: arXiv:2608.06246Compare with another technique Find in the explorer