Alignment, Reasoning, and Trustworthiness

SFT

D1Param. Upd.
D2Align.SafetyTask Spec.
D3Instr.-Resp. Corpus
D4Ad-hoc Perm.Sched. Perm.
D5Part.Whole
D6LLMMLLM

In brief — site editorial, not from the paper

Trains a model on curated instruction-and-response pairs so it answers in the format and manner people expect.

Definition

SFT, also termed instruction tuning, establishes the foundational behavioral capability for an LLM to interact via natural language instructions [152], and its taxonomic position demands precise separation from classical task-specific FT. Both employ Parametric Update via supervised GD, but the D3 (Data Requirements) divergence is categorical: task FT operates on Small Labeled datasets of narrowly scoped, single-function examples yielding a bounded operational envelope, whereas SFT employs a massively heterogeneous Instruction-Response Corpus—spanning question answering, summarization, code generation, reasoning chains, and multilingual instructions—to instill a general instruction-following capability that cuts across all task boundaries. This D3 (Data Requirements) divergence propagates into the D2 (Goal) profile: SFT achieves a multi-domain form of Task Specialization alongside baseline Alignment, Safety, rather than the single-function specialization of task FT. While traditional SFT relies on human-curated datasets, contemporary variants frequently utilize synthetic, model-generated data [209] (e.g., Instruction Backtranslation [120], RAFT [54], and Self-Rewarding models [225]). This introduces a critical D3 (Data Requirements) nuance: it shifts the compliance and bias-auditing burden away from human annotator governance and onto the upstream teacher model generating the synthetic corpus. Safety SFT is a specific instantiation that explicitly enforces Safety by restricting the corpus to refusal demonstrations and harm-category filtered examples. The D5 (Scope) coordinate—Partial, Whole-Model—is implementation-dependent: SFT applied to all parameters yields Whole-Model with a full-system regression burden. Because SFT reshapes the model's behavioral envelope across all tasks simultaneously—not just a single bounded function—it demands comprehensive cross-domain behavioral regression testing before re-deployment; a task-specific regression suite is insufficient; the validation protocol must cover the full instruction-following surface, including all task types represented in the instruction-response corpus.

Verbatim from the paper — Alignment, Reasoning, and Trustworthiness

Nearest profiles

DPO (0.22), RLAIF/CAI (0.22), RLHF (LLM Alignment) (0.28), RLVR/GRPO (0.33), Multimodal Instr. Tuning (0.38)

Computed from the taxonomy data (Gower distance over all six dimensions)

References

  1. [54] Hanze Dong et al. 2023. RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment. arXiv
  2. [120] Xian Li et al. 2023. Self-Alignment with Instruction Backtranslation.
  3. [152] Long Ouyang et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems
  4. [209] Yizhong Wang et al. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv
  5. [225] Weizhe Yuan et al. 2024. Self-rewarding language models. Proceedings of the 41st International Conference on Machine Learning

Numbered as in the paper

Cite this row

SFT — six-dimensional profile
D1: parametric-update
D2: alignment, safety, task-specialization
D3: instruction-response-corpus
D4: ad-hoc-permanent, scheduled-permanent
D5: partial, whole-model
D6: llm, mllm
Source: arXiv:2608.06246
Compare with another technique Find in the explorer