DPO

DPO eliminates the explicit reward model required by RLHF, recasting alignment as a supervised classification objective directly on preference pairs ^[164]. DPO shares D1 (Mechanism), D2 (Goal), D3 (Data Requirements), and D4 (Persistence) with RLHF—making the D5 (Scope) divergence the sole but operationally decisive difference between them. DPO's elimination of the reward model collapses D5 (Scope) to Partial, Whole-Model alone, removing the Surrogate Model component entirely. This D5 (Scope) reduction is the most system-engineering-consequential property of DPO: it eliminates the need for a separate reward model validation pipeline, and the three-stage orchestration infrastructure. Compared to RLAIF/CAI, DPO shares the same D5 (Scope) simplification but retains the Preference Pairs human-annotation requirement; RLAIF substitutes AI-generated constitutional preferences (Constitution-Derived), shifting the validation burden from annotator oversight to constitutional text auditing. DPO's structural simplicity narrows the validation surface to a single policy artifact; however, the preference pair dataset remains the sole provenance record determining the alignment boundary, and its annotator-selection protocol, demographic coverage, and inter-rater reliability metrics must be fully documented as the system's behavioral specification. Recent advancements have expanded this family into broader preference-optimization variants, including IPO ^[22], KTO ^[58], and SimPO ^[142]. Notably, ORPO ^[86] collapses the traditional multi-stage pipeline by integrating baseline instruction tuning and alignment into a single gradient update.