Alignment, Reasoning, and Trustworthiness

RLHF (LLM Alignment)

In brief — site editorial, not from the paper

Uses human preference comparisons to train a reward model, then tunes the model against it — the standard route to making outputs helpful and safe.

Definition

RLHF. RLHF trains a reward model on human preference pairs and optimizes an LLM policy against it, constituting the canonical alignment technique for generative models ^[43]. The Preference Pairs coordinate is the mechanism's defining input requirement and its most consequential data-governance obligation: unlike Small Labeled datasets annotated for a fixed objective, preference pairs encode comparative behavioral judgments—which of two model outputs a human annotator prefers—making annotator demographics, selection criteria, and inter-rater reliability first-class artifacts in the training provenance record. The Alignment, Safety goal and the strict LLM, MLLM bound reflect that preference-based alignment is architecturally contingent on the model's capacity for open-ended natural-language generation; the preference signal is structurally meaningless for discriminative or shallow architectures. The D5 (Scope) coordinate is implementation-dependent in a way that carries direct validation consequences: applying RL policy optimization across all network parameters yields Whole-Model, demanding full-system regression testing; applying it via PEFT adapters restricts the scope to Partial, isolating the change-control footprint to the adapter; this structural distinction is operationally critical, as it determines whether an alignment modification event demands full-system regression testing or a targeted, component-level validation assessment. Training the reward model introduces a third Surrogate Model component—a fully independent model artifact that triggers its own conformity validation chain. This last point is the taxonomically critical divergence from DPO: both share D1 (Mechanism), D2 (Goal), D3 (Data Requirements), and D4 (Persistence), but RLHF's reward model produces a Surrogate Model footprint that DPO eliminates entirely by recasting alignment as a direct classification objective. RLHF deployments require independent validation of the reward model artifact as a separate system component; any update to the reward model constitutes a distinct change-control event from the policy update, as the reward model defines the optimization target for all subsequent policy training.

Verbatim from the paper — Alignment, Reasoning, and Trustworthiness

Notes from the table

While preference-based RL predates foundation models, this row explicitly scopes its modern instantiation for generative alignment at the lm tiers. flushleft

Highly flexible techniques such as CE (e.g., prompting for step-by-step rationales) and RLHF (e.g., penalizing unexplainable refusals) can dynamically adopt Explainability as a primary goal when engineered specifically to satisfy regulatory transparency mandates.

Nearest profiles

DPO (0.06), RLAIF/CAI (0.22), SFT (0.28), RLVR/GRPO (0.39), Multimodal Instr. Tuning (0.42)

Computed from the taxonomy data (Gower distance over all six dimensions)

References

[43] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei 2017. Deep reinforcement learning from human preferences. Proceedings of the 31st International Conference on Neural Information Processing Systems

Numbered as in the paper

Cite this row

RLHF (LLM Alignment) — six-dimensional profile
D1: parametric-update
D2: alignment, safety
D3: preference-pairs
D4: ad-hoc-permanent, scheduled-permanent
D5: partial, surrogate-model, whole-model
D6: llm, mllm
Source: arXiv:2608.06246

Compare with another technique Find in the explorer