Skip to content

Alignment, Reasoning, and Trustworthiness

LEACE

In brief — site editorial, not from the paper

Surgically removes a concept's linear influence from the model's representations — used for fairness work — with a closed-form edit rather than training.

Definition

LEACE targets Fairness through a mechanistically novel D1 (Mechanism) pathway: rather than employing iterative GD, it operates via Parameter Composition—computing a covariance-based orthogonal projection from a Small Labeled and algebraically projecting the model's weight matrices into the null space of a target concept's linear representation [24], a parameter composition operation analogous to Task Arithmetic but operating on concept subspaces rather than task capability directions. This closed-form linear algebra operation bypasses the computational footprint of traditional parameter-updating methods entirely, producing a Ad-hoc Permanent, Partial modification at negligible training cost. Compared to LP, which trains a probe head on top of a frozen base to expose linear concept encodings, LEACE inverts the operation: it takes the same linear concept subspace identified by probing and algebraically projects it out of the base model's weight matrices—making LP and LEACE complementary, with LP as the diagnostic step and LEACE as the corrective intervention targeting the same identified representation. The Ad-hoc Permanent Permanent coordinate reflects that LEACE produces an irreversible, event-driven modification at negligible compute cost—unlike gradient-based fairness interventions that require training runs, LEACE can be applied as a post-hoc weight surgery. The Partial scope is bounded by the targeted layers: only the weight matrices encoding the concept's linear subspace are projected; all other parameters remain unchanged. Its operational and validation significance lies in the deterministic auditability of the intervention: because the modification is derived analytically from a strict concept specification, it is fully reproducible and mathematically characterizable; the concept subspace, projection matrix, and labeled probe dataset constitute a complete audit trail for the fairness intervention—validation requires verifying that the modified representation is linearly indecipherable for the target concept while baseline task performance is preserved, both of which are statistically testable properties.

Verbatim from the paper — Alignment, Reasoning, and Trustworthiness

Nearest profiles

Knowledge Editing (0.33), FT (partial) (0.42), Prompt Learning (0.46), Machine Unlearning (0.50), PEFT (LoRA, adapters) (0.50)

Computed from the taxonomy data (Gower distance over all six dimensions)

References

  1. [24] Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman 2023. LEACE: Perfect linear concept erasure in closed form. Advances in Neural Information Processing Systems

Numbered as in the paper

Cite this row

LEACE — six-dimensional profile
D1: parameter-composition
D2: fairness
D3: small-labeled
D4: ad-hoc-permanent
D5: partial
D6: dl, fm, llm, mllm
Source: arXiv:2608.06246
Compare with another technique Find in the explorer