Knowledge Transfer and Task Specialization

PEFT (LoRA, adapters)

D1Param. Upd.
D2Comp. Eff.Task Spec.
D3Small Labeled
D4Ad-hoc Perm.Sched. Perm.
D5ModularPart.
D6DLFMLLMMLLM

In brief — site editorial, not from the paper

Freezes the base model and trains a small add-on instead, so the original weights are never touched and the adapter can be swapped out or removed cleanly.

Definition

PEFT is taxonomically the most distinctive member of the FT family, differentiated by its Modular, Partial scope, which no full or partial FT variant achieves. The defining operational property is that the base model's weights remain strictly frozen throughout training; all adaptation occurs within a small set of newly introduced parameters appended to or inserted into the frozen architecture [90]. This produces the Modular coordinate: the adapter is a structurally independent artifact that can be loaded, swapped, or removed without touching the base checkpoint. This Modular property creates a structural bridge to Task Arithmetic and Model Merging, where adapter task-vectors can be algebraically combined. The data requirement contracts to Small Labeled relative to Full FT because the adapter's parameter count is orders of magnitude smaller than the base. At the LLM and MLLM tiers, PEFT simultaneously absorbs the distributional gap goal of DA when applied to domain-specific corpora, and its modular architecture makes it the preferred complement to alignment techniques (RLHF, DPO) in production stacks. Because a change-control event in a PEFT deployment affects only the adapter artifact—not the frozen base—regression testing scope is bounded to the adapter module alone; the base model's existing validation record remains unmodified, enabling clean rollback by removing the adapter checkpoint without any base-model reversion. While categorized as a single structural decision unit in the taxonomy, the PEFT paradigm encompasses a wide array of architectural variants. These include quantization-aware methods (e.g., QLoRA [52]), continuous prompt interventions (prefix-tuning, prompt-tuning, p-tuning), sparse bias updates (BitFit [25]), and attention-scaling (IA3 [125]). Furthermore, dynamic adapter composition strategies, such as adapter fusion and LoRA routing (e.g., X-LoRA [35], LoraHub [94], S-LoRA [177]), functionally blur the boundary between modular PEFT and MoE by dynamically routing inputs to specialized adapter modules.

Verbatim from the paper — Knowledge Transfer and Task Specialization

Related techniques

  • umbrella FT (full) — PEFT is a specialisation within the fine-tuning family
    Full FT is the maximally expressive parametric realization of TL Appendix C, Knowledge Transfer and Task Specialization
  • sub-technique FT (partial) — PEFT strictly extends Partial FT: same mechanism, plus a modular scope
    the base model's weights remain strictly frozen throughout training Appendix C, Knowledge Transfer and Task Specialization
  • bridge Task Arith. & Model Merging — PEFT's modular scope lets adapter task-vectors be combined algebraically
    structural bridge to Task Arithmetic and Model Merging, where adapter task-vectors can be algebraically combined Appendix C, Knowledge Transfer and Task Specialization
  • supersession DA — Domain-specific PEFT absorbs the distributional-gap goal at the LLM tier
    absorbs the distributional gap goal of DA when applied to domain-specific corpora Appendix C, Knowledge Transfer and Task Specialization

Nearest profiles

FT (partial) (0.17), Prompt Learning (0.21), LP (0.25), ReFT (0.25), MoE (0.32)

Computed from the taxonomy data (Gower distance over all six dimensions)

References

  1. [25] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel 2022. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
  2. [35] Eric L. Buehler, and Markus J. Buehler 2024. X-LoRA: Mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design. APL Mach. Learn.
  3. [52] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer 2023. QLoRA: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems
  4. [90] Edward J. Hu et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models.
  5. [94] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin 2024. LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition.
  6. [125] Haokun Liu et al. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Proceedings of the 36th International Conference on Neural Information Processing Systems
  7. [177] Ying Sheng et al. 2024. S-LoRA: Serving Thousands of Concurrent LoRA Adapters.

Numbered as in the paper

Cite this row

PEFT (LoRA, adapters) — six-dimensional profile
D1: parametric-update
D2: computational-efficiency, task-specialization
D3: small-labeled
D4: ad-hoc-permanent, scheduled-permanent
D5: modular, partial
D6: dl, fm, llm, mllm
Source: arXiv:2608.06246
Compare with another technique Find in the explorer