Efficiency and Composition

Task Arith. & Model Merging

D1Param. Comp.
D2Capab. Ext.Comp. Eff.
D3Param.-Only
D4Ad-hoc Perm.Sched. Perm.
D5Fused
D6FMLLMMLLM

In brief — site editorial, not from the paper

Combines separately tuned models by arithmetic on their weights — adding capabilities, or subtracting them by negating a task vector.

Definition

Task Arithmetic and Model Merging combine pre-trained and fine-tuned checkpoints into a single multi-capable model through arithmetic operations on parameter space, without any additional training data [100, 214, 218]. Its Parameter Composition and Parameter-Only classification suggest no gradient computation, no data consumption at merge time, no teacher model inference—only algebraic operations on weight tensors. Machine Unlearning shares Parameter Composition via task-vector negation (subtracting a forget-task vector to remove a learned capability), making Machine Unlearning a targeted polarity reversal of the exact same algebraic operation that Task Arithmetic uses to add capabilities. Compared to MTL, which achieves multi-task capability through joint optimization over a shared task-distribution corpus (Parametric Update, Task Distribution), Task Arithmetic achieves the same compound capability profile through post-hoc parameter composition requiring no additional training and no access to the original task datasets. To mitigate parameter interference and preserve specialized capabilities during the fusion process, practitioners employ advanced mathematical merge operators, including TIES-MERGING [218], DARE [223], SLERP [131, 180], and Fisher merging [135]. Consequently, the regression validation suites used to verify targeted machine unlearning—specifically, verifying the erasure of a specific capability while ensuring the preservation of adjacent foundational knowledge—are directly applicable to Task Arithmetic (task-vector operations). Because Model Merging bypasses traditional training telemetry, system validation must rely on downstream functional testing and post-merge automated regression benchmarks to verify that parameter-space interference has not degraded core model capabilities.

Verbatim from the paper — Efficiency and Composition

Related techniques

  • bridge PEFT (LoRA, adapters) — PEFT's modular scope lets adapter task-vectors be combined algebraically
    structural bridge to Task Arithmetic and Model Merging, where adapter task-vectors can be algebraically combined Appendix C, Knowledge Transfer and Task Specialization
  • hybrid Machine Unlearning — Task-vector-negation unlearning is mechanistically identical to task arithmetic
    mechanistically identical to Task Arithmetic executed with a negation operator Appendix C, Knowledge Modification and Activation-Based Adaptation

Nearest profiles

MoE (0.50), Model Compression (0.54), Long-Context Ext. (0.58), Modality-Spec. Adapters (0.61), KD (0.62)

Computed from the taxonomy data (Gower distance over all six dimensions)

References

  1. [100] Gabriel Ilharco et al. 2023. Editing Models with Task Arithmetic. arXiv
  2. [131] Wei Lu, Rachel K. Luu, and Markus J. Buehler 2025. Fine-tuning large language models for domain adaptation: exploration of training strategies, scaling, model merging and synergistic capabilities. npj Comput Mater
  3. [135] Michael Matena, and Colin Raffel 2022. Merging models with fisher-weighted averaging. Proceedings of the 36th International Conference on Neural Information Processing Systems
  4. [180] Ken Shoemake 1985. Animating rotation with quaternion curves. SIGGRAPH Comput. Graph.
  5. [214] Mitchell Wortsman et al. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. Proceedings of the 39th International Conference on Machine Learning
  6. [218] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal 2023. TIES-Merging: Resolving Interference When Merging Models. Advances in Neural Information Processing Systems
  7. [223] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li 2024. Language models are super mario: absorbing abilities from homologous models as a free lunch. Proceedings of the 41st International Conference on Machine Learning

Numbered as in the paper

Cite this row

Task Arith. & Model Merging — six-dimensional profile
D1: parameter-composition
D2: capability-extension, computational-efficiency
D3: parameter-only
D4: ad-hoc-permanent, scheduled-permanent
D5: fused-composition
D6: fm, llm, mllm
Source: arXiv:2608.06246
Compare with another technique Find in the explorer