Efficiency and Composition

Long-Context Ext.

D1Arch.Param. Upd.
D2Capab. Ext.
D3Small LabeledUnlabeled
D4Ad-hoc Perm.Sched. Perm.
D5Part.
D6FMLLMMLLM

In brief — site editorial, not from the paper

Extends how much text a model can attend to at once, so longer documents fit in a single pass.

Definition

Long-Context Extension methods (e.g., PI [40], YaRN [156], and PEPE [91]) extend a language model's operational context window by adapting its positional encoding (e.g., through RoPE [189] scaling) and, when required, fine-tuning on long-sequence data, enabling the model to process inputs far exceeding its original context length. This operates via a composite D1 (Mechanism): it requires an Architectural Modification to alter the frequency scaling of the RoPE, typically followed by a brief Parametric Update on long documents. The goal is strictly Capability Extension, as it structural expands the model's capacity rather than teaching a new task. It targets the Partial scope and utilizes Small Labeled, Unlabeled long-sequence corpora. Because it permanently alters the positional embedding architecture and corresponding parameters, it produces a versioned checkpoint exhibiting Ad-hoc Permanent, Scheduled Permanent persistence. Unlike techniques that require generation mechanics, context scaling can be applied to base GenFM, establishing its D6 (Model Type) footprint at FM, LLM, MLLM.

Verbatim from the paper — Efficiency and Composition

Nearest profiles

MoE (0.17), Cross-Modal Alignment (0.31), Prompt Learning (0.33), DA (0.38), FT (partial) (0.38)

Computed from the taxonomy data (Gower distance over all six dimensions)

References

  1. [40] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian 2023. Extending Context Window of Large Language Models via Positional Interpolation. arXiv
  2. [91] Jikun Hu et al. 2025. PEPE: Long-context Extension for Large Language Models via Periodic Extrapolation Positional Encodings. Findings of the Association for Computational Linguistics: EMNLP 2025
  3. [156] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole 2023. YaRN: Efficient Context Window Extension of Large Language Models.
  4. [189] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu 2024. RoFormer: Enhanced transformer with Rotary Position Embedding. Neurocomputing

Numbered as in the paper

Cite this row

Long-Context Ext. — six-dimensional profile
D1: architectural-modification, parametric-update
D2: capability-extension
D3: small-labeled, unlabeled
D4: ad-hoc-permanent, scheduled-permanent
D5: partial
D6: fm, llm, mllm
Source: arXiv:2608.06246
Compare with another technique Find in the explorer