Paper Detail

Copy-Paste Language Models

Nathan Godey, Yoav Artzi

huggingface Score 6.0

Published 2026-10-13 · First seen 2026-10-17

General AI

Abstract

Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{godey2026copy,
  title = {Copy-Paste Language Models},
  author = {Nathan Godey and Yoav Artzi},
  year = {2026},
  abstract = {Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models},
  url = {https://huggingface.co/papers/2610.19247},
  keywords = {Transformers, LLMs, Copy-Paste Language Models, language modeling head, long-context, retrieval, gradient bottleneck, huggingface daily},
  eprint = {2610.19247},
  archiveprefix = {arXiv},
}

Metadata

{}