Paper Detail

Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout

Tushaar Gangavarapu, Gil Tabak, Clemens JS Schaefer

huggingface Score 10.4

Published 2026-11-05 · First seen 2026-11-07

General AI

Abstract

Linear attention compresses context into a fixed-size key-value state for linear-time training and constant-memory inference. Gated delta-rule models decay this state and erase its prediction for an incoming key before writing a new value; Gated DeltaNet-2 (GDN-2) generalizes this with channel-wise decay and decoupled, full-rank erase and write gates. However, we show that state updates are far less sensitive to erasure than to writes: key activation and unit normalization concentrate key energy onto a sparse subset of channels, whereas value writes scale all channels, so the erase gate can be parameterized at lower rank. Furthermore, slow-decaying state rows accumulate much larger norms than fast rows, dominating unweighted readout and forcing stronger write-time erasure to suppress interference. Guided by this analysis, we introduce Muir, which fuses all gates into a sensitivity-allocated low-rank projection, re-weights state rows with a matched-filter readout gate, and reinvests the saved parameters in a wider feed-forward network. Because standard chunkwise algorithms become numerically unstable at the large chunk sizes needed to saturate wide matrix units, we design a parallel update for stable, efficient training. Across 166M-1.34B parameters, Muir lowers perplexity by 1.8-2.0% over GDN-2 and 2.5-3.6% over Transformer, raises commonsense accuracy by up to 1.9 points, and improves extractive recall over GDN-2. At 1.34B, Muir outperforms both baselines on multi-hop and multi-needle recall, extrapolates 2,560x from its 4K training length to 10M-token streams without perplexity drift, and maintains near-100% passkey retrieval to 1M tokens with static readout. Muir achieves these gains while matching GDN-2 throughput, reaching its 100B-token validation loss 1.2x faster, and training 2.7x faster than Transformer at 32K.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{gangavarapu2026muir,
  title = {Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout},
  author = {Tushaar Gangavarapu and Gil Tabak and Clemens JS Schaefer},
  year = {2026},
  abstract = {Linear attention compresses context into a fixed-size key-value state for linear-time training and constant-memory inference. Gated delta-rule models decay this state and erase its prediction for an incoming key before writing a new value; Gated DeltaNet-2 (GDN-2) generalizes this with channel-wise decay and decoupled, full-rank erase and write gates. However, we show that state updates are far less sensitive to erasure than to writes: key activation and unit normalization concentrate key energy},
  url = {https://huggingface.co/papers/2611.08376},
  keywords = {linear attention, fixed-size key-value state, linear-time training, constant-memory inference, gated delta-rule models, Gated DeltaNet-2, GDN-2, channel-wise decay, decoupled full-rank erase and write gates, state updates, erasure, writes, key activation, unit normalization, key energy, sparse subset of channels, value writes, erase gate, lower rank, slow-decaying state rows, unweighted readout, write-time erasure, interference, Muir, sensitivity-allocated low-rank projection, matched-filter readout gate, feed-forward network, chunkwise algorithms, numerical instability, large chunk sizes, wide matrix units, parallel update, perplexity, Transformer, commonsense accuracy, extractive recall, multi-hop recall, multi-needle recall, passkey retrieval, static readout, validation loss, huggingface daily},
  eprint = {2611.08376},
  archiveprefix = {arXiv},
}

Metadata

{}