Paper Detail

Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus

Zunhai Su, Bohan Sun, Xialie Zhuang, Shuibai Zhang, He Xiao, Jing Xiong, Hengyuan Zhang, Zhongzhu Zhou, Tiantian Zhang, Ngai Wong, Chuan-Wei Kuo

Browse

Workflow Queues

huggingface Score 6.4

Published 2026-08-12 · First seen 2026-08-14

General AI

Open paper source

Abstract

We present the first systematic study of Massive activations (MAs) in layer-interleaved HLA LLMs and uncover two architecture-aligned morphologies: MAs consistently spike immediately before full attention layers, forming pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP). As full attention becomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology of full attention LLMs. We establish the recurrence of this organization across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining of GDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removing GDN gates yields comparatively modest amplification. Mechanistically, our systematic-outlier analysis supports a shared lifecycle account governed by the timing of MA cancellation. PAS follows a localized write-sink-cancel process, while the extended persistence of ISP is consistent with delayed cancellation. At the full attention limit, this account recovers the stable MA morphology characteristic of full attention LLMs. Our code is available at https://github.com/StartluxLabs/Massive-Activations-HLA.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
later
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

Similar Papers

Continual Learning in Large Language Models: Methods, Challenges, and Opportunities 2026-03-13 · Score 22.5 Research Track A · General AI FriendBench: Benchmarking Dyadic Familiarity Inference in Humans and Multimodal Large Language Models 2026-07-31 · Score 14.3 General AI Hybrid Framework for Robotic Manipulation: Integrating Reinforcement Learning and Large Language Models 2026-03-31 · Score 18.3 General AI Using Large Language Models in Physics Education 2026-05-22 · Score 14.8 General AI Automated reproducibility assessments in the social and behavioral sciences using large language models 2026-06-11 · Score 6.3 General AI

BibTeX

@misc{su2026massive,
  title = {Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus},
  author = {Zunhai Su and Bohan Sun and Xialie Zhuang and Shuibai Zhang and He Xiao and Jing Xiong and Hengyuan Zhang and Zhongzhu Zhou and Tiantian Zhang and Ngai Wong and Chuan-Wei Kuo},
  year = {2026},
  abstract = {We present the first systematic study of Massive activations (MAs) in layer-interleaved HLA LLMs and uncover two architecture-aligned morphologies: MAs consistently spike immediately before full attention layers, forming pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP). As full attention becomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology of full },
  url = {https://huggingface.co/papers/2608.12149},
  keywords = {massive activations, layer-interleaved HLA LLMs, pre-attention spikes, inter-spike plateaus, linear attention, full attention, GDN, output gating, systematic-outlier analysis, code available, huggingface daily},
  eprint = {2608.12149},
  archiveprefix = {arXiv},
}

Metadata

{}