Paper Detail
Yifan Zhang, Steve Ta, Jasper Zhang, Jichen Feng, Shuzhen Li, Yongxin Zhang, Yifeng Liu, Huizhuo Yuan, Mengdi Wang, Quanquan Gu, Andrew Chi-Chih Yao
Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step $t$ is the prefix-aligned pair $(\mathbf{x}_t,\mathbf{y}_t)=(φ(\mathbf{k}_{t-1}),\mathbf{v}_t)$. The common same-step association $(φ(\mathbf{k}_t),\mathbf{v}_t)$ remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.
No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.
No ranking explanation is available yet.
No tags.
@article{zhang2026fast,
title = {Fast Weight Attention for Continual Learning},
author = {Yifan Zhang and Steve Ta and Jasper Zhang and Jichen Feng and Shuzhen Li and Yongxin Zhang and Yifeng Liu and Huizhuo Yuan and Mengdi Wang and Quanquan Gu and Andrew Chi-Chih Yao},
year = {2026},
abstract = {Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step \$t\$ is the prefix-aligned pair \$(\textbackslash{}mathbf\{x\}\_t,\textbackslash{}mathbf\{y\}\_t)=(φ(\textbackslash{}mathbf\{k\}\_\{t-1\}),\textbackslash{}mathbf\{v\}\_t)\$. The common same-step association \$(φ(\textbackslash{}mathbf\{k\}\_t),\textbackslash{}m},
url = {https://arxiv.org/abs/2608.27763},
keywords = {cs.LG, cs.CL, stat.ML},
eprint = {2608.27763},
archiveprefix = {arXiv},
}
{}