Paper Detail

Factored One-sided Adam-Moment for Practical and Scalable SOAP

Jaemyung Yu, Byeongho Heo, Sangdoo Yun, Dongyoon Han

huggingface Score 10.0

Published 2026-11-12 · First seen 2026-11-13

General AI

Abstract

Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored One-sided Adam-Moment (FOAM). The path to practical impact is twofold: first, FOAM employs input-only preconditioning, dropping the output-side Kronecker factor to reduce memory and remove one eigendecomposition. The Kronecker structure of the Hessian shows that this is exact for a diagonal output-side curvature and nearly lossless for a near-diagonal one, as we observe empirically. Second, FOAM replaces SOAP's dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. Under the K-FAC Fisher assumption with a diagonal output curvature, this estimator exactly recovers SOAP's dense second moment, and we measure only a 15-20% K-FAC violation, with negligible effect on the update. Our empirical results support the claim that FOAM matches strong SOAP and Muon baselines in validation loss across scales of large language model (LLM) pre-training, uses half of SOAP's optimizer state, runs at Muon-class wall-clock time, and retains robustness across a three-fold learning-rate range. The results hold from 124M to 760M parameters, demonstrating that SOAP's gains need not require SOAP's cost. Code: https://github.com/naver-ai/foam

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{yu2026factored,
  title = {Factored One-sided Adam-Moment for Practical and Scalable SOAP},
  author = {Jaemyung Yu and Byeongho Heo and Sangdoo Yun and Dongyoon Han},
  year = {2026},
  abstract = {Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored One-sided Adam-Moment (FOA},
  url = {https://huggingface.co/papers/2611.16905},
  keywords = {second-order optimizers, Shampoo, SOAP, AdamW, transformer training, optimizer-state memory, eigendecomposition, input-only preconditioning, output-side Kronecker factor, Kronecker structure of Hessian, diagonal output-side curvature, dense rotated second moment, factored Adam-moment estimator, row-by-column statistics, K-FAC Fisher assumption, dense second moment, K-FAC violation, Muon, LLM pre-training, Factored One-sided Adam-Moment (FOAM), code available, huggingface daily},
  eprint = {2611.16905},
  archiveprefix = {arXiv},
}

Metadata

{}