Paper Detail
Zailong Tian, Yanzhe Chen, Zhuoheng Han, Houfeng Wang, Lizi Liao
While Low-Rank Adaptation (LoRA) enables efficient task specialization, its learned updates can compromise capabilities beyond the target task. We identify \textbf{adaptation imbalance}: a few singular directions dominate the trained update, leaving its performance sensitive to how gains are allocated. We argue that \textbf{learning where to adapt does not ensure that adaptation gains are well balanced}. This motivates \textbf{LoRA-Norm}, a post-training normalization method that retains learned directions while rebalancing their gains. LoRA-Norm combines spectral rebalancing, a fixed nonlinear transformation of singular values, with nuclear-norm restoration, which preserves the original total spectral mass. It requires no calibration data or additional training and introduces no inference overhead. Across two backbones and three adaptation tasks, LoRA-Norm improves average specialization and capability retention, outperforming the evaluated post-hoc spectral pruning and gradient-guided editing configurations on both measures. Stronger functional equalization brings no consistent additional gains, revealing that balancing adapter gains and equalizing their responses are distinct objectives.
No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.
No ranking explanation is available yet.
No tags.
@article{tian2026learn,
title = {Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA},
author = {Zailong Tian and Yanzhe Chen and Zhuoheng Han and Houfeng Wang and Lizi Liao},
year = {2026},
abstract = {While Low-Rank Adaptation (LoRA) enables efficient task specialization, its learned updates can compromise capabilities beyond the target task. We identify \textbackslash{}textbf\{adaptation imbalance\}: a few singular directions dominate the trained update, leaving its performance sensitive to how gains are allocated. We argue that \textbackslash{}textbf\{learning where to adapt does not ensure that adaptation gains are well balanced\}. This motivates \textbackslash{}textbf\{LoRA-Norm\}, a post-training normalization method that retains learned},
url = {https://arxiv.org/abs/2610.02067},
keywords = {cs.LG},
eprint = {2610.02067},
archiveprefix = {arXiv},
}
{}