Paper Detail
Pengpeng Qiao, Yang Cao, Lingling Zhang, Guo Cheng, Junwei Chen, Manjiang Yu, Wei Yang Bryan Lim, Masatoshi Yoshikawa
Federated fine-tuning of large language models (LLMs) under heterogeneous client capacities is increasingly important, and low-rank adaptation (LoRA) with heterogeneous ranks makes it practical. Existing server-side aggregators compose local matrix updates $ΔW_i$ from client-specific LoRA factors $(A_i,B_i)$ before aggregating them. Although these matrix updates lie in the same weight space $\mathbb{R}^{m\times n}$ and can be averaged directly, the LoRA factors that generate them may expressed client-specific local basis. In the typical compose--aggregate--factorize--dispatch loop, this basis inconsistency can make the rank-$r_i$ reference prefix dispatched to lower-rank clients unstable after server-side refactorization. In this work, we propose \textbf{FedAbA}, a basis-consistent federated LoRA aggregation framework for heterogeneous ranks. FedAbA first extracts each client's local basis, aligns it to the corresponding prefix of the global reference basis, and constructs a basis-consistent reconstruction of each local matrix update before aggregation. It then aggregates these reconstructions with weights that combine client data size and consistency score, followed by exact coefficient-space SVD refactorization and gauge fixing before rank-prefix dispatch. Our theoretical analysis clarifies the role of basis consistency and supports the proposed consistency-aware aggregation mechanism. Extensive experiments show that FedAbA outperforms representative federated LoRA baselines, with the largest gains under the strongest rank heterogeneity.
No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.
No ranking explanation is available yet.
No tags.
@article{qiao2026align,
title = {Align Before Aggregation: Basis-Consistent Federated LoRA under Heterogeneous Ranks},
author = {Pengpeng Qiao and Yang Cao and Lingling Zhang and Guo Cheng and Junwei Chen and Manjiang Yu and Wei Yang Bryan Lim and Masatoshi Yoshikawa},
year = {2026},
abstract = {Federated fine-tuning of large language models (LLMs) under heterogeneous client capacities is increasingly important, and low-rank adaptation (LoRA) with heterogeneous ranks makes it practical. Existing server-side aggregators compose local matrix updates \$ΔW\_i\$ from client-specific LoRA factors \$(A\_i,B\_i)\$ before aggregating them. Although these matrix updates lie in the same weight space \$\textbackslash{}mathbb\{R\}\textasciicircum{}\{m\textbackslash{}times n\}\$ and can be averaged directly, the LoRA factors that generate them may expressed c},
url = {https://arxiv.org/abs/2611.07598},
keywords = {cs.AI},
eprint = {2611.07598},
archiveprefix = {arXiv},
}
{}