Paper Detail

Align Before Aggregation: Basis-Consistent Federated LoRA under Heterogeneous Ranks

Pengpeng Qiao, Yang Cao, Lingling Zhang, Guo Cheng, Junwei Chen, Manjiang Yu, Wei Yang Bryan Lim, Masatoshi Yoshikawa

arxiv Score 6.2

Published 2026-11-05 · First seen 2026-11-07

General AI

Abstract

Federated fine-tuning of large language models (LLMs) under heterogeneous client capacities is increasingly important, and low-rank adaptation (LoRA) with heterogeneous ranks makes it practical. Existing server-side aggregators compose local matrix updates $ΔW_i$ from client-specific LoRA factors $(A_i,B_i)$ before aggregating them. Although these matrix updates lie in the same weight space $\mathbb{R}^{m\times n}$ and can be averaged directly, the LoRA factors that generate them may expressed client-specific local basis. In the typical compose--aggregate--factorize--dispatch loop, this basis inconsistency can make the rank-$r_i$ reference prefix dispatched to lower-rank clients unstable after server-side refactorization. In this work, we propose \textbf{FedAbA}, a basis-consistent federated LoRA aggregation framework for heterogeneous ranks. FedAbA first extracts each client's local basis, aligns it to the corresponding prefix of the global reference basis, and constructs a basis-consistent reconstruction of each local matrix update before aggregation. It then aggregates these reconstructions with weights that combine client data size and consistency score, followed by exact coefficient-space SVD refactorization and gauge fixing before rank-prefix dispatch. Our theoretical analysis clarifies the role of basis consistency and supports the proposed consistency-aware aggregation mechanism. Extensive experiments show that FedAbA outperforms representative federated LoRA baselines, with the largest gains under the strongest rank heterogeneity.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
later
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@article{qiao2026align,
  title = {Align Before Aggregation: Basis-Consistent Federated LoRA under Heterogeneous Ranks},
  author = {Pengpeng Qiao and Yang Cao and Lingling Zhang and Guo Cheng and Junwei Chen and Manjiang Yu and Wei Yang Bryan Lim and Masatoshi Yoshikawa},
  year = {2026},
  abstract = {Federated fine-tuning of large language models (LLMs) under heterogeneous client capacities is increasingly important, and low-rank adaptation (LoRA) with heterogeneous ranks makes it practical. Existing server-side aggregators compose local matrix updates \$ΔW\_i\$ from client-specific LoRA factors \$(A\_i,B\_i)\$ before aggregating them. Although these matrix updates lie in the same weight space \$\textbackslash{}mathbb\{R\}\textasciicircum{}\{m\textbackslash{}times n\}\$ and can be averaged directly, the LoRA factors that generate them may expressed c},
  url = {https://arxiv.org/abs/2611.07598},
  keywords = {cs.AI},
  eprint = {2611.07598},
  archiveprefix = {arXiv},
}

Metadata

{}