Paper Detail

Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity

Lei Wang, Jieming Bian, Letian Zhang, Jie Xu

Browse

Workflow Queues

arxiv Score 9.3

Published 2026-09-01 · First seen 2026-09-03

General AI

Open paper source

Abstract

Large Language Models (LLMs) have achieved remarkable success across diverse domains, but their adaptation to privacy-sensitive, distributed datasets remains a challenge. While Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) provides a resource-efficient paradigm for collaborative fine-tuning, practical deployments are hindered by the dual challenges of resource heterogeneity and data heterogeneity. Existing rank-heterogeneous methods primarily focus on bridging dimension mismatches for aggregation but typically provide a unified global model for all clients sharing the same rank, failing to capture client-specific features in non-IID scenarios. In this paper, we propose FedRoRA (Federated Rank-wise Personalized LoRA), a novel framework that enables fine-grained personalization within rank-heterogeneous federations. FedRoRA decouples adaptation into shared global directions and personalized rank-wise magnitudes governed by learnable diagonal scales. On the server side, it extracts a global subspace via singular value decomposition (SVD) and redistributes client-specific initializations through a personalized projection and top-$k$ selection mechanism. Extensive experiments on NLU and NLG benchmarks demonstrate that FedRoRA consistently outperforms state-of-the-art methods.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

Similar Papers

Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment 2026-05-20 · Score 4.3 General AI CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning 2026-08-22 · Score 5.3 General AI Parameter-Efficient Fine-Tuning with Learnable Rank 2026-06-03 · Score 7.3 General AI LoRA vs. Full Fine-Tuning: A Theoretical Perspective 2026-05-18 · Score 3.3 General AI On the Efficiency of LoRA Fine-Tuning for Vision-Language-Action Models in Industrial Robotic Manipulation 2026-07-11 · Score 6.3 General AI

BibTeX

@article{wang2026breaking,
  title = {Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity},
  author = {Lei Wang and Jieming Bian and Letian Zhang and Jie Xu},
  year = {2026},
  abstract = {Large Language Models (LLMs) have achieved remarkable success across diverse domains, but their adaptation to privacy-sensitive, distributed datasets remains a challenge. While Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) provides a resource-efficient paradigm for collaborative fine-tuning, practical deployments are hindered by the dual challenges of resource heterogeneity and data heterogeneity. Existing rank-heterogeneous methods primarily focus on bridging dimension mismat},
  url = {https://arxiv.org/abs/2609.00632},
  keywords = {cs.LG, cs.AI},
  eprint = {2609.00632},
  archiveprefix = {arXiv},
}

Metadata

{}