Paper Detail

FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization

Hang Zou, Chao Zhang, Yuzhi Yang, Yu Tian, Samson Lasaulce, Mérouane Debbah

Browse

Workflow Queues

arxiv Score 7.6

Published 2026-10-01 · First seen 2026-10-02

General AI

Open paper source

Abstract

Federated Learning (FL) enables privacy-preserving fine-tuning of Large Language Models (LLMs), yet the massive communication overhead remains a critical bottleneck. Furthermore, applying Low-Rank Adaptation (LoRA) in FL faces a fundamental "aggregation dilemma" between the accurate Sum-of-Products (SoP) and the communication-efficient Product-of-Sums (PoS) implementations. To tackle these challenges, we propose FedFit. First, to significantly reduce communication overhead, we introduce a disjoint shared vector-bank parameterization that reconstructs high-dimensional adapter matrices from two compact and disjoint global vector banks. Second, to address the aggregation dilemma, we devise an alternating optimization schedule. By cycling between decoupled single-bank updates (which allow for accurate aggregation) and joint updates corrected by a Residual Spectral Aggregation mechanism, we resolve the conflict between SoP and PoS. Additionally, we integrate blockwise quantization with client-side error feedback to further compress the transmitted vectors. Furthermore, we establish theoretical convergence guarantees for the proposed algorithm. Extensive experiments on Qwen2.5 models demonstrate that FedFit achieves perplexity performance comparable to standard federated LoRA methods, while providing compression ratios up to 100x higher.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

Similar Papers

Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment 2026-05-20 · Score 4.3 General AI Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs 2026-09-16 · Score 5.3 General AI Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapters 2026-04-03 · Score 8.3 General AI Robust and Interpretable Adaptation of Equivariant Materials Foundation Models via Sparsity-promoting Fine-tuning 2026-06-17 · Score 3.3 General AI Debiasing LLMs by Fine-tuning 2026-04-03 · Score 8.3 General AI

BibTeX

@article{zou2026fedfit,
  title = {FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization},
  author = {Hang Zou and Chao Zhang and Yuzhi Yang and Yu Tian and Samson Lasaulce and Mérouane Debbah},
  year = {2026},
  abstract = {Federated Learning (FL) enables privacy-preserving fine-tuning of Large Language Models (LLMs), yet the massive communication overhead remains a critical bottleneck. Furthermore, applying Low-Rank Adaptation (LoRA) in FL faces a fundamental "aggregation dilemma" between the accurate Sum-of-Products (SoP) and the communication-efficient Product-of-Sums (PoS) implementations. To tackle these challenges, we propose FedFit. First, to significantly reduce communication overhead, we introduce a disjoi},
  url = {https://arxiv.org/abs/2610.01537},
  keywords = {cs.LG},
  eprint = {2610.01537},
  archiveprefix = {arXiv},
}

Metadata

{}