Paper Detail

Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization

Zebang Xie, Chuanyang Zheng, Yik-Chung Wu, Yihang Gao

Browse

Workflow Queues

arxiv Score 6.2

Published 2026-09-24 · First seen 2026-09-26

General AI

Open paper source

Abstract

Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective. In this work, we propose $\ell_p$-LoRA, a principled rank-allocation method based on $\ell_p$ regularization with $0<p<1$, which is a classical sparsity-inducing technique in signal processing and statistics. Specifically, we regularize the energy of each rank-one LoRA component, encouraging redundant components to vanish while preserving important ones. We derive the corresponding proximal subproblem and reduce the matrix optimization to a two-dimensional problem, leading to an implicit thresholding criterion for identifying redundant components. Experiments on natural language understanding and question-answering tasks demonstrate that the proposed method achieves competitive performance with existing LoRA baselines.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
later
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

Similar Papers

Low-Rank Adaptation Redux for Large Models 2026-04-23 · Score 3.3 General AI Statistical Inference for Rank Allocation in Low-Rank Adaptation 2026-07-22 · Score 6.3 General AI TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models 2026-04-07 · Score 5.3 General AI LoRAScan: Detecting Backdoor Prompts in Low-Rank Adapters for Large Language Models via Down-Projection Activation Spikes 2026-08-07 · Score 10.8 General AI TLoRA: Task-aware Low Rank Adaptation of Large Language Models 2026-04-20 · Score 9.3 General AI

BibTeX

@article{xie2026automatic,
  title = {Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization},
  author = {Zebang Xie and Chuanyang Zheng and Yik-Chung Wu and Yihang Gao},
  year = {2026},
  abstract = {Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective. In this work, we propose \$\textbackslash{}ell\_p\$-LoRA, a principled rank-allocation method based },
  url = {https://arxiv.org/abs/2609.28998},
  keywords = {cs.LG},
  eprint = {2609.28998},
  archiveprefix = {arXiv},
}

Metadata

{}