Paper Detail

MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

Qingyu Yang, Haonan He, Minglei Li, Jingqi Ye, Tao Chen, Lei Bai, Peng Ye

Browse

Workflow Queues

arxiv Score 25.0

Published 2026-07-24 · First seen 2026-07-27

Research Track A · General AI

Open paper source

Abstract

Mixture-of-Experts (MoE) architectures have been widely adopted in large language models, yet parameter-efficient fine-tuning (PEFT) for MoE models remains underexplored. Existing PEFT methods for MoE either ignore router priors with uniform adapters, reducing efficiency and risking forgetting, or rely on static expert selection, limiting per-token capacity and cross-expert feature learning. In this paper, we make the first attempt to fine-tune MoE models with MoE-style low-rank adaptation: our method, entitled MoE$^2$-LoRA, deeply couples the pretrained expert specialization with task-specific adaptivity via a dual-channel Routing-Conditioned Projection (RCP) module, which reuses base router activations to inform LoRA routing. We further introduce a single global LoRA expert pool shared across all layers, enabling model-wide adaptation with emergent layer-wise affinities and balanced expert utilization. MoE$^2$-LoRA simultaneously benefits from the advantages of prior reuse, dynamic adapter routing, and model-wide knowledge sharing. Evaluated on multiple MoE backbones with varying scales and expert granularities, MoE$^2$-LoRA consistently achieves state-of-the-art downstream accuracy while retaining stronger general capabilities.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
now
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

Similar Papers

Janus-LoRA: A Balanced Low-Rank Adaptation for Continual Learning 2026-05-27 · Score 21.5 Research Track A · General AI LoCA: Spatially-Aware Low-Rank Convolutional Adaptation of Vision Foundation Models 2026-07-08 · Score 23.0 Research Track A · General AI Null-Space Constrained Low-Rank Adaptation for Response-Specified Large Language Model Unlearning 2026-06-09 · Score 15.0 Research Track A · General AI On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR 2026-05-07 · Score 10.5 Research Track A · General AI Pruned Adaptation Modules: A Simple yet Strong Baseline for Continual Foundation Models 2026-03-22 · Score 14.0 Research Track A · General AI

BibTeX

@article{yang2026moe,
  title = {MoE\$\textasciicircum{}2\$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation},
  author = {Qingyu Yang and Haonan He and Minglei Li and Jingqi Ye and Tao Chen and Lei Bai and Peng Ye},
  year = {2026},
  abstract = {Mixture-of-Experts (MoE) architectures have been widely adopted in large language models, yet parameter-efficient fine-tuning (PEFT) for MoE models remains underexplored. Existing PEFT methods for MoE either ignore router priors with uniform adapters, reducing efficiency and risking forgetting, or rely on static expert selection, limiting per-token capacity and cross-expert feature learning. In this paper, we make the first attempt to fine-tune MoE models with MoE-style low-rank adaptation: our },
  url = {https://arxiv.org/abs/2607.21978},
  keywords = {cs.CL},
  eprint = {2607.21978},
  archiveprefix = {arXiv},
}

Metadata

{}