Paper Detail

SkillFormer: Skill-Decomposed Adaptation for Audio Language Models

Lee Seung-woo, Bowen Qi, Kim Min-jun, Jang Won-young

arxiv Score 6.8

Published 2026-10-06 · First seen 2026-10-08

General AI

Abstract

Audio language models must handle dozens of distinct skills, from pitch comparison and speaker counting to musical tempo estimation and emotion recognition. Joint training on all skills at once causes interference: gains on one skill often come at the cost of another. We propose \textbf{SkillFormer}, which decomposes audio understanding into skill-specific low-rank adapters and composes them at inference time through a learned router. The router examines the question to decide which adapters to activate and how much weight each should carry, so that a pitch query engages different parameters than a genre classification query. An alternating training schedule updates each adapter on its own skill cluster before jointly calibrating the router, preventing the gradient conflicts that arise in standard multi-task optimization. SkillFormer adds fewer than 4\% of the base model's parameters and requires no changes to the audio encoder or language backbone. Evaluated on three architecturally distinct models across MMSU, MMAU-Pro, and MMAR, it raises the average accuracy by 2.5 to 4.1 points, with balanced gains across perception, reasoning, and semantic subcategories.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@article{seungwoo2026skillformer,
  title = {SkillFormer: Skill-Decomposed Adaptation for Audio Language Models},
  author = {Lee Seung-woo and Bowen Qi and Kim Min-jun and Jang Won-young},
  year = {2026},
  abstract = {Audio language models must handle dozens of distinct skills, from pitch comparison and speaker counting to musical tempo estimation and emotion recognition. Joint training on all skills at once causes interference: gains on one skill often come at the cost of another. We propose \textbackslash{}textbf\{SkillFormer\}, which decomposes audio understanding into skill-specific low-rank adapters and composes them at inference time through a learned router. The router examines the question to decide which adapters to },
  url = {https://arxiv.org/abs/2610.07533},
  keywords = {cs.SD, cs.LG},
  eprint = {2610.07533},
  archiveprefix = {arXiv},
}

Metadata

{}