Paper Detail

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

Mohan Shi, Zilai Wang, Natarajan Balaji Shankar, Kaiyuan Zhang, Eray Eren, Abeer Alwan

Browse

Workflow Queues

arxiv Score 6.8

Published 2026-09-16 · First seen 2026-09-17

General AI

Open paper source

Abstract

Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-Rank Adapters (LoRA), have shown strong Automatic Speech Recognition (ASR) performance on general-domain speech. However, adapting them to domain-shifted speech, such as child or dialectal speech, remains challenging under limited target-domain data. Given the dominant role of the LLM in Speech-LLMs, with cross-entropy loss applied only at the LLM output, the speech encoder may receive insufficient adaptation to new acoustic conditions. In this paper, we propose Encoder Awakening via Adapters (EAVA), a simple yet effective domain-adaptive fine-tuning method for Speech-LLM-based ASR. First, lightweight adapters are inserted into each encoder layer and trained exclusively, enabling target-domain acoustic knowledge to be incorporated into the encoder while preserving its pre-trained knowledge. Second, the full model is jointly fine-tuned on the target domain with LoRA applied to the LLM. Experiments on three domain-shifted ASR datasets, covering child and dialectal speech, show that EAVA consistently outperforms vanilla fine-tuning and other baselines, achieving new state-of-the-art performance.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

Similar Papers

Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapters 2026-04-03 · Score 8.3 General AI Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment 2026-05-20 · Score 4.3 General AI Debiasing LLMs by Fine-tuning 2026-04-03 · Score 8.3 General AI NaRA: Noise-Aware LoRA for Parameter-Efficient Fine-Tuning of Diffusion LLMs 2026-05-28 · Score 14.3 General AI ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs 2026-09-05 · Score 27.5 Research Track A · General AI

BibTeX

@article{shi2026encoder,
  title = {Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs},
  author = {Mohan Shi and Zilai Wang and Natarajan Balaji Shankar and Kaiyuan Zhang and Eray Eren and Abeer Alwan},
  year = {2026},
  abstract = {Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-Rank Adapters (LoRA), have shown strong Automatic Speech Recognition (ASR) performance on general-domain speech. However, adapting them to domain-shifted speech, such as child or dialectal speech, remains challenging under limited target-domain data. Given the dominant role of the LLM in Speech-LLMs, with cross-entropy loss applied only at the LLM ou},
  url = {https://arxiv.org/abs/2609.17981},
  keywords = {eess.AS, cs.CL},
  eprint = {2609.17981},
  archiveprefix = {arXiv},
}

Metadata

{}