Paper Detail

Dive Into the Implicit Biases of Low-rank Vision-language Alignment

Mingjia Shi, Shuo Wang, Xiaobo Wang, Sifan Zhou, Kai Wang, Tianyu Fu, Chenxu Zhao, Anyang Su, Ping Jiang, Minghui Wu

Browse

Workflow Queues

arxiv Score 8.3

Published 2026-07-09 · First seen 2026-07-10

General AI

Open paper source

Abstract

Vision-language alignment, the stage that bridges pretrained vision encoders and large language models, is widely treated as a form of pretraining requiring full-parameter updates. We challenge this view and investigate what happens when low-rank adaptation is applied to the LLM during this stage instead. We find that low-rank alignment not only reduces computational costs but also outperforms full-parameter alignment on most benchmarks. To understand this phenomenon, we systematically characterize the implicit biases introduced by low-rank adaptation during alignment. Empirically, we find that low-rank alignment shifts model behavior from hallucinatory to conservative and preserves per-token linear separability of visual features that full-parameter alignment disrupts, a phenomenon we term LS-curse. Geometrically, low rank aligned models exhibit more homogeneous and structurally stable visual representations, maintaining modality-specific knowledge rather than prematurely fusing entity-level semantics. Theoretically, we establish two theorems showing that low-rank alignment induces preferences for parameter subspaces with flat gradients and feature subspaces robust to perturbations, providing a principled explanation for the observed structure-preserving behavior. Extensive experiments cover ablation over 100 alignment configurations, three families of low-rank operators, and various rank, encoder, and other settings.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

Similar Papers

Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Models 2026-06-20 · Score 7.8 General AI TLoRA: Task-aware Low Rank Adaptation of Large Language Models 2026-04-20 · Score 9.3 General AI LoCA: Spatially-Aware Low-Rank Convolutional Adaptation of Vision Foundation Models 2026-07-08 · Score 23.5 Research Track A · General AI On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR 2026-05-07 · Score 10.5 Research Track A · General AI TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models 2026-04-07 · Score 5.3 General AI

BibTeX

@article{shi2026dive,
  title = {Dive Into the Implicit Biases of Low-rank Vision-language Alignment},
  author = {Mingjia Shi and Shuo Wang and Xiaobo Wang and Sifan Zhou and Kai Wang and Tianyu Fu and Chenxu Zhao and Anyang Su and Ping Jiang and Minghui Wu},
  year = {2026},
  abstract = {Vision-language alignment, the stage that bridges pretrained vision encoders and large language models, is widely treated as a form of pretraining requiring full-parameter updates. We challenge this view and investigate what happens when low-rank adaptation is applied to the LLM during this stage instead. We find that low-rank alignment not only reduces computational costs but also outperforms full-parameter alignment on most benchmarks. To understand this phenomenon, we systematically character},
  url = {https://arxiv.org/abs/2607.08194},
  keywords = {cs.CV},
  eprint = {2607.08194},
  archiveprefix = {arXiv},
}

Metadata

{}