Paper Detail
Ellis Brown, Chris Hoang, Shengyi Qian, Shang-Wen Li, Mike Lewis, Marjan Ghazvininejad, Luke Zettlemoyer, Lucas Beyer, Mengye Ren, Jakob Verbeek, Saining Xie, Hu Xu
How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity (i.e., before the multimodal backbone). Because published models also differ in data, architecture, and compute, comparing them cannot isolate the effect of each choice. We disentangle these choices via a controlled framework where we pretrain all models from scratch on common data and compare them on visual question answering and language benchmarks at matched lifetime compute. We find that the visual prior is not necessary; an encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage. What matters instead is dedicated vision capacity: an encoder-free model trails on visual understanding even when its backbone is enlarged by the encoder's parameter count. Yet that capacity need not precede multimodal fusion, as vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder and can lower time to first token. These results also hold across backbone sizes, when encoder pretraining is treated as free, and when starting from a pretrained language model. Overall, our findings favor dedicated vision capacity trained jointly with the backbone and placed early within it.
No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.
No ranking explanation is available yet.
No tags.
@article{brown2026unbundling,
title = {Unbundling Multimodal Vision Encoding},
author = {Ellis Brown and Chris Hoang and Shengyi Qian and Shang-Wen Li and Mike Lewis and Marjan Ghazvininejad and Luke Zettlemoyer and Lucas Beyer and Mengye Ren and Jakob Verbeek and Saining Xie and Hu Xu},
year = {2026},
abstract = {How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity (i.e., before the multimodal backbone). Because published models also differ in data, architecture, and c},
url = {https://arxiv.org/abs/2611.17334},
keywords = {cs.CV, cs.LG, multimodal language models, vision encoder, pretrained vision encoder, visual prior, upstream pretraining, dedicated vision capacity, pre-fusion placement, multimodal backbone, visual question answering, language benchmarks, matched lifetime compute, jointly trained encoder, encoder-free model, multimodal fusion, vision-only modules, in-fusion, early backbone layers, pre-fusion encoder, time to first token, backbone sizes, encoder pretraining, pretrained language model, huggingface daily},
eprint = {2611.17334},
archiveprefix = {arXiv},
}
{}