Paper Detail

Where Do Vision Language Models Fail at 3D Reasoning?

Khai Le-Duc, Nguyen Kim Hai Bui, An Ngo, Nam D. Hoang, Yueqi Song, Jiarui Liu, Thai Quoc Hoang, Fang Wu, Chris Ngo, Terry Jingchen Zhang, Mona T. Diab, Ran Xu, Silvio Savarese, Xiao-Ming Wu

huggingface Score 11.0

Published 2026-10-29 · First seen 2026-11-03

General AI

Abstract

Vision-language models can get individual judgments right while struggling to recover and consistently use the exact properties of a 3D object. To characterize this gap, we introduce BedBench, a controlled benchmark of thirteen procedurally generated tasks spanning individual judgments and composite answers. Across twelve open-weight and four frontier models, per-color counting exposes a pronounced weakness: humans achieve 93.3% accuracy, compared with 7.2% for the best model. Targeted fine-tuning raises one model's accuracy from 3.0% to 68.6%, but reveals a further difficulty: its answers about the most common colors remain inconsistent with its own per-color counts. Supplying those counts as text improves its judgments, showing that recovering a property does not mean using it consistently. Reasoning, scaling, and added visual cues each improve individual judgments, but none makes exact answers reliable. To locate the difficulty, we probe Qwen3.5-27B: exact per-color counts remain recoverable at the language model's input, yet the model's answers rarely recover them (65.0% for the probe vs. 3.0% for the model). Image-removal controls further show that some apparent successes require no visual evidence. These findings motivate evaluating 3D reasoning by exact correctness, consistent use, and image dependence, rather than accuracy alone. Code and dataset are publicly available: https://github.com/leduckhai/BedBench

Workflow Status

Review status
pending
Role
unreviewed
Read priority
now
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{leduc2026where,
  title = {Where Do Vision Language Models Fail at 3D Reasoning?},
  author = {Khai Le-Duc and Nguyen Kim Hai Bui and An Ngo and Nam D. Hoang and Yueqi Song and Jiarui Liu and Thai Quoc Hoang and Fang Wu and Chris Ngo and Terry Jingchen Zhang and Mona T. Diab and Ran Xu and Silvio Savarese and Xiao-Ming Wu},
  year = {2026},
  abstract = {Vision-language models can get individual judgments right while struggling to recover and consistently use the exact properties of a 3D object. To characterize this gap, we introduce BedBench, a controlled benchmark of thirteen procedurally generated tasks spanning individual judgments and composite answers. Across twelve open-weight and four frontier models, per-color counting exposes a pronounced weakness: humans achieve 93.3\% accuracy, compared with 7.2\% for the best model. Targeted fine-tuni},
  url = {https://huggingface.co/papers/2610.39639},
  keywords = {vision-language models, 3D reasoning, BedBench, per-color counting, targeted fine-tuning, reasoning, scaling, visual cues, Qwen3.5-27B, image-removal controls, code available, huggingface daily},
  eprint = {2610.39639},
  archiveprefix = {arXiv},
}

Metadata

{}