Paper Detail

Jag: Direct Box Prediction for Efficient Visual Grounding

Xiuyuan Zhu, Ke Lu, Hao Wu, Zijin Du, Dongming Zhang, Jian Xue

huggingface Score 11.0

Published 2026-10-09 · First seen 2026-10-15

General AI

Abstract

Visual grounding combines image--language understanding with geometric prediction, yet autoregressive multimodal models often express a box as a coordinate sequence. Sequential decoding adds computation to the multimodal processing needed for localization. Adapting these representations to box geometry offers a route to direct prediction. We present Jag, which adapts a compact pretrained multimodal model for direct continuous box prediction. A lightweight regression head reads the last valid input-token state, conditioned on the image and referring expression, and predicts all four box coordinates together. Geometric supervision jointly adapts the language backbone, visual merger, and regression head, allowing localization losses to shape both the multimodal representations and the coordinate mapping. Inference retains image--language processing and returns the box in one forward pass, without coordinate generation or text parsing. Across benchmarks, Jag consistently improves grounding accuracy over its base model across the observed target sizes and expression groups. Component ablation show that jointly adapting the representations and head substantially improves accuracy over training the head alone. Against existing specialized grounding models, it achieves competitive accuracy with lower single-request latency and GPU memory use. These results show how supervision tailored to the geometric answer can specialize a compact pretrained multimodal model for accurate grounding and efficient direct prediction. Project page: https://xyzzzh.github.io/Jag/

Workflow Status

Review status
pending
Role
unreviewed
Read priority
now
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{zhu2026jag,
  title = {Jag: Direct Box Prediction for Efficient Visual Grounding},
  author = {Xiuyuan Zhu and Ke Lu and Hao Wu and Zijin Du and Dongming Zhang and Jian Xue},
  year = {2026},
  abstract = {Visual grounding combines image--language understanding with geometric prediction, yet autoregressive multimodal models often express a box as a coordinate sequence. Sequential decoding adds computation to the multimodal processing needed for localization. Adapting these representations to box geometry offers a route to direct prediction. We present Jag, which adapts a compact pretrained multimodal model for direct continuous box prediction. A lightweight regression head reads the last valid inp},
  url = {https://huggingface.co/papers/2610.13239},
  keywords = {visual grounding, autoregressive multimodal models, continuous box prediction, regression head, geometric supervision, multimodal representations, direct prediction, code available, huggingface daily},
  eprint = {2610.13239},
  archiveprefix = {arXiv},
}

Metadata

{}