Paper Detail

Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?

Aneerban Saha, Vijay Kumar, Sameer Madhav D., Priyam Srivastava, Dhruv Bansal

huggingface Score 12.0

Published 2026-10-12 · First seen 2026-10-15

General AI

Abstract

Knowledge-based Visual Question Answering (VQA) requires reasoning that extends beyond visible image content, demanding external commonsense and factual knowledge. While recent work leverages Large Language Models (LLMs) as knowledge sources, the question of how deeply such guidance should be integrated into a vision language architecture remains underexplored. In this work, we generate structured reasoning guidance, visual focus concepts, distractor elements and external knowledge using Llama~3.3~70B and systematically compare two integration strategies within a unified frozen ViT + FLAN-T5 framework on the OK-VQA benchmark: decision level late fusion and attention-matrix injection. Contrary to the prevailing assumption that deeper integration is superior, we find that shallow late fusion achieves the strongest results (39.4% exact match, +10.0 over the no guidance baseline), outperforming deep attention modification (31.8%) under data scarce fine-tuning. We further contribute an LLM based augmentation pipeline that expands training data more than tenfold via question paraphrasing. Our findings indicate that integration depth must be matched to the data regime.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
now
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{saha2026rethinking,
  title = {Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?},
  author = {Aneerban Saha and Vijay Kumar and Sameer Madhav D. and Priyam Srivastava and Dhruv Bansal},
  year = {2026},
  abstract = {Knowledge-based Visual Question Answering (VQA) requires reasoning that extends beyond visible image content, demanding external commonsense and factual knowledge. While recent work leverages Large Language Models (LLMs) as knowledge sources, the question of how deeply such guidance should be integrated into a vision language architecture remains underexplored. In this work, we generate structured reasoning guidance, visual focus concepts, distractor elements and external knowledge using Llama\textasciitilde{}3},
  url = {https://huggingface.co/papers/2610.17079},
  keywords = {Visual Question Answering, Large Language Models, ViT, FLAN-T5, late fusion, attention-matrix injection, cross-attention, OK-VQA, code available, huggingface daily},
  eprint = {2610.17079},
  archiveprefix = {arXiv},
}

Metadata

{}