Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning
This paper presents a new way for models to think and reason using both images and text. It shows how to better decide when to look at images while answering questions.
This video presentation explains the key concepts from the paper in plain language.
Content & Liability Disclaimer
This article and its accompanying video are automated summaries derived from the original research paper by Unknown authors. The original research was conducted solely by the paper's authors; PDFdigest did not conduct any of the research and makes no claims of ownership over the underlying scientific work.
The video narration is generated by artificial intelligence and references the paper's authors for attribution. The video is not narrated by any of the paper's authors. This content may contain inaccuracies, omissions, or misinterpretations of the original research. First-person language (e.g., "we found", "our results") reflects the original authors' voice, not PDFdigest's. Always read the original paper for accurate, verified information before making any decisions based on this content.
This content is provided "as is" without any warranties, express or implied. Simulated systems OÜ, its officers, directors, employees, and agents shall not be liable for any direct, indirect, incidental, special, consequential, or punitive damages arising from your use of, reliance on, or access to this content, including but not limited to errors, omissions, or misinterpretations of the original research. This disclaimer applies to the fullest extent permitted by applicable law.
- 1 Current models struggle with how to use visual information effectively.
- 2 The proposed framework improves accuracy by dynamically deciding when to gather visual evidence.
- 3 Experiments show significant performance improvements over existing methods.
Introduction
The introduction discusses the advancements in Vision-Language Models (VLMs) and categorizes existing multimodal reasoning approaches into two paradigms: text-centric pre-reasoning and unified multimodal representation.
Text-Centric Pre-Reasoning Paradigm
This section explains the first paradigm where visual inputs are converted into textual representations before reasoning, highlighting its limitations in capturing fine-grained visual details.
Unified Multimodal Representation Paradigm
This section describes the second paradigm that performs end-to-end reasoning by fusing visual and textual features, discussing its advantages and the issue of hallucinations due to linguistic dominance.
How PDFdigest Helps You Understand Research
Instant Paper Analysis
Get structured summaries and key findings from dense PDFs in seconds.
Visual Explanations
Turn complex methods, figures, and results into clearer visual breakdowns.
AI-Powered Q&A
Ask focused questions and get answers grounded in the paper.
Preliminaries
The preliminaries focus on the single-stream paradigm used in contemporary VLMs, detailing the process of encoding visual and textual information into a joint multimodal representation.
Experiments
Experiments demonstrate the effectiveness of the proposed framework across multiple multimodal reasoning benchmarks, showing consistent improvements in accuracy.
Figures Explained
The paper’s visual material highlights the workflow and the main system components.
- Figure 1 .: Figure 1. Illustration of two dominant multimodal reasoning paradigms and our framework.
- Figure 2 .: Figure 2. Layer-wise Mean Attention Scores (Text vs. Image). We report the average pre-softmax attention scores of the first generated token across all 35 Transformer layers on a ScienceQA subset. Text tokens consistently receive higher attention than visual tokens, indicating a systematic attention bias toward text.
- Figure 3 .: Figure3. Overview of the CSMR architecture and its reasoning workflow. The left panel illustrates the overall structure of the CSMR, which consists of a CRC and a PVP. Given an input image and a question, the CRC maintains the current reasoning state and generates targeted visual queries to invoke the PVP when necessary. The PVP independently analyzes the original image and returns textualized visual evidence that answers the issued query. This evidence is then integrated into the CRC’s reasoning state to support subsequent reasoning. The right panel presents a concrete example of reasoning. The CRC progressively generates visual queries based on the current reasoning state. Once the obtained textualized visual evidence is deemed sufficient, the CRC directly produces the final answer.
- Figure 4 .: Figure 4. Comparison of hallucination rates between DDCoT and CSMR on M3CoT. Hallucinations are identified by GPT-5 based on inconsistencies between generated dialogues and image content. CSMR exhibits a lower hallucination rate than DDCoT.
- Figure 5 .: Figure 5. Comparison of reasoning paths between DDCoT and CSMR. CSMR constructs a progressive, evidence-conditioned reasoning trajectory by dynamically generating sub-questions, while DDCoT relies on static and parallel sub-question decomposition, which leads to semantic drift and misaligned decision focus.
Frequently Asked Questions
This paper presents a new way for models to think and reason using both images and text. It shows how to better decide when to look at images while answering questions.
The introduction discusses the advancements in Vision-Language Models (VLMs) and categorizes existing multimodal reasoning approaches into two paradigms: text-centric pre-reasoning and unified multimodal representation.
Current models struggle with how to use visual information effectively. The proposed framework improves accuracy by dynamically deciding when to gather visual evidence. Experiments show significant performance improvements over existing methods.
Yes. PDFDigest can turn this paper into a structured explanation, key takeaways, visual summaries, and a narrated video when available.