PDFdigest

Cs.CV Research

DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning

This paper discusses a new approach to improve how models understand and reason about images by allowing them to interact with visual content directly.

4 min read July 6, 2026 Watch Video
Analyze with PDFdigest

Content & Liability Disclaimer

This article and its accompanying video are automated summaries derived from DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning by Chi Zhang, Haibo Qiu, Qiming Zhang, Zhixiong Zeng, Lin Ma, Jing Zhang. The original research was conducted solely by the paper's authors; PDFdigest did not conduct any of the research and makes no claims of ownership over the underlying scientific work.

The video narration is generated by artificial intelligence and references the paper's authors for attribution. The video is not narrated by any of the paper's authors. This content may contain inaccuracies, omissions, or misinterpretations of the original research. First-person language (e.g., "we found", "our results") reflects the original authors' voice, not PDFdigest's. Always read the original paper for accurate, verified information before making any decisions based on this content.

This content is provided "as is" without any warranties, express or implied. Simulated systems OÜ, its officers, directors, employees, and agents shall not be liable for any direct, indirect, incidental, special, consequential, or punitive damages arising from your use of, reliance on, or access to this content, including but not limited to errors, omissions, or misinterpretations of the original research. This disclaimer applies to the fullest extent permitted by applicable law.

Key Takeaways
  1. 1 This gap is critical, as the core objective for advanced reasoning models is to solve challenging, real world problems.
  2. 2 Crucially, beginning in this phase, the VLM consumes editor-produced visual tokens rather than ground-truth visual context, and the LM objective is conditioned on the editor's outputs.
  3. 3 The phase-1 language modeling objective then averages over all text tokens across the corpus and sums over examples, segments, and token positions:.
  4. 4 Finally, we unfreeze the LLM backbone while maintaining the Phase 2 objective.

Introduction

Recent progress shows that integrating step-by-step reasoning into VLMs has substantially improved their performance on complex tasks . However, current VLMs often exhibit a “thinking over see-ing” tendency : while they can generate lengthy and seemingly coherent reasoning traces, these traces are frequently detached from the actual visual input.

To address this, OpenAI introduced a new axis for VLM reasoning with “thinking with images” .

Instead of merely generating textual reasoning traces that overlook visual content, this approach enables models to actively interact with images through an explicit mechanism.

Important Note

Yet, these efforts have largely been confined to limited scenarios such as jigsaw puzzles and mazes.

Important Note

While promising, these paradigms struggle with grounding noise and limited precision, impeding the generation of consistent and controllable traces (Figure 1 ).

Research Question

The phase-1 language modeling objective then averages over all text tokens across the corpus and sums over examples, segments, and token positions:. Crucially, beginning in this phase, the VLM consumes editor-produced visual tokens rather than ground-truth visual context, and the LM objective is conditioned on the editor’s outputs.

Finally, we unfreeze the LLM backbone while maintaining the Phase 2 objective.

This gap is critical, as the core objective for advanced reasoning models is to solve challenging, real world problems.

Methodology

In both cases, the core mechanism relies heavily on precise coordinate regression. Compared to the prior approach that edits images in a highly compressed latent space, our method preserves richer semantic information: the editor operates directly on visual tokens with explicit conditioning on action embeddings.

Study Design

To better understand the effect of our method, we conduct an in-depth comparison against the base model Qwen2.5-VL-7B.

When breaking down results by task category, consistent patterns emerge: the most reliable gains appear in tasks involving geometry and counting, with particularly striking improvement on MathVision reaching 5.3 points.

How PDFdigest Helps You Understand Research

Instant Paper Analysis

Get structured summaries and key findings from dense PDFs in seconds.

Visual Explanations

Turn complex methods, figures, and results into clearer visual breakdowns.

AI-Powered Q&A

Ask focused questions and get answers grounded in the paper.

Try PDFdigest Free

Results & Findings

In many cases, the models misinterpret critical details in the image or even hallucinate content that is not present , suggesting that their reasoning is driven more by linguistic priors than by genuine visual perception . By zooming, cropping, and performing image-level manipulations, VLMs are encouraged to ground their reasoning in actual visual evidence.

  • In many cases, the models misinterpret critical details in the image or even hallucinate content that is not present , suggesting that their reasoning is driven.
  • By zooming, cropping, and performing image-level manipulations, VLMs are encouraged to ground their reasoning in actual visual evidence.
  • This paradigm represents a shift from “thinking over seeing” to “thinking through seeing,” enabling models to analyze visual information more deeply, more thoroughly, and ultimately achieve.
  • Following such an idea, recent efforts have explored stimulating the use of visual information in the reasoning process to enhance model performance in perception and reasoning.
  • VILASR defines a closed set of drawing operations and trains the model to decide when to invoke each of them.
Important Note

Despite their differences, these approaches share a common limitation: the supported action space remains relatively restricted, and they inevitably rely on accurate spatial grounding, which remains challenging: curated data seldom yield perfectly accurate annotations, and end-to-end reinforcement learning rollouts are.

Important Note

To overcome the constraints of a limited action space and to expand the model’s “thinking space,” another line of work makes a conceptual leap from execution to imagination, aiming to unify gen-.

Practical Applications

While such data may seem scarce, prior work has demonstrated its efficacy in enhancing VLM perception and reasoning across structured domains and beyond . It is worth noting that while programmatic edits serve as a strong reference, they are illustrative rather than absolute ground truth, as natural language instructions may admit multiple valid implementations.

First, the dataset is generated exclusively from code, which may limit the approach’s applicability to broader, open-world domains.

Code Editor

The Code Editor section presents a question regarding finding overlap areas and provides a source code snippet related to visual question answering.

Solver

The Solver section outlines the steps required to ensure accuracy in visual tasks, including the use of tools for zooming in on specific areas.

Source Paper Figures and Captions

Source-paper figure
Figure 2. Disciplinary coverage of our dataset. Figure 3. Wordcloud of manipulations.
Figure 2. Disciplinary coverage of our dataset. Figure 3. Wordcloud of manipulations.
Source-paper figure
Figure 4. Architecture of the proposed DeepSketcher model. A query Q and initial image I0 are encoded into the vision-language model, producing reasoning tokens Rt and edit instructions Actt. The Embedding Editor manipulates visual embeddings directly, supervised by code-rendered ground-truth edits, and inserts updated embeddings back into the VLM context. This process yields interleaved reasoning and visual manipulation traces, ultimately producing the final answer.
Figure 4. Architecture of the proposed DeepSketcher model. A query Q and initial image I0 are encoded into the vision-language model, producing reasoning tokens Rt and edit instructions Actt. The Embedding Editor manipulates visual embeddings directly, supervised by code-rendered ground-truth edits, and inserts updated embeddings back into the VLM context. This process yields interleaved reasoning and visual manipulation traces, ultimately producing the final answer.
Source-paper figure
Reasoning Warm-up. We initially optimize the model on interleaved image-text sequences using ground-truth image features, bypassing the embedding editor. The supervision signal is applied only to textual tokens and , tokens. These two special tokens serve as boundary markers for visual content in interleaved sequences. Visual features are inserted as continuous embeddings and serve only as conditioning context. This enables the model to learn proper structural demarcation between textual and visual modalities during generation. Formally, for the i-th training example with T (i) images {I (i) 0 , . . . , I (i)
Reasoning Warm-up. We initially optimize the model on interleaved image-text sequences using ground-truth image features, bypassing the embedding editor. The supervision signal is applied only to textual tokens and , tokens. These two special tokens serve as boundary markers for visual content in interleaved sequences. Visual features are inserted as continuous embeddings and serve only as conditioning context. This enables the model to learn proper structural demarcation between textual and visual modalities during generation. Formally, for the i-th training example with T (i) images {I (i) 0 , . . . , I (i)
Source-paper figure
Figure 5. Difference map visualizations. Each example shows the input image (left), the programmatic rendering (available only in Indicator-500) (middle), and the difference map between the embedding editor output and the original visual embedding (right).
Figure 5. Difference map visualizations. Each example shows the input image (left), the programmatic rendering (available only in Indicator-500) (middle), and the difference map between the embedding editor output and the original visual embedding (right).
Source-paper figure
Figure 6. Overview of img2code pipeline.
Figure 6. Overview of img2code pipeline.
Source-paper figure
Figure7. Overview of the DeepSketcher data curation pipeline. We first construct a dataset of VQA problems with images rendered directly from code. An agentic system is then designed to generate interleaved image-text reasoning traces.
Figure7. Overview of the DeepSketcher data curation pipeline. We first construct a dataset of VQA problems with images rendered directly from code. An agentic system is then designed to generate interleaved image-text reasoning traces.
Source-paper figure
Highlight the segment from AB perpendicular to CD in bright green, and label it "height = 4 m". Highlight segment BC in blue and label it "diameter = 2 cm". Highlight angle BCE at point C with a red arc. Mark EB and DC in bold. Highlight the two circular bases in different colors to clarify their locations. Draw a colored circle around each person's face in the image. (a) Cases where the model generally attends to the region in accordance with the instruction. Highlight the rectangle and label its length as "height". Highlight the base region of the cuboid formed after folding, showing the 10 cm × 5 cm rectangle. Highlight angle 1 in red and label it "95°" to confirm its value. Highlight triangle BEC in blue and mark the right angle at E with a small square.(b) Cases where the model's attention deviates from its textual intent.
Highlight the segment from AB perpendicular to CD in bright green, and label it "height = 4 m". Highlight segment BC in blue and label it "diameter = 2 cm". Highlight angle BCE at point C with a red arc. Mark EB and DC in bold. Highlight the two circular bases in different colors to clarify their locations. Draw a colored circle around each person's face in the image. (a) Cases where the model generally attends to the region in accordance with the instruction. Highlight the rectangle and label its length as "height". Highlight the base region of the cuboid formed after folding, showing the 10 cm × 5 cm rectangle. Highlight angle 1 in red and label it "95°" to confirm its value. Highlight triangle BEC in blue and mark the right angle at E with a small square.(b) Cases where the model's attention deviates from its textual intent.
Source-paper figure
Figure 8. Difference map visualization from public benchmarks. (a) Alignment between attention and instruction. (b) Cases with deviation from textual intent.
Figure 8. Difference map visualization from public benchmarks. (a) Alignment between attention and instruction. (b) Cases with deviation from textual intent.
Source-paper figure
More Visualization Results on Public BenchmarksIn Section 4.3, we examined "where the model is looking" by visualizing difference maps on our in-house benchmark. Here, we extend this analysis with additional examples on public benchmarks in Fig.8for a more comprehensive view. As shown in Fig.8a, the difference maps generally align with the model's textual edit intent. A notable case appears in the bottom-right example from MathVista, where the editor correctly attends to the faces of all three individuals in accordance with the instruction, despite the model being trained exclusively on code-rendered images that contain no such open-world scenarios. This result suggests that the model exhibits a certain degree of generalization, as it can attend to novel cases far beyond its training distribution. Then, we turn to failure cases in Fig.8b, where the model's attention seems to deviate from the intent expressed in natural language instructions. These examples might demonstrate the current limitations of the editor and the remaining challenges in faithful visual manipulation.10.2. Ablation: Interleaved vs. Pure-text CoTWe verify the necessity of visual-augmented reasoning by comparing our Interleaved CoT against a Pure-text CoT baseline. To ensure a fair comparison, both settings utilize the identical Solver model and training data (DeepSketcher 31k), differing only in that the baseline is text-only CoT traces without access to visual tool invocation.
More Visualization Results on Public BenchmarksIn Section 4.3, we examined "where the model is looking" by visualizing difference maps on our in-house benchmark. Here, we extend this analysis with additional examples on public benchmarks in Fig.8for a more comprehensive view. As shown in Fig.8a, the difference maps generally align with the model's textual edit intent. A notable case appears in the bottom-right example from MathVista, where the editor correctly attends to the faces of all three individuals in accordance with the instruction, despite the model being trained exclusively on code-rendered images that contain no such open-world scenarios. This result suggests that the model exhibits a certain degree of generalization, as it can attend to novel cases far beyond its training distribution. Then, we turn to failure cases in Fig.8b, where the model's attention seems to deviate from the intent expressed in natural language instructions. These examples might demonstrate the current limitations of the editor and the remaining challenges in faithful visual manipulation.10.2. Ablation: Interleaved vs. Pure-text CoTWe verify the necessity of visual-augmented reasoning by comparing our Interleaved CoT against a Pure-text CoT baseline. To ensure a fair comparison, both settings utilize the identical Solver model and training data (DeepSketcher 31k), differing only in that the baseline is text-only CoT traces without access to visual tool invocation.
Source-paper figure
Figure 9. Inference sample and response from pure text CoT training and interleaved CoT training. The interleaved CoT refines the reasoning by correcting initial textual guesses through inner visual thought and tool-driven verification.
Figure 9. Inference sample and response from pure text CoT training and interleaved CoT training. The interleaved CoT refines the reasoning by correcting initial textual guesses through inner visual thought and tool-driven verification.
Source-paper figure
Figure 10. Inference sample and response from pure text CoT training and interleaved CoT training.
Figure 10. Inference sample and response from pure text CoT training and interleaved CoT training.
Source-paper figure
Figure 11. Error analysis on responses from pure text CoT training and interleaved CoT training.
Figure 11. Error analysis on responses from pure text CoT training and interleaved CoT training.
Source-paper figure
Output MUST follow this template exactly: Step 1: … reasoning … Step 2: … reasoning … ... ... ⚠ IMPORTANT RULES ⚠ 1. If you are less than 99% confident in your answer, you MUST call the Renderer by filling with a specific drawing instruction (e.g., "Draw a red circle around triangle ABC").
Output MUST follow this template exactly: Step 1: … reasoning … Step 2: … reasoning … … … ⚠ IMPORTANT RULES ⚠ 1. If you are less than 99% confident in your answer, you MUST call the Renderer by filling with a specific drawing instruction (e.g., "Draw a red circle around triangle ABC").
PDFDIGEST AI

Struggling to understand complex research papers?

Upload any PDF and get instant AI-powered explanations, summaries, and visual breakdowns. Turn dense academic writing into clear, actionable insights.

Upload a Paper

Frequently Asked Questions

Crucially, beginning in this phase, the VLM consumes editor-produced visual tokens rather than ground-truth visual context, and the LM objective is conditioned on the editor’s outputs. This gap is critical, as the core objective for advanced reasoning models is to solve challenging.

It is worth noting that img2code is an extremely challenging task, as it requires the model to faithfully capture all fine-grained details in an image using programmatic language. This low success rate is attributable to the task’s sensitivity; even a minor error.

By zooming, cropping, and performing image-level manipulations, VLMs are encouraged to ground their reasoning in actual visual evidence. Following such an idea, recent efforts have explored stimulating the use of visual information in the reasoning process to enhance model performance in perception.

Therefore, the final overlapping area of the two circles is about 17.65 square meters. NONE 17.65 . First, the dataset is generated exclusively from code, which may limit the approach’s applicability to broader, open-world domains.

Despite their differences, these approaches share a common limitation: the supported action space remains relatively restricted, and they inevitably rely on accurate spatial grounding, which remains challenging: curated data seldom yield perfectly accurate annotations, and end-to-end reinforcement learning rollouts are similarly error-prone.

This paper discusses a new approach to improve how models understand and reason about images by allowing them to interact with visual content directly.