DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoning

Introduction

The introduction discusses the significant improvements in reasoning capabilities of large language models (LLMs) and the role of Process Reward Models (PRMs) in enhancing reasoning processes. It highlights the challenges of applying PRMs to multimodal large language models (MLLMs) due to distribution shifts and dataset quality imbalances.

The Proposed Domain-reweighting Method

This section details the DreamPRM framework, which addresses dataset quality imbalance and training-inference discrepancies through a bi-level optimization approach. It explains the lower-level optimization for updating PRM weights and the upper-level optimization for adjusting domain importance weights.

Related Works

This section reviews recent studies on multimodal reasoning and the limitations of existing methods, such as Chain-of-Thought (CoT) prompting. It discusses various approaches to enhance reasoning capabilities in MLLMs and the challenges associated with obtaining process supervision signals for PRMs.

Domain-reweighting

The domain-reweighting section outlines the problem setting and the need for effective training of PRMs in MLLMs. It introduces the concept of using Monte Carlo methods to generate approximated supervision signals for training PRMs.

Experimental settings

The experimental settings section describes the methodology used for testing the proposed DreamPRM framework, including the base models, training hyperparameters, and baseline comparisons with state-of-the-art models.

Figures Explained

The paper’s visual material highlights the workflow and the main system components.

• Figure 2 :: Figure 2: General flow of training PRM and using PRM for inference. Training phase: Train PRM with Monte Carlo signals from intermediate steps of Chain-of-Thoughts (CoTs). Inference phase: Use the trained PRM to verify CoTs step by step and select the best CoT. Conventional training of PRM has poor generalization capability due to distribution shift between training set and testing set.
• Figure 3 :: Figure 3: The proposed bi-level optimization based domain-reweighting method. Lower-level optimization: In this stage, PRM’s parameters are updated on multiple datasets with domain weights, allowing the PRM to prioritize domains with better quality. Upper-level optimization: In this stage, the PRM is evaluated on a separate meta dataset to compute an aggregation function loss and optimize the domain weights. DreamPRM helps address dataset quality imbalance problems and leads to stronger and more generalizable reasoning performance.
• Figure 3: V ϕ (x, ŷi ). With the PRM training loss on a single domain D k above, we next define the domain-reweighted training objective of PRM on multiple training domains D = {D k } K k=1 . The overall objective is a weighted sum of the single-domain PRM training losses, allowing the contribution of each domain to be adjusted during the learning process:.
• Figure 4 :: Figure 4: Leaderboard on MathVista (as ofOctober 15, 2025). The first column (“o4-mini + DreamPRM”) reports our own evaluation, while the remaining results are taken from the official MathVista leaderboard. The compared models include VL-Rethinker [62] , Step R1-V-Mini [58] , Kimi-k1.6-preview [43] , Kimi-k1.5 [24] , Doubao-pro-1.5 [60] , Ovis2-34B [1] , OpenAI o1 [45] , Llama 4 Maverick [41, 42] , and Vision-R1-7B [18] .
• Figure 5 :Figure 6 :: Figure 5: Comparative evaluation of DreamPRM on multimodal reasoning benchmarks. Radar charts report accuracy (%) on five datasets (WEMATH, MATHVISTA, MATHVISION, MMVET, and MMSTAR). (a) Impact of different data selection strategies. (b) Comparison with existing test-time scaling methods. (c) Ablation study of three key components, i.e. w/o aggregation function loss (AFL), w/o bi-level optimization (BLO), and w/o structural thinking (ST).