SIVA-RL: SENSITIVITY-INVARIANCE VISUAL ALIGNMENT FOR MULTIMODAL REINFORCEMENT LEARNING

Introduction

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language and vision-language models. Reward ambiguity is acute in vision-dependent tasks such as counting, geometry, and diagram understanding.

Recent multimodal RL methods strengthen visual dependence through auxiliary perception signals, visual credit assignment, or comparisons between clean and visually modified inputs.

Perturbation-driven methods regularize the policy by contrasting its behavior under original and corrupted images.

Research Question

A common design ties the intervention type to a predetermined objective where destructive interventions encourage separation and mild transformations encourage consistency. Large-drop pairs provide sensitivity signals and receive a margin-bounded separation objective.

Low-drop pairs provide outcome-stability signals and receive a conservative, clean-anchored consistency objective.

An audit policy evaluates each clean-intervention pair and its clean-anchored reward change determines soft routing weights for a dual-direction alignment objective.

Methodology

RLVR elicits multi-step reasoning by optimizing task-level correctness without densely annotated reasoning trajectories. The change may arise from task-relevant evidence or from intervention-induced artifacts.

Study Design

Low-drop pairs are not necessarily reliable invariance signals as they may arise from task-irrelevant perturbations, redundant evidence, or language-prior shortcuts.

These methods share the limitation that final-answer rewards give no direct signal about whether the policy grounds its predictions in task-relevant visual evidence.

Results & Findings

Final-answer rewards indicate whether a prediction is correct but not how it was obtained. A vision-language model can receive high reward while relying on language priors, answer-frequency biases, or memorized reasoning templates rather than required visual evidence.

• Final-answer rewards indicate whether a prediction is correct but not how it was obtained.
• A vision-language model can receive high reward while relying on language priors, answer-frequency biases, or memorized reasoning templates rather than required visual evidence.
• This operator-conditioned design assumes that the same visual operator induces a sufficiently similar effect across samples.
• SIVA-RL routes each valid pair toward sensitivity or clean-anchored consistency according to its observed outcome change.
• Our paired intervention audit reveals operator-effect heterogeneity at the sample level.

Related Work

This section reviews existing multimodal reinforcement learning methods and their limitations, particularly in providing direct signals about the grounding of predictions in visual evidence. It positions SIVA-RL as a complementary approach to existing RLVR backbones.

Overview And Base Rl Backbone

The method section outlines the dataset and the backbone algorithms (GRPO and DAPO) used in SIVA-RL. It describes the components of the framework, including intervention construction and outcome-conditioned soft routing.

Figures Explained

The paper’s visual material highlights the workflow and the main system components.

• Figure 1 :: Figure 1: Paired intervention audit and SIVA-RL overview. (a) Visual interventions induce four paired-outcome categories. (b) Black-image, random-mask, and PatchSwap interventions all produce mixtures of outcome-drop and outcome-stable cases among the 906 clean-correct examples. (c) SIVA-RL routes each valid pair toward sensitivity or clean-anchored consistency according to its observed outcome change.
• Figure 2: perception and intervention supervision. One line of work strengthens visual grounding through auxiliary perception rewards, clean-intervention policy contrast, or token-and trajectorylevel visual credit assignment Xiao et al. (2025); Wang et al. (2025b); Huang et al. (2025a); Wang et al. (2026c); Li et al. (2026). Another line introduces distorted, masked, mixed, or otherwise intervened inputs to improve visual exploration, sensitivity, and robustness Liu et al. (2026); Li et al. (2025); Zhang et al. (2025b); Gao et al. (2026). These methods establish the value of supervision beyond answer correctness. Many of them, however, determine the optimization target from the intervention type, or use interventions mainly to redistribute visual credit across tokens. SIVA-RL instead evaluates the sample-wise behavioral effect of each localized intervention. The paired outcome then determines both the direction and strength of supervision. This design separates intervention construction from outcome valuation and softly routes valid pairs toward sensitivity alignment or bounded clean-anchored consistency.
• Figure 2 :: Figure 2: Overview of SIVA-RL. The upper branch represents the standard clean-image RLVR backbone. (a) Token-aligned, distance-constrained PatchSwap constructs a localized candidate intervention xi . (b) A frozen audit policy performs paired decoding and converts the clean-tointervention reward drop into sensitivity and invariance weights. (c) The clean rollout responses are re-scored under both visual conditions. Large-drop pairs are encouraged to exceed a sensitivity margin, while valid low-drop pairs are constrained within an invariance margin.
• ( 3 ): How do intervention construction and outcomeconditioned alignment contribute to the final performance? (4) Does the same PatchSwap operator continue to induce heterogeneous routing behavior during training? 4.1 EXPERIMENTAL SETUP Models and RL backbones. We experiment with Qwen2.5-VL-3B and Qwen2.5-VL-7B Bai et al. (2025) on both GRPO-and DAPO-style policy-optimization backbones Shao et al. (2024); Yu et al. (2026), training on ViRL39K Wang et al. (2026a). All backbone-scale configurations share the same training data and SIVA-RL hyperparameters. For each configuration, SIVA-RL augments the corresponding RL objective with localized intervention construction, outcome-conditioned routing, and dual-direction alignment (Section 3). Training applies a uniform-sensitivity warm-up followed by outcome-conditioned dual-direction alignment. Appendix A provides the optimization and validation settings. Benchmarks and metrics. Following prior perception-aware multimodal RL evaluation protocols Wang et al. (2025b; 2026c), we organize the main benchmark suite into two groups. The general multimodal reasoning group includes Geometry3K (Geo3K) Lu et al. (2021), MathVista Lu et al. (2024), We-Math Qiao et al. (2025), MMK12 Meng et al. (2025), and MathVerse Zhang et al. (2024). The vision-dependent group includes LogicVista Xiao et al. (2024), the counting subset of Super-CLEVR Li et al. (2023), MMMU-Pro Yue et al. (2025), and the vision-only subset of MathVerse, denoted MathVerse-V Zhang et al. (.
• Figure 3 :: Figure 3: Training-time outcome-conditioned routing diagnostics. We report the sensitivity routing rate, invariance routing rate, and mean paired answer-reward drop for the four backbone-scale configurations.

Conclusion

The audit supports the conclusion that paired outcomes are useful for assigning perturbation-based supervision but do not identify the visual evidence responsible for a prediction. PatchSwap modifies local visual content and spatial relationships while preserving most unmodified regions and the overall image context.

The general, vision-dependent, and overall scores are benchmark-level macro averages.

Absolute overall gains range from 3.58 to 8.24 points corresponding to relative improvements from 6.83% to 14.98% over the matched RL baseline.