OR ELSE: A DIFFERENTIABLE TRUST REGION FOR POLICY OPTIMIZATION

Introduction

PPO-OR and GRPO-OR replace the clipped policy term with an OR squared-margin loss in rollout-relative token log-ratio space. Both group-relative methods exhibit substantially larger rollout-to-current log-ratio displacement than the GAE methods.

At G = 2, the GRPO-OR diagnostics do not translate into a reward-score gain.

Whether larger groups change this outcome remains open.

Research Question

PPO and the GRPO baseline use clipped surrogate objectives whose favorable-direction saturation introduces an abrupt change in the scalar objective’s derivative. The objective remains sensitive to movement in the opposite, disadvantageous direction.

We introduce PPO-OR and GRPO-OR, which replace the clipped policy term with an objective derived from Output Reset.

In forward-value form, the per-token objective is equivalent to the squared one-sided margin.

Methodology

PPO remains a common optimization method for LLM post-training and uses the clipped surrogate objective. PPO-OR remains an online policy-optimization method and specifically replaces the PPO-style clipped policy term with a C 1 squared one-sided margin in token-level log-ratio space.

Study Design

For each sampled token, the method measures the change in log-probability relative to the rollout policy and uses the sign of the estimated advantage to determine whether that token should become more or less probable.

Each method was evaluated with seeds 42, 43, and 44 on the same machine and with the same frozen reward model.

Results & Findings

We ask whether Output Reset offers a useful alternative for large language model post-training. We compare PPO-clip with PPO-OR under generalized advantage estimation and GRPO with GRPO-OR under group-relative advantages using Llama-3.2-1B-Instruct on Anthropic hh-rlhf.

• We ask whether Output Reset offers a useful alternative for large language model post-training.
• We compare PPO-clip with PPO-OR under generalized advantage estimation and GRPO with GRPO-OR under group-relative advantages using Llama-3.2-1B-Instruct on Anthropic hh-rlhf.
• Under GAE, PPO-OR has a mean final training-time reward-model score 0.305 higher than PPO-clip.
• Under group-relative advantages, GRPO-OR shows a smaller observed spread, a near-zero terminal OR residual, and a declining overshoot fraction.
• OR changes optimization behavior in both matched comparisons, but the observed reward effect differs between them.

Practical Applications

Under the implemented pipeline, this was the largest group size that could be run reliably together with four prompts per rollout batch. These are planning estimates rather than direct end-to-end measurements and do not include every possible memory optimization.

The favorable branch for a sampled token is defined using the current estimated advantage sign, while later rollouts may contain different tokens, states, and signs.

The mechanisms responsible for the larger group-relative displacement, and the factors that might change it, are left to Sec.

Output Reset

The Output Reset (OR) method is introduced as a target-adjustment procedure for classifier training, which is adapted for token-level policy optimization in this work.

Ppo-Or

PPO-OR is presented as a method that replaces the clipped policy-loss term while maintaining other components of the policy optimization pipeline, focusing on the implications of using OR in this context.

Figures Explained

The paper’s visual material highlights the workflow and the main system components.

• Figure 1: where r RM,b is the response-level reward-model score and T b is the final valid generated position. The resulting training loss is.
• Table 3 :Figure 1 :: Figure 1: Training-time reward-model score over 500 rollout steps for all four methods, with three seeds per method.Curves are smoothed with a moving window of 10 steps, and shaded regions denote ±1 standard deviation across seeds. The shared reward scale supports descriptive within-experiment comparisons, but its numerical zero is not calibrated as a success threshold.
• Figure 2 :: Figure 2: Policy-loss traces over training for three seeds per method; shaded regions denote ±1 standard deviation. The left panel shows the full plotted range, and the right panel enlarges the near-zero region. GRPO has substantially larger and more variable logged loss values than the other configurations. Absolute values across the clipped-surrogate and OR-MSE objectives are not directly comparable, so the figure is interpreted as an objective-specific optimization diagnostic rather than a common performance scale.
• Figure 3: Figure3summarizes the two matched substitutions without treating them as a single factorial effect. The GAE comparison shows a +0.305 difference in mean final score when the clipped surrogate is replaced by OR, together with a +0.048 increase in the observed standard deviation. The group-relative comparison shows a -0.031 difference in mean final score and a -0.053 difference in standard deviation. The two comparisons therefore exhibit different observed patterns: higher mean score with greater dispersion under GAE, and a similar but slightly lower mean with smaller dispersion under the group-relative configuration. The present experiment does not isolate an interaction between the loss and advantage estimator, because the two families are not otherwise matched.
• Figure 4 :: Figure 4: OR diagnostics over training for PPO-OR and GRPO-OR, with three seeds and ±1 standard-deviation bands. Left: overshoot fraction from Eq. (29). Tokens on this branch have zero direct OR residual at that evaluation. Right: mean squared OR target magnitude from Eq. (30), labeled “OR energy” in the training logs and capped at 25 for display. Neither quantity is a formal convergence measure or a bound on policy drift.