Release Preview

PRM-as-a-Judge 1.5 Is Coming

Date June 30, 2026 Target Mid-July 2026 By PRM-as-a-Judge Team

We are preparing PRM-as-a-Judge 1.5 for mid-July 2026: a more accessible toolkit, a more complete analysis package, and a technical report. This preview summarizes the main direction of the release: correcting a PPL sensitivity issue found in real evaluation practice, and lifting dense OPD curves into higher-level trajectory diagnostics.

In short: PRM-as-a-Judge 1.5 turns dense progress curves into higher-level trajectory diagnostics, so model developers can analyze failure-side progress, recovery after drawdown, and success-side execution quality more directly.

A PPL Correction Found Through Orca Collaboration

During our close collaboration with the BAAI Orca world-model team on embodied action evaluation, we found that the old PPL definition could be too sensitive to the final few frames. A rollout may already reach a meaningful high-progress state, then drop slightly at the end because of release motion, object contact, camera jitter, or a short unstable correction. In that case, using terminal progress can underestimate a trajectory that actually reached its best state efficiently.

Path Length

\mathrm{PathLen}=\sum_{t=1}^{T}|p_t-p_{t-1}|

Old PPL

\mathrm{PPL}_{old}=\mathrm{FinalP}^{2}/\mathrm{PathLen}

New PPL

\mathrm{PPL}_{new}=\mathrm{MaxP}^{2}/\mathrm{PathLen}

The new PPL keeps the original process-efficiency idea, but anchors it to the best progress state reached by the rollout. Repeated back-and-forth behavior is still penalized through a longer path length, while final-frame noise no longer dominates the score.

Higher-Level Trajectory Diagnostics

PRM-as-a-Judge 1.5 also introduces a compact set of higher-level diagnostics distilled from OPD signals. In plain terms, they answer three questions: when a rollout fails, how much did it still move the task forward; when progress drops, can the model recover; and when a rollout succeeds, how clean and stable was the execution.

Let a rollout produce normalized progress values p₀, ..., p_T. We use MaxP = max_t p_t, SR_τ = 1[MaxP reaches the success threshold], M50 = 1[MaxP ≥ 0.50], and M75 = 1[MaxP ≥ 0.75].

FNS

For failed rollouts only: how much useful progress the model still makes before it fails.

I_f=\mathbf{1}[\mathrm{SR}_{\tau}=0]

\mathrm{FNS}_{\tau}=I_f(0.5\,\mathrm{MaxP}+0.3\,M75+0.2\,M50)

DRR

After the largest setback: how much progress the rollout can recover.

\mathrm{MDD}=\max_t(\max_{i\le t}p_i-p_t)

t^*=\arg\max_t(\max_{i\le t}p_i-p_t)

\mathrm{Rec}=\max_{t\ge t^*}p_t-p_{t^*}

\mathrm{DRR}=\min(1,\mathrm{Rec}/\mathrm{MDD})

SQS

For successful rollouts only: how clean, stable, and efficient the execution is.

I_s=\mathbf{1}[\mathrm{SR}_{\tau}=1]

\mathrm{SQS}_{\tau}=I_s(0.5\,\mathrm{PPL}+0.3(1-\mathrm{CRA})+0.2(1-\mathrm{Stag}))

MaxP-F is reported separately as the raw average best progress under failed trajectories, $\mathrm{MaxP\text{-}F}=\mathbb{E}[\mathrm{MaxP}\mid \mathrm{SR}_{\tau}=0]$ . It is intentionally simpler than FNS: MaxP-F asks only how far failed rollouts got, while FNS adds milestone structure to show whether those failures were genuinely near-success cases.

Serving Stronger Embodied Evaluation Workflows

PRM-as-a-Judge has already served Orca, the new flagship world model from BAAI, through a deep collaboration with the BAAI Orca team. Orca first learns a latent world representation and then uses downstream readouts for language, vision, and action. In the action-generation evaluation, PRM-as-a-Judge provides trajectory-level diagnostics beyond binary success, making it possible to compare how deeply each method progresses, how much failed rollouts approach success, and how well execution recovers after regressions.

Setting / Model	M25 ↑	M50 ↑	SR ↑	MaxP-F ↑	FNS ↑	DRR ↑	SQS ↑
Environment OOD
V-JEPA 2.1	40	12	0	23.0	13.9	25.8	0.0
Qwen3.5	26	10	0	18.3	11.2	19.2	0.0
π_0.5	54	16	2	27.9	17.7	31.5	1.5
Orca	64	16	4	33.9	19.3	32.9	1.8
Object OOD
V-JEPA 2.1	14	2	0	11.8	6.3	15.2	0.0
Qwen3.5	10	0	0	7.9	4.0	4.61	0.0
π_0.5	54	12	8	25.1	12.9	21.9	4.5
Orca	46	12	8	21.8	10.8	27.7	3.9
Overall
V-JEPA 2.1	27	7	0	17.4	10.1	20.5	0.0
Qwen3.5	18	5	0	13.1	7.6	11.9	0.0
π_0.5	54	14	5	26.5	15.3	26.7	3.0
Orca	55	14	6	27.9	15.1	30.3	2.9

Source: Orca action-generation evaluation on real-robot OOD settings. All columns are PRM-as-a-Judge trajectory diagnostics, higher is better, and values are reported on the same percentage-style scale used by the Orca results.

The results show that Orca's world-learning representation transfers effectively to embodied action readouts: it leads the overall comparison on M25, SR, MaxP-F, and DRR, while remaining highly competitive on FNS and SQS. The strong DRR is especially meaningful because it suggests better recovery after execution errors, not only better early progress.

Orca action evaluation qualitative examples

1 / 6

Recovery example 1.

We welcome more teams to use PRM-as-a-Judge for model evaluation, benchmark analysis, and policy diagnosis. Our team can provide technical support and evaluation consulting for collaborators who want to deploy dense trajectory auditing in their own embodied AI workflows.

The PRM-as-a-Judge 1.5 code package, easier-to-use toolkit, fuller experimental analysis, and technical report are planned for mid-July 2026.