Quick Start

PRM-as-a-Judge

Evaluate three bundled robot rollout cases with a Process Reward Model (PRM) and generate progress curves, metrics, summaries, and an interactive report.

快速开始

PRM-as-a-Judge

使用过程奖励模型（Process Reward Model，PRM）评测仓库内的 3 个机器人执行案例，并生成进度曲线、评测指标、汇总结果和交互式报告。

Choose a Workflow

Bundled videos & tasks → Judge Model → progress curves → metrics & report

Notebook

Run the complete workflow cell by cell, inspect the cases, and open the generated report.

Open the Notebook

Command Line

Follow the steps below to run the same example with the standard shell entry point.

Run from the terminal

Run the Bundled Demo

1. Clone the repository

git clone https://github.com/Yuheng2000/PRM-as-a-Judge.git
cd PRM-as-a-Judge

2. Create and activate the environment

conda create -n prm-judge python=3.10 -y
conda activate prm-judge

3. Install the demo dependencies

python -m pip install --upgrade pip
python -m pip install -e ".[dopamine]" \
  -c constraints/dopamine-cu128-py310.txt
python -m pip check

4. Download the checkpoint

The bundled demo uses Robo-Dopamine-GRM-2.0-8B-Preview as its default PRM checkpoint.

PRM_PATH=$(hf download tanhuajie2001/Robo-Dopamine-GRM-2.0-8B-Preview)

5. Evaluate all three cases

MANIFEST=eval/examples/manifest_demo_cases.jsonl \
PRM_PATH="$PRM_PATH" \
VISUALIZE=1 \
bash eval/run_eval.sh

The final terminal line prints the generated run directory.

6. Open the interactive report

python eval/run_judge.py serve \
  --run-root eval/results/run_YYMMDD_HHMMSS

After the run completes, open the report URL printed by the command. The final performance report includes the Model Leaderboard shown below, which summarizes the core metrics for every evaluated model. You can also use the interactive report to filter and inspect every case, including its rollout video, progress curve, and per-case metrics.

Final Model Leaderboard summarizing performance metrics across evaluated policy models — **Final performance report.** The Model Leaderboard provides the final model-level performance summary for the evaluation run.

Failure Localization and Drawdown chart showing where failed rollouts stopped and their drawdown rate — **Failure localization and drawdown.** Inspect where failed rollouts stopped, their average MaxP-F, and the significant-drawdown rate.

Interactive trajectory explorer showing a rollout video, progress curve, and case metrics — **Interactive trajectory explorer.** Inspect rollout videos together with progress curves, live-frame values, per-case metrics, and milestones.

The report also supports failure localization and drawdown analysis, together with case-level inspection in the interactive trajectory explorer. See the Advanced Guide for the complete visualization reference.

7. Evaluate your own rollout

Create a JSONL manifest containing a case ID, task description, and video path:

{"case_id":"case_001","task":"put the can into the basket","video":"videos/case_001.mp4"}

MANIFEST=/path/to/cases.jsonl \
PRM_PATH="$PRM_PATH" \
VISUALIZE=1 \
bash eval/run_eval.sh

Configure additional views, evaluation modes, GPUs, curve processing, metrics, or another PRM.

Continue to the Advanced Guide →

WeChat Community

Join the group to discuss setup, evaluation, and rollout analysis with the community.

Scan the QR code to join the WeChat community.

选择使用方式

示例视频与任务 → Judge Model → 进度曲线 → 指标与报告

Notebook

逐单元运行完整流程，检查示例案例，并打开生成的报告。

打开 Notebook

命令行

按照下面的步骤，通过标准 Shell 入口运行相同示例。

在终端运行

运行仓库内示例

1. 克隆仓库

git clone https://github.com/Yuheng2000/PRM-as-a-Judge.git
cd PRM-as-a-Judge

2. 创建并激活环境

conda create -n prm-judge python=3.10 -y
conda activate prm-judge

3. 安装示例所需依赖

python -m pip install --upgrade pip
python -m pip install -e ".[dopamine]" \
  -c constraints/dopamine-cu128-py310.txt
python -m pip check

4. 下载 checkpoint

仓库内示例默认使用 Robo-Dopamine-GRM-2.0-8B-Preview 作为 PRM checkpoint。

PRM_PATH=$(hf download tanhuajie2001/Robo-Dopamine-GRM-2.0-8B-Preview)

5. 评测全部 3 个案例

MANIFEST=eval/examples/manifest_demo_cases.jsonl \
PRM_PATH="$PRM_PATH" \
VISUALIZE=1 \
bash eval/run_eval.sh

终端最后一行会输出本次评测结果目录。

6. 打开交互式报告

python eval/run_judge.py serve \
  --run-root eval/results/run_YYMMDD_HHMMSS

评测完成后，在浏览器中打开命令输出的报告地址。最终性能报告会生成下图所示的 Model Leaderboard，汇总展示每个被评测模型的核心指标；还可以通过交互式报告筛选并逐条查看每个案例（case）的执行视频、进度曲线和单案例指标。

汇总展示各策略模型性能指标的最终 Model Leaderboard — **最终性能报告。**Model Leaderboard 汇总展示本次评测中各模型的最终性能指标。

展示失败轨迹停止位置、平均 MaxP-F 和显著回撤率的失败定位与回撤报告 — **失败定位与回撤。**查看失败轨迹停止的位置、平均 MaxP-F 和显著回撤率。

同时展示执行视频、进度曲线和单案例指标的交互式轨迹查看器 — **交互式轨迹查看器。**同步查看执行视频、进度曲线、实时帧数值、单案例指标和里程碑。

报告还支持失败定位与回撤分析，以及通过交互式轨迹查看器逐案例检查执行过程。更多可视化模块和使用说明见高级指南。

7. 评测自己的轨迹

创建包含案例 ID、任务描述和视频路径的 JSONL manifest：

{"case_id":"case_001","task":"put the can into the basket","video":"videos/case_001.mp4"}

MANIFEST=/path/to/cases.jsonl \
PRM_PATH="$PRM_PATH" \
VISUALIZE=1 \
bash eval/run_eval.sh

配置更多视角、评测模式、GPU、曲线处理、指标或其他 PRM。

继续阅读高级指南 →

微信群

欢迎加入微信群，与社区交流环境配置、评测和轨迹分析问题。

扫描二维码加入微信群。