Eval-Actions
Fine-Grained Execution Quality Evaluation for Robotic Manipulation

Figure 1. Motivation and overview of the Eval-Actions diagnostic evaluation framework. Conventional manipulation evaluation often reduces each execution to a binary success/failure label, which cannot distinguish process quality among successful executions. Eval-Actions assesses execution quality from task completion, motion smoothness, execution efficiency, and visible collision-related events, provides EG/RG/CoT annotation labels, and introduces AutoEval as a reference multimodal evaluator for quality scores, rankings, and CoT-style diagnostic explanations.

Figure 2. Overview of the Eval-Actions benchmark. The figure illustrates task diversity across 150+ scenarios, including single-arm interactions and bimanual coordination tasks; a detailed case study with demonstrations ranging from high-quality successes to failures; and the data composition of each episode, including RGB-D sensory data, 7/14-DoF joint trajectories, task descriptions, success/failure labels, trajectory-source labels, and a fine-grained quality radar chart over task completion, smoothness, collision-related events, and efficiency.

Table I. Comparison of representative robotic manipulation datasets. Existing datasets target policy training, focusing on trajectory quantity and diversity. Eval-Actions is designed for diagnostic evaluation and includes failure cases, mixed trajectory sources, fine-grained quality scores, and CoT-style explanatory annotations based on RGB-D videos and robot trajectories.

Table IV. Statistics of the full Eval-Actions benchmark and the annotation-complete Eval-Actions Small (EAS) subset. The failure entry for EAS reports the failure ratio computed on the annotation-complete subset; AutoEval experiments are conducted on EAS unless otherwise specified.

Figure 5. Overview of the proposed AutoEval framework. AutoEval takes a robot manipulation video sequence and a compact kinematic summary as inputs. AutoEval-S is used for structured prediction under the EG and RG protocols, where Spatio-Temporal Aggregation tiles neighboring frames into composite visual inputs to preserve short-term motion cues under a fixed VLM visual-input budget. AutoEval-P is used for CoT-style diagnostic explanation generation and is optimized with Group Relative Policy Optimization, encouraging consistency between diagnostic explanations and structured predictions.

Table VI. Comparative performance analysis on the Eval-Actions benchmark. Results are reported across three protocols: Expert Grading (EG), Rank-Guided (RG), and Chain-of-Thought (CoT). To quantify the domain gap, the zero-shot performance of representative VLMs without supervised fine-tuning is included; the near-zero correlations indicate the importance of task-specific fine-tuning.

Table IX. Structured generalization on held-out tasks and robot arms. Arm and Task denote unseen-arm-only and unseen-task-only splits; RoboMIND-S and Open-X-S provide additional cross-dataset distribution shifts. Succ. and Src. report success-prediction and source-prediction accuracies.

Table X. Policy-level ranking using AutoEval-S under the EG protocol. Held-out denotes policy models whose rollout trajectories are excluded from the AutoEval training split.