← Back to the post

Detailed scores

Figure 5 in the post gives one summary per benchmark. This page breaks results down by task family, suite, difficulty level, perturbation type or scene. All results are from the Native GPT-6-Astra Policy behind starRoboEval.

How we count
  • Agents may stop voluntarily; each benchmark's native evaluator determines success.
  • Each table states its evaluation scope.
  • All runs use reasoning effort high and each benchmark's native step limit.
  • Groups follow the benchmark's own documentation where it has one. Groups marked (ours) are our own grouping by task name.

Overview

BenchmarkEvaluation scopeSuccess / episodesRate

SimplerEnvOct 6

15 tasks × 5 episodes.

RobotSuccess / episodesSR
Google Robot8 / 5514.5%
WidowX (Bridge)12 / 2060.0%
Total20 / 7526.7%
Task familySuccess / episodesSR
Pick coke can5 / 1533.3%
Move near0 / 50.0%
Open drawer0 / 150.0%
Close drawer2 / 1513.3%
Put apple into drawer1 / 520.0%
Spoon on towel3 / 560.0%
Carrot on plate4 / 580.0%
Stack green block on yellow3 / 560.0%
Eggplant into basket2 / 540.0%

LIBEROOct 27

4 suites × 10 tasks × 5 initial states.

SuiteSuccess / episodesSR
Spatial9 / 5018.0%
Object25 / 5050.0%
Goal22 / 5044.0%
Long (LIBERO-10)13 / 5026.0%
Total69 / 20034.5%

CALVINOct 23–24

34 tasks × 5 episodes. Each task is scored on its own.

Task family (ours)Success / episodesSR
Lift block41 / 4591.1%
Push block23 / 3076.7%
Rotate block20 / 3066.7%
Open / close drawer10 / 10100.0%
Move slider left / right10 / 10100.0%
Turn LED on / off10 / 10100.0%
Turn lightbulb on / off5 / 1050.0%
Place / push into drawer or slider14 / 1593.3%
Stack / unstack7 / 1070.0%
Total140 / 17082.4%

MetaWorldOct 23–24

50 tasks × 5 episodes.

Difficulty (Seo et al. 2022)Success / episodesSR
Easy102 / 14072.9%
Medium43 / 5578.2%
Hard14 / 3046.7%
Very hard19 / 2576.0%
Total178 / 25071.2%

RoboDojoOct 21 – Nov 3

42 tasks × 5 layouts. Scores use the native 0–100 scale.

TaskSuccess / episodesSRMean native score
match_and_pick_from_conveyor5 / 5100.0%100.0
general_pickup4 / 580.0%80.0
align_blocks4 / 580.0%80.0
stack_blocks3 / 560.0%66.0
stack_blocks_by_language3 / 560.0%60.0
classify_objects_by_language3 / 560.0%64.0
stack_bowls3 / 560.0%63.0
swap_T2 / 540.0%40.0
cover_blocks2 / 540.0%44.0
solve_equation2 / 540.0%40.0
arrange_largest_number2 / 540.0%54.0
fold_clothes2 / 540.0%48.0
pick_from_conveyor_by_image1 / 520.0%20.0
play_Xylophone1 / 520.0%20.0
put_bottles_into_dustbin2 / 540.0%52.0
fasten_screws1 / 520.0%44.0
pour_balls_into_vase1 / 520.0%20.0
classify_objects1 / 520.0%36.0
build_tower0 / 50.0%6.0
deposit_coin0 / 50.0%4.0
fill_egg_holder0 / 50.0%5.0
fill_pen_holder0 / 50.0%15.0
hang_mugs0 / 50.0%3.0
imitate_sorting_sequence0 / 50.0%12.0
insert_key0 / 50.0%15.0
insert_tubes0 / 50.0%0.0
make_kong0 / 50.0%0.0
make_toast0 / 50.0%0.0
organize_table0 / 50.0%25.0
pack_objects_into_box0 / 50.0%9.0
play_stacking_toy0 / 50.0%0.0
play_tic_tac_toe0 / 50.0%75.0
plug_in_charger0 / 50.0%0.0
pour_by_language0 / 50.0%0.0
pour_liquid_into_cup0 / 50.0%0.0
press_by_number0 / 50.0%0.0
push_T0 / 50.0%0.0
sort_nesting_dolls_by_size0 / 50.0%0.0
store_laptop_and_headphones0 / 50.0%4.0
store_tools_in_toolbox0 / 50.0%0.0
swap_blocks0 / 50.0%0.0
sweep_blocks0 / 50.0%0.0
Total42 / 21020.0%26.4

RoboCasa365Oct 28 – Nov 8

50 tasks × 5 episodes.

GroupSuccess / episodesSR
Atomic (seen)30 / 9033.3%
Composite (seen)4 / 805.0%
Composite (unseen)23 / 8028.8%
Total57 / 25022.8%
TaskSuccess / episodesSR
Atomic (seen)
CloseBlenderLid0 / 50.0%
CloseFridge5 / 5100.0%
CloseToasterOvenDoor2 / 540.0%
CoffeeSetupMug0 / 50.0%
NavigateKitchen5 / 5100.0%
OpenCabinet5 / 5100.0%
OpenDrawer0 / 50.0%
OpenStandMixerHead2 / 540.0%
PickPlaceCounterToCabinet2 / 540.0%
PickPlaceCounterToStove1 / 520.0%
PickPlaceDrawerToCounter0 / 50.0%
PickPlaceSinkToCounter2 / 540.0%
PickPlaceToasterToCounter1 / 520.0%
SlideDishwasherRack2 / 540.0%
TurnOffStove0 / 50.0%
TurnOnElectricKettle0 / 50.0%
TurnOnMicrowave3 / 560.0%
TurnOnSinkFaucet0 / 50.0%
Composite (seen)
DeliverStraw0 / 50.0%
GetToastedBread0 / 50.0%
KettleBoiling1 / 520.0%
LoadDishwasher0 / 50.0%
PackIdenticalLunches0 / 50.0%
PreSoakPan0 / 50.0%
PrepareCoffee1 / 520.0%
RinseSinkBasin0 / 50.0%
ScrubCuttingBoard1 / 520.0%
SearingMeat0 / 50.0%
SetUpCuttingStation0 / 50.0%
StackBowlsCabinet0 / 50.0%
SteamInMicrowave0 / 50.0%
StirVegetables1 / 520.0%
StoreLeftoversInBowl0 / 50.0%
WashLettuce0 / 50.0%
Composite (unseen)
ArrangeBreadBasket5 / 5100.0%
ArrangeTea1 / 520.0%
BreadSelection2 / 540.0%
CategorizeCondiments3 / 560.0%
CuttingToolSelection0 / 50.0%
GarnishPancake2 / 540.0%
GatherTableware1 / 520.0%
HeatKebabSandwich0 / 50.0%
MakeIceLemonade0 / 50.0%
PanTransfer2 / 540.0%
PortionHotDogs4 / 580.0%
RecycleBottlesByType2 / 540.0%
SeparateFreezerRack0 / 50.0%
WaffleReheat1 / 520.0%
WashFruitColander0 / 50.0%
WeighIngredients0 / 50.0%
Total57 / 25022.8%

RoboCasa TabletopOct 20 – Nov 3

24 tasks × 2 episodes.

Task group (ours)Success / episodesSR
Pick, place and close1 / 128.3%
From cutting board2 / 1020.0%
From placemat1 / 812.5%
From plate2 / 825.0%
From tray0 / 100.0%
Total6 / 4812.5%
TaskSuccess / episodesSR
Pick, place and close
PnPCupToDrawerClose0 / 20.0%
PnPPotatoToMicrowaveClose0 / 20.0%
PnPMilkToMicrowaveClose0 / 20.0%
PnPBottleToCabinetClose0 / 20.0%
PnPWineToCabinetClose1 / 250.0%
PnPCanToDrawerClose0 / 20.0%
From cutting board
PosttrainPnPNovelFromCuttingboardToBasketSplitA0 / 20.0%
PosttrainPnPNovelFromCuttingboardToCardboardboxSplitA1 / 250.0%
PosttrainPnPNovelFromCuttingboardToPanSplitA1 / 250.0%
PosttrainPnPNovelFromCuttingboardToPotSplitA0 / 20.0%
PosttrainPnPNovelFromCuttingboardToTieredbasketSplitA0 / 20.0%
From placemat
PosttrainPnPNovelFromPlacematToBasketSplitA0 / 20.0%
PosttrainPnPNovelFromPlacematToBowlSplitA1 / 250.0%
PosttrainPnPNovelFromPlacematToPlateSplitA0 / 20.0%
PosttrainPnPNovelFromPlacematToTieredshelfSplitA0 / 20.0%
From plate
PosttrainPnPNovelFromPlateToBowlSplitA0 / 20.0%
PosttrainPnPNovelFromPlateToCardboardboxSplitA1 / 250.0%
PosttrainPnPNovelFromPlateToPanSplitA1 / 250.0%
PosttrainPnPNovelFromPlateToPlateSplitA0 / 20.0%
From tray
PosttrainPnPNovelFromTrayToCardboardboxSplitA0 / 20.0%
PosttrainPnPNovelFromTrayToPlateSplitA0 / 20.0%
PosttrainPnPNovelFromTrayToPotSplitA0 / 20.0%
PosttrainPnPNovelFromTrayToTieredbasketSplitA0 / 20.0%
PosttrainPnPNovelFromTrayToTieredshelfSplitA0 / 20.0%
Total6 / 4812.5%

RoboTwinNov 2–4

50 tasks × 5 episodes, in clean scenes.

TaskSuccess / episodesSR
adjust_bottle0 / 50.0%
beat_block_hammer1 / 520.0%
blocks_ranking_rgb1 / 520.0%
blocks_ranking_size2 / 540.0%
click_alarmclock5 / 5100.0%
click_bell5 / 5100.0%
dump_bin_bigbin0 / 50.0%
grab_roller1 / 520.0%
handover_block0 / 50.0%
handover_mic0 / 50.0%
hanging_mug0 / 50.0%
lift_pot0 / 50.0%
move_can_pot0 / 50.0%
move_pillbottle_pad0 / 50.0%
move_playingcard_away1 / 520.0%
move_stapler_pad0 / 50.0%
open_laptop0 / 50.0%
open_microwave0 / 50.0%
pick_diverse_bottles0 / 50.0%
pick_dual_bottles0 / 50.0%
place_a2b_left1 / 520.0%
place_a2b_right0 / 50.0%
place_bread_basket0 / 50.0%
place_bread_skillet0 / 50.0%
place_burger_fries2 / 540.0%
place_can_basket0 / 50.0%
place_cans_plasticbox0 / 50.0%
place_container_plate2 / 540.0%
place_dual_shoes0 / 50.0%
place_empty_cup0 / 50.0%
place_fan0 / 50.0%
place_mouse_pad2 / 540.0%
place_object_basket1 / 520.0%
place_object_scale0 / 50.0%
place_object_stand0 / 50.0%
place_phone_stand0 / 50.0%
place_shoe2 / 540.0%
press_stapler5 / 5100.0%
put_bottles_dustbin0 / 50.0%
put_object_cabinet0 / 50.0%
rotate_qrcode0 / 50.0%
scan_object0 / 50.0%
shake_bottle2 / 540.0%
shake_bottle_horizontally3 / 560.0%
stack_blocks_three0 / 50.0%
stack_blocks_two1 / 520.0%
stack_bowls_three0 / 50.0%
stack_bowls_two2 / 540.0%
stamp_seal0 / 50.0%
turn_switch4 / 580.0%
Total43 / 25017.2%

BEHAVIORNov 3–8

6 household tasks × 5 episodes. Q-score measures goal completion from 0 to 1.

TaskSuccess / episodesSRMean Q
Picking up trash2 / 540.0%0.78
Clean up your desk0 / 50.0%0.32
Store honey0 / 50.0%0.00
Cook a brisket0 / 50.0%0.00
Putting shoes on rack1 / 520.0%0.50
Hanging pictures0 / 50.0%0.00
Total3 / 3010.0%0.27

SIMPLEOct 23–24

12 humanoid whole-body tasks × 5 episodes, at domain-randomization level 0.

TaskSuccess / episodesSR
Tabletop grasp4 / 580.0%
Close door5 / 5100.0%
Open faucet5 / 5100.0%
Open oven4 / 580.0%
Push office chair5 / 5100.0%
Open trash can5 / 5100.0%
Bend and pick0 / 50.0%
Walk and pick between tables0 / 50.0%
Move and pick0 / 50.0%
Move, bend and pick0 / 50.0%
Hand over0 / 50.0%
Pick, place and hug a container0 / 50.0%
Total28 / 6046.7%

VLA-ArenaOct 28–31

170 tasks × 5 episodes, levels L0–L2.

LevelSafetyDistractorExtrapolationLong horizonTotal
L072.0%
90 / 125
72.0%
36 / 50
45.3%
34 / 75
78.0%
39 / 50
66.3%
199 / 300
L164.0%
80 / 125
84.0%
42 / 50
68.0%
51 / 75
72.0%
18 / 25
69.5%
191 / 275
L252.0%
65 / 125
62.0%
31 / 50
66.7%
50 / 75
40.0%
10 / 25
56.7%
156 / 275
Total62.7%
235 / 375
72.7%
109 / 150
60.0%
135 / 225
67.0%
67 / 100
64.2%
546 / 850

LIBERO-PlusOct 26–28

40 tasks × 7 perturbation types × 5 variants, 1,396 episodes in total; four task–lighting combinations have only 4 variants.

PerturbationSuccess / episodesSR
Background texture60 / 20030.0%
Camera viewpoint90 / 20045.0%
Language instruction95 / 20047.5%
Lighting93 / 19647.4%
Object layout100 / 20050.0%
Robot initial state75 / 20037.5%
Sensor noise85 / 20042.5%
Total598 / 1,39642.8%
SuiteSuccess / episodesSR
Spatial149 / 34942.7%
Object219 / 34962.8%
Goal145 / 34941.5%
LIBERO-1085 / 34924.4%
Total598 / 1,39642.8%

DOMINONov 3–6

35 tasks × 5 episodes.

TaskSuccess / episodesSRMean MSMean RC
adjust_bottle5 / 5100.0%100.00100.00
beat_block_hammer1 / 520.0%30.2135.42
click_alarmclock5 / 5100.0%100.00100.00
click_bell5 / 5100.0%100.00100.00
dump_bin_bigbin0 / 50.0%2.312.31
grab_roller5 / 5100.0%100.00100.00
handover_block0 / 50.0%6.116.11
handover_mic0 / 50.0%7.977.97
hanging_mug1 / 520.0%37.4041.59
move_can_pot0 / 50.0%0.000.00
move_pillbottle_pad0 / 50.0%8.9717.94
move_playingcard_away0 / 50.0%7.387.38
move_stapler_pad0 / 50.0%2.985.95
place_a2b_left0 / 50.0%0.000.00
place_a2b_right0 / 50.0%18.6118.61
place_bread_basket0 / 50.0%7.7315.47
place_bread_skillet0 / 50.0%0.000.00
place_can_basket0 / 50.0%0.020.02
place_container_plate0 / 50.0%0.000.00
place_empty_cup0 / 50.0%4.529.04
place_fan0 / 50.0%0.651.29
place_mouse_pad1 / 520.0%20.0520.10
place_object_basket0 / 50.0%28.4628.46
place_object_scale0 / 50.0%2.124.24
place_object_stand0 / 50.0%5.8311.65
place_phone_stand1 / 520.0%30.5938.12
place_shoe0 / 50.0%0.000.00
press_stapler5 / 5100.0%100.00100.00
put_bottles_dustbin0 / 50.0%60.8960.89
put_object_cabinet1 / 520.0%45.0051.65
rotate_qrcode1 / 520.0%56.6961.20
scan_object0 / 50.0%33.1633.16
shake_bottle5 / 5100.0%100.00100.00
shake_bottle_horizontally5 / 5100.0%100.00100.00
stamp_seal0 / 50.0%6.6113.21
Total41 / 17523.4%31.8833.74

VLN-CEOct 28–30

373 of the 1,839 R2R val_unseen episodes: all 11 scenes, one instruction per route, at most 40 routes per scene.

SceneSuccess / routesSRSPL