Figure 5 in the post gives one summary per benchmark. This page breaks results down by task family, suite, difficulty level, perturbation type or scene. All results are from the Native GPT-6-Astra Policy behind starRoboEval.
high and each benchmark's native step limit.| Benchmark | Evaluation scope | Success / episodes | Rate |
|---|
15 tasks × 5 episodes.
| Robot | Success / episodes | SR |
|---|---|---|
| Google Robot | 8 / 55 | 14.5% |
| WidowX (Bridge) | 12 / 20 | 60.0% |
| Total | 20 / 75 | 26.7% |
| Task family | Success / episodes | SR |
|---|---|---|
| Pick coke can | 5 / 15 | 33.3% |
| Move near | 0 / 5 | 0.0% |
| Open drawer | 0 / 15 | 0.0% |
| Close drawer | 2 / 15 | 13.3% |
| Put apple into drawer | 1 / 5 | 20.0% |
| Spoon on towel | 3 / 5 | 60.0% |
| Carrot on plate | 4 / 5 | 80.0% |
| Stack green block on yellow | 3 / 5 | 60.0% |
| Eggplant into basket | 2 / 5 | 40.0% |
4 suites × 10 tasks × 5 initial states.
| Suite | Success / episodes | SR |
|---|---|---|
| Spatial | 9 / 50 | 18.0% |
| Object | 25 / 50 | 50.0% |
| Goal | 22 / 50 | 44.0% |
| Long (LIBERO-10) | 13 / 50 | 26.0% |
| Total | 69 / 200 | 34.5% |
34 tasks × 5 episodes. Each task is scored on its own.
| Task family (ours) | Success / episodes | SR |
|---|---|---|
| Lift block | 41 / 45 | 91.1% |
| Push block | 23 / 30 | 76.7% |
| Rotate block | 20 / 30 | 66.7% |
| Open / close drawer | 10 / 10 | 100.0% |
| Move slider left / right | 10 / 10 | 100.0% |
| Turn LED on / off | 10 / 10 | 100.0% |
| Turn lightbulb on / off | 5 / 10 | 50.0% |
| Place / push into drawer or slider | 14 / 15 | 93.3% |
| Stack / unstack | 7 / 10 | 70.0% |
| Total | 140 / 170 | 82.4% |
50 tasks × 5 episodes.
| Difficulty (Seo et al. 2022) | Success / episodes | SR |
|---|---|---|
| Easy | 102 / 140 | 72.9% |
| Medium | 43 / 55 | 78.2% |
| Hard | 14 / 30 | 46.7% |
| Very hard | 19 / 25 | 76.0% |
| Total | 178 / 250 | 71.2% |
42 tasks × 5 layouts. Scores use the native 0–100 scale.
| Task | Success / episodes | SR | Mean native score |
|---|---|---|---|
| match_and_pick_from_conveyor | 5 / 5 | 100.0% | 100.0 |
| general_pickup | 4 / 5 | 80.0% | 80.0 |
| align_blocks | 4 / 5 | 80.0% | 80.0 |
| stack_blocks | 3 / 5 | 60.0% | 66.0 |
| stack_blocks_by_language | 3 / 5 | 60.0% | 60.0 |
| classify_objects_by_language | 3 / 5 | 60.0% | 64.0 |
| stack_bowls | 3 / 5 | 60.0% | 63.0 |
| swap_T | 2 / 5 | 40.0% | 40.0 |
| cover_blocks | 2 / 5 | 40.0% | 44.0 |
| solve_equation | 2 / 5 | 40.0% | 40.0 |
| arrange_largest_number | 2 / 5 | 40.0% | 54.0 |
| fold_clothes | 2 / 5 | 40.0% | 48.0 |
| pick_from_conveyor_by_image | 1 / 5 | 20.0% | 20.0 |
| play_Xylophone | 1 / 5 | 20.0% | 20.0 |
| put_bottles_into_dustbin | 2 / 5 | 40.0% | 52.0 |
| fasten_screws | 1 / 5 | 20.0% | 44.0 |
| pour_balls_into_vase | 1 / 5 | 20.0% | 20.0 |
| classify_objects | 1 / 5 | 20.0% | 36.0 |
| build_tower | 0 / 5 | 0.0% | 6.0 |
| deposit_coin | 0 / 5 | 0.0% | 4.0 |
| fill_egg_holder | 0 / 5 | 0.0% | 5.0 |
| fill_pen_holder | 0 / 5 | 0.0% | 15.0 |
| hang_mugs | 0 / 5 | 0.0% | 3.0 |
| imitate_sorting_sequence | 0 / 5 | 0.0% | 12.0 |
| insert_key | 0 / 5 | 0.0% | 15.0 |
| insert_tubes | 0 / 5 | 0.0% | 0.0 |
| make_kong | 0 / 5 | 0.0% | 0.0 |
| make_toast | 0 / 5 | 0.0% | 0.0 |
| organize_table | 0 / 5 | 0.0% | 25.0 |
| pack_objects_into_box | 0 / 5 | 0.0% | 9.0 |
| play_stacking_toy | 0 / 5 | 0.0% | 0.0 |
| play_tic_tac_toe | 0 / 5 | 0.0% | 75.0 |
| plug_in_charger | 0 / 5 | 0.0% | 0.0 |
| pour_by_language | 0 / 5 | 0.0% | 0.0 |
| pour_liquid_into_cup | 0 / 5 | 0.0% | 0.0 |
| press_by_number | 0 / 5 | 0.0% | 0.0 |
| push_T | 0 / 5 | 0.0% | 0.0 |
| sort_nesting_dolls_by_size | 0 / 5 | 0.0% | 0.0 |
| store_laptop_and_headphones | 0 / 5 | 0.0% | 4.0 |
| store_tools_in_toolbox | 0 / 5 | 0.0% | 0.0 |
| swap_blocks | 0 / 5 | 0.0% | 0.0 |
| sweep_blocks | 0 / 5 | 0.0% | 0.0 |
| Total | 42 / 210 | 20.0% | 26.4 |
50 tasks × 5 episodes.
| Group | Success / episodes | SR |
|---|---|---|
| Atomic (seen) | 30 / 90 | 33.3% |
| Composite (seen) | 4 / 80 | 5.0% |
| Composite (unseen) | 23 / 80 | 28.8% |
| Total | 57 / 250 | 22.8% |
| Task | Success / episodes | SR |
|---|---|---|
| Atomic (seen) | ||
| CloseBlenderLid | 0 / 5 | 0.0% |
| CloseFridge | 5 / 5 | 100.0% |
| CloseToasterOvenDoor | 2 / 5 | 40.0% |
| CoffeeSetupMug | 0 / 5 | 0.0% |
| NavigateKitchen | 5 / 5 | 100.0% |
| OpenCabinet | 5 / 5 | 100.0% |
| OpenDrawer | 0 / 5 | 0.0% |
| OpenStandMixerHead | 2 / 5 | 40.0% |
| PickPlaceCounterToCabinet | 2 / 5 | 40.0% |
| PickPlaceCounterToStove | 1 / 5 | 20.0% |
| PickPlaceDrawerToCounter | 0 / 5 | 0.0% |
| PickPlaceSinkToCounter | 2 / 5 | 40.0% |
| PickPlaceToasterToCounter | 1 / 5 | 20.0% |
| SlideDishwasherRack | 2 / 5 | 40.0% |
| TurnOffStove | 0 / 5 | 0.0% |
| TurnOnElectricKettle | 0 / 5 | 0.0% |
| TurnOnMicrowave | 3 / 5 | 60.0% |
| TurnOnSinkFaucet | 0 / 5 | 0.0% |
| Composite (seen) | ||
| DeliverStraw | 0 / 5 | 0.0% |
| GetToastedBread | 0 / 5 | 0.0% |
| KettleBoiling | 1 / 5 | 20.0% |
| LoadDishwasher | 0 / 5 | 0.0% |
| PackIdenticalLunches | 0 / 5 | 0.0% |
| PreSoakPan | 0 / 5 | 0.0% |
| PrepareCoffee | 1 / 5 | 20.0% |
| RinseSinkBasin | 0 / 5 | 0.0% |
| ScrubCuttingBoard | 1 / 5 | 20.0% |
| SearingMeat | 0 / 5 | 0.0% |
| SetUpCuttingStation | 0 / 5 | 0.0% |
| StackBowlsCabinet | 0 / 5 | 0.0% |
| SteamInMicrowave | 0 / 5 | 0.0% |
| StirVegetables | 1 / 5 | 20.0% |
| StoreLeftoversInBowl | 0 / 5 | 0.0% |
| WashLettuce | 0 / 5 | 0.0% |
| Composite (unseen) | ||
| ArrangeBreadBasket | 5 / 5 | 100.0% |
| ArrangeTea | 1 / 5 | 20.0% |
| BreadSelection | 2 / 5 | 40.0% |
| CategorizeCondiments | 3 / 5 | 60.0% |
| CuttingToolSelection | 0 / 5 | 0.0% |
| GarnishPancake | 2 / 5 | 40.0% |
| GatherTableware | 1 / 5 | 20.0% |
| HeatKebabSandwich | 0 / 5 | 0.0% |
| MakeIceLemonade | 0 / 5 | 0.0% |
| PanTransfer | 2 / 5 | 40.0% |
| PortionHotDogs | 4 / 5 | 80.0% |
| RecycleBottlesByType | 2 / 5 | 40.0% |
| SeparateFreezerRack | 0 / 5 | 0.0% |
| WaffleReheat | 1 / 5 | 20.0% |
| WashFruitColander | 0 / 5 | 0.0% |
| WeighIngredients | 0 / 5 | 0.0% |
| Total | 57 / 250 | 22.8% |
24 tasks × 2 episodes.
| Task group (ours) | Success / episodes | SR |
|---|---|---|
| Pick, place and close | 1 / 12 | 8.3% |
| From cutting board | 2 / 10 | 20.0% |
| From placemat | 1 / 8 | 12.5% |
| From plate | 2 / 8 | 25.0% |
| From tray | 0 / 10 | 0.0% |
| Total | 6 / 48 | 12.5% |
| Task | Success / episodes | SR |
|---|---|---|
| Pick, place and close | ||
| PnPCupToDrawerClose | 0 / 2 | 0.0% |
| PnPPotatoToMicrowaveClose | 0 / 2 | 0.0% |
| PnPMilkToMicrowaveClose | 0 / 2 | 0.0% |
| PnPBottleToCabinetClose | 0 / 2 | 0.0% |
| PnPWineToCabinetClose | 1 / 2 | 50.0% |
| PnPCanToDrawerClose | 0 / 2 | 0.0% |
| From cutting board | ||
| PosttrainPnPNovelFromCuttingboardToBasketSplitA | 0 / 2 | 0.0% |
| PosttrainPnPNovelFromCuttingboardToCardboardboxSplitA | 1 / 2 | 50.0% |
| PosttrainPnPNovelFromCuttingboardToPanSplitA | 1 / 2 | 50.0% |
| PosttrainPnPNovelFromCuttingboardToPotSplitA | 0 / 2 | 0.0% |
| PosttrainPnPNovelFromCuttingboardToTieredbasketSplitA | 0 / 2 | 0.0% |
| From placemat | ||
| PosttrainPnPNovelFromPlacematToBasketSplitA | 0 / 2 | 0.0% |
| PosttrainPnPNovelFromPlacematToBowlSplitA | 1 / 2 | 50.0% |
| PosttrainPnPNovelFromPlacematToPlateSplitA | 0 / 2 | 0.0% |
| PosttrainPnPNovelFromPlacematToTieredshelfSplitA | 0 / 2 | 0.0% |
| From plate | ||
| PosttrainPnPNovelFromPlateToBowlSplitA | 0 / 2 | 0.0% |
| PosttrainPnPNovelFromPlateToCardboardboxSplitA | 1 / 2 | 50.0% |
| PosttrainPnPNovelFromPlateToPanSplitA | 1 / 2 | 50.0% |
| PosttrainPnPNovelFromPlateToPlateSplitA | 0 / 2 | 0.0% |
| From tray | ||
| PosttrainPnPNovelFromTrayToCardboardboxSplitA | 0 / 2 | 0.0% |
| PosttrainPnPNovelFromTrayToPlateSplitA | 0 / 2 | 0.0% |
| PosttrainPnPNovelFromTrayToPotSplitA | 0 / 2 | 0.0% |
| PosttrainPnPNovelFromTrayToTieredbasketSplitA | 0 / 2 | 0.0% |
| PosttrainPnPNovelFromTrayToTieredshelfSplitA | 0 / 2 | 0.0% |
| Total | 6 / 48 | 12.5% |
50 tasks × 5 episodes, in clean scenes.
| Task | Success / episodes | SR |
|---|---|---|
| adjust_bottle | 0 / 5 | 0.0% |
| beat_block_hammer | 1 / 5 | 20.0% |
| blocks_ranking_rgb | 1 / 5 | 20.0% |
| blocks_ranking_size | 2 / 5 | 40.0% |
| click_alarmclock | 5 / 5 | 100.0% |
| click_bell | 5 / 5 | 100.0% |
| dump_bin_bigbin | 0 / 5 | 0.0% |
| grab_roller | 1 / 5 | 20.0% |
| handover_block | 0 / 5 | 0.0% |
| handover_mic | 0 / 5 | 0.0% |
| hanging_mug | 0 / 5 | 0.0% |
| lift_pot | 0 / 5 | 0.0% |
| move_can_pot | 0 / 5 | 0.0% |
| move_pillbottle_pad | 0 / 5 | 0.0% |
| move_playingcard_away | 1 / 5 | 20.0% |
| move_stapler_pad | 0 / 5 | 0.0% |
| open_laptop | 0 / 5 | 0.0% |
| open_microwave | 0 / 5 | 0.0% |
| pick_diverse_bottles | 0 / 5 | 0.0% |
| pick_dual_bottles | 0 / 5 | 0.0% |
| place_a2b_left | 1 / 5 | 20.0% |
| place_a2b_right | 0 / 5 | 0.0% |
| place_bread_basket | 0 / 5 | 0.0% |
| place_bread_skillet | 0 / 5 | 0.0% |
| place_burger_fries | 2 / 5 | 40.0% |
| place_can_basket | 0 / 5 | 0.0% |
| place_cans_plasticbox | 0 / 5 | 0.0% |
| place_container_plate | 2 / 5 | 40.0% |
| place_dual_shoes | 0 / 5 | 0.0% |
| place_empty_cup | 0 / 5 | 0.0% |
| place_fan | 0 / 5 | 0.0% |
| place_mouse_pad | 2 / 5 | 40.0% |
| place_object_basket | 1 / 5 | 20.0% |
| place_object_scale | 0 / 5 | 0.0% |
| place_object_stand | 0 / 5 | 0.0% |
| place_phone_stand | 0 / 5 | 0.0% |
| place_shoe | 2 / 5 | 40.0% |
| press_stapler | 5 / 5 | 100.0% |
| put_bottles_dustbin | 0 / 5 | 0.0% |
| put_object_cabinet | 0 / 5 | 0.0% |
| rotate_qrcode | 0 / 5 | 0.0% |
| scan_object | 0 / 5 | 0.0% |
| shake_bottle | 2 / 5 | 40.0% |
| shake_bottle_horizontally | 3 / 5 | 60.0% |
| stack_blocks_three | 0 / 5 | 0.0% |
| stack_blocks_two | 1 / 5 | 20.0% |
| stack_bowls_three | 0 / 5 | 0.0% |
| stack_bowls_two | 2 / 5 | 40.0% |
| stamp_seal | 0 / 5 | 0.0% |
| turn_switch | 4 / 5 | 80.0% |
| Total | 43 / 250 | 17.2% |
6 household tasks × 5 episodes. Q-score measures goal completion from 0 to 1.
| Task | Success / episodes | SR | Mean Q |
|---|---|---|---|
| Picking up trash | 2 / 5 | 40.0% | 0.78 |
| Clean up your desk | 0 / 5 | 0.0% | 0.32 |
| Store honey | 0 / 5 | 0.0% | 0.00 |
| Cook a brisket | 0 / 5 | 0.0% | 0.00 |
| Putting shoes on rack | 1 / 5 | 20.0% | 0.50 |
| Hanging pictures | 0 / 5 | 0.0% | 0.00 |
| Total | 3 / 30 | 10.0% | 0.27 |
12 humanoid whole-body tasks × 5 episodes, at domain-randomization level 0.
| Task | Success / episodes | SR |
|---|---|---|
| Tabletop grasp | 4 / 5 | 80.0% |
| Close door | 5 / 5 | 100.0% |
| Open faucet | 5 / 5 | 100.0% |
| Open oven | 4 / 5 | 80.0% |
| Push office chair | 5 / 5 | 100.0% |
| Open trash can | 5 / 5 | 100.0% |
| Bend and pick | 0 / 5 | 0.0% |
| Walk and pick between tables | 0 / 5 | 0.0% |
| Move and pick | 0 / 5 | 0.0% |
| Move, bend and pick | 0 / 5 | 0.0% |
| Hand over | 0 / 5 | 0.0% |
| Pick, place and hug a container | 0 / 5 | 0.0% |
| Total | 28 / 60 | 46.7% |
G1Wholebody<Task>-v0, e.g. G1WholebodyCloseDoorTeleop-v0.170 tasks × 5 episodes, levels L0–L2.
| Level | Safety | Distractor | Extrapolation | Long horizon | Total |
|---|---|---|---|---|---|
| L0 | 72.0% 90 / 125 | 72.0% 36 / 50 | 45.3% 34 / 75 | 78.0% 39 / 50 | 66.3% 199 / 300 |
| L1 | 64.0% 80 / 125 | 84.0% 42 / 50 | 68.0% 51 / 75 | 72.0% 18 / 25 | 69.5% 191 / 275 |
| L2 | 52.0% 65 / 125 | 62.0% 31 / 50 | 66.7% 50 / 75 | 40.0% 10 / 25 | 56.7% 156 / 275 |
| Total | 62.7% 235 / 375 | 72.7% 109 / 150 | 60.0% 135 / 225 | 67.0% 67 / 100 | 64.2% 546 / 850 |
40 tasks × 7 perturbation types × 5 variants, 1,396 episodes in total; four task–lighting combinations have only 4 variants.
| Perturbation | Success / episodes | SR |
|---|---|---|
| Background texture | 60 / 200 | 30.0% |
| Camera viewpoint | 90 / 200 | 45.0% |
| Language instruction | 95 / 200 | 47.5% |
| Lighting | 93 / 196 | 47.4% |
| Object layout | 100 / 200 | 50.0% |
| Robot initial state | 75 / 200 | 37.5% |
| Sensor noise | 85 / 200 | 42.5% |
| Total | 598 / 1,396 | 42.8% |
| Suite | Success / episodes | SR |
|---|---|---|
| Spatial | 149 / 349 | 42.7% |
| Object | 219 / 349 | 62.8% |
| Goal | 145 / 349 | 41.5% |
| LIBERO-10 | 85 / 349 | 24.4% |
| Total | 598 / 1,396 | 42.8% |
35 tasks × 5 episodes.
| Task | Success / episodes | SR | Mean MS | Mean RC |
|---|---|---|---|---|
| adjust_bottle | 5 / 5 | 100.0% | 100.00 | 100.00 |
| beat_block_hammer | 1 / 5 | 20.0% | 30.21 | 35.42 |
| click_alarmclock | 5 / 5 | 100.0% | 100.00 | 100.00 |
| click_bell | 5 / 5 | 100.0% | 100.00 | 100.00 |
| dump_bin_bigbin | 0 / 5 | 0.0% | 2.31 | 2.31 |
| grab_roller | 5 / 5 | 100.0% | 100.00 | 100.00 |
| handover_block | 0 / 5 | 0.0% | 6.11 | 6.11 |
| handover_mic | 0 / 5 | 0.0% | 7.97 | 7.97 |
| hanging_mug | 1 / 5 | 20.0% | 37.40 | 41.59 |
| move_can_pot | 0 / 5 | 0.0% | 0.00 | 0.00 |
| move_pillbottle_pad | 0 / 5 | 0.0% | 8.97 | 17.94 |
| move_playingcard_away | 0 / 5 | 0.0% | 7.38 | 7.38 |
| move_stapler_pad | 0 / 5 | 0.0% | 2.98 | 5.95 |
| place_a2b_left | 0 / 5 | 0.0% | 0.00 | 0.00 |
| place_a2b_right | 0 / 5 | 0.0% | 18.61 | 18.61 |
| place_bread_basket | 0 / 5 | 0.0% | 7.73 | 15.47 |
| place_bread_skillet | 0 / 5 | 0.0% | 0.00 | 0.00 |
| place_can_basket | 0 / 5 | 0.0% | 0.02 | 0.02 |
| place_container_plate | 0 / 5 | 0.0% | 0.00 | 0.00 |
| place_empty_cup | 0 / 5 | 0.0% | 4.52 | 9.04 |
| place_fan | 0 / 5 | 0.0% | 0.65 | 1.29 |
| place_mouse_pad | 1 / 5 | 20.0% | 20.05 | 20.10 |
| place_object_basket | 0 / 5 | 0.0% | 28.46 | 28.46 |
| place_object_scale | 0 / 5 | 0.0% | 2.12 | 4.24 |
| place_object_stand | 0 / 5 | 0.0% | 5.83 | 11.65 |
| place_phone_stand | 1 / 5 | 20.0% | 30.59 | 38.12 |
| place_shoe | 0 / 5 | 0.0% | 0.00 | 0.00 |
| press_stapler | 5 / 5 | 100.0% | 100.00 | 100.00 |
| put_bottles_dustbin | 0 / 5 | 0.0% | 60.89 | 60.89 |
| put_object_cabinet | 1 / 5 | 20.0% | 45.00 | 51.65 |
| rotate_qrcode | 1 / 5 | 20.0% | 56.69 | 61.20 |
| scan_object | 0 / 5 | 0.0% | 33.16 | 33.16 |
| shake_bottle | 5 / 5 | 100.0% | 100.00 | 100.00 |
| shake_bottle_horizontally | 5 / 5 | 100.0% | 100.00 | 100.00 |
| stamp_seal | 0 / 5 | 0.0% | 6.61 | 13.21 |
| Total | 41 / 175 | 23.4% | 31.88 | 33.74 |
373 of the 1,839 R2R val_unseen episodes: all 11 scenes, one instruction per route, at most 40 routes per scene.
| Scene | Success / routes | SR | SPL |
|---|