What Does GPT-6-Astra Score When Nothing Is Tailored to a Benchmark?A Direct Test Across 14 Embodied Benchmarks

Figure 1. One recording from each of the 14 benchmarks, all from the Native GPT-6-Astra Policy. Drag to browse; click a tile to replay it with its decisions.
ABSTRACT

Frontier models such as GPT-6-Astra have shown that they can control robots, and many groups now use simulated benchmarks to explore how far they can go. But unless the evaluation setup rules it out, a score on an embodied benchmark can blend what the model can do, what the harness around it does, and what an agent could learn from simulator state within its reach. In this post, we set out to measure the model more directly. First, we built starRoboEval, which runs each benchmark as a separate service, and for every result we report, we held the agent to camera images, robot state and actions, with nothing else to work with. Second, we set up the Native GPT-6-Astra Policy, which uses one generic setup and one prompt with nothing tailored to any benchmark. Finally, we evaluate this policy broadly on 14 embodied benchmarks, from tabletop manipulation and household tasks to humanoids and navigation. It scores well below the published reference result on 11 of the 12 benchmarks that have one, for example 34.5% against 99.9% on LIBERO and 20.0% against 68.9% on RoboDojo; navigation on VLN-CE is the exception. The gap shows how far the model starts from those numbers when nothing is tailored to a benchmark; together, the results give a snapshot of GPT-6-Astra on its own across all 14 embodied benchmarks.

1Benchmarks as services

In most evaluation setups, the policy and the simulator run side by side, often in the same process. That is convenient, but it puts the agent right next to everything that defines the task: the scene layout, where the objects are, how success is scored. We pull the two apart with starRoboEval: each benchmark runs as its own service, on its own machine, isolated from the agent.

Each turn, the agent sends a short sequence of actions; the service checks them, runs them in the simulator and returns the next camera images and robot state. That is the whole interface. For every result in this post, we held the agent to it: nothing else from the benchmark reached the agent.

The interface between the agent and the Bench Server carries only observations and actions; it has no call for reading simulator files. AGENT SIDE agent host Agentic policy GPT-6-Astra direct control Receives images and robot state Chooses the next few actions Plans, takes notes and computes in its own workspace Not available Web search and web access Simulator files and assets Scores and success signals Other episodes NETWORK BOUNDARY BENCH SERVER benchmark host Service API Declares observations and actions Starts each episode exactly once Validates each action Runs it in the simulator Scores with its own evaluator Server-only Scene layouts and reset seeds Object poses and categories Reward and success Simulator assets and source observations robot_act read simulator files RGB + state · step 0 numeric actions cat layout.json
Red: a request to read simulator files. The interface has no call for it.
Figure 2. The boundary between agent and benchmark. Blue is the normal loop of images out and actions back; red is an attempt to read the simulator's files, for which the interface has no call.
✓The agent receives
  • The task instruction
  • Camera RGB images
  • Robot state (joints, end-effector pose)
  • The action space: dimensions, units, frames
  • Which of its actions were executed
✕The agent never receives
  • Ground-truth object poses or contacts
  • The scene layout or reset seed
  • Reward, success, or how scoring works
  • Simulator files and other episodes
  • Any built-in skill or fallback policy

Running benchmarks as services brings isolation: the interface connects the agent to the task through what it sees and the actions it sends, not through the simulator itself. It also lets us evaluate broadly: any benchmark that speaks this small interface becomes one more service, so the same agent can be run on many of them in exactly the same way. Because the interface only carries what a real robot could sense and do, the same interface can later connect physical hardware too.

Why isolation matters

Early on, before we isolated the simulator, we watched an agent do something sensible and troubling. Before its first move, it opened the simulator's layout file, counted the objects in the scene and noted roughly where each one was. Then it planned. Nothing in its instructions said not to. It was simply using what was within reach.

That is the problem with running the simulator next to the agent. The scene configuration, the assets, even the scoring code sit in the same workspace as the agent. An agent that reads them may still finish the task, but the score no longer says much about whether it can see and act. So rather than asking agents to look away, we think evaluations should take the shortcut off the table: keep the simulator isolated from the agent, so that a score can only come from what the agent sees and does, as on a real robot. That is what starRoboEval is built for.

Before looking at a single frame, the agent knows there are three cubes and one triangular prism, and where on the table each can be.
The workspace holds only public observations. How many objects there are and where they sit must come from the images.
Figure 3. The same agent with and without isolation. Reconstructed from a real RoboDojo layout file, not a raw log.

Why diversity matters

In our runs, isolation kept the simulator out of reach. Isolation alone does not stop an agent from fitting one test. A single benchmark covers one slice of robotics, with its own robots, scenes and tasks, and an agent can be tuned until it does well on that slice.

Robot agents are increasingly built to improve themselves. In a recursive self-improvement (RSI) loop, the agent studies its failures, rewrites its own prompts, tools and skills, and tries again, often with a person in the loop who decides which failures matter and which fixes to keep. Pointed at one benchmark, such a loop learns that benchmark: its scenes, its tasks and where its scorer is strict. The agent does much of this on its own, since every fix it writes answers a failure it saw there; the people steering it add what they notice. These adaptations become part of the agent method being evaluated.

Diversity makes this kind of fitting visible. Robots, scenes and tasks change from one benchmark to the next, so an adaptation that pays off on one slice stands out against the rest. An agent run unchanged across many benchmarks cannot quietly lean toward one of them: what it gains on one slice has to hold up on the others. Because such a loop fits whatever it is pointed at, we want to point it at as many benchmarks as we can. Services make this cheap: each new benchmark is just one more service, so we plan to keep adding them.

Figure 4. starRoboEval sits between policies and benchmarks. Lit items appear in this post; dimmed items mark planned extensions. Future real-robot adapters will connect hardware through the same observation-and-action interface.

2A Native GPT-6-Astra Policy

With the simulator isolated, we can ask the question in the title: before any self-improvement loop changes the agent, what does GPT-6-Astra do when nothing has been tailored to the benchmark in front of it? The Native GPT-6-Astra Policy is our attempt to measure that: one model, one short prompt and one setup, carried unchanged across all 14 benchmarks.

At a glance
The model seesthe task instruction, camera RGB and robot state. Nothing else.
The model outputsthe benchmark's own numeric actions, a few at a time, chosen by the model.
No specific harnessone generic setup for every benchmark, never tuned to its failure modes; no demonstrations and no task-specific prompts.
Whyit is the baseline: what a general model does before anything is built for a particular benchmark.
Prompt excerpt (verbatim)
You are the autonomous direct-control policy for one robot episode. Choose
native numeric actions from the supplied task instruction, current public RGB,
declared robot state and acknowledged execution. There is no fallback policy,
task-solving service, learned action proposal or hypothetical simulator.

Inspect action_spec and state_names before choosing actions. Their dimensions,
units, frame, rotation and gripper conventions belong to the selected service.
Do not assume a robot, number of arms, camera names, frequency, coordinate offset
or action meaning from a familiar benchmark. Missing state remains unknown.

For continued control, call robot_act with the current request_id, actions as
an ordered list of native vectors, a concise public reason, and stop_reason=null.
Each row requests one control. Choose between 1 and max_chunk_length rows;
repetitions and replan timing are your choices within that ceiling.

Compare the next RGB and declared robot state with the effect you expected.
Revise ineffective actions using public evidence; no scripted recovery is run.

Use only the actor-owned workspace and supplied public evidence. Do not connect
to a simulator, request resets or hidden observations, inspect object truth,
scores, layouts/seeds, native assets, evaluator files or other episodes.

The model keeps one running conversation per episode and looks at fresh images before each move. The prompt never changes during a run. In our runs, the agent had web search turned off, no network access from its shell and no MCP servers. Each benchmark's own evaluator determines success within the declared execution budget.

3How far untailored gets

We evaluate the Native GPT-6-Astra Policy on 14 benchmarks that span tabletop and bimanual manipulation, kitchens and households, humanoids, indoor navigation, long-horizon and safety tasks, perturbed scenes and moving targets.

Blue bars are the Native GPT-6-Astra Policy. Gray bars show the published reference result on the same benchmark: the best result we found from a frontier model, leaving out results in which a VLA or world-action model produces the actions.

Comparison data in preparation.
Figure 5.

The Native GPT-6-Astra Policy scores well below the reference result on 11 of the 12 benchmarks that have one. It reaches 34.5% on LIBERO against 99.9%, 20.0% on RoboDojo against 68.9%, and 42.8% on LIBERO-Plus, where every scene is perturbed, against 92.4%. It does better on CALVIN, 82.4% against 99.6%, and MetaWorld, 71.2% against 96.0%; on RoboCasa Tabletop both are low, 12.5% against 19.0%. The baseline struggles most on long tasks that need careful contact: bimanual sorting in RoboDojo, household chores in BEHAVIOR and humanoid tabletop work. On VLN-CE it reached the goal on 314 of 373 routes across the 11 unseen scenes (84.2%, SPL 0.72), against 83.0% (SPL 0.71) for the reference result, which used a 100-episode subset of the same split.

Many reference results come from setups built around their benchmark, with demonstrations, task-specific prompts or skills, larger step budgets or a subset of the tasks; hovering over a gray bar shows each one. That is the tailoring the Native GPT-6-Astra Policy leaves out, so the gap shows how far the model starts from these numbers rather than a like-for-like ranking.

4How GPT-6-Astra acts

Figure 5 gives one number per benchmark. Figure 6 shows what lies behind it: one episode from each benchmark, with every decision the model made, the reason it stated and the actions it sent.

The model's stated reason (verbatim)
Figure 6. We gave it no grasp signals, rewards or success. Click a number or drag the bar to jump.

5What the runs cost

We share what our runs used, as a reference for anyone planning their own evaluation. Figure 7 shows the tokens, estimated API cost and wall-clock time, per episode and scaled up to every task with five episodes each.

Cost data in preparation.
Figure 7.

6Evaluate your agent

If you have an agent, with or without a harness, you can run it against the same services. Your agent decides how it plans, which models it uses and what it remembers; the benchmark decides what happens when an action runs. You do not need a copy of the simulator.

  1. Start a benchmark service. Install the benchmark's simulator and assets following its setup guide, then launch its server with a fixed configuration, seed and output directory.
    rsi-bench list
    rsi-bench serve --benchmark simpler_env \
      --parameters /abs/benchmark-parameters.json \
      --seed 7 --episodes 1 --port 8765 \
      --output-dir /abs/new-run/server
  2. Connect your driver. Call describe to get the benchmark's observation and action format, open an episode and reset it once, then send actions with step and read the reply after each one until the episode ends. The repository includes a single-case driver template you can copy.
  3. Give your agent only what the protocol allows. The model sees the observations and sends robot actions; server credentials stay in your driver.
  4. Read the results. Every closed episode produces a result record with outcome, counters and errors. Interaction history and videos are recorded by default and can be turned off.
    python -B -m benchkit.reports.basic_result \
      --run-dir /abs/closed-run/client \
      --output-dir /abs/report/my-run

Results are reported per benchmark and per method. An agent with a harness, extra models or a tuned prompt shows up as its own entry, next to the Native GPT-6-Astra Policy.

What we take away

Untailored, GPT-6-Astra starts far below the published numbers. It scores well below the reference result on 11 of the 12 benchmarks that have one, most of all on long tasks that need careful contact, and matches it only on VLN-CE navigation. That distance is what a harness, demonstrations or a self-improvement loop would have to make up.

A baseline to build on. The Native GPT-6-Astra Policy ran the same way on all 14 benchmarks, with the simulator isolated and nothing tailored. A method built on top of the model, whether a harness, memory or a self-improvement loop, can run against the same services and be compared with it benchmark by benchmark.

Like-for-like numbers become cheap. Published results today come from different setups, so a fair comparison usually means rerunning everything. With benchmarks running as services, the next model or method is one more run under the same conditions, and each new benchmark is one more service.

Limitations

Environment access. The simulator, its assets and the scoring code run on a different host. In our runs, nothing from the benchmark side reached the agent except images, robot state and replies to its actions, and the only way the agent could affect the world was by sending actions.

Some paths lie outside the service. An agent's own setup can look things up on the open web, including a benchmark's public source code, and a user can hand it extra information. The benchmark side cannot prevent this. These are choices of the method, so they should be declared with its scores. In our runs, we turned off the agent's web search and its shell had no network access; its workspace held only its own notes and the observations it received.

Citation

If you use starRoboEval or its evaluation results, please cite this report:

@misc{starvla2026starroboeval,
  author = {{StarVLA Community}},
  title = {What Does {GPT-6-Astra} Score When Nothing Is Tailored to a Benchmark?},
  year = {2026},
  url = {https://starvla.github.io/blogs/starRoboEval/}
}

Download BibTeX · Project code

References

  1. ManiAgent: An Agentic Framework for General Robotic Manipulation. Yi Yang et al. arxiv.org/abs/2510.11660.
  2. Teach and Grow: An Agent-Centered Architecture for General Robot Learning. Chang Nie, Zhe Liu, and Hesheng Wang. arxiv.org/abs/2608.17209.
  3. Robotic Long-Horizon Manipulation with Progressive In-Context Code Generation and Episodic Feedback. Yuan Meng et al. arxiv.org/abs/2503.21969.
  4. Demonstration-Free Robotic Control via LLM Agents. Brian Y. Tsui, Alan Y. Fang, and Tiffany J. Hwu. arxiv.org/abs/2601.20334.
  5. RoboDojo Sim Leaderboard (agent track). Qoo Tech, Q0-Preview entry, accessed 2026-11-12. robodojo-benchmark.com/leaderboard.
  6. Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens. Ruiyang Si et al. arxiv.org/abs/2610.01939.
  7. Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning. Youwei Zhou et al. arxiv.org/abs/2610.28259.
  8. Has GPT-6-Astra Achieved Embodied Intelligence? SenseNova et al. arxiv.org/abs/2610.22471.
  9. Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation. Jian Zhou et al. arxiv.org/abs/2607.26148.
  10. RoboHarness: A Simple Harness Outperforms VLA and World Action Models. Bince Qu, Wei Chen, and Bo Zhang. arxiv.org/abs/2610.14018.

starRoboEval derives from StarVLA and keeps its MIT license.