SimplerEnv Evaluation

SimplerEnv is a ManiSkill-based simulation environment using the WidowX robotic arm for tabletop manipulation tasks (grasping, placing, drawer operations, etc.). It is widely used to evaluate VLA models trained on the Open X-Embodiment (OXE) dataset.

This document provides instructions for reproducing our experimental results with SimplerEnv.

The evaluation process consists of two main parts:

1. Setting up the simpler_env environment and dependencies.
2. Running the evaluation by launching services in both starVLA and simpler_env environments.

We have verified that this workflow runs successfully on both NVIDIA A100 and RTX 4090 GPUs.

---

SimplerEnv Evaluation

1. Environment Setup

To set up the environment, please first follow the official SimplerEnv repository to install the base simpler_env environment.

Afterwards, inside the simpler_env environment, install the following dependencies:

conda activate simpler_env
pip install tyro matplotlib mediapy websockets msgpack
pip install numpy==1.24.4  # Downgrade numpy for compatibility with the simulation environment

Common Issues: When testing SimplerEnv on NVIDIA A100, you may encounter the following error: libvulkan.so.1: cannot open shared object file: No such file or directory You can refer to this link to fix: Installation Guide – Vulkan Section

Verification

We provide a minimal environment verification script:

python examples/SimplerEnv/test_your_simplerEnv.py

If you see the ”✅ Env built successfully” message, it means SimplerEnv is installed correctly and ready to use.

---

2. Evaluation Workflow

Run the evaluation from the starVLA repository root using two separate terminals, one for each environment.

Why two terminals?

Model inference (starVLA environment) and the simulation (simpler_env environment) depend on different Python package versions that would conflict if installed in the same conda environment. Running them in separate terminals with separate conda environments avoids this.

• starVLA environment: runs the policy inference server.
• simpler_env environment: runs the simulation eval code.

Step 0. Download Checkpoint

Download the checkpoint: Qwen3VL-GR00T-Bridge-RT-1

Step 1. Start the server (starVLA environment)

In the first terminal, activate the starVLA conda environment and run:

bash examples/SimplerEnv/eval_files/run_policy_server.sh

Note: Open examples/SimplerEnv/eval_files/run_policy_server.sh, find the your_ckpt variable, and set it to your actual checkpoint path, e.g. results/Checkpoints/Qwen3VL-GR00T-Bridge-RT-1/checkpoints/steps_50000_pytorch_model.pt.

---

Step 2. Start the simulation (simpler_env environment)

In the second terminal, activate the simpler_env conda environment and run:

export MODEL_PATH=.../checkpoints/steps_50000_pytorch_model.pt
bash examples/SimplerEnv/start_simpler_env.sh ${MODEL_PATH}

This script will automatically launch the WidowX Robot evaluation tasks, reproducing the benchmark results reported above.

Note: Open examples/SimplerEnv/start_simpler_env.sh, find the SimplerEnv_PATH variable, and set it to your SimplerEnv repo clone path (e.g. /path/to/SimplerEnv).

Common Issues: If you encounter NotImplementedError: Framework QwenGR00T is not implemented when running the policy server, this usually means the Framework hasn’t been properly registered in Python’s import path. Run the smoke test first to trigger correct registration:

python starVLA/model/framework/QwenGR00T.py

If the smoke test passes, restart the policy server.

---

Training on OXE

Data Preparation

Steps:

1. Download a LeRobot-format OXE dataset:

   • bridge_orig_lerobot
   • fractal20220817_data_lerobot

2. Include modality.json in each *lerobot/meta/modality.json:

   • bridge modality - Rename as modality.json and put it as bridge_orig_lerobot/meta/modality.json
   • fractal modality - Rename as modality.json and put it as fractal20220817_data_lerobot/meta/modality.json

3. Add your dataset path to config.yaml:

   datasets:
     vla_data:
       dataset_py: lerobot_datasets
       data_root_dir: playground/Datasets/OXE_LEROBOT_DATASET
       data_mix: bridge_rt_1

Check Your Dataloader

We provide a simple way to check your dataloader. Make sure you can load batched data:

python starVLA/dataloader/lerobot_datasets.py --config_yaml examples/SimplerEnv/train_files/starvla_cotrain_oxe.yaml

Framework Preparation

Before running, you need to ensure that your framework can forward and predict_action using a fake data example.

Try the following command:

python starVLA/model/framework/QwenGR00T.py --config_yaml examples/SimplerEnv/train_files/starvla_cotrain_oxe.yaml

Start Training

Once everything is ready, use our provided script to start training:

bash ./examples/SimplerEnv/train_files/run_oxe_train.sh

Note: Ensure that the script explicitly uses the validated config path. If not already passed, add the --config_yaml argument.