Gauntlet
An evaluation harness for learned robot policies. It answers one question for VLA / diffusion / scripted policies: how does this policy fail, and has the latest checkpoint regressed against the last one?
Why it exists
Mean success rate hides the structure of failures. A policy that scores 78% overall may fail 100% of the time when cube texture is wood and lighting drops below 0.6. Gauntlet refuses that compression — every report leads with the failure axis combination, not the aggregate.
It is the robotics analogue of pytest + Sentry: pre-deployment regression testing plus structured failure-mode analytics for policies that can’t be debugged by reading code — because there is no code, only weights.
What it does
- Wraps any policy behind a uniform adapter and runs it across parameterized simulator perturbations — lighting, camera pose, textures, clutter, initial conditions — producing reports that break failures down by axis instead of hiding them in an aggregate mean.
- Four backends share action/observation spaces byte-for-byte and the canonical seven perturbation axes. Cross-backend compare requires --allow-cross-backend so simulator drift is never mistaken for policy regression.
- Sampling beyond full Cartesian grids: latin_hypercube and sobol for high-dimensional axes (e.g. 5⁵ = 3,125 cells → ~32 LHS/Sobol samples for ~98× fewer rollouts).
- Runtime drift detection fits an observation autoencoder on a known-good sweep and scores candidate trajectories for OOD rollouts.
- Image observations (render_in_obs / multi-camera CameraSpec) keep obs["image"] and obs["images"][name] contracts so single-view (OpenVLA) and multi-view (SmolVLA, ACT, Diffusion Policy) adapters work by swapping only the env factory.
- ROS 2 integration publishes episode results onto the graph and records live sessions back into Gauntlet’s artefact format for offline analysis.
Simulator backends
MuJoCo
Core tabletop env
PyBullet
Optional extra
Genesis
Optional extra
Isaac Sim
GPU / RTX required
Roadmap shipped
Phase 1 — MVP
- Tabletop MuJoCo environment with seven perturbation axes
- Parallel Runner with full seed control
- Breakdown-first HTML + JSON reports
- CLI: gauntlet run / report / compare
Phase 2 — Real policies & observability
- PyBullet, Genesis, and Isaac Sim backends
- OpenVLA and SmolVLA adapters
- Runtime drift detection (monitor)
- ROS 2 publishing + recording
- Multi-camera observations and structured report diffs
- Plugin system for third-party policies and envs
Phase 3 — Fleet tooling (partial)
- Fleet-wide failure-mode aggregator (gauntlet aggregate)
- Self-contained web dashboard
- Real-to-sim scene-ingestion pipeline
- RealSimRenderer Protocol for gaussian-splatting plugins
Try it
uv sync uv run gauntlet run examples/suites/tabletop-smoke.yaml --policy random --out out/ # open out/report.html
Full design and CLI docs live in the repository.