How LeRobot Splits Humanoid Planning From Whole-Body Control
The Unitree G1 workflow lets a vision-language-action policy choose motion tokens while a faster onboard controller handles balance and joint targets.
Edited by Tyronne Panaino
A September 25 LeRobot team article documents an open workflow for training and running whole-body policies on Unitree's G1 humanoid. Instead of asking one vision-language-action model to control every joint quickly enough to keep a biped balanced, the design separates task-level prediction from fast physical execution. The learned policy produces compact SONIC motion tokens; a lower-level controller converts them into feasible joint targets.
That division matters because humanoid control has two different timing problems. A policy needs enough context to interpret language, camera observations and robot state, while a standing robot must react continuously to balance and disturbances. LeRobot's new material turns that architectural distinction into a documented path spanning teleoperation, datasets, policy training, simulation and physical deployment.
Two control loops do different jobs
The LeRobot team article explains why a biped cannot rely on comparatively slow vision-language-action inference for every movement. The task policy consumes language, cameras and robot state, then predicts chunks of SONIC latent motion tokens. SONIC's decoder translates those tokens into joint targets that the robot's low-level controller tracks.
In practical terms, the task model decides what kind of motion should happen, while the faster controller handles how that motion stays dynamically executable. This avoids asking a task-specific dataset to teach balance and whole-body coordination from scratch. It also creates a clearer boundary for testing: failures can arise in task interpretation, token prediction, decoding, networking or the physical controller, and those layers should not be evaluated as one opaque system.
Demonstrations become latent actions
LeRobot's example begins with whole-body teleoperation. An operator supplies a kinematic reference through a virtual-reality headset, including tracked head and wrist motion and either ankle poses or navigation commands. The system converts that reference into a body-motion representation and passes it through SONIC's encoder, producing the latent action stream used for imitation learning.
For the can-moving demonstration described by the team, the dataset contains roughly 100 episodes recorded at 50 frames per second, totaling about 71 minutes of teleoperation. The team then fine-tuned its base policy for 12,000 steps on four H100 GPUs. Those details make the example inspectable, but they do not establish a general data or compute requirement for other tasks. A different object, environment, camera layout or controller could change the result substantially.
The robot-side controller stays close to the hardware
The accompanying Unitree G1 documentation says LeRobot now supports both 29- and 23-degree-of-freedom G1 variants for teleoperation, training and simulation. Its SONIC integration maps a 64-dimensional motion token, together with recent proprioceptive history, into residual actions and 50-hertz joint-position targets for the 29-degree-of-freedom setup.
For physical deployment, the documentation recommends asynchronous inference rather than putting a remote model call directly inside the balance loop. The policy runs on a GPU machine, while a robot client and SONIC controller remain on the G1's Jetson computer; compact token chunks cross the network instead of every low-level control decision. The stated reason is operational: a network delay in a synchronous loop could stall the controller while the robot is standing.
That arrangement is not a guarantee against falls or unsafe motion. It is an engineering boundary that keeps latency-sensitive control near the robot while allowing a larger learned policy to run elsewhere. Anyone reproducing the setup still needs task-specific safety limits, controlled test conditions and hardware-aware supervision beyond what a software integration alone can prove.
What the evidence does and does not show
The team reports a three-seed simulation experiment in which a released depth policy dodged 79.1% of resolved throws. The same article explicitly identifies that as a simulation result rather than a measured hardware success rate. A physical demonstration clip is also not a controlled evaluation of robustness, camera conditions or checkpoint selection.
The useful lesson is therefore architectural, not a headline performance claim. LeRobot is showing how open hardware modifications, demonstrations, learned policies and whole-body controllers can fit into one inspectable workflow. The material does not prove that the stack is safe, production-ready or transferable to every humanoid platform, and it does not independently validate the project's own measurements.
Status
Learning. The LeRobot team article and first-party Hugging Face documentation support the described G1 workflow and its software boundaries. Internal confidence is medium because both sources come from the project ecosystem and the cited performance evidence is limited to a small simulation evaluation and demonstrations rather than independent hardware testing.
Sources
- Hugging Face community article — Bringing Humanoids to LeRobot
- Hugging Face LeRobot documentation — Unitree G1
Update note: Last reviewed 2026-09-25. We will revise this post if the project publishes controlled hardware results, changes the G1 interface or updates the deployment guidance.
Sources
Drafted with AI assistance from source briefs; reviewed for citation completeness and label accuracy.