HALO
Apple's humanoid autonomy stack for the Unitree G1, trained only on synthetic home data
License
Apple sample code licence (permissive, no patent grant)
Apple's own sample code licence, not an OSI-approved one. It lets anyone use, change and redistribute the code, in source or binary form, with or without changes; an unmodified copy must keep Apple's notice. It grants no patent rights, and Apple's name and marks may not be used to promote derived work. The models HALO downloads (Qwen, Kimodo and the Llama 3 models Kimodo uses) and the Unitree G1 robot description keep their own licences, and the Llama 3 model is gated on Hugging Face.
Overview
HALO (Perceptive Humanoid Motion from Simulation) is a humanoid autonomy system from Apple's machine learning research group, released with its paper in November 2026. It drives a Unitree G1 from a camera image and a spoken request: it sits on chairs and sofas, walks to people and named objects, sidesteps through half-open doors, crouches and crawls under tables and pushes boxes. Its camera-driven skills were learned from synthetic data rendered in 1,000 procedurally generated homes, and the paper reports a 91% success rate over 360 trials in six real environments the robot had never seen. The repository holds the data pipeline, training code for both learned models, and deployment for MuJoCo and the real robot. It ships no trained weights and no dataset.
HALO runs as three layers on the robot. An orchestrator, an off-the-shelf vision-language model, turns the operator's request and the camera image into subtasks about every two seconds. A motion generator plans five seconds of whole-body motion, 150 frames of 36 values, about three times a second from the RGB image and the current subtask; it pairs a frozen Qwen3.5-2B vision encoder with a diffusion transformer decoder trained by flow matching, and commits only the first half-second of each plan. A tracker follows that plan at 50 Hz using a depth camera and the robot's own joint and IMU readings, and outputs targets for the G1's 29 joints. On Apple's robot the generator and orchestrator run on a Mac mini carried in a rear enclosure, and the tracker on the G1's own Jetson Orin NX.
The data pipeline is where the work is. NVIDIA's Kimodo generates G1 motion from text prompts, waypoints and end poses, and 33 geometric checks keep the best of several candidates. Infinigen builds the homes, each motion is placed where it collides with nothing, and Isaac Lab renders what a fisheye head camera would see, with eight kinds of visual domain randomization such as motion blur, low light and colour tints.
The tracker trains in Isaac Lab in three phases: PPO experts that see the full simulator state, distillation into one student that sees only onboard sensors, then PPO fine-tuning with one critic per expert. It exports to ONNX and deploys in Python for MuJoCo sim-to-sim tests and in real-time C++ with TensorRT on the robot. The system launchers run the whole loop in MuJoCo or on the G1. Apple's project page has the demo video, a gallery of real trials and a recorded run you can replay in 3D, and the method is in the paper.
Why it matters
Most humanoid skill data is collected by people, through teleoperation or motion capture, at roughly an hour of human time per hour of data. HALO shows a different route: a corpus generated entirely in simulation, which grows with compute rather than labour, carrying a G1 through real homes zero-shot. It is also a complete, readable pipeline rather than a single policy. Data generation, generator training, tracker training and real-robot deployment sit in one repository, so a lab with a G1 can study or rebuild every stage. Open humanoid control projects such as ASAP, Holosoma and GEAR-SONIC focus mainly on learning to track or execute motion. HALO pairs a tracker with the camera-driven layer that decides what motion to make in a room it has never seen.
Problem it solves
A humanoid that works in an ordinary home has to turn pixels and a spoken request into joint commands, and learning that mapping needs paired vision, language and motion data that does not exist at scale. Collecting it by teleoperation is slow and expensive. HALO generates that paired data synthetically in procedurally built homes, then trains a planner and a tracker on it that transfer to real rooms without fine-tuning.
Key takeaways
- Three onboard layers: a VLM orchestrator, a motion generator at about 3 Hz and a depth-based tracker at 50 Hz
- Its camera-driven skills are trained on synthetic data from 1,000 procedurally generated homes, with no real homes in training
- 91% success over 360 trials in six real environments not seen in training, as reported in the paper
- Includes the data pipeline, training for both models, and MuJoCo and real-robot deployment for the Unitree G1
- No trained weights or dataset ship: you generate the data and train both models yourself
Who it is for
- Labs with a Unitree G1 working on perceptive locomotion and whole-body skills
- Researchers studying synthetic data and zero-shot transfer from simulation to real robots
- Teams building humanoid motion trackers who want a tested three-stage training recipe in Isaac Lab
- Engineers looking for a worked example of real-time humanoid deployment with TensorRT on Jetson
Limitations
- No trained generator or tracker weights and no dataset are published, so nothing runs until you train both models
- The paper's scale is out of reach for most: the corpus was generated on 250 H100 GPUs and the generator trained on 64 B200 GPUs
- The data pipeline produces training data for the sit-on-chair task only; the other tasks need data that is not in the repository
- The multi-task orchestrator runs on MLX and needs an Apple silicon Mac with at least 32 GB of memory
- Built for one robot, the 29-joint Unitree G1, with a system-identified model of Apple's own unit
- Released under Apple's own permissive licence rather than a standard open-source one, with no patent grant