Open Source Robotics

A curated collection of high-quality open source robotics projects, tools, and software to propel the robotics community forward.

NVIDIA Cosmos 3

NVIDIA's open world models that reason, generate video and predict robot actions

License

OpenMDW 1.1

Code and weights: OpenMDW-1.1, read from the LICENSE files of NVIDIA/cosmos and NVIDIA/cosmos-framework and from every model card listed here; GitHub shows NOASSERTION only because it does not recognise this licence. It allows use, changes and redistribution, commercial use included, if the licence text and notices are kept, places no limits on outputs, and ends for anyone who sues claiming the model infringes a patent or copyright. The model cards say the models are ready for commercial and non-commercial use. Guardrail: the quick start for video generation needs nvidia/Cosmos-1.0-Guardrail, which is gated on Hugging Face and under the NVIDIA Open Model License. Dataset: NVIDIA releases its Cosmos3-DROID conversion under OpenMDW-1.1; the original DROID data comes from its own authors, so check their terms too. LeRobot: the LeRobot code for these policies is Apache-2.0. Checked 2026-11-11 against the LICENSE files, the Hugging Face model and dataset cards and openmdw.ai.

Overview

NVIDIA Cosmos 3 is NVIDIA's current family of open world models for physical AI, released on 31 May 2026. One model reads and writes text, images, video, sound and robot actions, so the same weights can describe a scene, generate video of what happens next, simulate the effect of an action, work out which action caused a change, and act as a robot policy. Earlier Cosmos releases split these jobs across separate models such as Cosmos Predict, Cosmos Transfer and Cosmos Reason. There are three base sizes: Cosmos3-Edge with 4B parameters for Jetson and other edge devices, Cosmos3-Nano with 16B for a single large GPU, and Cosmos3-Super with 64B for data-centre GPUs. NVIDIA also publishes post-trained example checkpoints for robot policies, driving and robot simulation and fast image and video generation, the Cosmos Framework for training and serving, and a cookbook of recipes. Code and weights are under OpenMDW-1.1, which allows commercial use; LeRobot 0.7 added both Nano and Edge as robot policies in November 2026.

NVIDIA released Cosmos 3 on 31 May 2026, with weights in a Hugging Face collection, code on GitHub and a technical report (arXiv 2606.02800). The NVIDIA/cosmos repository is the family's home: it lists every model, links the recipes and calls Cosmos 3 the current model family.

Cosmos 3 is a world model: it learns how scenes change over time, so it can predict and generate what happens next. NVIDIA calls it omnimodal because one model takes and produces text, images, video, audio and actions, and builds it as a Mixture-of-Transformers. In plain terms, every layer has two sets of weights that share one attention step. The reasoner reads text and images and writes text one token at a time, like a vision-language model. The generator produces images, video, sound and actions by removing noise step by step, like a video diffusion model. Because both sit in one network, the model can plan in words and then generate the matching video or action.

Three base models are open:

- Cosmos3-Edge, 4B parameters, released on 20 July 2026 for Jetson AGX Orin, Jetson Thor and RTX Pro 6000; it generates 256p and 480p video, reasons about scenes and outputs actions, but has no sound or transfer modes. - Cosmos3-Nano, 16B parameters, for one RTX Pro 6000, H100 or B200; NVIDIA presents it as the base model to post-train. - Cosmos3-Super, 64B parameters, for H200, B200 and GB200 systems, used for the highest quality, synthetic data and as a teacher for distillation.

Together they cover text-to-image, text-to-video and image-to-video with optional sound, video transfer from edge, blur, depth or segmentation maps, prediction of a video's next seconds, visual question answering and grounding, forward dynamics (the video that follows an action), inverse dynamics (the action behind a video) and robot policy. The action side supports camera motion, cars, egocentric hand poses, single-arm and dual-arm robots and the AgiBot humanoid.

The cosmos3-examples collection holds post-trained checkpoints. Cosmos3-Nano-Policy-DROID and Cosmos3-Edge-Policy-DROID are robot policies for the Franka arm of the DROID setup, trained on Cosmos3-DROID, NVIDIA's conversion of DROID to the LeRobot dataset format. Since October 2026 four simulation checkpoints turn Nano into an interactive simulator that generates multi-camera driving video with lidar, a bimanual robot's view as it acts, or photoreal video from a simulator's depth or segmentation output. Super has 4-step text-to-image and image-to-video students, and Nano and Super have FP8 versions.

The Cosmos Framework trains and serves the models: supervised fine-tuning, LoRA, distillation and reinforcement learning, with inference through Diffusers, Transformers, vLLM, vLLM-Omni and SGLang; NVIDIA also lists TensorRT-LLM and NIM. The Cosmos cookbook groups its recipes by goal and by GPU budget. Since LeRobot 0.7 of 5 November 2026, Nano and Edge are LeRobot policy types: the LeRobot guide explains how the policy predicts the next video frames and the next chunk of actions together, and gives fine-tuning recipes for LIBERO, DROID and new robots.

NVIDIA's Alpamayo 2 driving models are built on Cosmos 3 backbones.

Why it matters

Cosmos 3 is the newest generation of the NVIDIA Cosmos models we already list as Cosmos Predict 2.5, a video world model, and Cosmos Policy, which post-trains a Cosmos video model into a robot policy. Where those took separate models for prediction, transfer, reasoning and policy, NVIDIA describes Cosmos 3 as one model that replaces them, so a team can reason, simulate and act with the same weights and post-train one base instead of several. It is also one of the few open foundation models that is at once a video generator and a robot policy: like NVIDIA Isaac GR00T it can be fine-tuned into a VLA-style controller, but it predicts future video together with the actions. With Cosmos3-Edge at 4B and the LeRobot integration, a policy can be fine-tuned on one GPU and NVIDIA reports it running on Jetson Thor, which brings a world-model policy within reach of research labs rather than only data centres.

Problem it solves

Teams building robots and vehicles need models that understand scenes, generate training and test data, simulate what an action will do and control the machine, and until now that meant separate models with separate training code. Training such models from scratch takes video data and compute few groups have. Cosmos 3 releases one open model family that does all of these, in three sizes from edge devices to data-centre GPUs, with the code and recipes to post-train it on a team's own robot, vehicle or camera data.

Key takeaways

  • One omnimodal model reads and writes text, images, video, sound and robot actions
  • Three open base sizes: Cosmos3-Edge 4B, Cosmos3-Nano 16B and Cosmos3-Super 64B
  • Combines jobs earlier Cosmos releases split across Predict, Transfer and Reason models
  • Post-trained examples cover DROID robot policies, driving and robot simulation and fast generation
  • LeRobot 0.7 runs Nano and Edge as robot policies with LIBERO and DROID recipes
  • Code and weights under OpenMDW-1.1, which allows commercial use

Who it is for

  • Robot learning researchers who want a world-model policy or a simulator to post-train
  • Autonomous-driving and robotics teams generating synthetic or multi-camera training data
  • Developers deploying visual reasoning or robot policies on Jetson and other edge devices
  • LeRobot users who want to fine-tune a world action model on their own robot data

Limitations

  • Large GPUs except Edge: Nano post-training recipes ship for 8 GPUs, NVIDIA's recommended Super serving setup uses 8 data-centre GPUs, and full LeRobot fine-tuning of Nano needs at least 4 GPUs with 80 GB
  • On Jetson, Edge is far slower than on a data-centre GPU: NVIDIA measures about 6.3 s per DROID policy step on Jetson AGX Thor against about 1.4 s on an H100
  • The quick-start video generation needs the gated Cosmos-1.0-Guardrail model, which is under the NVIDIA Open Model License rather than OpenMDW-1.1
  • NVIDIA lists artifacts in long, high-resolution or physically complex outputs, such as objects changing shape and implausible motion, and says safety-critical uses need more validation
  • Benchmark rankings, policy success rates and speed figures are NVIDIA's and LeRobot's own measurements
  • The LeRobot integration is new, released with LeRobot 0.7 on 5 November 2026
  • Only BF16 is tested on the model cards; other precisions apart from the published FP8 checkpoints are not officially supported
  • Cosmos3-DROID is NVIDIA's conversion of the DROID dataset, which was collected by 13 institutions and has its own terms upstream

This website requires JavaScript to function properly. Please enable JavaScript to explore our collection of open source robotics projects.