FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control

Authors: Donghu Kim, Youngdo Lee, Minho Park, Kinam Kim, I Made Aswin Nahendra, Takuma Seno, Sehee Min, Daniel Palenicek, Florian Vogt, Danica Kragic, Jan Peters, Jaegul Choo, Hojoon Lee · RSS 2026

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

FlashSAC stabilises Soft Actor-Critic with large networks, large batches, few updates and bounded norms, beating PPO on high-dimensional tasks and cutting G1 sim-to-real training to minutes.

Why it matters

It shows off-policy RL can be used for sim-to-real humanoid control with one hyperparameter configuration across tasks, and that it scales better than PPO as state and action dimensions grow. Shorter training times make it practical to iterate on high-dimensional robots and slower, more realistic simulators.

Problem

PPO dominates sim-to-real RL because it is stable, but its narrow on-policy data makes value estimation hard in high-dimensional tasks such as dexterous hands and humanoids. Off-policy methods reuse broader data from a replay buffer, but fitting the critic needs many gradient updates, which is slow and lets bootstrapping errors compound, especially with larger networks.

Proposed solution

FlashSAC builds on Soft Actor-Critic and follows supervised-learning scaling trends: 1024 parallel environments, a replay buffer of up to 10M transitions, a 2.5M-parameter 6-layer network with batch size 2048, and only 2 gradient updates per 1024 new transitions. Stability comes from bounding weight, feature and gradient norms: inverted residual blocks, pre-activation batch normalization, a final RMSNorm, a distributional critic with adaptive reward scaling and weight projection onto the unit sphere. Exploration uses one entropy target set by a fixed action standard deviation (0.15) across all robots plus a cheap 'noise repetition' scheme for temporally correlated noise.

Key insights

In off-policy RL, larger models with larger batches and far fewer gradient updates converge faster in wall-clock time, provided the critic is stabilised
Bounding weight, feature and gradient norms keeps the critic's loss landscape well conditioned; each added component lowered the measured condition number
Replay-buffer data covers a much broader state-action region than on-policy rollouts, which the authors link to PPO's weaker results on high-dimensional tasks
A single entropy target defined by action standard deviation (best range 0.15-0.2) works across embodiments without per-task tuning
Repeating sampled action noise over several steps gives coherent exploration in massively parallel simulation at almost no memory cost

Main results

Unitree G1 (29-DoF) flat-ground walking on hardware after about 20 minutes of training, versus about 3 hours for PPO
Unitree G1 climbed real 15 cm stairs (dimensions unseen in training) after about 4 hours of training, versus nearly 20 hours for PPO
On 10 high-dimensional GPU-simulator tasks (Allegro, Shadow Hand, G1, H1, Booster T1), FlashSAC reached higher returns with less wall-clock time than PPO, even though PPO ran 200M steps (about 3x the compute) against FlashSAC's 50M; on 15 low-dimensional tasks it slightly outperformed PPO
On 40 single-environment CPU tasks (MuJoCo, DeepMind Control, MyoSuite, HumanoidBench) it outperformed PPO, XQC, SimbaV2, TD-MPC2 and MR.Q on representative tasks with one configuration
On 8 vision-based DeepMind Control tasks it matched or exceeded DrQ-v2 and MR.Q in final performance while converging faster in wall-clock time
Ablations: replay buffers up to 10M transitions improved results while 50M slowed learning; mixed precision cut wall-clock time by 5-10%

Limitations

The authors note the work focuses on state-based control and leave extending the critic-stabilisation principles to tactile-based learning to future work (no dedicated limitations section)
On low-dimensional tasks the gain over PPO is small: on-policy methods remain effective when state-action spaces are small and simulation is cheap
Very large replay buffers (50M) slowed learning because recent high-quality samples are drawn less often
Sim-to-real was shown on one robot (Unitree G1 blind locomotion), and FlashSAC and PPO used different reward weights there, including a termination penalty only PPO needed
Algorithm update time was profiled on one representative environment per setting and reused across all environments, so wall-clock totals are estimates

Field impact

Winner of the RSS 2026 Outstanding Paper Award; 17 citations so far. The authors provide a project webpage; the paper itself does not give a code link.

Abstract

Reinforcement learning (RL) is a core approach for robot control when expert demonstrations are unavailable. On-policy methods such as Proximal Policy Optimization (PPO) are widely used for their stability, but their reliance on narrowly distributed on-policy data limits accurate policy evaluation in high-dimensional state and action spaces. Off-policy methods can overcome this limitation by learning from a broader state-action distribution, yet suffer from slow convergence and instability, as fitting a value function over diverse data requires many gradient updates, causing critic errors to accumulate through bootstrapping. We present FlashSAC, a fast and stable off-policy RL algorithm built on Soft Actor-Critic. Motivated by scaling laws observed in supervised learning, FlashSAC sharply reduces gradient updates while compensating with larger models and higher data throughput. To maintain stability at increased scale, FlashSAC explicitly bounds weight, feature, and gradient norms, curbing critic error accumulation. Across over 60 tasks in 10 simulators, FlashSAC consistently outperforms PPO and strong off-policy baselines in both final performance and training efficiency, with the largest gains on high-dimensional tasks such as dexterous manipulation. In sim-to-real humanoid locomotion, FlashSAC reduces training time from hours to minutes, demonstrating the promise of off-policy RL for sim-to-real transfer.

Open Source Robotics