Predictive Auxiliary Objective for Payload-Adaptive Multi-Motion Humanoid Balancing

Overview

We train a single humanoid policy for balancing motions with unknown hand payloads, using future-observation prediction to learn its history representation.

Our method: payload-adaptive balancing on Unitree H1-2.

Abstract

Achieving versatile humanoid control requires a single policy that can execute many diverse motions while remaining robust to unobserved environment conditions, such as payloads carried in the robot's hands. Each of these requirements is demanding on its own, and combining them compounds the difficulty: the same policy must represent motions with very different dynamics and, for each of them, infer conditions it never observes directly. A common way to achieve such robustness in reinforcement learning is to condition the policy on a long history of observations, encoded by a history encoder trained end-to-end. We show that this history encoder can be trained more effectively with an auxiliary future-observation prediction objective, derived from intention-conditioned flow occupancy models, which treats each motion-payload combination as a latent intention to be inferred from history. The prediction objective is optimized online from a replay buffer of recent trajectories, alongside standard PPO. We evaluate on a challenging benchmark of dynamic whole-body balancing motions with unknown hand payloads on a full-size Unitree H1-2. Our method improves normalized return over A-RMA and long-short-history baselines in simulation, and increases single-leg balancing time under load on hardware.

Architecture

A history encoder infers the motion and payload conditions from past observations. The policy uses this representation to track the reference motion.

The prediction objective trains the history encoder; the reward objective trains the policy. The encoder latent conditions the policy and the future-observation decoder. — Actor architecture. Arrows show gradient flow; red marks the prediction branch.

The prediction objective trains the history encoder; PPO trains the policy.

Hardware experiments

The videos show example runs.

Kung fu - 4 kg right-arm payload

In these example runs, A-RMA and SH lose stability early. LSH balances longer, but for less time than our method and with poorer motion quality, as the slow-motion overlay shows.

Slow-motion overlay with a 4 kg right-arm payload. Ours is shown in green; LSH in red.

Kung fu - no payload

Without payload, all methods show similar balancing performance in these example runs.

Squat - 2 kg per arm

Pistol squat - no payload

Simulation and reference motions

LSH tracks the reference motions less closely, particularly the hand movements.

Kung fu

Squat

Pistol squat

Training and simulation evaluation

Training

All methods train for 500k steps. Our method achieves the highest final training return, with low variance across seeds.

Simulation evaluation

Evaluation uses 20 s episodes with full domain randomization and unknown payloads of 0-6 kg per hand. Our method achieves the highest return on all three motions.

Simulation return under full domain randomization. Mean +/- standard deviation across 3 seeds; higher is better.
Motion	Ours	LSH	A-RMA	SH
Kung fu	562.98 +/- 10.56	492.76 +/- 12.25	514.63 +/- 2.17	470.74 +/- 9.78
Pistol squat	518.59 +/- 14.21	423.49 +/- 8.36	425.02 +/- 2.92	404.61 +/- 28.41
Squat	579.43 +/- 12.76	479.51 +/- 27.94	522.42 +/- 2.23	500.64 +/- 11.01
Normalized mean	1.00 +/- 0.02	0.84 +/- 0.03	0.88 +/- <0.01	0.83 +/- 0.03

Key findings

Prediction learns useful history representations

Surprisingly, prediction learns a history representation better suited to the task than PPO-only training, despite the dense, DeepMimic-style tracking rewards.

The latent is a compact encoding of recent observations and actions. We train the encoder online so that a decoder can use this latent to predict future observations from recent trajectories. This encourages it to retain information about both the motion and the hidden payload conditions. LSH learns its encoding through the policy gradient; A-RMA learns to match a teacher's encoding of privileged dynamics parameters.

Our history encoder has the same architecture as A-RMA's and is larger than the standard LSH encoder: it adds an MLP after the CNN. Giving LSH the same larger encoder and training it with PPO alone reduces performance. The prediction objective makes this additional capacity useful.

The latent captures motion as well as payload

We collect the history encoders' outputs during simulation and project them into two dimensions with t-SNE. Coloring these projections by motion or payload shows what information the learned representations retain: all history-based methods encode payload information, while our method and LSH also show organization by motion. A-RMA's motion representations are more mixed.

t-SNE projections colored by motion, showing motion organization for our method and LSH. — Latent representations along trajectories, projected to 2D with t-SNE and colored by motion.

t-SNE projections colored by payload mass during single-leg balancing. — t-SNE projection of latents for different payloads, for the kung fu motion at the moment of balancing on one leg.

BibTeX

@inproceedings{mstepien2026pao,
  title={Predictive Auxiliary Objective for Payload-Adaptive Multi-Motion Humanoid Balancing},
  author={Stępień, Maciej and Roux, Constant and Stasse, Olivier},
  booktitle={2026 IEEE-RAS 25th International Conference on Humanoid Robots (Humanoids)},
  year={2026}
}