Contact-Mode Critics for Learning Loco-Manipulation from Diverse Demonstrations

Legged robots that push, lift, and roll objects can be trained with the most honest reward available: a penalty for every second the task is unfinished, and nothing else. This is called a sparse reward, because the learner gets almost no feedback until it succeeds. The approach works when the reinforcement learner is seeded with demonstrations from a planner. It fails in a specific way when those demonstrations disagree about how to do the task. For example, one demonstration pushes a box with the arm while another leans into it with the shoulder. The learner then blends the two strategies into an uncoordinated motion that does neither. Today the only fix is to throw away all but one strategy before training. That discards exactly the behavioral diversity a whole-body robot is built to exploit.

We propose to keep the diversity. The robot's choice of which body part makes contact with the object becomes an explicit discrete variable. The learner trains one action head per contact mode with a shared, mode-aware value function, and at each moment it picks the mode whose predicted return is highest. Contact mode is physically meaningful, and it is cheap to label automatically in simulation. That makes it a better handle on multimodality than a generic expressive policy. The central bet is that a robot allowed to switch strategies depending on the object's mass, height, and pose will finish faster than the best single-strategy policy, under the same sparse reward and the same data budget.

Core Claim

The closest work, Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL, motivates this proposal and supplies its pipeline. That paper uses sample-based model predictive control, a planner that scores thousands of random action sequences in parallel GPU simulation, to generate about 4 million demonstration samples per task. It seeds a FastTD3 learner's replay buffer with them, bounds the critic to the analytically known return range, and phases the demonstrations out as the learner succeeds on its own. Its learned policies run up to 50% faster than the planner that taught them. Its authors also report a "multimodality trap": training collapsed when the demonstrations mixed arm-pushing with torso or shoulder pushing, and filtering the data to a single mode was mandatory.

Our claim is that this trap is avoidable. Avoiding it is worth more than a robustness fix, because the robot can then select modes per state. That selection should beat the best single mode on task distributions where the right strategy depends on the object.

There is a mechanistic subtlety we will test rather than assume. A critic $Q(s,a)$, the network that estimates future return for state $s$ and action $a$, is action-conditioned. It cannot "average modes" in the way a regression onto actions does. The more likely failure point is the deterministic actor, together with any demonstration-anchoring term that pulls it toward the mean of the demonstrated actions. Our design addresses both sources. A first diagnostic experiment will say which one dominates.

Research Program

The high-level policy runs at task rate on top of the frozen ReLIC low-level controller from Versatile Loco-Manipulation through Flexible Interlimb Coordination. We keep that controller frozen deliberately, which isolates the multimodality question from the separate open problem of co-adapting the legs. ReLIC already treats each limb as switchable between support and manipulation, so contact modes map naturally onto its interface.

The policy is split into $K$ deterministic heads $\pi_z(s)$, one per contact mode $z \in \{1,\dots,K\}$. Typical modes are left arm, gripper, shoulder or torso, and a foot. A single critic $Q(s,a,z)$ sees the mode as an input. Action selection is

$$z^*(s) = \arg\max_{z} Q\big(s, \pi_z(s), z\big), \qquad a = \pi_{z^*}(s),$$

with a small switching cost added to the selection rule so the robot does not chatter between contact modes. Mid-task contact switching causes impact transients, a known weakness of ReLIC.

Each demonstration step is labeled with the robot link that is in contact with the object, which the simulator reports for free. The step then trains only its own head's demonstration term and contributes to the critic under its own mode label. This keeps one strategy's actions from dragging another head toward a blend. Online transitions are labeled with the mode that was actually executed. The bounded-critic trick and the demonstration phase-out of the Schuck et al. pipeline are kept unchanged, and phase-out happens per mode, so a rarely successful mode is not starved of demonstrations prematurely.

Contact labels are privileged simulator information, but only during training. At deployment the mode is chosen by the critic from the same observations the baseline policy uses. The method therefore adds no sensing requirement at run time. That matters, because perception is already the main deployment bottleneck in this literature, as Guided Reinforcement Learning for Robust Multi-Contact Loco-Manipulation and Curiosity-Driven Learning of Joint Locomotion and Manipulation Tasks both rely on motion capture or privileged state.

To create multimodal data, we regenerate the planner demonstrations with mode-specific contact costs, for example rewarding arm-object contact in one batch and shoulder-object contact in another. We also randomize object mass, size, and initial pose, so that different modes are faster in different regions of the task space. We first measure how often each mode is fastest by running each mode-specific planner on held-out object configurations. If no mode wins anywhere, the premise fails, and we report that directly.

Baselines

The naive reference point is the Schuck et al. pipeline trained on the raw mixed-mode dataset. It should reproduce the reported collapse, and it tells us how bad the trap is in our regenerated data.

The strongest existing method is the same pipeline trained on the single best unimodal subset, chosen by validation time-to-goal. This is exactly what Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL does. That paper reports near-perfect simulated success across all its tasks, up to 50% faster task completion than its planner teacher, an 11–45% reduction in the standard deviation of completion time, and convergence within 4 million samples on tire rolling. It reports no per-mode time-to-goal figures, so we recompute this baseline ourselves on the randomized task distributions. It is the strongest competitor because it already solves the tasks, and our method must show that diversity adds speed rather than just avoiding a failure. For fairness, every arm gets the same total demonstration count: the unimodal arm receives as many samples from its single mode as the mixture contains in total. Every arm also gets the same 4 million online samples per task, the same GPU, and the same hyperparameter search budget of 20 configurations, over 5 seeds.

We add a third baseline that trains one policy per mode and dispatches between them with an oracle that knows the best mode for each initial object configuration. This is an upper bound on per-episode mode selection. It cannot switch within an episode.

Two ablations isolate the key ingredient. One replaces the contact labels with unsupervised k-means clusters of demonstration actions, which tests whether physical labels matter. The other keeps the mode-split actor but drops the mode input to the critic, which tests where the collapse lives.

Beating the field means three things on the randomized distributions. The method must reach at least the unimodal baseline's success rate. It must achieve lower mean time-to-goal than the best unimodal policy, with non-overlapping 95% confidence intervals across seeds. It must approach the per-episode oracle within 10% on time-to-goal.

Why not just…?

Why not just filter the demonstrations to one mode, as the original authors did? It works, and that is why it is our main baseline rather than a straw man. Its cost is invisible on fixed tasks and appears only when the best strategy depends on the object, for instance shoulder-pushing a heavy low box versus arm-pushing a light tall one. If our first measurement shows that one mode is fastest across nearly all configurations, filtering is the right answer, and the honest conclusion is that multimodality is a nuisance rather than an opportunity. The randomized-distribution comparison settles this.

Why not just use an expressive policy, such as a diffusion model, that represents multiple modes without labels? This is the real threat. A diffusion policy generates actions by gradually denoising random noise, and MaskedManipulator: Versatile Whole-Body Control for Loco-Manipulation shows such generators can represent multimodal whole-body behavior. That work is tracking and imitation, however, not sparse-reward value learning, and nothing in our evidence shows an expressive actor resolving the trap under FastTD3-style updates. We therefore include a mixture-of-Gaussians actor with the same parameter count as a further baseline. Our prediction is that it can represent the modes but tends to drift between them during online updates. Explicit contact modes give the critic a discrete choice it can compare directly. If the mixture actor matches us, the contribution reduces to "use any multimodal actor," and we will say so.

Why not just add dense shaping terms that favor whichever contact mode is better? Doing so requires knowing in advance which mode is better where, which is exactly what we want learned. It also reintroduces the reward-weight tuning that Multi-critic Learning for Whole-body End-effector Twist Tracking found destabilizing for monolithic loco-manipulation policies. Schuck et al. further show that optimizing the true sparse objective already beats a dense-cost teacher.

Proposed Experiments

We use the SMPC-to-RL sparse loco-manipulation suite: box pushing with Spot and with the Unitree G1 humanoid, plus tire uprighting. Target reaching and tire rolling are omitted because they offer little contact-mode choice. Each task is extended with randomized mass (0.5–3× nominal), size, and initial pose, and evaluated on 1,000 held-out configurations. Metrics are success rate, mean and standard deviation of time-to-goal, mode-switch count per episode, and a base-stability measure (falls and peak base tilt), since a shoulder push loads the frozen controller differently from an arm push.

The first decisive check is the diagnostic. On the mixed dataset we inspect whether the baseline's actor outputs fall between the demonstrated mode clusters, which would indicate actor averaging. We also check whether $Q$ evaluated along a line between two mode-specific actions is bimodal, which would indicate a healthy critic. The two ablations then confirm which component the fix requires.

The second check is the headline comparison against the filtered baseline on the randomized distributions. We break the result down by configuration region to verify that the gains come from the critic picking the mode the planner benchmark identified as fastest. Gains on configurations with no mode advantage would point instead to a data-quantity effect.

The third check holds out one mode's demonstrations entirely during phase-out and asks whether its head survives online learning. This tests whether per-mode demonstration retention is necessary.

If the simulated results hold, we would deploy the Spot box-pushing policy with motion-capture object tracking, matching the original paper's setting. On boxes of two masses we would count how often the robot chooses each contact mode and its completion time against the filtered policy, over 10 trials each.