Pareto Front Interfaces for Hierarchical Robot Control

Modern robot stacks increasingly split into two parts. A slow "planner" (a large vision-language model or an image-generating diffusion model) decides what should happen next, and a fast "controller" (a small policy running at 10–50 Hz) decides how to move the motors. The message passed between them is almost always a single point: one goal image, one goal pixel, or one latent vector. We argue that this single point is a scalarization problem in disguise. It quietly collapses every physical trade-off the controller faces (how fast versus how gently, how much joint effort, how much clearance) into one fixed compromise baked in at training time. The planner also cannot tell whether that compromise is even achievable.

We propose replacing the point with a set. The controller learns to report, for any subgoal, the range of outcomes it can actually achieve, which we call its Pareto front. The planner then picks a specific target on that front, for example "reach it in under 2 seconds with contact force below a threshold." The controller is trained to hit that target directly. Two things follow. First, because the controller is conditioned on a desired outcome vector rather than on objective weights, it can in principle reach compromise behaviors in "dented" (concave) regions of the trade-off surface that weighted sums cannot. Second, the planner gets a built-in feasibility test: a subgoal whose predicted front does not contain the requested target is rejected before execution. This directly targets the unreachable-subgoal failure that existing image-subgoal planners report.

Core Claim

The claim has two parts, and each can be falsified on its own. The first part is that a continuous-control policy conditioned on target outcome vectors, together with a learned model of its own attainable front, covers the trade-off surface better than a policy conditioned on objective weights. "Better" means higher hypervolume, the volume of outcome space dominated by the achieved front, and better coverage of non-convex regions. The second part is that exposing this front at the planner–controller interface raises success and budget compliance under multi-criteria instructions. The comparison is against single-point interfaces, including ones augmented with a scalar reachability check. The controller's control rate must not drop.

Concretely, the controller is $\pi_\theta(a \mid s, g, \hat r)$. Here $s$ is the current observation, $g$ the subgoal from the planner, and $\hat r \in \mathbb{R}^k$ the desired vector of $k$ per-segment outcomes (for example time-to-subgoal, integrated joint effort, peak contact force). Alongside it we train a front model $\rho_\phi(s, g, u)$. For a direction $u$ on the probability simplex, it predicts how far from a reference point $z$ (a pessimistic corner of outcome space) the controller can push along $u$. A target is declared feasible when

$$\hat r = z + c\,u \ \text{ with } \ c \le \rho_\phi(s, g, u).$$

This radial description is the same geometry used by Tchebycheff scalarization, a method that scores a solution by its worst-weighted objective rather than a weighted sum. Unlike a weighted sum, it can represent star-shaped fronts with concave dents. This is why we parameterize the front this way, rather than through a supporting hyperplane per weight vector.

Relation to Prior Work

Our motivation comes from two documented failures. On the scalarization side, Pareto Multi-Task Learning recovers trade-offs in multi-task deep learning by decomposing the objective space along preference vectors, and shows that fixed weighted sums cannot reach concave regions. Advantages of Task-Specific Multi-Objective Optimisation in Evolutionary Robotics compares weighted-sum and multi-objective fitness in robot evolution, and shows that weighted sums drive robots into degenerate optima such as spinning in place or clumping. On the hierarchy side, SuSIE generates goal images with an image-editing diffusion model and executes them with a goal-conditioned diffusion policy. Its authors acknowledge that the generated subgoals are often kinematically unreachable. PixNav compresses an object-navigation plan into a single goal pixel. LeVERB passes a learned latent "verb" from a 10 Hz vision-language module to a 50 Hz humanoid whole-body controller. HiRT runs a 7B model asynchronously to feed a 10 Hz policy. All four pass one point and encode none of the physical trade-offs. We keep their split between slow planner and fast controller, and change only what crosses it.

Our method draws on three works. Pareto Conditioned Networks trains a policy by supervised learning to achieve a requested return vector over a requested horizon, avoiding scalarization entirely. However, it was tested only on small discrete domains such as Deep-Sea-Treasure and Minecart. We carry its return-conditioning idea into continuous, image-based, subgoal-conditioned control, and add the explicit front model it lacks. Learning Pareto Set for Multi-Objective Continuous Robot Control (Hyper-MORL) maps a preference vector to policy weights with a single hypernetwork, compressing parameters by roughly 200×. It still trains with a linear weighted sum, and its authors report degraded fidelity on the concave MO-HalfCheetah-v2 front. It is our main flat competitor. Smooth Tchebycheff Scalarization for Multi-Objective Optimization supplies the smooth worst-case scalarization we use as an auxiliary reinforcement-learning loss when the supervised targets are sparse.

Other work constrains the design or competes with it. Pareto Policy Adaptation computes common ascent directions for all objectives to reach Pareto stationarity in continuous control. It assumes linear user preferences and is flat, but it is a strong competitor for front quality. Pareto-Conditioned Diffusion Models for Offline Multi-Objective Optimization generates designs from a diffusion model conditioned on target objective values, and reports failure near 10,000-dimensional policy parameter spaces. This negative result is why we condition actions, not policy weights. Posetal Games replaces scalar costs with partially ordered priority metrics, but only over roughly 90 precomputed trajectories per agent. We borrow its idea that a planner should be able to express "safety strictly before speed": in our setting the planner selects among feasible front points in priority order, instead of choosing weights.

Research Program

The work proceeds in three stages. Each stage is designed so that a negative result changes what we claim, rather than being explained away.

Stage 1: flat continuous control. We test whether outcome-target conditioning plus a front model beats weight conditioning on the multi-objective MuJoCo suite used by Hyper-MORL and Pareto Policy Adaptation: Swimmer, HalfCheetah, Walker2d, Ant, Hopper, and 3-objective Hopper-v3. Training uses hindsight relabeling: every rollout segment is labeled with the outcome vector it actually achieved. The policy learns to reproduce it, and an auxiliary smooth-Tchebycheff reinforcement-learning loss pushes along sampled directions $u$ toward the frontier. The front model $\rho_\phi$ is trained on the nondominated outcomes observed per direction, with a pinball loss so that it estimates an upper quantile rather than a mean. MO-HalfCheetah-v2 is the decisive environment, because a concave failure of linear scalarization is already documented there.

Stage 2: measuring the shape of manipulation fronts. We need to know whether manipulation trade-offs are concave at all. In CALVIN, a simulated tabletop benchmark of chained language-instructed tasks, we define three per-segment outcomes that can be computed in simulation: time to subgoal, integrated squared joint torque, and peak end-effector contact force. Using a large population of controllers trained with Stage 1 methods, we estimate the empirical front per subgoal family and measure its deviation from its convex hull. This stage decides how much weight the concavity argument can carry. If the fronts are essentially convex, a weight-conditioned controller reaches the same points, and the remaining contribution is the feasibility signal alone. We will report that honestly.

Stage 3: the full hierarchy. We plug the Stage 1 controller and front model under a SuSIE-style planner, so that the low-level policy is conditioned on a generated goal image plus $\hat r$. The interface protocol is as follows. The planner samples several candidate subgoal images. The front model scores each one against the instruction's budget. Candidates that cannot meet the budget are discarded. Among the rest, the planner picks a target point by priority order rather than by weights. Because the front model is a small head on the controller's encoder, it runs at controller rate. Re-querying it every control step lets the controller report to the planner when an asynchronous, stale subgoal has become infeasible. This is precisely the regime where HiRT's moderate-dynamics results stop.

Baselines

The naive reference point is SuSIE as published: a single goal image and a controller trained by imitation with no outcome conditioning. Its reported CALVIN zero-shot result is 2.69 instructions completed in a row (SuSIE). We retrain it on our compute so its numbers are directly comparable. We expect our system to match it on unconstrained instructions; any loss there would be a cost of the method. The claim is decided on budgeted instructions.

The strongest competitor on our specific question is a weight-conditioned hierarchy. It uses the same planner and the same controller architecture, but the controller takes a preference vector $\omega$ and is trained on the linear objective $\omega^\top J$, as in Hyper-MORL. The planner selects $\omega$ by searching over weights on held-out validation episodes. It gets the same data, the same number of environment steps, and the same hyperparameter search budget (equal trials per method). For flat front quality we also run Pareto Policy Adaptation. The file reports a hypervolume for it only on SuperMario (1442.14 versus 1401.14 for the next baseline). We found no reported result on Lacuna for Hyper-MORL or PPA hypervolume on MO-HalfCheetah-v2, so we will reproduce both. Beating them means higher hypervolume on MO-HalfCheetah-v2 with non-overlapping seed intervals, plus measurably more achieved points inside the concave region identified in Stage 2.

The ablation that isolates our key ingredient keeps the outcome-conditioned controller but replaces the front model with a scalar reachability classifier, which predicts whether the subgoal will succeed. This separates "knowing whether a subgoal is reachable" from "knowing which trade-offs are reachable." A second ablation removes the Tchebycheff auxiliary loss to test whether hindsight relabeling alone extends to continuous control. For latency, the reference is HiRT's reported ~10 Hz controller versus ~4 Hz for a monolithic model (HiRT). Our controller, including front queries, must stay at or above 10 Hz.

Why not just…?

Why not just condition the controller on weights and let the planner pick them? This is the most serious alternative. If manipulation fronts turn out convex, it reaches every point our method reaches, at lower complexity. The evidence against it is geometric and specific to non-convex fronts (Pareto Multi-Task Learning), and Hyper-MORL documents the concave HalfCheetah failure itself. Stage 2 measures convexity directly, and the head-to-head in Stage 3 settles the question. Even in the convex case, weights do not tell the planner whether a numeric budget such as "peak force under 5 N" is achievable, because the mapping from weights to outcomes is unknown and state-dependent. Our budget-compliance metric tests exactly that.

Why not just filter subgoals with a learned success classifier? This would plausibly fix much of SuSIE's unreachable-subgoal problem, and on unconstrained CALVIN chains it may match us. It answers only "reachable?", not "reachable within this effort and force budget?", and it cannot help choose among feasible executions. The scalar-classifier ablation is built to show where the two diverge. If they do not diverge on budgeted instructions, the front model is not earning its cost and we will say so.

Why not train one constrained policy per budget, or an ensemble over weights? A separate policy per budget or weight grid works at the trained settings but cannot interpolate. Hyper-MORL reports 29 million parameters for a discrete PG-MORL archive on MO-Hopper-v3 versus 140,000 for one conditioned network (Learning Pareto Set). We include a small ensemble at matched parameter count and evaluate it on budgets between its training settings.

Why not drop the hierarchy and use one large end-to-end model? Monolithic vision-language-action models run at roughly 4 Hz. HiRT shows that decoupling raises dynamic-task success from 48% to 75%, and LeVERB reports a 7.8× gain over a naive whole-body baseline (LeVERB). Our proposal keeps the decoupling and is not competing with it.

Proposed Experiments

On multi-objective MuJoCo we report hypervolume, the fraction of points inside concave regions, and front-model calibration, meaning how often outcomes predicted as feasible are actually achieved. We use ten seeds per method. On CALVIN we report average chain length on standard instructions against SuSIE's 2.69. We then construct a budgeted variant: the same chains, each annotated with time, torque, or force limits computed from simulator ground truth. On that variant we report joint success, meaning the task completed and all budgets met, and the rate of rejected-but-actually-feasible subgoals. On Meta-World, following HiRT, we add moving targets at increasing speeds above the ~1 cm/s HiRT tested. We measure whether controller-rate front re-queries detect subgoals that become stale and infeasible, and what happens to success as speed rises.

Three outcomes decide the proposal. First, if outcome-target conditioning fails to beat weight conditioning on MO-HalfCheetah-v2 hypervolume, the concavity argument does not transfer to continuous control. Second, if the CALVIN fronts measured in Stage 2 are convex and the weight-conditioned hierarchy matches us on budgeted joint success, the set-valued interface is unnecessary. Third, if the scalar-classifier ablation matches the front model on budgeted instructions, the multi-criteria part of the interface adds nothing beyond reachability.