TL;DR
The Foundation Pareto Flow Policy (FP2) brings the "pre-mid-post-training" paradigm of foundation models to Multi-Objective Reinforcement Learning (MORL). By utilizing a multi-modal diffusion transformer and a novel three-stage training pipeline, FP2 learns a reusable "trade-off prior" that generalizes zero-shot to unseen user preferences and adapts rapidly to entirely new robotic tasks.
What Problem Does This Paper Address?
In the real world, reinforcement learning (RL) rarely has a single goal. A delivery robot needs to balance speed against battery life; a surgical arm must balance precision against safety. These are Multi-Objective Reinforcement Learning (MORL) problems, where the goal is to find a set of "Pareto-optimal" policies—solutions where you cannot improve one objective without hurting another.
The current bottleneck in MORL is generalization. Most existing methods are task-specific: a policy trained to balance speed and energy for a quadruped robot cannot easily transfer that "understanding" of trade-offs to a bipedal robot. Furthermore, these models often struggle to interpolate between known preferences or extrapolate to extreme ones. The authors aim to move away from these "siloed" models toward a foundation-level policy that understands the underlying structure of multi-objective trade-offs across different environments.
Methods
The authors introduce FP2, which stands out through its architecture and its training recipe:
- Generative Backbone: FP2 uses a Multi-Modal Diffusion Transformer (MMDiT). Unlike standard autoregressive models, this uses Rectified Flow Matching, a technique that learns to "straighten" the paths used to generate trajectories from noise. This makes the generation process more efficient and stable for continuous control.
- Dual Conditioning: The model is "instructable." It accepts both numerical preference vectors (e.g., $[0.8, 0.2]$) for precise control and textual prompts (e.g., "strongly prioritize speed over energy efficiency") for fuzzy, human-centric guidance.
- The Three-Stage Pipeline:
- Stage 1 (Pre-training): Trained on a massive, diverse, but noisy dataset to learn broad multi-objective control priors.
- Stage 2 (Mid-training): A novel "alignment" stage. The model is fine-tuned on a curated set of high-quality, Pareto-optimal trajectories. This "sharpens" the policy, ensuring that the agent’s behavior strictly follows the requested trade-offs.
- Stage 3 (Post-training): Efficient few-shot adaptation to new tasks or out-of-distribution preferences using parameter-efficient fine-tuning and $\ell_2$ regularization to prevent "catastrophic forgetting."
- Triplet Velocity Regularization: To ensure that similar preferences lead to similar (but distinct) behaviors, the authors introduced a contrastive loss that enforces a consistent structure in the learned preference space.
Datasets
The study primarily utilizes the D4MORL benchmark, which includes six complex MuJoCo robotic environments (Ant, HalfCheetah, Hopper, Walker2d, Swimmer, and a 3-objective Hopper).
To test generalization, they used "Shattered" and "Narrow" preference splits, where 30% of the preference space is hidden during training. For cross-task evaluation, they tested the model's ability to transfer knowledge from MuJoCo tasks to ten different Meta-World manipulation tasks (like opening doors or pressing buttons) in a zero-shot or few-shot manner.
Key Insights and Ideas
- The "Trade-off Prior" is Transferable: One of the most striking findings is that the logic of balancing objectives (e.g., "if I want more speed, I must accept more cost") can be learned in one environment and successfully applied to another. This suggests that Pareto manifolds have a shared latent structure across different robot morphologies.
- Mid-training as Alignment: Much like how LLMs use RLHF to align with human intent, MORL benefits from a dedicated mid-training stage. By filtering for "dataset-relative" Pareto fronts, the authors show you don't need perfect expert data—you just need to show the model the best examples currently available in your dataset to significantly improve preference alignment.
- Text as a Semantic Bridge: While numerical vectors provide the "knobs" for fine-tuning, the textual encoders (CLIP and L6) provide a semantic interface that allows the model to leverage high-level concepts, aiding in zero-shot transfer to tasks the model hasn't seen before.
Key Results
FP2 sets a new state-of-the-art (SOTA) for offline MORL:
- Superior Performance: Compared to the previous SOTA (MODULI), FP2 achieved median gains of +0.98% in Hypervolume (a measure of Pareto front quality) and a massive +29.44% improvement in Sparsity (meaning it covers the trade-off front much more uniformly).
- Zero-Shot Generalization: In tests with "held-out" preferences, FP2 demonstrated significantly better interpolation and extrapolation than baselines, effectively "filling in the gaps" of the training data.
- Rapid Adaptation: In cross-task settings, FP2 adapted to new environments in just 1,000 gradient steps, outperforming models trained from scratch by a wide margin.
- Efficiency: Despite its power, FP2 has roughly half the trainable parameters (12.8M vs 23.4M) and lower inference latency than MODULI.
Limitations and Future Work
The authors identify several areas for growth:
- Template-based Text: Currently, textual prompts are generated from templates. Moving to true free-form natural language would increase flexibility.
- State vs. Vision: The current model operates on low-dimensional states. Extending this to high-dimensional visual inputs (pixels) is a necessary step for real-world robotics.
- Disentangling Priors: It remains an open question how much of the transfer success comes from the "multi-objective structure" versus general trajectory pre-training. Future work could isolate these factors more cleanly.
Overall, FP2 represents a significant step toward "Foundation Policies" that don't just learn how to move, but how to choose between competing goals.
Summary updated