Foundation Pareto Flow Policy for Multi-Objective Reinforcement Learning

Zhanjiang Yang, Lijun Sun, Yueming Li, Meng Li, Yijun YangarXiv 2026 · arXiv

TL;DR

The Foundation Pareto Flow Policy (FP2) brings the "pre-mid-post-training" paradigm of foundation models to Multi-Objective Reinforcement Learning (MORL). By utilizing a multi-modal diffusion transformer and a novel three-stage training pipeline, FP2 learns a reusable "trade-off prior" that generalizes zero-shot to unseen user preferences and adapts rapidly to entirely new robotic tasks.

What Problem Does This Paper Address?

In the real world, reinforcement learning (RL) rarely has a single goal. A delivery robot needs to balance speed against battery life; a surgical arm must balance precision against safety. These are Multi-Objective Reinforcement Learning (MORL) problems, where the goal is to find a set of "Pareto-optimal" policies—solutions where you cannot improve one objective without hurting another.

The current bottleneck in MORL is generalization. Most existing methods are task-specific: a policy trained to balance speed and energy for a quadruped robot cannot easily transfer that "understanding" of trade-offs to a bipedal robot. Furthermore, these models often struggle to interpolate between known preferences or extrapolate to extreme ones. The authors aim to move away from these "siloed" models toward a foundation-level policy that understands the underlying structure of multi-objective trade-offs across different environments.

Methods

The authors introduce FP2, which stands out through its architecture and its training recipe:

  1. Generative Backbone: FP2 uses a Multi-Modal Diffusion Transformer (MMDiT). Unlike standard autoregressive models, this uses Rectified Flow Matching, a technique that learns to "straighten" the paths used to generate trajectories from noise. This makes the generation process more efficient and stable for continuous control.
  2. Dual Conditioning: The model is "instructable." It accepts both numerical preference vectors (e.g., $[0.8, 0.2]$) for precise control and textual prompts (e.g., "strongly prioritize speed over energy efficiency") for fuzzy, human-centric guidance.
  3. The Three-Stage Pipeline:
    • Stage 1 (Pre-training): Trained on a massive, diverse, but noisy dataset to learn broad multi-objective control priors.
  4. Stage 2 (Mid-training): A novel "alignment" stage. The model is fine-tuned on a curated set of high-quality, Pareto-optimal trajectories. This "sharpens" the policy, ensuring that the agent’s behavior strictly follows the requested trade-offs.
  5. Stage 3 (Post-training): Efficient few-shot adaptation to new tasks or out-of-distribution preferences using parameter-efficient fine-tuning and $\ell_2$ regularization to prevent "catastrophic forgetting."
  6. Triplet Velocity Regularization: To ensure that similar preferences lead to similar (but distinct) behaviors, the authors introduced a contrastive loss that enforces a consistent structure in the learned preference space.

Datasets

The study primarily utilizes the D4MORL benchmark, which includes six complex MuJoCo robotic environments (Ant, HalfCheetah, Hopper, Walker2d, Swimmer, and a 3-objective Hopper).

To test generalization, they used "Shattered" and "Narrow" preference splits, where 30% of the preference space is hidden during training. For cross-task evaluation, they tested the model's ability to transfer knowledge from MuJoCo tasks to ten different Meta-World manipulation tasks (like opening doors or pressing buttons) in a zero-shot or few-shot manner.

Key Insights and Ideas

  • The "Trade-off Prior" is Transferable: One of the most striking findings is that the logic of balancing objectives (e.g., "if I want more speed, I must accept more cost") can be learned in one environment and successfully applied to another. This suggests that Pareto manifolds have a shared latent structure across different robot morphologies.
  • Mid-training as Alignment: Much like how LLMs use RLHF to align with human intent, MORL benefits from a dedicated mid-training stage. By filtering for "dataset-relative" Pareto fronts, the authors show you don't need perfect expert data—you just need to show the model the best examples currently available in your dataset to significantly improve preference alignment.
  • Text as a Semantic Bridge: While numerical vectors provide the "knobs" for fine-tuning, the textual encoders (CLIP and L6) provide a semantic interface that allows the model to leverage high-level concepts, aiding in zero-shot transfer to tasks the model hasn't seen before.

Key Results

FP2 sets a new state-of-the-art (SOTA) for offline MORL:

  • Superior Performance: Compared to the previous SOTA (MODULI), FP2 achieved median gains of +0.98% in Hypervolume (a measure of Pareto front quality) and a massive +29.44% improvement in Sparsity (meaning it covers the trade-off front much more uniformly).
  • Zero-Shot Generalization: In tests with "held-out" preferences, FP2 demonstrated significantly better interpolation and extrapolation than baselines, effectively "filling in the gaps" of the training data.
  • Rapid Adaptation: In cross-task settings, FP2 adapted to new environments in just 1,000 gradient steps, outperforming models trained from scratch by a wide margin.
  • Efficiency: Despite its power, FP2 has roughly half the trainable parameters (12.8M vs 23.4M) and lower inference latency than MODULI.

Limitations and Future Work

The authors identify several areas for growth:

  • Template-based Text: Currently, textual prompts are generated from templates. Moving to true free-form natural language would increase flexibility.
  • State vs. Vision: The current model operates on low-dimensional states. Extending this to high-dimensional visual inputs (pixels) is a necessary step for real-world robotics.
  • Disentangling Priors: It remains an open question how much of the transfer success comes from the "multi-objective structure" versus general trajectory pre-training. Future work could isolate these factors more cleanly.

Overall, FP2 represents a significant step toward "Foundation Policies" that don't just learn how to move, but how to choose between competing goals.

Summary updated

Abstract

The overarching goal of reinforcement learning (RL) is to efficiently optimize the agent's policy for various environments, tasks, dynamics, and even user preferences, which often consist of multiple and potentially conflicting rewards/objectives. This necessitates the learning of Pareto-optimal policies that can navigate different trade-offs among objectives. However, existing multi-objective RL (MORL) methods, whether trained online or offline, are typically limited to task- or objective-specific policy learning and do not address generalization of learned trade-offs to unseen tasks/objectives. In this work, we propose the Foundation Pareto Flow Policy (FP2) for offline MORL in continuous robotic control, inspired by the pre-mid-post-training paradigm underlying recent foundation models. Specifically, we instantiate FP2 as a multi-modal diffusion transformer equipped with a likelihood-free flow matching loss, enabling unified, scalable, and instructable learning of diverse trade-off trajectories. We further develop a three-stage training pipeline to gradually optimize the policy: (1) pre-training on a large-scale MORL dataset, (2) mid-training on a carefully curated high-quality dataset, and (3) post-training on few collected trajectory data from new tasks and trade-offs. Empirical results demonstrate that FP2 outperforms previous state-of-the-art methods on most tasks of the D4MORL benchmark, achieving superior zero-shot interpolation and extrapolation on complex Pareto manifolds as well as efficient few-shot adaptation of the learned trade-off prior to new continuous-control tasks. Code is available at https://github.com/FibonacciRabbit/FP2.
BibTeX
@misc{yang2026foundation,
  author = {Zhanjiang Yang and Lijun Sun and Yueming Li and Meng Li and Yijun Yang},
  title = {{Foundation Pareto Flow Policy for Multi-Objective Reinforcement Learning}},
  year = {2026},
  eprint = {2611.19825},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19825},
}
Figure