Abstract
Multi-task imitation learning (MTIL) has shown significant potential in robotic manipulation by enabling agents to perform various tasks using a single policy. This simplifies policy deployment and enhances adaptability. However, challenges remain in maintaining action reliability and generalizing to unseen tasks. We introduce the Foresight-Augmented Manipulation Policy (FoAM), a novel MTIL policy that utilizes multi-modal goal conditions and introduces foresight augmentation alongside general action reconstruction. FoAM enables agents to reason about the visual consequences of their actions, learning expressive embeddings that capture nuanced task variations. Extensive experiments across 100+ tasks in simulation and real-world settings demonstrate that FoAM outperforms state-of-the-art baselines by up to 41% in success rate.
Introduction
In robotic manipulation, the goal of Multi-Task Imitation Learning (MTIL) is to develop a single policy capable of executing a wide array of chores. Despite its promise, MTIL faces two primary hurdles: action reliability and scenario ambiguity. Robots often fail to reason about the long-term consequences of their movements, leading to drifts from intended trajectories. Furthermore, language-only models struggle with generalization, while image-only models often fail to resolve ambiguities in complex environments, such as identifying which specific drawer in a cabinet is the target.
FoAM addresses these issues by equipping robots with "foresight"—the human-like ability to visualize the completed state of a task and use that mental image to guide real-time actions.
Methodology
FoAM introduces two primary innovations to the standard Transformer-based policy architecture to enhance task understanding and execution reliability.
Multi-modal Goal Conditioning
Rather than relying solely on text or images, FoAM utilizes both. A Goal Imagination Module, powered by a fine-tuned Vision-Language Model (InstructPix2Pix), takes a text command (e.g., "Put eggplant into the green bowl") and the current camera view to "hallucinate" a visual representation of the successful goal state.
Foresight Augmentation (FA)
During training, the model is tasked with predicting not only the next sequence of actions but also the visual consequence of those actions. A Foresight Loss ensures the robot's predicted future state aligns with the imagined goal image. This forces the model to learn a deeper understanding of how its movements alter the environment, acting as a powerful regularizer that prevents "nonsense" actions.
Experimental Setup
The researchers evaluated FoAM using a massive new benchmark and real-world tests:
- FoAM Benchmark: A MuJoCo-based simulation suite featuring a dual-arm UR3e robot with 10 scenarios and 86 tasks.
- Real-World Dataset: 14 tasks (e.g., test tube sorting, fruit placing) using a UFACTORY xArm 7.
- VLM Training: The imagination module was trained on 20,000 image pairs, including data from Google's RT-1 dataset.
Key Results and Discussion
FoAM demonstrated significant performance gains across all tested scenarios. In simulation, the model achieved an average success rate of 86% in dual-arm tasks, vastly outperforming baselines like BAKU (31%) and MT-ACT (33%).
Generalization and Robustness
On completely unseen tasks, FoAM maintained a 66% success rate, whereas language-based baselines often dropped to 0%. This suggests that aligning pixel-level information with semantic instructions allows for zero-shot potential in novel scenarios. In physical tests, FoAM proved reactive; for instance, if an object was removed from its gripper, the robot could re-grasp it and continue the task rather than blindly following a pre-planned path.
Limitations and Future Work
While FoAM sets a new standard for MTIL, some limitations persist. Tasks requiring extreme precision, such as inserting a test tube into a tight hole, sometimes suffered from slight misalignments in the VLM-generated goal images. Future research may focus on generating intermediate sub-goals to act as consequence controllers throughout long-horizon tasks, further refining the robot's ability to navigate complex, multi-step environments.
Summary updated