Introduction
In robotics, visuomotor control—the direct mapping of visual inputs to motor actions—remains a significant challenge. While current controllers can excel at specific tasks, they are often "one-trick ponies," requiring extensive retraining or fine-tuning to handle new objects or targets. To address this, researchers utilize goal-conditioning, where a robot is provided with a target state, such as an image of a completed task.
Existing goal-conditioning methods typically fall into two categories: Imitation Learning (IL), which often requires test-time human demonstrations, and Model Predictive Control (MPC), which is computationally intensive and prone to failure over long horizons. This paper introduces GEECO (Goal-conditioned End-to-End COntrol), a novel framework that achieves fast, versatile, and demo-free control by learning skill primitives through a single goal image.
The GEECO Architecture
The core innovation of GEECO is the use of Dynamic Images to represent temporal information. A dynamic image compresses a sequence of video frames into a single RGB image that highlights motion, allowing standard Convolutional Neural Networks (CNNs) to process temporal dynamics efficiently.
As shown in the architecture diagram below, GEECO utilizes three specialized CNN branches to process visual information:
- $\psi_{OBS}$: Encodes the current raw RGB observation frame ($I_t$).
- $\psi_{DYN}$: Encodes a dynamic image of the most recent frames to capture the robot's current motion state.
- $\psi_{DIFF}$: Encodes a dynamic image representing the visual difference between the current state ($I_t$) and the target goal image ($I_T$).
These visual features are combined with the robot's proprioceptive data (joint angles) and fed into a Long Short-Term Memory (LSTM) network. The LSTM is vital for multi-stage tasks, as it maintains an internal state that helps the robot track its progress (e.g., transitioning from "grasping" to "lifting"). During training, the model also utilizes auxiliary losses to predict the 3D coordinates of the gripper and the target object, which improves the vision system's spatial awareness.
Experimental Evaluation
The researchers evaluated GEECO using a new benchmark called GOAL2CUBE2, built in the MuJoCo physics engine with a simulated Fetch mobile manipulator. The model was trained on 4,000 expert demonstrations of pushing and pick-and-place tasks.
To test the versatility and robustness of the learned skills, the team introduced several challenging benchmarks:
- FG:CLUTTER: The workspace is filled with distracting, non-target objects.
- BG:PRISMA: A dynamic, rainbow-colored background designed to confuse visual sensors.
- NUTONCONE & BALLINCUP: Zero-shot generalization tests where the robot must apply its skills to entirely new object shapes (nuts, cones, balls) never seen during training.
Key Results and Insights
- Superior Performance: In pushing tasks, GEECO achieved an 89% success rate, vastly outperforming visual MPC (7%) and meta-learning baselines (15%).
- Multi-Stage Capability: GEECO successfully handled complex pick-and-place sequences where baseline models almost entirely failed.
- Perceptual Invariance: By focusing on the change required to reach a goal via dynamic images, GEECO proved robust to background noise. In the "BG:PRISMA" test, standard RGB models failed (0% success), while GEECO remained functional.
- Zero-Shot Generalization: The model successfully manipulated novel objects like nuts and cones, demonstrating that it had learned general skill primitives rather than specific, rigid trajectories.
Limitations and Future Work
Despite its success, GEECO faces some limitations. The reliance on motion for visual signals can lead to "depth ambiguity" when the robot pauses (e.g., just before dropping an object). Additionally, the current work is limited to rigid objects in simulation. Future research will focus on extending GEECO to deformable objects and bridging the sim-to-real gap for deployment on physical hardware.
Summary updated