Goal-Conditioned End-to-End Visuomotor Control for Versatile Skill Primitives

Oliver Groth, Chia-Man Hung, Andrea Vedaldi, Ingmar Posner

Introduction

In robotics, visuomotor control—the direct mapping of visual inputs to motor actions—remains a significant challenge. While current controllers can excel at specific tasks, they are often "one-trick ponies," requiring extensive retraining or fine-tuning to handle new objects or targets. To address this, researchers utilize goal-conditioning, where a robot is provided with a target state, such as an image of a completed task.

Existing goal-conditioning methods typically fall into two categories: Imitation Learning (IL), which often requires test-time human demonstrations, and Model Predictive Control (MPC), which is computationally intensive and prone to failure over long horizons. This paper introduces GEECO (Goal-conditioned End-to-End COntrol), a novel framework that achieves fast, versatile, and demo-free control by learning skill primitives through a single goal image.

High-level overview of GEECO's goal-conditioned control loop across time steps.

The GEECO Architecture

The core innovation of GEECO is the use of Dynamic Images to represent temporal information. A dynamic image compresses a sequence of video frames into a single RGB image that highlights motion, allowing standard Convolutional Neural Networks (CNNs) to process temporal dynamics efficiently.

As shown in the architecture diagram below, GEECO utilizes three specialized CNN branches to process visual information:

  • $\psi_{OBS}$: Encodes the current raw RGB observation frame ($I_t$).
  • $\psi_{DYN}$: Encodes a dynamic image of the most recent frames to capture the robot's current motion state.
  • $\psi_{DIFF}$: Encodes a dynamic image representing the visual difference between the current state ($I_t$) and the target goal image ($I_T$).
Detailed GEECO architecture featuring specialized CNN branches and an LSTM core for multi-stage task management.

These visual features are combined with the robot's proprioceptive data (joint angles) and fed into a Long Short-Term Memory (LSTM) network. The LSTM is vital for multi-stage tasks, as it maintains an internal state that helps the robot track its progress (e.g., transitioning from "grasping" to "lifting"). During training, the model also utilizes auxiliary losses to predict the 3D coordinates of the gripper and the target object, which improves the vision system's spatial awareness.

Experimental Evaluation

The researchers evaluated GEECO using a new benchmark called GOAL2CUBE2, built in the MuJoCo physics engine with a simulated Fetch mobile manipulator. The model was trained on 4,000 expert demonstrations of pushing and pick-and-place tasks.

To test the versatility and robustness of the learned skills, the team introduced several challenging benchmarks:

  • FG:CLUTTER: The workspace is filled with distracting, non-target objects.
  • BG:PRISMA: A dynamic, rainbow-colored background designed to confuse visual sensors.
  • NUTONCONE & BALLINCUP: Zero-shot generalization tests where the robot must apply its skills to entirely new object shapes (nuts, cones, balls) never seen during training.

Key Results and Insights

  • Superior Performance: In pushing tasks, GEECO achieved an 89% success rate, vastly outperforming visual MPC (7%) and meta-learning baselines (15%).
  • Multi-Stage Capability: GEECO successfully handled complex pick-and-place sequences where baseline models almost entirely failed.
  • Perceptual Invariance: By focusing on the change required to reach a goal via dynamic images, GEECO proved robust to background noise. In the "BG:PRISMA" test, standard RGB models failed (0% success), while GEECO remained functional.
  • Zero-Shot Generalization: The model successfully manipulated novel objects like nuts and cones, demonstrating that it had learned general skill primitives rather than specific, rigid trajectories.

Limitations and Future Work

Despite its success, GEECO faces some limitations. The reliance on motion for visual signals can lead to "depth ambiguity" when the robot pauses (e.g., just before dropping an object). Additionally, the current work is limited to rigid objects in simulation. Future research will focus on extending GEECO to deformable objects and bridging the sim-to-real gap for deployment on physical hardware.

Summary updated

Abstract

Visuomotor control (VMC) is an effective means of achieving basic manipulation tasks such as pushing or pick- and-place from raw images. Conditioning VMC on desired goal states is a promising way of achieving versatile skill primitives. However, common conditioning schemes either rely on task-specific fine tuning - e.g. using one-shot imitation learning (IL) - or on sampling approaches using a forward model of scene dynamics i.e. model-predictive control (MPC), leaving deployability and planning horizon severely limited. In this paper we propose a conditioning scheme which avoids these pitfalls by learning the controller and its conditioning in an end-to-end manner. Our model predicts complex action sequences based directly on a dynamic image representation of the robot motion and the distance to a given target observation. In contrast to related works, this enables our approach to efficiently perform complex manipulation tasks from raw image observations without predefined control primitives or test time demonstrations. We report significant improvements in task success over representative MPC and IL baselines. We also demonstrate our model's generalisation capabilities in challenging, unseen tasks featuring visual noise, cluttered scenes and unseen object geometries.
BibTeX
@inproceedings{groth2020goal,
  author = {Oliver Groth and Chia-Man Hung and Andrea Vedaldi and Ingmar Posner},
  title = {{Goal-Conditioned End-to-End Visuomotor Control for Versatile Skill Primitives}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  doi = {10.1109/icra48506.2021.9560752},
}
Figure