Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learning

Iou-Jen Liu, Zhongzheng Ren, Raymond A. Yeh, Alexander G. Schwing

Abstract

Multi-agent reinforcement learning (MARL) from visual inputs is a significant challenge due to the high dimensionality of pixel data and the non-stationarity of multi-agent environments. We propose Semantic Tracklets, an object-centric intermediate representation that characterizes environment states by explicitly tracking objects and their roles. By transforming raw pixels into a graph-based representation of tracked entities, our approach enables agents to learn coordinated strategies more efficiently. We evaluate this method on the Visual Multi-Agent Particle Environment (VMPE) and the Google Research Football (GFootball) simulator. Our results show that Semantic Tracklets consistently outperform pixel-based baselines, achieving a +2.4 higher score difference in GFootball and becoming the first method to successfully train a five-player team in this environment using only visual data.

Introduction

Multi-agent reinforcement learning is notoriously difficult because the environment is non-stationary; as one agent learns and changes its behavior, the "world" effectively changes for all other agents. This difficulty is compounded when agents must operate directly on high-dimensional visual inputs (pixels). Standard end-to-end approaches using Convolutional Neural Networks (CNNs) often struggle to extract meaningful features in complex, physics-based environments like Google Research Football. While imitation learning can mitigate these issues, it requires expensive expert demonstrations. This work investigates whether a disentangled, object-centric representation—Semantic Tracklets—can provide the necessary inductive bias to scale MARL to complex visual tasks without requiring expert data.

The Semantic Tracklets Pipeline

The proposed architecture decomposes the learning problem into perception and policy learning. Instead of mapping pixels directly to actions, the system extracts structured information about the entities in the scene.

Overview of the Semantic Tracklets pipeline, showing the transition from raw observations to tracked objects and graph-based policy learning.

The pipeline consists of four primary stages:

  1. Object Detection: A fast detector (e.g., YOLOv3) identifies key objects such as the ball and players.
  2. Tracking: To maintain agent identity across frames, the Hungarian algorithm solves an assignment problem, ensuring temporal consistency (e.g., identifying "Player A" across consecutive frames).
  3. Representation: Each object is defined by its "semantics" (type, such as teammate or opponent) and its "tracklet" (spatial trajectory over recent frames).
  4. Graph Neural Network (GNN) Processing: The agents and objects are treated as nodes in a graph. A Graph Convolutional Network (GCN) processes the $K$-nearest neighbors of the controlled agent. This architecture ensures permutation invariance, meaning the model's output is independent of the order in which identical teammates are listed in the input.
Detail of the preprocessing and object detection stage, where input frames are resized and cropped before being passed to the detector.

The entire system is trained using Proximal Policy Optimization (PPO), with the GCN serving as the backbone for both the policy ($π_θ$) and value ($V_ϕ$) functions.

Experimental Evaluation

We evaluated Semantic Tracklets on two benchmarks:

  • Visual Multi-Agent Particle Environment (VMPE): A visual variant of the classic MPE, featuring tasks like cooperative navigation and predator-prey.
  • Google Research Football (GFootball): A complex soccer simulator. We tested on "Academy" scenarios and the full 11-vs-11 game, controlling teams of 3 and 5 agents.

Key Results

  • GFootball Breakthrough: Our method is the first to learn a coordinated strategy for five players in GFootball using only visual data. In 11-vs-11 matches, it achieved a +2.4 higher score difference than CNN-based baselines.
  • Data Efficiency: In single-agent GFootball tasks, Semantic Tracklets reached target performance in just 7.5 million steps, compared to 100 million steps for the baseline—a 13x improvement.
  • Robustness: The system remains stable even with a 10% random drop in object detections, demonstrating resilience to noisy perception.

Discussion and Insights

The core advantage of Semantic Tracklets is the shift from "pixels" to "objects." Agents in a game interact with entities, not raw color values. By explicitly extracting these entities, the RL agent avoids wasting millions of training steps learning basic vision. Furthermore, providing object labels (bounding boxes) is significantly easier and cheaper than providing expert action demonstrations required for imitation learning.

While highly effective, the method currently relies on a pre-defined set of object categories. Future research could explore self-supervised object discovery to remove the need for explicit labels and investigate scaling these graph-based representations to environments with hundreds of agents.

Summary updated

Abstract

Solving complex real-world tasks, e.g., autonomous fleet control, often involves a coordinated team of multiple agents which learn strategies from visual inputs via reinforcement learning. Many existing multi-agent reinforcement learning (MARL) algorithms however don't scale to environments where agents operate on visual inputs. To address this issue, algorithmically, recent works have focused on non-stationarity and exploration. In contrast, we study whether scalability can also be achieved via a disentangled representation. For this, we explicitly construct an object-centric intermediate representation to characterize the states of an environment, which we refer to as `semantic tracklets.' We evaluate `semantic tracklets' on the visual multi-agent particle environment (VMPE) and on the challenging visual multi-agent GFootball environment. `Semantic tracklets' consistently outperform baselines on VMPE, and achieve a +2.4 higher score difference than baselines on GFootball. Notably, this method is the first to successfully learn a strategy for five players in the GFootball environment using only visual data.
BibTeX
@inproceedings{liu2021semantic,
  author = {Iou-Jen Liu and Zhongzheng Ren and Raymond A. Yeh and Alexander G. Schwing},
  title = {{Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=6qDPzo2LWc},
}
Figure