Learning What and Where - Unsupervised Disentangling Location and Identity Tracking

Manuel Traub, Sebastian Otte, Tobias Menge, Matthias Karlbauer, Jannik Thümmel, Martin V. Butz

Abstract

Tracking objects through complex interactions and occlusions remains a fundamental challenge in computer vision. We present Loci (Location and Identity tracking), a brain-inspired, self-supervised system that achieves state-of-the-art performance by strictly disentangling an object's appearance ("what") from its spatial properties ("where"). By mimicking the dual-pathway processing of the primate brain, Loci demonstrates superior object permanence, maintaining track of entities even when completely hidden. Evaluated on the rigorous CATER benchmark, Loci achieves 90.7% Top-1 accuracy with significantly fewer parameters than existing models, proving that strong architectural inductive biases can outperform raw scale in visual reasoning tasks.

Introduction

A central problem in visual perception is the "binding problem": the challenge of correctly associating visual features with specific objects and maintaining those associations over time. While deep learning has advanced object detection, many models struggle with object permanence—the understanding that an object continues to exist even when occluded. Most current systems conflate an object's identity with its location; when an object disappears behind another, the model often "forgets" it or loses its track.

To address this, we introduce Loci, a slot-based architecture designed to disentangle identity (the "Gestalt") from spatial properties (location, size, and priority). This separation allows the model to maintain a stable representation of an object's appearance even as its spatial coordinates become uncertain or hidden, enabling robust tracking through containment and occlusion.

Methodology

Loci is inspired by the dorsal ("where") and ventral ("what") pathways in the primate brain. It processes video streams without human-labeled supervision using a predictive coding framework.

Slot-based Architecture

The model distributes visual input into a set of "slots," each representing a potential object. Each slot generates two distinct latent codes:

  1. Gestalt Code: Captures appearance features such as shape, color, and texture.
  2. Spatial Code: Encodes X/Y coordinates, scale, and a "priority" value used for handling occlusions.
The Loci architecture, showing the temporal flow and the internal structure of the slot-based modules.

Temporal Dynamics and GateL0RD

To model object interactions and motion, Loci employs a Transition Module featuring Multi-Head Self-Attention (MHSA) and a specialized recurrent unit called GateL0RD. GateL0RD is designed for sparse updates, allowing the model to maintain stable object identities over long sequences. The system learns by minimizing the error between its predicted next frame and the actual visual input.

Efficient Training

Loci utilizes a combination of truncated backpropagation through time (BPTT) and "e-prop" (eligibility traces). This approach allows the model to learn from long video sequences without the prohibitive computational costs typically associated with standard recurrent neural network training.

Experimental Results

Loci was evaluated across several challenging benchmarks, demonstrating its ability to reason about physical scenes and maintain object permanence.

CATER Benchmark

The CATER dataset requires tracking a "Snitch" (a small yellow ball) as it is moved and often hidden inside other objects. Loci achieved a new state-of-the-art 90.7% Top-1 accuracy on the Snitch localization task. Notably, even in a purely unsupervised setting, Loci reached 78.4% accuracy, rivaling the performance of previous supervised models.

Example frame from the CATER benchmark, where the model must track the 'Snitch' (yellow ball) even when occluded.

Long-term Prediction and Real-world Scaling

On the Moving MNIST dataset, Loci maintained digit shapes for up to 100 frames, significantly outperforming models like PhyDNet which often suffer from "blurring" over time. Furthermore, tests on real-world aquarium footage demonstrated Loci's ability to track over 15 distinct fish simultaneously, proving its scalability beyond synthetic environments.

Discussion and Conclusion

The success of Loci highlights the power of architectural inductive biases. With only 3 million parameters—roughly 10x fewer than competing models—Loci achieves superior results by enforcing the separation of identity and location. This disentanglement is key to achieving human-like object permanence.

Future work will focus on extending Loci to handle dynamic backgrounds (ego-motion), transparent materials, and integrating the tracking system with reinforcement learning for goal-directed agents in complex environments.

Summary updated

BibTeX
@inproceedings{traub2023learning,
  author = {Manuel Traub and Sebastian Otte and Tobias Menge and Matthias Karlbauer and Jannik Thümmel and Martin V. Butz},
  title = {{Learning What and Where - Unsupervised Disentangling Location and Identity Tracking}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=NeDc-Ak-H_},
}
Figure