Abstract
Tracking objects through complex interactions and occlusions remains a fundamental challenge in computer vision. We present Loci (Location and Identity tracking), a brain-inspired, self-supervised system that achieves state-of-the-art performance by strictly disentangling an object's appearance ("what") from its spatial properties ("where"). By mimicking the dual-pathway processing of the primate brain, Loci demonstrates superior object permanence, maintaining track of entities even when completely hidden. Evaluated on the rigorous CATER benchmark, Loci achieves 90.7% Top-1 accuracy with significantly fewer parameters than existing models, proving that strong architectural inductive biases can outperform raw scale in visual reasoning tasks.
Introduction
A central problem in visual perception is the "binding problem": the challenge of correctly associating visual features with specific objects and maintaining those associations over time. While deep learning has advanced object detection, many models struggle with object permanence—the understanding that an object continues to exist even when occluded. Most current systems conflate an object's identity with its location; when an object disappears behind another, the model often "forgets" it or loses its track.
To address this, we introduce Loci, a slot-based architecture designed to disentangle identity (the "Gestalt") from spatial properties (location, size, and priority). This separation allows the model to maintain a stable representation of an object's appearance even as its spatial coordinates become uncertain or hidden, enabling robust tracking through containment and occlusion.
Methodology
Loci is inspired by the dorsal ("where") and ventral ("what") pathways in the primate brain. It processes video streams without human-labeled supervision using a predictive coding framework.
Slot-based Architecture
The model distributes visual input into a set of "slots," each representing a potential object. Each slot generates two distinct latent codes:
- Gestalt Code: Captures appearance features such as shape, color, and texture.
- Spatial Code: Encodes X/Y coordinates, scale, and a "priority" value used for handling occlusions.
Temporal Dynamics and GateL0RD
To model object interactions and motion, Loci employs a Transition Module featuring Multi-Head Self-Attention (MHSA) and a specialized recurrent unit called GateL0RD. GateL0RD is designed for sparse updates, allowing the model to maintain stable object identities over long sequences. The system learns by minimizing the error between its predicted next frame and the actual visual input.
Efficient Training
Loci utilizes a combination of truncated backpropagation through time (BPTT) and "e-prop" (eligibility traces). This approach allows the model to learn from long video sequences without the prohibitive computational costs typically associated with standard recurrent neural network training.
Experimental Results
Loci was evaluated across several challenging benchmarks, demonstrating its ability to reason about physical scenes and maintain object permanence.
CATER Benchmark
The CATER dataset requires tracking a "Snitch" (a small yellow ball) as it is moved and often hidden inside other objects. Loci achieved a new state-of-the-art 90.7% Top-1 accuracy on the Snitch localization task. Notably, even in a purely unsupervised setting, Loci reached 78.4% accuracy, rivaling the performance of previous supervised models.
Long-term Prediction and Real-world Scaling
On the Moving MNIST dataset, Loci maintained digit shapes for up to 100 frames, significantly outperforming models like PhyDNet which often suffer from "blurring" over time. Furthermore, tests on real-world aquarium footage demonstrated Loci's ability to track over 15 distinct fish simultaneously, proving its scalability beyond synthetic environments.
Discussion and Conclusion
The success of Loci highlights the power of architectural inductive biases. With only 3 million parameters—roughly 10x fewer than competing models—Loci achieves superior results by enforcing the separation of identity and location. This disentanglement is key to achieving human-like object permanence.
Future work will focus on extending Loci to handle dynamic backgrounds (ego-motion), transparent materials, and integrating the tracking system with reinforcement learning for goal-directed agents in complex environments.
Summary updated