Explaining human multiple object tracking as resource-constrained approximate inference in a dynamic probabilistic model

Ed Vul, Michael C. Frank, George A. Alvarez, Joshua B. TenenbaumNeurIPS 2009 · OpenReview

Multiple Object Tracking (MOT) is a classic experimental paradigm used by psychologists to probe the architecture of human visual attention. In a typical MOT task, subjects are asked to track a subset of identical moving items on a screen. While humans can generally track about 4 to 8 items, this capacity degrades significantly when objects move faster or get closer together.

For years, a central debate has persisted: do these tracking limits arise from specialized cognitive "hardware" (like a fixed number of tracking "slots"), low-level perceptual noise (like blurry peripheral vision), or a flexible cognitive resource (like attention)? This paper provides a unifying computational framework, proposing that human MOT performance is best understood as a Bayesian inference problem. By modeling the brain as a Rao-Blackwellized particle filter, the authors demonstrate that many human tracking limitations are actually optimal responses to perceptual noise, though a flexible, task-dependent resource is still required to explain the strict limit on the number of objects tracked.

A Computational Model of Tracking

To determine which human failures are inevitable consequences of uncertainty and which reflect true cognitive bottlenecks, the authors developed an "ideal observer" model based on a Rao-Blackwellized Particle Filter (RBPF).

The Multiple Object Tracking task and the corresponding probabilistic graphical model used by the authors.
  1. Generative Model: The model assumes objects follow an Ornstein-Uhlenbeck process, which acts like Brownian motion on a spring, providing a mean-reverting motion model.
  2. The Data Association Problem: At each time step, the observer receives a "bag" of unlabeled position and velocity measurements. The core computational challenge is deciding which new observation belongs to which previously tracked object.
  3. Inference Engine: The RBPF handles this uncertainty by using particles to represent different possible discrete "assignments" of observations to objects. Simultaneously, it uses a Kalman filter to analytically update the continuous state (estimated position and velocity) for each object within those assignments.
  4. Human-Like Noise: The model incorporates known human perceptual limits, such as position noise that increases with peripheral eccentricity and speed noise that follows Weber’s Law.
  5. Resource Constraint: To account for the performance drop when tracking more items, the authors introduce a "memory noise" term that scales linearly with the number of targets ($U \propto n$). This represents a finite pool of "samples" or "attentional resolution" shared among the tracked objects.

Key Insights and Results

The paper evaluates this model against both existing literature (e.g., classic speed-number tradeoffs) and novel behavioral experiments designed to map "iso-difficulty" curves—measuring how subjects trade off spatial spacing against speed or color-drift rates.

  • Uncertainty, Not Just Capacity: The model reveals that many perceived "limitations" are not bugs in our cognitive architecture, but features of optimal tracking under noise. For instance, tracking fewer objects at higher speeds is a natural consequence of the increased spatial uncertainty (entropy) in the prediction step of the Kalman filter.
  • The Speed-Space Tradeoff: The RBPF model accurately predicts the continuous tradeoff between object speed and spatial separation. As objects move faster, they must be kept further apart to maintain a baseline accuracy threshold.
Model thresholds demonstrating the tradeoff between object speed (velocity standard deviation) and spatial separation (position standard deviation).
  • Flexible Resources Over Fixed Slots: The authors argue strongly against the idea of "fixed slots" for attention. Instead, they show that a flexible resource—formalized as noise that increases as you spread it thinner across multiple targets—perfectly explains why tracking 8 objects is harder than tracking 4, even at identical speeds. By introducing the resource-dependent noise term ($U_n = U_1 n$), the model successfully replicates the human "speed limit" curve.
  • The Inertia Mismatch: Interestingly, humans do not seem to perfectly adapt to the "physics" of a display. The model fits human data best when it assumes humans use a single, low-inertia motion model, even if the objects on screen are moving with high inertia.
  • Feature Integration: The model also explains how humans use features like color to "rescue" tracking. When objects are spatially crowded, color provides an extra dimension to disambiguate targets, a prediction the authors confirmed behaviorally.

Limitations and Future Directions

While the RBPF provides a powerful algorithmic explanation for MOT, the authors note several limitations. The current model uses linear dynamics, whereas real-world motion and complex MOT displays with collisions are highly non-linear. Additionally, the paper assumes the cognitive resource is shared equally among all targets. Future work could explore whether humans dynamically shift "attention gain" to objects currently in high-risk, crowded areas of the screen. Finally, while the model is computationally sound, how the brain's neural networks actually implement "particles" or "Kalman updates" remains an exciting open question for neuroscience.

Summary updated

Abstract

Multiple object tracking is a task commonly used to investigate the architecture of human visual attention. Human participants show a distinctive pattern of successes and failures in tracking experiments that is often attributed to limits on an object system, a tracking module, or other specialized cognitive structures. Here we use a computational analysis of the task of object tracking to ask which human failures arise from cognitive limitations and which are consequences of inevitable perceptual uncertainty in the tracking task. We find that many human performance phenomena, measured through novel behavioral experiments, are naturally produced by the operation of our ideal observer model (a Rao-Blackwelized particle filter). The tradeoff between the speed and number of objects being tracked, however, can only arise from the allocation of a flexible cognitive resource, which can be formalized as either memory or attention.
BibTeX
@inproceedings{vul2009explaining,
  author = {Ed Vul and Michael C. Frank and George A. Alvarez and Joshua B. Tenenbaum},
  title = {{Explaining human multiple object tracking as resource-constrained approximate inference in a dynamic probabilistic model}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2009},
  url = {https://openreview.net/forum?id=HkWIbuZubB},
}
Figure