Faster Attend-Infer-Repeat with Tractable Probabilistic Models

Karl Stelzner, Robert Peharz, Kristian Kersting

TL;DR

SuPAIR (Sum-Product Attend-Infer-Repeat) significantly accelerates unsupervised scene understanding by replacing the intractable Variational Autoencoders (VAEs) in the original AIR framework with tractable Sum-Product Networks (SPNs). This architectural swap allows the model to marginalize out object appearances exactly, resulting in training speeds up to 10x faster and vastly improved robustness to noisy or structured backgrounds.

The Bottleneck in Unsupervised Scene Understanding

The goal of unsupervised scene understanding is to teach a computer to look at an image and decompose it into individual objects—identifying "what" and "where" they are—without any human-provided labels. A landmark approach to this is Attend-Infer-Repeat (AIR), which uses a Recurrent Neural Network (RNN) to attend to objects sequentially.

However, AIR has a major bottleneck: it uses VAEs to represent object appearance. Because VAEs rely on approximate inference for their high-dimensional latent codes, the overall model becomes highly intractable. This leads to two main issues:

  1. Slow Convergence: The inference network has to learn to predict complex latent codes for every object, which requires a long time and suffers from high-variance gradients.
  2. Fragility: AIR often gets stuck in poor local optima, especially when the background isn't perfectly black or when objects overlap (occlusion).

The SuPAIR Solution

The authors introduce SuPAIR, which retains the high-level "vision as inverse graphics" structure of AIR but swaps its internal components for more efficient, tractable alternatives.

  • Sum-Product Networks (SPNs) as Object Models: Unlike VAEs, SPNs are deep probabilistic models that allow for exact and efficient marginalization. This means the model can calculate the likelihood of an image patch belonging to an object class without needing to explicitly "guess" a latent appearance code.
  • Rao-Blackwellization: By using SPNs, the authors can analytically integrate out the appearance variables. In machine learning terms, this is a form of Rao-Blackwellization. It drastically reduces the variance of the gradients during training, leading to much faster and more stable learning.
  • Cut-and-Paste Interaction: The original AIR simply sums up the pixel intensities of objects. SuPAIR uses a more realistic "cut-and-paste" model where objects in the foreground physically occlude objects (and the background) behind them.
  • Explicit Background Modeling: SuPAIR includes a dedicated SPN to model the background. This prevents the object-detection part of the network from "wasting" an object slot trying to explain away background noise.

Key Insights

  • Tractability over Complexity: The core insight is that choosing a slightly more constrained but tractable model (SPN) over a more flexible but intractable one (VAE) actually yields better performance. The optimization problem becomes significantly easier for the RNN to solve.
  • Reducing the Inference Burden: In SuPAIR, the RNN only needs to figure out where objects are and how many there are. It no longer needs to predict the "what" (the latent appearance code), as that is handled by the SPN's ability to compute marginals.
  • The Importance of Background Noise: Without a background model, unsupervised systems often try to explain noise as "very faint objects." By giving the model a dedicated background SPN to absorb noise, the object detectors become much more reliable.

Key Results

The model was evaluated on several benchmarks designed to test counting and localization, including Multi-MNIST, Sprites, Noisy-MNIST, and Grid-MNIST.

10x Speedup and Superior Accuracy

SuPAIR reaches high counting accuracy on Multi-MNIST nearly an order of magnitude faster than the original AIR. Ultimately, SuPAIR achieved 98.3% counting accuracy compared to AIR's 94.7%.

Count accuracy over time comparing SuPAIR and AIR

Extreme Robustness

The performance gap is most evident in challenging environments. On Noisy-MNIST and Grid-MNIST, the original AIR failed completely (0% accuracy), unable to distinguish objects from the structured background. In contrast, SuPAIR maintained 93.8% and 97.5% accuracy, respectively.

Qualitative comparison of SuPAIR and AIR on Noisy and Grid MNIST

Additionally, because of the deterministic cut-and-paste model, SuPAIR handles overlapping objects in a more physically consistent way than the additive model of AIR.

Limitations and Future Work

  • Scaling to Natural Images: While SPNs work beautifully for digits and sprites, scaling them to the complexity of high-resolution natural photographs (like ImageNet) remains a challenge for the field of tractable modeling.
  • Inductive Bias Dependency: SuPAIR requires some hyperparameter tuning to ensure the background SPN stays "simpler" than the object SPNs; otherwise, the model might try to explain objects as part of a complex background.
  • Sequential Data: The authors suggest extending this logic to video (sequential AIR), where tractable models could help track objects through time more efficiently.

Summary updated

Abstract

The recent Attend-Infer-Repeat (AIR) framework marks a milestone in structured probabilistic modeling, as it tackles the challenging problem of unsupervised scene understanding via Bayesian inferen...
BibTeX
@inproceedings{stelzner2019faster,
  author = {Karl Stelzner and Robert Peharz and Kristian Kersting},
  title = {{Faster Attend-Infer-Repeat with Tractable Probabilistic Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
}
Figure