This paper introduces Neural Expectation Maximization (N-EM), a framework that marries the classic Expectation Maximization (EM) algorithm with deep learning to solve the "Binding Problem." By unrolling EM into a differentiable neural network, the authors demonstrate how a system can learn to segment and represent individual objects in a scene in a completely unsupervised manner.
TL;DR
Neural Expectation Maximization (N-EM) is a differentiable clustering framework that allows neural networks to perform unsupervised perceptual grouping. By treating objects as components in a spatial mixture model, it learns to "bind" pixels to specific entities, creating separate, symbol-like representations that improve next-step prediction and handle multi-object occlusions.
What Problem Does This Paper Address?
In machine learning, we often want "disentangled" representations—features that represent independent properties like shape or color. However, most models struggle when multiple objects are present. If you superimpose the representations of two different objects, you get an ambiguous "mess" where it is unclear which feature belongs to which object. This is known in neuroscience and AI as the Binding Problem.
To solve this, a model must perform perceptual grouping: it needs to dynamically decide which parts of an input (e.g., which pixels) belong to which entity. While humans do this effortlessly, teaching a machine to do it without labels (unsupervised) is incredibly difficult. Most existing unsupervised methods only work for single-object scenes or fail when objects overlap or move.
Methods
The authors formalize perceptual grouping as inference in a Parametrized Spatial Mixture Model.
- The Mixture Model: They treat an image as a composition of $K$ components. Each component $k$ is represented by a vector $\theta_k$.
- Neural Parametrization: Instead of using simple Gaussian distributions, they use a neural network $f_\phi$ to transform the representation $\theta_k$ into pixel-wise distribution parameters (like the mean of a Bernoulli or Gaussian).
- Differentiable EM: They use the Expectation Maximization framework to iteratively update the pixel assignments (the E-step) and the object representations (the M-step). Crucially, because the neural network is differentiable, they can replace the traditional M-step with a gradient ascent step.
- Unrolling and RNN-EM: By unrolling these iterations, the process looks like a Recurrent Neural Network (RNN). They introduce RNN-EM, which replaces the constrained gradient update with a fully parametrized RNN, providing more flexibility to learn complex grouping dynamics.
- The Objective Function: They use a two-term loss:
- Intra-cluster loss: A standard reconstruction loss weighted by pixel assignments.
- Inter-cluster loss: A KL-divergence penalty that prevents different components from trying to represent the same pixels, forcing them to specialize.
Datasets
The authors evaluated their method on three synthetic datasets designed to test grouping and temporal consistency:
- Static Shapes: $28 \times 28$ binary images containing three random shapes (triangles, squares, diamonds).
- Flying Shapes: A sequential version where these shapes move along random trajectories and bounce off walls, including frames where they overlap.
- Flying MNIST: A more complex grayscale sequence featuring two or three down-sampled MNIST digits moving and overlapping.
Key Insights and Ideas
- Objects as Common Causes: The core philosophy is that an "entity" is a common statistical cause for a group of observations. If knowing the state of one pixel helps you predict another, they likely belong to the same object.
- EM as Architecture: The most elegant insight is that the iterative nature of EM can be mapped directly onto the architecture of an RNN. This allows the model to "think" through the grouping process over several internal steps.
- Robustness to $K$: Surprisingly, the model is robust to the choice of $K$ (the number of possible objects). If $K$ is larger than the actual number of objects in the scene, the model simply learns to leave the extra components empty rather than splitting a single object into two.
- Prediction as a Grouping Signal: In the sequential tasks, the model uses "next-step prediction" as its training signal. To predict where a shape will be in the next frame, the model is forced to understand that the shape is a single, coherent entity.
Key Results
- Superior Grouping: On the Static Shapes task, RNN-EM achieved an Adjusted Mutual Information (AMI) score of 0.826, significantly outperforming the more rigid N-EM (0.475). For comparison, the contemporary "Tagger" model achieved 0.79 but required 20x more parameters.
- Handling Occlusion: In the Flying Shapes task, the model maintained high AMI (0.878 for 5 shapes) even when objects heavily overlapped.
- Generalization: A model trained on sequences with only two MNIST digits was able to successfully segment sequences with three digits at test time, achieving an AMI of 0.917 after stage-wise training.
- Better Prediction: By representing objects separately, RNN-EM achieved much lower prediction error (Binary Cross Entropy) compared to a standard recurrent autoencoder ($K=1$), which struggles to "track" multiple overlapping entities.
Limitations and Future Work
- Initialization Sensitivity: Like all clustering algorithms, N-EM is sensitive to how the pixel assignments ($\gamma$) are initialized.
- The "Object" Definition: Because the learning is unsupervised and based purely on statistics, the model's definition of an "object" might not always align with human intuition (e.g., it might group a shadow with an object if they always move together).
- Scalability: The experiments were limited to relatively small, synthetic images. Scaling this to high-resolution, natural images remains a significant challenge.
- Future Directions: The authors suggest extending the framework to hierarchical groupings (e.g., a car is an object, but its wheels are also sub-objects) and incorporating top-down attention mechanisms.
Summary updated