Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation

Michael Chang, Thomas L. Griffiths, Sergey Levine

TL;DR

Training models to recognize independent objects in a scene often requires "iterative refinement"—starting with a guess and polishing it to break symmetries among plausible explanations. This paper demonstrates that by treating the final object representations as fixed points and using implicit differentiation, we can train these models much more stably, with less memory, and without the fragile "optimization tricks" (like heavy gradient clipping) usually required.

The Problem with Unrolling

In object-centric learning, models try to decompose an image into a set of "slots," where each slot represents one object. Because objects can appear in any order, these models use iterative refinement (like Slot Attention) to settle on a specific explanation for the scene.

The standard way to train these models is to "unroll" the iterations and backpropagate through every single step. This is problematic for two main reasons:

  1. Instability: Gradients tend to explode as they travel back through many recursive steps, reflected by a high Jacobian spectral norm.
  2. Efficiency: It requires storing every intermediate state in memory, which is computationally expensive.

To make this work in practice, researchers usually rely on a "dark art" of hyperparameters: learning rate warmups, heavy gradient clipping, and careful decay schedules. This paper aims to replace these hacks with a mathematically grounded approach.

Methods: Implicit Slot Attention

The authors propose Implicit Slot Attention. Instead of differentiating through the entire unrolled path of the refinement process, they leverage the Implicit Function Theorem (IFT).

If the iterative process converges, the final output is a fixed point—a state where further updates do not meaningfully change the result. The IFT allows us to calculate the gradient of the loss with respect to the model weights using only the final fixed point, completely ignoring the intermediate steps.

Comparison of vanilla unrolled backpropagation versus implicit differentiation, alongside stability and loss metrics.

Specifically, they use a first-order Neumann approximation for the implicit gradient. In practice, this is incredibly simple to implement:

  • Forward Pass: Run the refinement as usual until it reaches a fixed point (without tracking gradients).
  • Backward Pass: Perform one additional "dummy" iteration at the fixed point and differentiate through just that single step.

This effectively decouples the forward and backward passes, leading to constant space and time complexity during backpropagation.

Key Insights and Results

The authors evaluated their method using the SLATE architecture across three benchmarks: CLEVR-Mirror (synthetic geometric objects with reflections), Shapestacks (3D stacked shapes), and COCO-2017 (complex natural scenes).

  • The "One Line of Code" Fix: The transition from vanilla to implicit slot attention is computationally trivial to implement but fundamentally changes the optimization landscape.
  • Stability via Decoupling: By not backpropagating through the unrolled iterations, the "gradient explosion" problem is bypassed. The Jacobian spectral norm remains low and stable throughout training.
  • Removing the "Tricks": With implicit differentiation, learning rate warmups and gradient clipping can often be removed entirely without the model crashing.
  • Superior Reconstruction: On CLEVR-Mirror, Implicit SLATE achieved a Mean Squared Error (MSE) of 10.66, compared to 67.04 for the vanilla version—nearly a 7x improvement.
  • Robustness: The model is far less sensitive to the number of refinement iterations used during training. While vanilla slot attention often fails if you increase the number of steps, the implicit version remains stable.

Limitations and Future Work

The primary limitation of this method is its reliance on convergence assumptions. The implicit differentiation approach assumes the refinement process actually reaches a fixed point.

Empirical observation of the forward residual approaching zero during training.

While there is no formal proof that Slot Attention must converge under all conditions, the authors observe this empirically (as seen in the forward residual dropping toward zero).

Looking forward, while performance improved on the COCO dataset, perfectly segmenting complex real-world scenes into objects remains an open challenge. However, the authors suggest this implicit approach could be broadly applied to other iterative models—such as diffusion models or energy-based models—where unrolling is currently a major optimization bottleneck.

Summary updated

Abstract

Current work in object-centric learning has been motivated by developing learning algorithms that infer independent and symmetric entities from the perceptual input. This often requires the use iterative refinement procedures that break symmetries among equally plausible explanations for the data, but most prior works differentiate through the unrolled refinement process, which can make optimization exceptionally challenging. In this work, we observe that such iterative refinement methods can be made differentiable by means of the implicit function theorem, and develop an implicit differentiation approach that improves the stability and tractability of training such models by decoupling the forward and backward passes. This connection enables us to apply recent advances in optimizing implicit layers to not only improve the stability and optimization of the slot attention module in SLATE, a state-of-the-art method for learning entity representations, but do so with constant space and time complexity in backpropagation and only one additional line of code.
BibTeX
@inproceedings{chang2022object,
  author = {Michael Chang and Thomas L. Griffiths and Sergey Levine},
  title = {{Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=-5rFUTO2NWe},
}
Figure