Differentiable Mathematical Programming for Object-Centric Representation Learning

Adeel Pervez, Phillip Lippe, Efstratios Gavves

Introduction: The Topology Gap in Object Discovery

Object-centric representation learning aims to decompose visual scenes into discrete "slots," where each slot corresponds to a single object. While models like Slot Attention have shown promise, they often treat image features as an unordered set, neglecting the spatial relationships between pixels. This lack of "topology awareness" makes it difficult for models to distinguish objects in complex, textured environments where color alone is an insufficient cue.

Classical computer vision addressed this through graph cuts, partitioning images by finding minimum-cost edges to sever. However, these combinatorial optimizations are typically non-differentiable and computationally expensive, making them incompatible with end-to-end deep learning. This paper bridges that gap by formulating feature partitioning as a series of differentiable $s-t$ graph cuts.

Methodology: Differentiable $k$-Partitioning

The authors propose a framework that treats object discovery as a partitioning problem solved via Differentiable Mathematical Programming. The core innovation is the $k$-partition module, which represents an image as a grid graph where nodes are pixels and edges represent neighborhood relationships.

The k-partition module integrates a differentiable Quadratic Programming (QP) solver to generate spatially coherent partitioning masks from feature maps.

Relaxing the Combinatorial Problem

To make graph cuts compatible with backpropagation, the authors relax the discrete problem into a regularized equality-constrained Quadratic Program (QP). By applying the implicit function theorem, gradients can flow through the optimization layer. To ensure efficiency during training, the authors leverage the static nature of the image grid. By pre-computing the Cholesky factorization of the KKT matrix, they reduce a complex optimization into a series of fast, sparse linear system solves. This allows the model to handle thousands of variables (e.g., for $64 \times 64$ images) in real-time.

Key Results and Empirical Performance

The model was benchmarked against standard datasets like Clevr and more challenging textured variants like ClevrTex and CAMO (camouflaged objects).

Quantitative comparison across benchmarks. The proposed method significantly outperforms existing pixel-space and glimpse-based methods on textured (ClevrTex) and camouflaged (CAMO) datasets.

Superiority on Complex Textures

On the ClevrTex dataset, the proposed method achieved an ARI-FG of 80.2%, a substantial improvement over Slot Attention’s 62.4%. This performance gap highlights that explicitly encoding spatial continuity is essential when objects and backgrounds share similar textures. Furthermore, in "camouflaged" settings where objects are nearly indistinguishable from their surroundings, the model maintained a high ARI-FG of 75.5%, whereas most competing pixel-space methods failed to segment the scene effectively.

Temporal Consistency in Video

Beyond static images, the framework extends to video via differentiable bipartite matching. On the MOVi-A dataset, the model achieved a foreground ARI of 65.6%, vastly outperforming standard Slot Attention (4.2%), which often fails to maintain slot consistency as objects move.

Conclusion and Future Work

By treating optimization as a neural network layer, this work demonstrates that classical structural priors can be effectively integrated into modern deep learning. While the current approach requires a fixed number of slots ($k$) and faces scaling challenges at very high resolutions, it sets a new standard for topology-aware object discovery. Future research may focus on hierarchical approximations to scale to megapixel resolutions and dynamic slot allocation.

Summary updated

BibTeX
@inproceedings{pervez2023differentiable,
  author = {Adeel Pervez and Phillip Lippe and Efstratios Gavves},
  title = {{Differentiable Mathematical Programming for Object-Centric Representation Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=1J-ZTr7aypY},
}
Figure