Deep Declarative Networks

Stephen Gould, Richard I. Hartley, Dylan CampbellIEEE TPAMI 2022 · OpenReview

TL;DR

This paper introduces "Deep Declarative Networks" (DDNs), a novel framework for building neural networks where layers are defined by their desired behavior (e.g., solving an optimization problem) rather than an explicit function. This allows for more principled integration of existing mathematical models and constraints into deep learning, enabling end-to-end learning through implicit differentiation.

What Problem Does This Paper Address?

Traditional deep learning models define each layer as an explicit function that transforms input to output. While powerful, this approach can be limiting when dealing with problems that are naturally expressed as optimization problems (e.g., geometric fitting, model predictive control) or when hard constraints need to be enforced. Re-learning these behaviors from scratch can be inefficient and may not guarantee desired properties. This paper addresses the challenge of seamlessly integrating these optimization-based behaviors into end-to-end trainable deep learning architectures, bridging the gap between traditional machine learning and established mathematical optimization techniques.

Methods

The core innovation is the concept of "declarative nodes," where a layer's forward pass is defined as the solution to an optimization problem. The key technical contribution is demonstrating how to backpropagate gradients through these declarative nodes using the implicit function theorem. This allows for end-to-end learning by treating the optimization problem's solution as an implicit function of the input and parameters. The authors provide theoretical results for computing these gradients for various types of optimization problems, including unconstrained, equality-constrained, and inequality-constrained scenarios. They also show how these declarative nodes can coexist with traditional "imperative" (explicitly defined) nodes within a single network, implemented in PyTorch.

Datasets

  • ModelNet40: Used for point cloud classification experiments with robust pooling.
  • ImageNet 2012: Used for image classification experiments with Lp-sphere and Lp-ball projection layers.

Key Insights and Ideas

  • Declarative Definition of Layers: Defining network layers by what they should achieve (solving an optimization problem) rather than how they achieve it (an explicit function) offers a more flexible and principled approach.
  • Implicit Differentiation for Gradients: The implicit function theorem provides a powerful tool to compute gradients through optimization problems without needing to explicitly unroll the solver, making end-to-end learning feasible.
  • Unified Framework: DDNs provide a unifying framework that encompasses existing deep learning models and a wide range of optimization-based tasks, from geometric model fitting to control algorithms.
  • Guaranteed Properties: Declarative nodes allow for the enforcement of hard constraints (e.g., valid rotations, normalized matrices), leading to more robust and interpretable models.
  • Composability: DDNs are inherently composable, allowing for complex pipelines of optimization problems to be nested and trained end-to-end.

Key Results

The paper demonstrates the effectiveness of DDNs on image and point cloud classification tasks.

  • Point Cloud Classification: Robust pooling layers, implemented as declarative nodes, significantly outperformed standard max pooling and quadratic pooling when dealing with outliers, achieving higher accuracy and mean Average Precision (mAP) on the ModelNet40 dataset, especially under high outlier rates.
  • Image Classification: Projection layers (Lp-sphere and Lp-ball) integrated into ResNet architectures improved mAP significantly, suggesting better model calibration and confidence in predictions.

Limitations and Future Work

  • Non-unique Solutions: The paper acknowledges that non-unique solutions to optimization problems can lead to difficulties in end-to-end learning, potentially causing oscillations. Strategies like minimal parametrization or adding proximal terms are suggested.
  • Non-smooth Objectives/Constraints: While the paper discusses handling non-smoothness using Clarke generalized gradients, further theoretical development is needed.
  • Computational Cost: Solving optimization problems in the forward pass can be computationally expensive, and efficient solvers are crucial. The paper suggests exploring trade-offs between exact gradients and efficient approximations.
  • Feasibility Issues: Parameterizations that lead to infeasible optimization problems during training need careful consideration.
  • Future Directions: The authors suggest exploring the combination of ODEs with constrained optimization, developing more sophisticated techniques for non-smooth problems, and investigating the theoretical guarantees of DDNs for safety-critical applications.

Summary updated

Abstract

We explore a class of end-to-end learnable models wherein data processing nodes (or network layers) are defined in terms of desired behavior rather than an explicit forward function. Specifically, the forward function is implicitly defined as the solution to a mathematical optimization problem. Consistent with nomenclature in the programming languages community, we name these models <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">deep declarative networks</i> . Importantly, it can be shown that the class of deep declarative networks subsumes current deep learning models. Moreover, invoking the implicit function theorem, we show how gradients can be back-propagated through many declaratively defined data processing nodes thereby enabling end-to-end learning. We discuss how these declarative processing nodes can be implemented in the popular PyTorch deep learning software library allowing declarative and imperative nodes to co-exist within the same network. We also provide numerous insights and illustrative examples of declarative nodes and demonstrate their application for image and point cloud classification tasks.
BibTeX
@article{gould2022deep,
  author = {Stephen Gould and Richard I. Hartley and Dylan Campbell},
  title = {{Deep Declarative Networks}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=9v4YSquhlm},
}
Figure