Deep Declarative Networks: A New Hope

Stephen Gould, Richard Hartley, Dylan CampbellarXiv 2019 · DOI · OpenReview

Abstract

We explore a new class of end-to-end learnable models wherein data processing nodes (or network layers) are defined in terms of desired behavior rather than an explicit forward function. Specifically, the forward function is implicitly defined as the solution to a mathematical optimization problem. Consistent with nomenclature in the programming languages community, we name these models deep declarative networks. Importantly, we show that the class of deep declarative networks subsumes current deep learning models. Moreover, invoking the implicit function theorem, we show how gradients can be back-propagated through many declaratively defined data processing nodes thereby enabling end-to-end learning. We show how these declarative processing nodes can be implemented in the popular PyTorch deep learning software library allowing declarative and imperative nodes to co-exist within the same network. We also provide numerous insights and illustrative examples of declarative nodes and demonstrate their application for image and point cloud classification tasks.

Introduction

In standard deep learning, layers are typically "imperative": they are defined by explicit functions, such as $y = \text{ReLU}(Wx + b)$. While these functions are computationally efficient and easy to differentiate, they struggle to incorporate complex domain-specific knowledge, such as physical laws, geometric constraints, or robust statistical estimators. Integrating such logic often requires "unrolling" iterative solvers, a process that is memory-intensive and prone to numerical instability.

Deep Declarative Networks (DDNs) address this by defining layers based on what they should achieve—the solution to an optimization problem—rather than how to compute them. This paradigm shift allows researchers to embed sophisticated mathematical logic directly into neural architectures.

The Deep Declarative Framework

The core innovation of DDNs is the declarative node. Unlike traditional nodes, a declarative node defines its output $y$ as the solution to an optimization problem: $$y \in \arg \min_{u \in C} f(x, u)$$ where $f$ is an objective function, $x$ is the input, and $C$ is a constraint set.

Architecture of a declarative node showing the forward pass (optimization) and the backward pass (implicit differentiation).

To enable end-to-end learning, the authors leverage implicit differentiation via the Implicit Function Theorem. Instead of backpropagating through the internal steps of a solver, gradients are calculated based on the optimality conditions (e.g., KKT conditions) at the solution point. This approach decouples the forward pass—which can utilize any state-of-the-art solver, including non-differentiable ones like RANSAC—from the backward pass, which uses a closed-form linear algebra expression.

Key Insights and Geometric Interpretation

A significant theoretical contribution of this work is the proof that traditional imperative layers are special cases of declarative nodes where the optimization problem has a trivial, closed-form solution. Furthermore, the framework is made computationally feasible for high-dimensional data by using Vector-Jacobian Products (VJP), avoiding the need to explicitly form large Jacobian matrices.

The paper also offers a compelling geometric interpretation of the gradients in constrained declarative nodes. For problems with constraints, the gradient of the loss with respect to the input is essentially the unconstrained gradient projected onto the tangent plane of the constraint surface.

Geometric representation of the gradient in a constrained optimization problem, illustrating the projection onto the tangent space.

Experimental Results

The authors demonstrated the power of DDNs in two key areas:

  1. Robustness to Outliers: On the ModelNet40 point cloud dataset, standard architectures like PointNet fail significantly when data is noisy. By replacing standard max-pooling with a declarative "robust pooling" node (using Welsch penalty functions), accuracy under 90% outlier conditions improved from 29.7% to 65.3%.
  2. Model Calibration: On ImageNet, integrating $L_2$-ball projection layers into a ResNet-18 backbone improved Mean Average Precision (mAP) from 58.97 to 72.43, suggesting that declarative constraints can help calibrate model confidence.

Conclusion and Future Work

Deep Declarative Networks provide a flexible and mathematically rigorous way to integrate optimization into deep learning. While the forward pass can be more computationally expensive than standard layers and requires twice-differentiable objectives for the current theory, the benefits in robustness and domain-integration are substantial. Future work aims to apply DDNs to complex tasks like geometric bundle adjustment in SLAM and model-predictive control in robotics.

Summary updated

Abstract

We explore a new class of end-to-end learnable models wherein data processing nodes (or network layers) are defined in terms of desired behavior rather than an explicit forward function. Specifically, the forward function is implicitly defined as the solution to a mathematical optimization problem. Consistent with nomenclature in the programming languages community, we name these models deep declarative networks. Importantly, we show that the class of deep declarative networks subsumes current deep learning models. Moreover, invoking the implicit function theorem, we show how gradients can be back-propagated through many declaratively defined data processing nodes thereby enabling end-to-end learning. We show how these declarative processing nodes can be implemented in the popular PyTorch deep learning software library allowing declarative and imperative nodes to co-exist within the same network. We also provide numerous insights and illustrative examples of declarative nodes and demonstrate their application for image and point cloud classification tasks.
BibTeX
@misc{gould2019deep,
  author = {Stephen Gould and Richard Hartley and Dylan Campbell},
  title = {{Deep Declarative Networks: A New Hope}},
  year = {2019},
  eprint = {1909.04866},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.04866},
}
Figure