NODIS: Neural Ordinary Differential Scene Understanding

Cong Yuren, Hanno Ackermann, Wentong Liao, Michael Ying Yang, Bodo RosenhahnarXiv 2020 · arXiv

Abstract

Scene Graph Generation (SGG) is a fundamental task in computer vision that involves identifying objects and their semantic relationships within an image. Traditional approaches often treat SGG as a discrete assignment problem, which can be difficult to integrate into end-to-end deep learning pipelines due to non-differentiability. This paper introduces NODIS (Neural Ordinary Differential Scene Understanding), which reformulates SGG as a continuous refinement process governed by Neural Ordinary Differential Equations (ODEs). By evolving features along a continuous trajectory, NODIS achieves state-of-the-art results on the Visual Genome benchmark, particularly in Scene Graph Classification (SGCLS).

Introduction

Scene Graph Generation aims to produce a structured representation of a scene, capturing relationships such as "man wearing hat" or "dog next to tree." Historically, researchers used (Mixed-)Integer Linear Programs (MILPs) to find optimal labels. However, MILPs are discrete and non-differentiable, preventing backpropagation from the final assignment back to the feature extractor. Previous end-to-end attempts used heuristics like message-passing or fixed-step recurrent neural networks (RNNs), which often lack the mathematical rigor of optimization-based methods.

Methodology

NODIS bridges the gap between optimization and deep learning by interpreting the assignment problem as an ODE. The architecture consists of four main stages: feature extraction, object classification, predicate prediction, and final graph generation.

The NODIS architecture, showing the flow from feature extraction to ODE-based classification modules (O-ODE and P-ODE).

Instead of a fixed stack of layers, NODIS uses two specialized modules: O-ODE (for object classification) and P-ODE (for predicate/relationship classification). While standard networks jump from one discrete representation to another, NODIS learns a continuous trajectory on an embedding manifold. Inside the ODE solver, the derivative function is modeled using a bidirectional LSTM, allowing the model to capture dependencies between different objects and relationships in the scene. Because the ODE solver is differentiable via the adjoint sensitivity method, the entire pipeline is trainable end-to-end.

Key Insights: Optimization as Flow

The central innovation of NODIS is viewing optimization as a flow. In this framework, the "time" variable $t$ represents the refinement process. As $t$ increases, the model iteratively improves its predictions, moving from a noisy initial guess to a dense, accurate representation.

Comparison between discrete refinement steps (left) and the continuous ODE trajectory (right) on an embedding manifold.

Ablation studies revealed that the power of the model lies primarily in the ODE formulation itself. Even simple linear layers inside the ODE module performed nearly as well as complex LSTMs, suggesting that the continuous refinement process is the key driver of performance.

Experiments and Results

The model was evaluated on the Visual Genome (VG) benchmark, the gold standard for scene graph tasks, involving 150 object categories and 50 predicate types. NODIS achieved impressive quantitative results, particularly in the SGCLS task:

  • State-of-the-Art Performance: On SGCLS, NODIS improved the Recall@50 metric to 41.9%, outperforming previous top models like CMAT (~39%) and MotifNet (~35%).
  • Robustness: The model showed strong results across all three standard tasks (PREDCLS, SGCLS, and SGGEN), proving that Neural ODEs are a viable alternative to message-passing heuristics.

Limitations and Future Work

Despite its success, NODIS faces several challenges. It is limited by the underlying object detector (Faster R-CNN); if the detector fails to propose a box, the ODE modules cannot recover that information. Additionally, performance on the PREDCLS task has plateaued across the field due to significant noise and missing labels in the Visual Genome dataset. Finally, while effective, ODE solvers can introduce computational overhead compared to fixed-depth networks.

Conclusion

NODIS provides a compelling mathematical framework for scene understanding, proving that the "black box" of neural networks can be successfully married to the classical logic of assignment problems through continuous refinement.

Summary updated

Abstract

Semantic image understanding is a challenging topic in computer vision. It requires to detect all objects in an image, but also to identify all the relations between them. Detected objects, their labels and the discovered relations can be used to construct a scene graph which provides an abstract semantic interpretation of an image. In previous works, relations were identified by solving an assignment problem formulated as Mixed-Integer Linear Programs. In this work, we interpret that formulation as Ordinary Differential Equation (ODE). The proposed architecture performs scene graph inference by solving a neural variant of an ODE by end-to-end learning. It achieves state-of-the-art results on all three benchmark tasks: scene graph generation (SGGen), classification (SGCls) and visual relationship detection (PredCls) on Visual Genome benchmark.
BibTeX
@misc{yuren2020nodis,
  author = {Cong Yuren and Hanno Ackermann and Wentong Liao and Michael Ying Yang and Bodo Rosenhahn},
  title = {{NODIS: Neural Ordinary Differential Scene Understanding}},
  year = {2020},
  eprint = {2001.04735},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.04735},
}
Figure