Abstract
Scene Graph Generation (SGG) is a fundamental task in computer vision that involves identifying objects and their semantic relationships within an image. Traditional approaches often treat SGG as a discrete assignment problem, which can be difficult to integrate into end-to-end deep learning pipelines due to non-differentiability. This paper introduces NODIS (Neural Ordinary Differential Scene Understanding), which reformulates SGG as a continuous refinement process governed by Neural Ordinary Differential Equations (ODEs). By evolving features along a continuous trajectory, NODIS achieves state-of-the-art results on the Visual Genome benchmark, particularly in Scene Graph Classification (SGCLS).
Introduction
Scene Graph Generation aims to produce a structured representation of a scene, capturing relationships such as "man wearing hat" or "dog next to tree." Historically, researchers used (Mixed-)Integer Linear Programs (MILPs) to find optimal labels. However, MILPs are discrete and non-differentiable, preventing backpropagation from the final assignment back to the feature extractor. Previous end-to-end attempts used heuristics like message-passing or fixed-step recurrent neural networks (RNNs), which often lack the mathematical rigor of optimization-based methods.
Methodology
NODIS bridges the gap between optimization and deep learning by interpreting the assignment problem as an ODE. The architecture consists of four main stages: feature extraction, object classification, predicate prediction, and final graph generation.
Instead of a fixed stack of layers, NODIS uses two specialized modules: O-ODE (for object classification) and P-ODE (for predicate/relationship classification). While standard networks jump from one discrete representation to another, NODIS learns a continuous trajectory on an embedding manifold. Inside the ODE solver, the derivative function is modeled using a bidirectional LSTM, allowing the model to capture dependencies between different objects and relationships in the scene. Because the ODE solver is differentiable via the adjoint sensitivity method, the entire pipeline is trainable end-to-end.
Key Insights: Optimization as Flow
The central innovation of NODIS is viewing optimization as a flow. In this framework, the "time" variable $t$ represents the refinement process. As $t$ increases, the model iteratively improves its predictions, moving from a noisy initial guess to a dense, accurate representation.
Ablation studies revealed that the power of the model lies primarily in the ODE formulation itself. Even simple linear layers inside the ODE module performed nearly as well as complex LSTMs, suggesting that the continuous refinement process is the key driver of performance.
Experiments and Results
The model was evaluated on the Visual Genome (VG) benchmark, the gold standard for scene graph tasks, involving 150 object categories and 50 predicate types. NODIS achieved impressive quantitative results, particularly in the SGCLS task:
- State-of-the-Art Performance: On SGCLS, NODIS improved the Recall@50 metric to 41.9%, outperforming previous top models like CMAT (~39%) and MotifNet (~35%).
- Robustness: The model showed strong results across all three standard tasks (PREDCLS, SGCLS, and SGGEN), proving that Neural ODEs are a viable alternative to message-passing heuristics.
Limitations and Future Work
Despite its success, NODIS faces several challenges. It is limited by the underlying object detector (Faster R-CNN); if the detector fails to propose a box, the ODE modules cannot recover that information. Additionally, performance on the PREDCLS task has plateaued across the field due to significant noise and missing labels in the Visual Genome dataset. Finally, while effective, ODE solvers can introduce computational overhead compared to fixed-depth networks.
Conclusion
NODIS provides a compelling mathematical framework for scene understanding, proving that the "black box" of neural networks can be successfully married to the classical logic of assignment problems through continuous refinement.
Summary updated