Vision-based Discovery of Nonlinear Dynamics for 3D Moving Target

Zhang Zitong, Yang Liu, Hao SunSubmission to ICLR 2024 · Withdrawn

TL;DR Researchers have developed a vision-based framework that automatically uncovers the hidden differential equations governing 3D moving objects using only raw video footage. By fusing deep learning-based tracking with physics-informed splines, the system can distill precise mathematical laws even when the video is noisy or the object is partially obscured.

The Challenge: From 2D Pixels to 3D Physics

In the realm of data-driven discovery, scientists aim to find governing equations (like $F=ma$ or the Lorenz equations) directly from observations. While tools like SINDy (Sparse Identification of Nonlinear Dynamics) have become popular, they typically require "clean" trajectory data—precise $x, y, z$ coordinates over time.

In reality, we rarely have sensors attached to every moving object; instead, we rely on cameras. Translating 2D pixels in a video to 3D governing equations is a massive hurdle. It requires overcoming camera calibration issues, perspective shifts, measurement noise, and "data gaps" where an object might disappear behind an obstacle. While previous work in this area was largely limited to 2D motion, this paper tackles the full complexity of 3D space.

A Three-Stage Discovery Pipeline

The authors propose a comprehensive pipeline to bridge the gap between computer vision and symbolic physics.

Overview of the vision-based trajectory reconstruction pipeline, moving from multi-camera video tracking to 3D coordinate transformation.
  1. Target Tracking: The system uses YOLO-v8 to identify and track the centroid of a moving object across frames in multiple video feeds (typically three cameras).
  2. Coordinate Transformation Learning: Instead of requiring all cameras to be perfectly calibrated, the framework calibrates just one. It then uses Rodrigues’ rotation formula within a learning module to automatically determine the scaling and rotation factors for the other cameras, reconstructing the object's 3D trajectory in a unified reference frame.
  3. Spline-Enhanced Sparse Regression: To find the actual equations, the system doesn't just look at the raw 3D points. It uses Cubic B-Splines as a differentiable surrogate. These splines smooth out noise and fill in missing data points. Simultaneously, a sparse regressor selects the most likely terms (e.g., $x^2$, $\sin(y)$) from a library of candidate functions to build the final Ordinary Differential Equations (ODEs).

The system is trained using Alternate Direction Optimization (ADO), which forces the splines to fit the visual data while ensuring the resulting trajectory obeys the discovered physics.

Key Insights

  • Physics as a Denoiser: One of the most compelling findings is that discovering the equation actually improves tracking. By enforcing a sparse physical law, the model can better estimate where the object should be during noisy frames or occlusions.
  • Bridging the Gap with Splines: Unlike standard neural networks, B-splines provide a mathematically convenient way to calculate derivatives (velocity and acceleration) directly, which are essential for identifying ODEs.
  • Minimal Calibration: By learning the camera transformation parameters on the fly, the framework becomes highly flexible for real-world scenarios where setting up a perfectly calibrated multi-camera rig is impractical.

Performance and Results

The researchers evaluated their method using nine synthetic datasets representing famous nonlinear and chaotic systems, including the Lorenz system, Sprott (E and F), Rayleigh-Benard, and Nose-Hoover. To simulate realistic challenges, they rendered these trajectories into videos with varied backgrounds, irregular object shapes, self-rotation, and simulated occlusions.

Comparison of true versus discovered 3D trajectories for various chaotic systems, demonstrating the framework's high accuracy.

Tested against the standard PySINDy baseline, the framework demonstrated superior performance:

  • Accuracy: In most cases, it achieved 100% recall, successfully finding every correct term in the underlying physical equation.
  • Robustness to Noise: The system accurately identified governing equations even with up to 20% Gaussian noise added to the video data.
  • Handling Occlusions: Remarkably, the model remained stable even when 50% of the data was missing due to random blocks (occlusions) in the video.
  • Trajectory Reconstruction: The "discovered" trajectories nearly perfectly coincided with the ground truth, even for highly chaotic systems.

Limitations and Future Work

While the results are impressive, the authors note a few areas for future development. First, the method relies on a predefined library of candidate functions; if the true physics involves an exotic term not in the library, the model will struggle. Integrating symbolic regression could help "evolve" new terms. Second, the current study focuses on a single moving object, whereas real-world scenes often involve multiple interacting objects requiring coupled equations. Finally, the next major step is testing the framework on real-world footage, where lighting, motion blur, and complex shadows will introduce new layers of difficulty.

Summary updated

Abstract

Data-driven discovery of governing equations has kindled significant interests in many science and engineering areas. Existing studies primarily focus on uncovering equations that govern nonlinear dynamics based on direct measurement of the system states (e.g., trajectories). Limited efforts have been placed on distilling governing laws of dynamics directly from videos for moving targets in a 3D space. To this end, we propose a vision-based approach to automatically uncover governing equations of nonlinear dynamics for 3D moving targets via raw videos recorded by a set of cameras. The approach is composed of three key blocks: (1) a target tracking module that extracts plane pixel motions of the moving target in each video, (2) a Rodrigues' rotation formula-based coordinate transformation learning module that reconstructs the 3D coordinates with respect to a predefined reference point, and (3) a spline-enhanced library-based sparse regressor that uncovers the underlying governing law of dynamics. This framework is capable of effectively handling the challenges associated with measurement data, e.g., noise in the video, imprecise tracking of the target that causes data missing, etc. The efficacy of the proposed method has been demonstrated through multiple sets of synthetic videos considering different nonlinear dynamics.
BibTeX
@misc{zitong2024vision,
  author = {Zhang Zitong and Yang Liu and Hao Sun},
  title = {{Vision-based Discovery of Nonlinear Dynamics for 3D Moving Target}},
  year = {2024},
  eprint = {2404.17865},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.17865},
}
Figure