Physics-Guided Fusion for Robust 3D Tracking of Fast Moving Small Objects

Prithvi Raj Singh, Raju Gottumukkala, Anthony S. Maida, Alan B. Barhorst, Vijaya GopuarXiv 2025 · DOI · arXiv

This paper, recently published by researchers from UL Lafayette and the Louisiana Transportation Research Center, tackles the "needle in a haystack" problem of computer vision: tracking tiny, high-speed objects in 3D space.

TL;DR

The researchers developed a robust 3D tracking system that fuses deep learning (YOLOv8) with a physics-informed kinematics model to track fast-moving small objects (FMOs). By explicitly modeling collision dynamics and using physics to fill in missing sensor data, the system reduces tracking error by up to 70% compared to state-of-the-art deep learning trackers.

What Problem Does This Paper Address?

While modern computer vision excels at tracking cars or pedestrians, it struggles significantly with Fast-Moving Small Objects (FMOs), such as a racquetball traveling at 180 mph.

Standard tracking-by-detection pipelines face three major hurdles here:

  1. Limited Spatial Features: A ball might only occupy 8x8 pixels, leaving very few discriminative features for a neural network to "grab" onto.
  2. Motion Blur: High velocities smear the object across frames, further degrading visual data.
  3. Erratic Trajectories: Unlike a car, a ball changes direction instantaneously upon hitting a wall. Standard trackers (like SORT or DeepOCSORT) assume relatively linear motion and often "lose" the object during these sudden bounces.

Methods

The authors propose a multi-stage system designed to run on edge hardware (NVIDIA Jetson TX2):

  • Detection: They utilize YOLOv8 for real-time 2D detection. After testing several models, YOLOv8 was chosen for its optimal balance of accuracy and inference speed (reaching 41 FPS at high resolution).
  • Depth Correction: Using an RGB-D camera (ZED 2i), the system retrieves 3D coordinates. However, depth sensors often fail on small, fast objects. The system includes a module that uses physics equations to estimate the Z-coordinate (depth) when the sensor returns "NaN" or noisy values.
  • Physics-Guided Tracking: Instead of relying solely on data-driven association, the system integrates kinematics motion equations. It predicts where the ball should be based on gravity and previous velocity.
  • Collision Response Modeling: This is a key innovation. The system explicitly models bounces using a Coefficient of Restitution (COR). When the ball's predicted path intersects a known boundary (floor or wall), the model calculates the post-collision velocity, allowing the tracker to "anticipate" the bounce rather than being confused by it.

Datasets

Because no suitable public datasets existed for this specific problem, the authors created a Custom Racquetball Dataset:

  • Content: 12 video sequences of high-intensity racquetball gameplay.
  • Scale: Over 80,000 frames, with 10,000 frames manually annotated with bounding boxes.
  • Complexity: Includes synchronized RGB and depth data, featuring ball speeds up to 180 mph and frequent collisions with walls, floors, and ceilings.

Key Insights and Ideas

  • Physics as a "Safety Net": The paper demonstrates that deep learning doesn't have to do all the heavy lifting. When visual features disappear due to blur or occlusion, the laws of physics provide a reliable fallback to maintain the trajectory.
  • Outlier Rejection via Kinematics: By comparing a YOLO detection against a physics-based prediction, the system can instantly reject "false positives" (like a mark on a wall) that don't follow a physically valid path.
  • The Importance of 3D Awareness: Most sports trackers operate in 2D. By incorporating depth and 3D boundaries, the system can model collisions much more accurately, leading to a massive jump in tracking stability.

Key Results

The system was evaluated across five challenging scenarios (multiple bounces, high-speed wall collisions, occlusions, etc.) against SOTA trackers like DeepOCSORT and ByteTrack:

  • Accuracy: The proposed system achieved an Average Displacement Error (ADE) of 8.42 pixels in complex bouncing scenarios, while the best deep-learning competitor (DeepOCSORT) lagged behind at 22.97 pixels.
  • Error Reduction: Overall, the physics-guided approach showed a 63-73% lower ADE than standard Kalman-filter-based trackers.
  • Real-time Performance: The pipeline maintains 41 FPS on an NVIDIA Jetson TX2, making it viable for real-world robotics or automated sports broadcasting.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Simplified Physics: The current model assumes a vacuum-like environment. Future iterations could incorporate air resistance, drag, and the Magnus effect (spin), which significantly influence ball flight at high speeds.
  • Dataset Diversity: The current dataset is limited to a specific racquetball court. Expanding to different lighting conditions and court materials would improve generalizability.
  • Robotics Integration: The ultimate goal is to deploy this on autonomous platforms (like mobile robots) to enable them to interact with fast-moving objects in real-time.

Summary updated

Abstract

While computer vision has advanced considerably for general object detection and tracking, the specific problem of fast-moving tiny objects remains underexplored. This paper addresses the significant challenge of detecting and tracking rapidly moving small objects using an RGB-D camera. Our novel system combines deep learning-based detection with physics-based tracking to overcome the limitations of existing approaches. Our contributions include: (1) a comprehensive system design for object detection and tracking of fast-moving small objects in 3D space, (2) an innovative physics-based tracking algorithm that integrates kinematics motion equations to handle outliers and missed detections, and (3) an outlier detection and correction module that significantly improves tracking performance in challenging scenarios such as occlusions and rapid direction changes. We evaluated our proposed system on a custom racquetball dataset. Our evaluation shows our system surpassing kalman filter based trackers with up to 70\% less Average Displacement Error. Our system has significant applications for improving robot perception on autonomous platforms and demonstrates the effectiveness of combining physics-based models with deep learning approaches for real-time 3D detection and tracking of challenging small objects.
BibTeX
@misc{singh2025physics,
  author = {Prithvi Raj Singh and Raju Gottumukkala and Anthony S. Maida and Alan B. Barhorst and Vijaya Gopu},
  title = {{Physics-Guided Fusion for Robust 3D Tracking of Fast Moving Small Objects}},
  year = {2025},
  eprint = {2510.20126},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.20126},
  howpublished = {arXiv},
}

Loading related content...

Figure