Real-Time Object Detection and Skeletonization for Motion Prediction in Video Streaming

Gavin Wong, Yulia Kumar, J. Jenny Li, Dov KrugerAAAI 2025 · DOI

Introduction

In fields such as biomechanics, digital animation, and autonomous driving, the structural movement of an object often carries more critical information than its visual texture. This structural data is typically captured through skeletonization—the process of reducing a complex shape to its central "bones" or medial axis. However, traditional skeletonization algorithms are often restricted by the requirement for pre-segmented images with clean backgrounds. In the context of high-speed, live video streaming, these requirements create significant bottlenecks.

SkelAI is a full-stack framework designed to overcome these limitations by performing real-time object detection and medial axis skeletonization directly from live video. By integrating state-of-the-art computer vision with custom geometric algorithms, SkelAI provides a robust pipeline for converting raw video into lightweight, motion-ready data.

Methodology

SkelAI utilizes a multi-stage pipeline implemented with a ReactJS frontend and a Flask/Python backend. The process moves from raw pixels to a geometric skeleton through the following steps:

  1. Object Detection & Segmentation: The system employs YOLOv8 ('yolov8n-seg') to identify objects and generate precise segmentation masks in real-time, eliminating the need for manual pre-masking.
  2. Boundary Extraction: Using OpenCV, the system extracts the contours of the detected objects from the YOLO-generated masks.
  3. Noise Reduction: A Gaussian filter is applied to smooth the contours, preventing the creation of "spurious branches" or artifacts in the final skeleton.
  4. Medial Axis Calculation: A custom algorithm calculates the geometric center-lines of the smoothed shape.
  5. Synthetic Data Generation: A unique feature of the framework is its integration with the OpenAI DALL-E 3 API. This allows for the generation of synthetic scenarios (e.g., specific athletic movements) to create training data when real-world footage is scarce.
  6. Data Export: The resulting skeletons are exported in PyTorch-compatible formats for use in deep learning models.
The multi-stage SkelAI pipeline, from initial YOLOv8 segmentation to the final skeleton overlay.

Key Insights: Skeletonization as Compression

A core finding of this research is the efficiency of skeletonization as a data compression technique. By representing a complex object as a series of medial axis points rather than a full bitmap or mask, the system achieves massive data reduction. This context-agnostic approach allows SkelAI to generate accurate motion data regardless of background complexity, removing the "pre-masked" bottleneck that has historically hindered real-time applications.

Furthermore, the integration of generative AI (DALL-E 3) creates a powerful feedback loop. Researchers can simulate rare or dangerous edge cases—such as specific sports injuries or pedestrian near-misses—to train motion prediction models without requiring dangerous real-world data collection.

Results and Performance

The system was evaluated using both live video streams and synthetic datasets. Key results include:

  • High Compression Efficiency: The system achieved an average compression ratio of 39:1. As shown in the performance data, original image representations were reduced to a fraction of their size while maintaining structural integrity.
  • Real-Time Performance: Unlike previous iterations that were limited to static images, SkelAI successfully maintains processing speeds suitable for live streaming environments.
  • Noise Resilience: The combination of CNN-based segmentation and Gaussian filtering allowed the system to remain stable even in visually noisy environments.
Compression metrics comparing original image size to medial axis points, showing an average 39:1 ratio.

Conclusion and Future Work

SkelAI represents a significant advancement in real-time motion analysis. By viewing skeletonization as both a geometric and a compression problem, the framework enables new possibilities in actionable analytics, such as the automatic detection of sports injury mechanics.

Future work will focus on expanding the system to support 3D object skeletonization, which is essential for advanced robotics. Additionally, the authors plan to integrate Transformer models to better utilize the exported skeleton data for long-term motion forecasting and apply the technology to autonomous vehicle systems for improved pedestrian path prediction.

Summary updated

BibTeX
@inproceedings{wong2025real,
  author = {Gavin Wong and Yulia Kumar and J. Jenny Li and Dov Kruger},
  title = {{Real-Time Object Detection and Skeletonization for Motion Prediction in Video Streaming}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35376},
}
Figure