Introduction
In fields such as biomechanics, digital animation, and autonomous driving, the structural movement of an object often carries more critical information than its visual texture. This structural data is typically captured through skeletonization—the process of reducing a complex shape to its central "bones" or medial axis. However, traditional skeletonization algorithms are often restricted by the requirement for pre-segmented images with clean backgrounds. In the context of high-speed, live video streaming, these requirements create significant bottlenecks.
SkelAI is a full-stack framework designed to overcome these limitations by performing real-time object detection and medial axis skeletonization directly from live video. By integrating state-of-the-art computer vision with custom geometric algorithms, SkelAI provides a robust pipeline for converting raw video into lightweight, motion-ready data.
Methodology
SkelAI utilizes a multi-stage pipeline implemented with a ReactJS frontend and a Flask/Python backend. The process moves from raw pixels to a geometric skeleton through the following steps:
- Object Detection & Segmentation: The system employs YOLOv8 ('yolov8n-seg') to identify objects and generate precise segmentation masks in real-time, eliminating the need for manual pre-masking.
- Boundary Extraction: Using OpenCV, the system extracts the contours of the detected objects from the YOLO-generated masks.
- Noise Reduction: A Gaussian filter is applied to smooth the contours, preventing the creation of "spurious branches" or artifacts in the final skeleton.
- Medial Axis Calculation: A custom algorithm calculates the geometric center-lines of the smoothed shape.
- Synthetic Data Generation: A unique feature of the framework is its integration with the OpenAI DALL-E 3 API. This allows for the generation of synthetic scenarios (e.g., specific athletic movements) to create training data when real-world footage is scarce.
- Data Export: The resulting skeletons are exported in PyTorch-compatible formats for use in deep learning models.
Key Insights: Skeletonization as Compression
A core finding of this research is the efficiency of skeletonization as a data compression technique. By representing a complex object as a series of medial axis points rather than a full bitmap or mask, the system achieves massive data reduction. This context-agnostic approach allows SkelAI to generate accurate motion data regardless of background complexity, removing the "pre-masked" bottleneck that has historically hindered real-time applications.
Furthermore, the integration of generative AI (DALL-E 3) creates a powerful feedback loop. Researchers can simulate rare or dangerous edge cases—such as specific sports injuries or pedestrian near-misses—to train motion prediction models without requiring dangerous real-world data collection.
Results and Performance
The system was evaluated using both live video streams and synthetic datasets. Key results include:
- High Compression Efficiency: The system achieved an average compression ratio of 39:1. As shown in the performance data, original image representations were reduced to a fraction of their size while maintaining structural integrity.
- Real-Time Performance: Unlike previous iterations that were limited to static images, SkelAI successfully maintains processing speeds suitable for live streaming environments.
- Noise Resilience: The combination of CNN-based segmentation and Gaussian filtering allowed the system to remain stable even in visually noisy environments.
Conclusion and Future Work
SkelAI represents a significant advancement in real-time motion analysis. By viewing skeletonization as both a geometric and a compression problem, the framework enables new possibilities in actionable analytics, such as the automatic detection of sports injury mechanics.
Future work will focus on expanding the system to support 3D object skeletonization, which is essential for advanced robotics. Additionally, the authors plan to integrate Transformer models to better utilize the exported skeleton data for long-term motion forecasting and apply the technology to autonomous vehicle systems for improved pedestrian path prediction.
Summary updated