TL;DR
This paper introduces a novel hybrid approach for real-time multi-object detection and tracking directly from H.264/AVC video bitstreams. By combining compressed domain processing with partial decoding of object regions, the method achieves high efficiency and robustness, even in challenging scenarios like long occlusions and articulated objects, which are difficult for existing compressed domain techniques.
What Problem Does This Paper Address?
Traditional object detection and tracking algorithms often operate on raw pixel data. While accurate, these "pixel domain" methods are computationally expensive, making them unsuitable for large-scale, real-time surveillance systems that require processing multiple video streams simultaneously. Furthermore, video data is frequently transmitted in compressed formats like H.264/AVC. Applying pixel domain methods to compressed video necessitates a full decoding step, adding significant computational overhead.
Compressed domain algorithms aim to reduce this burden by exploiting information directly from the bitstream, such as motion vectors. However, these methods often suffer from lower accuracy and struggle with complex scenarios. Motion vectors are block-based, not pixel-based, and can be unreliable or sparse, leading to difficulties in accurately segmenting and tracking objects, especially those that are articulated, change size, or have monotonous colors. Handling occlusions is also a significant challenge for these approaches. This paper addresses the need for a fast, accurate, and robust object detection and tracking system that can operate directly on H.264/AVC bitstreams, overcoming the limitations of both pure pixel domain and existing compressed domain methods.
Methods
The proposed method employs a two-phase hierarchical approach:
- Extraction Phase: This phase focuses on rapidly detecting and roughly tracking objects using a Probabilistic Spatiotemporal Macroblock Filtering (PSMF) technique.
- Block Clustering: It begins by identifying non-skip macroblocks (likely to contain moving objects) and clustering them into "block groups" based on connectivity.
- Spatial Filtering: These block groups are then filtered to remove those likely belonging to the background, based on integer transform (IT) coefficients and spatial structure.
- Temporal Filtering: Surviving "active block groups" are further filtered by analyzing their temporal consistency over an observation period. This step uses a Bayesian approach to estimate the probability of an entity being a real object versus background, merging coherent groups into potential object entities.
- Refinement Phase: This phase aims to accurately refine object trajectories.
- Partial Decoding and Background Subtraction (I-frames): For keyframes (I-frames), instead of full decoding, only the estimated object regions are partially decoded. This partially decoded image is then used with a background subtraction technique to obtain a more precise object blob. This partial decoding is made possible by substituting neighboring block pixels with background image pixels, a technique applicable to scenes with stationary backgrounds.
- Motion Interpolation (P-frames): For predictive frames (P-frames), motion interpolation is used to refine the object's location and size. This is based on the assumption that object motion is relatively smooth between I-frames and subsequent P-frames. The feature vector (location and size) is linearly interpolated between the refined I-frame information and the previous P-frame's estimated blob.
- Occlusion Handling: A method is introduced to handle occlusions. During occlusion, the system stores the hue color histogram of the object before occlusion (prior color distribution). After disocclusion, the hue color histogram of the recovered object is compared to stored prior distributions to identify the correct object.
Datasets
Not discussed in the paper. The authors mention using "two video sequences, which were taken by one fixed camera indoors and outdoors" for testing. These sequences were encoded using the JM12.4 reference software with the AVC baseline profile.
Key Insights and Ideas
- Hybrid Approach is Key: The paper demonstrates that combining compressed domain techniques (for speed) with pixel domain refinement (for accuracy) is a powerful strategy for real-time video analysis.
- Partial Decoding for Efficiency: The core innovation is the effective use of partial decoding in H.264/AVC bitstreams, specifically for object regions. This significantly reduces computational cost compared to full decoding, making real-time processing feasible.
- Robustness to Complex Scenes: The PSMF and the subsequent refinement steps allow the algorithm to handle challenging scenarios like articulated objects, objects with monotonous colors, and long occlusions, which are major limitations for many existing compressed domain trackers.
- Color Histograms for Identity: Using hue color histograms for occlusion handling proves effective in re-identifying objects after they reappear, a crucial feature for robust tracking.
- Overcoming H.264/AVC I-frame Decoding Challenges: The paper presents a method to overcome the inherent difficulty of decoding I-frames in H.264/AVC due to spatial prediction dependencies, by using background image pixels as references.
Key Results
- Real-time Performance: The proposed algorithm achieved frame rates of 49.5 fps (indoor) and 37.12 fps (outdoor) on a PC with a 3.2 GHz Pentium 4 CPU. This is significantly faster than full decoding, which yielded 20.46 fps (indoor) and 19.17 fps (outdoor).
- Extraction Efficiency: Spatial filtering removed a substantial portion of block groups: 70.8% in the indoor sequence and 64.2% in the outdoor sequence. Temporal filtering further reduced active group trains by 96%.
- Accurate Tracking: The refinement phase, particularly motion interpolation, was shown to accurately correct object blob estimations, enclosing the entire body even with articulated parts or changing sizes.
- Effective Occlusion Handling: The hue color histogram comparison successfully distinguished objects after occlusion, as demonstrated in Figure 10.
Limitations and Future Work
Limitations: * The method is primarily designed for scenes with stationary backgrounds and no illumination change. * The partial decoding technique for I-frames is restrictively applicable to such scene environments. * The assumption of linear motion for interpolation might become less reliable as the GOP length increases. * The current implementation uses the AVC Baseline profile and does not explicitly handle B-frames.
Future Work: * Extend the algorithm to incorporate more vision-based techniques for greater flexibility in various environments. * Support video bitstreams encoded with the Main profile (handling B-frames). * Develop advanced techniques to deal with illumination change and silhouette analysis in the compressed domain. * Apply the algorithm to large-scale distributed surveillance systems that process many compressed videos simultaneously across camera networks.
Summary updated