TL;DR
This paper introduces MVP-Occ, a novel synthetic dataset for dense multi-view pedestrian occupancy prediction in large-scale urban scenes, addressing limitations of existing datasets. They also propose OmniOcc, a robust baseline model that predicts both semantic and instance occupancy from multi-view images, demonstrating strong performance and synthetic-to-real transfer capabilities.
What Problem Does This Paper Address?
Predicting the precise location and occupancy of pedestrians in crowded, large-scale urban environments using multiple camera views is a significant challenge. Existing multi-view tasks like crowd counting, tracking, and detection often focus on 2D ground plane estimations or are limited to small environments with few individuals. While some methods can estimate 3D poses, they struggle with metric-level accuracy and dense occupancy prediction for the entire scene. This gap hinders applications requiring a comprehensive understanding of pedestrian presence and spatial distribution, especially in complex surveillance scenarios.
Methods
The authors propose a novel model called OmniOcc for multi-view pedestrian occupancy prediction. The model architecture consists of:
- Image Encoder: A ResNet backbone augmented with a Feature Pyramid Network (FPN) to extract multi-scale 2D features from input images.
- View Transformer: A non-parametric approach that projects multi-view 2D features onto a 3D voxel grid. This method assumes a uniform depth distribution along camera rays, effectively transferring features to the 3D space.
- Voxel Encoder: A 3D U-Net architecture processes the voxelized features to construct a rich feature volume, capturing hierarchical spatial information.
- Occupancy Decoder: A two-layer MLP network predicts semantic occupancy (five classes: pedestrian, background, etc.) for each voxel.
- Location Decoder: A single convolutional layer generates a 2D bird's-eye view (BEV) occupancy map, indicating pedestrian presence on the ground plane.
- Pedestrian Instance Grouping (PIG): This module leverages both semantic occupancy and the 2D BEV map to group pedestrian voxels into individual instances and then combines them with background classes to form panoptic occupancy predictions.
The model is trained using a composite loss function that includes weighted cross-entropy, Lovász-Softmax, and scene-class affinity losses for 3D occupancy, and Mean Squared Error (MSE) for 2D occupancy.
Datasets
- MVP-Occ (Multi-View Pedestrian Occupancy): A newly introduced synthetic dataset designed for dense pedestrian scenarios in large-scale urban environments. It features five expansive scenes (Alley, Plaza, Field, Park, Facade) with detailed 3D annotations, including voxel occupancy, semantic labels, instance labels, and panoptic labels. The dataset also includes RGB, depth, and semantic segmentation sensor data.
- WildTrack: A real-world multi-view dataset used for synthetic-to-real transfer evaluation. The authors manually annotated this dataset with 2D semantic, instance, and panoptic segmentation labels to serve as a proxy for 3D occupancy evaluation due to the lack of ground-truth occupancy data.
Key Insights and Ideas
- The necessity of 3D semantic context: The paper strongly advocates for predicting not just pedestrian locations but also the semantic context of the entire scene (e.g., walls, ground, free space). This richer understanding significantly improves pedestrian detection and scene reconstruction.
- Synthetic-to-real transfer is challenging but achievable: The study highlights the difficulty of transferring models trained on synthetic data to real-world scenarios, especially when scenes differ significantly. However, their proposed dataset and model demonstrate promising results in bridging this gap, particularly with the Facade scene.
- Pedestrian Instance Grouping (PIG) is crucial for panoptic prediction: While PIG might not always show significant quantitative gains in same-scene evaluation due to the sparsity of voxels vs. instances, it's essential for generating accurate panoptic occupancy by correctly associating semantic voxels with individual pedestrians.
- A unified approach for 2D and 3D occupancy: The model effectively combines 2D ground-plane localization with 3D volumetric occupancy prediction, leveraging the strengths of both to achieve a more comprehensive scene understanding.
Key Results
- 2D Occupancy Prediction (Same-Scene): OmniOcc outperforms previous methods on the MVP-Occ dataset, achieving an average F1 score of 95.2% and an average MODA of 90.8%.
- 3D Occupancy Prediction (Same-Scene): The model achieves high performance on MVP-Occ, with nearly 99% IoU for the Background class and 62-68% IoU for the Pedestrian class, depending on the scene.
- Synthetic-to-Real Evaluation (WildTrack): OmniOcc demonstrates superior performance compared to GMVD and MVFP on the WildTrack dataset, indicating its strong generalization capabilities. For example, on the Facade scene, it achieves an average 2D occupancy prediction MODA of 75.4% and F1 of 87.5%.
- Ablation Studies: The study confirms the importance of all proposed loss functions for optimal performance and highlights that semantic scene understanding significantly boosts both 2D and 3D occupancy prediction accuracy.
Limitations and Future Work
- Limitations:
- The paper notes that while PIG improves qualitative results, its quantitative impact on same-scene evaluation for panoptic occupancy is less pronounced due to the large number of voxels relative to instances.
- The synthetic-to-real transfer, while improved, still faces challenges when scenes are highly dissimilar.
- Future Work:
- The authors suggest further investigation into improving scene understanding in dissimilar scenes.
- The MVP-Occ dataset is designed to facilitate broader research in pedestrian recognition and scene understanding, implying future work could explore other perception tasks on this benchmark.
- The model's ability to reconstruct scenes with dynamic pedestrians when provided with ground-truth data opens avenues for real-time applications.
Summary updated