Abstract
Multi-view pedestrian detection aims to produce a unified bird’s-eye view (BEV) occupancy map from multiple synchronized camera feeds. Traditional methods often struggle with a trade-off between the computational intensity of 3D voxel grids and the geometric inaccuracies of 2D ground-plane homographies. This paper introduces Stacked HOmography Transformations (SHOT), a method that bridges this gap by projecting image features onto a "stack" of parallel planes at different heights. By using a soft-selection module to weight these projections, SHOT accurately aligns 3D human structures into a consistent BEV map with high efficiency.
Introduction
In multi-view pedestrian detection, the objective is to aggregate information from several cameras to determine the precise ground-plane locations of individuals. This task is critical for applications in surveillance, crowd management, and autonomous systems. However, accurately mapping 3D entities (humans) from 2D images to a top-down view is challenging due to perspective distortion and the "projection gap."
The Projection Gap and Computational Trade-offs
The primary difficulty in multi-view detection lies in how image features are projected onto the ground plane.
- 2D Homography Issues: Standard 2D homography maps the ground plane of the world to the image plane. Because humans are 3D objects extending vertically, projecting their upper body parts using a ground-level homography causes them to appear "shifted" or "leaning" away from their actual ground position. This leads to misalignment when merging features from different camera views.
- 3D Voxel Constraints: To resolve this, some approaches utilize 3D voxel grids. While accurate, these are extremely memory-intensive and require costly 3D convolutions, making them difficult to scale or deploy in real-time scenarios.
The authors propose SHOT as a "Goldilocks" solution: more accurate than a single 2D plane but significantly more efficient than a full 3D grid.
The SHOT Framework
The core innovation of SHOT is the use of a Homography Stack combined with a Soft Selection Module.
The Homography Stack
Instead of a single projection, the framework pre-computes a stack of $D+1$ homographies ($H_0, H_1, \dots, H_D$). Each homography corresponds to a specific height level (e.g., 0m, 0.5m, 1m, 1.5m, 2m). This allows the model to project image features to multiple potential heights simultaneously.
Soft Selection Module
Since the network does not inherently know the height of a specific pixel's content, it employs a small CNN to predict a "likelihood map." For every pixel, the network assigns a weight to each height level in the stack. The final projected feature for a given BEV cell is a weighted sum of all projections in the stack. This mechanism enables the network to learn semantically that heads should be projected via high-level homographies while feet should use ground-level ones.
The entire pipeline, which includes a ResNet-18 backbone for feature extraction and a final classifier for the aggregated BEV features, is end-to-end differentiable.
Experimental Evaluation
The authors evaluated SHOT on two major benchmarks:
- WILDTRACK: A real-world dataset with 7 synchronized cameras covering a 12m x 36m outdoor area, characterized by high density and frequent occlusions.
- MultiviewX: A synthetic dataset (Unity engine) with 6 cameras and 40 people per frame, providing a controlled environment for geometric testing.
Key Results
SHOT achieved state-of-the-art (SOTA) performance on both datasets:
- WILDTRACK: Reached 90.2% MODA (Multiple Object Detection Accuracy), surpassing the previous best of 88.2%.
- MultiviewX: Reached 88.3% MODA, a 4.4% improvement over the previous SOTA (83.9%).
- Robustness: In transfer learning tests where camera locations varied between training and testing, SHOT outperformed the MVDet baseline by 16.1% MODA, demonstrating superior robustness to viewpoint changes.
Discussion and Insights
- 2.5D Approximation: SHOT effectively treats the 3D world as a series of "slices." Mathematically, with enough slices, it can perfectly align 3D objects onto the BEV grid.
- Semantic Awareness: Visualizations confirm that the soft selection module correctly identifies body parts, using "higher" homographies for heads and "lower" ones for feet.
- Ease of Calibration: SHOT can be initialized with as few as two pedestrian annotations (head and feet) of a known height, reducing the need for exhaustive camera extrinsic parameters.
Limitations
While efficient, the stack's discrete nature means it may struggle with extreme height outliers if the stack density is low. Additionally, memory costs still scale with the number of cameras and stack levels, with the authors noting hardware limits at $D=6$. Future work may explore dynamic height estimation to further optimize the "2.5D" approach.
Summary updated