Stacked Homography Transformations for Multi-View Pedestrian Detection

Liangchen Song, Jialian Wu, Ming Yang, Qian Zhang, Yuan Li, Junsong Yuan

Abstract

Multi-view pedestrian detection aims to produce a unified bird’s-eye view (BEV) occupancy map from multiple synchronized camera feeds. Traditional methods often struggle with a trade-off between the computational intensity of 3D voxel grids and the geometric inaccuracies of 2D ground-plane homographies. This paper introduces Stacked HOmography Transformations (SHOT), a method that bridges this gap by projecting image features onto a "stack" of parallel planes at different heights. By using a soft-selection module to weight these projections, SHOT accurately aligns 3D human structures into a consistent BEV map with high efficiency.

Introduction

In multi-view pedestrian detection, the objective is to aggregate information from several cameras to determine the precise ground-plane locations of individuals. This task is critical for applications in surveillance, crowd management, and autonomous systems. However, accurately mapping 3D entities (humans) from 2D images to a top-down view is challenging due to perspective distortion and the "projection gap."

Overview of multi-view pedestrian detection: Input multi-view images are processed to output a Bird's-eye View Occupancy Map.

The Projection Gap and Computational Trade-offs

The primary difficulty in multi-view detection lies in how image features are projected onto the ground plane.

  1. 2D Homography Issues: Standard 2D homography maps the ground plane of the world to the image plane. Because humans are 3D objects extending vertically, projecting their upper body parts using a ground-level homography causes them to appear "shifted" or "leaning" away from their actual ground position. This leads to misalignment when merging features from different camera views.
  2. 3D Voxel Constraints: To resolve this, some approaches utilize 3D voxel grids. While accurate, these are extremely memory-intensive and require costly 3D convolutions, making them difficult to scale or deploy in real-time scenarios.

The authors propose SHOT as a "Goldilocks" solution: more accurate than a single 2D plane but significantly more efficient than a full 3D grid.

Comparison of projection methods: (a) SHOT approximates 3D with high efficiency and accuracy; (b) 3D projection is accurate but inefficient; (c) 2D projection is efficient but less accurate.

The SHOT Framework

The core innovation of SHOT is the use of a Homography Stack combined with a Soft Selection Module.

The Homography Stack

Instead of a single projection, the framework pre-computes a stack of $D+1$ homographies ($H_0, H_1, \dots, H_D$). Each homography corresponds to a specific height level (e.g., 0m, 0.5m, 1m, 1.5m, 2m). This allows the model to project image features to multiple potential heights simultaneously.

Soft Selection Module

Since the network does not inherently know the height of a specific pixel's content, it employs a small CNN to predict a "likelihood map." For every pixel, the network assigns a weight to each height level in the stack. The final projected feature for a given BEV cell is a weighted sum of all projections in the stack. This mechanism enables the network to learn semantically that heads should be projected via high-level homographies while feet should use ground-level ones.

The entire pipeline, which includes a ResNet-18 backbone for feature extraction and a final classifier for the aggregated BEV features, is end-to-end differentiable.

Experimental Evaluation

The authors evaluated SHOT on two major benchmarks:

  • WILDTRACK: A real-world dataset with 7 synchronized cameras covering a 12m x 36m outdoor area, characterized by high density and frequent occlusions.
  • MultiviewX: A synthetic dataset (Unity engine) with 6 cameras and 40 people per frame, providing a controlled environment for geometric testing.

Key Results

SHOT achieved state-of-the-art (SOTA) performance on both datasets:

  • WILDTRACK: Reached 90.2% MODA (Multiple Object Detection Accuracy), surpassing the previous best of 88.2%.
  • MultiviewX: Reached 88.3% MODA, a 4.4% improvement over the previous SOTA (83.9%).
  • Robustness: In transfer learning tests where camera locations varied between training and testing, SHOT outperformed the MVDet baseline by 16.1% MODA, demonstrating superior robustness to viewpoint changes.

Discussion and Insights

  • 2.5D Approximation: SHOT effectively treats the 3D world as a series of "slices." Mathematically, with enough slices, it can perfectly align 3D objects onto the BEV grid.
  • Semantic Awareness: Visualizations confirm that the soft selection module correctly identifies body parts, using "higher" homographies for heads and "lower" ones for feet.
  • Ease of Calibration: SHOT can be initialized with as few as two pedestrian annotations (head and feet) of a known height, reducing the need for exhaustive camera extrinsic parameters.

Limitations

While efficient, the stack's discrete nature means it may struggle with extreme height outliers if the stack density is low. Additionally, memory costs still scale with the number of cameras and stack levels, with the authors noting hardware limits at $D=6$. Future work may explore dynamic height estimation to further optimize the "2.5D" approach.

Summary updated

Abstract

Multi-view pedestrian detection aims to predict a bird’s eye view (BEV) occupancy map from multiple camera views. This task is confronted with two challenges: how to establish the 3D correspondences from views to the BEV map and how to assemble occupancy information across views. In this paper, we propose a novel Stacked HOmography Transformations (SHOT) approach, which is motivated by approximating projections in 3D world coordinates via a stack of homographies. We first construct a stack of transformations for projecting views to the ground plane at different height levels. Then we design a soft selection module so that the network learns to predict the likelihood of the stack of transformations. Moreover, we provide an in-depth theoretical analysis on constructing SHOT and how well SHOT approximates projections in 3D world coordinates. SHOT is empirically verified to be capable of estimating accurate correspondences from individual views to the BEV map, leading to new state-of-the-art performance on standard evaluation benchmarks.
BibTeX
@inproceedings{song2021stacked,
  author = {Liangchen Song and Jialian Wu and Ming Yang and Qian Zhang and Yuan Li and Junsong Yuan},
  title = {{Stacked Homography Transformations for Multi-View Pedestrian Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00599},
}
Figure