Choosing Which Frames to Label by Covering Metric Space-Time

Multi-view pedestrian detectors are trained on very small labeled sets. The standard real benchmark, WILDTRACK, has 400 annotated frames, and models trained on it collapse when the camera setup changes. This proposal asks a narrow question. If you can afford to label only a few hundred frames out of a large pool of multi-camera footage, which frames should you pick?

The proposed answer is to pick frames that cover the scene in physical units rather than in pixels. A space-time cube is a 3D volume whose floor is the scene and whose vertical axis is time. We build one whose floor is the real ground plane in meters, so that a crowd 30 meters from every camera counts exactly as much as a crowd 3 meters away. We then choose frames with a greedy algorithm whose coverage of that cube is provably near-optimal. Pixel-space selection cannot do this, because nearby people are large and fast in the image and far-away people are small and slow.

Core Claim

Frames chosen to maximize weighted coverage of a ground-plane space-time cube train better multi-view detectors than the same number of random, uniformly spaced, or pixel-feature-diverse frames. The gain should be largest on the far field and on unusual density regimes, because those are the parts of the scene that pixel-space criteria systematically under-sample. The cube's cells are bins over ground-plane position, local crowd density, and metric speed.

Because the coverage objective is monotone submodular (adding a frame never hurts, and each frame adds less once similar frames are already chosen), greedy selection achieves at least $(1-1/e)$ of the best possible coverage. This is the classical bound formalized by Kempe and Tardos. The guarantee concerns coverage, not detection accuracy. A central experiment therefore tests whether coverage actually predicts accuracy.

How This Relates to Prior Work

Four lines of prior work bear on the proposal, each in a different role.

The method's selection machinery comes from submodular summarization. Gygli et al. select video segments greedily under a mixture of interestingness, representativeness, and temporal uniformity terms, and reach 39.68% F-measure on SumMe. All their terms live in 2D pixel or appearance-feature space on single-camera user videos, and their goal is a human-watchable summary. We reuse the greedy coverage machinery but change both the space it covers (metric ground plane across cameras) and the purpose (choosing training data).

The motivation comes from perspective normalization. Liu et al. report that distant anomalies vanish because their pixel flow is tiny, and they leave depth-awareness as future work. Baradaran and Bergevin show that weighting flow by monocular depth lifts their motion branch on UCSD-Ped2 from 69.7% to 79.19% AUC. Older surveillance systems approximate the same correction by scaling motion with the image row, as Fudan at TRECVID 2008 does, or with clustered person heights, as NHK STRL at TRECVID 2009 does. These works establish that pixel-space motion statistics are biased toward the near field. All of them apply the correction to the detector's input. We apply it to the data-selection criterion instead.

The constraint comes from the multi-view data. WILDTRACK provides calibrated ground-plane geometry, built by bundle adjustment over about 1,400 points, but only 200 seconds of labels. Vora et al. show that MVDet falls to 38.9% MODA with 4 cameras and 27.8% under camera splits. Their fix is architectural: averaging over views plus random view dropping, which they call DropView, together with a synthetic dataset, GMVD. None of these works asks which frames or scenes the scarce labels should go to. Random choice is the implicit default.

The space-time cube idea is a visualization device in Andrienko's VAST work and in VolumeSTCube. We use it as the ground set of a coverage objective rather than something to look at. The perspective-warped slopes noted in pixel-space cubes are precisely the bias we remove.

Research Program

The pool and its cheap evidence. The pool is a large set of synchronized multi-camera timesteps. For each timestep we compute evidence without human labels:

  1. A detector trained on a small seed set produces ground-plane occupancy estimates.
  2. These are projected through the known homographies onto a ground grid with fixed metric cell size.
  3. Short-horizon tracking of the projected detections gives metric speeds.
  4. Local counts give density.

Each candidate timestep $t$ then induces a soft footprint $p_t(g) \in [0,1]$ over cube cells $g$. A cell indexes a ground-plane location, a density bin, and a metric-speed bin, and $p_t(g)$ is the estimated probability that timestep $t$ contains a well-observed example of that cell.

The objective. We maximize

$$f(S) = \sum_{g} w_g \left(1 - \prod_{t \in S} \bigl(1 - p_t(g)\bigr)\right),$$

where $S$ is the chosen set of timesteps and $w_g \ge 0$ is a cell weight. In the default setting $w_g$ is uniform over occupied cells, so far and near cells count equally. This probabilistic coverage function is monotone submodular, so the greedy guarantee above applies.

Computation. Lazy greedy evaluation keeps the oracle cost manageable, because each footprint is sparse over cells. This addresses the computational cost the research file flags for greedy selection over expensive video objectives.

When calibration is missing. For cameras without calibration, we replace the homography with row-based or height-based scale maps of the TRECVID kind. This tests whether a rough normalization is enough.

What we analyze. First, whether $f(S)$ across many subsets predicts held-out MODA. Second, where in the scene the gains concentrate, by binning detection errors by distance to the nearest camera. Third, how the advantage changes with budget. Selection should matter most at small budgets and fade as the budget approaches the full pool. This trend is the dataset-scale question stated directly.

Baselines

Training on the full pool. This is the reference ceiling: the same detector trained on every pool frame with the same number of optimizer steps. It tells us how much of the full-data accuracy a budgeted subset recovers.

Random and uniform temporal selection. These are the "do nothing" baselines. Random selection is the implicit practice behind every small benchmark. Uniform spacing is its strongest naive variant, because consecutive surveillance frames are highly redundant. We draw multiple random seeds and report the spread. Beating random means exceeding its mean by more than its seed-to-seed standard deviation at matched budget.

Pixel-space submodular selection. This is the strongest existing method. We use a Gygli-style representativeness-plus-uniformity objective (Gygli et al.) over per-view image embeddings and image-space motion. It uses the identical greedy solver and budget, so the only difference from our method is the space being covered. No reported result exists on Lacuna for this objective on multi-view detection. The closest number in the file is its 39.68% F-measure on SumMe, which is a different task.

The detector used for every arm. All arms train the same generalized multi-view detector of Vora et al.: view-averaged ground-plane features with DropView. Every arm gets identical hyperparameters, step counts, and tuning budget, with tuning done once on a random subset and then frozen. The full-data reference numbers from that paper are 87.2% MODA on WILDTRACK with pre-training and 70.7% MODA when transferring from MultiViewX to all WILDTRACK views. For the transfer setting, the comparison method SHOT reaches 53.6% at 6 views against their 66.1%.

Ablations isolating the key ingredient. Two ablations remove the perspective normalization while keeping everything else. The first builds the same cube but in each camera's pixel coordinates. The second replaces homographies with row-based scaling. If the pixel-coordinate cube matches the metric one, the perspective claim is false.

Why not just…?

Why not just sample frames uniformly in time? This is the real threat. In a single fixed scene with steady crowd flow, uniform spacing may already cover the space-time cube well. Gygli et al. found that learned submodular selection beat uniform sampling on user video. That evidence comes from heterogeneous summarization, not homogeneous surveillance, so it does not settle the question here. We expect our advantage mainly in pools that mix scenes and densities, such as GMVD, and possibly only a small advantage within a single WILDTRACK sequence. The within-WILDTRACK and cross-scene experiments are designed to show exactly where uniform spacing is good enough. We will report it honestly if that covers most settings.

Why not just run diverse selection on image features? Feature-diverse selection is the standard approach and needs no calibration. However, near-field pedestrians dominate both image appearance and pixel motion. The perspective literature shows this bias directly: Liu et al. document far-field misses, and Baradaran and Bergevin report a 9.49-point AUC gain from depth weighting. The pixel-space baseline and the pixel-coordinate cube ablation test this alternative head-on, and the distance-binned error analysis shows where any gap arises.

Why not just use more synthetic data or a better architecture? Synthetic frames are cheap, so selection seems unnecessary there. Our experiments do use synthetic pools, but only to measure how informative each frame is. The case that matters in practice is choosing real footage to label, where each frame costs human effort. Architectural fixes such as DropView are complementary rather than competing, which is why every arm uses that architecture. If full-pool synthetic training on GMVD already transfers as well as a selected real subset, then selection is unnecessary for real labeling, and we would report that.

Why not select by model uncertainty, as in active learning? Uncertainty sampling is a reasonable competitor and is included as an additional arm. It is known to pick redundant, near-identical hard frames, whereas coverage spreads the budget. The decisive comparison is at budgets of 50 to 100 frames, where redundancy is most costly.

Proposed Experiments

Within-scene selection on WILDTRACK. The pool is the first 360 annotated frames, the test set is the final 40, and budgets are 25, 50, 100, and 200. Labels exist for every pool frame, so we can measure how a labeled subset of a given size performs, while selection itself uses only pseudo-label footprints.

Cross-scene transfer. The pool combines MultiViewX and GMVD frames, budgets range from 50 to 400 frames, and testing is on WILDTRACK at 7 and 6 views. This setting has the most heterogeneity and is where the method should matter most. No reported result was found on Lacuna for GMVD itself.

Coverage as a proxy. For each budget we train on 30 or more random subsets, compute $f$ for each, and measure the rank correlation between $f$ and MODA. If this correlation is weak, the approximation guarantee is about the wrong quantity and the idea fails, whatever the headline numbers say.

Is the solver the bottleneck? On small instances we compare greedy coverage to an exact integer-program optimum. This confirms that near-optimal coverage is achieved and that any remaining gap comes from the objective, not the solver.

Success criteria. The method succeeds if three things hold. First, metric-cube selection beats random, uniform, and pixel-space selection by more than the random-seed spread at budgets of 100 frames or fewer. Second, the gains concentrate in far-from-camera cells. Third, the metric-coordinate cube beats its pixel-coordinate ablation.