Systems that watch a scene through many cameras, such as crowd counters or roadside traffic monitors, have to decide how much to trust each camera at each spot on the ground. A camera may be blocked by a pole or washed out by glare. Simulators can supply that answer for free during training, because they know exactly which camera sees which person. This lets a network learn a per-camera, per-location trust score. The trouble starts when the system is deployed on real cameras. Every real camera now looks "different" from the training data. A trust module learned in simulation cannot tell whether a camera looks different because real images have another style (the sim-to-real domain gap) or because that camera is actually blocked. It ends up either distrusting good cameras or over-trusting blocked ones.
We propose to separate these two causes using a structural fact about them. Domain shift changes a camera's output in a roughly consistent way everywhere and at all times: the same lens, the same texture statistics, the same rendering mismatch. Real view degradation is local in space and time. Concretely, for each camera we predict its ground-plane occupancy from the other cameras alone. We fit a per-camera correction over long windows to absorb the consistent part of the disagreement. The leftover, local disagreement is then used as an unsupervised target for retraining the trust module on unlabeled real video. The method needs no real labels, and a corrupted camera never votes on its own trustworthiness.
Core Claim
Self-supervised adaptive view weighting can be made robust to the sim-to-real gap if two kinds of cross-view disagreement are kept apart. Disagreement that a per-camera correction can explain counts as domain shift and should not lower trust. Disagreement that stays local after that correction counts as view degradation and should. We expect that retraining a synthetic-supervised view-weighting module on these "leave-one-camera-out residual" targets will improve zero-label transfer from synthetic to real multi-camera people detection. We also expect the learned weights to respond to real occlusions while ignoring global appearance changes. Existing self-supervised weighting schemes do not make this second distinction.
Where This Sits in Prior Work
The closest work, and the starting point for this proposal, is Multi-View People Detection in Large Scenes via Supervised View-Wise Contribution Weighting. It decodes an occupancy map per camera, supervises a per-camera ground-plane weight map against simulator visibility masks, fuses the weighted features, and adds a domain discriminator to align synthetic (CVCS) and real features. That paper shows view-wise weighting beats uniform aggregation in large scenes, which motivates us. Its weight predictor, however, is trained only on synthetic views and is never re-estimated on the real target. Our proposal keeps its architecture and replaces the missing real supervision with residual targets.
Self-supervised view weighting from multi-view clustering supplies the general mechanism. Examples include the Laplacian-trace weights of AMGL, the agreement-with-consensus terms of SWIB, the distributional-affinity weights of HSACC, and the per-sample weights of ADMC. These works also constrain the design in two ways. First, they score a view by how far it sits from the consensus, which in a sim-to-real setting penalizes any real camera whose style drifts. Second, their iterative loops are prone to confirmation bias, because a dominant corrupted view shapes the consensus it is then judged against, a weakness the research file attributes to SWIB and DuaLGR. We address the first by absorbing consistent per-camera offsets before scoring. We address the second by always excluding the camera being scored from its own consensus.
The nearest cross-domain reliability estimator is CoMAC. At test time it weights camera versus LiDAR predictions by the distance of each feature to its class centroid. That score is still a distance from a source-shaped reference, so a style-shifted but informative sensor looks unreliable. It is also an online method for two registered modalities, not many overlapping cameras.
Two papers motivate the sim-to-real side. MC-BEVRO reports that zero-shot transfer from CARLA to a real Arizona intersection was poor and needed few-shot fine-tuning, and its fusion has no explicit camera reliability. Adversarially Masking Synthetic to Mimic Real shows that clean synthetic inputs give a domain discriminator a shortcut. This warns us that a weighting module trained in simulation will learn "clean equals trustworthy." CTS shows that uncertainty-weighted self-training on unlabeled real data substantially closes a sim-to-real detection gap. That supplies the self-training recipe we build on, applied to weights over views rather than over pseudo-labeled boxes.
Research Program
We start from the supervised view-weighting detector of Zhang et al., trained on synthetic data (CVCS, and MultiviewX for the small-scene setting). It produces, for each camera $v$, frame $t$, and ground cell $x$, a single-view occupancy logit $s_v(x,t)$ and a weight $w_v(x,t)$.
On unlabeled real video we compute a leave-one-out consensus $\bar s_{-v}(x,t)$. This is the fused logit from all cameras except $v$ that cover cell $x$, weighted by the current weights. We then fit a per-camera correction by robust regression over all covered cells and a long window of frames. The residual is
$$r_v(x,t) = \left| s_v(x,t) - a_v\,\bar s_{-v}(x,t) - b_v \right|,$$
where $a_v$ and $b_v$ are a per-camera scale and offset in logit space. Long windows are used so that only consistent, camera-wide mismatch is absorbed. The residual target weight is a softmax over cameras of $-r_v(x,t)/\tau$, with temperature $\tau$.
Cells seen by only one camera have no leave-one-out evidence. There we keep the synthetic predictor's weight, so the method stays conservative and does not invent supervision. The weight predictor is fine-tuned on unlabeled real frames to match these targets. A small per-camera feature-normalization adapter is fitted alongside, so that the correction is also applied in feature space. We alternate a few rounds, and stop when the targets stop changing.
The affine logit correction is a deliberate simplification. Real style shift need not be affine. We therefore test a richer variant, a per-camera monotone spline in place of $(a_v, b_v)$, and report whether it changes the results. The analysis has three parts. The first measures how well the learned weights track true per-view visibility on real data. The second runs controlled corruption tests that separate global from local degradation. The third checks whether better weights translate into better detection.
Baselines
The naive reference point is uniform fusion: per-camera features averaged on the ground plane, in the style of the average-pooling and late-fusion variants studied in MC-BEVRO. It is trained on the same synthetic data and transferred zero-shot. Any weighting method that fails to beat it on real data is not earning its complexity.
The strongest existing method is the supervised view-wise contribution weighting detector of Zhang et al., including its domain discriminator, transferred without real labels. The file gives its average rank of 1.5 versus 3.0 for the next competitor across CVCS and CityStreet. It also gives F1 of 89.5% on Wildtrack and 91.4% on MultiviewX when fine-tuned with 5% of target labels. No reported result was found on Lacuna for its zero-label synthetic-to-real F1, so we will reproduce it ourselves. We use the same backbone, the same synthetic training data, the same unlabeled real frames for the discriminator, and the same tuning budget, with hyperparameters chosen on a held-out synthetic split for every method. Beating it means a statistically reliable gain in MODA and F1 on the real test splits under zero real labels, averaged over at least three seeds.
To isolate the key ingredient, we include three ablations that share the same self-training pipeline but use different weight targets. The first uses distance to the full consensus, including camera $v$ itself, with no per-camera correction. This is the HSACC/SWIB-style score. The second uses leave-one-out residuals without the per-camera correction. The third applies only the per-camera normalization adapter and keeps the synthetic weights frozen. If our full method does not beat the second and third ablations, the "separate consistent from local disagreement" claim fails.
We also run a pseudo-label self-training baseline in the spirit of CTS. It applies uncertainty-weighted soft pseudo-labels on real frames while keeping the synthetic weight predictor frozen, with an equal number of real-data passes.
Why not just…?
Why not just fine-tune on a few real labels? This is a real threat. Zhang et al. report 89.5% F1 on Wildtrack with 5% of labels, close to full supervision. Our proposal targets deployments where no real occupancy labels exist, such as newly installed roadside cameras like the Arizona intersection in MC-BEVRO, which needed drone footage to get ground truth. We will report label-efficiency curves at 0%, 1%, and 5% of labels. If our zero-label result already matches 1% fine-tuning, or our method still adds on top of 5% fine-tuning, the contribution stands. If 1% fine-tuning trivially dominates, we will say the method is useful only in strictly label-free settings.
Why not just let cross-attention or a deformable transformer learn the weights? Attention weights are learned on synthetic data exactly as Zhang et al.'s weight maps are, so they inherit the same clean-equals-trustworthy bias that the masking paper documents for discriminators. We include the deformable-transformer fusion (MVDeTr-style, the best fusion in MC-BEVRO with IoU 0.67759 on CARLA) as a zero-shot comparison. We also include it with our residual targets used as an auxiliary loss on its attention, which directly tests whether the residual signal or the architecture matters.
Why not just normalize each camera's features on real data and keep the synthetic weights? Per-camera normalization is the cheapest way to remove consistent style shift, and it may capture most of the gain. This is exactly the third ablation above. Our claim is that normalization fixes appearance but leaves the trust module blind to real occlusions that the simulator never rendered, such as foliage, haze, and street furniture. The corruption experiments below test that directly.
Why not just weight cameras by prediction entropy or centroid distance, as CoMAC does? Both scores measure distance from a source-shaped reference, so a style-shifted camera looks uncertain even when it is informative. We run entropy weighting and CoMAC-style centroid-distance weighting at test time as baselines. We expect them to downweight cameras under global appearance shifts, which is the failure our residual is designed to avoid.
Proposed Experiments
The main transfer settings are CVCS to CityStreet (large scenes) and MultiviewX to Wildtrack (small scenes). Both are evaluated with MODA, MODP, and F1 on the ground plane. Where the real data can be obtained, we add CARLA to the Arizona intersection with BEV occupancy IoU. For that setting no reported zero-shot number was found on Lacuna, so all methods will be run by us.
To evaluate the weights themselves, we use per-camera annotations where the dataset provides them; Wildtrack, for instance, labels people per camera view. From these we derive real per-view visibility of each person and report rank correlation between learned weights and visibility. This tests whether the module is right about which camera to trust, separately from whether detection improved.
The decisive check is a controlled corruption study on real test video. In one condition we apply a global, camera-wide appearance change to a single camera, using a color, blur, and noise transform of the kind that separates rendered from real images. In the other we paste local occluders or glare patches on part of that camera's field of view for a limited time. Under the global change, a correct method should keep that camera's weights nearly unchanged and keep detection close to uncorrupted. Under the local occlusion, it should cut weights only in the affected cells and frames. We predict that the consensus-distance baselines will drop the globally shifted camera everywhere, and that the frozen synthetic weights will fail to drop the occluded region. If our method does not show both behaviors, the core claim is wrong regardless of the detection numbers.
A final check measures confirmation bias. We initialize self-training with one camera artificially given high weight while its input is heavily corrupted. We then measure whether the leave-one-out targets recover the correct weights, and whether full-consensus targets lock in the error, as the file's account of SWIB and DuaLGR loops predicts.
References
- Multi-View People Detection in Large Scenes via Supervised View-Wise Contribution Weighting
- Parameter-Free Auto-Weighted Multiple Graph Learning: A Framework for Multiview Clustering and Semi-Supervised Classification
- Self-supervised Weighted Information Bottleneck for Multi-view Clustering
- Incomplete Multi-view Clustering via Hierarchical Semantic Alignment and Cooperative Completion
- Active Deep Multi-view Clustering
- Dual Label-Guided Graph Refinement for Multi-View Graph Clustering
- Multi-Modal Continual Test-Time Adaptation for 3D Semantic Segmentation
- MC-BEVRO: Multi-Camera Bird Eye View Road Occupancy Detection for Traffic Monitoring
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- CTS: Sim-to-Real Unsupervised Domain Adaptation on 3D Detection