Seeing Less to Generalize Across Viewpoints: Capacity-Limited Token Selection for View-Robust Recognition

Vision models fail badly when objects are photographed from unusual angles. On ObjectNet, every major architecture loses 40 to 45 points of accuracy relative to ImageNet-style photos, while humans stay near 95% (ObjectNet). Part of the reason is that these models depend on whatever image evidence happens to correlate with the label at the training viewpoint, including background context. Masking the background on CO3D-Retrieve drops retrieval mean average precision from 48.81% to 27.33% (A Light Touch Approach to Teaching Transformers Multi-view Geometry). Biological vision faces a strict limit on how much of the visual field it can process. Under selective attention, frontoparietal cortex carries strong evidence of zero information about unattended features rather than merely weaker information (Spatial and feature-selective attention interact...).

We propose to test whether such a hard limit on how much of the image a network may look at improves viewpoint robustness. The limit would be a small, fixed number of image patches selected early, with everything else multiplied to exactly zero. The selector would be trained so that the few patches it keeps support the same representation when the same object is seen from different angles. The claim fits in two sentences. Forcing a network to commit to a handful of patches, and rewarding it when those patches give view-consistent features, pushes it toward the parts of an object that survive rotation and away from view-specific context. Robustness should therefore improve as the patch budget tightens, up to a point, at little cost on familiar viewpoints.

Core Claim

The testable claim concerns robustness as a function of capacity. Let $k$ be the number of patch tokens a vision transformer may pass beyond its first few layers. We predict that the gap between in-distribution accuracy and viewpoint-shifted accuracy shrinks as $k$ decreases from all tokens to a small budget, before collapsing at very small $k$. We also predict this effect depends on two things. The gate must be hard, so unselected tokens contribute exactly zero rather than a down-weighted amount. The selector must be trained with a cross-view agreement signal rather than only the classification loss. No paper in the file measures how viewpoint generalization depends on attentional capacity.

The nearest existing work differs in specific ways. The Attentional Bottleneck driving network (Kim & Bansal) passes attention through an information bottleneck to get sparse, causal heatmaps. It works only on top-down orthographic maps where viewpoint never changes, so we borrow its bottleneck idea but apply it to perspective imagery. Viewpoint Bottleneck (Pointly-supervised 3D Scene Parsing with Viewpoint Bottleneck) uses a Barlow Twins–style cross-correlation objective between two geometric transforms of a point cloud. That bottleneck acts on feature channels, not on which spatial evidence is admitted. We reuse its objective but place the bottleneck on spatial selection. Perspective Invariant Attention in Discrete Latent Perspective Learning and the token lifting in 3DTRL both process every token with no capacity limit. The epipolar loss of Bhalgat et al. shapes where attention goes, but it needs ground-truth camera poses and never limits how much is attended. Our method needs no poses, only pairs of images known to show the same object.

Research Program

The model is a standard vision transformer, DeiT-Small in the main runs and DeiT-Tiny for sweeps, to match the 3DTRL code. A small scoring head is inserted after layer $L$ (we sweep $L \in \{2, 4\}$, testing whether early selection matters, as the rat cortex result in Early selection of task-relevant features through population gating suggests). The head assigns each patch token a score. The top $k$ tokens continue through the remaining layers, and all others are removed, which is the multiplicative zero gate. Gradients reach the scorer through a differentiable top-$k$ relaxation with a straight-through estimator, while the forward pass stays hard so that evaluation never sees leaked information.

Training uses pairs of views $(x_a, x_b)$ of the same object. These come from CO3D video frames and MVCap renderings (Omniview-Tuning), alongside ordinary single-view ImageNet batches. The objective is

$$ \mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \sum_i (1 - C_{ii})^2 + \lambda \mu \sum_{i \neq j} C_{ij}^2, $$

where $\mathcal{L}_{\text{task}}$ is the classification or retrieval loss and $C$ is the batch cross-correlation matrix between the pooled bottleneck outputs of view $a$ and view $b$. The weights $\lambda$ and $\mu$ balance agreement against channel decorrelation, as in Viewpoint Bottleneck. The agreement term is the only pressure on the scorer beyond the task. It rewards selecting patches whose features stay stable under rotation, without saying which patches those are.

One further variable is controlled. A single low-dimensional "periphery" token carries the mean and variance of the discarded tokens. This follows the idea that peripheral vision compresses rather than deletes (Statistical Summary Representation; What your visual system sees where you are not looking). Sweeping this token's dimension tests whether a little context helps or simply reopens the background shortcut.

The analysis focuses on what gets selected. On CO3D we measure the fraction of selected patches that fall inside the provided object masks. Using the dataset's camera poses at evaluation only, we also measure whether patches selected in two views reproject to the same surface region.

Baselines

The naive reference is the same DeiT backbone with all tokens, trained on the same mix of single-view and multi-view data with the task loss only. On ObjectNet this sits in the regime where ResNet-152 reaches about 30% top-1 against about 75% on the overlapping ImageNet classes (ObjectNet). Every comparison is reported as the change from this model.

The strongest existing method on our main backbone is 3DTRL, run from its official code. It is the most direct competitor: a pose-free add-on to a vision transformer, aimed at viewpoint robustness, costing about 2% extra compute. It reports gains of 3.3% to 4.1% on viewpoint-perturbed ImageNet-1K and ObjectNet, and 74.1% to 78.8% on CIFAR-10 with DeiT-Tiny. The file gives no absolute ObjectNet accuracy for 3DTRL, so we will reproduce it. We train it on exactly our data mix, schedule, and an equal hyperparameter-search budget. Beating it means a larger ObjectNet and viewpoint-perturbed ImageNet gain than reproduced 3DTRL, at no higher compute. We also test the two together, since they are not mutually exclusive.

For retrieval, the strongest baseline is the Reranking Transformer with epipolar loss, which reports CO3D-Retrieve mAP of 49.60% on full images and 28.21% on masked backgrounds (A Light Touch Approach...). It uses ground-truth poses during training and we do not, so matching its full-image number while clearly exceeding 28.21% on masked images would count as a win. The key diagnostic is the size of the drop from full to masked images. For vision-language models, we compare against Omniview-Tuning on its viewpoint out-of-distribution benchmarks, where it reports average top-1 gains of 8.9% to 10.2% over CLIP. Here we insert our bottleneck into the CLIP image encoder and train on the same MVCap data.

Three ablations isolate the key ingredient, each at matched $k$. The first replaces the hard gate with soft attention reweighting tuned to the same attention entropy, which tests whether exact zeros matter. The second selects $k$ tokens at random, or by the classification token's attention, with the agreement loss kept. The third keeps the learned hard selector but drops the cross-view agreement term. The first two ask whether learned selection matters beyond token dropping as regularization. The third asks whether capacity alone, without view-consistency pressure, does the work.

Why not just…?

Why not just augment harder? Augmentation is cheap and must be beaten. On DroneSeg, AutoAugment gave a 3.7% gain against 6.6% for latent perspective modeling (Discrete Latent Perspective Learning). Standard augmentation pipelines did not prevent the ObjectNet collapse. Still, 2D warps cannot produce out-of-plane rotations. Our multi-view pairs can, so all baselines receive the same multi-view data and the same augmentation, keeping the comparison about architecture rather than data.

Why not just segment the object and mask the background? This is a real threat, because one reading of our hypothesis is simply "ignore the background." Masking at test time hurts badly: CO3D masked-background mAP is 27.33% against 48.81% for full images. That gap shows models trained on full scenes depend on context, but it does not settle the question for a model trained on masked inputs. We therefore include a baseline trained and tested on foreground-masked crops from an off-the-shelf segmenter. If it matches our method on ObjectNet, the contribution reduces to background removal and we will say so. Our expectation is that selection inside the object also matters, choosing parts that survive rotation over silhouette regions that do not. The part-based gains of PANet on point clouds (Beyond Viewpoint) point the same way. The reprojection-consistency analysis tests this directly.

Why not just train a multi-view consistency objective on all tokens? This is the most serious alternative. The cross-view agreement loss alone might deliver most of the gain, as Viewpoint Bottleneck's results suggest for point clouds. The all-token model with the identical loss is therefore the main control, and the core claim stands only if tightening $k$ adds robustness on top of it. If the budget curve is flat, the bottleneck is unnecessary and the paper becomes a negative result about attentional capacity.

Why not just scale the model or the pre-training data? The ObjectNet gap stayed roughly constant from AlexNet to PNASNet. CLIP still fails at unusual 3D angles until targeted viewpoint alignment is added (Omniview-Tuning). The CLIP experiment tests whether a bottleneck still helps on top of web-scale pre-training.

Proposed Experiments

The decisive experiment is the budget sweep. We train with $k \in \{8, 16, 32, 64, \text{all}\}$ patches out of 196 and plot, for each $k$, ImageNet accuracy against ObjectNet accuracy, and CO3D full-image mAP against masked mAP. The hypothesis survives if three things hold: the ObjectNet gap and the full-to-masked drop shrink at intermediate $k$; this happens only when the gate is hard and the agreement loss is present; and the effect exceeds both the all-token agreement model and the segmentation-masking baseline.

The selection analysis must show where the gain comes from. Selected patches should increasingly fall on the object and reproject to corresponding surface points across views as $k$ shrinks. If the gain appears without this correspondence, it is better explained as regularization by token dropping.

A secondary, exploratory test probes the failure documented in Approximating Human-Level 3D Visual Inferences. There, multi-view networks matched humans within ShapeNet categories (accuracy 0.93) but fell to chance agreement with humans on novel ShapeGen shapes. We run the same match-to-sample protocol on bottlenecked and unbottlenecked encoders. Either outcome is informative about whether capacity limits change what transfers to novel shapes. We do not make it a headline claim.