Compiling Spherical Redundancy into Dense Tiles

Wide-angle and 360° vision models usually store the sphere as an equirectangular image. This is the familiar world-map layout, where every row of latitude gets the same number of pixels. Rows near the poles describe much less of the actual sphere than rows near the equator, yet the network spends the same arithmetic on each. That redundancy is a natural target for pruning, and the obvious move is to borrow activation-sparsity tricks that skip "unimportant" pixels at run time. Those tricks have a poor record on real hardware, though. Skipping scattered zeros costs indexing, masking and irregular memory access, and that cost often eats the savings. This proposal starts from one observation. The spherical redundancy is fixed by geometry and known before any image arrives, so it does not need to be treated as sparsity at all. It can be removed when the model is compiled, by repacking the sphere into dense, hardware-aligned bands. Ordinary dense matrix kernels then run on it with no mask and no index lookups.

The claim is deliberately bounded. On the sphere, area per latitude row scales with $\cos\theta$, where $\theta$ is latitude. Summing over all rows, an equirectangular grid holds $\pi/2 \approx 1.57$ times more samples than an equal-area grid with the same equatorial resolution. The geometric saving is therefore capped at roughly 36% of pixels, or about 1.57× in compute and activation memory. We propose to show that a banded equal-area layout comes close to this ceiling in measured latency, on both a datacenter GPU and an edge CPU, with no loss of depth accuracy. We also expect run-time spatial pruning given the same geometric prior to fall well short of that ceiling. If both hold, the lesson for the sparsity field is concrete: when the zero pattern is static and structured by geometry, change the layout instead of building a sparse kernel.

Core Claim

The proposed representation divides the sphere into horizontal latitude bands of $h$ rows each. Band $b$, centered at latitude $\theta_b$, is resampled to a width $$W_b = 32\left\lceil \frac{W\cos\theta_b}{32} \right\rceil,$$ where $W$ is the equatorial width of the original equirectangular image. The rounding to a multiple of 32 matches the tile width used by GPU matrix engines. Within a band, every convolution is an ordinary dense convolution with circular padding in longitude. Only the halo rows at band boundaries need special treatment: these are the $k-1$ rows a $k \times k$ kernel needs from the neighboring band, which has a different width. We fill them with a fixed interpolation operator that is computed once.

The overhead then has a simple form. Halo work grows roughly as $(k-1)/h$ of the band's cost, because each band of $h$ rows imports $k-1$ resampled rows. Area fidelity, by contrast, improves as $h$ shrinks, because the width can track $\cos\theta$ more closely. One parameter, $h$, therefore controls the whole trade-off between geometric savings and boundary cost. We will measure this trade-off rather than assume it.

The new element is the operating point. The work moves the spherical redundancy out of run-time masking and into a static dense layout. It then measures how much of the 1.57× geometric ceiling survives as wall-clock speedup, compared head-to-head with run-time sparsity methods given the same prior.

Relation to Prior Work

Two spherical-depth systems supply the application and the accuracy targets.

  • Depth Any Camera converts perspective, fisheye and panoramic inputs into a shared equirectangular frame. It reaches zero-shot metric depth across camera types but runs dense convolutions over the oversampled polar rows.
  • FoVA-Depth uses equirectangular or cubemap canonical formats with custom circular and cross-face padding. Those padding layers are implemented as dense operations, and their latency is never measured.

We keep both systems' geometry and change only how the sphere is laid out in memory. Two older results motivate the equal-area view:

  • Hansen et al. showed that doing scale-space analysis on the sphere itself makes radial distortion space-invariant.
  • PIT handles uneven pixel density by reweighting the training loss with the local area of each pixel. Its inference pass stays fully dense.

We use the same area factor PIT uses, but we spend it on removing computation rather than on rescaling gradients.

The sparsity literature both constrains the design and supplies baselines.

  • Grimaldi et al. prune whole spatial positions so that columns of the unrolled convolution matrix can be skipped. They report a 1.25× speedup on an ARM Cortex-A72 with a 1.1% accuracy drop. Their masks know nothing about geometry, and they hit an accuracy wall beyond moderate sparsity.
  • Sputnik and SMaT show that unstructured sparse kernels only beat dense cuBLAS above roughly 71–78% sparsity. That is far beyond the 36% geometric redundancy available here. SMaT also depends on offline row reordering.
  • Tile-wise sparsity and N:M sparsity recover speed by forcing structure. N:M needs specific Tensor Core hardware.

Together these results predict that run-time skipping of polar pixels will lose to a dense layout. Our design follows that prediction rather than arguing against it.

OmniMVS builds a dense four-dimensional cost volume over concentric spheres, with matching scores stored for every pixel at every candidate depth. Its authors name memory as the limit on resolution. Every depth slice in that volume carries the same equirectangular oversampling, so OmniMVS is our second application.

Research Program

We will build three things.

  1. Banded resampler and layout. This component converts between equirectangular images and the banded layout. Bands are stored contiguously, with each band's width a multiple of the tile size.
  2. Band-aware convolution operators. We will provide a 2D version and a 3D version for cost volumes. Each runs a dense GEMM, or a vendor dense convolution, per band. The halo rows come from a precomputed static interpolation matrix. Bands of equal width are fused into a single batched call so that kernel launch overhead stays small. Both a CUDA path (A100) and an XNNPACK path (Cortex-A72) will be implemented, so results can be compared directly with Grimaldi et al.
  3. Adaptation recipe. A model trained on equirectangular inputs must be adapted to the new layout. We start from released DAC- and FoVA-Depth-style weights and fine-tune briefly on the same training data. Equal-area bands make horizontal pixel spacing more uniform across latitude, and this could plausibly help accuracy rather than hurt it. We will test that rather than assume it.

As an optional second stage, we will add content-dependent skipping at the granularity of whole tiles inside a band, for example uniform ceiling or sky regions. Tiles are already aligned inside fixed-width bands, so no reordering is needed. This sidesteps the problem SMaT's preprocessing has with input-dependent patterns. This stage is optional because the geometric part must stand on its own first.

The analysis will plot measured speedup against band height $h$ and kernel size $k$. We will check whether halo overhead follows the predicted $(k-1)/h$ scaling, and how close each configuration gets to the 1.57× ceiling.

Baselines

Dense equirectangular model (do-nothing reference). This is the same network run at full resolution with circular padding, trained and fine-tuned for the same number of steps on the same data. For accuracy, the target figures are DAC's reported zero-shot delta-1 of 0.77 on Matterport3D and 0.85 on ScanNet++ fisheye, from Depth Any Camera. For stereo, it is OmniMVS's mean absolute error of 1.04 on OmniHouse at about 1.06 s per frame on a GTX 1080Ti, from OmniMVS. We will re-measure all of these on our own hardware rather than compare across machines. To count as beating this baseline, the banded model must stay within 0.01 delta-1, or within 2% relative error, while reaching at least 1.4× measured end-to-end speedup and a matching reduction in peak memory.

Cubemap with cross-face padding. This is the strongest dense competitor, following FoVA-Depth. FoVA-Depth reports an AbsRel of 0.232 on Matterport360 cubemaps for multi-view stereo. A cubemap is also roughly equal-area and fully dense, so it is the most direct threat to this proposal. We will train it with the same backbone, data and tuning budget, and compare at matched latency. Beating it means better accuracy at equal latency, or lower latency at equal accuracy, with seam errors measured separately along cube edges.

Latitude-weighted run-time activation skipping. This baseline is the obvious combination of the two research directions. We give the spatial pruning of Grimaldi et al. exactly the same $\cos\theta$ prior, dropping polar positions at the matching rate, and run it with their modified XNNPACK on the Cortex-A72. On the GPU, we run the equivalent unstructured pattern through Sputnik. Their reported 1.25× on ARM sets the scale. Beating this baseline means a higher measured speedup at the same pixel budget. This is the key ablation: the zeros are identical and only the execution strategy differs.

Uniform downsampling. This baseline shrinks the whole equirectangular image until its compute matches the banded model's. It isolates whether keeping equatorial resolution matters at all.

2:4 weight sparsity with SR-STE. Following Zhou et al., this is reported both on its own and combined with banding on the A100. The two methods act on different things, weights versus spatial positions, so we expect their savings to multiply. We will check whether they actually do.

Why not just…?

Why not just use a cubemap or tangent patches? This is the real threat. Cubemaps are dense and close to equal-area. However, FoVA-Depth needed special cross-face padding to suppress seams, and the cubemap's AbsRel rose from 0.232 to 0.413 without rotation augmentation. That suggests sensitivity to how content falls across faces. OmniMVS also reports seam artifacts and lost context when perspective patches are stitched. The bands are seamless in longitude and have only horizontal boundaries. If the matched-latency comparison shows the cubemap equal or better, the honest conclusion is that cubemaps already capture the geometric saving, and we will report it that way.

Why not just prune polar activations at run time with an existing kernel? The pattern here is only about 36% sparse. Sputnik only beats dense execution from 71% sparsity, and SMaT only from 78%. Even the column-skipping scheme of Grimaldi et al. reaches 1.25× at its operating point. The latitude-weighted skipping baseline tests this directly with identical zeros.

Why not just downsample the image, or reweight the loss as PIT does? Uniform downsampling throws away equatorial detail along with polar redundancy, and the matched-compute baseline measures that cost. Loss reweighting in the style of PIT changes training only and leaves inference fully dense, so it saves no latency.

Why not just apply 2:4 sparsity? It is orthogonal to banding, not a replacement, and it needs Ampere-class hardware, which gives nothing on edge CPUs. We will report the combination.

Proposed Experiments

Monocular depth. Fine-tune from equirectangular-trained weights and evaluate zero-shot on Matterport3D and ScanNet++ fisheye, and on KITTI-360 fisheye, where DAC's outdoor gains were weaker. We report delta-1 and AbsRel overall and per latitude band. The per-band breakdown checks that the poles do not degrade disproportionately.

Stereo. Apply the 3D band operator to the OmniMVS cost volume on OmniThings and OmniHouse. We report mean absolute error, runtime, and peak memory. We also report the highest depth-map resolution that fits in a fixed memory budget, because memory is the bottleneck OmniMVS names.

Decisive checks. There are three, and any one of them can falsify the proposal.

  1. Measured speedup must reach at least 1.4× of the 1.57× ceiling on both the A100 and the Cortex-A72, while latitude-weighted run-time skipping, given the same zeros, stays clearly below that on the same hardware.
  2. Accuracy must stay within the stated tolerance of the dense equirectangular model.
  3. The banded layout must match or beat the cubemap at equal latency.

If the first check fails because halo cost is larger than the $(k-1)/h$ prediction, that is itself a useful negative result about spherical operators. If the third fails, the contribution narrows to a careful measurement showing why run-time sparsity is the wrong tool for geometric redundancy.