Head-worn 360° cameras record everything around the wearer. When someone later watches that footage in a VR headset, they usually miss the moment that mattered, because they are free to look anywhere. The common fixes are overlays such as arrows, or rotating the whole sphere so the event sits in front of the viewer. Overlays break immersion, and forced rotation makes people sick. The alternative is implicit diegetic guidance: small changes inside the scene itself, such as a slight brightening or a local contrast lift, that draw the eye without being noticed. Current systems for this are either hand-tuned effects with no measured effect on real gaze, or optimized edits scored only by a gaze-prediction model. The second kind can fool the scoring model without moving any human eye.
We propose to generate guidance edits by optimizing them against a differentiable saliency model, which predicts where viewers will look. Each edit is a smooth brightness/contrast gain field defined directly on the sphere and restricted to low spherical-harmonic order. This restriction does two jobs. It makes an edit's strength independent of where it sits on the equirectangular image. It also stops the optimizer from exploiting high-frequency blind spots in the saliency model, so predicted gains have a chance of matching real human gaze shifts. The application is shaky first-person 360° footage. The decisive test is a head-mounted eye-tracking study that checks whether predicted and human gaze shifts agree as the allowed edit detail grows.
Core Claim
A saliency model is a useful guide for editing only within a band of edit detail where its predictions track human behavior. Spherical harmonics make that band an explicit, tunable parameter: the maximum harmonic order $L$. We claim three things. Edits limited to $L \le L^*$ for a modest $L^*$ produce real, measurable, largely undetected gaze shifts toward target events in egocentric 360° video. Above $L^*$, predicted gains keep rising while human gains flatten, which is the signature of the optimizer gaming its critic. Finally, doing the edit on the sphere rather than on the flat equirectangular grid matters most for targets far from the equator. First-person footage produces such targets constantly: hands, the ground, and objects held low.
A correction that shapes the design
The research file argues that head roll and pitch break latitude-dependent spherical kernels such as Kernel Transformer Networks, which learn row-specific versions of planar filters. That is not quite right. Equirectangular stretching depends only on the image row, whatever the head is doing. What egocentric motion actually breaks is the content prior. Saliency models learn that people look near the horizon and that objects appear upright. When the wearer tilts their head, the horizon slides off the image equator and those learned priors become wrong.
This is fixed by a rotation, not a new operator. We rotate every frame into a gravity-aligned, heading-smoothed frame using the stabilization of Joint Stabilization and Direction of 360° Videos, which estimates camera rotation for shaky head-worn 360° footage. We then apply this same preprocessing in every experimental arm. The first-person setting therefore enters the proposal in two places. It supplies the footage, and it imposes a constraint: edits live in the stabilized world frame, so they stay attached to scene content rather than to the jittering camera. Residual translation, parallax and motion blur remain. These are exactly the conditions under which subtle cues get drowned out, a failure mode noted in the file for motion-based cues such as illusory-motion attention cues. For that reason we deliberately use slowly varying brightness/contrast fields rather than motion cues.
Research Program
The edit is a multiplicative local-contrast gain on the stabilized sphere:
$$g(\omega,t) = 1 + \sum_{\ell=0}^{L}\sum_{m=-\ell}^{\ell} a_{\ell m}(t)\,Y_{\ell m}(\omega),$$
Here $\omega$ is a viewing direction on the unit sphere and $t$ is time. The $Y_{\ell m}$ are real spherical harmonics, the sphere's equivalent of Fourier modes. The $a_{\ell m}(t)$ are the learned coefficients, and $L$ caps how much spatial detail an edit can carry. The gain is applied to a luminance/contrast decomposition of each frame and then resampled to equirectangular form for display.
The coefficients are chosen to maximize a critic's predicted gaze mass on the target region $T$. The critic $S$ is a saliency predictor. The optimization is penalized for visible change and temporal flicker:
$$\max_{a}\; \mathbb{E}_t\!\left[\int_{T} S(I_t \odot g_t)\,d\omega\right] - \lambda_1 \,\mathrm{E}_{\text{vis}}(g) - \lambda_2 \sum_t \|a(t+1)-a(t)\|^2 .$$
$\mathrm{E}_{\text{vis}}$ is a contrast-visibility cost integrated over solid angle (not over pixels). $\lambda_1$ and $\lambda_2$ trade guidance against noticeability and flicker. The optimization is solved per clip, offline, with the target event's onset known, and the edit ramps in a second or two before the event.
The critic is a differentiable spherical saliency model. We use two families so we can test cross-critic transfer. The first is the tangent-plane transformer of SalViT360-AV, which splits frames into gnomonic tangent views and processes ambisonic spatial audio aligned to each view. The second is a KTN-backbone saliency network trained on the same data. Both are trained on YT360-EyeTracking, D-SAV360, AVS-ODV and VR-EyeTracking. Footage comes from EgoK360, a set of 640×320 equirectangular head-worn activity videos with fine-grained action labels. Its action onsets serve as target events. We supplement it with head-worn clips of the kind used by Tang et al.
This work builds on three strands in different roles. Target Scanpath-Guided 360-Degree Image Enhancement (PSEM) shows the method motivation: it removes distracting objects by inpainting to reshape scanpaths. However, it works on still images, edits on the flat grid, and reports its gains against predicted scanpaths. We keep its scanpath metrics but move to video, work on the sphere, and require human validation. Focus360 applies hand-set blur, darkening and fade effects around tracked targets and reports no behavioral numbers. It becomes our heuristic baseline. SphereNet and KTN supply the evidence that distortion-aware processing beats flat processing on the sphere. That evidence justifies placing both the edit and the critic on the sphere rather than on the equirectangular grid.
Baselines
The do-nothing reference is the stabilized clip with no edit. Every gain is measured against it with the same participants, clips and targets.
The strongest existing method is kinematic re-direction from Joint Stabilization and Direction of 360° Videos. It rotates the whole sphere so salient events land in the viewer's forward direction. The paper reports that the percentage of key events seen rose from 34.79% in raw videos to 56.18% with its method. We run their energy formulation on the same clips, with the same target events supplied as saliency. Beating it means matching or exceeding its percentage seen, or matching it with lower simulator-sickness and detectability scores. The second outcome would still be a meaningful win, because it means comparable guidance at less cost to comfort and immersion.
The heuristic diegetic baseline is Focus360-style effects: radial darkening and background blur around the target. We apply them at the same solid-angle budget as our edits, both on the equirectangular image as that system does and on the sphere. No reported quantitative result was found on Lacuna for Focus360.
The audio baseline boosts the ambisonic gain of the target's sound source by the same amount as a matched loudness budget. This is motivated by SalViT360-AV's report that adding ambisonics raised NSS (a measure of how much predicted saliency lands on actual fixations) on YT360-EyeTracking from 1.738 to 2.449 over its visual-only version.
For critic quality, the reference is CASP, which reports CC 0.604 and NSS 4.545 on AVS-ODV. Our critics need not beat it, but we report both critics on the same splits so readers can see how edit quality depends on critic quality.
Two ablations isolate the key ingredients. The first optimizes an unconstrained per-pixel gain map on the equirectangular grid against the same critic. This removes both the band limit and the spherical parametrization. The second keeps the spherical harmonic parametrization but sweeps $L$ from 2 to 32, and each setting is tuned with the same optimization budget and the same visibility budget.
Why not just…?
Why not just rotate the sphere toward the event, as Tang et al. do? This is the real threat, since it has a reported 21-point gain in percentage seen. Its costs are the ones the same paper acknowledges: it needs a smoothness term to avoid rotational snaps, and rotation the viewer did not initiate conflicts with their own head motion in a headset. It also overrides viewer autonomy, which diegetic guidance tries to preserve. We do not assume it loses. The headset study measures percentage seen, simulator sickness and detection for both methods. If rotation wins on every axis, the right conclusion is that implicit edits are only worth using where rotation is unacceptable, such as with seated viewers who are already turning their heads. We also test the combination.
Why not just add hand-tuned blur or darkening around the target? It is the obvious move, but it has no behavioral evidence. There is also a geometric problem specific to this footage: an effect with fixed pixel radius on the equirectangular image covers a very different solid angle near the poles, where egocentric targets such as hands and the ground often sit. The latitude-binned analysis described below shows whether this matters in practice. If heuristic spherical vignettes match optimized edits, the critic is unnecessary and we will say so.
Why not just use spatial audio? Audio is a strong, natural cue, as the NSS jump in SalViT360-AV shows. But many egocentric targets are silent, such as an object being picked up, and crowded soundscapes defeat audio-query models, as CASP itself notes. The audio arm measures the size of this effect directly. We report results separately for sound-emitting and silent targets, and we test audio and visual edits combined.
Why not trust the saliency model's predicted gain and skip human studies? Because the claim is about the gap between prediction and behavior. PSEM's reported 10.98% average reduction in scanpath edit distance is measured against predicted scanpaths. An optimizer can inflate such numbers without moving real eyes. The bandwidth sweep with eye tracking exists specifically to measure that gap.
Proposed Experiments
We plan a within-subject headset study with eye tracking. Each participant sees stabilized EgoK360 and head-worn clips under each arm, with clip-to-arm assignment counterbalanced. Target events are the annotated action onsets, restricted to targets initially outside the viewer's likely forward view.
Primary measures are the percentage of target events seen (Tang et al.'s metric) and time to first fixation on the target. Secondary measures are PSEM's scanpath edit-distance and recurrence scores against a target scanpath, a post-clip forced-choice test of whether the clip was modified (to check that guidance was implicit), and standard presence and simulator-sickness questionnaires.
The decisive check is the bandwidth sweep. For each $L$ we plot critic-predicted gain against human gain in time to first fixation. The hypothesis predicts close agreement at low $L$ and divergence beyond some $L^*$. Two results would falsify it: human gain tracking predicted gain across all $L$, which would mean the band limit is unnecessary, or no human gain at any $L$, which would mean the critic is useless as a guide.
A second check tests robustness to critic choice. Edits optimized against the SalViT360-AV critic are scored by the KTN critic, and vice versa. Cross-critic agreement should predict human gain better than the optimizing critic's own score does.
A third check bins targets by latitude in the stabilized frame. It compares spherical edits with the equirectangular per-pixel ablation at matched solid-angle visibility. We expect the spherical advantage to grow toward the poles. If there is no advantage at any latitude, the spherical parametrization is working only as a regularizer, and its geometric justification should be dropped.
References
- Kernel Transformer Networks for Compact Spherical Convolution
- Joint Stabilization and Direction of 360°Videos
- Generating spatial attention cues via illusory motion
- Spherical Vision Transformers for Audio-Visual Saliency Prediction in 360$^{\circ }$∘ Videos
- Egok360: A 360 Egocentric Kinetic Human Activity Video Dataset
- Target Scanpath-Guided 360-Degree Image Enhancement
- Focus360: Guiding User Attention in Immersive Videos for VR
- SphereNet: Learning Spherical Representations for Detection and Classification in Omnidirectional Images
- CASP: Consistency-aware Audio-induced Saliency Prediction Model for Omnidirectional Video