Audio-visual models usually assume that whatever you hear is visible on screen. Real video breaks that assumption all the time: a television behind the camera, a speaker around a corner, a train heard but not yet in frame. When it breaks, models attach the sound to the wrong thing, for example a silent guitar on the wall or the railway tracks instead of the train. Teaching a model to say "this sound comes from something you cannot see, behind you and to the left of the door" needs labels for invisible things. Real video never has those labels. Simulators do. A 3D acoustic simulator knows every object, where every sound comes from, and which of them the camera can see. The open question is how to move that supervision to real footage without corrupting it along the way.
We propose to render simulated rooms, convert the rendered views into realistic-looking images with unpaired image-to-image translation (learning a mapping between two image collections that have no matched pairs), and train an off-screen-aware sound localizer on the result. The new ingredient is the constraint on the translator. We do not ask it to reproduce pixels on a round trip. We ask it to preserve the scene graph, meaning the set of visible objects and the relations between them. We also ask it to preserve absences: it must not paint into the frame any object of the class that the label says is off-screen. If a translator invents a television in the corner, it silently flips an "off-screen television" label into a wrong one. The core claim is that preserving relational structure, absences included, is the property sim-to-real translation needs when the labels are themselves statements about what is not visible.
Core Claim
Off-screen sound attribution should be posed as structured prediction over an extended scene graph. Nodes are visible objects plus "off-screen" nodes created on demand. Edges relate an off-screen source to the visible layout and to the camera ("behind camera," "beyond the left edge," "behind the door"). Each sound event is assigned either to a visible node or to an off-screen node. That assignment can only be supervised in simulation, so the bottleneck is transfer. Our claim is that a translator constrained by scene-graph and absence consistency gives better real-world off-screen attribution than three alternatives: no translation, cycle-consistent translation, and detector-only consistency. Most of the gain should show up as fewer false on-screen attributions to silent same-class objects.
Positioning Against Prior Work
The motivating evidence comes from localization work that handles off-screen sound only as a yes/no decision. SLAVC splits localization into a spatial term and a "does audio match this video" term, and uses heavy visual dropout so it can reject absent sources. A Proposal-based Paradigm gates on a global on-screen activation before choosing among object proposals. Both output a scalar "not here." Neither says where the source is relative to the room. SLAVC also reports near-zero accuracy for sounding objects smaller than 32×32 pixels.
Object-aware SSL separates an active foreground object from a silent same-class one, using captions produced by a multimodal large language model during training. It works within the 2D frame and cannot place a source outside it.
Egomotion-supervised localization learns sound direction from camera rotation, including directions behind the camera. It does not tie that direction to any entity or surface.
ASMP is the closest structural precedent. It builds a 2.5D scene graph from detected objects and uses subgraphs to separate sounds. Its nodes come only from detections inside the visible frame, so an undetected or off-screen source has nowhere to go. We keep its idea that scene graphs should condition audio. We add nodes for things the camera cannot see.
The method side draws on two kinds of work. On the simulation side, SoundSpaces 2.0 supplies geometry-aware sound propagation on real scanned meshes, through the Habitat-Lab platform. SoundLoc3D shows that simulated supervision can localize visually unremarkable sources, but it assumes the source sits on a surface the camera observes.
On the translation side, Patch Your Matcher replaces generic realism with a discriminator that scores two-view correspondences through a frozen matcher. ToDayGAN adds gradient discriminators so that night-to-day translation keeps the structure needed for retrieval. Both establish the principle we reuse: constrain the translator with the downstream task's invariant. Both preserve dense geometry, not object identity or absence. Neither prevents hallucinated objects, which is the failure that matters here.
Our structural prior is the scene-graph-to-image generator of sg2im, which uses an object-level discriminator. That model is supervised. We use graph consistency as the content constraint in an unpaired setting instead.
Research Program
The data engine renders egocentric clips in HM3D and Matterport3D rooms with SoundSpaces 2.0. Each clip comes with binaural audio and the full 3D scene graph. The audio uses real recorded source sounds convolved with simulated room responses, so that source timbre is real. For each clip we compute the visible subgraph $S_{vis}$ and the set of off-screen sounding entities $U$. About half the clips are built to contain a "distractor": a silent object of the same class as an off-screen source, placed inside the frame. That pattern is the one Object-aware SSL identifies as the dominant false-attribution failure.
The translator $G$ maps rendered frames to the distribution of real egocentric video, drawn from StereoWalks and real indoor 3RScan imagery. We train it with an adversarial loss plus a graph term computed by a frozen scene-graph parser $P$:
$$\mathcal{L}_{graph}(x) = \mathrm{CE}\big(P(G(x)),\, S_{vis}\big) \;-\; \lambda \sum_{c \in U} \log\big(1 - p_c(G(x))\big).$$
Here $x$ is a rendered frame. $\mathrm{CE}$ is the cross-entropy between the parser's predicted nodes and predicates and the simulator's visible subgraph. $p_c(G(x))$ is the parser's highest detection score for off-screen class $c$ anywhere in the translated frame. $\lambda$ weights the absence term. In plain terms, the first term keeps what the label says is present, and the second keeps out what the label says is absent. We test a GAN and a neural Schrödinger bridge as the translation backbone, so the result does not hinge on one generator family.
The localizer parses the frame into a visible graph. Sound-event embeddings then attend over visible nodes plus a pool of learnable off-screen slots. Each event is assigned to one node. When it lands on an off-screen slot, the model predicts a direction relative to the camera and a predicate linking the slot to a visible node. On real binaural video we add the egomotion rotation-consistency loss as an unlabeled regularizer.
Baselines
The naive reference is the same localizer trained on raw renders, with no translation and only standard color and noise augmentation. It shows how much of the problem is the visual domain gap rather than the model.
The strongest existing off-screen method is SLAVC, which we retrain on our data and code. A Closer Look at Weakly-Supervised Audio-Visual Source Localization reports 32.95 average precision on Extended VGG-SS and 83.60% localization accuracy on Flickr-SoundNet without early stopping. On mono benchmarks we run it with the published protocol. We count SLAVC as beaten if we exceed 32.95 average precision on Extended VGG-SS. On the distractor split we also require a lower false-attribution rate.
For direction, the reference is the egomotion model. Its paper reports a mean absolute error of 34.0° on YT-Stereo-iPhone, and our model must match or improve on that while also naming the entity. For simulated 3D localization, the reference is SoundLoc3D, which reports 0.518 mAP and 0.320 m mean localization error on its simulated dataset. No reported result was found on Lacuna for relational off-screen attribution, so on that task all comparisons are internal.
Translation ablations hold the localizer, data and tuning budget fixed. We use the same number of generator updates and the same hyperparameter sweep size for each, and change only the content constraint. The variants are cycle consistency, which is CycleGAN as reviewed in GAN developments; a ToDayGAN-style gradient discriminator; detector-only consistency with no predicates and no absence term; and the full objective with and without the absence term. The absence-term ablation isolates the key ingredient. If removing it does not raise the false-attribution rate on distractor clips, the central claim fails.
Why not just…?
Why not just train on raw renders and rely on audio, since audio carries the off-screen information anyway? Audio says that something is off-screen and roughly where. Vision supplies the anchor graph and rules out the silent distractor. A localizer trained on renders will misparse real frames, and every misparsed node corrupts attribution. ToDayGAN's own comparison, 13.9% versus 52.9% of night queries localized within 5 m and 10°, shows how badly a visual domain gap can hurt a downstream geometric task. The naive baseline above measures this directly. If it comes within noise of our model on real data, translation is unnecessary and we would drop it.
Why not just use CycleGAN or another standard unpaired translator? Cycle consistency only asks that the input be recoverable. A generator can add a plausible television to a corner, hide the cue it needs to undo the change in high-frequency detail, and pass. That edit is harmless for appearance and fatal for an "off-screen television" label. We measure this directly as the rate at which the parser finds off-screen classes in translated frames, compared across constraints. This alternative is a real threat if the hallucination rate turns out to be low in practice, and the ablation will tell.
Why not just skip simulation and use self-supervision on real video, egomotion for direction plus SLAVC-style gating for presence? This is the most serious competitor. It may well match us on direction error. It cannot output which entity, or which relation to the room, an off-screen sound belongs to, because no real signal supervises that. We include the combination as a baseline. Our claim survives only if the combination loses on entity and relation accuracy and on distractor rejection, not merely on direction.
Why not just ask a multimodal language model? On EgoSound, the best model reaches 56.7% against 83.9% for humans, with the largest failures on off-camera and causal questions. We report our model's accuracy on EgoSound's localization questions alongside these numbers.
Proposed Experiments
The decisive real-world test is a new evaluation set. We will annotate several hundred real binaural clips from StereoWalks plus clips recorded in scanned rooms. Each sound event gets one of three labels: a visible object, an off-screen entity with a coarse direction and a relation to a visible object, or ambient. Metrics are attribution accuracy, false-attribution rate to silent same-class objects, direction error, and predicate accuracy.
On existing benchmarks we report average precision on Extended VGG-SS and localization accuracy on Flickr-SoundNet, both for on/off-screen rejection; direction error on StereoWalks; and SoundLoc3D's metrics on its simulated split, to check transfer of the 3D localization ability. For the translator itself we report visible-graph recall and the hallucination rate of off-screen classes on held-out renders. We also break results down by object size, to see whether graph-level anchoring reduces SLAVC's small-object collapse.
The result that would most support the proposal is on the real annotated set. The full model must reduce distractor false-attribution relative to the egomotion-plus-gating baseline and relative to the cycle-consistent translator, and the absence-term ablation must recover most of that gap. If cycle-consistent and graph-consistent translation tie, the structural constraint is not earning its place.
References
- A Closer Look at Weakly-Supervised Audio-Visual Source Localization
- A Proposal-based Paradigm for Self-supervised Sound Source Localization in Videos
- Object-aware Sound Source Localization via Audio-Visual Scene Understanding
- Supervising Sound Localization by In-the-wild Egomotion
- Learning Audio-Visual Dynamics Using Scene Graphs for Audio Source Separation
- SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning
- SoundLoc3D: Invisible 3D Sound Source Localization and Classification Using a Multimodal RGB-D Acoustic Camera
- Patch Your Matcher: Correspondence-Aware Image-to-Image Translation Unlocks Cross-Modal Matching via Single-Modality Priors
- Night-to-Day Image Translation for Retrieval-based Localization
- Image Generation From Scene Graphs
- Unpaired Image-to-Image Translation via Neural Schrödinger Bridge
- Generative Adversarial Networks: Recent Developments
- EgoSound: Benchmarking Sound Understanding in Egocentric Videos