Listening to Trajectories: Audio-Guided Segmentation of Moving 3D Gaussian Scenes

Most sounds in a room are made by things moving. A hand hits a table, a door swings shut, a person claps, a ball bounces. Current methods that use audio to find "which object is making the sound" in a 3D reconstruction assume nothing in the scene moves. Current methods that reconstruct moving scenes ignore the soundtrack. We propose to join the two at the one place where sound and motion are physically linked, which is how each small piece of the scene moves over time. The scene is represented with 4D Gaussian splatting, a set of small colored ellipsoids that can be rendered quickly and that each carry their own motion through time. An audio encoder then asks "which of these motion histories caused this sound?" It does this by attending over short per-primitive trajectories, not over image pixels.

The payoff is a 4D sounding-object segmentation: a 3D mask of the emitting object that moves through time and can be rendered from any viewpoint. A second payoff is a sharper split between objects exactly where visual-only methods blur them, at the moment of contact. Two things touching look like one blob to motion clustering, but the sound tells you which one was struck. The two-sentence version: sound in dynamic scenes is caused by motion events, so audio should be matched against per-primitive acceleration and velocity histories rather than against pixels. That matching separates adjacent sounding and silent objects that spatial audio intensity and pixel attention both confuse.

Core Claim

The closest existing system, 3D Audio-Visual Segmentation (EchoSegnet), reconstructs a static Gaussian scene, lifts 2D sounding-object masks onto it, and refines them with an audio intensity map. It assumes a static scene. It also documents a failure: a silent oven next to an active microwave gets segmented because spatial audio intensity bleeds across the boundary. Visual-only dynamic decomposition, exemplified by Split4D, gives each Gaussian a velocity and an identity feature and clusters Gaussians that move together. It reports 0.893 mIoU on Multi-Human against 0.592 for the tracking-based SA4D. Split4D is silent, though, and its authors' setting leaves Gaussians shared across objects that touch or move along similar paths.

Our claim is that these two failures have a common fix. Spatial audio resolves position coarsely, much more coarsely than the neighboring objects are separated. Audio does resolve time finely, and the time course of a sound (its onsets, its periodicity, its decay) is caused by the time course of some object's motion. Adjacent objects are close in space but usually distinct in their motion histories. Attention computed in trajectory space should therefore separate them where attention in image space or intensity space cannot. This is different from EchoSegnet, which has no time axis. It is different from Split4D, which has no audio. It is also different from audio-driven Gaussian avatars like GaussianTalker, which use audio to generate the motion of a single parametric head rather than to explain the observed motion of an arbitrary scene.

Research Program

The backbone is a Split4D-style reconstruction trained on multi-view synchronized video. Each Gaussian carries a velocity and an identity feature, and training is chronological. Split4D's linear velocity holds only over short windows, as UFO-4D also notes for its own velocity model. We therefore read motion over sliding windows of roughly half a second. Within each window we record every Gaussian's position, velocity, and acceleration magnitude at the video frame rate.

Fusing audio with millions of Gaussians is too expensive. We pool Gaussians into a few hundred trajectory tokens using Split4D's identity clusters, subdivided spatially so no token spans a large region. Each token summarizes the motion history of its members.

Audio enters as a sequence of log-mel frames over the same window, encoded by a pretrained audio encoder. The fusion is a cross-attention decoder in which audio frames are the queries and trajectory tokens are the keys and values. This mirrors the design that Audio-Visual Synchronisation in the wild found better than global pooling for fleeting, localized sounds (52.7% versus 49.4% on VGG-Sound Sync). Our keys are motion histories of 3D primitives rather than image patches. A learned gate, following gated audio attention for video-text retrieval, lets the model attend to nothing when the sound is ambient hum or comes from off screen. This addresses the missing-modality failure that the synchronisation paper reports. The output is a per-token, per-time sounding score, which is broadcast to member Gaussians and rendered as masks from any view.

Training is mostly self-supervised. Following the contrastive objective in the synchronisation work, the correct window of audio must score its trajectory tokens higher than time-shifted audio from the same clip. This teaches the model to use timing rather than scene priors. On synthetic data we add direct mask supervision.

The same scores feed back into decomposition. When a Split4D cluster contains tokens whose sounding scores diverge sharply at an audio event, we add a repulsion term to the contrastive clustering loss for that pair. This lets the sound split Gaussians that visual motion alone had merged, such as a hand and the cup it strikes.

A deliberately simple non-learned version is central to the program and serves as a strong baseline. Let $e(t)$ be the audio onset envelope and $a_k(t)$ the mean acceleration magnitude of token $k$. The handcrafted score is

$$s_k = \max_{|\tau| \le \tau_0} \mathrm{corr}\big(e(t),\, a_k(t-\tau)\big),$$

where $\tau_0$ is a small lag allowance of a few frames for audio-video offset. If this score matches the learned attention everywhere, the paper's conclusion becomes "trajectory-space correlation suffices," and we would report that. We expect the learned model to win only where onsets are absent or overlapping. Continuous sources (speech, running engines, water) and simultaneous sources have spectral structure that must be tied to periodic or sustained motion, and an onset envelope cannot capture this.

Data is the main practical obstacle. No benchmark in our file pairs dynamic multi-view capture with sounding-object 3D masks. We build a synthetic dynamic extension of the 3DAVS-S34-O7 setup, which uses Habitat scenes with SoundSpaces 2.0 spatial audio. In it, rigid objects are animated through struck, dropped, pushed, and silently carried motions, with sounds emitted only at physically plausible events. The benchmark deliberately includes a proximity subset in which a sounding object sits within a few centimeters of a moving silent object. For real data we use Replay-NVAS, the multi-view apartment captures with conversations and exercise used by AV-Cloud. We will hand-annotate sounding-person masks on a held-out subset of frames and views.

Baselines

The naive reference point is motion-only saliency: label as "sounding" whatever Gaussians move most in the window, with no audio at all. This baseline matters because in many clips the moving thing is the sounding thing. Any audio method must beat it clearly on clips where a silent object also moves.

The strongest existing competitor is a dynamic port of EchoSegnet. In each frame we produce 2D sounding masks with the audio-guided decoder of Open-Vocabulary Audio-Visual Semantic Segmentation, which reports 29.14% mIoU on novel AVSBench-OV classes against 8.53% for Sam4AVS. We lift those masks onto the same Split4D Gaussians by per-frame voting, then apply EchoSegnet's audio-intensity refinement at each time step. It is the strongest competitor because it combines the best reported 2D audio-guided segmentation with the only 3D audio-informed refinement in the literature, and it runs on the same 4D backbone. For this port on our benchmarks we have no reported result found on Lacuna. EchoSegnet's own paper, as summarized in our file, reports only that removing its refinement module costs 0.06 mIoU and 0.10 F-score single-instance, and 0.04 mIoU and 0.11 F-score multi-instance, on the static 3DAVS-S34-O7. Our comparison therefore rests on runs we make.

For fairness, all methods share the identical Split4D reconstruction, token pooling, training clips, and hyperparameter search budget. The 2D decoder is fine-tuned on our synthetic training split with the same number of GPU hours as our fusion module. Beating it means a clear 4D mIoU and F-score gain on the full test set, and a large gain on the proximity subset. That subset is where we predict intensity-based refinement fails. We also require that novel-view PSNR not drop relative to visual-only Split4D.

The visual-only decomposition baseline is Split4D itself, evaluated on decomposition mIoU in windows around contact events. This tests whether audio-driven repulsion helps separate touching objects.

Two ablations isolate the key ingredient. The first is the handcrafted correlation score above. The second is our architecture with keys replaced by pixel-patch features from rendered frames instead of trajectory tokens. This tests directly whether trajectory space, rather than cross-attention per se, is what matters.

Why not just…?

Why not just segment sounding objects in 2D and lift the masks onto the 4D Gaussians? This is the dynamic EchoSegnet port, and it is a real threat on easy clips. Two pieces of evidence suggest it falls short where it counts. Split4D shows that relying on per-frame 2D evidence propagated through time is what drove SA4D to 0.592 mIoU. EchoSegnet documents that audio intensity cannot separate adjacent objects. The proximity subset and the contact-event windows are designed to settle this. If the lifted-mask baseline matches us there, the trajectory-space claim is wrong.

Why not just use onset timing without any learned attention? It may suffice for impulsive sounds, and we treat it as a first-class result rather than a strawman. We report results separately for impulsive and continuous sources and for single versus overlapping sources. The learned model must justify its cost on the continuous and overlapping splits. If it does not, we adopt the handcrafted score.

Why not just use spatial audio to localize the source and pick the nearest moving object? Binaural or ambisonic direction is exactly the coarse spatial cue that EchoSegnet shows leaking across neighbors. We include direction-of-arrival cues as an input to all audio methods, so the comparison asks only whether trajectory matching adds anything beyond localization. The proximity subset measures that directly.

Why not just reconstruct visually first and attach audio afterwards? For segmentation this is roughly what we do, since the backbone is visual. The decomposition feedback is the part that requires audio during training. AV-GS found that frozen visual Gaussians miss structure that audio needs. The contact-window comparison against plain Split4D tests whether the analogous effect holds for object boundaries.

Proposed Experiments

On the synthetic dynamic benchmark we measure 4D sounding-object mIoU and F-score, following 3DAVS-S34-O7's protocol extended over time. Scores are reported overall and on the proximity, contact, silent-motion, and off-screen subsets. On annotated Replay-NVAS frames we measure rendered-mask mIoU across held-out views. On Split4D's Multi-Human data we check that decomposition quality away from contacts is not harmed relative to the reported 0.893. Contact-window decomposition gains are measured on our synthetic data, where ground truth exists.

Four checks decide the claim. First, shifting the audio by half a second at test time must collapse our gains to the motion-only baseline, which shows the model uses timing rather than scene priors. Second, on the proximity subset, trajectory-token keys must beat pixel-patch keys and the lifted-mask baseline by a clear margin. Third, on silent-motion clips the gate must keep false positives near the motion-only baseline's rate or below it. Fourth, the learned model must beat the handcrafted correlation on continuous and overlapping sources; if it does not, the handcrafted score becomes the contribution. A negative result on the second check would falsify the central thesis, and we will report it as such.