Sound and sight rarely line up in time. A pan starts sizzling a moment after it touches the flame, a cymbal rings long after the stick has left it, and a narrator names an object seconds before it appears. Most audio-visual models fuse the two streams by letting every audio moment look at every video moment. That blurs event boundaries, and it lets the model assume a sound is present whenever its source is visible. The DAVE benchmark measures the damage: Gemini 1.5 Pro scores about 55% on audio-visual synchronization but about 1% on detecting that a sound is absent. We propose a different route. Audio and video never talk to each other directly. Each modality separately reports evidence about a shared, discrete vocabulary of concepts ("cello", "frying", "speech"). Those concepts become explicit graph vertices, and each concept learns its own asymmetric temporal kernel that says how long its sound can lead or trail its visual evidence.
This borrows the central trick of attribute-augmented graphs: turn attributes into extra nodes, so that fusing features becomes ordinary message passing on a graph. Here the trick is applied to a new problem, which is synchronizing two streams in time. The shared concept node is also a response to the cross-modal semantic gap, meaning that audio and visual features live in incompatible geometric spaces. The node sidesteps that gap because neither modality has to be projected into the other's space. Each only has to say whether a discrete concept is active. The concept node's temporal kernel absorbs the lag between modalities. Disagreement between a concept's audio-side and visual-side activity carries information of its own: "visible but silent" is exactly the case today's models get wrong.
Core Claim
Route all cross-modal information through time-stamped concept vertices, each with a learned, asymmetric lead/lag kernel. This should beat both unconstrained fusion and pipeline-based boundary detection on audio-visual segmentation when sound sources change over time or are silent. It should also degrade much more gracefully when audio is artificially shifted against video. The claim has three separable parts, each with an ablation: the concept bottleneck (no direct audio-to-region edges), the per-concept asymmetric kernel (versus zero-lag or symmetric windows), and message passing over the concept graph (versus concept-level late fusion with the same kernels). We treat the third as the part most at risk.
Relation to Prior Work
The method comes from the attribute-as-vertex line of work. SA-Cluster and PANE add attribute vertices so that random walks mix structural and semantic proximity. GCL-JAM builds the same "attribute-interpolated" graph for contrastive learning. All three assume attributes and edges are static. We make attribute edges time-stamped and give each attribute a temporal extent.
Virtual Nodes Go Temporal adds virtual nodes to dynamic graphs and proves they shorten long-range information paths. It reports gains of up to 40% in MRR over TGN, but its virtual nodes are topological cluster hubs on single-modality interaction graphs. Ours are semantic, and they bridge two modalities. Understanding Virtual Nodes acts as a constraint on the design: virtual nodes with unbounded degree cause over-squashing and heterogeneity. We therefore instantiate concept nodes per clip and per time window rather than as global hubs, which bounds their degree.
Two papers motivate the problem and supply the main competitor. Co-Prop shows that anchoring audio boundaries before propagating masks helps segmentation. It obtains its boundaries from an offline, retrieval-prompted LLM whose errors carry straight into the masks. We learn boundaries end to end, as a byproduct of concept-node activity. MM-Align aligns sequences with a locally constrained optimal transport plan, a matching that assigns each timestep in one stream a weighted mix of nearby timesteps in the other. The plan is pairwise and symmetric in time, and it is not concept-specific, whereas physical lag clearly depends on the sound source.
On the semantic gap, EGO/LEGO avoids shared vector spaces by comparing modalities through relational graphs. It reaches 97.26% AUC on ShanghaiTech with 0.091M parameters, which suggests that relational or discrete intermediaries can beat direct projection. EGO assumes the modalities are already aligned in time; we keep its spirit and drop that assumption. Learning Robust Visual-Semantic Embeddings argues that pointwise contrastive matching leaves the two modalities' overall distributions misaligned, which is part of why we choose a discrete concept bridge over a CLIP-style adapter.
Research Program
For each video clip we build a graph with three node types. There are visual region nodes (per-frame object proposals from a frozen segmentation backbone), audio segment nodes (short windows from a frozen audio encoder), and concept nodes indexed by the sound-source class vocabulary. We start from the AVSS semantic labels, so no new annotation is needed. Region and audio nodes connect only to concept nodes, never to each other. For concept $c$, frame time $\tau$ and audio window time $t$, the audio-side activity is
$$a_c(\tau) = \sum_t k_c(\tau - t)\, p^A_c(t),$$
Here $p^A_c(t)$ is the audio encoder's probability that concept $c$ is sounding at time $t$. The kernel $k_c$ is a learned, asymmetric, two-sided exponential. It has three scalars per concept: an onset lead, a decay tail, and a peak offset. This lets a resonating object keep its concept node active after the visual contact ends, while speech gets a short symmetric window. A visual-side activity $v_c(\tau)$ is computed from region-to-concept edge weights.
A few rounds of message passing then run in both directions. They pass along concept-to-concept edges, which are initialized from label co-occurrence, and along concept-to-region edges. The final mask logit for region $r$ is gated by $a_c(\tau)$ for the concepts $r$ attaches to. A region can be segmented as sounding only if its concept is currently active in the audio, however visually salient it is. The gap $v_c(\tau) - a_c(\tau)$ serves directly as a sound-absence score.
Training uses the standard segmentation loss plus a weak consistency term that encourages $a_c$ and $v_c$ to agree only where ground-truth masks say the object is sounding. Kernel parameters are shared across clips, so they are learned from the whole corpus and can be inspected. We will check whether learned tails are longer for struck instruments than for voices. If they are, this is evidence the kernels capture physics rather than dataset artifacts.
Baselines
The naive reference point is AVSegFormer as reported inside Co-Prop, which fuses without any mechanism for temporal misalignment. The file gives 58.36% mean Jaccard and 69.3% mean F-score on M3 with a PVT-v2 backbone. This number tells us whether any alignment machinery helps at all.
The strongest existing method is Co-Prop (Collaborative Hybrid Propagator), which reports 63.58% mean Jaccard and 73.96% mean F-score on M3 with PVT-v2. It is the right competitor because it explicitly targets boundary shifts in exactly this task. We will run it with the same PVT-v2 backbone and the same training split. The file links the codebase Cyyzpoi/SelM in connection with this line of work; we will use it where it applies and reimplement the remaining pieces otherwise. Beating Co-Prop means matching it on overall M3, because we do not expect large gains on clips with a single steady source. It also means clearly exceeding it on the 17-clip MOC subset, where sound sources switch over time. The MOC subset is small, so we will report per-clip results and bootstrap intervals rather than a single mean.
The second strong competitor is a locally banded cross-attention layer between audio and region tokens, with a learned global offset. It uses the same frozen encoders and a parameter count matched to within 10%, and it is tuned with an equal number of hyperparameter trials. This is the simplest learned-lag alternative.
Three ablations isolate the key ingredients. First, concept late fusion keeps the same concept vocabulary and kernels but drops all message passing: region score times lagged audio concept probability. This tests whether the graph adds anything. Second, zero-lag and symmetric kernels test the asymmetric-kernel claim. Third, direct region-to-audio edges added alongside the concept path test whether the bottleneck itself is what suppresses visual bias.
Why not just…?
Why not just use windowed or sparse cross-attention with a learned offset? Structured sparsity already synchronizes audio and video in generation (Synchrony-Aware Cross-Modal Sparse Attention), so this is a real threat. Its weakness is that a single window or offset is shared across all sources. Nothing forces silence to suppress a visible object, so the positive-association bias measured in DAVE can survive inside the window. The banded-attention baseline, together with the synthetic lag sweep below, settles this. If banded attention matches us on MOC and under shifted audio, the concept bottleneck is unnecessary and we will say so.
Why not just align the streams with optimal transport, as MM-Align does? MM-Align reports gains of 6.6% to 9.3% over unimodal baselines on CMU-MOSI and a 3x to 4x speedup over generative imputation. Its plan, however, is a symmetric, local, concept-agnostic matching between whole sequences, and it has no notion of "this source is silent now." We will run a transport-aligned fusion variant on the same features and compare. The prediction is that it helps under uniform shifts but not under source-dependent lag or on silent-but-visible frames.
Why not just tag concepts per modality and combine the outputs with a temporal dilation? This is the concept late-fusion ablation. It is the most dangerous alternative, because it keeps most of our inductive bias and drops the graph. We expect message passing to matter mainly when several sources co-occur or switch, where co-occurrence edges let one concept's evidence sharpen another's boundary. If late fusion matches the full model, the paper's contribution narrows honestly to "discrete concept bottleneck plus per-concept lag kernels," and the graph is removed.
Why not just fine-tune an omni-modal LLM? The Daily-Omni results cut the other way. A modular agent with explicit alignment reaches 61.82%, beating several end-to-end proprietary systems, although Gemini 2.5 Flash reaches 73.06% (Daily-Omni). On DAVE, Video-LLaMA-2 reaches only 31.3%. We do not claim to beat frontier models on question answering. The question here is whether an explicit concept-lag structure fixes a specific failure that scale has not fixed.
Proposed Experiments
The primary evaluation is audio-visual segmentation on S4, M3 and AVSS, reporting mean Jaccard and mean F-score, with the MOC subset as the diagnostic for source switching.
The decisive check is a controlled lag sweep. On the M3 and AVSS test sets we shift audio relative to video by known offsets from −1 s to +1 s, applied either uniformly or per source class, and plot accuracy against offset for every system. The prediction is a flat curve for our model inside the range covered by its learned kernels, and steeper degradation for AVSegFormer, Co-Prop and banded attention. Per-class shifts should separate us from the single-offset baselines.
A second check targets absence. We build silent-but-visible test clips by replacing M3 audio with audio from a different clip, which provides ground truth that nothing should be segmented. We then report the false-positive mask area. For external grounding we train a light absence head on the $v_c - a_c$ gap and evaluate it on DAVE's sound-absence task. The reported MLLM numbers (about 1% for Gemini 1.5 Pro, 84.7% human average across tasks) serve only as reference points, since the systems and training data differ.
Finally, we inspect the learned kernel parameters per class. We also report concept-node degree statistics, to confirm that bounding degree per window avoids the over-squashing regime described in Understanding Virtual Nodes.
References
- DAVE: Diagnostic benchmark for Audio Visual Evaluation
- Graph Clustering Based on Structural/Attribute Similarities
- PANE: scalable and effective attributed network embedding
- Graph Contrastive Learning with Joint Spectral Augmentation of Attribute and Topology
- Virtual Nodes Go Temporal
- Understanding Virtual Nodes: Oversquashing and Node Heterogeneity
- Collaborative Hybrid Propagator for Temporal Misalignment in Audio-Visual Segmentation
- MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences
- Learnable Expansion of Graph Operators for Multi-Modal Feature Fusion
- Learning Robust Visual-Semantic Embeddings
- Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention
- Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities