Graph-based video question-answering models turn a video into a graph of clips or tracked objects and let each node absorb information from its neighbors in time, as well as from the words of the question. Every one of these exchanges is attractive: a node can only become more like the things it listens to. This proposal tests a simple hypothesis. Attractive-only exchange blurs neighboring moments into one another, so the model loses track of what came before and what came after. Once that ordering information is gone, the model falls back on guessing from the question text alone. The proposed fix gives each edge a sign. Clips inside the same event pull together, clips on opposite sides of an event boundary push apart, and clips that are irrelevant to the question are pushed away from it rather than merely ignored. A bounded physical update rule keeps the pushing from blowing up.
The motivation is a well-documented failure. On NExT-QA, models score near 60% on descriptive questions but only 46–48% on causal "Why" questions, against 42% for a model that never sees the video (NExT-QA). On ActivityNet-QA, a soft-attention baseline reaches 41.2% on free-form questions and 2.5% on temporal-relationship questions (ActivityNet-QA). If part of this gap comes from temporal representations collapsing toward each other, the gains from signed messages should appear on temporal and causal questions and not on descriptive ones. That prediction is sharp enough to fail.
Core Claim
Our claim is that the useful form of over-smoothing mitigation in video graphs is not about depth. Video graph encoders are shallow, so the "deep GNNs collapse to a constant" story does not apply directly. What matters is temporal identity: whether clip representations stay distinguishable along the time axis after cross-modal fusion. We further claim that a negative message does something a zero message cannot. Suppressing an irrelevant clip (weight zero) leaves it where it is. Repelling it (weight negative) actively increases the separation that ordering questions depend on.
The new object is a signed graph whose edge signs cross the modality boundary and are conditioned on the question. Existing signed and repulsive methods all work on static, single-modality graphs; the research file finds no work that places signed edges between visual frames and text tokens.
Relation to Prior Work
The mechanism comes from the repulsive message-passing literature. ACMP treats node features as particles that attract when their similarity exceeds a bias β and repel otherwise. It adds an Allen-Cahn double-well term that bounds features in [−1, 1], which yields a positive lower bound on Dirichlet energy (a standard measure of how different neighboring nodes remain). ACMP reports 80.30% on Cora at 128 layers and 86.2% on the heterophilic Texas benchmark. SBP reads over-smoothing through structural balance: positive edges should sit inside clusters and negative edges between them. It also proves a constraint we must respect. There is a critical repulsion strength, below which collapse persists and above which feature norms explode. SBP further shows that normalization and residual connections act as implicit negative edges, which makes them a necessary baseline. wsGAT supplies negative attention coefficients inside attention layers. The negative-sampling line (D2GCN, layer-diverse sampling) subtracts messages from sampled non-neighbors. We borrow the double-well bound from ACMP and the negative-attention form from wsGAT. Where these methods fix signs from labels or static feature similarity on a unimodal graph, we condition signs on the question and on temporal structure.
The application regime comes from video graph models. VGT builds dynamic object graphs tracked through time and reasons with temporal transformers and spatial graph convolutions. HGA places shots and question words in one heterogeneous graph. Both use only nonnegative (softmax or co-attention) weights. HGA itself reports that graph propagation hurt static frame questions, which we take as evidence that indiscriminate attraction adds noise. We modify their operators rather than propose a new backbone.
The identity-preservation literature shapes the regularization side. BSG adds a hinge penalty when a node becomes too similar to its neighbors, and UniEA separates an intra-graph repulsive loss from an inter-space alignment loss. These are loss-level fixes, and they motivate a loss-only baseline. BSG also assumes homophily, whereas video graphs are partly heterophilic across event boundaries. This is one reason we expect sign-aware propagation to beat a uniform margin.
On the cross-modal side, ReViSE argues that pairwise alignment leaves modalities separated by an empty region of the shared space, while distribution matching can erase fine distinctions within a modality. We see the attractive question-to-clip edges in VideoQA as a small version of this tension. Pulling all clips toward the question embedding closes the gap between text and video, but does so by making the clips resemble each other.
Two papers constrain how we interpret results. The Oversmoothing Fallacy and Are we measuring oversmoothing correctly? question whether collapse metrics predict accuracy. We therefore treat our temporal-collapse diagnostic as a hypothesis to test, not a given.
Research Program
We build a drop-in signed temporal layer for VGT's temporal graph transformer and for HGA's shot-word graph. For clip or object node $i$ with feature $x_i$, the update is
$$ x_i \leftarrow x_i + \tau \Big[ \sum_{j \in \mathcal{N}(i)} s_{ij}\,(x_j - x_i) + \delta\, x_i \odot (1 - x_i^{2}) \Big]. $$
Here $\mathcal{N}(i)$ contains temporal neighbors and question tokens. $\tau$ is a step size, $\delta$ is the strength of ACMP's double-well term, and $\odot$ is elementwise multiplication. The signed weight is $s_{ij} = \tanh(g_q(x_i, x_j) - \beta)$, where $g_q$ is a small scoring network conditioned on the question $q$ and $\beta$ is a learned bias. A negative $s_{ij}$ moves $x_i$ away from $x_j$, and the double-well term keeps features bounded no matter how strong the repulsion becomes. This is the reason to use ACMP's form rather than plain subtraction, given SBP's blow-up result. We use three to six explicit steps, comparable to VGT's depth, and add no neural ODE solver.
The sign has two sources, and we ablate each separately. Temporal edges carry a sign from a weak event-boundary signal: the change in frozen clip features between adjacent clips, in the spirit of SBP's feature-based variant, refined by $g_q$. Cross-modal edges carry a sign from question relevance, so distractor clips are repelled from the question tokens rather than zero-weighted.
The first analysis is diagnostic and comes before any new model is trained. We take trained VGT and HGA models and measure temporal collapse, defined as the mean cosine similarity between clip representations as a function of their temporal distance, before and after fusion. We also measure order sensitivity, the change in answer probability when clip order is reversed or shuffled. We then ask, per question type, whether these quantities predict correctness. If collapse does not predict failures on temporal and causal questions, the core motivation is wrong, and we report that negative result.
Baselines
The naive reference point is the text-only BlindQA model, which scores 42% on NExT-QA "Why" questions (NExT-QA). Every method's gain will be reported as distance above it per question type.
The strongest existing method in the file is VGT, which reports 53.68% overall accuracy on NExT-QA multi-choice without pretraining and 55.7% with 0.18M WebVid clips (VGT). It is the right competitor because it already builds explicit temporal graphs and is the architecture we modify. The file does not give VGT's per-type causal and temporal breakdown, so we will re-run it ourselves. HGA's reported 49.7% on NExT-QA serves as a second graph baseline. For ActivityNet-QA, the file reports E-SA at 31.8% overall and 2.5% on temporal relationships (ActivityNet-QA). For TGIF-QA state transition, VGT reports 97.6%, close to ceiling, so we use it only as a check against regressions.
Every comparison uses VGT's released features and training recipe. All methods share the same number of parameters to within 2%, the same epochs, the same hyperparameter search budget (an equal number of trials over learning rate and each method's own strength coefficients), and five seeds. We report means with standard deviations.
Beating VGT means the following on the non-pretrained setting: an overall gain larger than two seed standard deviations, with at least two-thirds of that gain coming from the causal and temporal splits. A gain that is uniform across types, including descriptive questions, would suggest generic extra capacity rather than the mechanism.
Four ablations isolate the key ingredient. The zero-gate ablation keeps the identical scoring network but clips $s_{ij}$ to $[0,1]$, so irrelevant clips are suppressed rather than repelled. This is the decisive test of whether sign matters. The no-well ablation removes the double-well term and sweeps repulsion strength, to test SBP's critical-value prediction in this setting. The normalization ablation adds LayerNorm plus residuals, or pair-wise normalization, to VGT with no signed edges, which SBP argues is implicit repulsion. The single-source ablations sign only temporal edges or only cross-modal edges.
Why not just…?
Why not just use a larger transformer or multimodal LLM on flattened frames? At scale this is a real competitor, and we do not claim to beat large pretrained models. The file's evidence is only suggestive here. Attention-only baselines collapse on temporal-relation questions (2.5% on ActivityNet-QA), and VGT's structured graph with modest pretraining reached 55.7%. Our question is narrower: does sign-aware propagation improve temporal identity within a fixed backbone? The same layer can be inserted on top of frozen features from a larger encoder in a follow-up. The claim would weaken if collapse turns out to be absent in large transformers.
Why not just gate irrelevant clips to zero? This is the most serious threat, because query-conditioned sparse attention is simpler and already standard. The zero-gate ablation answers it directly. If zero-gating matches signed messages on the causal and temporal splits, we conclude that repulsion adds nothing beyond selection and drop the claim.
Why not just add a temporal-order contrastive loss, using shuffled videos as negatives, or a BSG-style divergence hinge? A loss can in principle enforce the same separation without changing propagation. We will run both at the matched tuning budget, including a sweep of BSG's margin, which BSG itself flags as sensitive. Our expectation is that a uniform margin conflicts with within-event smoothing, while signs can separate across boundaries and smooth within events. If the loss-only baseline matches us, the honest conclusion is that the gain comes from the objective, not from message passing.
Why not just add normalization and residuals? SBP's analysis says these already inject negative edges, but uniformly rather than selectively. The normalization ablation tests whether uniform implicit repulsion suffices.
Decisive Checks
We will consider the idea supported only if four conditions hold together. Temporal collapse in baseline VGT and HGA must predict errors on temporal and causal questions more strongly than on descriptive ones. The signed layer must reduce that collapse and raise order sensitivity. Its accuracy gain must be concentrated on the NExT-QA causal and temporal splits and on ActivityNet-QA temporal-relationship questions. It must beat both the zero-gate and the order-contrastive baselines on those splits.
We will also measure the cross-modal gap, the distance between the text and clip embedding distributions, to check that repulsion does not reopen it. A widening gap accompanied by accuracy loss on descriptive questions would indicate over-repulsion. The no-well sweep should reproduce an SBP-style threshold between collapse and divergence. A threshold appearing in video graphs would itself be a useful transfer result, whatever the accuracy outcome.
References
- NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions
- ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
- ACMP: Allen-Cahn Message Passing with Attractive and Repulsive Forces for Graph Neural Networks
- A Signed Graph Approach to Understanding and Mitigating Oversmoothing
- wsGAT: Weighted and Signed Graph Attention Networks for Link Prediction
- Negative Samples-enhanced Graph Convolutional Neural Networks
- Layer-diverse Negative Sampling for Graph Neural Networks
- Video Graph Transformer for Video Question Answering
- Reasoning with Heterogeneous Graph Alignment for Video Question Answering
- Balancing Graph Embedding Smoothness in Self-Supervised Learning via Information-Theoretic Decomposition
- Unifying Dual-Space Embedding for Entity Alignment via Contrastive Learning
- Learning Robust Visual-Semantic Embeddings
- The Oversmoothing Fallacy: A Misguided Narrative in GNN Research
- Are we measuring oversmoothing in graph neural networks correctly?