Shared Attention as Controlled Over-Smoothing

When several people in a video look at the same thing, a model that reasons over them should end up with partly identical internal descriptions of those people: the part that says "what I am looking at" ought to merge, and the part that says "who I am" ought not to. Graph neural networks offer a known mechanism for this. In a graph neural network, each node repeatedly averages information from its neighbors, a process called message passing. The usual failure is over-smoothing: every node's representation drifts toward the same vector until nodes become indistinguishable. Current models of multi-person gaze, such as the spatio-temporal graph of Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning and the person-token transformer MTGS, mix all people with soft attention. These models therefore smooth the wrong things: they blur identities across the whole scene, and they cannot make two people's target descriptions exactly equal.

We propose to treat shared attention as over-smoothing that the model controls on purpose. Each person's representation is split into two parts. A "focus" channel is allowed to smooth, but only along hard, sparse edges defined by gaze. An "identity" channel never mixes across people. Over-smoothing theory then gives a concrete, testable prediction: after enough propagation, the focus channel collapses to exactly as many distinct vectors as there are groups of people sharing a target. Shared attention, mutual gaze, and the number of attention groups can then be read directly from that collapsed structure instead of being guessed by a separate classifier head.

Core Claim

Demystifying Oversmoothing in Attention-Based Graph Neural Networks proves two things that matter here. First, softmax attention gives every edge a strictly positive weight, so it cannot cut a connected graph (their Lemma 2). Second, as a result, attention-based message passing still collapses exponentially toward a single shared vector (their Theorem 1). We use this result in two roles. As a constraint, it tells us that MTGS-style all-to-all person attention cannot keep two distinct attention groups apart as depth grows. As a method, the same ergodicity argument applied to a graph with $K$ disconnected components gives convergence to $K$ distinct vectors rather than one, because each component's product of averaging matrices collapses separately.

Our focus-channel update is

$$h_i^{(l+1)} = h_i^{(l)} + \sum_j g_{ij}\, W\big(h_j^{(l)} - h_i^{(l)}\big),$$

where the symbols are as follows. $h_i^{(l)}$ is person or object $i$'s focus vector at layer $l$. $W$ is a learned matrix. $g_{ij}$ is a gate computed from person $i$'s predicted gaze cone, passed through a sparse normalizer such as sparsemax or top-1 selection, which can output exact zeros.

The sparse gate is motivated by Predicting Primary Gaze Behavior Using Social Saliency Fields. That work found that human gaze selects the strongest "social charge", taking a maximum rather than averaging over candidates. A person therefore sends one dominant edge to one target, and people who follow another person's gaze inherit that person's edge. Because the gates can be exactly zero, the gated graph breaks into components. The claim is that this sparse two-channel design does three things. It identifies shared-attention groups more accurately than soft all-to-all mixing in crowded scenes. It keeps individual identity intact at depth. And it lets depth carry multi-hop chains (A follows B, who follows C, who looks at an object) that one or two layers cannot reach.

Research Program

The model starts from the MTGS pipeline: person tokens built from head crops and gaze embeddings, plus scene features. We keep its person-to-scene cross-attention for the identity channel, so each person still reads the image, but we remove person-to-person mixing from that channel. The focus channel runs the gated update above over a graph of people and candidate target regions. The gates are recomputed at every layer from the current gaze estimate, which allows a gaze-following chain to resolve over several hops.

Outputs are read from the focus channel. Shared attention between two people is scored by how close their focus vectors are. The attended point is decoded from the target node at the root of each person's component. Mutual gaze ("looking at each other") is a two-node component that contains no object. Temporal context comes from the existing MTGS temporal encoding.

Running alongside the model is a diagnostic study, which responds to Are we measuring oversmoothing in graph neural networks correctly? and its warning that global similarity scores confuse useful agreement with harmful collapse. We measure similarity separately within ground-truth attention groups and between groups, layer by layer. Useful smoothing should raise within-group similarity. Harmful smoothing should raise between-group similarity. We apply this measurement to MTGS, to the graph model of Understanding Human Gaze Communication, and to our model. This answers the prior question directly: do existing social-gaze models actually collapse at the depths they use, or does over-smoothing only appear when they are made deeper?

Where this differs from the existing bridges is specific. A Gravity-Informed Spatiotemporal Transformer adds a physical prior to attention to reduce smoothing across urban regions, but it does not use smoothing as an output signal. The role-constrained graph network of Towards Consumer Loan Fraud Detection keeps role clusters separate in transaction graphs with a structured penalty, which is close in spirit to our identity channel but has no gaze-defined topology. The two-channel filters of Complete the Missing Half pair a smoothing channel with a sharpening channel over the same fixed edges. We instead change which edges exist. Finally, the advection-diffusion operator of Advection-Diffusion on Graphs learns a node potential that steers flow; this is a natural extension we deliberately defer, because our hard gating is the simpler test of the core claim.

The main risk is that the gates depend on predicted gaze, so gaze errors produce wrong components. This is exactly the cascading-pipeline fragility that Attention Flow criticizes. We address it in two ways. Gates are relaxed during training with a temperature-annealed sparse normalizer, so the model is trained end-to-end. And we report results with oracle gaze to separate gating quality from gaze quality.

Baselines

The naive reference point is MTGS with person-to-person attention removed entirely: each person sees only the scene. This measures how much any cross-person reasoning contributes. A second naive point is pairwise geometric intersection of predicted gaze targets, which declares shared attention when two predicted points fall within a threshold of each other. Both use the same backbone, gaze head, and training data as our model.

The strongest existing method is MTGS. It is the closest competitor because it already predicts gaze targets together with the three social labels (looking at humans, looking at each other, shared attention) using dense person-token attention. That is precisely the design our claim says is flawed. MTGS reports an Average Precision of 0.974 on UCO-LAEO. We found no reported result on Lacuna for MTGS on its own VSGaze benchmark, so we will retrain it on VSGaze under our splits with the same backbone, number of epochs, and hyperparameter-search budget as our model. We will run it at depths 2, 4, 8, and 16, so that the comparison is depth-matched rather than pitting a deep model against a shallow one.

On VideoCoAtt, the reported reference points are as follows. Attention Flow reports 78.1% accuracy and a mean $L_2$ error of 62.84. Detecting attended visual targets in video reports 83.3% shared-attention accuracy without task-specific training. On VACATION, the graph model of Understanding Human Gaze Communication reports 37.1% event-level accuracy and 55.02% top-1 atomic accuracy; we reimplement that model as the shallow-graph baseline.

Beating the strongest method means the following. Our model must match MTGS's looking-at-each-other AP within noise on UCO-LAEO. It must exceed MTGS's shared-attention AP on VSGaze at matched depth, with the gap widening in scenes with four or more people. And it must improve on 83.3% on VideoCoAtt when trained comparably.

Three ablations isolate the key ingredients. The first keeps the two channels but uses softmax gates instead of sparse gates; this tests whether exact zeros matter, which is the specific mechanism that Lemma 2 rules out for softmax. The second keeps sparse gates but merges the two channels into one; this tests whether protecting identity matters. The third replaces gaze-derived gates with sparse gates learned freely from features; this tests whether the gaze-cone prior matters or whether sparsity alone does the work.

Why not just…?

Why not just keep the social graph shallow, at one or two layers? This is the most serious threat, because social scenes often contain only two to five people, and two layers may already cover every useful hop. Our diagnostic study tests it directly. If MTGS shows no rise in between-group similarity at its native depth and gains nothing from more depth, then the over-smoothing motivation is wrong for ordinary scenes. In that case we would restrict the claim to crowded scenes and to multi-hop following chains. The event-level gap on VACATION, 37.1% from the model versus 55.9% when that model is given ground-truth atomic labels (Understanding Human Gaze Communication), suggests that compositional reasoning is lacking. However, it does not prove that depth is the missing piece, and we do not assume it is.

Why not just add residual connections and normalization to a standard transformer? MTGS already has residual connections. The proof in Demystifying Oversmoothing rests on connectivity that softmax cannot cut, and residual connections slow collapse without making two groups' target vectors separable by construction. Our softmax-gate ablation is the direct test: if two channels with softmax gates match two channels with sparse gates, then sparsity is unnecessary and the simpler fix wins.

Why not just classify every pair of people independently? A pairwise classifier avoids message passing altogether and cannot over-smooth. It also cannot represent following chains, in which two people share attention only through a third person, and its pairwise decisions can contradict each other (A matches B, B matches C, A does not match C). Component structure enforces consistency for free. We include the geometric-intersection reference, and we will also train a learned pairwise classifier on the same features. If either matches our model on scenes with four or more people, the relational machinery is not justified.

Proposed Experiments

Training and main evaluation use VSGaze, which merges GazeFollow, VideoAttentionTarget, ChildPlay, VideoCoAtt, and UCO-LAEO. We additionally test on VideoCoAtt (accuracy and $L_2$ error), UCO-LAEO (AP), and VACATION (atomic and event accuracy). All results are stratified by the number of people in the scene and by whether the ground truth contains a following chain of two or more hops.

The first decisive check is the rank prediction. On held-out clips, the number of distinct clusters in the focus channel at the final layer should equal the ground-truth number of attention groups. Within-group similarity should rise and between-group similarity should stay flat as depth grows. MTGS's person tokens should show between-group similarity rising with depth. If our model's groups also merge at depth, the core mechanism has failed.

The second check is a controlled synthetic test. We compose scenes in which gaze chains of known length $k$ connect people to a target. We expect accuracy to rise with depth up to roughly $k$ for our model, while degrading with depth for softmax-mixing baselines, which is the exponential collapse that Theorem 1 predicts.

The third check reruns everything with oracle gaze directions. This shows how much of any remaining gap comes from gating errors rather than from the propagation design, and therefore tells us whether the end-to-end relaxation removes the pipeline fragility that motivated end-to-end models in the first place.