Exclusion as a Change in Covariance: Within-Person Contrastive Subspaces for Detecting Social Stress

When people are socially excluded, their bodies change: they look away, touch their faces, and stop nodding and smiling. Automated measurement of this has stalled because each person moves and breathes in their own way. An excluded fidget and an included fidget can look the same once you average across a room of people. The standard response is to reduce rich video and physiology to a few hand-counted behaviors or group means, and those means often wash out. We propose a different measurement. Record each person's multimodal behavior (body pose, facial muscle activity, heart signals) during a stretch where they are included. Treat that stretch as a personal reference. Then use a PCA-style eigen-decomposition to find the directions of behavior whose variance grows when the same person is excluded. PCA here means principal component analysis, the method that finds the axes along which data varies most. The exclusion signature is then defined by which patterns of movement become more coordinated and more variable relative to that person's own baseline, not by shifts in average posture.

The opportunity matters for two reasons. First, the clearest human result in the file is that physiological markers of exclusion failed because of large differences between individuals. Nonverbal responses to social inclusion and exclusion collected heart rate and cortisol and found no condition effect for exactly this reason. A within-person, covariance-level measure attacks that failure directly. Second, the same decomposition asks a scientific question the field has not posed in a testable form. Recent animal work suggests exclusion changes how signals are organized rather than simply turning a signal up. If that holds in human behavior, the directions found for one person should be shared across people, and they should collapse back when the person is re-included.

Core Claim

The claim is that the behavioral effect of social exclusion is better captured by a change in each person's behavioral covariance than by a change in their behavioral means. It is also claimed that these per-person "exclusion directions" are consistent enough across people to form a shared low-dimensional subspace.

Concretely, let $\Sigma_{\text{inc}}$ be the covariance matrix of a person's feature vectors during the inclusion phase and $\Sigma_{\text{exc}}$ the same during exclusion. We solve the generalized eigenproblem

$$\Sigma_{\text{exc}}\, v = \lambda\, \Sigma_{\text{inc}}\, v,$$

where each eigenvector $v$ is a direction in feature space and its eigenvalue $\lambda$ is the ratio of exclusion variance to inclusion variance along it. Directions with $\lambda \gg 1$ are behaviors that exclusion newly amplifies for that person. Dividing by the person's own inclusion covariance factors out their idiosyncratic style. At test time, a sliding window $w$ of new data is scored by $s(w) = \sum_{j=1}^{k} \log\big(v_j^\top \hat\Sigma_w v_j \,/\, v_j^\top \Sigma_{\text{inc}} v_j\big)$. Here $\hat\Sigma_w$ is the window covariance and $v_1,\dots,v_k$ are the top $k$ exclusion directions, either the person's own or the shared cross-person ones.

This generalized-eigenvector contrast is a known piece of linear algebra, and we do not claim it as a new algorithm. What is new is using a person's own inclusion phase as the reference distribution for social-stress measurement. We also test three things for the first time: whether the resulting directions are shared across people, whether they are separable from time-on-task effects, and whether they reverse under re-inclusion. Section 2 of the research file reports no paper that applies any PCA-family decomposition to exclusion or vicarious-stress data. The closest work either hand-codes behavior or uses supervised classifiers.

Where This Sits in Prior Work

The social-stress papers play three roles. They motivate the proposal, they constrain the design, and they supply the testable predictions.

Motivation comes from two papers. Nonverbal responses to social inclusion and exclusion showed that excluded people display more flight and displacement behavior and less affiliation and relaxation. It did so by manually grouping ethological codes into four categories, and it abandoned physiology because baselines varied too much. We keep its behavioral constructs as an interpretive target but replace manual grouping with a decomposition that is referenced to each individual. Social Exclusion Amplifies Behavioral Responses to Physical Pain via Insular Neuromodulation found in mice that exclusion blurs population-level valence encoding in the insula instead of activating a dedicated set of neurons. It labeled behavior with a supervised pose classifier and syllable segmentation. That "reorganization rather than a dedicated signal" result is the reason we look for a covariance-level signature rather than a mean shift, though we measure behavior in humans rather than neurons.

The predictions come from two further papers. Social inclusion regulates the effect of social exclusion on adaptation to emotional conflict showed that re-inclusion restores cognitive-control effects that exclusion degrades. This gives us a directional prediction: variance along exclusion directions should fall back toward $\lambda \approx 1$ in a re-acceptance phase. Altered Interpersonal Neural Synchronization during Social Interaction After Shared Excluded Experiences in Depressed Adolescents measured coordination between two people's brains with channel-to-channel wavelet coherence during Cyberball. Cyberball is a computer ball-tossing game in which virtual players first include and then ignore the participant. We borrow its three-phase protocol but study coordination within a single person's multimodal behavior.

Fairness constrains how results must be reported. Do we like honest signal givers? Macro and micro expressions of disgust found that women who suppress negative affect are judged less trustworthy than men who do so (4.83 vs 5.41). Krahmer et al. found exclusion easier to detect in men. Any detector must therefore report detection quality separately by gender.

The PCA papers supply method components and one caution. Loki: Low-rank Keys for Efficient Sparse Attention fits a PCA projection on a small calibration set and applies it at inference. Our inclusion phase plays the role of a per-person calibration set. Effective Dimensionality Reduction for Word Embeddings showed that the top few principal directions of an embedding space often encode nuisance structure (word frequency) and that removing them helps. This is the strongest unsupervised alternative to our within-person reference, and we test it as a baseline. Efficient fair PCA for fair representation learning gives an analytic way to constrain a projection so it carries no linear covariance with a protected attribute. We use it as a constraint on the shared subspace.

The caution comes from Simple and Effective Unsupervised Redundancy Elimination to Compress Dense Vectors for Passage Retrieval. It reported that unsupervised PCA beat supervised linear projections at 256 and 128 dimensions on retrieval embeddings. This is suggestive for small-sample regimes, but it comes from a different domain, so we treat supervised projection as a live competitor rather than a settled loser. The same paper also reported PCA degrading out of domain. That warns us that a shared subspace fitted in one lab protocol may not transfer, so we test transfer explicitly.

Research Program

There is no public exclusion dataset in the file that pairs video with physiology across phases, so the program starts with data collection. We will run a three-phase Cyberball study (inclusion, exclusion, re-acceptance, roughly three minutes each) with about 120 adults, balanced by gender. Each session records frontal and side video and a chest-strap ECG. A control arm of equal size stays included for all three phases. This control arm is essential. Exclusion always follows inclusion in time, so any variance growth could be boredom or fatigue. The control arm lets us estimate and project out pure time-on-task directions, found by the same generalized eigenproblem applied to control participants' phase 2 versus phase 1.

A smaller replication, about 40 participants, uses the confederate group-discussion protocol of Krahmer et al. It checks whether directions found in a computer game carry over to a face-to-face exclusion.

Per frame, we extract body keypoints, facial action-unit intensities, and head pose from off-the-shelf trackers. We also compute heart-rate and heart-rate-variability features in rolling windows. The concatenated vector has a few hundred dimensions. Covariance estimation from about 5,000 autocorrelated frames per phase is noisy at that size. We therefore first project everyone onto a global PCA basis of about 30 dimensions fit on inclusion data only, then estimate per-phase covariances with shrinkage toward the diagonal.

The analysis has four parts. The first is per-person exclusion directions and their eigenvalue spectra. The second is a shared subspace, found by aggregating per-person top directions and taking their dominant common span; we measure cross-person consistency by average principal angles against a permutation null. The third is interpretation, correlating the shared directions' loadings with hand-coded flight, displacement, and affiliation behaviors on a coded subset. The fourth is a Fair PCA variant of the shared subspace that removes linear covariance with gender, then re-evaluates detection by gender.

Baselines

The naive reference point is a per-person mean-shift detector. Each test participant's features are z-scored using their own inclusion phase, and each window is scored by its distance from the inclusion mean. This gets exactly the same calibration data as our method, which makes it the direct test of "covariance versus means." If it matches us, the core claim is wrong.

The strongest competitor is supervised: a logistic regression and a gradient-boosted tree classifier trained across participants on the same z-scored features to classify exclusion windows. This is the approach the field would reach for, in the spirit of the supervised pose classifier in Social Exclusion Amplifies Behavioral Responses to Physical Pain via Insular Neuromodulation. It is run leave-one-subject-out with the same inclusion-phase normalization and the same hyperparameter search budget as our choice of $k$ and shrinkage.

The file gives no automated benchmark for this task. We found no reported result on Lacuna for automated exclusion-detection accuracy on any exclusion corpus. The only human reference is from Nonverbal responses to social inclusion and exclusion: judges' mean certainty ratings on a −5 to +5 scale were 1.60 for included and −1.45 for excluded participants from 8-second clips. That is not an accuracy number. We will therefore also collect human-judge ratings of our own 8-second clips to report a human area under the ROC curve (AUC) on identical data.

Two unsupervised baselines isolate our key ingredient, the per-person reference. The first is global PCA with the top components removed as in Effective Dimensionality Reduction for Word Embeddings, with a pooled contrast of exclusion versus inclusion covariance. It asks whether removing dominant shared nuisance directions is enough without per-person referencing. The second is our method with a pooled rather than per-person $\Sigma_{\text{inc}}$. A final ablation drops the control-arm time-direction removal, to show how much apparent signal was fatigue.

Beating the baselines means a leave-one-subject-out window-level AUC for exclusion-versus-inclusion that exceeds the per-person mean-shift detector by a margin whose bootstrap 95% interval excludes zero. The gain must survive the time-direction removal, and the method must be at least as good as the supervised classifiers.

Why not just…?

Why not just z-score each person against their own baseline and compare means? This is the most serious threat, because it uses the same calibration idea with one fewer moving part. It is built into the design as the naive baseline with identical data. Krahmer et al.'s finding that condition means of cortisol and heart rate did not separate groups is suggestive but does not settle it, since they did not normalize within person. If z-scored means perform as well, we will report that exclusion is a within-person mean shift and drop the covariance claim.

Why not just train a supervised classifier on all the raw features? With about 120 participants, a flexible classifier can learn person- and session-specific cues, and it gives no shared, inspectable axis to test for re-inclusion reversal. Simple and Effective Unsupervised Redundancy Elimination to Compress Dense Vectors for Passage Retrieval found supervised projections overfitting relative to PCA, but in retrieval, so we do not rely on it. The leave-one-subject-out comparison, and the transfer test to the confederate protocol, will show whether supervision wins. If it does on accuracy, the interpretability and reversal analyses still require the unsupervised directions, and we will say so plainly.

Why not just use a large pretrained facial-action model? From Pixels to Semantics: Unified Facial Action Representation Learning for Micro-Expression Analysis reports UF1 0.8943 on a composite micro-expression benchmark. However, it needed about a million face-video pairs for pretraining, dropped to UF1 0.6807 on CAS(ME)$^3$, and was trained on pre-clipped expressions rather than minutes-long exclusion episodes. Its features are compatible with our method rather than competing with it: we will run the contrast on its embeddings as an alternative input and report whether they carry more exclusion variance than tracker features.

Why not just hand-code flight and displacement behaviors? Hand coding is slow and was limited to 20 participants in Krahmer et al. It also inherits the gender asymmetries documented there and in Do we like honest signal givers? Macro and micro expressions of disgust. We use hand codes as the interpretive ground truth for the shared directions, not as a competitor. If the shared directions simply reproduce the four categories, that is a useful validation, not a failure.

Proposed Experiments and Decisive Checks

The first decisive check is covariance versus means. It uses the leave-one-subject-out AUC comparison against the z-scored mean-shift detector after time-direction removal from the control arm. The claim needs the covariance method to win with an interval excluding zero.

The second check is sharedness. The average principal angle between individuals' top exclusion directions must be significantly smaller than under a null built by contrasting two halves of the inclusion phase. The shared subspace, applied to held-out people without their own directions, must beat chance and approach per-person performance.

The third check is reversal. In re-acceptance, variance along the shared exclusion directions should fall back toward inclusion levels, as the repair effect in Social inclusion regulates the effect of social exclusion on adaptation to emotional conflict predicts, and the control arm should show no such drop. Failure here would suggest the directions track accumulated arousal rather than exclusion itself.

The fourth check is transfer. The shared subspace from Cyberball is evaluated without refitting on the confederate replication. The out-of-domain degradation reported by Ma et al. for PCA makes this a genuine risk.

The fifth check is fairness. We report AUC separately for women and men, before and after the Fair PCA constraint of Efficient fair PCA for fair representation learning, and compare the gap to the human-judge gap on the same clips. Success means the automated gender gap is smaller than the human one with no more than a small loss in overall AUC.

Together these checks separate three outcomes: a real, shared, reversible covariance signature of exclusion; a mean-shift effect dressed up in eigenvectors; and a time-on-task artifact.