TL;DR
While state-of-the-art video foundation models excel at identifying actions like "cooking" or "dancing," they are surprisingly blind to the subtle social dynamics—like competing, collaborating, or comforting—that humans perceive effortlessly. This paper introduces Behavioral Geometric Supervision (BGS), a lightweight fine-tuning method that uses human similarity judgments to restructure a model's embedding space. By updating less than 2% of parameters, BGS allows video models to surpass text-based language models in social understanding and even generalize zero-shot to abstract, out-of-distribution social animations.
What Problem Does This Paper Address?
When humans watch a brief video of two people interacting, we don't just see body movements; we instantly read intentions, emotions, and power dynamics. Current video foundation models (such as V-JEPA or TimeSformer) are heavily optimized for pixel reconstruction or action classification, leaving a massive gap in higher-level social-visual understanding.
In fact, prior benchmarks show that text embeddings of video captions (like MPNet) are significantly better at predicting human social similarity judgments than raw video models. This raises a fundamental question: Can purely visual models develop a human-like relational structure for social perception, or is this nuance something that can only be captured through the lens of language?
Methods
The authors introduce Behavioral Geometric Supervision (BGS), a hybrid training objective implemented via Low-Rank Adaptation (LoRA) to keep the computational footprint minimal (updating <2% of the backbones' parameters).
Instead of training the model on explicit labels (e.g., "friendly" or "aggressive"), BGS shapes the geometry of the model's embedding space using a dual-component loss function:
- Triplet Loss (Local Constraints): Based on human odd-one-out choices, this loss ensures that the embedding distance between a reference video (Anchor) and a socially similar video (Positive) is smaller than the distance to a socially distinct video (Negative).
- Representational Similarity Analysis (RSA) Loss (Global Geometry): To prevent the model from only fixing local relationships, a differentiable RSA loss aligns the model's overall pairwise distance matrix with the global structure of human judgments across batches of videos.
The authors tested this approach across four distinct Vision Transformer (ViT) architectures: V-JEPA (2 and 2.1), TimeSformer, VideoMAE, and CLIP.
Datasets
The study leverages three main datasets for training and evaluation:
- New Social Video Similarity Benchmark: The authors collected a novel dataset consisting of 49,484 human odd-one-out judgments from 245 participants watching 250 naturalistic, two-person social video clips (curated from the Moments in Time dataset).
- PHASE (PHysically-grounded Abstract Social Events): A dataset of out-of-distribution social interactions depicted entirely by 2D geometric shapes moving in a physical environment. This was used to test zero-shot generalization.
- UCF101: A standard action recognition dataset used to ensure the fine-tuning process didn't degrade the models' core capabilities.
Key Insights and Ideas
- Emergent Social Dimensions: Remarkably, after undergoing BGS, the video models naturally developed linearly decodable representations of core social-affective attributes—specifically valence, arousal, and dominance—despite never being exposed to these labels during training.
- Beyond Language Distillation: Through variance partitioning, the authors proved that BGS-aligned models don't just mimic text captions. The fine-tuned models captured unique visual-social variance that text-based language models completely miss.
- Spontaneous Attention Shift: Without any explicit spatial supervision, the internal attention maps of BGS-tuned models automatically shifted away from generic background contexts and reorganized around socially informative regions: faces, gaze directions, and interacting hands.
Key Results
- Closing the Human Gap: Fine-tuning with BGS nearly tripled the performance of the distilled V-JEPA 2.1 model, boosting its explained variance ($R^2$) from a baseline of 0.058 to 0.165. This surpassed the strongest text-embedding baseline (MPNet at 0.134) and approached the human split-half noise ceiling of 0.213.
- Language vs. Video Variance: As shown in the variance partitioning analysis, fine-tuning successfully shifted the explained variance from language-specific components into shared and video-unique social structures.
- OOD Generalization: On the abstract PHASE benchmark, the BGS-aligned V-JEPA 2.1 achieved 81% accuracy in classifying social interactions—a massive 26-point leap over its pretrained baseline (55%), effectively matching human agreement levels.
- No Catastrophic Forgetting: The models fully retained their original action-recognition capabilities on UCF101 (e.g., V-JEPA 2.1 slightly improved from 78.17% to 80.31% top-1 accuracy).
Limitations and Future Work
While highly effective, the authors note several limitations:
- Dataset Scale: The 250-video stimulus set is relatively small compared to massive image-domain alignment datasets. Scaling up the video count is a natural next step.
- Cultural Homogeneity: The dataset reflects predominantly Western social norms, as the participants were recruited from a U.S. university platform and the captions were written by native English speakers. Cross-cultural validation is required to ensure the model generalizes across different global social norms.
- Consensus Smoothing: The training relies on aggregate human consensus, which inherently smooths over individual differences in how people perceive social interactions. Future work could explore personalized or subgroup-specific social alignment.
Summary updated