Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception

Kathy Garcia, Leyla Isik

TL;DR

While state-of-the-art video foundation models excel at identifying actions like "cooking" or "dancing," they are surprisingly blind to the subtle social dynamics—like competing, collaborating, or comforting—that humans perceive effortlessly. This paper introduces Behavioral Geometric Supervision (BGS), a lightweight fine-tuning method that uses human similarity judgments to restructure a model's embedding space. By updating less than 2% of parameters, BGS allows video models to surpass text-based language models in social understanding and even generalize zero-shot to abstract, out-of-distribution social animations.

What Problem Does This Paper Address?

When humans watch a brief video of two people interacting, we don't just see body movements; we instantly read intentions, emotions, and power dynamics. Current video foundation models (such as V-JEPA or TimeSformer) are heavily optimized for pixel reconstruction or action classification, leaving a massive gap in higher-level social-visual understanding.

In fact, prior benchmarks show that text embeddings of video captions (like MPNet) are significantly better at predicting human social similarity judgments than raw video models. This raises a fundamental question: Can purely visual models develop a human-like relational structure for social perception, or is this nuance something that can only be captured through the lens of language?

Methods

The authors introduce Behavioral Geometric Supervision (BGS), a hybrid training objective implemented via Low-Rank Adaptation (LoRA) to keep the computational footprint minimal (updating <2% of the backbones' parameters).

Instead of training the model on explicit labels (e.g., "friendly" or "aggressive"), BGS shapes the geometry of the model's embedding space using a dual-component loss function:

  1. Triplet Loss (Local Constraints): Based on human odd-one-out choices, this loss ensures that the embedding distance between a reference video (Anchor) and a socially similar video (Positive) is smaller than the distance to a socially distinct video (Negative).
  2. Representational Similarity Analysis (RSA) Loss (Global Geometry): To prevent the model from only fixing local relationships, a differentiable RSA loss aligns the model's overall pairwise distance matrix with the global structure of human judgments across batches of videos.
Figure 1: Triplet OOO Dataset & Video Model Hybrid Fine-Tuning. Human similarity judgments are collected in 20-trial odd-one-out sets and define positive and negative training signals. The model is updated with triplet loss (blue) on Anchor (A), Positive (P), and Negative (N) batches. 6x per epoch, an RSA loss (purple) is applied to a 24-video subset by aligning model pairwise distances with triplet-derived human similarities (see Eq. 4 for combined triplet-RSA objective).

The authors tested this approach across four distinct Vision Transformer (ViT) architectures: V-JEPA (2 and 2.1), TimeSformer, VideoMAE, and CLIP.

Datasets

The study leverages three main datasets for training and evaluation:

  • New Social Video Similarity Benchmark: The authors collected a novel dataset consisting of 49,484 human odd-one-out judgments from 245 participants watching 250 naturalistic, two-person social video clips (curated from the Moments in Time dataset).
  • PHASE (PHysically-grounded Abstract Social Events): A dataset of out-of-distribution social interactions depicted entirely by 2D geometric shapes moving in a physical environment. This was used to test zero-shot generalization.
  • UCF101: A standard action recognition dataset used to ensure the fine-tuning process didn't degrade the models' core capabilities.

Key Insights and Ideas

  • Emergent Social Dimensions: Remarkably, after undergoing BGS, the video models naturally developed linearly decodable representations of core social-affective attributes—specifically valence, arousal, and dominance—despite never being exposed to these labels during training.
  • Beyond Language Distillation: Through variance partitioning, the authors proved that BGS-aligned models don't just mimic text captions. The fine-tuned models captured unique visual-social variance that text-based language models completely miss.
  • Spontaneous Attention Shift: Without any explicit spatial supervision, the internal attention maps of BGS-tuned models automatically shifted away from generic background contexts and reorganized around socially informative regions: faces, gaze directions, and interacting hands.

Key Results

  • Closing the Human Gap: Fine-tuning with BGS nearly tripled the performance of the distilled V-JEPA 2.1 model, boosting its explained variance ($R^2$) from a baseline of 0.058 to 0.165. This surpassed the strongest text-embedding baseline (MPNet at 0.134) and approached the human split-half noise ceiling of 0.213.
Explained variance across different modalities and fine-tuned ablations against the human noise ceiling.
  • Language vs. Video Variance: As shown in the variance partitioning analysis, fine-tuning successfully shifted the explained variance from language-specific components into shared and video-unique social structures.
Figure 3: Variance partitioning between V-JEPA 2.1 and MPNet before and after finetuning. Fine- tuning shifts explained variance from language- specific to shared and video-specific structure, bringing total variance near the split-half reliabil- ity ceiling. Black outlines indicate total variance explained by video-model.
  • OOD Generalization: On the abstract PHASE benchmark, the BGS-aligned V-JEPA 2.1 achieved 81% accuracy in classifying social interactions—a massive 26-point leap over its pretrained baseline (55%), effectively matching human agreement levels.
  • No Catastrophic Forgetting: The models fully retained their original action-recognition capabilities on UCF101 (e.g., V-JEPA 2.1 slightly improved from 78.17% to 80.31% top-1 accuracy).

Limitations and Future Work

While highly effective, the authors note several limitations:

  • Dataset Scale: The 250-video stimulus set is relatively small compared to massive image-domain alignment datasets. Scaling up the video count is a natural next step.
  • Cultural Homogeneity: The dataset reflects predominantly Western social norms, as the participants were recruited from a U.S. university platform and the captions were written by native English speakers. Cross-cultural validation is required to ensure the model generalizes across different global social norms.
  • Consensus Smoothing: The training relies on aggregate human consensus, which inherently smooths over individual differences in how people perceive social interactions. Future work could explore personalized or subgroup-specific social alignment.

Summary updated

Abstract

Current video foundation models, including the strongest self-supervised models such as V-JEPA2, fail to capture how humans organize social information in dynamic scenes. For example, across a range of diverse vision models tested, none were able to predict human similarity judgments to social video clips as well as a sentence embedding model of the caption text (MPNet). We show this gap in vision model performance can be closed by a compact behavioral supervisory signal. We introduce behavioral geometric supervision (BGS): a hybrid objective that constrains local and global pairwise embedding geometry to match the relational similarity structure across videos. We apply this method using a new human similarity dataset, containing 49,484 odd-one-out judgments from 250 naturalistic social video clips, and low-rank adaptation across four ViT backbones (V-JEPA 2/2.1, TimeSformer, VideoMAE, and CLIP). We find that one of the best fine-tuned models, V-JEPA 2.1, nearly triples in performance compared to the pre-trained baseline and reaches close to the noise ceiling, exceeding the strongest sentence-embedding baseline. In addition, finetuned models (i) capture unique variance in human judgments that caption-based language embeddings do not, (ii) develop interpretable social-affective attributes (valence, arousal, and dominance) despite never being trained on any of these attributes, (iii) zero-shot transfer to a separate dataset of out-of-distribution abstract social interactions, and (iv) shift spatial attention from scene context to socially informative regions (faces, gaze, and interacting bodies). A matched language-distillation control fails to reproduce these gains, ruling out caption transfer as the mechanism. Our results show how a modest amount of human behavioral data can steer video models toward human-like social visual understanding.
BibTeX
@misc{garcia2025behavioral,
  author = {Kathy Garcia and Leyla Isik},
  title = {{Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception}},
  year = {2025},
  eprint = {2510.01502},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.01502},
}
Figure