Before Words, Beyond Speech: Evaluating Nonverbal Social Reasoning in Early Childhood

Marie Amale Huynh, Laura Bravo-Sánchez, Lauren Klein, Nick Haber, Philip Fisher, Serena Yeung-Levy

TL;DR

Researchers from Stanford have introduced NEST (Naturalistic Early-childhood Social inTeractions), the first benchmark specifically designed to evaluate how well Vision-Language Models (VLMs) understand non-verbal social reasoning in young children. Their findings reveal a "performance collapse": while state-of-the-art models are excellent at recognizing broad context, they struggle significantly to decode the subtle, sub-second behavioral and interpersonal cues—like a shared gaze or a specific reaction—that define early human development.

What Problem Does This Paper Address?

Most current video understanding benchmarks are "adult-centric" and "speech-centric." They rely heavily on dialogue to infer social meaning. However, children navigate the world through a sophisticated set of non-verbal tools—gaze, gesture, and physical proximity—long before they speak their first words.

Interpreting these signals is crucial for child development research and clinical interventions, but it remains a manual, labor-intensive process for human experts. While VLMs have advanced rapidly, they have a blind spot for these subtle interactions. The authors argue that we lack the tools to measure if AI can actually "read the room" in a nursery or classroom setting, where social meaning is built through composition (e.g., a child points, a mother looks, and they both smile).

Methods

The authors developed a compositional evaluation schema to move beyond simple "accuracy" scores. As shown in Figure 2, NEST breaks down social reasoning into four hierarchical levels:

  1. Contextual: Basic scene properties (e.g., "Where is this taking place?").
  2. Behavioral: Atomic actions and temporal signals (e.g., "What is the child doing at second 3?").
  3. Interpersonal: Causal and reactive structures (e.g., "How did the caregiver respond to the child's point?").
  4. Open-ended Descriptions: Free-form narratives of the interaction.
Figure 2: (Left) NEST’s hierarchical task ontology for non-verbal social cues. (Right) Example of dense per-task VQA annotations (avg. ∼12 per clip) used to decompose model performance across the NEST schema.

To ensure the benchmark was rigorous, they used AutoConverter, a tool that employs an LLM to generate "hard" distractors for multiple-choice questions. These distractors are designed to be both visually grounded and linguistically plausible, preventing models from simply guessing based on language priors.

Datasets

NEST comprises 1,208 manually annotated clips (~12s each) and over 14,000 structured questions. The data is curated for diversity across 10 countries and various developmental stages (infancy to school age).

The clips are sourced from three distinct corpora:

  • ChildPlay: Naturalistic videos from YouTube.
  • CHILDES: A longitudinal research corpus.
  • FIND: A private dataset from a real-world caregiver coaching intervention.
Figure 4: NEST Statistics. A) Child age distribution (677/1,208 clips with reported age). B) Recording country distribution (801/1,208 with reported origin). C) Duration distributions for actions (a behavior directed at another participant) and reactions (the responding behavior), covering both adult→child and child→adult pairs.

Key Insights and Ideas

  • The Performance Collapse: Models exhibit a sharp drop in accuracy as they move from perception to reasoning. A model might correctly identify that a child is in a library (Contextual) but fail to see that the child is reacting to a caregiver's frown (Interpersonal).
  • Causal Asymmetry: Models are significantly better at identifying the cause of a reaction than predicting the consequence of an action. It is easier for a VLM to work backward from a cry to a taken toy than to predict how a child will respond to a toy being offered.
  • Temporal Sensitivity: There is a direct correlation between action duration and model accuracy. Brief, sub-second cues—which are often the most meaningful in child development—are frequently missed by current VLMs.
  • Non-Verbal Necessity: Ablation studies (Figure 5B) proved that NEST is fundamentally visual. Removing the video stream caused performance to crater, confirming that the benchmark cannot be "cheated" using just audio or text transcripts.

Key Results

The study evaluated a wide range of models, including Qwen 3.5, LLaVA-OV, and GPT-5 mini.

  • Human vs. Model: While humans achieve ~87.5% accuracy across the board, the best open-source model (Qwen 3.5-27B) hits ~88.5% on contextual tasks but drops to ~65.8% on interpersonal reasoning.
  • Open-Ended Failure: In free-form descriptions, models often collapsed into generic phrases like "playing with toys," with recall scores below 30%.
  • Sankey Analysis: Figure 5A illustrates the flow of model success: many models that get the "Context" right still fail the "Interpersonal" questions for the same clip.
Figure 5: A) Sankey plot of the best open-source model Qwen 3.5-27B, showing how correct (purple) and incorrect (gray) questions change across NEST’s levels. B) Model performance by input type relative to the visual input, the results highlight the non-verbal and visual nature of NEST.

Limitations and Future Work

The authors acknowledge several key limitations:

  • Data Privacy: A portion of the dataset (FIND) is private due to the sensitive nature of filming minors, meaning the full benchmark results cannot be perfectly replicated by the public.
  • Standardized Resolution: To ensure fair comparison, models were capped at 512px resolution, which might bottleneck models capable of higher fidelity.
  • LLM-as-Judge: While they validated their automated scoring against human experts, they noted that LLM judges can be slightly conservative and may miss subtle factual errors.

Future work points toward building "socially grounded" AI that doesn't just narrate video, but understands the intricate, non-verbal "serve and return" dynamics essential to human connection.

Summary updated

Abstract

Children's earliest years are a period of intense social learning, during which meaning is built through gaze, gesture, and physical contact. Yet, this critical aspect of social interaction remains under-evaluated in multimodal video understanding, as existing benchmarks prioritize the conversation-driven interactions of adults. We address this gap with \bench ($\textbf{N}$aturalistic $\textbf{E}$arly-childhood $\textbf{S}$ocial in$\textbf{T}$eractions), the first child-centric benchmark for nonverbal social reasoning. \bench comprises \totalvids manually annotated clips ($\sim$12s) curated for diversity across naturalistic settings, cultural contexts, and developmental stages. Beyond a lack of data, targeted model improvement in this domain is hindered by coarse evaluation schemas that conflate basic perception with higher-order reasoning. To resolve this, \bench introduces a compositional schema that builds atomic interaction cues into progressive levels of social reasoning: contextual, behavioral, interpersonal, and open-ended descriptions. Extensive analysis of state-of-the-art Vision-Language Models (VLMs) reveals a stark performance collapse: despite strong contextual recognition, models exhibit substantial gaps in decoding precise behavioral and interpersonal cues. We trace these failures to specific biases, including poor consequence grounding and temporal sensitivity. Finally, we ground \bench in real-world applications by evaluating VLMs using domain adaptation strategies. Ultimately, \bench provides a rigorous testbed for advancing socially grounded artificial intelligence in early childhood. Project page: https://laubravo.github.io/nest_website/.
BibTeX
@misc{huynh2026before,
  author = {Marie Amale Huynh and Laura Bravo-Sánchez and Lauren Klein and Nick Haber and Philip Fisher and Serena Yeung-Levy},
  title = {{Before Words, Beyond Speech: Evaluating Nonverbal Social Reasoning in Early Childhood}},
  year = {2026},
  eprint = {2611.13213},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13213},
}
Figure