From Action Recognition to Social Intent Inference

103 concepts80 papers1 proposal

Current artificial intelligence excels at identifying the "what" and "how" of a scene. Computer vision models can reliably label a person (a noun) and detect that they are running (a verb). However, as Bo Pang et al. observe, AI has historically ignored the adverb—the descriptor of how and why an action is performed. This represents a "social gap" where models can describe physical movements but fail to interpret the underlying mental states that make those movements meaningful.

The Social Perception Gap

Humans are "social detectives" who naturally perform Theory of Mind (ToM): the attribution of mental states like beliefs, goals, and intentions to others. When we see someone reach for a cup, we don't just see a trajectory; we see a thirsty person. Haojun Shi et al. argue that while AI has mastered physical action recognition, it still fails to bridge the gap to deep social understanding.

This failure is quantified in benchmarks like Social-IQ, where Amir Zadeh et al. found that while humans navigate social interactions with 95% accuracy, AI models struggle to break 65%. The difficulty lies in the fact that identical physical actions can stem from entirely different motivations. As the authors of IntentQA suggest, intent is the "dark matter" of social intelligence—unobservable in the visual stream but governing the behavior of every participant.

Modeling the Unobservable via Inverse Planning

To move beyond surface-level pattern matching, researchers are shifting toward intuitive psychology. Tianmin Shu et al. demonstrate that models with built-in "core knowledge" of physics and planning are far better at generalizing to novel social situations than standard neural networks.

The primary mechanism for this is Inverse Planning. Instead of just predicting the next frame, the system works backward:

  1. Observe the agent's physical actions and constraints.
  2. Assume the agent is acting rationally to achieve a goal.
  3. Infer which hidden goal or belief would most likely lead to that specific action.

Katherine M. Collins et al. implemented this in the Neuro-Symbolic Inverse Planning Engine (NIPE), which uses Large Language Models to translate descriptions into symbolic world models, allowing for human-like accuracy in goal inference. Similarly, Aviv Netanyahu showed that explicitly reasoning about physical constraints and mental goals allows models to distinguish between helping, hindering, and collaborating in abstract animations.

The Causal Chain of Social Reasoning

A recurring finding across recent work is that identifying a social cue is not the same as understanding it. Matteo Bortoletto notes that models often succeed at identifying immediate perceptions (e.g., "Alice saw the book") but fail to propagate that information through a causal chain to update internal beliefs (e.g., "Therefore, Alice knows where the book is").

This lack of logical consistency is a major hurdle. Even advanced Vision-Language Models (VLMs) exhibit a "performance cliff" when transitioning from identifying physical attributes to inferring the purpose of a social group. Simindokht Jahangard et al. found that models often fail to distinguish between similar physical behaviors that have different social intents, such as "wandering" versus a "purposeful excursion."

Integrating Multimodal Context

True social intelligence requires fusing disparate signals. Ping Wei et al. demonstrate that gaze and intent are deeply coupled; knowing where someone is looking significantly improves the ability to predict what they intend to do. Muhammet Furkan Ilaslan et al. further show that eye-gaze data is a critical "missing link" for resolving ambiguities in human-robot collaboration.

Beyond visual cues, the social context—relationships, personality traits, and cultural norms—acts as a "premise" for reasoning. Qingxiu Dong et al. argue that predicting behavior requires integrating these abstract social premises to resolve the ambiguity of future actions. Without this, AI remains a passive observer, unable to transition into the socially active participant described by John F. Canny et al., capable of providing meaningful support in human environments.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_social_intent_inference_33467c8d,
  title = {social intent inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/social-intent-inference/rdr_33467c8d946c4577bf4e35c3e084f3e5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_social_intent_inference_33467c8d_v1,
  title = {social intent inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/social-intent-inference/rdr_33467c8d946c4577bf4e35c3e084f3e5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake