IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Models

Hamed Rahimi, Clemence Grislain, Adrien Jacquet Cretides, Olivier Sigaud, Mohamed ChetouaniarXiv 2026 · arXiv

TL;DR

IntentVLM is a novel two-stage video-language framework that enables social robots to infer open-vocabulary human intentions from continuous video streams. By mimicking the "forward-inverse" modeling processes found in cognitive science, the model first generates plausible goal candidates and then selects the most likely intention. This structured approach achieves state-of-the-art performance on intention recognition benchmarks, matching or exceeding human-level accuracy without sacrificing general scene-understanding capabilities.

What Problem Does This Paper Address?

For autonomous robots to seamlessly integrate into human-centered environments (like homes or hospitals), they must look beyond observable actions and understand why a person is doing something. For example, if someone is scanning a room, they aren't just looking around out of casual curiosity—they might be looking for assistance.

Inferring these latent mental states is incredibly challenging because intentions are inherently unobservable and highly context-dependent. Traditional intention recognition systems simplify the problem by treating it as a closed-set classification task over a predefined vocabulary of labels. However, real-world human intentions are diverse and open-ended. While modern Vision-Language Models (VLMs) offer zero-shot reasoning capabilities, relying on a single-pass text generation often leads to "hallucinations" and inconsistent predictions in complex, temporally evolving video scenes.

Methods

IntentVLM addresses these limitations by breaking down intention recognition into a structured, two-stage pipeline inspired by cognitive forward-inverse reasoning:

  1. Stage 1: Proposal (Goal Candidate Generator): Acting as a cognitive "forward model," this component simulates plausible future states. Given an input video and a textual query, a large VLM backbone (Qwen3-VL 8B) generates a diverse set of $K$ candidate goals. This expands the exploration space while grounding the possibilities in the video's context.
  2. Stage 2: Selection (Intention Inference Module): Acting as the "inverse model," this component evaluates the generated candidates. A smaller VLM instance (Qwen3-VL 2B or 4B) scores each candidate based on its consistency with the visual evidence and selects the goal with the highest posterior probability as the final inferred intention.

Both stages utilize frozen VLM backbones augmented with parameter-efficient Low-Rank Adaptation (LoRA) adapters, ensuring computational efficiency during training.

Architecture of IntentVLM

Datasets

The authors evaluated IntentVLM on two distinct multimodal benchmarks:

  • IntentQA: A video dataset specifically designed for context-aware human intention reasoning. It features four question-answering categories: Causal What (explaining an action), Causal How (how an action leads to an outcome), Temporal Next (predicting future actions), and Temporal Past (inferring past intentions).
  • Inst-IT Bench: A fine-grained benchmark evaluating instance-level comprehension (attributes, interactions, and temporal changes) across images and videos. This dataset was used to test whether fine-tuning the model for intention recognition caused it to forget general visual understanding capabilities.

Key Insights and Ideas

  • Structured Reasoning Mitigates Hallucinations: Unconstrained text generation in standard VLMs often leads to erratic outputs. By constraining the final inference step to a dynamically generated set of candidates, IntentVLM significantly reduces the solution space and eliminates common reasoning hallucinations.
  • Cognitive Science Parallels Work in AI: Decomposing intent attribution into a forward proposal step and an inverse selection step closely mirrors how the human brain is thought to predict and interpret the actions of others.
  • Adaptation Without Catastrophic Forgetting: A major concern with fine-tuning foundation models on niche tasks is the degradation of their broader capabilities. IntentVLM maintains high performance on general instance-level tasks (Inst-IT Bench) even after being optimized for intention recognition, proving that task-specific adaptation can preserve foundational knowledge.

Key Results

  • Surpassing Human Performance: On the IntentQA multiple-choice benchmark, IntentVLM (4B variant) achieved a total accuracy of 85.15% on the test set. This represents a massive ~30% absolute improvement over previous state-of-the-art baselines (such as VideoQA at 57.64%) and slightly outperforms the human baseline of 78.49%.
Figure 1: IntentVLM for open-vocabulary human intention recognition from video streams. Given a continuous video stream of a human in a kitchen environment, IntentVLM integrates multimodal visual cues — scene context, object interactions, and body movements — to infer free-form user intentions without relying on a predefined action vocabulary. The model reasons over temporal video evidence to produce natural language intention descriptions ("Make coffee", "Heat water for tea"), demonstrating the capacity of video-language understanding to ground open-vocabulary human goals in real-world robotic settings.
  • Open-Vocabulary Success: In free-form evaluation settings where no multiple-choice options are provided, the two-stage proposal-selection framework consistently outperformed direct answer generation by approximately 5 points across automatic text metrics (ROUGE, Cosine Similarity, and BERTScore).
  • Efficiency and Scale: The 4B parameter version of IntentVLM achieved performance competitive with or superior to general video foundation models that are more than double its size (7B+ parameters), demonstrating an excellent performance-to-size trade-off.

Limitations and Future Work

Despite its strong performance, the authors note that IntentVLM's accuracy still degrades significantly in purely open-ended generation settings compared to multiple-choice scenarios. Removing all structural constraints expands the output space drastically, which remains an open challenge for unconstrained generation.

Additionally, the authors include a responsible innovation statement highlighting the privacy risks of continuous video monitoring (which can resemble surveillance) and the potential for cultural biases if the model is deployed in sensitive caregiving or assistive robotics contexts. Future work will need to address these ethical and deployment constraints.

Summary updated

Abstract

Improving the effectiveness of human-robot interaction requires social robots to accurately infer human goals through robust intention understanding. This challenge is particularly critical in multimodal settings, where agents must integrate heterogeneous signals including text, visual cues to form a coherent interpretation of user intent. This paper presents IntentVLM, a novel two-stage video-language framework designed for open-vocabulary human intention recognition. The approach is inspired by forward-inverse modeling in cognitive science by decomposing intention understanding into goal candidate generation followed by structured inference through selection, effectively reducing hallucinations in latent reasoning. Evaluated on the IntentQA and Inst-IT Bench datasets, IntentVLM achieves state-of-the-art results with up to 80% accuracy, notably surpassing the baseline performance by 30% and matches human performance. Our findings demonstrate that this structured reasoning approach enhances open-vocabulary intention understanding without catastrophic forgetting, offering a robust foundation for human-centered robotics.
BibTeX
@misc{rahimi2026intentvlm,
  author = {Hamed Rahimi and Clemence Grislain and Adrien Jacquet Cretides and Olivier Sigaud and Mohamed Chetouani},
  title = {{IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Models}},
  year = {2026},
  eprint = {2604.24002},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.24002},
}

Loading related content...

Figure