TL;DR
IntentVLM is a novel two-stage video-language framework that enables social robots to infer open-vocabulary human intentions from continuous video streams. By mimicking the "forward-inverse" modeling processes found in cognitive science, the model first generates plausible goal candidates and then selects the most likely intention. This structured approach achieves state-of-the-art performance on intention recognition benchmarks, matching or exceeding human-level accuracy without sacrificing general scene-understanding capabilities.
What Problem Does This Paper Address?
For autonomous robots to seamlessly integrate into human-centered environments (like homes or hospitals), they must look beyond observable actions and understand why a person is doing something. For example, if someone is scanning a room, they aren't just looking around out of casual curiosity—they might be looking for assistance.
Inferring these latent mental states is incredibly challenging because intentions are inherently unobservable and highly context-dependent. Traditional intention recognition systems simplify the problem by treating it as a closed-set classification task over a predefined vocabulary of labels. However, real-world human intentions are diverse and open-ended. While modern Vision-Language Models (VLMs) offer zero-shot reasoning capabilities, relying on a single-pass text generation often leads to "hallucinations" and inconsistent predictions in complex, temporally evolving video scenes.
Methods
IntentVLM addresses these limitations by breaking down intention recognition into a structured, two-stage pipeline inspired by cognitive forward-inverse reasoning:
- Stage 1: Proposal (Goal Candidate Generator): Acting as a cognitive "forward model," this component simulates plausible future states. Given an input video and a textual query, a large VLM backbone (Qwen3-VL 8B) generates a diverse set of $K$ candidate goals. This expands the exploration space while grounding the possibilities in the video's context.
- Stage 2: Selection (Intention Inference Module): Acting as the "inverse model," this component evaluates the generated candidates. A smaller VLM instance (Qwen3-VL 2B or 4B) scores each candidate based on its consistency with the visual evidence and selects the goal with the highest posterior probability as the final inferred intention.
Both stages utilize frozen VLM backbones augmented with parameter-efficient Low-Rank Adaptation (LoRA) adapters, ensuring computational efficiency during training.
Datasets
The authors evaluated IntentVLM on two distinct multimodal benchmarks:
- IntentQA: A video dataset specifically designed for context-aware human intention reasoning. It features four question-answering categories: Causal What (explaining an action), Causal How (how an action leads to an outcome), Temporal Next (predicting future actions), and Temporal Past (inferring past intentions).
- Inst-IT Bench: A fine-grained benchmark evaluating instance-level comprehension (attributes, interactions, and temporal changes) across images and videos. This dataset was used to test whether fine-tuning the model for intention recognition caused it to forget general visual understanding capabilities.
Key Insights and Ideas
- Structured Reasoning Mitigates Hallucinations: Unconstrained text generation in standard VLMs often leads to erratic outputs. By constraining the final inference step to a dynamically generated set of candidates, IntentVLM significantly reduces the solution space and eliminates common reasoning hallucinations.
- Cognitive Science Parallels Work in AI: Decomposing intent attribution into a forward proposal step and an inverse selection step closely mirrors how the human brain is thought to predict and interpret the actions of others.
- Adaptation Without Catastrophic Forgetting: A major concern with fine-tuning foundation models on niche tasks is the degradation of their broader capabilities. IntentVLM maintains high performance on general instance-level tasks (Inst-IT Bench) even after being optimized for intention recognition, proving that task-specific adaptation can preserve foundational knowledge.
Key Results
- Surpassing Human Performance: On the IntentQA multiple-choice benchmark, IntentVLM (4B variant) achieved a total accuracy of 85.15% on the test set. This represents a massive ~30% absolute improvement over previous state-of-the-art baselines (such as VideoQA at 57.64%) and slightly outperforms the human baseline of 78.49%.
- Open-Vocabulary Success: In free-form evaluation settings where no multiple-choice options are provided, the two-stage proposal-selection framework consistently outperformed direct answer generation by approximately 5 points across automatic text metrics (ROUGE, Cosine Similarity, and BERTScore).
- Efficiency and Scale: The 4B parameter version of IntentVLM achieved performance competitive with or superior to general video foundation models that are more than double its size (7B+ parameters), demonstrating an excellent performance-to-size trade-off.
Limitations and Future Work
Despite its strong performance, the authors note that IntentVLM's accuracy still degrades significantly in purely open-ended generation settings compared to multiple-choice scenarios. Removing all structural constraints expands the output space drastically, which remains an open challenge for unconstrained generation.
Additionally, the authors include a responsible innovation statement highlighting the privacy risks of continuous video monitoring (which can resemble surveillance) and the potential for cultural biases if the model is deployed in sensitive caregiving or assistive robotics contexts. Future work will need to address these ethical and deployment constraints.
Summary updated