TL;DR
THG-Omni is a new framework that brings "traceability" to video reasoning by grounding Multimodal Large Language Models (MLLMs) in structured, human-centric graphs. By combining SAM 3 tracking with verified semantic annotations, it teaches models to link specific participants to their actions and emotions across time, resulting in significant performance gains on complex social reasoning benchmarks like IntentBench and NExT-QA.
What Problem Does This Paper Address?
Current MLLMs are surprisingly good at answering questions about videos, but they often act as "black boxes." They might correctly identify that a person is lying or feeling sad, but they rarely provide evidence for which participant, what specific event, or when in the timeline that evidence occurred. This lack of traceability makes it hard to trust their reasoning.
Furthermore, most models are trained using "answer-level supervision"—they are simply told the right answer. This doesn't teach the model how to interpret the underlying structure of a video, such as maintaining a person's identity across camera cuts or associating a facial expression with a specific interaction. Human-centric reasoning is particularly difficult because it requires integrating subtle cues (like vocal delivery and body motion) that are often distributed across a long video.
Methods
The authors propose THG-Omni, a framework that constructs a human-centric graph and uses it for targeted supervision. The process involves several key stages:
- Question-Guided Grounding: The system identifies human references in a question and uses Segment Anything Model 3 (SAM 3) to detect and track those participants. This provides deterministic geometric data: bounding boxes, trajectories, and spatial relations (e.g., "Person A is to the left of Person B").
- Semantic Annotation: Qwen3-VL generates semantic claims about the tracked individuals, including their attributes (blue shirt), actions (kneeling), affective states (serious), and interactions (facing each other).
- Cross-Model Verification: To ensure the training data is high-quality, the authors introduce a verification step. A second model, HumanOmniV2, checks each semantic claim against the source video and audio. Only claims that are explicitly "supported" are kept, avoiding the pitfalls of learning from hallucinations.
- Graph-Specific Supervision: Instead of just giving the model the graph as extra text, the authors use Low-Rank Adaptation (LoRA) to train the model to interpret the graph. They generate specific questions about the graph fields (e.g., "What is man_1 doing?") to ensure the model learns the mapping between the structured data and the video evidence.
Datasets
The framework was evaluated on several challenging benchmarks:
- IntentBench: A comprehensive suite for social reasoning, emotion understanding, and deception perception (comprising Social-IQ, EMER, and MDPE).
- NExT-QA: A benchmark focused on causal, temporal, and descriptive reasoning in videos.
- MER24: Used for a "graph-information diagnostic" to test how much reasoning information is preserved in the graph itself when the source video and audio are removed.
Key Insights and Ideas
- Construction vs. Utilization: The authors argue that building a good graph and getting a model to use it are two different problems. Even a perfect graph is useless if the model hasn't been supervised to pay attention to its specific fields.
- Traceability as a Learning Target: By supervising identity, events, and time, the model learns a "traceable evidence path." This forces the model to ground its final answer in identifiable participants and intervals.
- Positive-Only Protocol: By only using verified positive claims for training, the framework avoids the "lexical shortcuts" that often occur when models are trained on synthetically generated negative/counterfactual statements.
- The Power of Compactness: Interestingly, experiments on NExT-QA showed that a "compact" graph (removing emotion and intent hypotheses) performed slightly better than a full graph, suggesting that filtering out semantic noise is as important as providing data.
Key Results
THG-Omni demonstrated substantial improvements over unadapted backbones:
- IntentBench: Achieved a sample-weighted overall score of 71.72%, a massive jump from the Qwen2.5-Omni-7B baseline of 55.27%.
- Social-IQ: Accuracy improved from 53.99% to 71.18%.
- NExT-QA: Reached 89.54% accuracy with the matched graph, compared to 85.10% without graph input.
- Traceability Diagnostic: In the MER24 experiment (where the model only saw the graph and the question, no video), the exact match rate rose from 14.67% to 24.00%, proving the graph successfully captures and preserves reasoning-relevant evidence.
Limitations and Future Work
- Tracking Failures: The system still struggles with persistent identity tracking across complex camera transitions or long occlusions. If the tracker fails, the semantic claims can be attached to the wrong person.
- Automatic Inference Noise: While training data is verified by humans and secondary models, the graphs used during inference are generated automatically. This means errors in the initial annotation can still propagate to the final answer.
- Dynamic Field Selection: The authors suggest future work could involve teaching the model to dynamically select which parts of a graph are relevant to a specific question, further reducing the impact of irrelevant semantic noise.
- Scaling: Testing the framework on even longer videos with more complex, multi-party interactions remains a goal for future research.
Summary updated