THG-Omni: Traceable Human-Centric Graph Supervision for Video Reasoning

Jisheng Dang, Kai Wei, Ronghao Lin, Bimei Wang, Hong Peng, Wei-Shi Zheng, Bin Hu, Qi Tian, Tat-Seng ChuaarXiv 2026 · arXiv

TL;DR

THG-Omni is a new framework that brings "traceability" to video reasoning by grounding Multimodal Large Language Models (MLLMs) in structured, human-centric graphs. By combining SAM 3 tracking with verified semantic annotations, it teaches models to link specific participants to their actions and emotions across time, resulting in significant performance gains on complex social reasoning benchmarks like IntentBench and NExT-QA.

What Problem Does This Paper Address?

Current MLLMs are surprisingly good at answering questions about videos, but they often act as "black boxes." They might correctly identify that a person is lying or feeling sad, but they rarely provide evidence for which participant, what specific event, or when in the timeline that evidence occurred. This lack of traceability makes it hard to trust their reasoning.

Furthermore, most models are trained using "answer-level supervision"—they are simply told the right answer. This doesn't teach the model how to interpret the underlying structure of a video, such as maintaining a person's identity across camera cuts or associating a facial expression with a specific interaction. Human-centric reasoning is particularly difficult because it requires integrating subtle cues (like vocal delivery and body motion) that are often distributed across a long video.

Motivation for traceable graph supervision: contrasting answer-level vs. grounded graph supervision

Methods

The authors propose THG-Omni, a framework that constructs a human-centric graph and uses it for targeted supervision. The process involves several key stages:

  1. Question-Guided Grounding: The system identifies human references in a question and uses Segment Anything Model 3 (SAM 3) to detect and track those participants. This provides deterministic geometric data: bounding boxes, trajectories, and spatial relations (e.g., "Person A is to the left of Person B").
  2. Semantic Annotation: Qwen3-VL generates semantic claims about the tracked individuals, including their attributes (blue shirt), actions (kneeling), affective states (serious), and interactions (facing each other).
  3. Cross-Model Verification: To ensure the training data is high-quality, the authors introduce a verification step. A second model, HumanOmniV2, checks each semantic claim against the source video and audio. Only claims that are explicitly "supported" are kept, avoiding the pitfalls of learning from hallucinations.
  4. Graph-Specific Supervision: Instead of just giving the model the graph as extra text, the authors use Low-Rank Adaptation (LoRA) to train the model to interpret the graph. They generate specific questions about the graph fields (e.g., "What is man_1 doing?") to ensure the model learns the mapping between the structured data and the video evidence.
Overview of the THG-Omni framework: from video evidence to verified graph supervision

Datasets

The framework was evaluated on several challenging benchmarks:

  • IntentBench: A comprehensive suite for social reasoning, emotion understanding, and deception perception (comprising Social-IQ, EMER, and MDPE).
  • NExT-QA: A benchmark focused on causal, temporal, and descriptive reasoning in videos.
  • MER24: Used for a "graph-information diagnostic" to test how much reasoning information is preserved in the graph itself when the source video and audio are removed.

Key Insights and Ideas

  • Construction vs. Utilization: The authors argue that building a good graph and getting a model to use it are two different problems. Even a perfect graph is useless if the model hasn't been supervised to pay attention to its specific fields.
  • Traceability as a Learning Target: By supervising identity, events, and time, the model learns a "traceable evidence path." This forces the model to ground its final answer in identifiable participants and intervals.
  • Positive-Only Protocol: By only using verified positive claims for training, the framework avoids the "lexical shortcuts" that often occur when models are trained on synthetically generated negative/counterfactual statements.
  • The Power of Compactness: Interestingly, experiments on NExT-QA showed that a "compact" graph (removing emotion and intent hypotheses) performed slightly better than a full graph, suggesting that filtering out semantic noise is as important as providing data.

Key Results

THG-Omni demonstrated substantial improvements over unadapted backbones:

  • IntentBench: Achieved a sample-weighted overall score of 71.72%, a massive jump from the Qwen2.5-Omni-7B baseline of 55.27%.
  • Social-IQ: Accuracy improved from 53.99% to 71.18%.
  • NExT-QA: Reached 89.54% accuracy with the matched graph, compared to 85.10% without graph input.
  • Traceability Diagnostic: In the MER24 experiment (where the model only saw the graph and the question, no video), the exact match rate rose from 14.67% to 24.00%, proving the graph successfully captures and preserves reasoning-relevant evidence.

Limitations and Future Work

  • Tracking Failures: The system still struggles with persistent identity tracking across complex camera transitions or long occlusions. If the tracker fails, the semantic claims can be attached to the wrong person.
  • Automatic Inference Noise: While training data is verified by humans and secondary models, the graphs used during inference are generated automatically. This means errors in the initial annotation can still propagate to the final answer.
  • Dynamic Field Selection: The authors suggest future work could involve teaching the model to dynamically select which parts of a graph are relevant to a specific question, further reducing the impact of irrelevant semantic noise.
  • Scaling: Testing the framework on even longer videos with more complex, multi-party interactions remains a goal for future research.

Summary updated

Abstract

Omni-modal models can answer questions about video, but their predictions rarely show which participant, event, or interval supports an answer. Moreover, answer-level supervision alone does not teach a model to interpret structured video evidence. To systematically mitigate this issue, we present THG-Omni, a human-centric graph-supervision framework that links Segment Anything Model~3 participant tracks with temporally grounded Qwen3-VL annotations. Candidate semantic claims are checked against the source video and available audio before they are used to construct training questions, while track geometry provides explicit spatial and temporal supervision. The resulting questions teach an omni-modal reasoning model to interpret participant attributes, actions, affect, interactions, and their timing from a structured graph. At inference, THG-Omni uses a predicted graph constructed from the evaluation video. Experiments on IntentBench show consistent improvements over the unadapted backbone across social reasoning, emotion understanding, and deception perception. On NExT-QA, THG-Omni achieves 89.54\% accuracy with the matched graph, compared with 85.10\% without graph input. Further validation experiments show that compact graph input achieves 78.22\% accuracy, compared with 78.02\% using the full graph and 77.58\% without graph input. A complementary diagnostic demonstrates that the predicted graph preserves evidence useful for answering questions even without the source video and audio. Together, these findings show that THG-Omni improves human-centric video reasoning while grounding its evidence in identifiable participants and events. Our codes are available at https://anonymous.4open.science/r/review-n94hhfcwp5666/
BibTeX
@misc{dang2026thg,
  author = {Jisheng Dang and Kai Wei and Ronghao Lin and Bimei Wang and Hong Peng and Wei-Shi Zheng and Bin Hu and Qi Tian and Tat-Seng Chua},
  title = {{THG-Omni: Traceable Human-Centric Graph Supervision for Video Reasoning}},
  year = {2026},
  eprint = {2611.07836},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07836},
}
Figure