Traditional video models operate under a "closed-book" assumption: they assume that every piece of information needed to understand a scene is contained within the pixels of the frames. However, human comprehension is rarely so isolated. When we watch a basketball game or a medical procedure, we bring a lifetime of external context—rules, names, and physics—to bear on the visual data.
Researchers are now moving toward Knowledge-Augmented Video Understanding, a paradigm that bridges the gap between raw visual perception and structured world knowledge.
The Perception Reasoning Gap
The fundamental problem in modern video AI is that identifying an object is not the same as understanding its significance. As Andong Wang, Bo Wu, and colleagues demonstrate in SOK-Bench, there is a sharp divide between "situated" knowledge (what is happening in the frame) and "open-world" knowledge (what we know about those objects).
A model might see a man on a boat, but without external context, it cannot know if he is an "astronomer" or if his actions violate a specific maritime rule. This limitation is highlighted in the KnowIT VQA dataset by Mayu Otani and Noa Garcia, which shows that human fans of a TV show significantly outperform AI because they possess show-specific background knowledge that isn't visible in any single clip.
Mapping Pixels to Entities with Knowledge Graphs
One way to bridge this gap is by grounding visual features in Knowledge Graphs (KGs). Instead of treating a video as a sequence of generic labels, these models treat it as a series of entities with rich, pre-existing relationships.
In the realm of sports, Zeyu Xi introduced KEANet, which uses a basketball-specific KG to move beyond descriptions like "a person shoots a ball" to "Brandon Ingram misses a 2pt jump shot." By aligning visual detections with player rosters and historical data, the model transforms generic motion into specific, professional-grade commentary. Similarly, Jiaxin Deng and researchers at KuaiShou developed a unified framework that uses heterogeneous KGs to perform "cross-video" reasoning, allowing the system to infer identities and professional roles that are never explicitly mentioned in the audio or text.
Retrieval as a Cognitive Anchor
When structured graphs are unavailable, models can use Retrieval-Augmented Generation (RAG) to pull in unstructured text as a reference. This acts as a "cognitive anchor," helping the model resolve visual ambiguities.
For instructional videos, Yeonjoon Jung proposed a system that retrieves external textual scripts to clarify what a "blurry bowl" might contain during a cooking tutorial. In narrative contexts, Kevin Qinghong Lin developed MovieSeq, which interleaves video clips with plot summaries and character photos. This allows the model to understand long-term character motivations that aren't visible in a single 5-second snippet.
Even for spatial tasks, H. Lee showed that generating "environmental captions" serves as a common-sense filter. If a user asks to find a scene in a kitchen, the model uses its internal knowledge of what a kitchen looks like to prune irrelevant footage of laundry rooms, effectively using common sense to narrow the search space.
Reasoning with Symbolic Logic
The most advanced frontier of this research involves using Large Language Models (LLMs) as reasoning agents that consult external "experts." Zongxin Yang and the creators of DoraemonGPT use a symbolic memory approach. When a video shows a complex physical interaction, the system doesn't just guess what happens next; it uses a planner to query specialized domain knowledge—like the laws of physics or chemistry—to explain the underlying causes of visual phenomena.
By moving from pure visual pattern matching to a synthesis of perception and external facts, these models are beginning to replicate the "world model" that humans use to navigate and interpret the visual world.