Explaining Vision and Language through Graphs of Events in Space and Time

Mihai Masala, Nicolae Cudlenco, Traian Rebedea, Marius Leordeanu

Abstract

Artificial Intelligence makes great advances today and starts to bridge the gap between vision and language. However, we are still far from understanding, explaining and controlling explicitly the visual content from a linguistic perspective, because we still lack a common explainable representation between the two domains. In this work we come to address this limitation and propose the Graph of Events in Space and Time (GEST), by which we can represent, create and explain, both visual and linguistic stories. We provide a theoretical justification of our model and an experimental validation, which proves that GEST can bring a solid complementary value along powerful deep learning models. In particular, GEST can help improve at the content-level the generation of videos from text, by being easily incorporated into our novel video generation engine. Additionally, by using efficient graph matching techniques, the GEST graphs can also improve the comparisons between texts at the semantic level.

The Challenge of Explainable Vision-Language Models

While AI has made massive leaps in generating images and videos from text, these models often act as "black boxes." We can give them a prompt, but we have very little control over the specific flow of events, and the models often struggle with long-term logical consistency. For example, current state-of-the-art models can produce inappropriate or nonsensical content for simple prompts like "A woman goes to the bedroom" because they lack a grounded understanding of the "story" being told.

The core issue is the lack of a common explainable representation. Text is symbolic; video is a dense stream of pixels. Bridging them requires a middle ground that captures the meaning (the who, what, where, and when) in a way that both humans and machines can inspect and manipulate.

The GEST Framework

Researchers have introduced GEST (Graph of Events in Space and Time), a hierarchical graph representation that acts as a "common language" between text and video. By mapping stories into structured events and relations, GEST allows for more explainable text-to-video generation and more accurate semantic comparisons between different pieces of text.

Overview of the GEST framework connecting video and text representations.

In the GEST framework:

  • Nodes represent Events: These range from simple actions (opening a door) to complex activities (a political revolution).
  • Edges represent Relations: These define how events interact, covering temporal (before/after), spatial (inside/near), or semantic (causality) relationships.

A key feature of GEST is its recursive hierarchy. A whole graph representing a complex story can be "collapsed" into a single event node at a higher level of abstraction. Conversely, a single node can be "expanded" into a detailed sub-graph. This mimics how humans summarize stories—we don't remember every pixel; we remember the sequence of key events.

Applications and Methodology

To test this, the authors developed two main applications:

  1. Semantic Comparison: Using graph matching algorithms (Spectral Matching and Neural Graph Matching) to compare GEST representations of different texts. This proved that structural matching is better at identifying if two descriptions refer to the same video than standard word-overlap metrics.
  2. Video Generation Engine: A system that takes a GEST graph and "renders" it into a video using a game engine (via Multi Theft Auto). This allows for perfect control over actors, locations, and actions, ensuring the video strictly follows the "script" defined by the graph.

Key Results and Performance

The study primarily utilized the Videos-to-Paragraphs dataset to evaluate how well GEST could identify semantic similarities between different natural language descriptions of the same visual events.

Comparison of overall semantic scores between GEST (Ours) and baseline models.
  • Superior Text Similarity: Combining GEST with metrics like BLEURT saw a consistent performance boost. The "GEST-SM" (Spectral Matching) approach achieved the highest correlation and accuracy in several tests.
  • Human Preference: In a head-to-head comparison, human evaluators preferred GEST-generated videos over those from high-end diffusion models (like CogVideo or Text2VideoZero) in 87.39% of cases.
  • Semantic Consistency: On a scale of 1 to 10 for semantic content, GEST-generated videos scored an average of 7.93, significantly higher than CogVideo (4.03) and Text2VideoZero (5.55). This suggests that for complex storytelling, structure beats raw pixel quality.

Limitations and Future Work

The most obvious limitation is visual fidelity. Because the system currently relies on an older game engine, the videos look like video games rather than real life. However, while they aren't yet photorealistic, they are semantically "truthful."

Future work aims to integrate the structured, explainable power of GEST with the high-end rendering capabilities of modern deep learning models. The goal is a hybrid system: the "brain" of GEST handles the logic and story structure, while a generative model handles the "skin" to produce photorealistic, yet perfectly controlled, video content.

Summary updated

Abstract

Artificial Intelligence makes great advances today and starts to bridge the gap between vision and language. However, we are still far from understanding, explaining and controlling explicitly the visual content from a linguistic perspective, because we still lack a common explainable representation between the two domains. In this work we come to address this limitation and propose the Graph of Events in Space and Time (GEST), by which we can represent, create and explain, both visual and linguistic stories. We provide a theoretical justification of our model and an experimental validation, which proves that GEST can bring a solid complementary value along powerful deep learning models. In particular, GEST can help improve at the content-level the generation of videos from text, by being easily incorporated into our novel video generation engine. Additionally, by using efficient graph matching techniques, the GEST graphs can also improve the comparisons between texts at the semantic level.
BibTeX
@misc{masala2023explaining,
  author = {Mihai Masala and Nicolae Cudlenco and Traian Rebedea and Marius Leordeanu},
  title = {{Explaining Vision and Language through Graphs of Events in Space and Time}},
  year = {2023},
  eprint = {2309.08612},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.08612},
}
Figure