Grounding Spatio-Temporal Language with Transformers

Tristan Karch, Laetitia Teodorescu, Katja Hofmann, Clément Moulin-Frier, Pierre-Yves Oudeyer

Introduction

In the world of embodied AI, "language grounding" is the process of mapping words to physical realities. While we have made great strides in teaching robots simple, static commands like "pick up the red ball," most current models struggle with language that involves time and relative history.

This paper introduces a novel benchmark and study for grounding language that describes how objects move and change over time. By testing various Transformer architectures in a custom 2D environment, the researchers discovered that maintaining distinct identities for objects within a model's attention mechanism is the critical "secret sauce" for understanding complex spatio-temporal descriptions.

The Challenge of Spatio-Temporal Language

Consider a sentence like: "You previously grasped the thing that used to be at the right of the parrot." To understand this, an agent cannot simply look at a single snapshot of the world. It must simultaneously understand:

  1. Spatial relations (e.g., "right of the parrot").
  2. Temporal tenses (e.g., "previously grasped," "used to be").
  3. Object persistence (e.g., tracking the "thing" even as it moves across the scene).

The authors identify a significant gap in research regarding these spatio-temporal concepts and propose a systematic way to study how neural architectures—specifically Transformers—handle this complexity.

The Temporal Playground Dataset

To test these capabilities, the authors introduced the Temporal Playground, a 2D environment where an agent (represented by a gripper) interacts with various objects like animals and plants.

Examples of interactions in the Temporal Playground, showing traces of actions alongside their corresponding spatio-temporal descriptions.

The environment features 32 object types across 5 categories. Objects can move, grow (if watered), or be shaken and grasped. To accompany this environment, the researchers developed a synthetic grammar generating 2,672 unique descriptions. These are categorized into:

  • Basic: Simple attributes.
  • Spatial: Relative positions.
  • Temporal: Past actions.
  • Spatio-Temporal: The most complex category, combining both spatial and temporal elements.

Over 56,000 trajectories (video-like histories of an agent's actions and object states) were generated to train and test the models.

Methods: Learning a Truth Function

The researchers frame the problem as learning a truth function. Given a "trace" of the agent's history and a synthetic sentence, the model must predict if the sentence is true or false. They compared several Transformer-based architectures to see how different "inductive biases" (architectural shortcuts) affect learning:

  • Unstructured Transformer (UT): Processes all object states and words across all time steps simultaneously with no specific hierarchy.
  • Spatial-First Transformer (SFT): First aggregates information across all objects at each time step, then processes the sequence over time.
  • Temporal-First Transformer (TFT): First summarizes the history of each individual object into a single representation, then looks at how those objects relate to each other and the language.
  • Word Aggregation (-WA): A variant where the entire sentence is compressed into a single vector before interacting with the visual data.

Key Insights and Results

Object Identity is Essential

The most significant finding was the failure of the Spatial-First (SFT) model. By collapsing all objects into a single scene-vector early on, the model lost the ability to track individual object identities. This made it nearly impossible to ground predicates to specific items over time.

Temporal Summarization is Viable

Interestingly, the Temporal-First (TFT) model performed exceptionally well. This suggests that while we must keep objects separate, we can successfully summarize an object's entire history into a single "summary token" without losing too much meaning.

The "Word Aggregation" Paradox

Summarizing a sentence into one vector (the -WA approach) is common in AI, but this paper shows it is a double-edged sword. While it helps the model learn the "past tense" (like the word was), it actively hurts the model's ability to handle "systematic generalization"—the ability to understand a new combination of words it has never seen together before.

Overall Performance

The Unstructured (UT) and Temporal-First (TFT) models were the clear winners. They achieved near-perfect F1 scores (~1.0) on basic sentences and maintained high scores (~0.89) even on the most difficult spatio-temporal sentences. Standard LSTM baselines were significantly outperformed by the Transformer variants, proving that attention mechanisms are better suited for these relational tasks. However, while models handled randomly held-out sentences well, they still struggled with systematic splits (e.g., learning the concept of "left of" in the present and having to apply it to the past for the first time).

Limitations and Future Work

The authors acknowledge that their study uses a synthetic language and high-level object features (like coordinates and RGB values) rather than raw pixels. In the real world, an agent would have to deal with noisy visual data and the "object permanence" problem (objects being hidden behind others).

The next logical step is to use this "truth function" as a reward signal in Reinforcement Learning. If an agent can accurately judge whether it has fulfilled a complex spatio-temporal command, it can use that internal judgment to teach itself how to perform those actions autonomously in the future.

Summary updated

Abstract

Language is an interface to the outside world. In order for embodied agents to use it, language must be grounded in other, sensorimotor modalities. While there is an extended literature studying how machines can learn grounded language, the topic of how to learn spatio-temporal linguistic concepts is still largely uncharted. To make progress in this direction, we here introduce a novel spatio-temporal language grounding task where the goal is to learn the meaning of spatio-temporal descriptions of behavioral traces of an embodied agent. This is achieved by training a truth function that predicts if a description matches a given history of observations. The descriptions involve time-extended predicates in past and present tense as well as spatio-temporal references to objects in the scene. To study the role of architectural biases in this task, we train several models including multimodal Transformer architectures; the latter implement different attention computations between words and objects across space and time. We test models on two classes of generalization: 1) generalization to new sentences, 2) generalization to grammar primitives. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall, and that summarizing object traces in a single token has little influence on performance. We then discuss how this opens new perspectives for language-guided autonomous embodied agents.
BibTeX
@inproceedings{karch2021grounding,
  author = {Tristan Karch and Laetitia Teodorescu and Katja Hofmann and Clément Moulin-Frier and Pierre-Yves Oudeyer},
  title = {{Grounding Spatio-Temporal Language with Transformers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=ZQQqo8H1qjC},
}
Figure