Learning Perceptually-Grounded Semantics in The L0 Project

Terry RegierACL 1991 · DOI

Abstract

This paper introduces a connectionist approach to the acquisition of spatial semantics, focusing on how language can be grounded in visual perception. The research addresses two fundamental challenges in artificial intelligence: generalizing spatial concepts across landmarks of varying geometries and learning effectively without explicit negative evidence. By utilizing parameterized regions and weakened mutual exclusivity, the system demonstrates robust learning and generalization, mirroring the constraints of human language acquisition.

Introduction

A central problem in cognitive science is how learners—whether human or machine—acquire the meaning of spatial terms. For instance, how does one learn that a trajector (TR) is "above" a landmark (LM), regardless of whether the landmark is a small square or a wide rectangle?

A basic spatial scene where a trajector (TR) is positioned "above" a landmark (LM).

The L0 project explores this through a miniature language acquisition task, identifying two primary hurdles:

  1. Geometric Generalization: Spatial terms are relative. The region defined as "above" must adapt its shape and extent based on the landmark's geometry. A static coordinate system is insufficient for this task.
  2. The "No Negative Evidence" Problem: Children are rarely told what a word does not mean. Most machine learning models, however, rely on labeled negative data to establish semantic boundaries. The goal is to learn these boundaries using only positive examples.

Methodology: Parameterized Regions and Weakened Mutual Exclusivity

The proposed neural network architecture combines several novel techniques to ground language in visual scenes.

Parameterized Regions

To achieve geometric invariance, the network learns regions relative to five "key points" of the landmark: its center of mass and the four corners of its bounding box. By anchoring the coordinate system to these points, the definition of a relation like "above" scales and moves dynamically with the landmark. This allows the model to learn the underlying logic of "above-ness" rather than just a fixed spatial coordinate.

The system uses key points of the landmark to define spatial regions, ensuring geometric invariance across different shapes.

Bitmap "Interior Maps"

Beyond bounding boxes, the system uses a bitmap representation to handle the actual shape of objects. Receptive fields "view" the area surrounding the landmark to determine if a trajector is inside or outside its boundary, providing the necessary detail for terms like "in" and "out."

Weakened Mutual Exclusivity

To solve the lack of negative evidence, the model employs a heuristic: if a point is labeled "on," it is treated as an implicit negative for "off" or "above." However, because spatial terms often overlap (e.g., "above" is also "outside"), these implicit negatives are "weakened" during training. Their error signal is attenuated (divided by a factor of 32), allowing the model to accommodate natural semantic overlap while still learning where boundaries lie.

Results and Discussion

The system successfully learned a vocabulary of eight core English spatial prepositions: above, below, on, off, in, out, left, right.

Successful Generalization

Most impressively, the model demonstrated robust generalization. After being trained exclusively on rectangular landmarks, it correctly applied spatial terms to a novel triangular landmark—a shape it had never encountered during training. This confirms that the parameterized region approach effectively captures the relative nature of spatial language.

Qualitative Accuracy

Without explicit negative evidence, standard models often over-generalize, potentially failing to distinguish the interior of an object from the region "outside" it. The "weakened" negative evidence approach correctly identified these boundaries, excluding the landmark's interior from the "outside" or "above" regions. This suggests that weak negative signals are sufficient for learning precise semantic boundaries.

Conclusion

The L0 project provides a compelling framework for understanding how perceptually-grounded semantics can be acquired. By mimicking the environmental constraints of human development—such as the absence of negative evidence—the model demonstrates that geometric constraints and simple learning heuristics are powerful tools for language acquisition. Future work will extend this model to handle 3D spaces, complex trajector shapes, and cross-linguistic semantic variations.

Summary updated

BibTeX
@inproceedings{regier1991learning,
  author = {Terry Regier},
  title = {{Learning Perceptually-Grounded Semantics in The L0 Project}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {1991},
  doi = {10.3115/981344.981362},
}
Figure