Robust Natural Language Processing - Combining Reasoning, Cognitive Semantics, and Construction Grammar for Spatial Language

Michael Spranger, Jakob Suchan, Mehul BhattIJCAI 2016

Introduction

In real-world robotics, natural language processing (NLP) often fails due to the inherent "messiness" of the environment. Visual sensors jitter, objects are occluded, and human speech is frequently ungrammatical or incomplete. Most existing systems are brittle; a single missing word or a momentary loss of visual tracking can lead to total system failure. This paper presents a robust framework designed for "graceful degradation," allowing robots to understand and generate spatial language even when faced with significant sensory and linguistic noise.

System Architecture

The proposed system integrates three distinct frameworks to bridge the gap between raw sensory data and high-level communication.

The integrated pipeline for speaker and listener agents, showing the flow from perception to utterance.
  1. Qualitative Spatial Reasoning (CLP(QS)): Instead of relying on precise numerical coordinates, the system abstracts the world into qualitative relations. It utilizes the RCC8 calculus for topology (e.g., "touching," "inside") and LR calculus for orientation. Dynamic events, such as "moving across," are defined as sequences of these qualitative states using Allen’s interval algebra.
  2. Cognitive Semantics (IRL): Incremental Recruitment Language (IRL) treats the meaning of a phrase as a "program" of cognitive operations. The robot attempts to assemble a program (e.g., filtering objects by color) that aligns with its current visual context.
  3. Fluid Construction Grammar (FCG): FCG provides a flexible, bidirectional mapping between meanings and words. Unlike rigid parsers, it is designed to extract partial meanings from ungrammatical or incomplete sentences by applying as many rules as possible.

Experimental Setup and Datasets

The system was evaluated using a dataset of 200 pre-recorded spatial scenes involving two humanoid robots interacting with various objects.

An example of a spatial scene where a robot tracks an object's trajectory relative to a box and other landmarks.

From these scenes, 10,000 interaction trials were generated. In each trial, robots took turns as "speaker" and "listener." The resulting utterances were verified for correctness by human speakers to ensure a reliable ground truth for evaluation.

Key Insights: Robustness through Abstraction

A core finding is that qualitative abstraction acts as a shield against sensor noise. By focusing on states like "inside" rather than specific coordinates, the system remains stable despite pixel-level jitter. Furthermore, the reasoning engine can "fill the gaps" in visual data. If a robot misses frames where a block is in the middle of a region but sees it enter and exit, it can logically infer the "across" movement.

The "Best-Fit" listener model also contributes to robustness. Because the listener uses its own visual context to interpret incoming language, it can repair broken syntax. For instance, hearing "block red region across" allows the listener to look at the scene and correctly identify the intended action based on the available visual evidence.

Results and Performance

The system demonstrated remarkable resilience:

  • Visual Robustness: Performance remained stable even with a 50% random drop in visual frames. Significant degradation only occurred after 75% of the data was lost.
  • Linguistic Robustness: Dropping or permuting words often had minimal impact. In some cases, dropping a word actually improved success rates (from 0.78 to 0.89) by reducing linguistic ambiguity and forcing a greater reliance on accurate spatial reasoning.
  • Reasoning Impact: Enabling predictive reasoning about subsequent events boosted communicative success by approximately 10%.

Conclusion and Future Work

While the system excels at handling noise in simple spatial instructions, future work is needed to scale this "fluid" approach to complex, multi-clause sentences. Additionally, real-time testing with human subjects is necessary to validate the system's robustness against the unpredictable nature of natural human speech and the computational overhead of heuristic searches in high-stakes environments.

Summary updated

BibTeX
@inproceedings{spranger2016robust,
  author = {Michael Spranger and Jakob Suchan and Mehul Bhatt},
  title = {{Robust Natural Language Processing - Combining Reasoning, Cognitive Semantics, and Construction Grammar for Spatial Language}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure