Multi-modal Discriminative Model for Vision-and-Language Navigation

Haoshuo Huang, Vihan Jain, Harsh Mehta, Jason Baldridge, Eugene Ie

Abstract

Vision-and-Language Navigation (VLN) is a natural language grounding task where agents must interpret natural language instructions in the context of visual scenes in a dynamic environment to achieve prescribed navigation goals. Successful agents must have the ability to parse natural language of varying linguistic styles, ground them in potentially unfamiliar scenes, and plan with ambiguous environmental feedback. Generalization ability is often limited by the scarcity of human-annotated data, as paired vision-language sequence data is expensive to collect. We develop a discriminator that evaluates how well an instruction explains a given path using multi-modal alignment. Our study reveals that only a small fraction of high-quality augmented data is useful for training VLN agents. We also demonstrate that a VLN agent warm-started with pre-trained components from the discriminator outperforms benchmark success rates by 10% relative measure on previously unseen environments.

Introduction

In Vision-and-Language Navigation (VLN), agents follow instructions like "Walk past the kitchen and stop at the rug" through 3D environments. The primary bottleneck in this field is data: human-annotated instruction-path pairs are difficult to scale. While "speaker models" can synthesize massive amounts of augmented data, these synthetic instructions are often noisy—garbled, nonsensical, or poorly aligned with the visual path. Training on such low-quality data can hinder an agent's ability to generalize to new environments. This research addresses how to identify high-quality synthetic data and leverage that knowledge to build more robust navigators.

Methodology

The Discriminator Model

The proposed discriminator uses a "two-tower" architecture to evaluate the compatibility of an instruction and a visual path:

  1. Language Tower: A bi-directional LSTM that encodes the instruction text.
  2. Visual Tower: A bi-directional LSTM that encodes the sequence of panoramic images along a path.
The Discriminator Model architecture featuring separate language and visual encoders connected by an alignment layer.

The core innovation is the Alignment Layer. Rather than relying solely on final LSTM states, the model computes a similarity matrix between every word in the instruction and every visual step in the path. Using a "softmin" operation, it identifies the best and worst alignments, forcing the model to learn specific correspondences (e.g., matching the word "bedroom" to visual features of a bedroom).

Training and Datasets

The model was trained using Negative Sampling on the Room-to-Room (R2R) dataset, which contains 21,567 human-annotated instructions. By intentionally corrupting real paths—reversing order, shuffling steps, or selecting random paths—the discriminator learned to assign low scores to "fake" pairs and high scores to authentic human pairs. This trained discriminator was then used to rank the Fried-Augmented dataset, a large-scale synthetic collection.

Results and Discussion

Quality Over Quantity: The 1% Rule

A critical finding of this study is that data quality is more important than volume. Using only the top 1% of augmented data, as ranked by the discriminator, yielded performance gains nearly identical to using the entire noisy dataset. This suggests that a small amount of high-fidelity data is significantly more valuable for generalization than a large volume of low-quality samples.

Cumulative distribution of alignment scores, showing that synthetic data (data_augmentation) generally scores lower than real human-annotated data (val_seen).

Performance Gains

The discriminator-enhanced agent achieved a 39.1% success rate on unseen environments, a significant improvement over the 35.5% benchmark. Additionally, the model reduced the average navigation error from 6.6 meters to 5.9 meters. By "warm-starting" a navigation agent with the pre-trained encoders from the discriminator, the agent begins with a superior foundational understanding of language-vision grounding.

Implicit Grounding

Visualizing the alignment matrix confirmed that the model learned to ground language implicitly. For instance, the phrase "exit the door" correctly aligned with visual frames containing doors, despite the absence of explicit object detection labels during training.

Conclusion

While the model shows strong generalization in unseen environments, there is a slight performance trade-off in previously seen environments when using discriminator pre-training. Future work may involve using the discriminator as a real-time reward signal in a Reinforcement Learning framework, potentially pushing both navigators and speaker models toward higher-quality, more human-like performance.

Summary updated

Abstract

Vision-and-Language Navigation (VLN) is a natural language grounding task where agents have to interpret natural language instructions in the context of visual scenes in a dynamic environment to achieve prescribed navigation goals. Successful agents must have the ability to parse natural language of varying linguistic styles, ground them in potentially unfamiliar scenes, plan and react with ambiguous environmental feedback. Generalization ability is limited by the amount of human annotated data. In particular, \emph{paired} vision-language sequence data is expensive to collect. We develop a discriminator that evaluates how well an instruction explains a given path in VLN task using multi-modal alignment. Our study reveals that only a small fraction of the high-quality augmented data from \citet{Fried:2018:Speaker}, as scored by our discriminator, is useful for training VLN agents with similar performance on previously unseen environments. We also show that a VLN agent warm-started with pre-trained components from the discriminator outperforms the benchmark success rates of 35.5 by 10\% relative measure on previously unseen environments.
BibTeX
@inproceedings{huang2019multi,
  author = {Haoshuo Huang and Vihan Jain and Harsh Mehta and Jason Baldridge and Eugene Ie},
  title = {{Multi-modal Discriminative Model for Vision-and-Language Navigation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure