Proceedings of the First Workshop on Advances in Language and Vision Research

Xin Wang, Jesse Thomason, Ronghang Hu, Xinlei Chen, Peter Anderson, Qi Wu, Asli Celikyilmaz, Jason Baldridge, William Yang WangACL 2020

TL;DR

This document presents the proceedings of the first workshop on Advances in Language and Vision Research (ALVR), bringing together researchers from NLP and Computer Vision. It highlights the growing importance of this interdisciplinary field, which is moving towards more active perception and real-world applications, and showcases recent work in areas like image-to-text translation, scene graph generation, and visual question answering.

What Problem Does This Paper Address?

The field of language and vision research sits at the intersection of Natural Language Processing (NLP) and Computer Vision (CV). Traditionally, research in this area has focused on more passive tasks like image captioning or object recognition. However, there's a clear shift towards more dynamic and interactive applications, such as agents that can understand and act upon natural language commands in visual environments, or systems that can generate questions about images. This workshop aims to foster progress in these more active and real-world-oriented language and vision tasks, bridging the gap between the NLP and CV communities and encouraging collaboration to solve complex problems.

Methods

This document is a collection of papers presented at a workshop, rather than a single research paper with a unified methodology. Therefore, the methods section will summarize the approaches used in the individual papers as presented in the Table of Contents and Workshop Program.

  • Extending ImageNet to Arabic using Arabic WordNet: This paper likely focuses on adapting image classification datasets to a new language (Arabic) by leveraging linguistic resources like WordNet. This could involve techniques for cross-lingual word embeddings, semantic mapping, or data augmentation.
  • Toward General Scene Graph: Integration of Visual Semantic Knowledge with Entity Synset Alignment: This work aims to create more comprehensive scene graphs by integrating visual and semantic knowledge. Methods might include graph neural networks, knowledge graph embeddings, and techniques for aligning visual entities with semantic concepts (synsets).
  • Visual Question Generation from Radiology Images: This paper addresses the task of generating relevant questions about medical images. This likely involves deep learning models, possibly sequence-to-sequence architectures, trained on radiology images and associated question-answer pairs or clinical reports.
  • On the role of effective and referring questions in GuessWhat?!: This research investigates the characteristics of effective questions in a visual guessing game. It likely involves analyzing dialogue data, potentially using NLP techniques to identify question properties that lead to successful object identification.
  • Latent Alignment of Procedural Concepts in Multimodal Recipes: This paper focuses on understanding procedural steps described in recipes, which involve both text and potentially visual elements (e.g., images or videos). Methods could include multimodal learning, latent variable models, and sequence alignment techniques to connect textual descriptions with underlying actions.

Datasets

The workshop features several papers that likely utilize or contribute to various datasets:

  • ImageNet: Mentioned in the context of extending it to Arabic, suggesting its use as a foundational dataset for image classification tasks.
  • VATEX: This is a newly introduced large-scale multilingual video description dataset used for the Video-guided Machine Translation (VMT) challenge. It contains over 41,250 videos and 825,000 high-quality captions in both English and Chinese, with a focus on English-Chinese translation pairs.
  • GuessWhat?!: This is a visual guessing game dataset, likely used for the paper investigating effective questions.
  • Radiology Images: The paper on Visual Question Generation from Radiology Images explicitly mentions using this type of data, implying a specialized dataset for medical imaging.
  • Multimodal Recipes: The paper on Latent Alignment of Procedural Concepts in Multimodal Recipes suggests the use of datasets containing recipes with both textual and visual components.

Key Insights and Ideas

  • The growing synergy between NLP and CV: The workshop itself signifies the increasing recognition of the need for interdisciplinary research, moving beyond siloed approaches.
  • Shift towards active and embodied AI: The focus on tasks like visual question generation and agents navigating environments indicates a move from passive understanding to active interaction with the visual world.
  • Importance of linguistic resources for cross-modal tasks: Leveraging resources like WordNet for adapting models to new languages highlights the crucial role of linguistic knowledge in bridging modalities.
  • The complexity of real-world visual understanding: Tasks like localizing objects in unseen rooms (REVERIE challenge) underscore the challenges of grounding language in dynamic, real-world scenarios.
  • The value of specialized datasets: The introduction of datasets like VATEX for specific tasks (VMT) is essential for driving progress in niche areas.

Key Results

Not discussed in the paper. As this is a proceedings document, specific quantitative results for each paper are not summarized here. The full proceedings would contain the detailed findings of each presented work.

Limitations and Future Work

Not discussed in the paper. This document serves as an overview of the workshop and its presented papers. Detailed limitations and future work for each individual paper would be found within their respective contributions in the full proceedings.

Summary updated

BibTeX
@inproceedings{wang2020proceedings,
  author = {Xin Wang and Jesse Thomason and Ronghang Hu and Xinlei Chen and Peter Anderson and Qi Wu and Asli Celikyilmaz and Jason Baldridge and William Yang Wang},
  title = {{Proceedings of the First Workshop on Advances in Language and Vision Research}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure