Robust Visual Reasoning via Language Guided Neural Module Networks

Arjun R. Akula, Varun Jampani, Soravit Changpinyo, Song-Chun ZhuNeurIPS 2021 · OpenReview

TL;DR

This paper introduces LG-Conv, a novel language-guided adaptive convolution layer that allows Neural Module Networks (NMNs) to better associate visual features with textual context. By explicitly conditioning convolutional filters on the input query, the model achieves state-of-the-art performance on visual reasoning benchmarks, showing a massive 11.6% improvement on the CLOSURE dataset and superior generalization to unseen concept combinations.

What Problem Does This Paper Address?

Visual reasoning tasks like Visual Question Answering (VQA) and Referring Expression Recognition (REF) require a model to understand complex relationships between objects in an image based on a natural language query. Neural Module Networks (NMNs) are a popular solution because they decompose these queries into executable "programs" of specialized sub-networks (modules).

However, standard NMNs suffer from a "generalization gap." They often fail when encountering novel combinations of known concepts. For example, a model trained on "blue sphere" and "yellow cube" might fail to understand "yellow sphere" because its internal modules don't effectively "listen" to the textual context while processing the image. The authors argue that existing modules are too rigid; they don't adapt their visual processing based on the specific nuances of the language query.

Methods

The core innovation is the Language-Guided Adaptive Convolution (LG-Conv) layer.

Architecture of the Language-Guided Adaptive Convolution (LG-Conv) layer.

In a standard convolution, filter weights are fixed after training. In LG-Conv, the filter weights are explicitly multiplied by a spatially varying guidance kernel ($G$). This kernel is dynamically generated based on the joint context of the image and the text query, allowing the network to adaptively co-attend over potential objects of interest.

The authors propose two ways to generate this guidance kernel:

  1. BiSAtt (Bi-Salient Attentional Guidance): Encodes text and image separately to learn adaptive weights.
  2. CoSAtt (Co-Salient Attentional Guidance): Uses "early fusion" to jointly attend over image and text inputs, identifying co-salient regions and relationships. This proved to be the more effective method.

Additionally, the authors parameterize module arguments. Instead of having separate, hard-coded modules for every possible attribute (e.g., one for filter_red, one for filter_blue), they use a single generic filter_color module that takes the text "red" or "blue" as an input argument. This reduces the number of distinct modules by 75%, making the model more efficient and easier to train.

Datasets

The model was evaluated on several rigorous benchmarks:

  • CLEVR: The standard synthetic benchmark for visual reasoning.
  • CLOSURE: A benchmark specifically designed to test systematic generalization (unseen combinations of objects and relations).
  • CLEVR-Ref+: A dataset for referring expression recognition.
  • C3-Ref+ (New): Introduced in this paper, the "Compositional and Contrastive CLEVR-Ref+" benchmark tests a model's ability to handle adversarial perturbations and novel concept compositions.

Key Insights and Ideas

  • Dynamic Filtering is Key: Simply "feeding" text into a network isn't enough. By allowing the text to directly alter the convolutional filters (the "eyes" of the module), the model can "look" for different things depending on the query context.
  • Parameterization Improves Generalization: Treating attributes (like colors or shapes) as arguments to a module rather than separate modules allows the network to learn the concept of filtering rather than just memorizing specific attribute-image pairs.
  • Early Fusion Matters: The CoSAtt encoder's success suggests that understanding the relationship between language and vision needs to happen early in the feature extraction process, not just at the final decision layer.

Key Results

Performance comparison on CLEVR and CLOSURE datasets.
  • CLOSURE Benchmark: As shown in the table above, the model achieved 88.0% accuracy with CoSAtt, a massive 11.6% leap over the previous state-of-the-art (NS-VQA at 76.4%).
  • CLEVR-Ref+: On the single-referent (S-Ref) test split, the model outperformed competitors by +9.8%.
  • Robustness: On the new C3-Ref+ benchmark, standard models saw performance drops of up to 18% when faced with novel compositions. The proposed model's performance dropped by less than 5%, demonstrating significantly higher robustness against adversarial perturbations.

Limitations and Future Work

  • Synthetic vs. Real World: The evaluation is primarily on CLEVR-based synthetic datasets. While these are excellent for testing logic and composition, real-world images have much higher visual variance and noise.
  • Program Dependency: The system still relies on a "Program Generator" (a seq2seq model) to parse the query. If the initial program parsing is incorrect, the execution engine—no matter how robust—will likely fail.
  • Kernel Complexity: The authors used a fixed parametric Gaussian for the guidance kernel. They suggest that exploring more complex, non-parametric kernels could further improve performance.

Summary updated

Abstract

Neural module networks (NMN) are a popular approach for solving multi-modal tasks such as visual question answering (VQA) and visual referring expression recognition (REF). A key limitation in prior implementations of NMN is that the neural modules do not effectively capture the association between the visual input and the relevant neighbourhood context of the textual input. This limits their generalizability. For instance, NMN fail to understand new concepts such as “yellow sphere to the left" even when it is a combination of known concepts from train data: “blue sphere", “yellow cube", and “metallic cube to the left". In this paper, we address this limitation by introducing a language-guided adaptive convolution layer (LG-Conv) into NMN, in which the filter weights of convolutions are explicitly multiplied with a spatially varying language-guided kernel. Our model allows the neural module to adaptively co-attend over potential objects of interest from the visual and textual inputs. Extensive experiments on VQA and REF tasks demonstrate the effectiveness of our approach. Additionally, we propose a new challenging out-of-distribution test split for REF task, which we call C3-Ref+, for explicitly evaluating the NMN’s ability to generalize well to adversarial perturbations and unseen combinations of known concepts. Experiments on C3-Ref+ further demonstrate the generalization capabilities of our approach.
BibTeX
@inproceedings{akula2021robust,
  author = {Arjun R. Akula and Varun Jampani and Soravit Changpinyo and Song-Chun Zhu},
  title = {{Robust Visual Reasoning via Language Guided Neural Module Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=gcjVW83kww},
}
Figure