TL;DR
This paper introduces MTXNet, a novel end-to-end trainable multimodal architecture for generating both visual and textual explanations for TextVQA (Visual Question Answering that relies on text in images). They also introduce TextVQA-X, a new dataset of human-annotated multimodal explanations, addressing the need for more interpretable AI systems that can justify their decisions.
What Problem Does This Paper Address?
Deep learning models, particularly in tasks like Visual Question Answering (VQA), are often "black boxes." While they can achieve high accuracy, they don't inherently explain why they arrive at a particular answer. This lack of transparency is problematic for debugging, building trust, and complying with regulations like GDPR's "right to explanation." Existing explanation methods are often post-hoc (applied after the model has made a decision) and may not fully leverage all available information, especially the text present within images, which is crucial for tasks like TextVQA. The authors aim to bridge this gap by developing a system that generates explanations that are an integral part of the model's design and directly incorporate visual and textual evidence from the image.
Methods
The core of the paper is the MTXNet (Multimodal Text-in-Image Explanation Network). This architecture is designed for end-to-end multitask training, encompassing three key tasks:
- Answer Prediction: The primary goal of VQA.
- Textual Explanation Generation: Producing natural language justifications.
- Visual Explanation via Semantic Segmentation: Highlighting relevant image regions.
MTXNet comprises two main components:
- Graph Attention Network (GAT): To model the spatial relationships between objects and OCR (Optical Character Recognition) tokens in the image. This helps the model understand how different visual elements relate to each other and to the question.
- Multimodal Transformer (MMT): This component processes information from three modalities: question words, visual objects (extracted using Faster R-CNN), and OCR tokens. Crucially, it also incorporates embeddings for the textual explanations themselves, allowing for a feedback loop.
The training process is phased, randomly switching between predicting the answer first, then the explanation, or vice-versa, or doing them independently. This encourages cross-modal feedback. The overall loss function uses a multitask learning approach with homoscedastic uncertainty to dynamically weigh the contributions of the three tasks.
Datasets
- TextVQA Dataset: The authors use a subset of this existing dataset as the basis for their new benchmark.
- TextVQA-X Dataset: This is a novel dataset introduced by the authors. It contains human-annotated multimodal explanations for TextVQA tasks. Key features include:
- Multi-reference textual explanations: Each question-image pair has an average of 3.71 textual explanations, reflecting human diversity in justification.
- Ground truth segmentation maps: For visual explanations.
- Focus on text in the image: Unlike VQA-X, TextVQA-X is specifically designed for tasks requiring text understanding.
Key Insights and Ideas
- Multimodal Explanations are Mutually Beneficial: The study shows that jointly optimizing for visual and textual explanations significantly improves both the answer prediction accuracy and the quality of the explanations themselves. This highlights that these modalities aren't just additive but actively inform each other.
- Text in Images is Crucial for Explanations: The TextVQA task, by its nature, requires understanding text within images. The MTXNet's ability to leverage this text for explanations is a key differentiator and leads to more grounded justifications.
- End-to-End Trainable Explanations: Integrating explanation generation directly into the model's training objective, rather than as a post-hoc step, leads to more faithful and useful explanations that are tightly coupled with the model's decision-making process.
- The Value of Multi-Reference Data: Using multiple human-annotated explanations during training significantly boosts the performance of textual explanation generation, underscoring the subjective nature of explanations and the benefit of diverse ground truth.
Key Results
- Improved Answer Accuracy: MTXNet achieved an answer accuracy of 36.27% on the TextVQA-X test set, a slight improvement over the baseline M4C model (35.23%), demonstrating that explanations complement performance.
- Significant Gains in Explanation Quality:
- Excluding visual explanations led to a drop of up to 7% in CIDEr scores for textual explanations.
- Excluding textual explanations resulted in a 2% drop in IoU for visual explanations.
- Multi-reference Training Boost: Training with multi-references led to a substantial increase of up to 25% in CIDEr scores for text generation, with consistent improvements across other metrics.
- Qualitative Success: The model can accurately answer questions and provide relevant visual and textual justifications. Failure cases often stem from OCR errors or incorrect visual localization, which the explanations can help diagnose.
Limitations and Future Work
- Limitations:
- The paper acknowledges that some textual explanations can be "generic and dull."
- Failure cases can arise from inaccurate OCR or incorrect visual localization, impacting the explanation's faithfulness.
- The current work focuses on specific TextVQA tasks; broader applicability needs further investigation.
- Future Work:
- The authors suggest incorporating visual features more deeply into the transformer architecture.
- Developing more fine-grained evaluation methodologies for multimodal explanations, considering different stakeholder needs.
- Exploring applications in e-commerce for automated compliance checks and product detail validation.
Summary updated