TL;DR
This paper introduces DocAssistant, a novel approach to document visual question answering (DVQA) that tackles the limitations of existing models in handling complex documents and reasoning. By integrating key-region reading and step-wise reasoning, DocAssistant achieves more robust and interpretable answers, outperforming larger models on challenging datasets.
What Problem Does This Paper Address?
Understanding information within documents, especially those with complex layouts, charts, and extensive text, is a crucial but challenging task for AI. Existing document visual question answering (DVQA) models often struggle to:
- Focus on relevant information: They tend to process the entire document without prioritizing key areas related to the question, leading to errors.
- Perform complex reasoning: Many models generate answers directly, lacking the step-by-step reasoning process humans use, making it difficult to verify their accuracy.
- Handle diverse document types: Models often perform well on simple documents but falter on complex layouts like posters or charts.
- Provide interpretable answers: The lack of intermediate reasoning steps makes it hard to understand how a model arrived at its answer.
This paper aims to bridge this gap by developing a model that can read, reason, and explain its answers like a human.
Methods
DocAssistant introduces several key innovations:
- Mixture-of-Modality Adaptation: This lightweight module is inserted into the visual encoder of existing models. It allows the model to dynamically focus on visual features relevant to the question's keywords without compromising the original model's performance. This helps in filtering out noisy information and improving accuracy.
- Step-wise Data Generation: To address the lack of step-wise reasoning data, the authors leverage large multimodal language models (MLLMs) to generate high-quality, step-wise question-and-answer pairs. This process involves:
- Data Generator: Using templates and few-shot learning to create triplets of (question, rationale, answer).
- Data Checker: Employing a multi-agent collaborative filtering and rule-based pipeline to filter out noisy or incorrect data generated by the MLLMs. This checker verifies both information extraction and reasoning steps.
- Efficient SLVM (Small Language Model): The generated high-quality data is used to train a smaller, efficient multimodal model (DocAssistant) that excels in both question-aware document visual context understanding and reasoning.
Datasets
The paper evaluates DocAssistant on three established DVQA datasets:
- DocVQA: Primarily consists of scanned documents with simpler questions focused on text extraction.
- InfographicVQA: Features posters with more complex layouts and questions requiring some reasoning.
- ChartQA: Includes synthetic and human-marked chart data (bar, line, pie charts) involving mathematical logic operations and complex reasoning.
The authors also generated new data to augment these datasets, specifically focusing on step-wise reasoning.
Key Insights and Ideas
- Focusing on "Key Regions" is Crucial: The Mixture-of-Modality adaptation highlights that directing visual attention to question-relevant areas significantly improves performance, especially on complex documents.
- Step-wise Reasoning is Key for Robustness: Generating and training with intermediate reasoning steps is essential for tackling complex questions and making answers interpretable. This mimics human problem-solving.
- MLLMs as Powerful Data Augmenters: The paper demonstrates the effectiveness of using MLLMs not just for model training but also for generating high-quality, specialized training data that addresses specific model limitations.
- Data Quality Matters: The rigorous data filtering process is critical for ensuring that the generated data is reliable and effectively trains the model, mitigating MLLM hallucinations.
Key Results
- DocAssistant achieved state-of-the-art or near state-of-the-art results on DocVQA, InfographicVQA, and ChartQA, often outperforming larger models.
- The Mixture-of-Modality Adaptation module led to significant performance improvements (e.g., 1.6%, 2.6%, 2.7% on the three datasets) when added to a baseline model.
- Training with the generated step-wise data improved performance by 1.3%, 5.2%, and 2.5% on InfoVQA and ChartQA, demonstrating its effectiveness for complex reasoning tasks.
- Qualitative analysis showed DocAssistant providing detailed reasoning steps and locating correct visual regions, leading to more interpretable and accurate answers compared to base models.
Limitations and Future Work
- Dataset Generalizability: The authors acknowledge that the generalizability of their constructed dataset to entirely new domains might be limited, potentially requiring dataset reconstruction.
- Backbone Limitations: While the visual encoder is enhanced, its performance is still constrained by the original backbone's ability to handle complex, multi-structured data.
- Single-Page Focus: DocAssistant is primarily effective for single-page document understanding and cannot efficiently handle multi-page documents.
- Future Directions: The paper suggests future research should focus on improving dataset generalizability, exploring more advanced backbone architectures for complex data, and extending the model's capabilities to multi-page documents.
Summary updated