TL;DR
Skill-Based Agentic DocMiner is a modular, multi-agent system designed to extract information from complex scientific figures in the highly specialized domain of Atomic Layer Deposition and Etching (ALD/E). By combining multimodal context enrichment (leveraging captions and surrounding text) with iterative prompt refinement and multi-agent verification, the system achieved top-5 finishes across figure classification, table extraction, and visual question answering tasks at the Sci-ImageMiner 2026 competition.
What Problem Does This Paper Address?
Scientific papers are packed with domain-specific figures—charts, schematics, and micrographs—that are notoriously difficult for standard vision-language models to parse accurately. In specialized fields like ALD/E, plots often share similar visual layouts but represent completely different physical processes or experimental measurements.
To unlock this data, researchers need systems that can accurately classify figure types, extract underlying data into structured tables, and answer complex reasoning questions about the visual content. This paper addresses these challenges by moving away from monolithic, single-prompt AI approaches toward a structured, multi-agent workflow that mimics how a human researcher reviews a paper.
Methods
Instead of relying on a single prompt to handle everything, the authors developed Skill-Based Agentic DocMiner, a lightweight orchestration framework built on top of Gemini 3.1 Pro and Qwen-Plus/3.5-Plus APIs.
As shown in Figure 1, the architecture relies on four modular components:
- Context Builder: Enriches the visual input by pulling in figure captions, nearby paper paragraphs, and relevant few-shot examples.
- Primary Specialist: Generates initial predictions using task-specific instructions (or "skills").
- Verifier/Reviewer: Catches formatting errors and checks semantic plausibility when outputs are uncertain.
- Coordinator: Aggregates candidate outputs and resolves any multi-agent disagreements before final submission.
The authors tailored this setup for each task. For Figure Classification (Task 1), they introduced "class prototype documents" from the training data to give the model explicit reference points for distinguishing visually similar charts. For Data Table Extraction (Task 2), they implemented a two-stage validation process that separates structural checking (Markdown syntax) from content verification. For Visual Question Answering (Task 4), a router splits questions into four distinct categories (e.g., Process-Oriented vs. Comparative/Trend) and sends them to specialized agents with tailored checklists.
Datasets
The system was evaluated on the Sci-ImageMiner 2026 benchmark dataset. This dataset focuses exclusively on scientific figures extracted from Atomic Layer Deposition and Etching (ALD/E) literature. It features 49 distinct figure classes and requires fine-grained, multimodal reasoning across classification, structured table reconstruction, and diverse question-answering categories.
Key Insights and Ideas
- Images Aren't Enough: Scientific charts are often low-resolution or feature overlapping lines that make direct visual parsing error-prone. Treating extraction as a multimodal grounding problem—where the surrounding text acts as a semantic regularizer—drastically reduces numerical hallucinations.
- Decouple Structure from Content: In table extraction, separating the validation of Markdown syntax from the verification of data plausibility makes it much easier to systematically repair errors. If a table is malformed, the system fixes the schema first before worrying about the numbers.
- Divide and Conquer for VQA: Routing questions to specialized agents with tailored reasoning habits (e.g., an agent focused purely on axis reading and trends vs. one focused on molecular structure) yields sharper, more accurate answers than a general-purpose prompt.
Key Results
The system performed exceptionally well on the official Sci-ImageMiner 2026 leaderboards, securing top positions across all three targeted tasks:
- Task 1 (Figure Classification): Ranked 4th with an accuracy of 0.75 and an F1-score of 0.76 (see Table 1).
- Task 2 (Data Table Extraction): Ranked 5th with a weighted score of 33.19 and a TEDS score of 53.72 (see Table 2).
- Task 4 (Visual Question Answering): Ranked 3rd overall with a weighted score of 0.27, showing particularly strong performance on Yes/No questions with an accuracy of 0.89 (see Table 3).
Ablation studies highlighted in the paper (Table 4) confirmed the value of their modular design. For instance, adding the Markdown structural check boosted the table extraction TEDS score from 50.60 to 53.72, while prototype-guided labeling raised the classification F1-score from 0.72 to 0.76.
Limitations and Future Work
While the agentic workflow is highly effective, reading exact numerical values from complex, multi-line charts remains a bottleneck due to visual resolution limits. The authors note that future iterations will focus on integrating tool-grounded chart parsers for precise pixel-to-data mapping. They also plan to implement uncertainty-aware verification to automatically flag ambiguous predictions and evaluate how well this skill-refinement workflow generalizes to scientific domains outside of ALD/E.
Summary updated