[DocMiner] at Sci-ImageMiner 2026 (Tasks 1, 2, and 4): Skill-Based Agentic DocMiner

Shutao Gong, Jing HuangLLM、VLM\Agent 2026 · OpenReview

TL;DR

Skill-Based Agentic DocMiner is a modular, multi-agent system designed to extract information from complex scientific figures in the highly specialized domain of Atomic Layer Deposition and Etching (ALD/E). By combining multimodal context enrichment (leveraging captions and surrounding text) with iterative prompt refinement and multi-agent verification, the system achieved top-5 finishes across figure classification, table extraction, and visual question answering tasks at the Sci-ImageMiner 2026 competition.

What Problem Does This Paper Address?

Scientific papers are packed with domain-specific figures—charts, schematics, and micrographs—that are notoriously difficult for standard vision-language models to parse accurately. In specialized fields like ALD/E, plots often share similar visual layouts but represent completely different physical processes or experimental measurements.

To unlock this data, researchers need systems that can accurately classify figure types, extract underlying data into structured tables, and answer complex reasoning questions about the visual content. This paper addresses these challenges by moving away from monolithic, single-prompt AI approaches toward a structured, multi-agent workflow that mimics how a human researcher reviews a paper.

Methods

Instead of relying on a single prompt to handle everything, the authors developed Skill-Based Agentic DocMiner, a lightweight orchestration framework built on top of Gemini 3.1 Pro and Qwen-Plus/3.5-Plus APIs.

Technical System Architecture of Skill-Based Agentic DocMiner

As shown in Figure 1, the architecture relies on four modular components:

  1. Context Builder: Enriches the visual input by pulling in figure captions, nearby paper paragraphs, and relevant few-shot examples.
  2. Primary Specialist: Generates initial predictions using task-specific instructions (or "skills").
  3. Verifier/Reviewer: Catches formatting errors and checks semantic plausibility when outputs are uncertain.
  4. Coordinator: Aggregates candidate outputs and resolves any multi-agent disagreements before final submission.

The authors tailored this setup for each task. For Figure Classification (Task 1), they introduced "class prototype documents" from the training data to give the model explicit reference points for distinguishing visually similar charts. For Data Table Extraction (Task 2), they implemented a two-stage validation process that separates structural checking (Markdown syntax) from content verification. For Visual Question Answering (Task 4), a router splits questions into four distinct categories (e.g., Process-Oriented vs. Comparative/Trend) and sends them to specialized agents with tailored checklists.

Datasets

The system was evaluated on the Sci-ImageMiner 2026 benchmark dataset. This dataset focuses exclusively on scientific figures extracted from Atomic Layer Deposition and Etching (ALD/E) literature. It features 49 distinct figure classes and requires fine-grained, multimodal reasoning across classification, structured table reconstruction, and diverse question-answering categories.

Key Insights and Ideas

  • Images Aren't Enough: Scientific charts are often low-resolution or feature overlapping lines that make direct visual parsing error-prone. Treating extraction as a multimodal grounding problem—where the surrounding text acts as a semantic regularizer—drastically reduces numerical hallucinations.
  • Decouple Structure from Content: In table extraction, separating the validation of Markdown syntax from the verification of data plausibility makes it much easier to systematically repair errors. If a table is malformed, the system fixes the schema first before worrying about the numbers.
  • Divide and Conquer for VQA: Routing questions to specialized agents with tailored reasoning habits (e.g., an agent focused purely on axis reading and trends vs. one focused on molecular structure) yields sharper, more accurate answers than a general-purpose prompt.

Key Results

The system performed exceptionally well on the official Sci-ImageMiner 2026 leaderboards, securing top positions across all three targeted tasks:

  • Task 1 (Figure Classification): Ranked 4th with an accuracy of 0.75 and an F1-score of 0.76 (see Table 1).
  • Task 2 (Data Table Extraction): Ranked 5th with a weighted score of 33.19 and a TEDS score of 53.72 (see Table 2).
  • Task 4 (Visual Question Answering): Ranked 3rd overall with a weighted score of 0.27, showing particularly strong performance on Yes/No questions with an accuracy of 0.89 (see Table 3).
Task 1 Figure Classification Leaderboard
Task 2 Data Extraction Leaderboard
Table 3. Task 4 VQA Leaderboard with Answer-Type Metrics.

Ablation studies highlighted in the paper (Table 4) confirmed the value of their modular design. For instance, adding the Markdown structural check boosted the table extraction TEDS score from 50.60 to 53.72, while prototype-guided labeling raised the classification F1-score from 0.72 to 0.76.

Table 4. Component-Level Ablations Used During Development.

Limitations and Future Work

While the agentic workflow is highly effective, reading exact numerical values from complex, multi-line charts remains a bottleneck due to visual resolution limits. The authors note that future iterations will focus on integrating tool-grounded chart parsers for precise pixel-to-data mapping. They also plan to implement uncertainty-aware verification to automatically flag ambiguous predictions and evaluate how well this skill-refinement workflow generalizes to scientific domains outside of ALD/E.

Summary updated

Abstract

We present Skill-Based Agentic DocMiner, our submission to Sci-ImageMiner 2026 for Task 1 (figure classification), Task 2 (data table extraction), and Task 4 (visual question answering). Our system follows a unified agentic workflow with three recurring components: multimodal context enrichment from captions and paper text, iterative skill refinement using development-set feedback, and collaborative verification by specialist reviewers. For classification, we use class prototype documents to distinguish visually similar figure types. For table extraction, we combine chart parsing with Markdown structure validation and content checking. For VQA, we route each question to a specialist agent aligned with the official reasoning categories. On the official leaderboard, our submission ranked fourth on classification, fifth on table extraction, and third on VQA. The framework is simple, modular, and well matched to structured scientific reasoning. Our code is publicly available at https://github.com/1gst/Sci-ImageMiner.
BibTeX
@misc{gong2026docminer,
  author = {Shutao Gong and Jing Huang},
  title = {{[DocMiner] at Sci-ImageMiner 2026 (Tasks 1, 2, and 4): Skill-Based Agentic DocMiner}},
  year = {2026},
  url = {https://openreview.net/forum?id=iHgg8YKcet},
  howpublished = {LLM、VLM\textbackslash\{\}Agent},
}
Figure