TL;DR
DocFS transforms complex, multi-page documents into hierarchical file systems, allowing AI agents to navigate sections and access multimodal content using standard tools like ls, grep, and read_file. This approach achieves state-of-the-art performance on long-document benchmarks while being up to 10.3x faster and using 10.1x fewer tokens than previous top-tier methods.
What Problem Does This Paper Address?
Multimodal Document Question Answering (DocQA) is a significant challenge for Vision Language Model (VLM) agents. Real-world documents—like financial reports or academic papers—are long, hierarchical, and contain information scattered across text, tables, figures, and equations.
Current approaches generally fall into two camps:
- Flat Organization: Treating the document as a sequence of pages. This loses the structural context (e.g., knowing a figure belongs to "Section 3.2").
- Custom Structured Representations: Building complex graphs or trees. While these capture structure, they often lack intuitive, standardized tools for an agent to actually explore that structure efficiently.
The authors identify a gap: we need an environment that not only represents document structure but also provides a set of "aligned tools" that allow an agent to navigate, search, and verify information just like a human researcher would.
Methods
The core innovation is DocFS, which treats a document as a virtual operating system environment. The process follows three main stages:
- Document Preprocessing: Raw PDFs are parsed into multimodal elements. Crucially, the authors use an LLM to reconstruct the section hierarchy (fixing "flattened" headers) and a VLM to correct errors in parsed equations and tables.
- File System Construction: The document is reorganized into a directory tree.
- Directories mirror the section hierarchy.
- Files store content in native formats (Markdown for text, CSV for tables, JPG for images).
- Metadata Files:
index.mdfiles provide navigation entry points, whilerelation.jsonfiles explicitly link text to relevant figures or tables. - Image Descriptions: VLMs generate textual summaries of visual elements, making them searchable via keyword search.
- Agentic Reasoning: Instead of a specialized API, the agent uses standard file system tools:
ls: To see what sections and assets are available.grep: For global keyword search across the entire "document-drive."read_file: To examine specific content.
glob: To collect files of a specific type (e.g., all figures).
Datasets
The authors evaluated DocFS on two rigorous benchmarks:
- MMLongBench-Doc: A dataset of lengthy, multi-domain documents requiring models to integrate evidence across different regions and modalities.
- FinRAGBench-V: A benchmark focused on financial reports with dense, complex layouts and heavy reliance on tables and charts.
Key Insights and Ideas
- The File System Metaphor Works: File systems are a mature solution for organizing hierarchical data. By mapping documents to this structure, agents can use well-understood logic (navigation and search) to solve QA tasks.
- Tool-Structure Alignment: The "aha moment" is that an agent's effectiveness depends on its tools. A structured document is useless if the agent can't "walk" through it. Standard OS tools provide a flexible, robust interface for this exploration.
- Searchable Visuals: By generating textual descriptions for images and storing them as searchable files, DocFS allows agents to "find" a figure via
grepbefore "looking" at it with a VLM. - Efficiency through Localization: Because the agent can navigate directly to a relevant "folder" (section), it avoids processing irrelevant pages, leading to massive savings in time and compute costs.
Key Results
DocFS set new state-of-the-art records across both benchmarks using backbone models like Qwen3.5-Plus and DeepSeek-V4.1-Flash.
- Accuracy: On MMLongBench-Doc, DocFS achieved 71.6% accuracy, outperforming the strongest training-free baseline (DocLens) by 2.9 percentage points. On FinRAGBench-V, it reached 82.3%, a 4.3-point lead.
- Speed and Cost: The efficiency gains were dramatic. As shown in the table below, DocFS is significantly faster and more token-efficient. For instance, using DeepSeek-V4.1-Flash on financial reports, DocFS was 10.3x faster and used 10.1x fewer tokens than DocLens.
Ablation studies confirmed that grep (keyword search) and ls (directory listing) are the most critical tools; removing grep caused the largest performance drop, as agents were forced to "guess" which files contained the answers.
Limitations and Future Work
While DocFS is highly effective, the authors note several areas for growth:
- Adaptive Organization: The current hierarchy is fixed during preprocessing. Future versions could allow agents to reorganize the "file system" dynamically based on the specific question.
- Complex Cross-Modal Reasoning: While
relation.jsonhelps, navigating extremely dense associations between hundreds of entities across modalities remains a challenge. - Cost-Reliability Balance: There is still a trade-off between how much an agent explores (cost) and the final answer's accuracy. Optimizing this "stopping criteria" is a promising research direction.
Summary updated