TL;DR
This paper introduces FigEx, a novel vision-language model that automatically separates complex scientific figures into their constituent subfigures and aligns each with its corresponding caption segment. This is crucial for downstream tasks like visual question answering and image captioning, and the authors also release BioSci-Fig, a large, curated dataset to advance research in this area.
What Problem Does This Paper Address?
Scientific papers often contain "compound figures" – images composed of multiple subfigures, each with its own specific caption. Current methods struggle to accurately separate these subfigures and link them to their correct captions. This limitation hinders a deeper understanding of scientific content, as treating a compound figure as a single unit forces all subfigures to share one caption, leading to loss of specific information. This is particularly problematic in fields like biomedicine, where a majority of figures are compound.
Methods
FigEx is a vision-language model that leverages a multimodal approach. It first uses a vision-language model (VLM) to process the entire figure and its overall caption, generating VLM-guided features and initial subcaption information. These features are then fed into a vision transformer (DeiT) to predict bounding boxes for each subfigure. A key innovation is the injection of a special <DET> token, which acts as a bridge between the VLM and the detection module, allowing the model to learn from both visual and textual cues. An attention mechanism further refines the process by highlighting relevant caption snippets for each panel. The model is trained in a three-stage process: initial LoRA adaptation for the language decoder, detection head training for the vision encoder and MLP, and finally, joint fine-tuning of all components.
Datasets
The authors introduce BioSci-Fig, a new dataset comprising 7,174 compound figures with meticulously annotated subfigure bounding boxes and aligned subcaptions. This dataset was created to address the limitations of existing benchmarks, which are often small, restricted by licenses, or limited to specific scientific fields. They also utilize the MedICaT dataset, which contains 2,089 human-verified compound figures with annotated captions and bounding boxes.
Key Insights and Ideas
- Vision and Language Synergy is Key: Combining visual features with language understanding is crucial for accurately separating subfigures and their captions, especially for complex or rare cases.
- A Dedicated Detection Token: The
<DET>token acts as a critical bridge, enabling the model to learn from both visual and textual information for improved detection. - Dataset Size Matters: While smaller datasets might favor specialized models, larger, more diverse datasets like BioSci-Fig allow multimodal models like FigEx to significantly outperform pure vision or language models.
- Caption Complexity Drives Model Strength: FigEx shows particular strength in handling longer and more complex captions, suggesting its utility in detailed scientific figure analysis.
Key Results
- On the MedICaT dataset, FigEx-7B achieved an AP of 0.203, surpassing Grounding DINO by a notable margin.
- On the larger BioSci-Fig dataset, FigEx-7B recorded an AP of 0.753, outperforming YOLO111 by 0.108 and showing significant gains in long-tail metrics.
- For caption separation, FigEx-7B achieved a BLEU score of 0.722 on BioSci-Fig, nearly tripling the performance of the best baseline (Llama-2-13B).
- Ablation studies showed that combining VLM-guided features with textual cues provided complementary gains, leading to better overall performance.
Limitations and Future Work
- Generalization: The current work is confined to biomedical figures. The model's ability to generalize to other domains, such as engineering diagrams, remains unverified.
- Open-Ended Tasks: The current focus is on subfigure detection and caption separation. The paper does not assess performance on more open-ended multimodal tasks like visual question answering or iterative subcaption refinement.
- Pre-training Data Exposure: The authors note that public checkpoints may have been exposed to some evaluation samples during pre-training, which could slightly inflate baseline results.
- Future Directions: The authors suggest further investigating how caption length and complexity influence the advantages of vision-language integration and exploring targeted augmentation for rare classes.
Summary updated