Introduction
Large Language Models (LLMs) have achieved remarkable success in text summarization, yet they frequently struggle with visual logic. Block diagrams—essential tools in engineering, business, and science—require an understanding of both high-level flow and precise local connections. Current Document AI models often fail to bridge the "local-global" gap, either missing specific connections or failing to synthesize them into a coherent narrative. Furthermore, the field suffers from a scarcity of large-scale, high-quality datasets for diagram summarization, particularly in non-English languages.
In "Unveiling the Power of Integration: Block Diagram Summarization through Local-Global Fusion," researchers Shreyanshu Bhushan, Eun-Soo Jung, and Minho Lee introduce BlockNet, a framework designed to integrate local connection data with global visual context.
The BlockNet Architecture
BlockNet employs a three-pillar architecture to ensure both factual accuracy and linguistic fluency:
- Local Information Extractor (BlockSplit): Utilizing a "divide-and-conquer" strategy, this module employs YOLOv5 for object detection and Pororo OCR for text extraction. The "BlockSplit" algorithm then maps these elements into logical triplets:
<Head> <Relation> <Tail>. - Global Information Extractor: This OCR-free transformer, based on the Donut architecture (Swin Transformer + BART), processes the entire image to capture the holistic flow and "vibe" of the diagram.
- The Integrator: An LLM (GPT-4V) serves as the reasoning engine, synthesizing the outputs from both extractors. The precise local triplets are used to refine and correct the global summary, ensuring the final output is both detailed and accurate.
Dataset Construction: BD-EnKo
To address the lack of training data, the authors developed BD-EnKo, a multilingual dataset comprising 83,394 diagrams with summaries in English and Korean. The dataset was constructed using a synthetic pipeline that ensures perfect alignment between the diagram's logic and its textual description.
The pipeline begins with GPT-3.5 generating Mermaid code, which is then rendered into diverse images. Simultaneously, GPT-3.5 generates natural language descriptions based on the source code, providing high-quality ground truth summaries.
As shown in the dataset statistics, BD-EnKo covers a wide range of diagram types, including flowcharts, sequence diagrams, and state diagrams, providing a robust foundation for training and evaluation.
Experimental Results and Insights
BlockNet establishes a new state-of-the-art in block diagram summarization. In English (BD-En), it achieved a BLEU score of 58.61, outperforming GPT-4V (50.65). The performance gap was even more pronounced in Korean (BD-Ko), where BlockNet reached 62.41 BLEU, nearly tripling GPT-4V's score of 20.72.
Key findings from the study include:
- Fusion is Critical: Ablation studies demonstrated that local connection information (triplets) is vital; its removal led to a significant drop in accuracy compared to removing global context.
- Synthetic-to-Real Transfer: Training on large-scale synthetic data allowed the model to generalize effectively to real-world and handwritten diagrams.
- Multilingual Superiority: Domain-specific fine-tuning on BD-EnKo allowed BlockNet to significantly outperform general-purpose frontier models in non-English contexts.
Conclusion
BlockNet demonstrates that integrating local structural details with global visual perception is key to mastering diagrammatic reasoning. By providing both a novel architecture and a massive multilingual dataset, this work sets a new standard for Document AI. Future research will focus on increasing visual complexity, expanding language support, and optimizing the integrator for greater efficiency.
Summary updated