Metaknowledge Extraction Based on Multi-Modal Documents

Shukan Liu, Ruilin Xu, Bo-Ying Geng, Qiao Sun, Li Duan, Yiming Liu

Abstract: The triplet-based knowledge in large-scale knowledge bases is most likely lacking in structural logic and problematic of conducting knowledge hierarchy. In this paper, we introduce the concept of metaknowledge to knowledge engineering research for the purpose of structural knowledge construction. Therefore, the Metaknowledge Extraction Framework and Document Structure Tree model are presented to extract and organize metaknowledge elements (titles, authors, abstracts, sections, paragraphs, etc.), so that it is feasible to extract the structural knowledge from multi-modal documents. Experiment results have proved the effectiveness of metaknowledge elements extraction by our framework. Meanwhile, detailed examples are given to demonstrate what exactly metaknowledge is and how to generate it. At the end of this paper, we propose and analyze the task flow of metaknowledge applications and the associations between knowledge and metaknowledge.

Introduction

In the current landscape of Artificial Intelligence, knowledge extraction has largely focused on "triplets"—simple facts represented as (Subject, Relation, Object). While these flat knowledge bases (such as DBpedia or YAGO) are invaluable, they often lack the structural logic and hierarchy that humans naturally use to organize information. When reading a complex document, humans do not perceive a mere list of facts; they see a hierarchy of titles, sections, and paragraphs that provide essential context and logical flow.

This research proposes a shift in focus toward metaknowledge: the structural "skeleton" that describes how knowledge is organized within documents. By extracting this structure, we can bridge the gap between flat data and the hierarchical associations found in laws, scientific papers, and textbooks.

The Metaknowledge Extraction Framework (MEF)

To address the limitations of single-modal extraction, the authors introduce the Metaknowledge Extraction Framework (MEF). This multi-modal system combines text analysis with visual layout detection to reconstruct document hierarchies into a logical Document Structure Tree (DST).

The Metaknowledge Extraction Framework (MEF) illustrates the pipeline from multimodal document data to the final Document Structure Tree and Metaknowledge layers.

Multi-Modal Components

The framework operates through three primary modules:

  1. Text Modal Extraction: This module utilizes a BERT + BiLSTM + CRF architecture. BERT provides deep contextual embeddings, BiLSTM captures the sequence of the text, and the Conditional Random Field (CRF) layer ensures that predicted labels (e.g., "Title" or "Section") follow a logical sequence.
  2. Image Modal Extraction: This module treats document structure as an object detection task. It uses YOLOv4 to identify bounding boxes for layout elements and PaddleOCR to recognize the text within those boxes.
  3. Alignment and Organization: A decision matrix aligns results from both modalities. The text modal is used to correct OCR errors via Levenshtein distance, and the elements are organized into the DST.
Detailed view of the Text Modal Extraction process using BERT, BiLSTM, and CRF layers to label document elements.

Experimental Results and Insights

The study utilized two primary datasets: GovDoc-CN (6,816 Chinese government document pages) and DocBank (a large-scale benchmark for layout analysis).

Key Findings

  • Modality Complementarity: The image modal performs exceptionally well on visually rich pages (like headers with logos), while the text modal excels on inner pages where linguistic cues (like "Section 1.1") are stronger.
  • Performance Gains: On the GovDoc-CN dataset, the multi-modal MEF achieved an average F1-Score of 0.9048, significantly outperforming text-only (0.8668) and image-only (0.7412) baselines.
  • State-of-the-Art (SOTA): On the DocBank dataset, the image modal extraction achieved SOTA performance on 7 out of 12 element types, including Abstract, Author, and Caption, even when compared to models like LayoutLM trained on larger samples.

Conclusion and Future Work

Metaknowledge serves as a critical bridge between raw data and structured understanding. By extracting the "skeleton" of a document first, subsequent extraction of entities and relations becomes context-aware. Future research will focus on expanding the framework to multilingual datasets and diverse document types, as well as developing representation learning methods to better encode hierarchical metaknowledge for downstream tasks like automated question answering.

Reference: Liu, S.-K., et al. (2021). "Metaknowledge Extraction Based on Multi-Modal Documents." IEEE Access, vol. 9, pp. 50050-50060.

Summary updated

Abstract

The triplet-based knowledge in large-scale knowledge bases is most likely lacking in structural logic and problematic of conducting knowledge hierarchy. In this paper, we introduce the concept of metaknowledge to knowledge engineering research for the purpose of structural knowledge construction. Therefore, the Metaknowledge Extraction Framework and Document Structure Tree model are presented to extract and organize metaknowledge elements (titles, authors, abstracts, sections, paragraphs, etc.), so that it is feasible to extract the structural knowledge from multi-modal documents. Experiment results have proved the effectiveness of metaknowledge elements extraction by our framework. Meanwhile, detailed examples are given to demonstrate what exactly metaknowledge is and how to generate it. At the end of this paper, we propose and analyze the task flow of metaknowledge applications and the associations between knowledge and metaknowledge.
BibTeX
@inproceedings{liu2021metaknowledge,
  author = {Shukan Liu and Ruilin Xu and Bo-Ying Geng and Qiao Sun and Li Duan and Yiming Liu},
  title = {{Metaknowledge Extraction Based on Multi-Modal Documents}},
  booktitle = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2021.3068728},
}
Figure