Metaknowledge Extraction Based on Multi-Modal Documents

Shukan Liu, Ruilin Xu, Geng Bo-ying, Qiao Sun, Li Duan, Yiming Liu

Abstract

The triple-based knowledge in large-scale knowledge bases is most likely lacking in structural logic and problematic of conducting knowledge hierarchy. In this paper, we introduce the concept of metaknowledge to knowledge engineering research for the purpose of structural knowledge construction. Therefore, the Metaknowledge Extraction Framework and Document Structure Tree model are presented to extract and organize metaknowledge elements (titles, authors, abstracts, sections, paragraphs, etc.), so that it is feasible to extract the structural knowledge from multi-modal documents. Experiment results have proved the effectiveness of metaknowledge elements extraction by our framework. Meanwhile, detailed examples are given to demonstrate what exactly metaknowledge is and how to generate it. At the end of this paper, we propose and analyze the task flow of metaknowledge applications and the associations between knowledge and metaknowledge.

Introduction

In the current landscape of Knowledge Graphs, information is primarily stored as entity-relation triples (e.g., "Paris is the capital of France"). While effective for simple fact retrieval, this "flat" structure fails to capture the inherent logical hierarchy of human knowledge. When reading complex documents like research papers or government reports, the structural layout—titles, sections, and paragraphs—provides a critical logical flow that triples miss.

This paper addresses the challenge of structural knowledge extraction by introducing "metaknowledge": knowledge about how information is organized. By identifying the "scaffolding" of a document, we can move beyond isolated facts toward a more intuitive, hierarchical understanding of information.

The Metaknowledge Extraction Framework (MEF)

To extract these structural elements, the researchers propose the Metaknowledge Extraction Framework (MEF). This framework is designed to process multi-modal document data by leveraging both textual content and visual layout cues.

The Metaknowledge Extraction Framework (MEF) architecture, integrating text and image modalities.

The MEF operates through three primary stages:

  1. Text Modal Extraction: Utilizing a BERT + BiLSTM + CRF stack, the system performs Named Entity Recognition (NER) to identify textual strings that likely represent specific metaknowledge elements, such as titles or addressees.
  2. Image Modal Extraction: Simultaneously, the system employs YOLOv4 and PaddleOCR to analyze the document's visual layout. This allows the model to "see" bounding boxes, recognizing that a large, centered block of text at the top of a page is likely a title based on its spatial positioning.
  3. Alignment and Organization: The framework aligns the results from both modalities using Levenshtein distance to correct OCR errors. These elements are then organized into a Document Structure Tree (DST), a hierarchical model that ensures the extracted components form a coherent skeleton of the original document.

Experimental Results

The framework was evaluated using two datasets: GovDoc-CN, a newly introduced dataset of 6,816 annotated Chinese government document pages, and DocBank, a large-scale open-source dataset.

The MEF demonstrated significant improvements over single-modal approaches. On the GovDoc-CN dataset, the framework achieved an average Micro-F1 score of 0.9048, representing a 3.8% improvement over text-only models and a 16.36% improvement over image-only models. The model proved exceptionally accurate in identifying administrative elements like the "Sign of issuing authority" (0.9999 F1) and "Addressee" (0.9905 F1).

Visual comparison of ground truth vs. model predictions on Chinese government documents.

Conclusion and Future Work

By integrating visual layout data with textual analysis, the MEF successfully bridges the gap between flat data and structured knowledge. While the current framework shows high precision in administrative contexts, future work will focus on expanding its capabilities to more complex, multilingual document types and developing "Metaknowledge Representation Learning" to enable AI systems to reason directly using these hierarchical structures.

Summary updated

Abstract

The triple-based knowledge in large-scale knowledge bases is most likely lacking in structural logic and problematic of conducting knowledge hierarchy. In this paper, we introduce the concept of metaknowledge to knowledge engineering research for the purpose of structural knowledge construction. Therefore, the Metaknowledge Extraction Framework and Document Structure Tree model are presented to extract and organize metaknowledge elements (titles, authors, abstracts, sections, paragraphs, etc.), so that it is feasible to extract the structural knowledge from multi-modal documents. Experiment results have proved the effectiveness of metaknowledge elements extraction by our framework. Meanwhile, detailed examples are given to demonstrate what exactly metaknowledge is and how to generate it. At the end of this paper, we propose and analyze the task flow of metaknowledge applications and the associations between knowledge and metaknowledge.
BibTeX
@inproceedings{liu2021metaknowledge,
  author = {Shukan Liu and Ruilin Xu and Geng Bo-ying and Qiao Sun and Li Duan and Yiming Liu},
  title = {{Metaknowledge Extraction Based on Multi-Modal Documents}},
  booktitle = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2021.3068728},
}
Figure