TL;DR
This paper introduces a novel document classification method for biomedical texts that leverages a "literature graph" incorporating bibliographic and entity information. By representing documents and their relationships within this graph, the model achieves state-of-the-art performance on challenging biomedical classification tasks, outperforming existing methods that rely solely on textual content.
What Problem Does This Paper Address?
Large language models like BERT have significantly advanced document classification. However, they often struggle with documents that require understanding beyond the immediate text. This includes crucial contextual information like author affiliations, publication venues, citation networks, and the relationships between medical entities. These external knowledge sources are vital for accurate classification, especially in specialized domains like biomedicine, where subtle distinctions and complex relationships are common. Existing methods that attempt to incorporate external information often do so in a limited way, focusing on only one or two types of relationships (e.g., word co-occurrence or citations). This paper aims to bridge this gap by proposing a unified approach that simultaneously considers multiple types of bibliographic and entity information within a structured literature graph.
Methods
The core of the proposed method lies in constructing and utilizing a "literature graph." This graph represents biomedical documents and their associated information as nodes and relationships as edges.
- Literature Graph Construction: The graph includes nodes for papers, authors, publication years, journals, and biomedical entities (e.g., MeSH terms). Edges connect these nodes based on bibliographic information (citations, authorship, publication details) and entity relationships (hypernymy, supplementary concepts). Textual information, such as paper titles/abstracts and entity descriptions, is associated with the respective nodes.
- Node Initialization: Both paper and entity nodes are initialized using BERT embeddings derived from their associated text. This injects textual understanding into the graph structure from the outset.
- Representation Learning: To capture the complex relationships within the literature graph, the authors employ TransE, a knowledge graph embedding technique. TransE learns vector representations for nodes and relations, aiming to model the graph's structure and interconnections.
- Document Classification: The learned vector representations of the target paper node and relevant entity nodes are then concatenated with the target text's BERT representation. This combined input is fed into a BERT model for the final classification task.
Datasets
The paper evaluates its method on two established biomedical document classification datasets:
- Ohsumed: A dataset of abstracts categorized into 23 cardiovascular disease categories.
- Hallmarks of Cancer (HoC): A dataset where documents are assigned categories related to 10 different cancer features.
The authors also utilized the MEDLINE database (2020 version) to construct their literature graph, which contains over 30 million articles.
Key Insights and Ideas
- Holistic Information Integration: The paper demonstrates that a unified literature graph, encompassing both bibliographic and entity information, is more effective than relying on isolated external sources. This suggests that the interplay between different types of knowledge is crucial for robust classification.
- Entity Representations Matter: Entity representations derived from the literature graph (Elg) were found to be more effective than those from a standalone entity graph (Eeg). This highlights the importance of contextualizing entities within their document and bibliographic relationships, rather than treating them in isolation.
- Textual Information is Still Key, but Enhanced: While the model integrates graph representations, it still relies on textual input. However, the graph representations act as a powerful enhancement, compensating for information lost when textual entities are masked.
- Simultaneous Use is Tricky: The paper notes that simply combining multiple types of representations (paper, entity) doesn't always yield better results than using a single, well-chosen representation. This suggests that the interaction and potential redundancy between different information sources require careful handling.
Key Results
- The proposed model achieved state-of-the-art performance on both the Ohsumed and HoC datasets.
- On Ohsumed, the model achieved an accuracy of 78.08%, and on HoC, 84.50%. These results are competitive with or surpass existing methods.
- When comparing different representation types, using the paper representation from the literature graph (Pig) for Ohsumed and the entity representation from the literature graph (Elg) for HoC yielded the best results.
- Experiments showed that even without textual information, the graph representations alone could achieve a reasonable level of classification, demonstrating their inherent value.
- Masking entities in the target text significantly degraded performance, but the literature graph's entity representations (Elg) helped to compensate for this loss.
Limitations and Future Work
The authors acknowledge three main limitations:
- TransE Limitations: Due to the massive scale of the literature graph, TransE was used for representation learning. However, TransE struggles with relations that have multiple "tail" nodes (e.g., a paper having multiple authors). More expressive methods like GCN or RotatE could potentially capture these complex relationships better.
- Sentence Truncation: Appending graph representations to the target text can lead to exceeding BERT's maximum input length, requiring truncation of textual information for longer documents.
- Lack of Detailed Analysis: The paper did not deeply analyze how the proposed method positively or negatively impacts document classification, leaving this for future work.
Future work suggested includes exploring GCN for representation learning on large-scale literature graphs and developing methods for simultaneously learning graph representations and document classification.
Summary updated