TL;DR
This paper introduces Roof-Transformer, a novel architecture that addresses the limitations of standard BERT models in incorporating external knowledge. By using two BERTs – one for text and one for knowledge – and a fusion layer, Roof-Transformer effectively integrates rich knowledge graphs into language understanding tasks, leading to improved performance on QA and GLUE benchmarks.
What Problem Does This Paper Address?
Large language models like BERT have revolutionized NLP, but they often struggle to leverage external knowledge effectively. A key challenge is BERT's fixed input sequence length (typically 512 tokens). When trying to inject knowledge from sources like knowledge graphs (KGs) into BERT, this limitation forces the model to discard valuable information, especially for tasks involving long documents or complex reasoning. Existing methods often concatenate knowledge directly into the input, further exacerbating the sequence length issue. This paper tackles the problem of how to efficiently and comprehensively integrate external knowledge into BERT-based models without being severely constrained by input length limitations.
Methods
The core innovation is the Roof-Transformer architecture. It employs a "divide and join" strategy:
- Two Underlying BERTs:
- TASK-BERT: Encodes the original input text (e.g., a question-paragraph pair for QA).
- KB-BERT: Encodes knowledge triples extracted from a knowledge base (KB).
- Fusion Layer: A Transformer Encoder acts as a "roof" over the two BERTs, integrating their respective outputs. This layer is crucial for combining the textual and knowledge-based representations.
- Prediction Layer: A standard linear layer for downstream task prediction.
This architecture allows for a more flexible integration of knowledge, as the KB-BERT can process knowledge independently, and the fusion layer can then intelligently combine it with the text representation. The authors also explore various knowledge representation formats (Exp0, Exp1, Exp2) and selection strategies (Has_Tail vs. No_Tail) to optimize knowledge integration.
Datasets
The paper evaluates Roof-Transformer on two main types of tasks:
- Question Answering (QA): The DRCD dataset, a Chinese QA benchmark.
- General Language Understanding Evaluation (GLUE): A suite of English NLP tasks. Specifically, they used eight GLUE tasks: SST-2, CoLA, MRPC, STS-B, QNLI, QQP, RTE, and MNLI-m.
For knowledge, they utilized:
- Chinese KBs: HowNet and CN-DBpedia.
- English KB: KELM corpus (which converts Wikidata into natural text sentences).
Key Insights and Ideas
- Dual BERTs are Crucial: Sacrificing text input length to accommodate knowledge within a single BERT often leads to performance degradation. Using a separate BERT for knowledge is more effective.
- Knowledge Selection Matters: Simply adding more knowledge isn't always better. The "Has_Tail" selection strategy (where both the head and tail of a triple appear in the passage) significantly improves performance, even with less knowledge, suggesting that relevant, well-integrated knowledge is key.
- Fusion Layer Design: The Transformer Encoder proves to be a powerful fusion mechanism. Its depth and learning rate are critical hyperparameters, with a higher learning rate for the fusion layer showing benefits.
- Cached KB-BERT Efficiency: Using a cached (frozen) KB-BERT encoder for knowledge can reduce computational cost by 33% without significant performance drop, making inference more efficient.
- Task-Specific Knowledge Needs: Knowledge integration doesn't benefit all tasks equally. Tasks like sentiment analysis (SST-2) or sentence similarity (STS-B) might rely more on emotional words or semantic similarity rather than factual knowledge, showing less improvement.
Key Results
- QA Performance: Roof-Transformer achieved an EM score improvement of over 1.5% on the DRCD QA task compared to baselines, demonstrating the value of integrated knowledge.
- GLUE Performance: Roof-Transformer outperformed baselines like BERTbase and ERNIE on several GLUE tasks (e.g., RTE, CoLA, MRPC), showing its effectiveness in general language understanding.
- KB Format Impact: The "Exp2 + Has_Tail" KB format consistently yielded the best results, highlighting the importance of both representation and selection.
- Fusion Layer Efficiency: Experiments showed that using the last 4 TE layers for the fusion layer on QA tasks and 3 layers for GLUE tasks provided optimal performance. A learning rate 10 times that of the underlying BERTs also boosted performance.
Limitations and Future Work
- Input Length Limitation Persists: While Roof-Transformer alleviates the problem, the KB-BERT still has a 512-token limit for knowledge. Precise knowledge selection is therefore critical.
- Future Work: The authors plan to explore replacing the underlying BERTs with other pre-trained LMs like RoBERTa and XLNet to further consolidate the framework's effectiveness. They also suggest that this architecture is generalizable and could benefit from further exploration with different knowledge sources and NLP tasks.
Summary updated