Abstract
GiBERT (Gated Injection BERT) is a novel architecture designed to enhance the BERT model with external linguistic knowledge, such as synonymy and dependency structures. By utilizing a lightweight gated injection mechanism, GiBERT integrates high-quality static embeddings into BERT's contextual representations. This method allows the model to leverage specialized linguistic information while maintaining the integrity of pre-trained knowledge. Experimental results on five semantic similarity tasks demonstrate that GiBERT significantly outperforms vanilla BERT and other injection baselines, particularly on challenging "non-obvious" cases, while adding only a fraction of the parameters required by attention-based alternatives.
Introduction
Large Language Models (LLMs) like BERT have revolutionized NLP through statistical learning from massive text corpora. However, these models often lack explicit linguistic knowledge, leading to "lexical gaps." For instance, BERT may struggle to recognize semantic equivalence between sentences using different synonyms or fail to grasp complex syntactic dependencies.
Historically, the NLP community developed "static" word embeddings (e.g., counter-fitted vectors) specifically designed to capture these linguistic nuances. GiBERT addresses the challenge of integrating this "old-school" linguistic wisdom into modern contextual models without disrupting their pre-trained capabilities.
Methodology
The GiBERT framework introduces a gated mechanism to inject external information into BERT's internal layers. The process consists of three primary stages:
- Alignment: External static word embeddings are mapped to BERT's WordPiece tokens.
- Projection: A feed-forward layer (linear + tanh) projects external embeddings into the same dimensionality as BERT’s hidden states.
- Gated Injection: A learned gating vector ($g$) controls the flow of external information.
The gate is initialized at zero, ensuring that the model initially behaves as vanilla BERT. During fine-tuning, the model learns to "open the gate" for useful dimensions of external information. This "zero-start" approach prevents catastrophic forgetting of pre-trained knowledge.
Experimental Setup
GiBERT was evaluated on five English-language semantic similarity datasets:
- MSRP: Paraphrase detection in news articles.
- Quora: Duplicate question identification.
- SemEval 2017 (Subtasks A, B, and C): Community Question Answering tasks.
The authors compared GiBERT against vanilla BERT and AiBERT, an alternative method using Multi-Head Attention for injection.
Results and Discussion
GiBERT consistently outperformed vanilla BERT and other baselines across the evaluated datasets. Notably, GiBERT with counter-fitted embeddings (focused on synonyms) achieved the highest average F1 scores.
Efficiency and Precision
A key advantage of GiBERT is its efficiency. The gated injection mechanism adds only 0.23M parameters, significantly less than the 1.64M parameters required by the attention-based AiBERT, while delivering comparable or superior performance.
The Importance of Location
Ablation studies revealed that the optimal injection point varies by information type. Semantic information (synonyms) is most effective when injected in middle layers (e.g., Layer 6), whereas syntactic information (dependencies) yields better results in later layers (e.g., Layer 11). This suggests that BERT processes syntax later in its pipeline than basic lexical meanings.
Handling "Non-Obvious" Cases
GiBERT showed substantial improvements in "non-obvious" F1 scores. These metrics target difficult examples where simple word-overlap strategies fail, proving that the model successfully utilizes external linguistic cues to resolve complex semantic relationships.
Conclusion
GiBERT provides a lightweight and effective method for augmenting BERT with explicit linguistic knowledge. By using a gated injection mechanism, the model achieves state-of-the-art performance on semantic similarity tasks with minimal computational overhead. Future research may explore simultaneous injection of multiple information sources and the application of this method to a broader range of NLP tasks.
Summary updated