TL;DR
Large language models (LLMs) that process text in only one direction (unidirectional) struggle with tasks requiring understanding of the full context, like token classification. This paper proposes a simple yet effective method: train a small "backward" language model and concatenate its outputs with a standard "forward" unidirectional LLM to achieve pseudo-bidirectionality, significantly boosting performance on these tasks.
What Problem Does This Paper Address?
Modern large language models (LLMs) often fall into two categories: unidirectional (UniLMs) and bidirectional (BiLMs). UniLMs, like GPT and Llama, are excellent for text generation because they process text sequentially, predicting the next word based on what came before. BiLMs, like BERT, are historically dominant in tasks requiring a deep understanding of the entire sentence's context, such as named entity recognition (NER) or part-of-speech (POS) tagging. Despite the impressive capabilities of large UniLMs, they haven't fully displaced BiLMs in these "token-level classification" tasks. This paper hypothesizes that the core reason is the UniLMs' inherent lack of bidirectionality – they can't "see" the words that come after the current word when making a prediction. The challenge is to equip powerful UniLMs with this crucial bidirectional understanding without the prohibitive cost of retraining them from scratch.
Methods
The paper introduces a straightforward yet powerful technique to imbue UniLMs with bidirectionality. The core idea is to train a separate, small unidirectional language model that processes text in reverse (a "backward LM"). This backward LM generates token representations based on the context following a token.
The key innovation lies in how these representations are combined:
- Concatenation: The representations from the standard "forward" UniLM (which sees context from the left) and the newly trained "backward" LM (which sees context from the right) are concatenated. This creates a richer, pseudo-bidirectional representation for each token.
- Drop-in Replacement: This concatenated representation is then fed into a simple classification layer, effectively replacing the output of a traditional BiLM. Crucially, the original large UniLM is not fine-tuned; only the classification layer is trained for the downstream task.
This approach offers flexibility: the backward LM can be reused with different forward UniLMs, and it doesn't require access to the internal workings or gradients of the forward model, making it applicable even to "black-box" LLMs.
Datasets
- Training: The backward LM was trained on BookCorpus and Wikitext-103.
- Evaluation: The method was evaluated on several token-classification tasks:
- Chunking
- Part-of-Speech (POS) tagging
- Named Entity Recognition (NER) on the CoNLL-2003 dataset.
- NER on the Few-NERD dataset (specifically for rare domains).
Key Insights and Ideas
- Bidirectionality is Key for Token Classification: The paper strongly suggests that the unidirectional nature of LLMs is a significant bottleneck for token-level classification tasks, even for very large models.
- Simple Concatenation Works Wonders: The proposed method of simply concatenating forward and backward LM representations is surprisingly effective, achieving performance comparable to or better than dedicated BiLMs like BERT.
- Reusability of Backward Models: A significant advantage is that the trained backward LM can be paired with various forward UniLMs, as long as they share a vocabulary. This offers a "one-for-many" solution.
- Addressing the "Valley" in Few-Shot Learning: The method shows particular promise in low-data regimes (few-shot learning), helping to bridge the performance gap between very limited and moderately sufficient training examples.
- Effectiveness for Rare Domains: The approach demonstrates improved performance on specialized datasets like Few-NERD, indicating its utility for domain-specific challenges.
Key Results
- Significant Performance Gains: The proposed method improved performance on CoNLL-2003 NER by over 10 points compared to using the forward UniLM alone.
- Competitive with BERT: The concatenated models achieved performance comparable to or better than BERT (a strong BiLM) across various tasks.
- Few-Shot Improvement: In few-shot NER settings on CoNLL-2003, the proposed method outperformed BERT when training data was very limited (less than 16 shots per entity).
- Rare Domain Success: On the Few-NERD dataset, Llama-2 with the proposed method outperformed GPT-2, suggesting larger forward LMs are more effective for rare domains.
- Qualitative Improvements: Case studies showed the method could correctly identify entities at the beginning of sentences or in conjoined phrases where forward-only models struggled.
Limitations and Future Work
- Computational Cost: The proposed method increases inference cost due to the additional backward LM. However, this is mitigated by the fact that the backward LM is small and can be run alongside existing large UniLMs.
- Task Scope: The evaluation was primarily focused on token-level classification tasks in English. Future work could explore applying this method to other tasks like text classification (by pooling representations) or cross-lingual settings.
- Model Size: The experiments used models up to 7B parameters. Investigating the impact of even larger UniLMs is a potential future direction.
- Strict Comparability: While BERT and GPT-2 were chosen for their relevance, future work could involve training comparable models from scratch for a more direct comparison.
- Alternative Backward Providers: The paper notes that using existing BiLMs as a source of backward context, rather than training a new small LM, is another avenue to explore.
Summary updated