TL;DR
This paper introduces a novel Bidirectional Long Short-Term Memory (bi-LSTM) model for Combinatory Categorial Grammar (CCG) supertagging, achieving state-of-the-art performance. By effectively capturing long-range syntactic dependencies, their model significantly outperforms previous approaches, demonstrating the power of deep learning architectures for complex linguistic tasks.
What Problem Does This Paper Address?
The paper tackles the challenge of CCG supertagging, a crucial step in syntactic and semantic parsing. Supertagging assigns complex lexical labels to words, which are essential for accurately parsing sentences according to Combinatory Categorial Grammar. Unlike simpler Part-of-Speech (POS) tagging, which has a limited set of labels, CCG supertagging involves a vast label space (over 1,200 labels). Crucially, correctly assigning a supertag often requires understanding syntactic relationships that can span across the entire sentence, making it a difficult task for models that only consider local context. Existing approaches, while improving, struggled to fully capture these long-range dependencies.
Methods
The authors propose and evaluate several neural network architectures for supertagging:
- Feed-Forward Neural Networks (FFNNs): They use FFNNs with two hidden layers. For POS tagging, they employ a five-word window. For supertagging, they use a slightly smaller window (3 words left/right) with suffix and capitalization features, but importantly, they train on the full set of observed supertag categories, unlike some prior work.
- Bidirectional Long Short-Term Memory (bi-LSTM) Models:
- Basic bi-LSTM: This model uses separate forward and backward LSTMs to process the input sentence, combining their hidden states to predict the supertag for each word.
- bi-LSTM-LM (Language Model): This novel architecture augments the bi-LSTM with an LSTM language model that operates over the predicted supertags. This allows the model to explicitly learn and leverage the sequential dependencies between supertags, further enhancing its ability to capture sentence structure.
- Training Techniques:
- Gold Supertag Training (g-train): Standard training where the model uses the correct supertag at each step.
- Scheduled Sampling (ss-train): To bridge the gap between training (where gold labels are available) and inference (where errors can propagate), they employ scheduled sampling. This technique probabilistically uses the model's own predictions from previous steps as input for the current step, improving robustness. They experiment with using the single most likely prediction (ss-train-1) and the 5-best predictions (ss-train-5).
Datasets
The experiments are conducted on the CCGbank dataset, using standard splits for training, development, and testing. For POS tagging, they use WSJ sections 00-18 for training, 19-21 for development, and 22-24 for testing. They note that their CCGbank training section contains 1,284 lexical categories, with a long tail, and they allow their models to predict all categories, not just those above a certain frequency threshold.
Key Insights and Ideas
- Long-Range Dependencies are Crucial: The paper strongly argues that supertagging, unlike POS tagging, necessitates understanding syntactic information from across the entire sentence. This is evidenced by the significant gains achieved by bidirectional models over unidirectional ones.
- Language Model on Supertags Boosts Performance: The introduction of an LSTM language model that predicts the next supertag based on previous supertags (bi-LSTM-LM) provides substantial improvements, particularly in parseability. This highlights the importance of modeling the sequential structure of supertag sequences themselves.
- Scheduled Sampling Improves Robustness: Techniques like scheduled sampling are vital for making deep learning models more robust to their own prediction errors during inference, leading to better overall performance.
- Feed-Forward Models Can Be Surprisingly Strong: The paper shows that well-designed feed-forward networks can achieve competitive results on POS tagging, even outperforming some bi-LSTM baselines, suggesting that for simpler tasks, local context might suffice.
- Embeddings Reveal Model Strengths: Visualizing learned embeddings shows how different models capture different aspects of syntactic relationships, with the bidirectional model integrating the strengths of both forward and backward contexts.
Key Results
- The proposed bi-LSTM-LM model with scheduled sampling (ss-train-1 and beam decoding) achieves a new state-of-the-art test accuracy of 94.5% on CCG supertagging, representing a 1.5% absolute gain over previous approaches.
- Their Feed-Forward POS tagging model achieves 97.28% on the development set and 97.4% on test, demonstrating strong performance with a simpler architecture.
- The bi-LSTM-LM model with beam decoding improves parseability by up to 15% compared to models without the language model.
- Table 2 shows that the bidirectional model significantly outperforms unidirectional models on many common and difficult supertags, underscoring the need for global context.
Limitations and Future Work
- Limitations:
- While the bi-LSTM-LM with scheduled sampling and beam decoding achieves state-of-the-art, the paper notes that greedy decoding with ss-train-5 can perform as well as beam decoding with ss-train-1, suggesting potential for further optimization in decoding strategies.
- The large number of unused categories (≥120) indicates that there is still substantial room for improvement, particularly for rare categories.
- Future Work:
- The authors suggest investigating methods to improve performance on rare categories.
- Further exploration of language model integration and training techniques for enhanced robustness and accuracy.
Summary updated