TL;DR
This paper introduces a novel method for enhancing stochastic unification-based grammars (SUBGs) by incorporating external linguistic knowledge, specifically lexical selectional preferences, through "auxiliary distributions." This approach allows SUBGs to leverage information that would be too sparse to estimate directly from limited parsed corpora, leading to more robust and informative grammatical models.
What Problem Does This Paper Address?
Developing accurate probabilistic models for "unification-based" grammars (UBGs), like Lexical-Functional Grammar (LFG), is challenging. Unlike simpler models like Probabilistic Context-Free Grammars (PCFGs), UBGs can capture complex, non-local linguistic constraints. This complexity makes it difficult to estimate the vast number of parameters required for a stochastic UBG (SUBG) from typical parsed corpora, which are often small. Specifically, incorporating fine-grained lexical information, such as which verbs prefer which types of arguments, is practically impossible with limited data. This paper tackles this data sparsity problem by proposing a way to integrate pre-existing statistical models of lexical preferences, estimated from larger, less structured corpora, into SUBGs.
Methods
The core contribution is a new estimation procedure for SUBGs that allows the incorporation of "auxiliary distributions." These auxiliary distributions, which can be estimated from external sources (like large text corpora with shallow parsing), are treated as additional features within an exponential family model framework.
Here's a breakdown:
- Exponential Family Models: The paper frames SUBGs within the standard exponential family model, where the probability of a parse tree $\omega$ is given by $P(\omega) = \frac{1}{Z_\lambda} e^{\lambda \cdot f(\omega)}$, where $f(\omega)$ is a vector of features and $\lambda$ are the learned parameters.
- Auxiliary Features: The novelty lies in defining new features $f_{m+j}(\omega) = \log Q_j(\omega)$, where $Q_j(\omega)$ is an auxiliary distribution. These are essentially the log probabilities from external models.
- Parameter Estimation: The parameters $\lambda$ for both the original and auxiliary features are estimated using a pseudo-likelihood maximization approach, similar to previous work on SUBGs. This method is computationally feasible for parsing applications.
- "Weight" Parameters: Crucially, each auxiliary distribution $Q_j$ is associated with its own parameter $\lambda_{m+j}$. This allows the estimation procedure to learn how much "weight" to give to each auxiliary distribution, effectively allowing it to scale, down-weight, or even ignore the external information if it's not deemed relevant. This is a key advantage over simpler methods that might force the external information to be used directly.
- Lexical Selectional Preferences: The specific auxiliary distribution used in their experiments is based on a model of lexical selectional preferences derived from shallow parses of a large corpus. This model captures relationships between governors (verbs/prepositions) and their arguments, smoothed across hidden classes to handle data sparsity.
Datasets
- Training Data:
- Verbmobil Corpus: 324 ambiguous sentences, with 3,245 parses.
- Homecentre Corpus: 481 ambiguous sentences, with 3,169 parses.
- These corpora were provided with manual annotations indicating the correct parse for each sentence.
- Data for Auxiliary Distribution:
- British National Corpus (BNC): A 117 million word corpus used to estimate the lexical selectional preference model. Shallow parses were generated for this corpus.
Key Insights and Ideas
- Bridging the Data Gap: The most significant insight is that external, potentially less structured, but larger datasets can be effectively leveraged to enrich sparse, structured linguistic models like SUBGs. This is a generalizable strategy for many NLP tasks where high-quality structured data is scarce.
- Flexible Integration of External Knowledge: The proposed method doesn't rigidly impose external knowledge. By learning "weight" parameters for each auxiliary distribution, the model can adaptively decide how much to rely on this external information, making it robust even if the auxiliary distribution is only partially relevant.
- Auxiliary Distributions as Features: Treating the log of auxiliary distributions as features is a simple yet powerful mechanism. It elegantly fits into the existing exponential family framework for SUBGs without requiring fundamental changes to the estimation machinery.
- Pseudo-likelihood for Practicality: The use of pseudo-likelihood estimation is crucial for making the approach computationally tractable for parsing, as it avoids full integration over the entire parse space.
Key Results
The paper evaluates the impact of adding auxiliary lexical dependency features to a Stochastic Lexical-Functional Grammar (SLFG).
- Verbmobil Corpus:
- Adding auxiliary features slightly improved the "correct parses" measure (from 180 to 183 for feature $f_1$, and 180.5 for $f_t, f_c, f_n$).
- The pseudo-likelihood measure showed a slight degradation or minimal improvement (from 401.3 to 401.6 for $f_1$, and 404.0 for $f_t, f_c, f_n$).
- Homecentre Corpus:
- The "correct parses" measure saw a marginal improvement (from 283.25 to 284 for $f_1$, and 285 for $f_t, f_c, f_n$).
- The pseudo-likelihood measure also showed a slight improvement (from 580.6 to 580.6 for $f_1$, and 582.2 for $f_t, f_c, f_n$).
- Indistinguishable Sentences: Adding auxiliary features reduced the number of indistinguishable sentences, particularly for the Homecentre corpus (by 24%), suggesting the features help disambiguate.
- Overall Performance: Even without auxiliary features, the system achieved good performance, selecting the correct parse for 56% of ambiguous Verbmobil sentences and 59% of ambiguous Homecentre sentences.
The authors express disappointment that the improvements were not more substantial, especially on the pseudo-likelihood measure, suggesting potential issues with the mismatch between the source of auxiliary data and the target parses, or the increased dimensionality of the feature space.
Limitations and Future Work
- Limitations:
- Data Mismatch: The authors acknowledge that the lexical dependencies estimated from shallow parses might not perfectly align with the detailed structures produced by LFG. This mismatch could limit the effectiveness of the auxiliary features.
- Small Corpora: The small size of the UBG-parsed corpora remains a fundamental limitation, hindering the direct estimation of complex features.
- Pseudo-likelihood Degradation: The slight degradation in pseudo-likelihood scores with added features is noted, possibly due to increased feature dimensionality and the nature of the pseudo-likelihood measure itself.
- Future Work:
- Broader Application: The authors suggest applying these techniques to other SUBG formalisms like HPSGs, categorial grammars, and transformational grammars.
- TAG Parsers: A promising extension is to apply these methods to broad-coverage feature-based Tree-Adjoining Grammar (TAG) parsers.
- Improved Auxiliary Data: Investigating auxiliary distributions estimated from corpora more closely related to the target domain or linguistic phenomena could yield better results.
Summary updated