TL;DR
This paper proposes a novel log-bilinear model that leverages pretrained static word embeddings to improve the estimation of word co-occurrence probabilities, especially in low-data scenarios. By incorporating semantic and syntactic information from embeddings, the model achieves better accuracy than traditional methods, offering a more robust way to capture linguistic relationships.
What Problem Does This Paper Address?
Understanding how words relate to each other is fundamental to many areas of linguistics and computational linguistics, including psycholinguistics, corpus linguistics, and cognitive modeling. Specifically, researchers are interested in conditional probabilities, like the probability of an adjective given the noun it modifies (e.g., P(red | ball)) or the probability of a direct object given its verb (e.g., P(ball | kicked)). These probabilities are crucial for measures like Pointwise Mutual Information (PMI), which is used in grammar induction, sentence comprehension models, and theories of word order.
However, accurately estimating these probabilities from text data is challenging due to the sparsity of word co-occurrences. Even with powerful pretrained language models, directly extracting these specific conditional probabilities can be difficult because they often rely on relationships not explicitly captured by linear word order or simple templates. This paper aims to bridge this gap by using pretrained static word embeddings to enhance generalization and provide more accurate estimates, particularly when training data is limited.
Methods
The core of the paper is a log-bilinear model adapted for estimating conditional probabilities. The model takes pretrained static word embeddings for two words, a target word ($w$) and a context word ($c$), as input. These embeddings are then passed through separate encoder functions ($\phi$ and $\psi$) to produce latent representations. The probability of $w$ given $c$ is then modeled as:
$q(w | c) = Z(c)^{-1} \exp(\phi(w)^T A \psi(c))$
where $Z(c)$ is a normalization term (the partition function), and $A$ is an interaction matrix. The model parameters ($\phi$, $\psi$, and $A$) are trained by minimizing the cross-entropy loss on a dataset of word pairs.
A key aspect of this approach is its ability to generalize. By using word embeddings, the model can infer relationships between words that haven't been seen together in the training data, based on their semantic and syntactic similarities captured in the embeddings. The authors also explore variations, such as using a single encoder for both target and context words (tying parameters) and investigate the impact of the target word vocabulary size ($W$).
Datasets
The experiments were conducted on several languages (Arabic, English, Korean, and Spanish) using data from:
- Universal Dependencies (UD) 2.8: Used for extracting dependency relations like "amod" (attributive modifier) for adjective-noun pairs and "obj" (direct object) for verb-noun pairs.
- Automatically-parsed Wikipedia datasets (CoNLL 2017 Shared Task): Also used for extracting adjective-noun and verb-object pairs.
For training, a fixed number of adjective-noun pairs (99,000) were randomly sampled from the Wikipedia datasets for each language. Development sets of 1,000 pairs were used for hyperparameter tuning. Test sets were derived from the relevant UD corpora. The target word vocabulary ($W$) was set to the 10,000 most frequent wordforms for adjectives and 20,000 for nouns.
Key Insights and Ideas
- Embeddings as a powerful prior: Pretrained word embeddings provide a rich source of semantic and syntactic information that can significantly improve the estimation of co-occurrence probabilities, especially when empirical counts are sparse.
- Generalization beyond co-occurrence: The log-bilinear model, by leveraging embeddings, can generalize to unseen word pairs by understanding their underlying semantic relationships, a feat difficult for purely count-based methods.
- Task-dependent performance: The model performs better on predicting attributive adjectives given nouns than on predicting direct objects given verbs. This is attributed to the inherent nature of the tasks and potentially the larger vocabulary size required for nouns.
- Word encoders can be optional: In some cases, particularly for predicting adjectives, the log-bilinear model without explicit word encoders (i.e., using the raw embeddings directly) performed comparably or even better than models with encoders. This suggests that the input embeddings themselves are already well-suited for this task.
Key Results
The proposed log-bilinear model consistently outperformed baseline methods (additive smoothing, interpolated smoothed estimator, and softmax) across all languages and tasks.
- Attributive Adjectives: The log-bilinear models (both with and without encoders) achieved significantly lower Negative Log Likelihood (NLL) scores compared to baselines. In some cases (e.g., Spanish and Korean adjectives), only the log-bilinear model could outperform the interpolated baseline.
- Direct Objects: While overall performance was worse than for adjectives, the log-bilinear models still showed a clear advantage over baselines.
- Unseen Contexts: The model demonstrated robustness on pairs where the context word (head noun or verb) was not seen during training, indicating strong generalization capabilities.
- Fine-tuning vs. Static Embeddings: Fine-tuning the input word embeddings led to severe overfitting, highlighting the benefit of using static embeddings as a fixed prior.
- Vocabulary Size: Restricting the target word vocabulary ($W=V$) negatively impacted performance, suggesting that a carefully chosen, smaller target vocabulary is beneficial.
Limitations and Future Work
- Limitations:
- The paper notes that when predicting adjectives, models without explicit word encoders sometimes performed better, suggesting that the encoders might not always add value or could even be detrimental if not properly trained.
- The performance on predicting direct objects from verbs was generally worse than for adjectives, indicating that this task is inherently harder.
- Future Work:
- The authors suggest that a hybrid model combining their approach with methods that handle unknown or infinite vocabularies (like neural-Bayesian nonparametric estimators) could be a promising direction.
- Exploring different encoder architectures and hyperparameter settings could further optimize performance.
- Investigating the impact of different types of pretrained embeddings (e.g., contextual embeddings) could be a valuable extension.
Summary updated