TL;DR
This paper introduces a novel framework for automatically generating rich feature sets for Semantic Role Labeling (SRL) by composing small, task-agnostic units called "featlets." This approach rivals state-of-the-art performance on standard SRL datasets with significantly reduced reliance on human linguistic expertise.
What Problem Does This Paper Address?
Feature engineering is a crucial but labor-intensive aspect of building robust Natural Language Processing (NLP) models. Traditionally, this involves experts manually crafting features, a process that is often opaque, difficult to reproduce, and can lead to stagnation in feature design. The authors highlight that while advancements in NLP models are often methodologically driven, the underlying feature sets, which are critical for performance, are frequently described vaguely. This lack of clarity hinders reproducibility and can bias practitioners towards copying existing, potentially suboptimal, feature sets. The paper aims to automate this feature generation process, making it more systematic, reproducible, and less reliant on domain-specific knowledge.
Methods
The core innovation is the concept of featlets, which are small, atomic operations that can be composed to create more complex features. These featlets are task-agnostic and can be combined to form templates. The authors distinguish between featlets (building blocks) and templates (sets of featlets that produce features).
The process involves:
- Featlet Composition: Featlets are categorized into types like Label Extractors (e.g.,
TARGETSPAN), Token Extractors (e.g.,WORD,POS), Value Mutators (e.g.,LCfor lowercase), Dependency Walkers (e.g.,PARENTD), and Distance Functions. These can be combined in various ways to construct complex feature templates. - Template Generation: A brute-force search with heuristics is used to find "legal" templates (combinations of featlets that produce meaningful output).
- Template Transformation: Automatic transforms like
TOPN(frequency-based filtering) andCNTC(count-based filtering) are applied to manage the number of generated features. - Feature Selection: To handle the potentially massive number of generated features, mutual information (information gain) is used to score templates, and a greedy pruning strategy is employed to select a manageable, informative feature set.
Datasets
The paper evaluates its approach on two standard Semantic Role Labeling (SRL) datasets:
- FrameNet 1.5: A dataset derived from the FrameNet lexical database.
- CoNLL 2012 data: Derived from the OntoNotes 5.0 corpus.
The authors also mention using data from the Stanford CoreNLP toolset for annotations.
Key Insights and Ideas
- Decomposition into Featlets: Breaking down complex features into simple, reusable "featlets" is the key to automating feature generation. This allows for systematic exploration of feature combinations that might not be intuitive to human experts.
- Task Agnosticism of Featlets: The ability to define featlets that are not tied to a specific NLP task (like SRL) makes the framework generalizable and reusable across different problems.
- Information Gain for Feature Selection: Using mutual information as a scoring mechanism effectively filters the vast space of generated features, prioritizing those that are most informative for the prediction task.
- Expert-Free Feature Engineering: The framework demonstrates that it's possible to achieve state-of-the-art performance without extensive manual feature engineering, shifting the burden from human intuition to computational search and scoring.
- Sensitivity to Feature Stages: The experiments reveal that the model is more sensitive to the removal of argument identification features than role classification features, offering insights into where to focus computational resources for optimization.
Key Results
- The proposed automatic feature generation method achieves performance comparable to state-of-the-art systems that rely on hand-engineered features for SRL.
- On the FrameNet dataset, the best model used approximately 2500 features, while on Propbank, the optimal feature set size varied significantly with the
βparameter (ranging from 320 to 1280 features). - The paper shows that feature sets optimized for one dataset do not necessarily transfer perfectly to another, highlighting the importance of dataset-specific feature selection.
Limitations and Future Work
- Symmetry in Featlets: The paper acknowledges that while template products are commutative, featlets are not, and identifying equivalent featlet combinations can be challenging and computationally expensive.
- Large Feature Sets: Even with feature selection, the best models still require a significant number of features (e.g., over 2500 for FrameNet).
- Future Work: The authors suggest exploring low-rank or embedding methods for parameterizing featlet strings, rather than storing individual weights. They also propose investigating step-wise methods that iteratively select features based on mutual information with residuals. Further research into handling symmetries and improving the efficiency of feature selection is also suggested.
Summary updated