Abstract
This paper describes the WHUNlp team's participation in the SemEval-2016 DiMSUM task, which focuses on identifying multiword expressions (MWEs) and assigning them broad semantic labels known as "supersenses." We implement a two-stage pipeline utilizing Conditional Random Fields (CRFs) for MWE detection and a Maximum Entropy (MaxEnt) classifier for supersense tagging. Using a minimal set of linguistic features, our pilot study establishes a baseline performance and identifies critical challenges such as error propagation in pipeline architectures and the difficulty of detecting non-contiguous MWEs. Post-evaluation analysis demonstrates that incorporating contextual POS bigrams and morphological features significantly enhances performance.
Introduction
Natural Language Processing (NLP) often struggles with identifying "minimal units" of meaning. Multiword Expressions (MWEs), such as "a bit of" or "take a hike," function as single semantic units but are frequently non-contiguous, containing "gaps" (e.g., "he gave the book up"). The DiMSUM (Detecting Minimal Semantic Units and their Meanings) task at SemEval-2016 addresses this by requiring systems to both detect MWEs and label them with supersenses—high-level semantic categories like n.location or v.change. This study explores a supervised learning approach using basic linguistic features to assess baseline capabilities without extensive external knowledge bases.
Methodology
The proposed system follows a two-stage pipeline:
- MWE Detection: We employ first-order Conditional Random Fields (CRFs) with an expanded labeling scheme (O, B, I, o, b, i) to accommodate gappy MWEs. The lowercase tags represent tokens within an MWE span that do not belong to the expression itself.
- Supersense Tagging: Identified units are classified by a Maximum Entropy (MaxEnt) model into one of 41 possible supersense labels (26 for nouns, 15 for verbs).
The model relies on simple features: the current word, a +/- 1 word window, and their Part-of-Speech (POS) tags. The dataset comprises annotated English sentences from service reviews (TrustPilot), social media (Tweebank), and TED talk transcripts.
Results
The system's performance in the official competition was modest, reflecting the difficulty of the task under "super-closed" conditions. The WHUNlp system achieved a Macro-F1 score of 25.71%.
Following the official evaluation, the authors conducted a feature enrichment study. This analysis revealed that the initial feature set was significantly under-powered. The addition of contextual POS bigrams yielded the most dramatic improvement, boosting the MWE detection F1 score by 21.4 points. Morphological features, such as word prefixes, suffixes, and capitalization, also provided substantial gains of 10-12 points.
Error analysis further clarified performance bottlenecks. The most frequent MWE detection errors occurred with NOUN NOUN compounds. In supersense tagging, n.artifact (e.g., "thing," "watch") and v.stative (state-of-being verbs) categories were the most prone to misclassification due to their high semantic ambiguity.
Discussion and Future Work
The pilot study highlights several key insights regarding the pipeline architecture. One of the biggest hurdles was the "pipeline penalty": errors in the initial CRF segmentation stage cascade into the MaxEnt classifier, leading to incorrect labeling of non-existent or poorly defined units. Additionally, "weak" MWEs, such as noun-noun compounds, proved harder to detect than "strong" idioms due to the commonality of their individual components in non-MWE contexts.
Future work will focus on two primary areas: integrating external resources like WordNet and Brown clusters to enrich the feature space, and developing joint models that predict segmentation and supersenses simultaneously to mitigate error propagation.
Summary updated