Abstract: Multi-word expressions (MWE) have seen much attention from the NLP community. In this paper, we investigate their impact on the recognition of textual entailment (RTE). Using the manual Microsoft Research annotations, we first manually count and classify MWEs in RTE data. We find few, most of which are arguably unlikely to cause processing problems. We then consider the impact of MWEs on a current RTE system. We are unable to confirm that entailment recognition suffers from wrongly aligned MWEs. In addition, MWE alignment is difficult to improve, since MWEs are poorly represented in state-of-the-art paraphrase resources, the only available sources for multi-word similarities. We conclude that RTE should concentrate on other phenomena impacting entailment, and that paraphrase knowledge is best understood as capturing general lexico-syntactic variation.
Introduction
In Natural Language Processing (NLP), Multi-word Expressions (MWEs)—such as idioms ("kick the bucket"), compound nouns ("executive director"), and light verb constructions—are often viewed as a major obstacle. Because MWEs possess meanings that cannot be derived solely from their individual components, standard models that process text word-by-word are expected to struggle. This is particularly relevant for Recognizing Textual Entailment (RTE), where the goal is to determine if a hypothesis sentence can be logically inferred from a premise.
Methodology and Data
The researchers quantified the impact of MWEs through three phases:
- Manual Annotation: Analyzing the MSR word alignment corpus for RTE2, they classified MWEs by cardinality and decomposability (whether the phrase's meaning is captured by its parts).
- Resource Analysis: Evaluating how well paraphrase resources like DIRT and GALE MT cover MWEs found in RTE data.
- System Comparison: Comparing two alignment strategies in the Stanford RTE system: UNIQ (one-to-one word mapping) and MANLI (many-to-many phrase alignment).
Key Insights
The study revealed several surprising findings that challenge the conventional wisdom regarding MWEs in inference tasks:
- Rarity: "True" MWEs appeared in only about 8% of sentence pairs in standard benchmarks like RTE2 and RTE4.
- Decomposability: Over half of the identified MWEs were decomposable. For instance, in "executive director," aligning just "director" is often sufficient for the entailment decision.
- Paraphrase Noise: Existing resources intended to help with MWEs are often noisy, sometimes assigning higher similarity scores to incorrect matches than to actual MWEs.
Results and Performance
Contrary to expectations, the simpler one-to-one aligner (UNIQ) consistently outperformed the more complex many-to-many aligner (MANLI) on the end task. On RTE2, UNIQ achieved 63.8% accuracy compared to MANLI's 60.6%. On RTE4, the gap remained, with UNIQ at 63.3% and MANLI at 59.0%. Furthermore, adding paraphrase knowledge to MANLI actually decreased performance on RTE4 to 57.9%, likely due to the introduction of false-positive alignments from noisy data.
Conclusion
The findings suggest that the "idiosyncrasy" of MWEs may not be the primary bottleneck for textual entailment. The authors argue that the NLP community should shift focus from specialized MWE processing toward general lexico-syntactic variation and context-dependent semantic similarity. Improving how models handle context-sensitive nuances—such as how "be awarded" relates to "obtain" depending on the object—is likely to yield more significant gains in inference accuracy.
Summary updated