Abstract
Standard metrics for evaluating dependency parsers are fundamentally biased toward "analytic" languages like English that use many function words, unfairly penalizing "synthetic" languages like Finnish that use inflection. This paper introduces CLAS (Content-Word Labeled Attachment Score), a new metric that levels the cross-linguistic playing field by focusing evaluation strictly on the relationships between content-carrying words.
The Arithmetic Bias in Dependency Parsing
In Natural Language Processing (NLP), the Labeled Attachment Score (LAS) is the standard for measuring dependency parser performance. LAS calculates the percentage of words assigned the correct head and grammatical label. However, this metric is structurally biased due to the diversity of human languages.
Consider the difference between English and Finnish. English is an analytic language, relying on separate function words (determiners, prepositions) to express grammatical relationships. Finnish is a synthetic, morphologically rich language that uses suffixes (inflections) to achieve the same goals. As shown in Figure 1, an English sentence may require eight words to express a thought that Finnish captures in four.
If a parser makes a single error in both sentences, the English parser receives an 87.5% score, while the Finnish parser drops to 75%. This "arithmetic bias" suggests that parsers for morphologically rich languages are underperforming, when in fact the metric is simply rewarding the correct identification of "easy" function words.
Methodology: Introducing CLAS
To address this, the authors propose CLAS (Content-Word Labeled Attachment Score). The metric is built by categorizing the 37 standard Universal Dependencies (UD) syntactic relations into five functional groups:
- CORE: Essential arguments (subjects, objects).
- NON-CORE: Modifiers (adjectives, adverbs).
- FUN: Functional relations (determiners, auxiliaries, case markers).
- MWE: Multiword expressions.
- PUNCT: Punctuation.
CLAS is defined as the labeled $F_1$-score calculated over all relations excluding the FUN and PUNCT groups. By stripping away these auxiliary relations, the metric focuses on the semantic backbone of the sentence—the dependencies between content-carrying words.
Experimental Setup
The study evaluated 42 treebanks from the Universal Dependencies release v2.0. The authors selected one representative treebank per language, requiring at least 30,000 words and full morphological features to ensure a robust comparison across language families, including Slavonic, Germanic, Romance, and Finno-Ugric.
Key Results and Typological Insights
The transition from LAS to CLAS reveals significant shifts in perceived parser performance. The researchers found that functional relations (FUN) are generally parsed with high accuracy (~90%), while core arguments (CORE) hover around 75%. Including function words thus artificially inflates scores for languages that use them heavily.
Key findings include:
- Average Performance Drop: Across the 42 languages, the average score decreased by 4.32 points when moving from LAS to CLAS.
- Typological Sensitivity: The gap between LAS and CLAS serves as a "typological profile." Japanese and Romance languages (French, Italian, Spanish) saw massive drops (e.g., Japanese dropped 13.18 points), whereas agglutinating languages like Hungarian or Finnish saw minimal changes (Hungarian dropped only 1.08 points).
- Punctuation Stability: Excluding punctuation makes the metric more stable, as punctuation annotation is often inconsistent across treebanks and contributes little to syntactic understanding.
Conclusion
CLAS provides a more equitable framework for cross-linguistic parser evaluation by focusing on the semantic core of sentences. While LAS remains useful for full sentence reconstruction, CLAS reveals the "real" difficulty of parsing core syntactic relationships. Future work will explore whether CLAS correlates better with downstream tasks like Machine Translation, where the relationship between content words is paramount.
Summary updated