Investigating the cross-lingual translatability of VerbNet-style classification

Olga Majewska, Ivan Vulic, Diana McCarthy, Yan Huang, Akira Murakami, Veronika Laippala, Anna KorhonenLang. Resour. Evaluation 2018 · OpenReview

Abstract

VerbNet—the most extensive online verb lexicon currently available for English—has proved useful in supporting a variety of NLP tasks. However, its exploitation in multilingual NLP has been limited by the fact that such classifications are available for few languages only. Since manual development of VerbNet is a major undertaking, researchers have recently translated VerbNet classes from English to other languages. However, no systematic investigation has been conducted into the applicability and accuracy of such a translation approach across different, typologically diverse languages. Our study is aimed at filling this gap. We develop a systematic method for translation of VerbNet classes from English to other languages which we first apply to Polish and subsequently to Croatian, Mandarin, Japanese, Italian, and Finnish. Our results on Polish demonstrate high translatability with all the classes (96% of English member verbs successfully translated into Polish) and strong inter-annotator agreement, revealing a promising degree of overlap in the resultant classifications. The results on other languages are equally promising. This demonstrates that VerbNet classes have strong cross-lingual potential and the proposed method could be applied to obtain gold standards for automatic verb classification in different languages. We make our annotation guidelines and the six language-specific verb classifications available with this paper.

Introduction

In Natural Language Processing (NLP), understanding verb behavior is fundamental. Verbs serve as the "engines" of sentences, dictating argument structures and syntactic movement. VerbNet is the gold standard for English, grouping verbs into classes based on shared syntactic and semantic behaviors, known as diathesis alternations. For example, verbs like "break" and "shatter" are grouped because they both allow the causative-inchoative alternation (I broke the window $\rightarrow$ The window broke).

While VerbNet is invaluable, building similar resources for other languages from scratch is a massive undertaking. This paper investigates whether a systematic "translation approach"—mapping English VerbNet classes to other languages—can serve as an effective shortcut for creating multilingual lexical resources, even for typologically diverse languages like Mandarin, Japanese, and Finnish.

Methodology

The researchers developed a two-stage translation methodology to test cross-lingual applicability:

  1. Direct Translation: Translating the 12 core member verbs of an English VerbNet class into the target language using their most frequent senses.
  2. Syntactic Filtering: Evaluating whether these translated verbs participate in the same syntactic "frames" (sentence structures) in the target language. Verbs that do not fit the behavioral pattern of the class in the new language are discarded.

To evaluate the effectiveness of this method and the level of expertise required, the authors conducted a study with six annotators (three linguists and three non-linguists). Participants were asked to answer a specific set of questions for each VerbNet class to quantify translatability and effort.

Table 1: Participant questionnaire used to evaluate the translatability and effort required for each VerbNet class.

Key Results and Efficiency

The study first "stress-tested" the method on Polish before expanding to Croatian, Italian, Mandarin, Japanese, and Finnish. The results across all languages were remarkably consistent:

  • High Translatability: In Polish, 96% of English member verbs were successfully translated. Across all six languages, success rates ranged from 82% (Japanese) to 98% (Mandarin).
  • Frame Overlap: Approximately 74% of syntactic frames could be directly transferred, suggesting the underlying logic of these verb groupings holds up cross-lingually.
  • The "Expert" Advantage: While non-linguists could perform the task, trained linguists were significantly faster and more consistent, particularly in identifying subtle language-specific requirements like complex case-marking.
Table 2: Comparative results between linguists and non-linguists across the seven evaluation questions, showing means and standard deviations.

As shown in Table 2, the average time to translate and refine one class (Q7) was approximately 32 minutes. This efficiency proves that the translation method is vastly faster than building a lexicon from scratch.

Discussion

Several key insights emerged from the study:

  • Universal Foundations: The core principle of VerbNet—that a verb's meaning determines its syntactic behavior—appears largely universal. English classes provided a stable starting point even for typologically distant languages.
  • Lexicalization Hurdles: The primary challenges involved "Path" vs. "Manner" distinctions in motion verbs. English often lexicalizes how something moves (e.g., amble), while languages like Japanese or Italian often prioritize the direction of motion, leading to smaller class sizes in those target languages.
  • Manual vs. Automatic: Manual translation proved far superior to semi-automatic mapping methods, which often produced noisy results and missed over half of the valid verb candidates.

Conclusion

This research demonstrates that VerbNet classes have strong cross-lingual potential. The proposed systematic translation method offers a viable path for creating high-quality gold standards for automatic verb classification in diverse languages. Future work will explore testing this method on more linguistically "exotic" structures, such as ergative languages, and investigating the role of Machine Translation in further automating the initial stages of resource creation.

Summary updated

Abstract

VerbNet—the most extensive online verb lexicon currently available for English—has proved useful in supporting a variety of NLP tasks. However, its exploitation in multilingual NLP has been limited by the fact that such classifications are available for few languages only. Since manual development of VerbNet is a major undertaking, researchers have recently translated VerbNet classes from English to other languages. However, no systematic investigation has been conducted into the applicability and accuracy of such a translation approach across different, typologically diverse languages. Our study is aimed at filling this gap. We develop a systematic method for translation of VerbNet classes from English to other languages which we first apply to Polish and subsequently to Croatian, Mandarin, Japanese, Italian, and Finnish. Our results on Polish demonstrate high translatability with all the classes (96% of English member verbs successfully translated into Polish) and strong inter-annotator agreement, revealing a promising degree of overlap in the resultant classifications. The results on other languages are equally promising. This demonstrates that VerbNet classes have strong cross-lingual potential and the proposed method could be applied to obtain gold standards for automatic verb classification in different languages. We make our annotation guidelines and the six language-specific verb classifications available with this paper.
BibTeX
@inproceedings{majewska2018investigating,
  author = {Olga Majewska and Ivan Vulic and Diana McCarthy and Yan Huang and Akira Murakami and Veronika Laippala and Anna Korhonen},
  title = {{Investigating the cross-lingual translatability of VerbNet-style classification}},
  booktitle = {Lang. Resour. Evaluation},
  year = {2018},
  url = {https://openreview.net/forum?id=lDOWceOrOo},
}
Figure