Development of the Multilingual Semantic Annotation System

Scott Piao, Francesca Bianchi, Carmen Dayrell, Angela D’Egidio, Paul RaysonNAACL 2015

TL;DR

This paper presents a method for rapidly developing semantic annotation tools for new languages by automatically translating existing English semantic lexicons. The authors demonstrate that this "bootstrapping" approach, combined with minor manual refinement, can achieve promising lexical coverage and annotation precision for languages like Italian, Chinese, and Brazilian Portuguese. This work is crucial for advancing multilingual natural language processing (NLP) and text mining capabilities.

What Problem Does This Paper Address?

Developing sophisticated NLP tools, particularly those involving semantic understanding, often relies on extensive, manually curated lexical resources and annotation schemes. While such tools exist for English, creating them for other languages is a time-consuming and labor-intensive process. This paper addresses the challenge of efficiently extending semantic annotation capabilities to multiple languages by proposing a method to automatically generate the necessary semantic lexicons, thereby overcoming the bottleneck of manual resource creation. This is important for building intelligent multilingual NLP systems that can understand and process text across different linguistic boundaries.

Methods

The core of the paper's methodology is the bootstrapping of semantic lexicons for target languages (Italian, Chinese, Brazilian Portuguese) by leveraging an existing English semantic annotation system (USAS). This involves:

  • Automatic Translation of Lexicons: Existing English semantic lexicons, classified under the USAS scheme (21 main discourse fields and 232 sub-fields), are translated into the target languages.
  • Bilingual Lexical Resources: A combination of high-quality, manually edited bilingual dictionaries and larger, less curated bilingual word lists (from sources like FreeLang and LDC) were used for translation.
  • Handling Ambiguity and POS Tagging: The process accounts for the fact that English words can have multiple Part-of-Speech (POS) tags and semantic categories. To avoid losing information, all possible POS tags and their corresponding semantic tags were passed to translation equivalents. This introduces noise but is deemed preferable to missing potential tags.
  • POS Tagset Simplification: To bridge differences in POS tagsets used by various resources, a simplified common POS tagset was designed for each target language.
  • Manual Improvement: After automatic generation, a manual refinement stage is employed to filter incorrect entries, select correct semantic tags, and add missing tags. This stage is crucial for improving precision.
  • System Architecture: The generated semantic lexicons are integrated into a software architecture that includes POS taggers (TreeTagger for Italian/Portuguese, Stanford POS Tagger for Chinese) and disambiguation methods to annotate words in running text.

Datasets

The paper utilizes existing semantic lexicons and bilingual resources for its bootstrapping process. For evaluation, it employs:

  • Reference Corpora:
    • PAISÀ Italian corpus (Borghetti et al., 2011)
    • LCMC Corpus (Lancaster Corpus of Mandarin Chinese) (McEnery and Xiao, 2004)
    • Lacio-Ref Portuguese corpus (Aluisio et al., 2003)
    • Subsections of approximately 1.5 million Italian words and 1.7 million Portuguese words were extracted from these corpora for evaluation.
  • Bilingual Lexical Resources:
    • English-Italian FreeLang wordlist
    • Chinese/English dictionary
    • LDC Eng-Chi bilingual wordlist
    • Portuguese/English dictionary
    • English-Portuguese (Brazilian version) FreeLang wordlist
    • Corpus-based frequency dictionaries for Chinese and Portuguese.

No new benchmarks were introduced; rather, existing corpora were used to evaluate the performance of the developed annotation tools.

Key Insights and Ideas

  • Feasibility of Bootstrapping: The most significant insight is that automatically translating existing English semantic lexicons is a feasible and rapid way to develop prototype semantic annotation tools for new languages.
  • Trade-off Between Coverage and Precision: The use of larger, less curated bilingual word lists, while introducing noise, significantly increases lexical coverage. This highlights a trade-off that can be managed through subsequent manual refinement.
  • Importance of Manual Refinement: While automatic translation is efficient, manual improvement is essential for achieving high annotation precision. The extent of manual effort required is directly related to the quality of the initial bilingual resources.
  • Handling Lexical Ambiguity: The paper demonstrates a practical approach to managing the inherent ambiguity of language, particularly the multiple senses and POS tags of English words, during the translation process.
  • Potential for Multilingual NLP: The success of this approach suggests a scalable path towards creating comprehensive multilingual semantic annotation systems, which is a critical step for advanced cross-lingual NLP applications.

Key Results

  • Lexical Coverage: The prototype annotation tools achieved an average lexical coverage of 79.86% across Italian, Chinese, and Brazilian Portuguese. Italian had the highest coverage (85.53%), and Portuguese the lowest (73.40%).
  • Annotation Precision:
    • First-tag precision: The Portuguese tagger achieved the highest first-tag precision (82.58%), while Italian was lower (55.91%).
  • Including partially correct annotations: When partially correct annotations were included, precision scores became more consistent, with an average of 84.64%.
  • Recall: Estimated recall was around 55.39% to 69.46% for the three languages, indicating room for improvement.
  • Lexicon Sizes: The process generated substantial semantic lexicons: approximately 38,720 entries for Italian, 83,600 for Chinese, and 15,700 for Portuguese.

Limitations and Future Work

  • Limitations:
    • The current recall is still relatively low.
  • The precision of the Italian tagger was significantly lower than others, suggesting that the quality of bilingual resources and the complexity of the language play a role.
  • Errors arise from homonym translations, translation errors, and missing translations in the English lexicons.
  • The current approach does not fully leverage parallel or comparable corpora for disambiguation, which could resolve issues like the erroneous annotation of "爸爸" (father) with a discourse marker tag.
  • Future Work:
    • Improve Precision: Continue manual cleaning and improvement of lexicons.
  • Leverage Parallel/Comparable Corpora: Use these resources for more accurate disambiguation and context-aware translation.
  • Expand Language Coverage: Extend the system to include more languages, such as Spanish and Dutch.
  • Develop Large-Scale System: Aim to build a comprehensive multilingual semantic annotation and analysis system.
  • Task-Based Evaluation: Conduct evaluations of manually checked versus automatically generated lexicons in specific NLP tasks.

Summary updated

BibTeX
@inproceedings{piao2015development,
  author = {Scott Piao and Francesca Bianchi and Carmen Dayrell and Angela D’Egidio and Paul Rayson},
  title = {{Development of the Multilingual Semantic Annotation System}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure