TexSmart: A Text Understanding System for Fine-Grained NER and Enhanced Semantic Analysis

Haisong Zhang, Lemao Liu, Haiyun Jiang, Yangming Li, Enbo Zhao, Kun Xu, Linfeng Song, Suncong Zheng, Botong Zhou, Jianchen Zhu, Xiao Feng, Tao Chen, Tao Yang, Dong Yu, Feng Zhang, Zhanhui Kang, Shuming ShiarXiv 2020 · OpenReview

TL;DR

TexSmart is a comprehensive text understanding system that significantly expands the capabilities of Named Entity Recognition (NER) by supporting over 1,000 fine-grained entity types. It also introduces novel semantic analysis features like semantic expansion and deep semantic representation, offering a richer understanding of text beyond traditional NLP tools.

What Problem Does This Paper Address?

Natural Language Processing (NLP) aims to enable computers to understand human language as well as humans do. This involves various tasks like lexical, syntactic, and semantic analysis. However, existing text understanding systems often fall short in several areas:

  • Limited Entity Granularity: Traditional NER systems typically recognize only a handful of broad entity types (e.g., person, location, organization). This lack of detail hinders deeper analysis and application.
  • Lack of Advanced Semantic Features: Many systems lack advanced semantic analysis capabilities, such as providing related entities for context (semantic expansion) or parsing temporal and quantitative expressions into precise structured formats (deep semantic representation).
  • One-Size-Fits-All Algorithms: Most systems offer a single algorithm for a given task, which may not be optimal for the diverse needs of academic research and industrial applications, where a trade-off between speed and accuracy is often necessary.
  • Reliance on Extensive Human Annotation: Developing and updating NLP models often requires significant human annotation, which is costly and time-consuming.

TexSmart aims to address these limitations by providing a more granular, feature-rich, and flexible text understanding system.

Methods

TexSmart introduces several key innovations and approaches:

  • Fine-Grained NER: Instead of a few broad categories, TexSmart supports over 1,000 fine-grained entity types. This is achieved through a semi-automatically built ontology and a combination of unsupervised and hybrid methods. The unsupervised method leverages semantic expansion to derive entity types, while the hybrid method combines this with a coarse-grained NER model for improved robustness.
  • Semantic Expansion: This novel feature provides a list of related entities for a given entity in a sentence, enriching its contextual understanding. The approach involves offline training (Hearst patterns for hyponymy, clustering) and online testing (retrieval and disambiguation). A unique scoring mechanism based on cosine similarity of word embeddings is used to determine cluster relevance.
  • Deep Semantic Representation: For temporal and quantitative entities, TexSmart goes beyond simple recognition to parse them into structured representations (e.g., JSON format for dates). This is achieved using Context Free Grammars (CFGs) and the Earley algorithm, enabling precise interpretation for applications like chatbots.
  • Spectrum of Algorithms: For each core function (e.g., POS tagging, NER), TexSmart implements a range of algorithms, from very fast but less accurate to slower but highly precise ones. This allows users to choose the best trade-off for their specific application needs.
  • Emphasis on Unsupervised/Weakly-Supervised Learning: A core design principle is the use of unsupervised or weakly-supervised algorithms. This aims to facilitate easier model updates with fresh data and reduced reliance on extensive human annotation.

In addition to these unique features, TexSmart also includes traditional NLP modules like word segmentation, POS tagging, coarse-grained NER, constituency parsing, semantic role labeling, text classification, and text matching.

Datasets

The paper mentions the use of several datasets for training and evaluation:

  • Ontonotes 5.0 dataset: Used for training coarse-grained NER models and for creating a larger fine-grained NER dataset by having human annotators label coarse-labeled entities with fine-grained types.
  • CoNLL 2003, BTC, GMB, SEC_FILING, WikiGold, WNUT17: Publicly available datasets used for training English coarse-grained NER models.
  • Penn Treebank (PTB): Used for English POS tagging and constituency parsing experiments.
  • Penn Chinese Treebank (CTB): Used for Chinese POS tagging and constituency parsing experiments.
  • MRPC, QUORA: Datasets used for evaluating English text matching.
  • LCQMC, AFQMC, BQ_CORPUS, PAWS-zh: Datasets used for evaluating Chinese text matching.

The paper also notes the creation of a new fine-grained NER dataset based on Ontonotes 5.0, which includes a sub-ontology of 140 types.

Key Insights and Ideas

  • The Power of Granularity: The paper strongly advocates for fine-grained NER, demonstrating that over 1,000 entity types can be effectively recognized and utilized, offering a significant leap in semantic understanding compared to traditional coarse-grained approaches.
  • Semantic Expansion as a Contextual Enricher: Semantic expansion is presented not just as an entity recognition enhancement but as a tool to provide richer contextual information, which can be invaluable for applications like web search and recommendation systems.
  • Structured Representation for Temporal/Quantitative Data: The ability to parse temporal and quantitative expressions into precise, structured formats is crucial for task-oriented applications like chatbots, where exact interpretation is paramount.
  • Algorithmic Spectrum for Flexibility: Offering a range of algorithms for each task acknowledges the practical trade-offs between speed and accuracy, making the system adaptable to diverse real-world scenarios.
  • Reducing Annotation Burden: The emphasis on unsupervised and weakly-supervised methods is a critical step towards making NLP model development more scalable and cost-effective.

Key Results

  • Semantic Expansion: Achieved an averaged human score of approximately 80.0 for quality on both English and Chinese datasets.
  • Fine-Grained NER: The hybrid approach outperformed the supervised model (LUA), achieving an F1 score of 53.8.
  • POS Tagging: DNN models achieved the highest accuracy (up to 98.08% F1 for Chinese), while Log-linear models were significantly faster (up to 3.9K sents/sec for English).
  • Coarse-Grained NER: DNN models achieved higher accuracy (up to 83.12% F1 for English) compared to CRF models, which were faster (up to 1.1K sents/sec for Chinese).
  • Constituency Parsing: Achieved F1 scores of 95.42 for English and 92.25 for Chinese, with speeds of 16.60 and 16.00 sents/sec, respectively.
  • Semantic Role Labeling: Achieved F1 scores of 86.7 for English and 82.1 for Chinese, with speeds of 10.2 and 11.5 sents/sec, respectively.
  • Text Matching: Both ESIM and Linkage algorithms demonstrated efficient performance, processing thousands of sentences per second. ESIM showed better accuracy on some datasets, while Linkage was competitive.

Limitations and Future Work

  • Unsupervised NER Limitation: The unsupervised fine-grained NER method has a drawback: it cannot recognize entity mentions that are not present in its term clusters.
  • SRL Speed: The speed for Semantic Role Labeling (around 10 sentences per second) is noted as potentially too slow for some applications.
  • English ESIM Training: The English version of the ESIM text matching algorithm was not yet trained due to data availability.
  • Future Work: The authors plan to design more efficient syntactic parsing and SRL algorithms. They also aim to make the English version of ESIM available.

Summary updated

Abstract

This technique report introduces TexSmart, a text understanding system that supports fine-grained named entity recognition (NER) and enhanced semantic analysis functionalities. Compared to most previous publicly available text understanding systems and tools, TexSmart holds some unique features. First, the NER function of TexSmart supports over 1,000 entity types, while most other public tools typically support several to (at most) dozens of entity types. Second, TexSmart introduces new semantic analysis functions like semantic expansion and deep semantic representation, that are absent in most previous systems. Third, a spectrum of algorithms (from very fast algorithms to those that are relatively slow but more accurate) are implemented for one function in TexSmart, to fulfill the requirements of different academic and industrial applications. The adoption of unsupervised or weakly-supervised algorithms is especially emphasized, with the goal of easily updating our models to include fresh data with less human annotation efforts. The main contents of this report include major functions of TexSmart, algorithms for achieving these functions, how to use the TexSmart toolkit and Web APIs, and evaluation results of some key algorithms.
BibTeX
@misc{zhang2020texsmart,
  author = {Haisong Zhang and Lemao Liu and Haiyun Jiang and Yangming Li and Enbo Zhao and Kun Xu and Linfeng Song and Suncong Zheng and Botong Zhou and Jianchen Zhu and Xiao Feng and Tao Chen and Tao Yang and Dong Yu and Feng Zhang and Zhanhui Kang and Shuming Shi},
  title = {{TexSmart: A Text Understanding System for Fine-Grained NER and Enhanced Semantic Analysis}},
  year = {2020},
  eprint = {2012.15639},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.15639},
}
Figure