More than Text: Multi-modal Chinese Word Segmentation

Dong Zhang, Zheng Hu, Shoushan Li, Hanqian Wu, Qiaoming Zhu, Guodong ZhouACL 2021 · OpenReview

TL;DR

This paper tackles Chinese Word Segmentation (CWS) by moving beyond text-only approaches to incorporate audio information. They introduce a novel "Time-dependent Multi-modal Interactive Network" (TMIN) that effectively fuses text and audio features, achieving state-of-the-art results and demonstrating the significant benefit of multi-modal information for this task.

What Problem Does This Paper Address?

Chinese, like many East Asian languages, lacks explicit word delimiters (spaces) between words. This makes Chinese Word Segmentation (CWS) a crucial, yet challenging, preprocessing step for many Natural Language Processing (NLP) tasks, including named entity recognition, sentiment analysis, and machine translation. While existing CWS models achieve high accuracy on text-only data, they often overlook the rich information present in real-world scenarios like news broadcasts or dialogues, which involve text, audio, and video. The authors argue that relying solely on text can be insufficient, especially when punctuation is absent, making word boundaries ambiguous. They propose to leverage audio information to improve CWS, particularly in scenarios where text alone is not enough to disambiguate word boundaries.

Methods

The core of this paper is the proposed Time-dependent Multi-modal Interactive Network (TMIN). This architecture is designed to capture temporal correspondences and interactive semantics between text and audio. It consists of three main modules:

  1. Time-dependent Uni-modal Interaction:
    • Textual Modality: BERT is used as an encoder to capture contextual character representations from the input text.
  2. Acoustic Modality: Audio features (MFCC, LP-coefficients, FFT spectrum) are extracted using OpenSMILE and then processed by multiple Transformer layers to capture intra-modal interactions.
  3. Crucially, text and audio are aligned using phoneme and character timestamps obtained from the Montreal Forced Aligner.
  4. Time-dependent Multi-modal Interaction:
    • This module employs a Long- and Short-Term Hybrid Memory Gating (LSTHMG) block, an extension of standard LSTMs.
  5. It uses a multi-attention gating mechanism to dynamically control the contribution of each modality's hidden state. This allows the model to mine potential dimension-aware importance for each modality, effectively fusing them.
  6. CRF Labeling:
    • Finally, a Conditional Random Field (CRF) layer is used for sequence labeling. This joint modeling considers correlations between labels in neighborhoods, further refining the segmentation.

Datasets

The authors introduce a newly annotated dataset for CWS containing text and audio. This dataset was collected from a Chinese news reporting platform ("Xuexi") and comprises audio with machine-transcribed text.

  • They crawled 120 short videos, resulting in approximately 2000 sentences.
  • For annotation, they selected 250 sentences to manually annotate word boundaries according to CTB guidelines.
  • The remaining data is reserved for future semi-supervised or unsupervised learning.
  • The annotated data statistics include sentence count, average and maximum lengths (character, word, and time), total characters, total words, and total time.

Key Insights and Ideas

  • Audio is more than just noise: The paper strongly suggests that audio information is not merely a supplementary signal but can provide crucial cues for disambiguating word boundaries, especially in text-heavy scenarios.
  • Time-dependent fusion is key: Simply concatenating features from different modalities (early fusion) is shown to be less effective. The TMIN's time-dependent, interactive fusion mechanism allows for a more nuanced integration of information.
  • Multi-modal interaction improves robustness: The proposed TMIN not only achieves better overall performance but also demonstrates improved accuracy in recognizing Out-of-Vocabulary (OOV) words, suggesting it learns more robust representations.
  • Transformer and LSTM synergy: The combination of Transformer for uni-modal feature extraction and a modified LSTM (LSTHMG) for multi-modal interaction proves effective.

Key Results

  • The proposed TMIN significantly outperforms both single-modal state-of-the-art approaches and multi-modal approaches with early feature fusion.
  • On the newly annotated dataset, the TMIN achieved an overall F-score of 96.62% with 150 training sentences, outperforming baselines like WMSEG and BC.
  • The TMIN also showed superior performance in recalling Out-of-Vocabulary (OOV) words, achieving 96.73% recall with 150 training sentences, compared to baselines that struggled with OOV words.
  • A case study (Figure 3) illustrates how the TMIN correctly segments "历史性" (historic) by leveraging audio cues, while baselines incorrectly split it.

Limitations and Future Work

  • Visual Modality Ignored: The authors acknowledge that visual modality was not utilized for simplicity, as the news videos primarily focused on the scene rather than the speaker's face.
  • Dataset Size: The current annotated dataset is relatively small (250 sentences for training).
  • Future Work:
    • Annotate more samples to expand the dataset.
  • Incorporate more modalities, such as visual information from social media, monologues, and dialogues with continuous front-facing speakers.
  • Explore neural active learning approaches to reduce annotation effort and improve performance.

Summary updated

BibTeX
@inproceedings{zhang2021more,
  author = {Dong Zhang and Zheng Hu and Shoushan Li and Hanqian Wu and Qiaoming Zhu and Guodong Zhou},
  title = {{More than Text: Multi-modal Chinese Word Segmentation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=Ivdv6DN--c},
}
Figure