N-Best ASR Transformer: Enhancing SLU Performance using Multiple ASR Hypotheses

Karthik Ganesan, Pakhi Bamdev, Jaivarsan B, Amresh Venugopal, Abhinav TusharACL 2021

TL;DR

This paper proposes a simple yet effective method to improve Spoken Language Understanding (SLU) by concatenating multiple ASR hypotheses as input to Transformer models. This approach achieves state-of-the-art performance on the DSTC2 dataset, particularly excelling in low-data scenarios, and offers a practical solution for users of third-party ASR APIs.

What Problem Does This Paper Address?

Spoken Language Understanding (SLU) systems are crucial for interpreting user utterances in dialog systems, converting spoken language into structured semantic representations like dialog acts and slots. A major hurdle in this process is the inherent noise and errors introduced by Automatic Speech Recognition (ASR) systems. These transcription errors can significantly degrade the performance of downstream SLU tasks. While existing methods have explored using richer ASR outputs like word-lattices or word confusion networks, these often require deep integration with the ASR system. This paper addresses the need for a more accessible approach that leverages multiple ASR hypotheses without requiring such deep integration, aiming to improve SLU performance by effectively utilizing these hypotheses.

Methods

The core of this paper's contribution lies in its novel input representation for Transformer-based SLU models. Instead of relying on a single ASR transcript, the authors propose concatenating the top N-best ASR hypotheses for a user utterance, separated by a special [SEP] delimiter. This concatenated sequence, along with the last system utterance (dialog context), is then fed into pre-trained Transformer encoders like BERT and XLM-RoBERTa. The output of the Transformer encoder is then passed to a Semantic Tuple Classifier (STC) that uses binary classifiers to predict act-slot pairs and multi-class classifiers to predict their corresponding values. This approach is designed to be compatible with standard Transformer architectures and does not require specialized handling of word-lattices or confusion networks.

Datasets

The experiments in this paper are conducted on the DSTC2 dataset. This dataset includes pairs of utterances and their corresponding semantic representations (act-slot-value triplets). For each utterance, the dataset provides manual transcriptions and up to 10-best ASR hypotheses. The dataset is used for training (11,677 samples), development (3,934 samples), and testing (9,890 samples).

Key Insights and Ideas

  • Simplicity is Key: The paper demonstrates that a straightforward concatenation of N-best ASR hypotheses, when fed into powerful pre-trained Transformer models, can yield significant performance gains. This challenges the notion that complex ASR representations are always necessary.
  • Accessibility Matters: By avoiding word-lattices and word confusion networks, the proposed method becomes readily usable with off-the-shelf ASR APIs, democratizing access to advanced SLU techniques.
  • Low-Data Regime Champion: The approach shows remarkable performance improvements in low-data scenarios, highlighting the effectiveness of transfer learning from pre-trained Transformers when data is scarce.
  • Dialog Context is Crucial: Including the last system utterance (dialog context) alongside the user utterance hypotheses helps disambiguate similar ASR outputs and further boosts performance.

Key Results

  • The proposed N-Best ASR Transformer (using BERT and XLM-RoBERTa) achieves performance equivalent to the prior state-of-the-art model on the DSTC2 dataset in terms of F1-score.
  • In the low-data regime (5% and 10% of training data), the N-Best ASR BERT model significantly outperforms the previous state-of-the-art (WCN-BERT STC), achieving up to a 3.2% and 2.2% increase in F1-score respectively.
  • Including dialog context (last system utterance) improved the F1-score by 1.3% and utterance-level accuracy by 1.6% compared to using only the user utterance hypotheses.

Limitations and Future Work

The authors acknowledge several avenues for future research:

  • Generalization to OOV Slot Values: Extending the model to handle out-of-vocabulary slot values.
  • Multi-lingual Evaluation: Assessing the approach's effectiveness in multi-lingual settings.
  • Varying N: Investigating the impact of using different numbers of N-best hypotheses.
  • ASRs with Different WERs: Evaluating the model's performance with ASR systems that have varying Word Error Rates.

Summary updated

BibTeX
@inproceedings{ganesan2021n,
  author = {Karthik Ganesan and Pakhi Bamdev and Jaivarsan B and Amresh Venugopal and Abhinav Tushar},
  title = {{N-Best ASR Transformer: Enhancing SLU Performance using Multiple ASR Hypotheses}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure