Constructing Interpretive Spatio-Temporal Features for Multi-Turn Responses Selection

Junyu Lu, Chenbin Zhang, Zeying Xie, Guang Ling, Tom Chao Zhou, Zenglin XuACL 2019 · DOI · OpenReview

TL;DR

This paper introduces the Spatio-Temporal Matching (STM) network, a novel approach for response selection in multi-turn dialogues. By leveraging 3D convolutions on attention maps, STM effectively captures local semantic relationships across both text segments and their temporal evolution, leading to significant performance improvements over existing methods.

What Problem Does This Paper Address?

In automated dialogue systems, selecting the most appropriate response from a large set of candidates is crucial. This task, known as response selection, is challenging due to the inherent semantic gap between dialogue context and potential replies, and the sheer volume of candidates. Existing methods often struggle to capture the nuanced, localized temporal dependencies that characterize natural conversations, where the relevance of a response might hinge on a specific recent turn rather than the entire dialogue history.

Methods

The proposed Spatio-Temporal Matching (STM) network tackles this by introducing a novel way to represent and match dialogue context with candidate responses. The core idea is to treat the attention maps generated between context utterances and candidate responses as "images" that evolve over time.

Here's a breakdown of the key components:

  • Dual Encoders: The model first uses bidirectional GRUs (or pre-trained BERT embeddings) to encode both the dialogue context and each candidate response, capturing sequential information within each utterance.
  • Spatio-Temporal Matching Block: This is the heart of the STM.
  • Soft Alignment: An attention mechanism is used to compute local relevance between words in context utterances and words in candidate responses, generating attention matrices.
  • Spatio-Temporal Feature Construction: These attention matrices, representing local matching information, are aggregated across the temporal dimension (dialogue turns) to form a 4D "cube." This cube can be thought of as a sequence of attention images.
  • 3D Convolutional Neural Networks (3D CNNs): Inspired by video analysis, 3D CNNs are applied to this spatio-temporal cube. This allows the model to extract local matching features not just within a single turn's attention map (spatial) but also across the sequence of attention maps over time (temporal). This captures how matching patterns evolve.
  • Max-Pooling: 3D pooling operations are used to downsample the feature maps, further consolidating the extracted spatio-temporal information.
  • Classification: Finally, a fully-connected layer predicts a matching score for each candidate response, and a softmax cross-entropy loss is used for training.

Datasets

The paper evaluates the STM model on two large-scale multi-turn response selection datasets:

  • Ubuntu Corpus: Extracted from Ubuntu chat logs, this dataset consists of nearly a million two-person conversations used for technical support. It's noted for having a large set of candidates.
  • Advising Dataset: This dataset comprises advisor dialogues aimed at guiding students in course selection, considering curriculum, personal preferences, and career paths. It offers a substantial number of partial conversations and candidates.

Key Insights and Ideas

  • Treating Attention as Spatio-Temporal Data: The most significant insight is the novel idea of treating attention maps as a temporal sequence of images and applying 3D CNNs. This allows the model to capture dynamic matching patterns that are missed by traditional 2D approaches.
  • Local Temporal Relevance is Key: The paper emphasizes that in multi-turn dialogues, relevance often appears in local time periods. The STM's ability to aggregate and process attention across turns directly addresses this by capturing how matching evolves.
  • Interpretability through Visualization: The authors highlight that the spatio-temporal features are not only effective but also interpretable. Visualizing attention maps reveals how the model identifies important word-level and even segment-level matches across time, providing insights into the model's decision-making process.
  • 3D CNNs for Text Matching: The successful application of 3D CNNs, typically used for video analysis, to text-based dialogue tasks demonstrates the versatility of these architectures for capturing spatio-temporal patterns in sequential data.

Key Results

The STM model achieved state-of-the-art performance on both the Ubuntu Corpus and the Advising Dataset.

  • Ubuntu Corpus: The STM (Ensemble) model achieved an R100@10 of 0.797 and an MRR of 0.616, outperforming other models like DAM and DME. The STM (BERT) variant further improved these metrics.
  • Advising Dataset: The STM (Ensemble) model achieved an R100@10 of 0.385 and an MRR of 0.502, also demonstrating superior performance compared to baselines.
  • Visualization: Qualitative analysis showed that the model could effectively identify relevant word-level matches (e.g., "vlc," "wma" matching "drm") and capture segment-level and sentence-level information through its spatio-temporal processing.

Limitations and Future Work

  • Limitations: The paper notes that while 3D CNNs are effective, using more than three layers can degrade performance in their experiments. The computational complexity of cross-attention is also mentioned as a concern for large candidate sets, leading to the use of dual-encoder architectures.
  • Future Work: The authors express interest in exploring the effectiveness of various attention methods to handle more complex, ambiguous choices in dialogue and further leveraging interpretive features. They also suggest exploring different granularities of attention (e.g., segment-level vs. word-level) for matching.

Summary updated

Abstract

Response selection plays an important role in fully automated dialogue systems. Given the dialogue context, the goal of response selection is to identify the best-matched next utterance (i.e., response) from multiple candidates. Despite the efforts of many previous useful models, this task remains challenging due to the huge semantic gap and also the large size of candidate set. To address these issues, we propose a Spatio-Temporal Matching network (STM) for response selection. In detail, soft alignment is first used to obtain the local relevance between the context and the response. And then, we construct spatio-temporal features by aggregating attention images in time dimension and make use of 3D convolution and pooling operations to extract matching information. Evaluation on two large-scale multi-turn response selection tasks has demonstrated that our proposed model significantly outperforms the state-of-the-art model. Particularly, visualization analysis shows that the spatio-temporal features enables matching information in segment pairs and time sequences, and have good interpretability for multi-turn text matching.
BibTeX
@inproceedings{lu2019constructing,
  author = {Junyu Lu and Chenbin Zhang and Zeying Xie and Guang Ling and Tom Chao Zhou and Zenglin Xu},
  title = {{Constructing Interpretive Spatio-Temporal Features for Multi-Turn Responses Selection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1006},
}
Figure