PENTATRON: PErsonalized coNText-Aware Transformer for Retrieval-based cOnversational uNderstanding

U.n. Niranjan, Ziyan Jiang, Ankit, Sungjin Lee, Jie Hao, Xing Fan, Chenlei Guo

TL;DR

This paper introduces PENTATRON, a novel system designed to improve conversational understanding in intelligent devices by accurately correcting user queries. By combining personalized user history with conversational context, PENTATRON significantly reduces errors in entity recognition, leading to a much smoother user experience.

What Problem Does This Paper Address?

Modern intelligent devices, like smart assistants, often struggle with understanding user queries, leading to frustrating interactions. This misunderstanding stems from various sources, including mispronunciations, background noise, and faulty signal processing. Crucially, two major deficiencies exacerbate these issues: (1) devices aren't tailored to individual users' unique patterns, and (2) they often fail to leverage the ongoing conversation's context. This paper tackles these problems by focusing on entity correction within conversational understanding, aiming to make devices more robust and personalized.

Methods

PENTATRON employs a retrieval-based approach, viewing entity correction as a search problem. Its core components include:

  • Parametric Transformer-based Language Model: This model learns patterns from user-device interactions within a session.
  • Non-Parametric Personalized Entity Index: This index stores historical user data (e.g., frequently played songs, artists) to provide personalized entity suggestions.
  • Bi-encoder Architecture: Based on MiniLM, this architecture jointly encodes queries and entities for efficient comparison.
  • Multi-task Learning: PENTATRON is trained to simultaneously predict the correct entity and its domain. This helps disambiguate entities across different conversational domains (e.g., music vs. smart home).
  • Task Markers (Prompts): Special tokens are used to guide the model during training, further enhancing its ability to handle specific tasks.
  • Hard Negatives Mining: A technique to improve training by identifying and learning from challenging, incorrect predictions.

Datasets

The authors used logs from a commercial voice assistant, specifically multi-turn dialogue sessions from English-speaking users over a one-month period. They filtered for sessions where the last two turns involved a rephrase pair, focusing on cases with a single erroneous entity for prediction. They also mention using a "second" training set for hard negatives mining.

Key Insights and Ideas

  • Personalization and Context are Crucial: The paper strongly demonstrates that combining user-specific history with the ongoing dialogue context is key to overcoming entity recognition errors.
  • Entity Correction as Retrieval: Framing entity correction as a retrieval problem, rather than pure generation, allows for more controlled and stable rewrites, which is vital for production systems.
  • Multi-tasking for Domain Disambiguation: Training the model to predict both the entity and its domain significantly improves its ability to handle ambiguous queries across different application areas.
  • The Power of Prompts: Using task markers (prompts) as special tokens during training provides an additional boost in performance, guiding the model more effectively.

Key Results

PENTATRON achieved remarkable improvements in its primary metric, Exact Match (EM). Across various configurations, the system showed significant gains:

  • Up to 500.97% improvement in Exact Match (relative to the baseline DPR-EC).
  • Even a purely personalized system (PENTATRON-N) without contextualization showed a +432.11% improvement.
  • The fully contextualized and personalized system (PENTATRON-CC) with multi-tasking and prompts achieved the highest gains.
  • In an A/B test, PENTATRON-N showed a 47.5% improvement in user experience and over 40% reduction in turn error rate.
  • The system demonstrated low latency, with a P90 latency of less than 30ms for end-to-end execution at 120 queries per second.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Limited Entity Coverage: Personalization relies on historical data, which can lead to limited coverage of desired entities (around 20% in their studies). Future work could involve multi-level indexes and user clustering.
  • Long Sequence Length: While natural language prompts can improve the system, very long sequences pose latency and memory challenges. Investigating low-rank factorization in attention mechanisms is proposed.
  • Production Deployment Costs: Large-scale in-memory indexes for multiple locales can be costly. Future studies could explore hybrid storage mechanisms and high-performance cache designs.
  • Self-Learning Systems: The authors plan to investigate designing self-learning systems using reinforcement learning.
  • Multi-level Indexing: Exploring multi-level indexing to improve entity coverage and address the cold-start problem for new users is a promising direction.

Summary updated

Abstract

Conversational understanding is an integral part of modern intelligent devices. In a large fraction of the global traffic from customers using smart digital assistants, frictions in dialogues may be attributed to incorrect understanding of the entities in a customer’s query due to factors including ambiguous mentions, mispronunciation, background noise and faulty on-device signal processing. Such errors are compounded by two common deficiencies from intelligent devices namely, (1) the device not being tailored to individual customers, and (2) the device responses being unaware of the context in the conversation session. Viewing this problem via the lens of retrieval-based search engines, we build and evaluate a scalable entity correction system, PENTATRON. The system leverages a parametric transformer-based language model to learn patterns from in-session customer-device interactions coupled with a non-parametric personalized entity index to compute the correct query, which aids downstream components in reasoning about the best response. In addition to establishing baselines and demonstrating the value of personalized and context-aware systems, we use multitasking to learn the domain of the correct entity. We also investigate the utility of language model prompts. Through extensive experiments, we show a significant upward movement of the key metric (Exact Match) by up to 500.97% (relative to the baseline).
BibTeX
@inproceedings{niranjan2022pentatron,
  author = {U.N. Niranjan and Ziyan Jiang and Ankit and SUNGJIN LEE and Jie Hao and Xing Fan and Chenlei Guo},
  title = {{PENTATRON: PErsonalized coNText-Aware Transformer for Retrieval-based cOnversational uNderstanding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=l8otIX2UQzO},
}
Figure