TL;DR
Current addressee recognition models struggle in real-world "noisy" environments because they try to force every utterance—even unrelated background chatter—into a fixed set of known participants. This paper introduces RARM (Robust Addressee Recognition Model), which uses a discrete VQ-VAE-inspired codebook to represent addressees, allowing the system to effectively identify and "categorize" noise rather than misattributing it to a speaker in the conversation.
The Challenge of Noisy Multi-party Conversations
In a multi-party conversation (like a group chat or a busy meeting), "addressee recognition" is the task of identifying who a speaker is talking to. While current AI models perform well in "quiet" datasets, they often break down in real-world applications.
The core issue is the closed-set assumption. Most models are trained to pick an addressee from a pre-defined list of participants. If a "noise" utterance occurs—such as someone nearby asking "What's for lunch?" during a technical meeting—the model is forced to assign that utterance to one of the technical participants. This leads to a total system failure in noisy environments. The authors aim to build a model that is robust enough to recognize when an utterance simply doesn't belong to the current conversation's addressee set.
Methodology: The RARM Architecture
The authors propose the Robust Addressee Recognition Model (RARM). Its architecture consists of three main components designed to move beyond simple classification:
- Transformer Encoder: A standard 12-layer Transformer that takes the query and the conversation context to produce a hidden state representation ($h$).
- Discrete Addressee Codebook: Instead of mapping $h$ directly to a participant ID, the model uses a Vector Quantized-Variational AutoEncoder (VQ-VAE) approach. It maintains a "codebook" (a table of latent embedding vectors). The hidden state $h$ is mapped to the nearest codes in this table. To make the representation richer, the system discretizes the addressee into a set of codes ($Z$) by iteratively calculating the difference between the hidden state and the chosen code.
- Classifier: A final layer that uses the set of codes to predict the addressee (either a specific participant or a "Noise" category).
The model is trained using a combination of standard classification loss and a specialized Addressee Discrete loss (a bag-of-words loss on latent variables), which ensures the codebook captures meaningful semantic information.
Experimental Evaluation
The researchers evaluated their model using the Ubuntu IRC benchmarks, a classic dataset for multi-party dialogue. To test robustness, they created two types of "noisy" versions:
- ID-AN (In-Domain Addressee Noise): They injected random queries sampled from other Ubuntu conversations. This is "hard" noise because it shares the same technical vocabulary as the main conversation.
- OD-AN (Out-Domain Addressee Noise): They injected queries from the DailyDialog dataset (general chit-chat). This is "easier" noise because the topics (e.g., food, weather) differ significantly from technical Ubuntu chat.
Key Insights and Results
RARM consistently outperformed baseline models like BERT and MPC-BERT (a state-of-the-art multi-party model) across all tests. On the Hu et al. (2019) version of the dataset, RARM achieved 85.1% accuracy, compared to BERT's 80.4% and MPC-BERT's 83.8%. The gap widened in noisy conditions; in the "length-10" Ouchi and Tsuboi (2016) dataset, RARM maintained 73.8% accuracy, while the standard BERT model dropped to 67.3%.
Analysis of the model revealed two critical findings:
- Discretization as a Filter: By forcing the model to represent addressees through a discrete codebook, it learns to cluster "types" of utterances. Noise utterances end up mapped to specific codes that the classifier learns to associate with the "Noise" label.
- Semantic Clustering: Visualization (via T-SNE) showed that the codebook naturally organizes itself into semantic clusters. For example, one code might represent "Operating Systems" (close to words like 'ubuntu' and 'linux'), while another represents "Hardware" (close to 'CPU' and 'memory').
Conclusion
The primary limitation acknowledged by the authors is that in-domain noise remains a significant challenge. If the content of the noise is semantically very close to the actual conversation, even the codebook approach struggles to distinguish them perfectly. Future work will focus on improving accuracy on these "hard" in-domain samples and scaling the codebook approach to more complex multi-party environments beyond text-based chats.
Summary updated