A Cooperative Memory Network for Personalized Task-oriented Dialogue Systems with Incomplete User Profiles

Jiahuan Pei, Pengjie Ren, Maarten de RijkeWWW 2021 · DOI · OpenReview

Abstract

There is increasing interest in developing personalized Task-oriented Dialogue Systems (TDSs). Previous work on personalized TDSs often assumes that complete user profiles are available for most or even all users. This is unrealistic because (1) not everyone is willing to expose their profiles due to privacy concerns; and (2) rich user profiles may involve a large number of attributes. In this paper, we study personalized TDSs without assuming that user profiles are complete. We propose a Cooperative Memory Network (CoMemNN) that has a novel mechanism to gradually enrich user profiles as dialogues progress and to simultaneously improve response selection based on the enriched profiles. CoMemNN consists of two core modules: User Profile Enrichment (UPE) and Dialogue Response Selection (DRS). The former enriches incomplete user profiles by utilizing collaborative information from neighbor users as well as current dialogues. The latter uses the enriched profiles to update the current user query so as to encode more useful information, based on which a personalized response to a user request is selected. We conduct extensive experiments on the personalized bAbI dialogue benchmark datasets. We find that CoMemNN is able to enrich user profiles effectively, which results in an improvement of 3.06% in terms of response selection accuracy compared to state-of-the-art methods. We also test the robustness of CoMemNN against incompleteness of user profiles by randomly discarding attribute values from user profiles. Even when discarding 50% of the attribute values, CoMemNN is able to match the performance of the best performing baseline without discarding user profiles, showing the robustness of CoMemNN.

Introduction

Personalization is a critical component for making digital assistants feel helpful and intuitive. To achieve this, systems typically rely on structured user profiles containing attributes like gender, age, and dietary preferences. However, a significant real-world challenge is that user profiles are rarely complete. Users may withhold information for privacy reasons, or the system may not have had sufficient interaction to populate all attributes.

Existing models often struggle when profile data is sparse. For instance, if a system is unaware of a user's vegetarian preference, it might inappropriately suggest a steakhouse. This research addresses two primary challenges: accurately inferring missing profile values using current conversation context and collaborative data, and integrating these inferences in real-time to improve dialogue quality.

The Cooperative Memory Network (CoMemNN)

The authors propose CoMemNN, an architecture that treats profile enrichment and response selection as a joint, "cooperative" task. Unlike static models, CoMemNN views the user profile as a dynamic variable that evolves through dialogue.

The cooperative interaction between user profiles and dialogue in CoMemNN.

The system is composed of two primary modules:

  1. User Profile Enrichment (UPE): This module acts as the "detective." It initializes memory by identifying "neighbor" users with similar traits to predict missing values via collaborative filtering. It then updates this memory using a Multi-Layer Perceptron (MLP) that links dialogue history with profile data. For example, if a user asks about "French fries," the model updates the profile to reflect a likely interest in fast food.
  2. Dialogue Response Selection (DRS): This module utilizes the enriched query from the UPE to match against candidate responses, ensuring the selected response is personalized to the inferred user profile.

The model employs a multi-hop mechanism, iterating through memory several times to refine its understanding of the user before committing to a response. It is trained using a joint loss function that optimizes both response selection and profile prediction accuracy.

Experimental Results

The researchers evaluated CoMemNN using the Personalized bAbI dialogue (PbAbI) dataset, which includes profiles with attributes like gender, age, dietary preference, and favorite food. To test robustness, they simulated real-world conditions by randomly discarding attribute values at rates from 0% to 100%.

Performance and Robustness

CoMemNN achieved state-of-the-art performance, outperforming the best baseline (PMemNN) by 3.06% on the small dataset and 2.80% on the large dataset in terms of Response Selection Accuracy (RSA). Notably, the UPE module correctly predicted missing profile values with over 98% accuracy in most scenarios.

Performance comparison across different discard ratios, highlighting CoMemNN's robustness.

A key finding was the model's resilience to data sparsity. As shown in the experimental results, even when 50% of the profile data was discarded, CoMemNN maintained performance levels comparable to the best baseline models using 100% of the data. While baseline models saw significant performance degradation as profiles became more incomplete, CoMemNN exhibited much higher stability.

Conclusion

CoMemNN demonstrates that dialogue and user profiles can form a "two-way street" where each improves the other. By leveraging collaborative information from similar users, the system can effectively bridge gaps in individual profiles. While the results on the simulated PbAbI dataset are promising, future research should focus on testing the model against the unpredictable nature of real-world human speech and exploring more complex profile structures.

Summary updated

Abstract

There is increasing interest in developing personalized Task-oriented Dialogue Systems (TDSs). Previous work on personalized TDSs often assumes that complete user profiles are available for most or even all users. This is unrealistic because In this paper, we study personalized TDSs without assuming that user profiles are complete. We propose a Cooperative Memory Network (CoMemNN) that has a novel mechanism to gradually enrich user profiles as dialogues progress and to simultaneously improve response selection based on the enriched profiles. Cooperative Memory Network (CoMemNN) consists of two core modules: User Profile Enrichment (UPE) and Dialogue Response Selection (DRS). The former enriches incomplete user profiles by utilizing collaborative information from neighbor users as well as current dialogues. The latter uses the enriched profiles to update the current user query so as to encode more useful information, based on which a personalized response to a user request is selected. We conduct extensive experiments on the personalized bAbI dialogue benchmark datasets. We find that CoMemNN is able to enrich user profiles effectively, which results in an improvement of 3.06% in terms of response selection accuracy compared to state-of-the-art methods. We also test the robustness of CoMemNN against incompleteness of user profiles by randomly discarding attribute values from user profiles. Even when discarding 50% of the attribute values, CoMemNN is able to match the performance of the best performing baseline without discarding user profiles, showing the robustness of CoMemNN.
BibTeX
@inproceedings{pei2021cooperative,
  author = {Jiahuan Pei and Pengjie Ren and Maarten de Rijke},
  title = {{A Cooperative Memory Network for Personalized Task-oriented Dialogue Systems with Incomplete User Profiles}},
  booktitle = {The Web Conference},
  year = {2021},
  doi = {10.1145/3442381.3449843},
}
Figure