Direct optimization of F-measure for retrieval-based personal question answering

Rasool Fakoor, Amanjit Kainth, Siamak Shakeri, Christopher Winestock, Abdel-rahman Mohamed, Ruhi Sarikaya

Abstract

Recent advances in spoken language technologies and the introduction of many customer-facing products have given rise to a wide customer reliance on smart personal assistants for many of their daily tasks. In this paper, we present a system to reduce users' cognitive load by extending personal assistants with long-term personal memory where users can store and retrieve, by voice, arbitrary pieces of information. The problem is framed as a neural retrieval-based question answering system where answers are selected from previously stored user memories. We propose to directly optimize the end-to-end retrieval performance, measured by the F1-score, using reinforcement learning, leading to better performance on our experimental test sets.

Introduction

Smart assistants like Alexa or Siri are increasingly serving as "external brains," allowing users to store arbitrary personal information (e.g., "Remember that I put Ben's phone in the kitchen drawer"). Retrieving this information later is a retrieval-based Question Answering (QA) task: given a user query, the system must identify the relevant "memories" from a potentially large and noisy list of stored items.

This task faces two primary challenges. First, there is a metric mismatch in standard training. Most models are trained using Cross-Entropy (CE) loss, which evaluates individual question-answer pairs. However, real-world performance is measured by the F1-score—a balance of precision and recall—across a group of candidate answers. Because the F1-score is non-differentiable, it cannot be optimized directly using standard gradient descent. Second, the system must handle a massive class imbalance, as the vast majority of stored memories are irrelevant to any specific query.

Methodology

Architectures

We introduced two primary neural architectures for this task:

  • TEFF: A fast, efficient model using fully-connected layers and max-pooling to aggregate information.
  • TEFFCH: An extension of TEFF that incorporates a Character-level CNN (CharCNN) to better handle sub-word information, which is particularly useful for dealing with typos and transcription errors.

Direct F1 Optimization via Reinforcement Learning

To bridge the gap between training loss and evaluation metrics, we employed Reinforcement Learning (RL). Specifically, we used the REINFORCE algorithm, treating the retrieval model as an agent that receives a reward based on the F1-score of its predictions for a given QA group.

To ensure training stability, we utilized Multi-Task Learning (MTL) as a "kick-start" strategy. Models were first pre-trained with standard cross-entropy to establish a baseline understanding before gradually introducing the RL-based F1 optimization. We also designed custom reward functions—referred to as RV1 and RV2 (Reduced Variance)—to help the model navigate the noisy data and penalize false positives and false negatives appropriately.

Experimental Results

We evaluated our models on two distinct test sets: TEST-1 (clean, human-typed questions) and TEST-2 (noisy questions generated by Automatic Speech Recognition).

The proposed TEFF and TEFFCH architectures significantly outperform standard cosine similarity baselines across all thresholds on both clean (TEST-1) and noisy (TEST-2) data.

As shown in the figure above, both TEFF and TEFFCH provide a substantial performance leap over traditional cosine similarity baselines. Interestingly, the Character-CNN (TEFFCH) did not provide a massive boost over the simpler TEFF model. This suggests that because personal memories are typically very short (averaging 3-4 words), pre-trained word embeddings already capture the majority of the necessary context.

Impact of RL Optimization

Directly optimizing for the F1-score consistently yielded better results than the standard cross-entropy baseline or previous "Smooth F1" approximation methods.

Comparison of optimization strategies on TEST-1 and TEST-2. The RL-based methods (RV1, RV2) consistently achieve higher F1-scores than the standard TEFF/TEFFCH baselines.

On the clean TEST-1 set, the TEFFCH model achieved a relative F1 improvement of up to 5.2% when using the RV2 optimization strategy. While improvements were also observed on the noisy TEST-2 set, they were more modest due to the inherent difficulty of recovering from fundamental ASR transcription errors.

Conclusion

By optimizing for the F1-score directly, our models learn to make holistic decisions across an entire set of candidate memories, effectively balancing precision and recall. Our hybrid approach, combining the stability of cross-entropy with the performance ceiling of RL, demonstrates a robust path forward for personal memory retrieval. Future work will focus on improving robustness to ASR noise and exploring curriculum tuning schedules to further enhance performance gains.

Summary updated

Abstract

Recent advances in spoken language technologies and the introduction of many customer facing products, have given rise to a wide customer reliance on smart personal assistants for many of their daily tasks. In this paper, we present a system to reduce users' cognitive load by extending personal assistants with long-term personal memory where users can store and retrieve by voice, arbitrary pieces of information. The problem is framed as a neural retrieval based question answering system where answers are selected from previously stored user memories. We propose to directly optimize the end-to-end retrieval performance, measured by the F1-score, using reinforcement learning, leading to better performance on our experimental test set(s).
BibTeX
@inproceedings{fakoor2018direct,
  author = {Rasool Fakoor and Amanjit Kainth and Siamak Shakeri and Christopher Winestock and Abdel-rahman Mohamed and Ruhi Sarikaya},
  title = {{Direct optimization of F-measure for retrieval-based personal question answering}},
  booktitle = {SLT},
  year = {2018},
  url = {https://openreview.net/forum?id=bkqx_ZunZpC},
}
Figure