An Auxiliary Task Boosted Multi-task Learning Method for Service Account Retrieval with Limited Human Annotation

Yuanzhou Yao, Zhao Zhang, Kaijia Yang, Huasheng Liang, Qiang Yan, Yongjun Xu

Abstract

Service account retrieval in platforms like WeChat presents unique challenges compared to traditional document search, as users seek functional services rather than just text matches. While human-annotated data for ranking is scarce and expensive, user log data is abundant but noisy. We present AuxBoost-MTL, a multi-task learning framework that leverages auxiliary tasks derived from user logs—such as dwell time and retention—to enhance retrieval performance. By employing an Adaptive Hierarchical Fusion (AHF) module that incorporates Knowledge Graphs and Query Entropy, the model intelligently weights different quality signals based on account category and user intent. Experimental results on massive real-world datasets demonstrate that AuxBoost-MTL significantly outperforms state-of-the-art single-task and multi-task baselines.

Introduction

Retrieving service accounts (e.g., mini-programs or official accounts) requires understanding the functional quality of the service. Traditional text-matching approaches are often insufficient for these functional entities. Ideally, models would be trained on extensive human-labeled data, but such annotations are costly and limited. In contrast, industrial platforms generate billions of user log entries. The core challenge addressed in this work is effectively transforming these messy behavioral signals into reliable quality indicators to boost ranking performance when human labels are limited.

Methodology: AuxBoost-MTL

The proposed AuxBoost-MTL architecture is designed to bridge the gap between sparse human labels and dense user logs through three primary components.

Shared and Task-Specific Modules

The framework utilizes a shared-bottom architecture with "expert" networks to extract general features. It branches into five distinct "towers": one main task focused on human-annotated ranking and four auxiliary tasks derived from user behavior: click-through rate, retention, exit clicks, and dwell time.

The AuxBoost-MTL architecture featuring shared layers, task-specific gates, and the Adaptive Hierarchical Fusion (AHF) module.

Adaptive Hierarchical Fusion (AHF)

The AHF module is the central innovation, dynamically weighting auxiliary signals based on context:

  • Knowledge-Aware Gating: By integrating a Knowledge Graph (KG) of 10 million entities, the model understands account categories. For instance, retention is prioritized for games but deprioritized for one-off utility services (like reporting a lost ID).
  • Entropy-Aware Gating: The model calculates "Query Entropy" to distinguish between navigational queries (low entropy, specific intent) and general queries (high entropy, broad intent). This allows the model to prioritize text matching for specific names and service quality for general needs.

Results and Discussion

AuxBoost-MTL was compared against ten single-task models (e.g., DeepFM, DCN) and four state-of-the-art multi-task models (e.g., MMOE, PLE, MetaBalance) using two massive datasets derived from WeChat logs: the Official-Account and Mini-Program datasets.

Performance comparison of AuxBoost-MTL against single-task and multi-task baselines across Mini-Program and Official-Account datasets.

As shown in the results, AuxBoost-MTL consistently outperforms all baselines across metrics like NDCG@5, MRR, and Hit Ratio. Notably, it achieved an 8.5% improvement in HR@1 on the Mini-Program dataset over the strongest multi-task baseline. Ablation studies confirmed that both the Knowledge Graph and Query Entropy components are vital for these gains, as removing either led to performance degradation. This proves that the "intelligent fusion" of these features is what enables the model to effectively utilize noisy log data.

Conclusion

AuxBoost-MTL demonstrates that user logs can serve as powerful proxies for service quality when fused intelligently. By accounting for account categories and query intent, the model provides a robust solution for service retrieval in industrial settings where human annotation is a bottleneck. Future work may address the "cold start" problem for new accounts and the handling of rapidly shifting "trending" query entropy.

Summary updated

Abstract

Service accounts, including organizations’ official accounts and mini-programs, provide various convenient services for users, and have become crucial components of a number of applications. Therefore, retrieving service accounts quickly and accurately is vital. However, this task suffers from the problem of limited human annotation, i.e., manually assessing account functionality and assigning ratings based on user experience is both labor-intensive and time-consuming. To this end, this paper proposes a novel approach, the Auxiliary task Boosted Multi-Task Learning method (AuxBoost-MTL). Specifically, the proposed method introduces multiple auxiliary tasks, which is able to utilized the log data from our application as supervision, and enhance the performance of the main task, service account retrieval. Furthermore, we introduce an Adaptive Hierarchical Fusion Module (AHF module) into our approach. This module is designed to adaptively perform hierarchical fusion of embeddings from auxiliary tasks into the main task, thereby enhancing the model efficacy. Experiments on two real-world industrial datasets demonstrate the effectiveness of our proposed approach.
BibTeX
@inproceedings{yao2023auxiliary,
  author = {Yuanzhou Yao and Zhao Zhang and Kaijia Yang and Huasheng Liang and Qiang Yan and Yongjun Xu},
  title = {{An Auxiliary Task Boosted Multi-task Learning Method for Service Account Retrieval with Limited Human Annotation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-industry.50},
}
Figure