Towards Effective Automatic Debt Collection with Persona Awareness

Tong Zhang, Junhong Liu, Chen Huang, Jia Liu, Hongru Liang, Zujie Wen, Wenqiang LeiEMNLP 2023 · OpenReview

TL;DR

This paper introduces PAD, a novel persona-aware automatic debt collection agent that significantly improves recovery rates by tailoring strategies to individual debtor characteristics. By developing a debtor persona taxonomy and a corresponding dataset, PAD demonstrates a 3.31% increase in recovery and an additional ~$100K in collections in a real-world commercial setting, offering a practical solution for the debt collection industry.

What Problem Does This Paper Address?

Collecting overdue debts is a complex and often challenging task, especially for novice debt collectors who lack the experience to effectively navigate diverse debtor excuses and tailor their approach. Existing automatic debt collection agents, while helpful, often fail to account for crucial individual debtor characteristics, such as their financial situation, attitude, and personal circumstances. This lack of personalization leads to less effective collection strategies and missed opportunities. The paper aims to bridge this gap by demonstrating the critical role of "debtor personas" in developing more effective and empathetic debt collection strategies.

Methods

The core of this work lies in the development of a Persona-Aware Debt collection agent (PAD). PAD operates in two main stages:

  1. Persona Extractor (PE): This component first filters out irrelevant conversational turns that don't reveal persona information. It then uses a BERT-based classifier to identify relevant utterances and a UniLM model for abstractive summarization to extract and synthesize debtor personas.
  2. Suggestion Generator: This component takes the conversation history and the extracted debtor personas as input. It employs a BART model enhanced with a Persona-Aware Attention (PAA) mechanism. This PAA mechanism is key, allowing the model to effectively fuse persona embeddings with conversation history embeddings to generate tailored strategies and responses. Unlike previous methods that relied on pre-defined templates, PAD generates dynamic suggestions.

The researchers also introduced a persona taxonomy to categorize debtor characteristics and a persona-aware conversation dataset (PCD) to train and evaluate their models.

Datasets

The paper introduces a novel Persona-aware Conversation Dataset (PCD). This dataset was constructed by collecting and transcribing conversations from 30 experienced debt collectors. It was then meticulously annotated by 22 experienced collectors to identify both debtor personas and collection strategies. The dataset includes statistics on conversations, turns, tokens, and the average number of personas and strategies per conversation.

Key Insights and Ideas

  • Debtor Personas are Crucial: The study strongly emphasizes that understanding and incorporating debtor personas is not just beneficial but essential for effective debt collection. Generic strategies are significantly less effective than personalized ones.
  • Persona Taxonomy is a Novel Contribution: The paper introduces a systematic taxonomy for debtor personas, categorizing them into Financial Health (FH), Family Status (FS), Cooperation Degree (CD), and Debt Status (DS). This provides a structured way to represent and utilize debtor information.
  • Persona-Aware Attention (PAA) Matters: The PAA mechanism in PAD's suggestion generator is highlighted as a key innovation. It allows the model to effectively leverage persona information, leading to superior strategy planning and response generation compared to models that simply concatenate persona information.
  • Commercial Viability Demonstrated: The paper showcases a successful real-world commercial deployment of PAD, proving its practical utility and significant financial benefits beyond academic research.
  • Novice Collector Empowerment: PAD acts as a valuable assistant for novice collectors, bridging the experience gap and improving their performance, thereby reducing training costs.

Key Results

  • Improved Recovery Rate: PAD achieved a 3.31% increase in recovery rate in a two-month online test.
  • Increased Collections: PAD helped collect an additional ~100K RMB in debt.
  • Reduced Call Number: PAD-assisted novices also contributed to the lowest call number (-0.37 improvement).
  • Superior Strategy Planning: PAD consistently outperformed baseline automatic agents in strategy planning, showing an average of 13% better performance in F1 and ROC AUC metrics compared to the SOTA agent (P2T).
  • Enhanced Response Quality: Both automatic and human evaluations indicated PAD's responses were more human-like, with improvements in perplexity, lexical features, and semantic aspects.
  • Persona Effectiveness: Ablation studies showed that incorporating debtor personas significantly improved the performance of various models, including PAD, P2T, and ChatGLM.
  • High-Quality Persona Extraction: Human evaluations of the extracted personas showed an 88% "Reasonable" rating, with good inter-annotator agreement.

Limitations and Future Work

  • Proactive Strategy for Ability: The paper notes that PAD's performance is less significant for debtors' repayment ability (e.g., DS) compared to willingness (CD). They suggest that a future research direction could involve developing proactive strategies that prompt debtors to disclose more information about their financial situation.
  • LLM Deployment Challenges: While LLMs like ChatGLM are powerful, their real-time inference speed can be a bottleneck for high-volume scenarios. The authors plan to explore deploying distilled LLMs or smaller models in the future.
  • Strategic Differences: The analysis of strategy transitions revealed differences between PAD and experienced collectors, particularly concerning married debtors and non-cooperative debtors. Future work could explore these nuanced strategic differences further.
  • Reducing Training Costs: The authors aim to further explore the potential of persona-aware agents in reducing the capital expenditure associated with training and coaching novice agents.

Summary updated

Abstract

Understanding debtor personas is crucial for collectors to empathize with debtors and develop more effective collection strategies. In this paper, we take the first step towards comprehensively investigating the significance of debtor personas and present a successful commercial practice on automatic debt collection agents. Specifically, we organize the debtor personas into a taxonomy and construct a persona-aware conversation dataset. Building upon it, we implement a simple yet effective persona-aware agent called PAD. After two-month online testing, PAD increases the recovery rate by 3.31% and collects an additional ~100K RMB. Our commercial practice brings inspiration to the debt collection industry by providing an effective automatic solution.
BibTeX
@inproceedings{zhang2023towards,
  author = {Tong Zhang and Junhong Liu and Chen Huang and Jia Liu and Hongru Liang and Zujie Wen and Wenqiang Lei},
  title = {{Towards Effective Automatic Debt Collection with Persona Awareness}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=Vr8tBvjkeN},
}
Figure