Detecting Continuously Evolving Scam Calls under Limited Annotation: A LLM-Augmented Expert Rule Framework

Haoyu Ma, Qinliang Su, Minhua Huang, Wu KaiEMNLP 2025 · OpenReview

TL;DR

Scammers are constantly changing their tactics, making traditional scam call detection methods struggle. This paper introduces a novel framework that uses Large Language Models (LLMs) and expert-defined rules to detect these evolving scams, even with limited labeled data. The approach achieves superior performance by breaking down the detection process into discrimination, reflection, and summarization steps, making it adaptable to new scam scenarios.

What Problem Does This Paper Address?

The proliferation of scam calls, especially on platforms like recruitment, ride-hailing, and delivery services, poses a significant economic and social problem. Existing methods for detecting these calls often rely on large amounts of labeled data and assume that scam patterns remain static. However, scammers are highly adaptive, continuously evolving their deceptive narratives to evade detection. This "data distribution shift" makes models trained on older data quickly become ineffective. The challenge is to build a robust scam detection system that can keep up with these evolving tactics without requiring constant retraining on massive, newly labeled datasets.

Methods

The core of this paper's contribution is a LLM-Augmented Expert Rule Framework. Instead of solely relying on data-driven learning, the authors leverage LLMs' reasoning capabilities guided by human expertise. The framework operates in a hierarchical, few-shot prompting manner:

  1. Pre-Selection: A lightweight LLM is used with broad, loosely defined "short names" for scam types (e.g., "offering many job positions"). This step acts as a filter to quickly identify potentially suspicious calls, reducing the workload for the more powerful LLM. This is done per scam type to maintain focus.
  2. Rule-Guided Detection: Suspicious calls are then passed to a larger LLM. This stage involves three key reasoning steps:
  3. Discrimination Module: The LLM is prompted with expert-defined "scam rules" (natural language descriptions of scam characteristics) to determine if a call exhibits scam traits.
  4. Reflection Module: To combat false positives arising from the similarity between scam and normal calls, the LLM is also prompted with "normal rules" (characteristics of legitimate calls). This step helps to re-evaluate calls flagged as suspicious by the discrimination module, checking if they also possess normal call features.
  5. Summarization Step: The LLM synthesizes the findings from the discrimination and reflection modules to produce a final, structured detection result.

This hierarchical approach, combined with expert rules, allows the LLM to reason about scam characteristics more effectively and adapt to evolving patterns.

Datasets

The authors used two datasets for their experiments:

  • RealScamCall (Real-World Scam Call Dataset): A large-scale, real-world dataset collected from an online recruiting platform. Due to privacy concerns, this dataset cannot be publicly released. It was collected at two different time periods (May and July) to simulate data distribution shifts.
  • SynthScamCall (Synthetic Scam Call Dataset): A publicly available dataset constructed using LLMs to rewrite real-world call texts. This dataset was designed to mimic the characteristics of real-world calls, including data distribution shifts, and is available for research.

Key Insights and Ideas

  • Expert Rules as a Compass: The paper strongly emphasizes that expert-defined rules, expressed in natural language, are crucial for guiding LLMs. These rules act as a "compass," helping the LLM focus on relevant scam characteristics and adapt to evolving tactics, rather than just learning from potentially outdated data.
  • Reflection is Key for Precision: The "reflection module" is a critical innovation. By explicitly prompting the LLM to consider normal call characteristics, the framework significantly reduces false positives, a common issue when distinguishing between similar-sounding scam and legitimate calls.
  • Adaptability Through Rules: The framework's design highlights its inherent adaptability. When new scam types emerge or existing ones evolve, the system can be updated simply by modifying or adding expert rules, without needing to retrain the entire model on new labeled data.
  • Hierarchical Reasoning Pays Off: Decomposing the detection process into pre-selection, discrimination, reflection, and summarization allows the LLM to perform more nuanced reasoning, mimicking how human experts might approach the problem.

Key Results

  • Superior Performance: The proposed LLM-augmented expert rule framework consistently outperformed traditional pre-trained language model-based methods and other prompting methods on both SynthScamCall and RealScamCall datasets.
  • Improved Precision: On the challenging RealScamCall dataset, the method achieved a significant improvement in precision (nearly 0.3) compared to other methods, indicating a better ability to accurately identify scam calls and avoid misclassifications.
  • Effectiveness with Limited Data: The framework demonstrated strong performance even with minimal labeled data, a critical advantage given the high cost of annotation.
  • Robustness to Evolution: The framework showed superior performance in detecting continuously evolving scam calls, a key problem addressed by the paper.
  • Adaptability Demonstrated: The paper showed that the framework could be quickly adapted to a new scam detection scenario by simply changing the expert rules.

Limitations and Future Work

  • Inference Time and Cost: The primary limitation acknowledged is the long inference time and high computational cost associated with using large LLMs, making it challenging for real-time, large-scale deployment.
  • Unfine-tuned Pre-selection Model: The pre-selection stage used unfine-tuned open-source models, which resulted in suboptimal performance. Fine-tuning this component is suggested as a potential improvement.
  • Future Work: The authors plan to focus on optimizing LLM usage to reduce processing time and costs. They also aim to explore using LLMs to assist in training smaller, more efficient models for detection.

Summary updated

Abstract

The increasing prevalence of scam calls, particularly on online platforms for recruitment, ride-hailing, and delivery services, has become a significant social and economic issue. Traditional approaches to scam call detection rely on labeled data and assume a static distribution of scam narratives. However, scammers continuously evolve their tactics, making these methods less effective. In this paper, we propose a novel approach leveraging large language models (LLMs) to detect continuously evolving scam calls. By abstracting scam and normal call rules based on expert knowledge, we develop a hierarchical few-shot prompting framework. This framework consists of a discrimination module to identify scam characteristics, a reflection module to reduce false positives by comparing with normal call features, and a summary step to synthesize the final detection results. Our method is evaluated on real-world and synthesized datasets, demonstrating superior performance in detecting evolving scam calls with minimal labeled data. Furthermore, we show that the framework is highly adaptable to new scam detection scenarios, requiring only modifications to the expert rules.
BibTeX
@inproceedings{ma2025detecting,
  author = {Haoyu Ma and Qinliang Su and Minhua Huang and Wu Kai},
  title = {{Detecting Continuously Evolving Scam Calls under Limited Annotation: A LLM-Augmented Expert Rule Framework}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=FIFPUSKdOm},
}
Figure