Online Fraud Detection via Test-Time Retrieval-Based Representation Enrichment

Yiran Qiao, Ningtao Wang, Yuncong Gao, Yang Yang, Xing Fu, Weiqiang Wang, Xiang Ao

In the high-stakes world of online finance, fraud detection is a perpetual arms race. Machine learning models are typically trained on historical data, but fraudsters are professional pivoters who constantly evolve their tactics to exploit new vulnerabilities—a phenomenon known as concept drift. By the time a model is deployed, the "fraudulent patterns" it learned may already be obsolete.

Presented at AAAI-25, the paper "Online Fraud Detection via Test-Time Retrieval-Based Representation Enrichment" introduces TRE (Test-time Retrieval-based Representation Enrichment), a lightweight plug-in that allows fraud detection models to adapt to new patterns at inference time without the massive overhead of constant retraining.

The Challenge of Concept Drift

The core challenge in online fraud detection is concept drift, which manifests in two primary ways:

  1. Incremental Drift: Fraudsters slowly refine their methods over months.
  2. Recurring Drift: Patterns that repeat during specific events, such as the Chinese New Year, where unique spending behaviors create new opportunities for scams.

Standard models are static; they only know what they saw during training. Periodically retraining these models is the traditional fix, but it is prohibitively expensive and slow for massive online platforms. Existing "Test-Time Training" (TTT) methods often rely on auxiliary tasks that don't translate well to the abstract sequence data found in user behavior logs.

The TRE Framework

TRE functions as a "memory bank" for the model. Instead of relying solely on its internal weights, the model looks at recent platform activity to enrich its current decision-making process. The pipeline consists of four main modules:

  1. Index Building: As new, unlabeled test data arrives, an encoder maps these instances into a latent space stored in an efficient retrieval index (e.g., FAISS or HNSW).
  2. Query Generation: New transactions are encoded into a "query" vector.
  3. Retrieval: The system identifies the top-$K$ most similar recent samples from the index.
  4. Response Utilization: This module aggregates neighbors using a Gaussian kernel (giving closer neighbors more influence) and employs an Adaptive-$k$ mechanism to select the optimal number of neighbors for each specific sample.
The TRE architecture showing the entire process (a) and the detailed structure of the aggregator (b).

To handle the extreme class imbalance typical in fraud (where 99% of transactions are legitimate), the authors incorporate a logit offset during training. Additionally, a lightweight non-parametric version using SimHash is provided, offering a 2x speedup with minimal accuracy loss.

Experimental Results and Insights

The researchers evaluated TRE on three massive, real-world industrial datasets from a major Chinese mobile payment platform: General Fraud, Account Takeover (ATO), and Money Laundering. Using an Out-of-Time (OOT) testing setup—training on September 2022 data and testing on February 2023 data—they ensured the models were tested against significant drift.

Key Findings

  • Unlabeled Data as a Signal: TRE demonstrates that we don't need labels to adapt. Knowing a transaction resembles a cluster of other recent transactions provides the distributional context needed to refine judgment.
  • The "Holiday Effect": All baseline models experienced a significant performance dip in January 2023, coinciding with the Chinese New Year. TRE mitigated this dip by "seeing" the shift in behavior through its retrieval bank.
AUC performance over time, highlighting the dip during the Chinese New Year (01/23) and how retrieval-based enrichment (SHORING/TRE) maintains higher accuracy.
  • Performance Boost: On the Fraud dataset, adding TRE to a standard Transformer increased the AUC from 0.7992 to 0.8233. The combination of TextCNN + TRE achieved the highest overall scores, hitting an AUC of 0.8276 on fraud detection and 0.8624 on money laundering.

Limitations and Future Work

While powerful, TRE requires a "warm-up" period to populate the retrieval database and is dependent on the quality of the initial encoder. Future research may explore more complex aggregation methods and integration with graph neural networks (GNNs) to capture social-network-based fraud patterns.

Summary updated

Abstract

Anti-fraud machine learning systems are perpetually confronted with the significant challenge of concept drift, driven by the continuous and intense evolution of fraudulent techniques. That is, outdated models trained on historical fraudulent behaviors often fall short in addressing the evolving tactics of malicious users over time. The key issue lies in effectively tackling the rapid and significant evolution of fraudsters' behaviors to detect these emerging and unforeseen anomalies. In this paper, we propose a solution by directly accessing real-time data and introducing a lightweight plug-in approach named TRE (Test-time Retrieval-based Representation Enrichment). Considering the similarity among samples, TRE employs a retriever to efficiently identify the top-K most relevant recent samples and implements an aggregation strategy to provide neighboring embeddings to the predictor. It thus adjusts the trained classifiers during the test time, providing them with the information from the latest unlabeled data. Extensive experiments on three large-scale real-world datasets demonstrate the superiority of TRE. By consistently incorporating information from the nearest neighbors, TRE demonstrates high adaptability and surpasses existing methods in performance.
BibTeX
@inproceedings{qiao2025online,
  author = {Yiran Qiao and Ningtao Wang and Yuncong Gao and Yang Yang and Xing Fu and Weiqiang Wang and Xiang Ao},
  title = {{Online Fraud Detection via Test-Time Retrieval-Based Representation Enrichment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i12.33359},
}
Figure