PU GNN: Chargeback Fraud Detection in P2E MMORPGs via Graph Attention Networks with Imbalanced PU Labels

Jiho Choi, Junghoon Park, Woocheol Kim, Jin-Hyeok Park, Yumin Suh, Minchang Sung

TL;DR

This paper introduces PU GNN, a novel method for detecting chargeback fraud in Play-to-Earn (P2E) MMORPGs. By combining player behavior logs with transaction graph data and employing a Positive-Unlabeled (PU) learning approach to handle imbalanced and partially labeled data, PU GNN significantly outperforms existing methods.

What Problem Does This Paper Address?

The rise of Play-to-Earn (P2E) Massively Multiplayer Online Role-Playing Games (MMORPGs) has blurred the lines between virtual in-game assets and real-world value. Players can now exchange in-game goods for cryptocurrencies, making these assets directly convertible to real-world currency. However, this also introduces a significant challenge: chargeback fraud. Unlike traditional in-game items, P2E assets recorded on the blockchain are difficult to recover once fraudulent transactions occur (e.g., payment fraud, cancellations, or refunds). Existing fraud detection methods often struggle with the unique characteristics of P2E MMORPGs, particularly the imbalanced nature of fraud data and the difficulty in obtaining definitive labels for all players, as some players who haven't yet committed fraud might do so later. This paper aims to address this gap by developing a robust chargeback fraud detection system specifically for P2E MMORPGs.

Methods

The proposed PU GNN model integrates two key data sources: player in-game activity logs and P2E token transaction graphs. The architecture consists of four main components:

  1. Player Behavior Modeling: This layer processes players' sequential in-game activity logs (e.g., logins, item purchases, quest completions) using a self-attention mechanism. This captures nuanced behavioral patterns that might indicate fraudulent activity. The output is an initial behavioral representation for each player.
  2. Graph Attention Networks (GATv2): The P2E token transaction graph, where nodes represent players and edges represent token transfers, is utilized. GATv2 is employed to learn contextual relationships between players by aggregating information from their neighbors in the transaction network. This component captures how players interact and transact, which can reveal collusive or fraudulent behavior.
  3. Imbalanced Label Handling (GraphSMOTE): To address the severe imbalance between fraudulent and benign players, the authors adapt GraphSMOTE. This technique oversamples the minority class (fraudulent players) by generating synthetic graph data, helping to balance the dataset.
  4. Positive & Unlabeled Learning (PU Loss): A crucial aspect of the method is its handling of unlabeled data. In the context of chargeback fraud, players who have not yet committed fraud but could potentially do so are treated as unlabeled. The PU learning framework, specifically using a non-negative PU (nnPU) risk estimator, allows the model to learn from both confirmed fraud (positive) and unlabeled instances, rather than treating all unlabeled instances as definitively benign. This is essential because players might cancel payments after mining P2E tokens.

The model combines the player behavior representations and the graph-based representations, then uses a PU loss function to train the classifier.

Datasets

The authors evaluated their model on three real-world datasets collected from two popular P2E MMORPGs. These datasets consist of player behavior activity logs and P2E token transaction graphs. The game and token titles are anonymized as MMORPG01, MMORPG02, TOKEN01, TOKEN02, and TOKEN03. The datasets were split into training, validation, and test sets. For the PU learning setup, half of the positive (fraud) labels were treated as positive, and the other half were treated as unlabeled.

Key Insights and Ideas

  • The Value of Integrated Data: Combining player behavior logs with transaction graph data is crucial for effective chargeback fraud detection in P2E MMORPGs. Neither source alone is sufficient; their synergy provides a more comprehensive view of player activities.
  • Unlabeled Data is Not Necessarily Benign: In the P2E context, players who haven't yet committed fraud but have paid for in-game goods are best treated as "unlabeled" rather than definitively "benign." This is because they might still initiate a chargeback after acquiring P2E tokens. PU learning effectively handles this uncertainty.
  • GraphSMOTE for Imbalanced Graphs: Adapting SMOTE for graph-structured data (GraphSMOTE) is effective in mitigating the severe class imbalance common in fraud detection tasks, leading to better model performance.
  • Automated Fraud Patterns: Chargeback fraud in P2E MMORPGs often stems from automated programs (macros) rather than individual manual play, leading to distinct, collusive activity patterns that can be detected.

Key Results

The PU GNN model demonstrated superior performance across all three datasets compared to various baseline models, including MLP, GRU, GCN, GraphSAGE, and GAT.

  • F1-score and AUC: PU GNN achieved the best F1-scores and AUC values on all datasets. For instance, on DATASET01, PU GNN achieved an F1-score of 0.855 and an AUC of 0.854, outperforming the next best baseline (DAGNN) which scored 0.846 and 0.835 respectively.
  • Sensitivity and Specificity: The model also showed strong performance in terms of sensitivity (true positive rate) and specificity (true negative rate), indicating its ability to correctly identify both fraudulent and benign players.
  • Ablation Studies: Ablation studies confirmed the effectiveness of both the PU learning approach and the GraphSMOTE oversampling technique. Models trained without these components performed significantly worse.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Temporal Features: The current model does not explicitly consider time-related temporal features, such as the time interval between payment and cancellation, or collusive work times. Incorporating survival analysis techniques like hazard functions could enhance detection.
  • Heterogeneous Graphs: The current graph representation is homogeneous. Extending the model to handle heterogeneous graphs with various node and edge types could capture more complex relationships.
  • Early Detection: Developing methods for earlier detection of chargeback fraud is crucial for minimizing losses.
  • Unlabeled Data Interpretation: While PU learning is effective, further research into the precise interpretation and utilization of unlabeled data could yield even better results.

Summary updated

Abstract

The recent advent of play-to-earn (P2E) systems in massively multiplayer online role-playing games (MMORPGs) has made in-game goods interchangeable with real-world values more than ever before. The goods in the P2E MMORPGs can be directly exchanged with cryptocurrencies such as Bitcoin, Ethereum, or Klaytn via blockchain networks. Unlike traditional in-game goods, once they had been written to the blockchains, P2E goods cannot be restored by the game operation teams even with chargeback fraud such as payment fraud, cancellation, or refund. To tackle the problem, we propose a novel chargeback fraud prediction method, PU GNN, which leverages graph attention networks with PU loss to capture both the players' in-game behavior with P2E token transaction patterns. With the adoption of modified GraphSMOTE, the proposed model handles the imbalanced distribution of labels in chargeback fraud datasets. The conducted experiments on three real-world P2E MMORPG datasets demonstrate that PU GNN achieves superior performances over previously suggested methods.
BibTeX
@inproceedings{choi2022pu,
  author = {Jiho Choi and Junghoon Park and Woocheol Kim and Jin-Hyeok Park and Yumin Suh and Minchang Sung},
  title = {{PU GNN: Chargeback Fraud Detection in P2E MMORPGs via Graph Attention Networks with Imbalanced PU Labels}},
  booktitle = {ECML/PKDD},
  year = {2022},
  doi = {10.1007/978-3-031-43427-3\_15},
}
Figure