Abstract
This study investigates the viability of ChatGPT as a replacement for traditional supervised spam filters. By comparing Large Language Model (LLM) performance against industry-standard baselines like BERT and SVM, the research identifies a significant performance "flip" based on data availability. While ChatGPT underperforms on large-scale English datasets, it emerges as the superior solution in low-resource scenarios, such as small-scale Chinese datasets, where traditional training data is insufficient for supervised learning.
Introduction
Spam remains a primary vector for cybersecurity threats, ranging from marketing nuisances to sophisticated phishing schemes. Current state-of-the-art spam filters typically rely on supervised models trained on thousands of labeled examples. However, a critical gap exists for languages or niche domains where such massive datasets are unavailable. This paper explores whether the "zero-shot" or "few-shot" capabilities of ChatGPT can provide an effective "out-of-the-box" solution for spam detection without the need for traditional, resource-intensive training.
Methods and Experimental Setup
The researchers evaluated ChatGPT (specifically the GPT-3.5 API) using In-Context Learning (ICL). Unlike traditional models that require weight updates through fine-tuning, ICL utilizes the prompt to guide the model's reasoning. Three configurations were tested:
- Zero-shot: A prompt defining ChatGPT as a "junk mail detection expert" with no examples.
- One-shot: The prompt plus one labeled example of spam and one of ham.
- Five-shot: The prompt plus five labeled examples of each.
These were compared against five baselines: traditional machine learning models (Naive Bayes, SVM, and Logistic Regression using TF-IDF) and deep learning models (DNN and BERT).
Datasets
Two distinct datasets were utilized to test performance across different scales and languages:
- Email Spam Detection Dataset (ESD): A large English dataset (5,572 emails) representing a high-resource scenario.
- Chinese Spam Dataset (CSD): A small dataset (58 emails) compiled to mimic real-world low-resource environments.
Results and Comparative Analysis
The experimental results revealed a stark contrast in performance based on the dataset size. On the high-resource ESD dataset, BERT was the clear leader, achieving an F1-score of 0.98 and an accuracy of 0.99. ChatGPT (5-shot) lagged behind with an F1-score of 0.80.
However, the roles reversed on the low-resource CSD dataset. In this environment, ChatGPT (5-shot) achieved an F1-score of 0.79 and an accuracy of 0.89. Conversely, BERT’s performance plummeted to an F1-score of 0.56, and Naive Bayes fell to 0.42. This demonstrates that while supervised models are "data-hungry" and crumble when data is sparse, ChatGPT maintains stable performance regardless of dataset size.
The Impact of Few-Shot Learning
A key insight from the study is the efficacy of prompt engineering as a form of "mini-training." ChatGPT's performance consistently improved as the researchers moved from zero-shot to five-shot prompts. This suggests that providing even a handful of examples allows the model to calibrate its internal detection logic to the specific nuances of the target dataset.
Discussion and Qualitative Findings
Qualitative error analysis highlighted a specific "blind spot": ChatGPT often misclassifies spam that mimics "small talk" or references public information. The model's conversational pre-training appears to make it more likely to trust emails that adopt a friendly, informal tone, even when they are unsolicited.
Furthermore, while ChatGPT is easier to deploy because it requires no training, it faces practical hurdles. The per-message cost of API calls is significantly higher than running local, specialized models like Naive Bayes. Additionally, API token limits currently restrict the model's ability to process long-form, complex spam emails.
Conclusion
ChatGPT is a robust generalist for spam detection, particularly valuable in low-resource settings where traditional models fail. While it cannot yet match the precision of a fully-trained BERT model on large datasets, its ability to provide high accuracy with minimal examples makes it a powerful tool for emerging threats and niche languages. Future work should explore whether newer iterations like GPT-4 can bridge the performance gap in high-resource environments.
Summary updated