Temporal Context-Aware Spam Detection Using Transformer-Based NLP Models

Fariz Zeynalov, Razim MammadliarXiv 2026 · arXiv

TL;DR

This paper shifts SMS spam detection from isolated message classification to modeling a sender's behavioral history. By augmenting Transformer-based text encoders with inter-message temporal features and causal sequence modeling, the authors demonstrate a more robust way to catch spam campaigns that use "spintax" to evade traditional content filters.

What Problem Does This Paper Address?

Most modern spam filters treat each message as an independent event. However, professional spammers are adept at evading these "stateless" filters using spintax—automatically generating thousands of slight variations of the same message to avoid signature-based detection.

While the content might vary, the behavior of a spammer is often distinct: they send messages in high-volume bursts, at suspiciously regular intervals, or to a massive number of recipients in a short window. This paper tackles outbound SMS spam detection at the carrier level. Unlike inbound filters that protect users, outbound filters help platforms predict if a carrier will block a message at send time, preventing legitimate senders from being penalized and helping platforms maintain their reputation.

Methods

The authors explore three main modeling approaches:

  1. Text-Only Baselines: A standard TF-IDF + Logistic Regression model and a more sophisticated DeBERTa-v3-base classifier to establish how much can be learned from content alone.
  2. DistilBERT + Temporal Features: This is the model currently deployed in production. It takes the [CLS] representation from a DistilBERT encoder and augments it with two temporal features: the time delta since the sender's last message ($\Delta t$) and its logarithm ($\log(1 + \Delta t)$).
  3. TemporalBertSpam (Proposed): A more advanced architecture designed to model the entire sequence of a sender's history. It uses a two-stage process:
  4. Stage 1: Each message is encoded into a vector using DistilBERT.
  5. Stage 2: These vectors, combined with temporal features and sinusoidal positional encodings, are fed into a 4-layer causal Transformer. This allows the model to "remember" previous messages and identify patterns like burst sending across a window of the last 32 messages.

Datasets

The study uses a proprietary production database of 43,741 outbound SMS messages sent over an 89-day period.

  • Labels: Unlike many datasets that rely on human annotation, these labels come from actual carrier delivery statuses (e.g., "delivered" vs. "spam/blocked").
  • Features: Crucially, the dataset includes UTC transmission timestamps at microsecond precision, which are absent from common benchmarks like the SMS Spam Collection.
  • Class Distribution: The set is relatively balanced, with 41.8% spam and 58.2% delivered messages.

Key Insights and Ideas

  • Behavior vs. Content: The authors argue that content-only models are fundamentally limited because they cannot see a sender's history. Two identical messages (e.g., "Your appointment is confirmed") carry vastly different risks if one is a single notification and the other is the 500th message sent in a one-minute burst.
  • Bimodal Confidence: The model's predictions tend to be highly polarized. As seen in the distribution of predicted probabilities, most messages are scored either very close to 0.0 or very close to 1.0, with very few "unsure" cases in the middle.
Figure 2: Distribution of predicted pspam on the 20,000- message production evaluation set (0.05-wide bins). The dashed line marks threshold 0.5. Most predictions fall below 0.20 or above 0.70, with few in between.
  • The "Cold Start" Problem: For a sender's very first message, no history exists. In these cases, the model gracefully degrades to relying solely on content and the "zero" temporal feature.

Key Results

The results demonstrate that while content is a strong signal, temporal context is vital for production-grade reliability.

  • DeBERTa-v3-base (text-only) achieved an AUC-ROC of 0.9011 and an F1 of 0.8412 on a held-out test set.
  • The deployed DistilBERT + temporal model reached an accuracy of 0.8238 and a high recall of 0.9128 for the spam class at a 0.5 threshold. This indicates a design preference for catching as much spam as possible, even at the cost of some false positives.

The authors also performed a threshold sweep to find the optimal balance for production, noting that F1 performance is relatively stable between thresholds of 0.45 and 0.65.

Figure 1: Precision, recall, and F1 across decision thresholds on the production evaluation set (from Ta- ble 3). The dotted line marks the production threshold of 0.55.

Limitations and Future Work

  • Evaluation Mismatch: The text-only baselines and the deployed temporal model were evaluated on different datasets with different class balances, making a direct "head-to-head" comparison difficult.
  • Data Leakage: The initial text baselines used a random split rather than a time-ordered split. This means messages from the same sender or template could appear in both training and testing, potentially inflating scores.
  • Label Noise: Carrier statuses are an imperfect proxy for "true" spam; filtering policies change, and some "spam" labels might be triggered by technical errors rather than malicious intent.
  • Future Directions: The authors plan to evaluate all models on a single, time-ordered split to rigorously answer their research questions. They also suggest exploring newer encoders like ModernBERT and implementing semantic deduplication to better handle paraphrased spintax.

Summary updated

Abstract

Spam detection is usually treated as single-message classification, ignoring behavioral patterns that emerge across sequences of messages from the same sender. We study outbound SMS spam detection at the carrier level, where the goal is to predict at send time whether a carrier will flag a message as spam. Using a proprietary production database of 43,741 outbound SMS messages with UTC transmission timestamps and carrier delivery statuses, we report three sets of results. On an 8,749-message held-out test set, a TF-IDF + logistic regression baseline reaches AUC-ROC 0.8721, and a text-only DeBERTa-v3-base classifier reaches AUC-ROC 0.9011 and F1 0.8412. A deployed DistilBERT model augmented with inter-message temporal features reaches accuracy 0.8238, spam-class F1 0.8382, and recall 0.9128 at threshold 0.5 on a separate, class-balanced 20,000-message production evaluation set; these results are not directly comparable with the held-out test results. We also describe TemporalBertSpam, an architecture that encodes each sender's message history with a causal Transformer, and outline the same-split evaluation needed to measure its contribution.
BibTeX
@misc{zeynalov2026temporal,
  author = {Fariz Zeynalov and Razim Mammadli},
  title = {{Temporal Context-Aware Spam Detection Using Transformer-Based NLP Models}},
  year = {2026},
  eprint = {2611.16139},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16139},
}
Figure