SMS Spam Detection Through Skip-gram Embeddings and Shallow Networks

Gustavo Sousa, Daniel Carlos Guimarães Pedronette, João Paulo Papa, Ivan Rizzo GuilhermeACL 2021

TL;DR

This paper introduces a novel text embedding technique for SMS spam detection that leverages "skip-gram" patterns to represent messages as vectors. This approach, combined with shallow machine learning models, achieves competitive accuracy with state-of-the-art deep learning methods while offering better interpretability and efficiency, making it suitable for mobile devices.

What Problem Does This Paper Address?

The proliferation of SMS spam poses a significant annoyance and potential security risk to mobile users. While various methods exist for detecting spam, many rely on simple keyword matching or complex deep learning architectures that can be computationally expensive and less interpretable. This paper aims to bridge this gap by developing a lightweight yet effective embedding technique that captures nuanced patterns in SMS messages, enabling accurate spam classification with shallow models. The authors highlight that simply looking for individual spam-indicative words isn't enough; the context and sequence of words, especially in relation to specific patterns (like "text [WORDS] to"), are crucial discriminators.

Methods

The proposed method consists of three main steps:

  1. Tokenization: SMS messages are first broken down into "attribute-set sequences." This involves splitting text into fragments based on digits, alphabetic characters, or other non-space characters. Alphabetic fragments are further split into sub-word units of length 3. Each fragment is then assigned attributes: type (e.g., digit sequence length, sub-word, other), pos (for sub-words: start, middle, end), and sub (the actual sub-word or fragment).
  2. Skip-gram Pattern Mining: This is the core novelty. The authors define a "k-skip-n-gram" pattern as a tuple of attribute sets. These patterns capture sequences of attributes with a flexible skip distance (up to k+1) between consecutive elements. An algorithm interactively constructs a set of relevant skip-gram patterns by iteratively expanding and scoring candidate patterns based on their support (how many messages they match) and a "GY-Score" that measures dependency between the pattern and the spam/non-spam label.
  3. Embedding Model: A d-dimensional vector representation v(s) for each SMS message s is constructed. This vector is a weighted combination of two components:
  4. u1(s): A binary vector indicating the presence or absence of the top d most informative skip-gram patterns in the message.
  5. u2(s): A vector reflecting contextual information, calculated as the average probability of a pattern matching given relevant attributes in the message. The parameter α (U_ALPHA) controls the weighting between these two components.

Datasets

The primary dataset used is the UCI SMS Spam Collection (Almeida et al., 2011). This dataset contains 4,827 legitimate messages and 747 spam messages, with an average length of 16±12 tokens. The authors used this dataset for both training their embedding model and evaluating its performance. They also re-executed experiments using different data splits to compare with related work.

Key Insights and Ideas

  • Contextual Patterns Over Keywords: The paper strongly emphasizes that the sequence and context of words, captured by skip-gram patterns of attributes, are far more discriminative for spam detection than individual keywords. For instance, the pattern "text [WORDS] to" is a much stronger indicator of spam than just the word "text."
  • Lightweight and Interpretable Embeddings: The proposed skip-gram embedding is designed to be lightweight, suitable for shallow classifiers and even mobile devices, unlike many complex deep learning models. The authors also claim the model allows for interpretation, as the dimensions of the embedding vector correspond to specific, meaningful skip-gram patterns.
  • Attribute-Set Sequences: Representing text not just as tokens but as sequences of attribute sets (type, position, sub-word content) provides a richer representation for pattern mining.
  • GY-Score for Pattern Relevance: The GY-Score effectively balances the probability of a pattern appearing in spam messages against its probability of appearing in non-spam messages, guiding the selection of informative patterns.

Key Results

  • The proposed method, when combined with shallow classifiers like Linear SVM, achieved an average accuracy of 99.03% on 10-fold cross-validation experiments using the UCI SMS Spam Collection.
  • This accuracy is competitive with recent deep learning approaches.
  • Prediction experiments on an ARM Cortex-A53 mobile processor yielded an average processing time of 22 milliseconds per SMS message, highlighting its efficiency.
  • The method showed significant increases in spam detection rates while keeping the rate of blocked legitimate messages (hams) very low (less than 0.5%).
  • A visualization using UMAP showed that the generated embeddings successfully separate spam from non-spam messages in a 2D space.

Limitations and Future Work

  • Limited Dataset Scope: The current evaluation is primarily based on the UCI SMS Spam Collection. Future work could involve evaluating the method on a broader range of textual datasets.
  • Integrating Word Vectors: The authors suggest investigating ways to integrate pre-trained word vectors into their skip-gram mechanism. This could help capture patterns with semantically similar but not identical attributes.
  • Contextual Component Refinement: The u2(s) component of the embedding function, which captures contextual information, is currently based on a simple average of conditional probabilities. More robust predictive models could be explored for this component.
  • Alternative Pattern Definitions: Exploring alternative ways of defining the contextual component of the embedding function is also suggested.

Summary updated

BibTeX
@inproceedings{sousa2021sms,
  author = {Gustavo Sousa and Daniel Carlos Guimarães Pedronette and João Paulo Papa and Ivan Rizzo Guilherme},
  title = {{SMS Spam Detection Through Skip-gram Embeddings and Shallow Networks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure