Continuous Multi-Task Pre-training for Malicious URL Detection and Webpage Classification

Yujie Li, Yanbin Wang, Haitao Xu, Zhen-Hao Guo, Zheng CaoarXiv 2024 · DOI · arXiv

Abstract

Malicious URL detection and webpage classification are critical tasks in cybersecurity and information management. In recent years, extensive research has explored using BERT or similar language models to replace traditional machine learning methods for detecting malicious URLs and classifying webpages. While previous studies show promising results, they often apply existing language models to these tasks without accounting for the inherent differences in domain data (e.g., URLs being loosely structured and semantically sparse compared to text), leaving room for performance improvement. Furthermore, current approaches focus on single tasks and have not been tested in multi-task scenarios. To address these challenges, we propose urlBERT, a pre-trained URL encoder leveraging Transformer to encode foundational knowledge from billions of unlabeled URLs. To achieve it, we propose to use 5 unsupervised pretraining tasks to capture multi-level information of URL lexical, syntax, and semantics, and generate contrastive and adversarial representations. Furthermore, to avoid inter-pre-training competition and interference, we proposed a grouped sequential learning method to ensure effective training across multi-tasks. Finally, we leverage a two-stage fine-tuning approach to improve the training stability and efficiency of the task model. To assess the multitasking potential of urlBERT, we fine-tune the task model in both single-task and multi-task modes. The former creates a classification model for a single task, while the latter builds a classification model capable of handling multiple tasks. We evaluate urlBERT on three downstream tasks: phishing URL detection, advertising URL detection, and webpage classification. The results demonstrate that urlBERT outperforms standard pre-trained models, and its multi-task mode is capable of addressing the real-world demands of multitasking.

Introduction

While Large Language Models (LLMs) like BERT have revolutionized natural language processing, applying them directly to cybersecurity tasks like malicious URL detection is often a "square peg in a round hole" problem. URLs are fundamentally different from natural language: they are loosely structured, semantically sparse (containing few "words" and many random strings), and filled with noise like session IDs and cache-busters. General-purpose models trained on Wikipedia or books struggle to capture the structural and sequential nuances of a web address. Furthermore, most existing security models are "one-trick ponies"—trained for a single specific task (like phishing detection) and unable to generalize to broader multi-task scenarios required in real-world security operations.

To bridge this gap, researchers have developed urlBERT, a Transformer-based model specifically designed to understand the "language" of URLs. By pre-training on over 3 billion unlabeled URLs using five specialized tasks and a novel staged learning strategy, the model significantly outperforms general-purpose language models in detecting phishing, identifying advertising links, and classifying webpage topics.

Methodology

The urlBERT model adapts the BERT-base architecture but overhauls the training process for the URL domain. The core of the approach lies in how it handles the unique structure of URLs through domain-specific tokenization and a multi-stage pre-training regimen.

The urlBERT architecture featuring two-stage pre-training with five specialized tasks (RTD, STD, MLM, VAT, SCL) and a shared Transformer encoder.

Domain-Specific Tokenization

Instead of using a standard English vocabulary, the researchers trained a custom tokenizer on a massive corpus of URLs. This allows the model to better handle protocol identifiers, paths, and query parameters that are often mangled by standard NLP tokenizers.

Five Specialized Pre-training Tasks

To capture different levels of information, urlBERT employs five distinct tasks:

  • Replaced Token Detection (RTD) & Shuffled Token Detection (STD): These tasks focus on learning the structural "grammar" and sequential dependencies of URLs.
  • Masked Language Modeling (MLM): Used to capture semantic context within the URL strings.
  • Self-supervised Contrastive Learning (SCL) & Virtual Adversarial Training (VAT): These tasks build robustness against noise and small perturbations, such as typo-squatted URLs.

Grouped Sequential Learning (GSL)

To prevent different training tasks from interfering with each other—a common issue in multi-task learning—the authors implemented a two-stage approach. As shown in the architecture diagram, the model is first trained on structural tasks (RTD/STD) to establish a foundational understanding of URL syntax. In the second stage, it moves to more complex semantic and adversarial tasks (MLM, VAT, SCL).

Two-Stage Fine-Tuning

To improve efficiency during downstream task adaptation, a two-stage fine-tuning process is used. The pre-trained encoder is initially frozen to train task-specific layers, followed by a full-stack fine-tuning phase.

Experimental Setup

The scale of the data used to train and evaluate urlBERT is substantial:

  • Pre-training Corpus: A massive collection of 3.28 billion unique URLs curated from sources like Common Crawl, Open PageRank, and various malicious URL repositories.
  • Downstream Tasks:
    • Phishing Detection: Combined GramBedding and Mendeley datasets (~364k samples).
  • Webpage Topic Classification: DmoZ dataset covering five categories like Games, Health, and Shopping (~356k samples).
  • Advertising URL Detection: A custom-crawled dataset of ~88k samples.

Results and Discussion

urlBERT consistently outperformed baselines, including standard BERT, RoBERTa, and traditional CNN/LSTM architectures, across all evaluated metrics.

Performance Metrics

  • Phishing Detection: urlBERT achieved an AUC of 0.9921, significantly higher than standard BERT (0.9759).
  • Webpage Classification: The model reached an AUC of 0.8111, outperforming the best baseline by 7%.
  • Data Efficiency: When trained on only 10% of the available data, urlBERT still achieved 95% accuracy in phishing detection, whereas standard BERT's performance dropped significantly.
  • Training Speed: The two-stage fine-tuning approach (urlBERTc) reduced training time by over 50% compared to standard fine-tuning while maintaining or improving accuracy.

Key Insights

The success of urlBERT highlights several critical principles for domain-specific modeling. First, Structure First, Semantics Second: The GSL strategy suggests that for non-natural languages like URLs, teaching the model syntax before meaning leads to better stability. Second, Robustness via Adversarial Training: Incorporating VAT during pre-training allows the model to ignore "noise" like random session IDs. Finally, Efficiency in Multi-tasking: A single "urlBERT-MT" model can handle multiple security tasks simultaneously with negligible performance loss, making it highly suitable for real-world deployment.

Conclusion and Future Work

While urlBERT represents a significant advancement, there are areas for future improvement. The computational overhead of Transformer models remains a challenge for deployment on mobile devices or real-time browser plugins; model compression techniques like quantization or pruning are potential solutions. Additionally, the evolving nature of the web necessitates research into Class Incremental Learning to incorporate new URL types without full retraining. Finally, the authors aim to explore Few-Shot Learning to detect emerging threats with minimal labeled examples.

Summary updated

Abstract

Malicious URL detection and webpage classification are critical tasks in cybersecurity and information management. In recent years, extensive research has explored using BERT or similar language models to replace traditional machine learning methods for detecting malicious URLs and classifying webpages. While previous studies show promising results, they often apply existing language models to these tasks without accounting for the inherent differences in domain data (e.g., URLs being loosely structured and semantically sparse compared to text), leaving room for performance improvement. Furthermore, current approaches focus on single tasks and have not been tested in multi-task scenarios. To address these challenges, we propose urlBERT, a pre-trained URL encoder leveraging Transformer to encode foundational knowledge from billions of unlabeled URLs. To achieve it, we propose to use 5 unsupervised pretraining tasks to capture multi-level information of URL lexical, syntax, and semantics, and generate contrastive and adversarial representations. Furthermore, to avoid inter-pre-training competition and interference, we proposed a grouped sequential learning method to ensure effective training across multi-tasks. Finally, we leverage a two-stage fine-tuning approach to improve the training stability and efficiency of the task model. To assess the multitasking potential of urlBERT, we fine-tune the task model in both single-task and multi-task modes. The former creates a classification model for a single task, while the latter builds a classification model capable of handling multiple tasks. We evaluate urlBERT on three downstream tasks: phishing URL detection, advertising URL detection, and webpage classification. The results demonstrate that urlBERT outperforms standard pre-trained models, and its multi-task mode is capable of addressing the real-world demands of multitasking.
BibTeX
@misc{li2024continuous,
  author = {Yujie Li and Yanbin Wang and Haitao Xu and Zhen-Hao Guo and Zheng Cao},
  title = {{Continuous Multi-Task Pre-training for Malicious URL Detection and Webpage Classification}},
  year = {2024},
  eprint = {2402.11495},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.11495},
}
Figure