NorBERT: NetwOrk Representations through BERT for Network Analysis and Management

Franck Le, Davis Wertheimer, Seraphin B. Calo, Erich M. Nahum

Introduction

Deep neural networks have revolutionized fields like Natural Language Processing (NLP) and computer vision, but their application to network analysis and management remains a developing frontier. A significant hurdle in this domain is the "generalization gap." Traditional deep learning models often rely on categorical features, such as Fully Qualified Domain Names (FQDNs), treating them as atomic, discrete strings.

When a model is trained in one environment, it learns specific identifiers (e.g., server-54.us-east-1.update.xyz.com). However, when deployed in a different network, it may encounter a variation (e.g., server-27.us-west-2.update.xyz.com). To a standard model, these are entirely unrelated categories, leading to performance degradation. NorBERT addresses this by learning semantically meaningful numerical representations—contextual embeddings—that allow models to generalize across diverse network environments.

Methodology: Adapting BERT for Networks

NorBERT adapts the BERT (Bidirectional Encoder Representations from Transformers) architecture to the specific structure of network data. The core innovation lies in treating sequences of DNS queries similarly to sentences in a language.

Hierarchy-Aware Tokenization

Unlike standard NLP models that use WordPiece tokenization, NorBERT leverages the inherent hierarchy of the DNS system. FQDNs are split by their delimiters (dots) and processed from right to left (Top-Level Domain, then Second-Level Domain, etc.). A truncation hyperparameter $k$ is utilized to manage vocabulary size while preserving the most significant structural information.

Contextual Embeddings and Training

Rather than using static lookup tables like Word2Vec, NorBERT employs a Transformer architecture to generate embeddings based on the sequence of FQDNs. The representation of a specific domain is influenced by its "neighbors" in the traffic sequence.

The model is pre-trained using Masked Language Modeling (MLM). By hiding 15% of the FQDNs in a sequence and requiring the model to predict the missing tokens, NorBERT learns the underlying "grammar" of network traffic—the relationships between different domains—without requiring human-labeled data.

Experimental Setup

The effectiveness of NorBERT was evaluated using three distinct packet capture (pcap) environments:

  1. Smart Lab (UNSW): A public dataset featuring traffic from 21 IoT devices (cameras, switches, etc.).
  2. Smart Lab (Private): A larger lab setup with 65 IoT devices from 31 different vendors.
  3. Large Enterprise Network: A real-world trace from a North American enterprise with over 3,000 hosts, used primarily for unsupervised learning and visualization.

Key Results and Insights

The study compared NorBERT against models using random initialization and context-independent embeddings (such as GloVe).

Bridging the Generalization Gap

The most striking result was observed in device type classification. A standard Gated Recurrent Unit (GRU) model achieved an F1-score of 0.997 on its training set but plummeted to 0.592 when tested on an unseen network. By incorporating NorBERT embeddings, the model maintained a high performance of 0.965 on the same unseen dataset. This demonstrates that the model successfully learned the semantic similarity between different identifiers representing the same underlying services.

Semantic Proximity and Visualization

Using t-SNE visualizations, the researchers confirmed that NorBERT naturally clusters similar entities. For instance, various "mirror" and "repository" servers were grouped together in the embedding space despite having different top-level domains. This proves that domain names are semantically rich and that NorBERT can capture structural and organizational relationships automatically.

Extensibility

Preliminary experiments suggest that this approach extends beyond DNS. NorBERT correctly identified semantic relationships between other categorical fields, such as grouping port 80 (HTTP) closer to port 443 (HTTPS) than to port 22 (SSH), and successfully clustering similar TLS ciphersuites.

Conclusion and Future Work

NorBERT demonstrates that network traffic possesses a "grammar" dictated by software and device purpose that can be captured through Transformer-based pre-training. By mapping categorical identifiers to a rich semantic embedding space, network management models can achieve significantly higher robustness and generalization.

Future research directions include scaling the model beyond the 4-layer architecture used in this study, refining tokenization strategies to better weight hierarchical levels, and investigating the adversarial robustness of these embeddings against attackers attempting to spoof DNS sequences.

Summary updated

Abstract

Deep neural network models have been very successfully applied to Natural Language Processing (NLP) and Image based tasks. Their application to network analysis and management tasks is just recently being pursued. Our interest is in producing deep models that can be effectively generalized to perform well on multiple network tasks in different environments. A major challenge is that traditional deep models often rely on categorical features, but cannot handle unseen categorical values. One method for dealing with such problems is to learn contextual embeddings for categorical variables used by deep networks to improve their performance. In this paper, we adapt the NLP pre-training technique and associated deep model BERT to learn semantically meaningful numerical representations (embeddings) for Fully Qualified Domain Names (FQDNs) used in communication networks. We show through a series of experiments that such an approach can be used to generate models that maintain their effectiveness when applied to environments other than the one in which they were trained.
BibTeX
@inproceedings{le2022norbert,
  author = {Franck Le and Davis Wertheimer and Seraphin B. Calo and Erich M. Nahum},
  title = {{NorBERT: NetwOrk Representations through BERT for Network Analysis and Management}},
  booktitle = {MASCOTS},
  year = {2022},
  url = {https://openreview.net/forum?id=0G6S1be7hn},
}
Figure