NorBERT: NetwOrk Representations Through BERT for Network Analysis & Management

Franck Le, Davis Wertheimer, Seraphin B. Calo, Erich M. Nahum

TL;DR

NorBERT adapts the BERT language modeling architecture to learn "contextual embeddings" for network data, specifically Fully Qualified Domain Names (FQDNs). By treating sequences of DNS queries like sentences, it allows deep learning models to generalize across different network environments, boosting device classification F1-scores from 0.59 to 0.96 on unseen datasets.

What Problem Does This Paper Address?

Deep learning is increasingly used for network management tasks like identifying IoT devices or detecting malware. However, these models suffer from a massive "generalization gap." They often rely on categorical features—like the FQDNs a device contacts—but treat them as simple, atomic strings.

If a model learns that a specific camera connects to server-54.us-east-1.update.xyz.com, it will likely fail when deployed in a different region where the same camera connects to server-27.us-west-2.update.xyz.com. To a traditional model, these are two entirely different, unrelated categories. To a human (or a better model), they are clearly the same service. This inability to understand the "semantic" meaning of network identifiers causes models to perform well in the lab but fail in the wild.

Methods

The authors introduce NorBERT, which applies the NLP technique of pre-training Transformers to network traffic.

  1. Hierarchy-Aware Tokenization: Unlike standard BERT which uses WordPiece, NorBERT exploits the DNS hierarchy. It splits FQDNs by their delimiters (dots) and processes them from right to left (TLD, then SLD, etc.). They use a truncation hyperparameter $k$ to control vocabulary size.
  2. Contextual Embeddings: Instead of a static lookup table (like Word2Vec or GloVe), NorBERT uses a Transformer architecture to generate embeddings based on the sequence of FQDNs. If a device queries a series of domains, the representation of each domain is influenced by its "neighbors" in that sequence.
  3. Masked Language Modeling (MLM): The model is trained by hiding 15% of the FQDNs in a sequence and tasking the model with predicting the missing ones. This allows the model to learn the relationships between different domains without needing any human-labeled data.

Datasets

The study utilizes three distinct packet capture (pcap) environments:

  • Smart Lab - UNSW: A public dataset containing traffic from 21 IoT devices (cameras, switches, etc.).
  • Smart Lab - Private: A similar lab setup but larger, featuring 65 IoT devices from 31 different vendors.
  • Large Enterprise Network: A real-world trace from a North American enterprise with over 3,000 hosts (laptops, phones, servers). This was used primarily for unsupervised learning and visualization.

Key Insights and Ideas

  • Network Traffic as Language: The core "aha" moment is that a device's DNS activity is not random; it follows a "grammar" dictated by its software and purpose. A sequence of queries to windowsupdate.com followed by bing.com tells a specific story about the host.
  • FQDNs are Semantically Rich: Domain names aren't just addresses; they contain structural information (e.g., .gov vs .com) and organizational info (e.g., apple.com vs icloud.com). NorBERT successfully captures these relationships.
  • Generalization via Semantics: By learning that server-54 and server-27 are semantically similar in the context of update.xyz.com, the model can handle "unseen" values in new environments. It maps the new value to a known region in the embedding space rather than treating it as an error or "unknown" token.

Key Results

The authors compared NorBERT against models using random initialization and context-independent embeddings (GloVe).

  • Massive Generalization Boost: In a device type classification task, a standard GRU model's performance dropped from 0.997 (on the training set) to 0.592 when tested on a different network. Using NorBERT embeddings, the performance on that same unseen network stayed at 0.965.
  • Semantic Proximity: Using t-SNE visualizations, the authors showed that NorBERT naturally clusters similar entities. For example, it automatically grouped various "mirror" and "repository" servers together in the embedding space, even though they had different top-level domains.
  • Beyond DNS: Preliminary results showed the method also works for other categorical fields. It correctly identified that port 80 (HTTP) is semantically closer to port 443 (HTTPS) than to port 22 (SSH), and grouped similar TLS ciphersuites together.

Limitations and Future Work

  • Model Scale: The authors used a relatively small BERT model (4 layers, 8 heads). Larger models might capture even more nuance but would require more computational resources for real-time network monitoring.
  • Tokenization Complexity: The current tokenization is a simple split-and-truncate method. The authors suggest that more advanced tokenizers that weight hierarchical levels differently could improve results.
  • Downstream Tasks: While the paper focuses heavily on device classification, the authors note that the "downstream evaluation" for other categorical variables (like ports and ciphers) is still in the preliminary stages.
  • Adversarial Robustness: The paper does not explicitly address how an attacker might "spoof" DNS sequences to confuse the embeddings, which is a common concern in network security.

Summary updated

Abstract

Deep neural network models have been very successfully applied to Natural Language Processing (NLP) and Image based tasks. Their application to network analysis and management tasks is just recently being pursued. Our interest is in producing deep models that can be effectively generalized to perform well on multiple network tasks in different environments. A major challenge is that traditional deep models often rely on categorical features, but cannot handle unseen categorical values. One method for dealing with such problems is to learn contextual embeddings for categorical variables used by deep networks to improve their performance. In this paper, we adapt the NLP pre-training technique and associated deep model BERT to learn semantically meaningful numerical representations (embeddings) for Fully Qualified Domain Names (FQDNs) used in communication networks. We show through a series of experiments that such an approach can be used to generate models that maintain their effectiveness when applied to environments other than the one in which they were trained.
BibTeX
@inproceedings{le2022norbert,
  author = {Franck Le and Davis Wertheimer and Seraphin B. Calo and Erich M. Nahum},
  title = {{NorBERT: NetwOrk Representations Through BERT for Network Analysis \& Management}},
  booktitle = {MASCOTS},
  year = {2022},
  url = {https://openreview.net/forum?id=0G6S1be7hn},
}

Loading related content...

Figure