Introduction
Deep neural networks have revolutionized fields like Natural Language Processing (NLP) and computer vision, but their application to network analysis and management remains a developing frontier. A significant hurdle in this domain is the "generalization gap." Traditional deep learning models often rely on categorical features, such as Fully Qualified Domain Names (FQDNs), treating them as atomic, discrete strings.
When a model is trained in one environment, it learns specific identifiers (e.g., server-54.us-east-1.update.xyz.com). However, when deployed in a different network, it may encounter a variation (e.g., server-27.us-west-2.update.xyz.com). To a standard model, these are entirely unrelated categories, leading to performance degradation. NorBERT addresses this by learning semantically meaningful numerical representations—contextual embeddings—that allow models to generalize across diverse network environments.
Methodology: Adapting BERT for Networks
NorBERT adapts the BERT (Bidirectional Encoder Representations from Transformers) architecture to the specific structure of network data. The core innovation lies in treating sequences of DNS queries similarly to sentences in a language.
Hierarchy-Aware Tokenization
Unlike standard NLP models that use WordPiece tokenization, NorBERT leverages the inherent hierarchy of the DNS system. FQDNs are split by their delimiters (dots) and processed from right to left (Top-Level Domain, then Second-Level Domain, etc.). A truncation hyperparameter $k$ is utilized to manage vocabulary size while preserving the most significant structural information.
Contextual Embeddings and Training
Rather than using static lookup tables like Word2Vec, NorBERT employs a Transformer architecture to generate embeddings based on the sequence of FQDNs. The representation of a specific domain is influenced by its "neighbors" in the traffic sequence.
The model is pre-trained using Masked Language Modeling (MLM). By hiding 15% of the FQDNs in a sequence and requiring the model to predict the missing tokens, NorBERT learns the underlying "grammar" of network traffic—the relationships between different domains—without requiring human-labeled data.
Experimental Setup
The effectiveness of NorBERT was evaluated using three distinct packet capture (pcap) environments:
- Smart Lab (UNSW): A public dataset featuring traffic from 21 IoT devices (cameras, switches, etc.).
- Smart Lab (Private): A larger lab setup with 65 IoT devices from 31 different vendors.
- Large Enterprise Network: A real-world trace from a North American enterprise with over 3,000 hosts, used primarily for unsupervised learning and visualization.
Key Results and Insights
The study compared NorBERT against models using random initialization and context-independent embeddings (such as GloVe).
Bridging the Generalization Gap
The most striking result was observed in device type classification. A standard Gated Recurrent Unit (GRU) model achieved an F1-score of 0.997 on its training set but plummeted to 0.592 when tested on an unseen network. By incorporating NorBERT embeddings, the model maintained a high performance of 0.965 on the same unseen dataset. This demonstrates that the model successfully learned the semantic similarity between different identifiers representing the same underlying services.
Semantic Proximity and Visualization
Using t-SNE visualizations, the researchers confirmed that NorBERT naturally clusters similar entities. For instance, various "mirror" and "repository" servers were grouped together in the embedding space despite having different top-level domains. This proves that domain names are semantically rich and that NorBERT can capture structural and organizational relationships automatically.
Extensibility
Preliminary experiments suggest that this approach extends beyond DNS. NorBERT correctly identified semantic relationships between other categorical fields, such as grouping port 80 (HTTP) closer to port 443 (HTTPS) than to port 22 (SSH), and successfully clustering similar TLS ciphersuites.
Conclusion and Future Work
NorBERT demonstrates that network traffic possesses a "grammar" dictated by software and device purpose that can be captured through Transformer-based pre-training. By mapping categorical identifiers to a rich semantic embedding space, network management models can achieve significantly higher robustness and generalization.
Future research directions include scaling the model beyond the 4-layer architecture used in this study, refining tokenization strategies to better weight hierarchical levels, and investigating the adversarial robustness of these embeddings against attackers attempting to spoof DNS sequences.
Summary updated