Abstract
Parser-based log compressors have been widely explored in recent years because the explosive growth of log volumes makes the compression performance of general-purpose compressors unsatisfactory. These compressors typically preprocess logs by grouping them based on parsing results before feeding them into general-purpose tools. However, they suffer from misaligned goals, instability based on sample logs, and high processing overhead. We propose Denum, a simple, general log compressor that prioritizes the efficient handling of numeric tokens—which can constitute up to 70% of log data. Denum utilizes a Numeric Token Parsing (NTP) module to extract and process numeric tokens with tailored methods (e.g., delta encoding) and a String Processing module for the remaining content. Evaluated on 16 datasets, Denum achieves an 8.7%–434.7% higher average compression ratio and 2.6x–37.7x faster speeds than state-of-the-art baselines.
Introduction
Modern cloud systems generate logs at a staggering scale, sometimes reaching petabytes per day. While general-purpose compressors like gzip or LZMA are versatile, they fail to exploit the semi-structured nature of logs. Existing "parser-based" compressors attempt to solve this by grouping logs into templates (e.g., "User <*> logged in"). However, this approach has significant flaws. First, log parsing is designed for analysis, not compression; grouping by template often breaks arithmetic sequences of numbers, such as incrementing IDs, that span different templates. Second, performance is often unstable and dependent on training samples. Finally, the parsing step itself is a significant computational bottleneck.
The Denum Architecture
Denum shifts the focus from finding templates to parsing numbers. The framework decouples numeric data from static text to optimize the compression of each independently.
As shown in the architecture above, the system consists of two primary modules:
- Numeric Token Parsing (NTP): This module extracts numeric tokens (pure numbers, IPs, timestamps, and hex variables) using lightweight regular expressions. These tokens are then tagged, grouped, and processed using mathematical strategies like delta encoding for timestamps and elastic encoding for variable-length integers.
- String Processing Module: Once numbers are replaced by tags, the remaining text is processed using a dictionary-index method or can be passed to other specialized log compressors.
Finally, the outputs from both modules are combined and processed by a general-purpose compressor (such as LZMA) to produce the final file.
Numeric Token Parsing (NTP) Logic
The core innovation of Denum lies in how it classifies and handles different numeric patterns. Rather than treating all variables as generic wildcards, Denum applies specific heuristics to maximize redundancy reduction.
As illustrated in the decision logic, numeric tokens are categorized into three types:
- Pure Numbers: These are processed based on their length. For example, very long numbers (length > 15) are treated as generic variables, while mid-length numbers are stored using a combination of their length and first digit to preserve structural patterns.
- Tokens with Special Characters: Denum uses empirical knowledge to identify specific formats like IP addresses (
<I>) or timestamps (<T>). Timestamps are particularly effective for compression when stored as incremental differences (delta encoding). - Numeric Variables: Variables that contain both alphabetic and numeric characters are replaced with a generic tag (
<*>) to simplify the remaining string structure.
Experimental Evaluation
The researchers evaluated Denum using the Loghub benchmark, comprising over 378 million log entries (77GB) from 16 diverse systems, including HDFS, Windows, and Apache.
Key Results
- Compression Efficiency: Denum outperformed all baselines on 12 out of 16 datasets. It achieved an average compression ratio (CR) significantly higher than gzip, LZMA, and specialized tools like LogZip.
- Processing Speed: In a C++ implementation, Denum achieved an average throughput of 26.2 MB/s, making it 2.6x to 37.7x faster than the state-of-the-art LogShrink.
- Modularity: Integrating Denum’s NTP module into existing compressors like LogShrink improved their compression ratios by approximately 12% and nearly tripled their processing speed.
Conclusion
Denum demonstrates that treating numbers as first-class citizens in log data—rather than just variable placeholders—unlocks massive gains in both storage efficiency and processing speed. By replacing heavy template-based parsing with lightweight numeric extraction and tailored mathematical encoding, Denum provides a scalable solution for the mounting challenges of modern log management. Future work may explore more complex arithmetic correlations between different numeric variables to further enhance compression ratios.
Summary updated