Unlocking the Power of Numbers: Log Compression via Numeric Token Parsing

Siyu Yu, Yifan Wu, Ying Li, Pinjia He

Abstract

Parser-based log compressors have been widely explored in recent years because the explosive growth of log volumes makes the compression performance of general-purpose compressors unsatisfactory. These compressors typically preprocess logs by grouping them based on parsing results before feeding them into general-purpose tools. However, they suffer from misaligned goals, instability based on sample logs, and high processing overhead. We propose Denum, a simple, general log compressor that prioritizes the efficient handling of numeric tokens—which can constitute up to 70% of log data. Denum utilizes a Numeric Token Parsing (NTP) module to extract and process numeric tokens with tailored methods (e.g., delta encoding) and a String Processing module for the remaining content. Evaluated on 16 datasets, Denum achieves an 8.7%–434.7% higher average compression ratio and 2.6x–37.7x faster speeds than state-of-the-art baselines.

Introduction

Modern cloud systems generate logs at a staggering scale, sometimes reaching petabytes per day. While general-purpose compressors like gzip or LZMA are versatile, they fail to exploit the semi-structured nature of logs. Existing "parser-based" compressors attempt to solve this by grouping logs into templates (e.g., "User <*> logged in"). However, this approach has significant flaws. First, log parsing is designed for analysis, not compression; grouping by template often breaks arithmetic sequences of numbers, such as incrementing IDs, that span different templates. Second, performance is often unstable and dependent on training samples. Finally, the parsing step itself is a significant computational bottleneck.

The Denum Architecture

Denum shifts the focus from finding templates to parsing numbers. The framework decouples numeric data from static text to optimize the compression of each independently.

The Denum framework architecture, featuring Numeric Token Parsing and String Processing modules.

As shown in the architecture above, the system consists of two primary modules:

  1. Numeric Token Parsing (NTP): This module extracts numeric tokens (pure numbers, IPs, timestamps, and hex variables) using lightweight regular expressions. These tokens are then tagged, grouped, and processed using mathematical strategies like delta encoding for timestamps and elastic encoding for variable-length integers.
  2. String Processing Module: Once numbers are replaced by tags, the remaining text is processed using a dictionary-index method or can be passed to other specialized log compressors.

Finally, the outputs from both modules are combined and processed by a general-purpose compressor (such as LZMA) to produce the final file.

Numeric Token Parsing (NTP) Logic

The core innovation of Denum lies in how it classifies and handles different numeric patterns. Rather than treating all variables as generic wildcards, Denum applies specific heuristics to maximize redundancy reduction.

Decision logic for classifying and processing different types of numeric tokens.

As illustrated in the decision logic, numeric tokens are categorized into three types:

  • Pure Numbers: These are processed based on their length. For example, very long numbers (length > 15) are treated as generic variables, while mid-length numbers are stored using a combination of their length and first digit to preserve structural patterns.
  • Tokens with Special Characters: Denum uses empirical knowledge to identify specific formats like IP addresses (<I>) or timestamps (<T>). Timestamps are particularly effective for compression when stored as incremental differences (delta encoding).
  • Numeric Variables: Variables that contain both alphabetic and numeric characters are replaced with a generic tag (<*>) to simplify the remaining string structure.

Experimental Evaluation

The researchers evaluated Denum using the Loghub benchmark, comprising over 378 million log entries (77GB) from 16 diverse systems, including HDFS, Windows, and Apache.

Key Results

  • Compression Efficiency: Denum outperformed all baselines on 12 out of 16 datasets. It achieved an average compression ratio (CR) significantly higher than gzip, LZMA, and specialized tools like LogZip.
  • Processing Speed: In a C++ implementation, Denum achieved an average throughput of 26.2 MB/s, making it 2.6x to 37.7x faster than the state-of-the-art LogShrink.
  • Modularity: Integrating Denum’s NTP module into existing compressors like LogShrink improved their compression ratios by approximately 12% and nearly tripled their processing speed.

Conclusion

Denum demonstrates that treating numbers as first-class citizens in log data—rather than just variable placeholders—unlocks massive gains in both storage efficiency and processing speed. By replacing heavy template-based parsing with lightweight numeric extraction and tailored mathematical encoding, Denum provides a scalable solution for the mounting challenges of modern log management. Future work may explore more complex arithmetic correlations between different numeric variables to further enhance compression ratios.

Summary updated

Abstract

Parser-based log compressors have been widely explored in recent years because the explosive growth of log volumes makes the compression performance of general-purpose compressors unsatisfactory. These parser-based compressors preprocess logs by grouping the logs based on the parsing result and then feed the preprocessed files into a general-purpose compressor. However, parser-based compressors have their limitations. First, the goals of parsing and compression are misaligned, so the inherent characteristics of logs were not fully utilized. In addition, the performance of parser-based compressors depends on the sample logs and thus it is very unstable. Moreover, parser-based compressors often incur a long processing time. To address these limitations, we propose Denum, a simple, general log compressor with high compression ratio and speed. The core insight is that a majority of the tokens in logs are numeric tokens (i.e. pure numbers, tokens with only numbers and special characters, and numeric variables) and effective compression of them is critical for log compression. Specifically, Denum contains a Numeric Token Parsing module, which extracts all numeric tokens and applies tailored processing methods (e.g. store the differences of incremental numbers like timestamps), and a String Processing module, which processes the remaining log content without numbers. The processed files of the two modules are then fed as input to a general-purpose compressor and it outputs the final compression results. Denum has been evaluated on 16 log datasets and it achieves an 8.7%-434.7% higher average compression ratio and 2.6x-37.7x faster average compression speed (i.e. 26.2MB/S) compared to the baselines. Moreover, integrating Denum's Numeric Token Parsing into existing log compressors can provide an 11.8% improvement in their average compression ratio and achieve 37% faster average compression speed.
BibTeX
@inproceedings{yu2024unlocking,
  author = {Siyu Yu and Yifan Wu and Ying Li and Pinjia He},
  title = {{Unlocking the Power of Numbers: Log Compression via Numeric Token Parsing}},
  booktitle = {ASE},
  year = {2024},
  doi = {10.1145/3691620.3695474},
}
Figure