Field-Temperature Encoding for Compressed Log Archives

Production systems write logs at petabyte-per-day scale, and two kinds of tools exist for storing them. Archival log compressors split each line into a fixed template (the hard-coded text of a print statement) and its runtime variables, then compress each variable stream separately. They shrink logs by two orders of magnitude or more, but the archive has to be decompressed before anyone can search it. Searchable stores such as Parquet files or full-text indexes answer queries quickly, but they cost several times more space. Every system we found picks one physical format and applies it to all of the data. This proposal argues for a different split. In log analytics, what makes data "hot" is less often its age than which field it is. Operators repeatedly filter on a small set of variables: block IDs, request IDs, error codes, URIs. Most other variables, such as counters, thread IDs and padding fields, are almost never named in a predicate.

The proposal is a single-copy log store that watches which (template, variable) columns queries actually touch, and how. It then re-encodes each column chunk in place. Untouched columns stay in the densest archival encoding. Filtered columns get encodings that support lookups or scans without decompressing the whole block. A cost model decides when re-encoding pays for itself. The claim to test is whether field-level, workload-driven selection can approach the query latency of a uniformly searchable format while keeping close to the footprint of a uniformly archival one. The decisive comparison is against the simpler policy a practitioner would actually deploy: searchable format for recent data, archive for old data.

Core Claim

Log query workloads are concentrated along the field axis. A store that adapts the encoding of each (template, variable, time-chunk) column to the observed predicate types should reach a better storage–latency trade-off than three alternatives: any uniform format, age-based tiering, and data-statistics-only adaptive encoding. It should do this without storing more than one copy.

The "single copy" constraint is what distinguishes the proposal from the closest query-adaptive work. Each chunk exists in exactly one encoding at a time, and adaptation means rewriting it. The "field-level" granularity is what distinguishes it from the closest log work, whose physical layout is fixed.

Relation to Prior Work

Three log compressors supply the decomposition we reuse, and all of them stop at archival output.

  • Logzip clusters a sample of lines to extract templates and groups parameters per template before applying gzip or lzma. It reports 15.1x over plain gzip on Windows logs.
  • LogShrink adds delta encoding for arithmetic variable sequences and a column-oriented layout. The layout alone cuts compressed size by 36%–103%, and it reaches a compression ratio of 456.3 on Windows. It runs at 2.95 MB/s in Python.
  • LogLite skips parsing by using line length as a proxy for template identity. It reaches 2.24 million rows/s.

None of the three can answer a predicate without decompressing. We reuse LogShrink's template/variable columns as the archival encoding. We also treat LogShrink's observation that the top 3–5% of templates carry over half the volume as motivating evidence that access will be similarly skewed.

LogCloud is the closest log system. It builds an FM-index (a compressed substring-search structure) that is chunked for object storage, together with a range-reduced suffix array, and keeps the data in Parquet. It reports an 11.8x geomean smaller footprint than OpenSearch, and 2.2x and 5.0x geomean keyword-search speedups over OpenSearch and LogGrep respectively. Its authors also note that Parquet costs 2.8x more space than LogGrep's domain-specific encoding, and that they accepted this cost to keep SQL interoperability. Our proposal targets exactly that 2.8x. LogCloud pays it uniformly; we pay it only on the columns queries touch.

Query-adaptive compression supplies the selection method, and it also constrains it.

  • HippogriffDB picks compressed formats per query using a 2-approximation greedy algorithm, but it keeps multiple compressed versions of each table. That storage amplification is what our single-copy constraint rules out.
  • Compressed Linear Algebra chooses offset-list, run-length or uncompressed encoding per column group from sampled data statistics, at around 100 MB/s planning bandwidth. It is data-adaptive but not query-adaptive. We use it as the ablation that isolates our key ingredient.
  • HocoPG runs text operators directly on run-length and TADOC-compressed data inside PostgreSQL. It leaves scheme choice to an advisory tool, and it notes that unsupported operators fall back to decompression. That fallback is why we use a cost model that charges for decompression rather than insisting on fully homomorphic encodings.

Research Program

Storage unit and ingestion. Ingestion parses lines with Drain from Tools and Benchmarks for Automated Log Parsing, which reports 0.865 average accuracy across Loghub. Each line becomes a template ID plus variable slots. The storage unit is a column chunk: all values of one variable slot of one template within one time window, sized to a few megabytes so that each chunk is one object-store read. Every chunk starts in the archival encoding, so ingestion cost is LogShrink-like. Ingestion speed is a constraint here, not a contribution. We report it, and we leave substituting LogLite-style grouping as a follow-on.

Encoding menu. Each chunk can hold one of three encodings, or a fourth for string columns:

  • Archival: delta or dictionary transform plus lzma over the whole chunk. Densest, and requires full decompression.
  • Scan: dictionary or frame-of-reference pages with per-page min/max summaries, compressed with zstd. Range and equality filters skip pages and decompress only the matching ones.
  • Lookup: a dictionary plus per-value compressed bitmaps of row positions. A point query reads only the bitmaps, without touching the values.
  • Substring: for high-cardinality string columns such as URIs, LogCloud's chunked FM-index restricted to that column.

Workload log and cost model. The system records each query's predicates as (column, operator class) pairs with exponentially decayed frequencies $f_{c,o}$. Here $c$ is a column, $o$ ranges over equality, range, substring and full scan, and the decay half-life is a tunable parameter. For each chunk $c$, the selector picks the encoding $e$ that minimizes

$$ J(c,e) = \lambda\, S(c,e) + \sum_{o} f_{c,o}\, T(c,e,o) + \frac{R(c,e)}{H}, $$

where:

  • $S(c,e)$ is the chunk's stored size under encoding $e$, estimated by encoding a sample;
  • $\lambda$ converts bytes to the same cost units as time, using the object-store price per byte-month relative to the compute price;
  • $T(c,e,o)$ is the measured expected latency for operator class $o$ on that encoding, including object-store round trips and any decompression;
  • $R(c,e)$ is the one-off cost of rewriting the chunk into encoding $e$;
  • $H$ is the expected horizon over which the current workload persists.

A chunk is rewritten only when the saving in $J$ exceeds a hysteresis margin, which prevents the system from rewriting back and forth. Because each chunk's choice is independent given the workload frequencies, the selection decomposes per chunk. Unlike HippogriffDB's coupled problem, it needs no approximation algorithm. The research questions are whether the independence assumption holds in practice and how fast the decayed estimates track workload shifts.

Analysis. Beyond end-to-end numbers, we will measure how query predicates are distributed over columns. We will also measure the fraction of total bytes held in columns that receive more than a threshold share of queries. If that fraction is not small, the core claim fails before any system comparison.

Baselines

The naive reference point is uniform archival storage: LogShrink's encoding everywhere, with every query decompressing every candidate chunk of the relevant templates. It defines the footprint floor and the latency ceiling. The opposite pole is uniform scan-format storage: Drain-parsed columns in Parquet with zstd and page statistics. It defines the latency the proposal hopes to approach.

The strongest existing system is LogCloud LogCloud. It is the only system in the file built for searching compressed logs on object storage, and it reports a 2.2x geomean speedup over OpenSearch and 5.0x over LogGrep on keyword search, with a footprint 2.8x larger than LogGrep. We will run it as published, on the same object store, the same instance types and the same query sets. Beating it means matching its p95 latency within 1.2x on the evaluated workloads while storing at least 2x fewer bytes. The 2x figure is roughly the gap between its Parquet layout and domain-specific encoding.

The most dangerous practical baseline is age-based tiering. Chunks younger than a cutoff use the scan encoding and older chunks use the archival one. The cutoff is tuned on a held-out workload slice with the same tuning budget we give the decay half-life and hysteresis margin. If field-temperature selection does not Pareto-dominate the best age cutoff, the idea is not worth its complexity.

The ablation that isolates the key ingredient is data-only adaptive selection in the style of Compressed Linear Algebra. It uses the same encoding menu, but each chunk's encoding is chosen from data statistics (cardinality, run lengths, sortedness) to minimize size subject to a fixed decode-speed floor, without looking at queries. It gets the same planning budget. If it matches our system, then query information adds nothing.

Finally, an oracle that sees the full future workload gives an upper bound on what any online policy can achieve. A replicated variant in the style of HippogriffDB, allowed to keep a second encoding of hot chunks within a 1.2x storage budget, tells us whether single-copy rewriting gives up anything meaningful.

Why not just…?

Why not just keep recent logs in Parquet and archive the rest? This is the real threat, because log queries are strongly recency-biased. The proposal's bet is that even within a recent window, most bytes sit in columns nobody filters on. In that case age tiering pays the Parquet premium on every column of recent data, while field-temperature selection pays it only on queried columns. The bet also covers the case where forensic queries reach old data on a few fields, where age tiering pays full decompression. The head-to-head storage–latency curves settle this. We also include a hybrid policy (age × field) to check whether the two axes are complementary rather than competing.

Why not just choose encodings from the data, as Compressed Linear Algebra does? Data statistics tell you which encoding is smallest or fastest to decode. They cannot tell you which columns will be filtered. A high-cardinality request-ID column looks like poor dictionary material, yet it may be the most-queried field. The data-only ablation measures exactly this. If it comes within noise of our system, we will report that query adaptivity is unnecessary for logs.

Why not just run LogCloud, or use Zstd over raw text? General-purpose compressors lose to structural log compressors by large margins: 15.1x on Windows for Logzip, and 16%–356% for LogShrink. They also offer no selective access. LogCloud does offer selective access, but its uniform Parquet layout carries the 2.8x footprint its own authors report. We include it as the main baseline rather than argue past it.

Why not just keep multiple encodings, as HippogriffDB does? At log volumes, each extra copy is a direct multiple of the dominant cost. The 1.2x-budget replicated variant quantifies how much latency a bounded second copy would buy. If it is large, a hybrid of rewriting and limited replication becomes the better design, and we will say so.

Proposed Experiments

Data. We will use the Loghub datasets from Loghub. The emphasis is on the large ones LogCloud also used, HDFS, Thunderbird, Hadoop and Windows, plus BGL and Spark. Data is stored on an S3-compatible object store with realistic per-request latency.

Workloads. Loghub ships no query traces, which is the main threat to validity, so we construct workloads three ways:

  • Investigative sessions derived from the labeled anomaly datasets. For each labeled anomaly in HDFS or BGL: look up its block or node identifier, filter the error templates around it, then aggregate over its time range.
  • Keyword and URI substring queries following LogCloud's query classes, for direct comparability.
  • Dashboard workloads with periodic aggregations over numeric variables.

Each workload is generated with Zipf-skewed field popularity. We report results across skew levels, including uniform, which is where we expect the proposal to lose.

Metrics. We report:

  • stored bytes per raw gigabyte;
  • p50 and p95 query latency;
  • object-store requests per query;
  • bytes rewritten per day;
  • a total-cost-of-ownership curve over query volume, in LogCloud's style.

Decisive checks. First, the field-concentration measurement: the fraction of bytes in columns receiving most queries. Second, Pareto comparison against the best-tuned age tiering. Third, the data-only ablation. Fourth, a drift experiment that switches workloads mid-stream and measures rewrite volume and the time until latency returns to steady state, compared with the oracle. A positive result requires the proposal to dominate age tiering and the data-only ablation on at least the skewed workloads, and to come within 1.5x of the oracle's cost. If either dominance fails, the honest conclusion is that log hotness is temporal, not field-wise, and the simpler policy should be used.