Abstract
In large scale learning, disk I/O for data loading is often the runtime bottleneck. We propose a lossy data compression scheme with a fast decompression to reduce disk I/O, allocating fewer than the standard 32 bits for each feature value. This approach, termed Scaled Rounding, significantly reduces data size and speeds up loading by up to 5.4x with negligible impact on model accuracy.
Introduction
In the era of big data, training machine learning models on massive datasets often exceeds the capacity of a single machine's RAM. Consequently, systems must rely on sequential reads from secondary storage, such as Hard Disk Drives (HDDs). While CPU speeds have increased dramatically, disk I/O speeds have not kept pace, leading to a "Data Loading Bottleneck" where the processor remains idle while waiting for data.
Existing solutions like distributed systems introduce significant complexity, and standard compression tools (e.g., gzip) often decompress too slowly to provide a net runtime benefit. This paper introduces Scaled Rounding, a method designed to compress real-valued feature vectors with extremely fast decompression, specifically tailored for high-performance machine learning pipelines.
Methodology: Scaled Rounding
The proposed method replaces standard 32-bit single-precision floating-point representations with a "coarse" bit-depth (e.g., 8 bits).
Compression and Storage
For each feature vector, the algorithm identifies the maximum absolute value ($x_{max}$) and defines a grid of $2^b$ points, where $b$ is the number of bits. A scale factor $s$ is calculated, and each feature is rounded to the nearest integer on this grid. The system stores these $b$-bit integers alongside a single real-valued scale factor for the entire vector. This per-vector scaling ensures that the available bits are used efficiently regardless of the magnitude of the features within a specific instance.
Decompression
Decompression is designed for maximum speed, consisting of a simple scalar-vector multiplication ($s \times \text{integer vector}$). This operation is computationally trivial and can be performed on-the-fly during the training process, effectively shifting the bottleneck from I/O to the CPU where there is often spare capacity.
Sparse Data Support
For sparse datasets, Scaled Rounding is combined with Variable Byte Code (the "VW method") to compress the gaps between feature indices, further reducing the overall footprint.
Experimental Setup
The effectiveness of Scaled Rounding was evaluated across four diverse public datasets representing various machine learning tasks, ranging from binary classification to regression.
The datasets were initially converted to a binary 32-bit format to establish a rigorous baseline for comparison against the compressed versions.
Results
Accuracy Preservation
Empirical results demonstrate that using as few as 8 bits per value achieves training loss and prediction performance nearly identical to the original 32-bit data. This confirms that the high resolution of 32-bit floats is often unnecessary for convergence in practical ML tasks. The authors show that it is mathematically and empirically better to use a "noisy" version of the entire dataset (via compression) than a "perfect" version of only a fraction of the data (subsampling).
Throughput and Speedup
Scaled Rounding achieved significant speedups in total training time. For the Webspam dataset, a 5.4x speedup in data loading was observed, while the AdClick dataset saw a 3.3x improvement. The method outperformed general-purpose compression tools like LZ4 and specialized toolkits like Vowpal Wabbit (VW) in terms of the compression-speed tradeoff. For example, on Webspam, Scaled Rounding combined with VW index compression reduced data size to 22.6% of the original, whereas VW alone only reached 67.4%.
Conclusion
Scaled Rounding addresses the critical I/O bottleneck in large-scale learning by utilizing coarse feature vectors. By prioritizing decompression speed and per-vector scaling, the method allows for significant reductions in data size and training time without sacrificing model accuracy. While hardware has evolved since the paper's publication, the principle of optimizing data representation to fit within faster memory tiers remains a cornerstone of efficient large-scale machine learning.
Summary updated