Predicting CNN Learning Accuracy Using Chaos Measurement

Rémi Piau, Thomas Maugey, Aline RoumyWorkshops @ ICASSP 2023 · DOI

TL;DR

This paper introduces a novel "chaos measurement" metric to predict how well a Convolutional Neural Network (CNN) will perform when trained on entropy-coded image data. By quantifying the disruption caused by entropy coding, this metric offers a way to estimate accuracy loss without costly retraining, paving the way for more efficient deep learning on compressed visual data.

What Problem Does This Paper Address?

The increasing volume of visual data necessitates efficient storage and processing. While deep learning models, particularly CNNs, excel at analyzing images, training them directly on compressed data could save significant storage and computational resources by bypassing the decoding step. However, entropy coding, a crucial part of image compression, fundamentally alters the data's structure in ways that are detrimental to CNNs, which rely on spatial and semantic closeness of pixel values. Previous work showed that learning on entropy-coded data is possible but results in accuracy loss, with the cause remaining unclear. This paper aims to quantify this loss and predict it using a new metric.

Methods

The authors propose a new metric, denoted as $D_f(X)$, to measure the "chaos" introduced by an entropy coding function $f$ on input data $X$. This metric is based on the Approximate Entropy (ApEn), a measure of the regularity or predictability of a time series. Specifically, $D_f(X)$ is calculated as the sum of the ApEn of each bitplane of the entropy-coded data, normalized by the sum of the ApEn of the original bitplanes. The intuition is that as entropy coding introduces more randomness (chaos), the ApEn of the coded bitplanes increases, leading to a higher $D_f(X)$. This metric is designed to capture the combined effect of disruptions to both spatial and semantic closeness properties that CNNs rely on.

Datasets

The paper evaluates its proposed metric using two standard image datasets:

  • MNIST: A dataset of handwritten digits.
  • YCIFAR-10: The grayscale version of the CIFAR-10 dataset, containing images from 10 different classes.

Key Insights and Ideas

  • Entropy coding disrupts CNN assumptions: Entropy coding, by design, breaks the spatial and semantic closeness properties that CNNs typically exploit, leading to a performance drop.
  • Chaos as a predictor of accuracy loss: The proposed "chaos measurement" metric, derived from Approximate Entropy of bitplanes, shows a strong correlation with the accuracy loss experienced by CNNs when trained on entropy-coded data.
  • Efficiency gain: This metric allows for predicting learning performance without the need for computationally expensive retraining of the CNN for each entropy coding configuration.
  • Unified measure: The metric effectively captures the combined impact of entropy coding on both spatial and semantic properties, which are difficult to distinguish in the coded domain.

Key Results

The paper demonstrates a good correlation between the proposed chaos measurement metric ($D_f(X)$) and the actual accuracy of a CNN (specifically, UVGG11) on both the MNIST and YCIFAR-10 datasets.

  • For YCIFAR-10, figures show that as $D_f(X)$ increases, the accuracy of the CNN generally decreases. This correlation is observed for both Huffman and arithmetic coding with varying input symbol sizes.
  • Arithmetic coding generally introduces more chaos and results in lower accuracy compared to Huffman coding, which is also reflected in the higher $D_f(X)$ values for arithmetic coding.
  • Similar trends are observed for the MNIST dataset, further validating the metric's predictive power.

Limitations and Future Work

  • Limitations: The paper acknowledges that measuring accuracy loss by retraining the CNN is computationally intensive, which is the primary motivation for their metric. While the correlation is good, the metric is a predictor and not a perfect substitute for actual evaluation. The metric is based on Approximate Entropy, which has its own parameters ($m$ and $r$) that might influence the results.
  • Future Work: The authors suggest that their chaos measurement metric can be used to guide the selection of optimal entropy coding parameters or even to develop new entropy coding schemes that are more amenable to deep learning. Further research could explore the application of this metric to other compression techniques or different types of neural networks.

Summary updated

BibTeX
@inproceedings{piau2023predicting,
  author = {Rémi Piau and Thomas Maugey and Aline Roumy},
  title = {{Predicting CNN Learning Accuracy Using Chaos Measurement}},
  booktitle = {Workshops @ ICASSP},
  year = {2023},
  doi = {10.1109/icasspw59220.2023.10193083},
}
Figure