Exploring the Relative Value of Collaborative Optimisation Pathways (Student Abstract)

Sudarshan SreeramAAAI 2023 · DOI

Introduction

Deploying machine learning models on "the Edge"—such as microcontrollers (MCUs) in smart speakers or wearables—presents significant challenges due to extremely limited memory and processing power. To address these constraints, researchers employ compression techniques like pruning (removing unnecessary weights), clustering (grouping similar weights), and quantization (reducing weight precision). While these techniques are effective individually, "collaborative optimization" chains them into a multi-stage pipeline. This study investigates whether the increased training overhead of complex pipelines is justified for small, dense models like those used in keyword spotting (KWS).

Methodology

The study utilizes the DS-CNN-S (Depthwise Separable Convolutional Neural Network - Small) architecture as a case study. Four optimization pathways were evaluated using the TensorFlow Model Optimization Toolkit (TF-MOT):

  1. QAT (Quantization-Aware Training): A baseline approach reducing weights to INT8.
  2. PQAT (Pruning + QAT): Pruning weights to specific sparsity levels before quantization.
  3. CQAT (Clustering + QAT): Grouping weights into clusters (8, 16, or 32) before quantization.
  4. PCQAT (Pruning + Clustering + QAT): A full collaborative pipeline involving all three techniques.

Performance metrics, including model size and inference latency, were estimated using Ethos-U Vela for the Arm Ethos-U55 microNPU.

Dataset and Infrastructure

The model was trained on the Google SpeechCommands v0.0.2 dataset, which contains one-second audio clips of 12 distinct keyword labels. Audio data is pre-processed into Mel Frequency Cepstral Coefficients (MFCCs), serving as visual "fingerprints" for the neural network.

Spectrogram visualizations of the 12 keyword labels from the Google SpeechCommands dataset.

To facilitate rapid experimentation, the training infrastructure was optimized by moving MFCC computation to the GPU and implementing parallel pre-fetching. These improvements reduced training time by 3x, from approximately 25 minutes to 8.5 minutes.

Results and Discussion

The experimental results reveal a clear trade-off between optimization complexity and hardware performance.

Performance comparison of different optimization pathways on the DS-CNN-S model.

The Law of Diminishing Returns

While the full PCQAT pipeline achieved the smallest model size (24.66 KB) and fastest inference (113.20 µs), the gains over the simpler PQAT pathway (25.18 KB, 113.42 µs) were marginal. Given that PCQAT requires significantly more training time, the added complexity may not be warranted for "small-dense" models like DS-CNN-S, which already possess high parameter efficiency.

Clustering and Sparsity

Interestingly, CQAT models occasionally outperformed the uncompressed baseline in accuracy, suggesting that weight clustering can serve as a form of regularization. Conversely, the study identified a "catastrophic sparsity" threshold; pruning beyond 50% sparsity led to a sharp decline in model accuracy for this specific architecture.

Conclusion

For small-scale Edge AI models, a full-scale collaborative optimization pipeline often yields diminishing returns. While PCQAT provides the absolute best performance, simpler pathways like PQAT or even basic QAT offer a more efficient balance between training overhead and deployment performance. Future research should explore whether larger, more redundant architectures (e.g., MobileNetV2) derive more substantial benefits from these complex optimization chains.

Summary updated

BibTeX
@inproceedings{sreeram2023exploring,
  author = {Sudarshan Sreeram},
  title = {{Exploring the Relative Value of Collaborative Optimisation Pathways (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27028},
}
Figure