Does synthetic pre-training pay off on a microcontroller? Accuracy, latency and energy across two MCUs

Achintya YadavarXiv 2026 · arXiv

TL;DR

Synthetic pre-training (using rendered images) offers negligible accuracy gains for microcontroller-scale models when sufficient real-world data is available. However, it becomes a powerful tool when real data is scarce, providing a significant boost (~3 percentage points) for ultra-small CNNs. Crucially, the study reveals that vendor-optimized kernels and host-side interpreters often fail to match reference arithmetic, making on-device verification essential.

What Problem Does This Paper Address?

In the world of TinyML, training data is the most expensive ingredient. Researchers often look to "synthetic" data—images rendered by game engines—as a cheap, infinitely scalable alternative to manual labeling. While this works for massive models on GPUs, the "Tiny" in TinyML introduces two major hurdles: capacity and precision.

First, can a model with only 60K parameters even "remember" the features learned from synthetic data once it's fine-tuned on real images? Second, does the benefit of pre-training survive INT8 quantization? Most microcontrollers (MCUs) lack floating-point units efficient enough for deep learning, requiring models to be squashed into 8-bit integers. This paper investigates whether the effort of building a synthetic training stage actually pays off in terms of accuracy, latency, and energy once the model hits the silicon.

Methods

The author re-examines a previous study (Bird et al.) using a rigorous, pre-registered experimental design. The methodology centers on two structures:

  1. The Four-Arm Comparison: To isolate the effect of synthetic data, models were initialized in four ways:
  2. Arm A: Random weights.
  3. Arm B: ImageNet weights (standard transfer learning).
  4. Arm C: Random weights $\rightarrow$ Synthetic pre-training $\rightarrow$ Real data.
  5. Arm D: ImageNet weights $\rightarrow$ Synthetic pre-training $\rightarrow$ Real data.
  6. The Model Ladder: Five models of decreasing size were tested, ranging from a full MobileNetV2 (2.27M parameters) down to a tiny 61K-parameter CNN.
  7. Hardware Deployment: Models were deployed on two popular MCUs: the ESP32-S3 (high performance, specialized instructions) and the RP2040 (low cost, general purpose).
  8. Reference Verification: Instead of trusting host-side accuracy scores, the author compared on-device INT8 outputs element-by-element against TensorFlow Lite’s reference kernels to ensure "bit-exact" correctness.

Datasets

The study utilizes the dataset from Bird et al., which consists of:

  • Real Data: 3,013 photographs across six scene classes (bathroom, bedroom, computer lab, etc.).
  • Synthetic Data: 5,928 images rendered in virtual environments corresponding to the same six classes.
  • Imagenette: A subset of ImageNet used for pre-training the smallest CNN where standard ImageNet weights were unavailable.

Key Insights and Ideas

  • The "Scarce Data" Sweet Spot: Synthetic pre-training is not a "more is always better" solution. With full real data, the gain was a measly +0.312 percentage points (pp). But when real data was cut to 10%, the tiny CNN saw a jump of +3.438 pp. Synthetic data acts as a vital scaffold when the real-world signal is weak.
  • Quantization is Not the Enemy of Transfer: A common fear is that the subtle features learned during pre-training are lost during 8-bit quantization. This study found that the "transfer benefit" (the gap between pre-trained and non-pre-trained models) remained virtually identical in both float32 and INT8.
  • The "Vendor Kernel" Trap: To get high performance, developers use vendor-optimized libraries (like ESP-NN). The author found a critical out-of-bounds memory write in the ESP32-S3’s depthwise convolution kernel. Without the rigorous "bit-exact" verification step, this bug might have gone unnoticed, leading to silent accuracy degradation.
  • Hardware Divergence: The RP2040 failed to reproduce reference arithmetic at the global average pooling stage. This highlights a scary reality for TinyML engineers: your model might "work" on your PC but compute something slightly different on the actual chip.

Key Results

  • Accuracy: For the main deployable model (MobileNetV2 width 0.35), synthetic pre-training provided no detectable gain with full data. At 10% data, the gain for the small CNN was +2.715 pp averaged over three seeds.
  • Latency: The ESP32-S3 is a powerhouse for its size, processing an inference in 152.7 ms. The RP2040, lacking specialized neural network instructions, was roughly 12 times slower (1.80 s) for the same model.
  • Energy: On the ESP32-S3, an inference cost 16.3 mJ above idle. Interestingly, adding the synthetic pre-training stage did not change the final model's on-device latency or energy consumption—the cost is purely in the "offline" training time.
  • Memory Constraints: The largest 96px MobileNetV2 model exceeded the RP2040’s 2MB flash limit, illustrating that for many MCUs, model size is dictated by storage, not just RAM.

Limitations and Future Work

The author notes that this is a case study of a single dataset and one specific renderer. It remains unclear if more diverse synthetic data (more "scenes" rather than just more "images") would change the results for larger models.

Future work is suggested to:

  1. Disentangle the effects of input resolution vs. parameter count on the synthetic gain.
  2. Test newer hardware like the Cortex-M33 (RP2350) to see if modern entry-level MCUs close the performance gap.
  3. Investigate why the RP2040’s global average pooling departs from reference arithmetic.

Summary updated

Abstract

Rendered images are cheap to label, but a microcontroller classifier must survive 8-bit integer (INT8) quantisation and compute correctly on the chip. Five models, from MobileNetV2 at 224 px to a 61K-parameter convolutional neural network (CNN), were fine-tuned on a six-class scene dataset with and without a synthetic stage, against thresholds fixed in advance, and deployed on an ESP32-S3 and an RP2040. With full real data the stage gave no detectable gain. For the main deployable model the INT8 difference was +0.312 percentage points (pp), its interval below the +1.6 pp gain recomputed from the source study; quantisation did not detectably change it, nor did model size. With 10% of the real data the stage paid off for the CNN, by +3.438 pp on one seed and +2.715 pp over three; its cause is open. On the ESP32-S3 the main model took 152.7 ms and 16.3 mJ above idle per inference, and adding the stage changed neither beyond preset bounds. The model's INT8 outputs matched TensorFlow Lite's reference kernels exactly, but only after a memory overrun in a vendor kernel was patched. On the RP2040 the same model took 1.80 s and 27.3 mJ above idle, about twelve times slower, and the largest 96 px model exceeded the board's flash-capacity limit. Neither the RP2040 nor the host interpreter used for scoring reproduced the reference arithmetic. A synthetic stage is worth adding only when real data are scarce, and on-device INT8 outputs should be checked against reference kernels, not inferred from host scores.
BibTeX
@misc{yadav2026does,
  author = {Achintya Yadav},
  title = {{Does synthetic pre-training pay off on a microcontroller? Accuracy, latency and energy across two MCUs}},
  year = {2026},
  eprint = {2611.19931},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19931},
}
Figure