ImageSig: A signature transform for ultra-lightweight image recognition

Mohamed R. Ibrahim, Terry J. LyonsWorkshops @ CVPR 2022 · DOI · arXiv

Abstract

ImageSig is a novel, ultra-lightweight image recognition framework designed for resource-constrained edge devices. By applying signature transforms from rough path theory, ImageSig treats images as streams of data rather than static 2D grids. This approach eliminates the need for computationally expensive 2D convolutions and attention mechanisms. Experimental results demonstrate that ImageSig can achieve competitive accuracy with a memory footprint as small as 44.2 KB and high-speed inference exceeding 140 FPS on a Raspberry Pi 4, significantly outperforming standard mobile architectures like MobileNetV2 in efficiency.

Introduction

Modern computer vision relies heavily on deep Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). While these models achieve state-of-the-art accuracy, their computational demands—often involving millions of parameters and high memory usage—make them unsuitable for microcontrollers or embedded devices with minimal resources. Even optimized architectures like MobileNet are frequently too heavy for the most restricted "edge" environments.

The authors of ImageSig address this gap by proposing a mathematical representation that is both compact and descriptive, avoiding expensive 2D spatial operations while capturing essential image features for classification tasks.

Methodology: Images as Streams

The core innovation of ImageSig is the application of Rough Path Theory to static images. The framework follows a multi-step process to transform raw pixels into a lightweight feature set.

  1. Images as Streams: Instead of a 2D grid, an image is interpreted as a "stream of paths" where pixels evolve over a "virtual time" across spatial dimensions.
  2. The Signature Transform: The model extracts a "signature"—a unique mathematical summary of these paths. This signature is invariant to path parameterization, capturing the underlying order and structure of the data.
  3. Lightweight Backend: Since the signature transform handles feature extraction, the learning component can be minimalist. The authors evaluated three simple backends: a single Fully Connected (FC) layer, a 1D Convolutional block, and a small Transformer encoder.
Overview of the ImageSig architecture and the representation of images as streams of paths.

Experimental Setup

The framework was evaluated across four diverse datasets to demonstrate its versatility:

  • Fire Detection: A custom dataset of 18,912 images for anomaly detection.
  • Fog Benchmark: 4,345 images for detecting street-level weather conditions.
  • Concrete Crack Detection: 40,000 images for structural health monitoring.
  • CelebA: A large-scale dataset (202,599 images) used for facial attribute recognition.

Key Results and Performance

ImageSig demonstrates exceptional performance on edge hardware. By converting 2D spatial data into 1D path signatures, the model avoids the heavy FLOPs associated with 2D sliding windows.

  • Extreme Efficiency: On the fire dataset, ImageSig required only 0.77 Million FLOPs, compared to over 2,500 Million for VGG16.
  • High Throughput: On a Raspberry Pi 4 (CPU), ImageSig achieved 142.3 FPS, whereas MobileNetV2 reached only 13.9 FPS. On a Jetson Nano, throughput increased to 442.8 FPS.
  • Compact Footprint: A quantized ImageSig + CNN1D model occupies only 44.2 KB, making it compatible with almost any microcontroller.
ROC curves comparing ImageSig variants against MobileNetV2 for fire and fog detection.

As shown in the ROC curves, ImageSig variants consistently outperform MobileNetV2 on low-resolution (64x64) tasks, achieving higher Area Under the Curve (AUC) scores while maintaining a fraction of the parameter count.

Discussion and Future Work

A key insight from the study is the "sweet spot" for signature depth; a depth of 4 provides the optimal balance between accuracy and model size. Furthermore, the signature acquisition can be performed once and shared across multiple detectors on the same chip, enhancing energy efficiency.

Current limitations include scaling to very high-resolution images, which may increase the computational cost of the signature transform itself. Future work will focus on extending ImageSig to more complex tasks such as object detection and semantic segmentation, as well as integrating local and global signatures to better capture both fine details and overall context.

Summary updated

BibTeX
@inproceedings{ibrahim2022imagesig,
  author = {Mohamed R. Ibrahim and Terry J. Lyons},
  title = {{ImageSig: A signature transform for ultra-lightweight image recognition}},
  booktitle = {Workshops @ CVPR},
  year = {2022},
  doi = {10.1109/cvprw56347.2022.00409},
}
Figure