Two-dimensional signature of images and texture classification

Sheng Zhang, Guang Lin, Samy Tindel

TL;DR

This paper introduces a novel way to represent images as "2D signatures," inspired by rough paths theory, to extract low-dimensional, powerful features for texture classification. They demonstrate that these signatures, particularly when incorporating second-order information, achieve near-perfect accuracy on a challenging texture dataset, outperforming traditional methods.

What Problem Does This Paper Address?

Extracting meaningful, low-dimensional features from images is crucial for efficient and accurate machine learning. Traditional methods like Principal Component Analysis (PCA) capture linear relationships but often miss complex, nonlinear patterns inherent in image textures. The authors aim to bridge this gap by leveraging the power of "signatures," a mathematical tool from rough paths theory, to capture these intricate, nonlinear image structures. While 1D signatures have seen success in time-series analysis, their application to 2D data like images was underdeveloped, presenting a challenge for image processing tasks.

Methods

The core of this paper is the introduction and application of 2D signatures for image analysis.

  • Continuous 2D Signatures: The authors define 2D signatures as iterated integrals of a 2D field (representing an image). These integrals capture how the image's intensity or color values change across its spatial dimensions. They focus on first and second-order increments, which are analogous to derivatives and second-order derivatives in calculus, but generalized for 2D fields.
  • Discretization: To apply these signatures to digital images (pixel grids), they develop discrete versions of these iterated integrals. This involves summing over pixel neighborhoods, effectively approximating the continuous integrals.
  • Feature Engineering: For texture classification, they construct a feature vector for each image. This vector includes:
  • First-order increments: These capture basic spatial changes.
  • Second-order increments: These capture more complex, nonlinear interactions between spatial changes, which are crucial for texture.
  • Symmetry: To make the features invariant to image rotation and reflection (important for texture), they average the signature features over eight different orientations.
  • PCA Integration: They also explore combining these signature features with principal components derived from raw pixel data or first-order increments.
  • Classification: A random forest classifier is trained using these engineered features to predict texture classes.

Datasets

  • CuRRET Database: The experiment uses the CuRRET (Columbia-Utrecht Reflectance and Texture Database), which contains 42 different texture categories.
  • Training Data: 10 samples per texture category were used for training (10 x 42 = 420 images).
  • Testing Data: 100 samples per texture category were used for testing (100 x 42 = 4200 images).

Key Insights and Ideas

  • Signatures Capture Nonlinear Texture: The paper strongly suggests that 2D signatures, especially second-order ones, are adept at capturing the nonlinear, intricate patterns that define textures, which linear methods like PCA might miss.
  • Symmetry is Key for Texture: Incorporating rotational and reflectional symmetry into the signature features significantly improves classification performance, as texture is inherently orientation-invariant.
  • Low-Dimensional Power: Despite the complexity of the underlying mathematical framework, the authors demonstrate that a relatively small set of signature-based features (around 15-52 dimensions) can achieve very high classification accuracy.
  • Complementary to PCA: Signature features act as powerful nonlinear complements to the linear features provided by PCA, leading to superior performance when combined.

Key Results

  • Near-Perfect Classification: Using a combination of the first 3 principal components and 12 symmetric second-order signature features, the random forest classifier achieved close to 100% classification accuracy on the CuRRET dataset with only 10 training samples per class.
  • Significant Improvement: The inclusion of second-order signature features, especially when made symmetric, consistently improved classification accuracy compared to using only PCA features.
  • Visualization: A t-SNE visualization of the 15-dimensional feature space showed excellent separation between different texture classes, even before classification, highlighting the discriminative power of the features.

Limitations and Future Work

  • Computational Complexity: While the paper focuses on second-order signatures for tractability, the number of terms in higher-order signatures can explode, posing a computational challenge.
  • Generalization: The authors acknowledge that their work is a first step and that more extensive experiments across various image categories are needed to fully assert the validity of 2D signatures.
  • Algebraic Structures: Further investigation into the underlying algebraic structures of 2D signatures, potentially involving Hopf algebras, is suggested for a more systematic construction of signatures.
  • Broader Applications: The authors envision applications beyond texture classification, including material science and civil engineering, where texture analysis is important.
  • Advanced Calculus: Exploring connections to paracontrolled calculus and Fourier modes for image processing is also proposed.

Summary updated

Abstract

We introduce a proper notion of two-dimensional signature for images. This object is inspired by the so-called rough paths theory, and it captures many essential features of a two-dimensional object such as an image. It thus serves as a low-dimensional feature for pattern classification. Here, we implement a simple procedure for texture classification. In this context, we show that a low-dimensional set of features based on signatures produces an excellent accuracy.
BibTeX
@inproceedings{zhang2022two,
  author = {Sheng Zhang and Guang Lin and Samy Tindel},
  title = {{Two-dimensional signature of images and texture classification}},
  booktitle = {Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences},
  year = {2022},
  doi = {10.1098/rspa.2022.0346},
}
Figure