K-textures, a self-supervised hard clustering deep learning algorithm for satellite image segmentation

Fabien Wagner, Ricardo Dalagnol, Alber Sánchez, Mayumi C. M. Hirye, Samuel Favrichon, Jake Lee, Steffen Mauceri, Yan Yang, Sassan SaatchiFrontiers in Environmental Science 2022 · DOI

This paper, published in Frontiers in Environmental Science, introduces K-textures, a self-supervised deep learning algorithm designed to segment satellite imagery into discrete "hard" clusters. By combining the logic of k-means with the spatial awareness of Convolutional Neural Networks (CNNs), the authors provide a way to automate the creation of high-quality training labels for environmental monitoring.

TL;DR

K-textures is a self-supervised algorithm that performs hard clustering on satellite images by learning to reconstruct them using discrete binary masks and generated textures. It overcomes the "differentiability hurdle" of discrete classification in deep learning, allowing for automated, texture-aware segmentation without manual labels.

What Problem Does This Paper Address?

Mapping land use and land cover (LULC) typically requires massive amounts of manually labeled data, which is expensive and slow to produce. While unsupervised methods like k-means exist, they are "pixel-wise"—they only look at the color of a single pixel and ignore its neighbors. This is a problem in remote sensing, where the texture of a forest canopy or the pattern of an urban area is often more informative than raw spectral values.

Furthermore, most deep learning clustering methods produce "soft" assignments (probabilities). In the real world, a pixel is usually either forest or not forest; we need "hard" clusters. However, making discrete 0-or-1 decisions is traditionally non-differentiable, meaning you can't easily train a standard neural network to do it using gradient descent.

Methods

The K-textures model uses a two-path architecture to "simulate" the original image:

  1. The Mask Path: An encoder generates $k$ discrete binary masks. To make this differentiable, the authors developed a novel "hard sigmoid" activation function with an extremely steep linear part. They even trained a small sub-network ($\sigma_{CNN}$) to approximate this function, ensuring stable gradients during training.
  2. The Texture Path: A generator creates $k$ independent textures (one for each class).

The model multiplies the masks by the textures and sums them to create a simulated version of the input image.

The Loss Function: Instead of comparing raw pixels, the model uses a "perceptual loss." It compares the high-level features of the original and simulated images using a pre-trained VGG-16 network and a custom Variational Autoencoder (VAE). This forces the model to care about spatial patterns and textures rather than just matching colors.

Datasets

  • Planet Satellite Imagery: High-resolution (4.78m) 4-band imagery (RGB + Near-Infrared) covering a 20km x 20km area of the Brazilian Amazon.
  • MapBiomas (Collection 6): A 30m resolution land-cover map used as a "ground truth" reference to validate how well the unsupervised clusters matched real-world categories like "Forest," "Pasture," and "Urban."
  • Synthetic Data: Used specifically to train the $\sigma_{CNN}$ sub-network to handle the hard sigmoid activation.

Key Insights and Ideas

  • Discrete Search is Possible: The most significant "aha" moment is that you can perform discrete search (finding 0/1 masks) within a gradient-descent framework by using a steep sigmoid and adding a tiny amount of Gaussian noise to prevent the model from getting stuck on the "slopes" of the function.
  • Context is King: By using a feature-based loss, the model naturally groups pixels that "look" like forest (texture) even if their specific color values vary due to shadows or leaf tilt.
  • Automatic Ordering: Interestingly, the model tends to sort its learned textures in a 1D hue/intensity order naturally, which makes the resulting cluster tables much easier for humans to interpret.

Key Results

  • Binary Integrity: The masks produced were over 99.999% discrete (either 0 or 1), proving the effectiveness of the hard sigmoid approach.
  • Forest Segmentation: When compared to MapBiomas data, K-textures achieved a Precision of 0.815 and an Intersection over Union (IoU) of 0.771.
  • Comparison to k-means: While standard k-means had slightly higher precision in some cases, K-textures provided much better Recall (0.934) and produced segmentations that were more spatially consistent and less "noisy" than pixel-wise methods.
  • Efficiency: The model converged in roughly 18 to 80 minutes (depending on the number of clusters) on a standard mid-range GPU (Nvidia RTX2080).

Limitations and Future Work

  • Sensitivity to Illumination: The model currently struggles with large-scale variations in atmospheric conditions or heavy shadows, which it might mistake for separate textures.
  • Manual $k$ Selection: Like k-means, the user must still manually choose the number of clusters ($k$). The authors suggest that future work could involve automated methods to find the optimal $k$.
  • Computational Overhead: While more accurate than k-means, it is significantly more computationally expensive to train.

The authors have made their code and weights publicly available, positioning K-textures as a powerful tool for researchers needing to generate rapid, high-fidelity training samples for supervised LULC models.

Summary updated

Abstract

Deep learning self-supervised algorithms that can segment an image in a fixed number of hard clusters such as the k-means algorithm and with an end-to-end deep learning approach are still lacking. Here, we introduce the k-textures algorithm which provides self-supervised segmentation of a 4-band image (RGB-NIR) for a k number of classes. An example of its application on high-resolution Planet satellite imagery is given. Our algorithm shows that discrete search is feasible using convolutional neural networks (CNN) and gradient descent. The model detects k hard clustering classes represented in the model as k discrete binary masks and their associated k independently generated textures, which combined are a simulation of the original image. The similarity loss is the mean squared error between the features of the original and the simulated image, both extracted from the penultimate convolutional block of Keras “imagenet” pre-trained VGG-16 model and a custom feature extractor made with Planet data. The main advances of the k-textures model are: first, the k discrete binary masks are obtained inside the model using gradient descent. The model allows for the generation of discrete binary masks using a novel method using a hard sigmoid activation function. Second, it provides hard clustering classes–each pixel has only one class. Finally, in comparison to k-means, where each pixel is considered independently, here, contextual information is also considered and each class is not associated only with similar values in the color channels but with a texture. Our approach is designed to ease the production of training samples for satellite image segmentation and the k-textures architecture could be adapted to support different numbers of bands and for more complex self-segmentation tasks, such as object self-segmentation. The model codes and weights are available at https://doi.org/10.5281/zenodo.6359859 .
BibTeX
@inproceedings{wagner2022k,
  author = {Fabien Wagner and Ricardo Dalagnol and Alber Sánchez and Mayumi C. M. Hirye and Samuel Favrichon and Jake Lee and Steffen Mauceri and Yan Yang and Sassan Saatchi},
  title = {{K-textures, a self-supervised hard clustering deep learning algorithm for satellite image segmentation}},
  booktitle = {Frontiers in Environmental Science},
  year = {2022},
  doi = {10.3389/fenvs.2022.946729},
}
Figure