Convolutional Set Transformer

Federico Chinello, Giacomo BoracchiarXiv 2025 · DOI · arXiv

Abstract

The analysis of image sets—where information is distributed across multiple instances—poses a significant challenge for traditional computer vision models that process images in isolation. Existing "cascaded" architectures typically perform independent feature extraction before contextualizing the set, leading to a loss of spatial information and context-blind features. This paper introduces the Convolutional Set Transformer (CST), a novel architecture that integrates feature extraction and contextual modeling within a single framework using the SetConv2D layer. By employing Multi-Head Self-Attention to compute dynamic biases, CST enables simultaneous learning across the set. Experimental results across classification and anomaly detection tasks demonstrate that CST consistently outperforms state-of-the-art cascaded models while providing superior explainability through standard tools like Grad-CAM.

Introduction

In modern computer vision, the analysis of individual images has reached a high level of maturity. However, many real-world applications require processing a set of images where the relevant information is distributed across multiple instances. Examples include medical diagnostics involving multiple scans of a single patient, or social media platforms categorizing photo albums. Traditional neural networks often struggle in these scenarios because they treat each image in isolation, failing to capture the inter-dependencies within the set.

In their paper, "Convolutional Set Transformer," Federico Chinello and Giacomo Boracchi propose a novel architecture (CST) designed to address these challenges. The CST allows neural networks to perform feature extraction and contextual modeling simultaneously, enabling the network to "talk" across a set of images during the learning process.

The Limitation of Cascaded Pipelines

Existing architectures for set-input tasks, such as Deep Sets or the original Set Transformer, are fundamentally limited to processing vector embeddings rather than 3D image tensors. Consequently, researchers typically employ a "cascaded" pipeline. In this setup, a standard Convolutional Neural Network (CNN) first processes each image independently to generate a feature vector. These vectors are then passed to a set-input network for contextualization.

This design has two significant drawbacks. First, the initial feature extractor is "context-blind," meaning it cannot adjust its focus based on other images in the set. Second, the spatial information is collapsed into a vector before set-processing occurs, which prevents the use of standard explainability tools like Grad-CAM to identify which specific image regions influenced the set-level decision.

The Convolutional Set Transformer (CST)

The primary innovation of the CST is the SetConv2D layer. Unlike traditional layers that operate on images in isolation, SetConv2D integrates information sharing directly into the convolutional block.

The SetConv2D layer operates through the following steps:

  1. Shared Convolution: A standard 2D convolution is applied to every image in the set using shared weights.
  2. Global Average Pooling (GAP): Spatial features are summarized into representative vectors.
  3. Self-Attention: These vectors interact via Multi-Head Self-Attention (MHSA), allowing each image to learn from every other image in the set.
  4. Dynamic Biases: The output of the attention mechanism is transformed into context-aware biases, which are then added back to the original spatial feature maps.
  5. Activation: The enriched features pass through a non-linear activation function.

By stacking these layers, the CST can model hierarchical relationships, ranging from low-level texture similarities in early layers to high-level semantic connections in deeper layers.

Experimental Evaluation

The authors evaluated the CST across diverse benchmarks, including ImageNet classification, Tiny ImageNet, and CIFAR-10/100. They also explored Set Anomaly Detection using the CelebA dataset, where the model was tasked with identifying the "odd-one-out" in a set of faces (e.g., finding a person without a hat among hat-wearers). Finally, the model was applied to the Personal Event Collection (PEC) dataset for event categorization in photo albums.

Additionally, the authors introduced CST-15, a compact yet powerful backbone pre-trained on ImageNet specifically for set-learning tasks. This model (28M parameters) outperformed much larger architectures like VGG-19 (144M parameters) on ImageNet classification while remaining significantly more efficient.

Key Insights and Results

The CST demonstrated superior performance and interpretability compared to traditional cascaded methods:

  • Performance Gains: CST outperformed competing models in 58 out of 60 test cases, showing relative accuracy improvements of up to 20% in some scenarios.
  • Native Explainability: Because the CST maintains spatial dimensions throughout the set-processing stages, researchers can apply Grad-CAM directly. In anomaly detection tasks, the CST accurately highlighted specific anomalous features, such as a missing accessory or a distinct facial expression.
  • Contextual Sharpening: The research found that as set size increases, the network's internal representations become more distinct, effectively using context as a "denoiser" for visual ambiguity.
  • Combinatorial Training (CT): Training on small, varying set sizes allows the model to generalize to much larger sets during inference, significantly reducing memory requirements during the training phase.

Conclusion

The Convolutional Set Transformer represents a significant step forward in set-based image analysis. By integrating contextual modeling into the feature extraction process, it achieves higher accuracy and provides better explainability than previous cascaded approaches. While the inclusion of self-attention adds computational overhead, the benefits in performance and transparency make it a compelling choice for complex tasks in medical imaging, surveillance, and e-commerce.

Summary updated

BibTeX
@misc{chinello2025convolutional,
  author = {Federico Chinello and Giacomo Boracchi},
  title = {{Convolutional Set Transformer}},
  year = {2025},
  eprint = {2509.22889},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.22889},
}
Figure