Multi-task Learning of Classification and Generation for Set-structured Data

Fumioki Sato, Hideaki Hayashi, Hajime NagaharaWACV 2025 · DOI

Abstract

Researchers from Osaka University have developed a multi-task learning framework that simultaneously classifies and generates set-structured data, such as point clouds. By integrating a classifier into a Variational Autoencoder (VAE) with hard parameter sharing, the model achieves significantly higher accuracy in low-data regimes and produces more reliable confidence scores compared to standard classifiers.

Introduction

Deep learning for "set-structured" data—unordered collections of points like LiDAR scans or bounding boxes—faces two major challenges: label scarcity and overconfidence. While raw data is abundant, manual labeling is expensive and slow. Furthermore, modern neural networks are often poorly calibrated, providing high-confidence predictions even when incorrect, which is unacceptable in safety-critical applications like autonomous driving. Standard architectures often fail to respect the permutation invariance required for sets, where the order of elements should not change the underlying meaning.

Methodology

The proposed framework utilizes a multi-task approach where a single encoder is shared between a generative model and a classifier. This "hard parameter sharing" forces the encoder to learn robust, general-purpose features representing the data's underlying structure rather than just shortcuts for classification.

Architecture of the proposed multi-task learning framework.

Key Components

  • Induced Set Attention Blocks (ISAB): These blocks in the encoder efficiently capture complex relationships between set elements while maintaining permutation invariance.
  • Attentive Bottleneck Layers (ABL): Used in the decoder to handle stochastic sampling needed for generation while preserving the set structure.
  • Joint Loss Function: The model is trained on a weighted sum of reconstruction error (generative task) and classification error.

Experiments and Datasets

The model was evaluated on three distinct set-based benchmarks:

  • Set-MNIST: A version of handwritten digits where pixels are treated as a set of coordinates.
  • Set-MultiMNIST: A multi-label version where multiple digits appear in one set.
  • ShapeNet: A large-scale dataset of 3D point clouds representing 55 different categories of real-world objects.

Results and Discussion

The generative task acts as a powerful regularizer, preventing overfitting on small labeled datasets. By forcing the model to recreate the input, it learns the actual data distribution $p(x)$. This allows the model to recognize ambiguous regions in the feature space, leading to lower, more honest confidence scores near decision boundaries.

Conceptual comparison of classification, generation, and the proposed joint approach.

In low-data scenarios, the performance gains were most dramatic. On Set-MNIST with only 10% of labels available, the proposed model maintained a high 92.6% accuracy, whereas a baseline classifier failed completely (9.8% accuracy). Additionally, the Expected Calibration Error (ECE) was consistently lower, indicating that the model's predicted probabilities are much closer to its actual accuracy.

Conclusion

While the model significantly improves calibration and low-data performance, challenges remain in scaling raw accuracy on complex datasets like ShapeNet due to extreme class imbalance. Future work will explore adding more tasks, such as segmentation, and testing the model on even more diverse 3D environments to replicate the accuracy gains seen in simpler set-based data.

Summary updated

BibTeX
@inproceedings{sato2025multi,
  author = {Fumioki Sato and Hideaki Hayashi and Hajime Nagahara},
  title = {{Multi-task Learning of Classification and Generation for Set-structured Data}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00656},
}
Figure