Atomized Deep Learning Models

Yi-Lin Tuan, Zih-Yun Chiu, William Yang Wang

Abstract

Deep learning models often tackle the intra-sample structure, such as the order of words in a sentence and pixels in an image, but have not pay much attention to the inter-sample relationship. In this paper, we show that explicitly modeling the inter-sample structure to be more discretized can potentially help model's expressivity. We propose a novel method, Atom Modeling, that can discretize a continuous latent space by drawing an analogy between a data point and an atom, which is naturally spaced away from other atoms with distances depending on their intra structures. Specifically, we model each data point as an atom composed of electrons, protons, and neutrons and minimize the potential energy caused by the interatomic force among data points. Through experiments with qualitative analysis in our proposed Atom Modeling on synthetic and real datasets, we find that Atom Modeling can improve the performance by maintaining the inter-sample relation and can capture an interpretable intra-sample relation by mapping each component in a data point to electron/proton/neutron.

Introduction

Deep learning models often struggle to capture the "nuanced uniqueness" of complex data. While massive architectures like Transformers excel at processing intra-sample structures, they often lack expressiveness—the ability to clearly distinguish between similar but different inputs—and interpretability. Existing solutions like contrastive learning or vector quantization help but often require prior knowledge of positive/negative pairs or rely on rigid architectural changes. This paper introduces "Atom Modeling," a technique to boost a model's ability to separate data in its internal latent space without needing explicit supervision.

Methods: Atom Modeling

The core innovation is a technique that treats a data sample (like an image) as a collection of sub-units (like patches or tokens) that behave like sub-atomic particles.

Overview of the Atom Modeling framework, illustrating the transition from raw data through a latent function to a discretized latent space modeled as interacting atoms.
  1. Token Importance ($q_i$): The model learns to assign an "importance score" to every sub-component. A score of +1 or -1 indicates the component is crucial for distinguishing the sample, while 0 is irrelevant.
  2. Atomic Distance: Inspired by the Coulomb force (the physical force between charged particles), a "dynamic distancing loss" pushes data samples apart in the latent space.
  3. The Physics Analogy: The average of the most crucial tokens forms a "nucleus" ($\mu$), while other tokens orbit this nucleus with a "radius" ($r$). Importance scores act like electrical charges.
  4. Dynamic Separation: Unlike standard methods that use fixed distances, this approach calculates an optimal "balance distance" between samples based on their internal structures. If two samples have very similar internal "atomic" structures, the loss forces the model to work harder to separate them.

Key Insights and Results

Atom Modeling leads to several emergent properties across various tasks:

  • Emergent Interpretability: Without being told what to look for, the model naturally assigned high importance scores to semantically meaningful areas—like facial features in images or keywords in a sentence.
  • Self-Organizing Latent Space: By minimizing the atomic loss, the model "spreads out" its representations. This prevents the "crowding" effect often seen in standard training, where different classes overlap in high-density regions.
  • Performance Gains: On fine-grained classification tasks like identifying pet breeds or flower species, Atom Modeling consistently outperformed standard Cross-Entropy and SimCLR.
Comparison of classification accuracy across different loss functions, showing Atom Modeling achieving significantly higher accuracy with lower variance compared to standard methods.

On the Oxford-Flowers102 dataset, the method achieved 59.1% accuracy compared to 56.7% for standard training. In generative tasks, the method significantly lowered the Frechet Inception Distance (FID); on MNIST, the FID dropped from 88.4 to 49.0. For text, fine-tuning BERT with Atom Modeling improved the Matthews Correlation Coefficient (MCC) on the CoLA dataset from 60.0 to 61.3.

Limitations and Future Work

The primary limitation is computational cost; the "multiplication-inverse" term in the loss function increases training time by roughly 3% to 12%. Additionally, selecting the optimal hidden layer for modeling remains empirical, as the choice varies between classifiers and generators. Future research could explore more efficient ways to calculate these atomic distances or apply the method to multi-modal learning.

Summary updated

Abstract

Deep learning models often tackle the intra-sample structure, such as the order of words in a sentence and pixels in an image, but have not pay much attention to the inter-sample relationship. In this paper, we show that explicitly modeling the inter-sample structure to be more discretized can potentially help model's expressivity. We propose a novel method, Atom Modeling, that can discretize a continuous latent space by drawing an analogy between a data point and an atom, which is naturally spaced away from other atoms with distances depending on their intra structures. Specifically, we model each data point as an atom composed of electrons, protons, and neutrons and minimize the potential energy caused by the interatomic force among data points. Through experiments with qualitative analysis in our proposed Atom Modeling on synthetic and real datasets, we find that Atom Modeling can improve the performance by maintaining the inter-sample relation and can capture an interpretable intra-sample relation by mapping each component in a data point to electron/proton/neutron.
BibTeX
@inproceedings{tuan2023atomized,
  author = {Yi-Lin Tuan and Zih-Yun Chiu and William Yang Wang},
  title = {{Atomized Deep Learning Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=_lPNXhQ4uvS},
}
Figure