A Quantum Field Theory of Representation Learning

Robert Bamler, Stephan MandtarXiv 2019 · OpenReview

Abstract

Continuous symmetries and their breaking play a prominent role in contemporary physics. Effective low-energy field theories around symmetry-breaking states explain diverse phenomena such as superconductivity, magnetism, and the mass of nucleons. We show that such field theories can also be a useful tool in machine learning, particularly for loss functions with continuous symmetries that are spontaneously broken by random initializations. By drawing deep analogies between superconductivity and temporal representation learning, we formulate a gauge theory of 'charged' embedding vectors in time-series models. We demonstrate that making the loss function gauge invariant significantly speeds up convergence and improves model alignment.

The Challenge of Continuous Symmetries

In representation learning, "dense embeddings" are vectors where no specific direction has a fixed meaning. This creates a continuous rotational symmetry: if every embedding vector in the system is rotated by the same amount, the internal relationships and the loss function remain unchanged.

The problem arises in time-series models, such as tracking how word meanings evolve over decades. While we want embeddings at time $t$ to be similar to those at $t-1$, the model's rotational invariance allows the system to "drift" or rotate slightly at each time step without a significant penalty. This results in "flat" directions in the loss landscape, known in physics as Goldstone modes.

The loss landscape of a system with continuous symmetry often features "flat" valleys or Goldstone modes, where the optimizer can wander without reducing loss.

Mathematically, this creates an ill-conditioned Hessian (the matrix of second derivatives). Gradient Descent (GD) struggles in these flat regions, leading to a convergence rate that slows down quadratically as the length of the time series ($T$) increases.

Goldstone-GD: A Gauge Theory for Optimization

To address this optimization "drag," the authors propose a solution inspired by the Higgs mechanism in superconductivity. In physics, when charged particles are coupled to a gauge field, the previously flat energy spectrum acquires a "gap" (curvature).

Translating this to Machine Learning, they introduce Goldstone-GD:

  1. Minimal Coupling: Auxiliary "gauge fields"—specifically, rotation matrices $\Gamma_t$—are introduced for every time step.
  2. Gauge-Invariant Loss: The loss function is redefined to be invariant under local rotations, allowing the model to distinguish between real data changes and artificial coordinate rotations.
  3. Optimization Loop: The algorithm alternates between solving for optimal rotations to align embedding spaces across time and performing standard GD updates on the embeddings themselves.

Empirical Validation: Dynamic Word Embeddings

The approach was evaluated using dynamic word embeddings trained on the Google Books N-grams corpus, covering 188 years (1800 to 2008). The task was to maintain semantic alignment across time—for example, finding the 1800s equivalent of a modern word.

Standard GD failed significantly due to artificial drift; the embedding spaces for 1800 and 2008 became misaligned, leading to nonsensical semantic mappings. In contrast, Goldstone-GD correctly identified semantically related words across centuries. Quantitative analysis confirms that Goldstone-GD maintains high cosine similarity for words over time, reflecting the reality that most word meanings are relatively stable.

Distribution of cosine distances between word embeddings from 1800 to 2008. Goldstone-GD maintains significantly higher similarity, indicating better semantic alignment over time compared to the baseline.

Conclusion

The "gapless excitation spectrum" of a physical system is mathematically identical to the slow convergence of an ill-conditioned optimization problem in ML. By treating symmetry as a bug that creates optimization drag, the Goldstone-GD algorithm effectively "tilts" the landscape. This physics-inspired approach provides a robust framework for improving the efficiency and stability of representation learning in time-series data.

Summary updated

Abstract

Continuous symmetries and their breaking play a prominent role in contemporary physics. Effective low-energy field theories around symmetry breaking states explain diverse phenomena such as superconductivity, magnetism, and the mass of nucleons. We show that such field theories can also be a useful tool in machine learning, in particular for loss functions with continuous symmetries that are spontaneously broken by random initializations. In this paper, we illuminate our earlier published work (Bamler & Mandt, 2018) on this topic more from the perspective of theoretical physics. We show that the analogies between superconductivity and symmetry breaking in temporal representation learning are rather deep, allowing us to formulate a gauge theory of `charged' embedding vectors in time series models. We show that making the loss function gauge invariant speeds up convergence in such models.
BibTeX
@misc{bamler2019quantum,
  author = {Robert Bamler and Stephan Mandt},
  title = {{A Quantum Field Theory of Representation Learning}},
  year = {2019},
  eprint = {1907.02163},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.02163},
}
Figure