The Copycat Perceptron: Smashing Barriers Through Collective Learning

Giovanni Catania, Aurélien Decelle, Beatriz Seoane

TL;DR

This paper demonstrates that coupling multiple binary perceptron "students" together via a simple imitation mechanism (ferromagnetic coupling) allows them to collectively bypass the rugged, glassy energy landscapes that typically trap individual learning algorithms. By deriving the complete temperature-capacity phase diagram, the authors show that this collective learning approach—Replicated Simulated Annealing—enables networks to achieve perfect generalization faster and with significantly fewer training examples, closely approaching the theoretical Bayes-optimal limit.

What Problem Does This Paper Address?

Training neural networks with discrete weights (like the binary perceptron) is notoriously difficult. In the classic "teacher-student" scenario, a student network tries to learn a classification rule from data labeled by a teacher network. Even when there are theoretically enough examples to find the perfect solution (the "easy" inference phase), standard optimization algorithms like Simulated Annealing (SA) get trapped in suboptimal, glassy metastable states (spurious local minima).

The paper addresses this algorithmic bottleneck: how can we alter the optimization landscape to prevent algorithms from getting stuck, allowing them to find the true underlying rule efficiently?

Methods

The authors use tools from statistical mechanics to analyze a system of $y$ coupled binary perceptrons. They introduce a Hamiltonian where the students minimize classification errors (using the "perceptron rule" potential) while simultaneously being pulled toward each other by a pairwise ferromagnetic coupling proportional to their Hamming distance.

To map out the system's equilibrium properties, they employ Mean-Field Theory and the Replica Trick under a Replica Symmetric (RS) ansatz. This analytical approach allows them to derive the full $(\alpha, T)$ phase diagram, where $\alpha$ is the constraint density (ratio of examples to parameters) and $T$ is the temperature. They complement this theoretical work with numerical experiments using Replicated Simulated Annealing (RSA).

Datasets

The paper relies entirely on synthetic data generated within the teacher-student scenario. The training samples consist of $N$-dimensional random binary vectors $\xi^\mu \in \{-1, 1\}^N$, and the labels $\sigma_0^\mu$ are noiselessly assigned by a hidden teacher perceptron with weight vector $w_0$. Performance is evaluated based on the generalization error relative to this teacher.

Key Insights and Ideas

  • Landscape Smoothing via Imitation: The core "aha moment" is that forcing replicas to imitate each other ($y > 1$) fundamentally alters the free entropy landscape. It bends the phase diagram boundaries toward lower $\alpha$ values, effectively smoothing out the rugged terrain around the perfect generalization solution.
  • Bypassing Glassy Barriers: In a single perceptron ($y=1$), the "easy" phase is plagued by subdominant 1RSB metastable states that act as algorithmic traps. Coupling the perceptrons allows the collective system to melt directly into the teacher solution before encountering these glassy traps.
  • Relevance to Federated Learning: This provides a strong theoretical foundation for collaborative learning. Multiple agents reviewing the same data can learn the correct rule significantly faster and with fewer examples than a single agent working in isolation.

Key Results

  • Phase Diagram Bending: The authors mapped the full $(\alpha, T)$ phase diagram for the replicated perceptron, showing a clear shift in the spinodal lines toward lower $\alpha$ as the number of replicas increases, as illustrated in Figure 1:
Figure 1. (α, T)-Equilibrium phase diagram of the single, non- replicated, binary perceptron (top panel) and the replicated perceptron according to Eq. (2) for a fixed coupling γ = 1 and y multiple replicas (bottom panel). The color shades refer respectively to the impossible (red), hard (yellow), easy (green) inference phases as extrapolated from T =0. The (a)- (e) regions are discussed in the text.
  • Algorithmic Speedup: Numerical simulations with a slow cooling rate ($\eta = 10^{-5}$) showed that a single perceptron fails to find the teacher solution until $\alpha \approx 1.6$. In contrast, adding just a few replicas ($y \ge 2$) shifts the success threshold sharply toward the theoretical algorithmic limit $\alpha_c \approx 1.49$, which matches the Approximate Message Passing (AMP) threshold (see Figure 2).
Numerical performances of replicated SA
  • Better Suboptimal Solutions: Even in the "hard" inference regime where perfect recovery is impossible, the replicated system finds solutions with lower generalization errors than the non-replicated baseline.

Limitations and Future Work

  • Simplified Model: The study is restricted to the binary perceptron—a classic but highly simplified toy model—using a fully connected interaction topology (schematized in Figure S1/Figure 3).
  • Theoretical Gap: A complete understanding of how the glassy states disappear in the coupled system requires a full 1RSB analysis, which the authors leave for future investigations.
  • Scaling to Deep Learning: While highly suggestive for decentralized and federated learning, it remains to be seen how well these insights transfer to complex, continuous multi-layer deep neural network architectures.

Summary updated

Abstract

We characterize the equilibrium properties of a model of $y$ coupled binary perceptrons in the teacher-student scenario, subject to a suitable cost function, with an explicit ferromagnetic coupling proportional to the Hamming distance between the students' weights. In contrast to recent works, we analyze a more general setting in which thermal noise is present that affects each student's generalization performance. In the nonzero temperature regime, we find that the coupling of replicas leads to a bend of the phase diagram towards smaller values of $α$: This suggests that the free entropy landscape gets smoother around the solution with perfect generalization (i.e., the teacher) at a fixed fraction of examples, allowing standard thermal updating algorithms such as Simulated Annealing to easily reach the teacher solution and avoid getting trapped in metastable states as it happens in the unreplicated case, even in the computationally \textit{easy} regime of the inference phase diagram. These results provide additional analytic and numerical evidence for the recently conjectured Bayes-optimal property of Replicated Simulated Annealing (RSA) for a sufficient number of replicas. From a learning perspective, these results also suggest that multiple students working together (in this case reviewing the same data) are able to learn the same rule both significantly faster and with fewer examples, a property that could be exploited in the context of cooperative and federated learning.
BibTeX
@inproceedings{catania2024copycat,
  author = {Giovanni Catania and Aurélien Decelle and Beatriz Seoane},
  title = {{The Copycat Perceptron: Smashing Barriers Through Collective Learning}},
  booktitle = {Physical},
  year = {2024},
  doi = {10.1103/physreve.109.065313},
}
Figure