Transformer needs NMDA receptor nonlinearity for long-term memory

Dong-Kyum Kim, Jea Kwon, Meeyoung Cha, C. Justin LeeICLR 2023 · OpenReview

Abstract

The NMDA receptor (NMDAR) in the hippocampus is essential for learning and memory. We find an interesting resemblance between deep models' nonlinear activation function and the NMDAR's nonlinear dynamics. In light of a recent study that compared the transformer architecture to the formation of hippocampal memory, this paper presents new findings that NMDAR-like nonlinearity may be essential for consolidating short-term working memory into long-term reference memory. We design a navigation task assessing these two memory functions and show that manipulating the activation function (i.e., mimicking the Mg$^{2+}$-gating of NMDAR) disrupts long-term memory formation. Our experimental data suggest that the concept of place cells and reference memory may reside in the feed-forward network layer of transformers and that nonlinearity plays a key role in these processes. Our findings propose that the transformer architecture and hippocampal spatial representation resemble by sharing the overlapping concept of NMDAR-like nonlinearity.
BibTeX
@inproceedings{kim2023transformer,
  author = {Dong-Kyum Kim and Jea Kwon and Meeyoung Cha and C. Justin Lee},
  title = {{Transformer needs NMDA receptor nonlinearity for long-term memory}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=0z_cXcu1N6o},
}

Loading related content...

Figure