Distributed Memory based Self-Supervised Differentiable Neural Computer

Taewon Park, Inchul Choi, Minho Lee

Introduction

Memory Augmented Neural Networks (MANNs), such as the Differentiable Neural Computer (DNC), were designed to bridge the gap between the flexible learning capabilities of neural networks and the structured, long-term storage of traditional computer architectures. Despite their promise, these models often struggle with complex relational reasoning—tasks that require understanding the intricate dependencies between disparate pieces of information over long sequences.

This paper identifies a primary bottleneck in conventional MANNs: the reliance on a single, monolithic external memory matrix. When a model attempts to encode complex data and its internal relationships into a single vector representation for storage, it often results in "lossy" compression. While some researchers have turned to computationally expensive self-attention mechanisms ($O(n^2)$ complexity) to resolve this, this work proposes a more efficient, brain-inspired alternative: the Distributed Associative Memory (DAM) network combined with a Memory Refreshing Loss (MRL).

Architectural Innovations

Distributed Associative Memory (DAM)

Instead of a single large memory, the DAM architecture utilizes $K$ smaller, independent sub-memory blocks. This design is inspired by the distributed nature of information processing in the human brain.

  • Writing: The controller generates independent writing operators for each of the $K$ blocks, allowing the system to store different "views" or feature sets of the same input simultaneously. This multi-faceted storage prevents the representational collapse seen in single-memory systems.
  • Reading: To retrieve information, the model employs an attentive gate. This soft-attention mechanism interpolates between the read-outs of all $K$ blocks, dynamically selecting the most relevant distributed representations for the current task.

Crucially, because these sub-memory blocks can be processed in parallel, the DAM architecture maintains $O(n)$ computational complexity, offering a significant speed advantage over quadratic self-attention models.

Memory Refreshing Loss (MRL)

To further enhance the richness of the stored representations, the authors introduce the Memory Refreshing Loss (MRL), an unsupervised auxiliary task inspired by human "rehearsal." In biological systems, repeating information helps consolidate it into long-term memory; MRL mimics this by forcing the model to reconstruct stochastically sampled previous inputs using only its current memory contents.

To prevent the model from simply "echoing" the input, the authors use Bernoulli trials to sample only a percentage of inputs for reconstruction. Furthermore, a dynamic re-weighting strategy balances this auxiliary loss with the primary task objective, ensuring that the rehearsal process supports, rather than overwhelms, the main goal.

Experimental Evaluation

The model was evaluated across a diverse suite of benchmarks designed to test both basic memorization and high-level reasoning:

  1. Algorithmic Tasks: Standard tests including "Copy" and "Associative Recall."
  2. Representation Recall (RR): A new task introduced by the authors where the model must remember $2N$ sub-parts of a binary vector and use $N$ parts as a cue to recall the remainder, specifically testing representational richness.
  3. bAbI QA Tasks: A set of 20 reasoning tasks (e.g., deduction, induction) involving short natural language stories.
  4. Relational Tasks: The "$N^{th}$ Farthest" vector task and the "Convex Hull" task (predicting points that form a boundary around a set).

Key Results and Insights

State-of-the-Art Performance

The DAM-MR model achieved a 0.14% mean error rate on the bAbI tasks, setting a new state-of-the-art record. This performance surpasses established models like the Universal Transformer and the Sparse Transformer Memory (STM).

Superior Relational Reasoning

On the $N^{th}$ Farthest task, the model achieved 98.2% accuracy, significantly outperforming the Relational Memory Core (RMC), which achieved 91%. This suggests that distributed representations can capture high-order dependencies as effectively as—or better than—self-attention mechanisms.

Scalability and Convergence

Experiments demonstrated that as the number of sub-memory blocks ($K$) increased, the model's error rate on complex tasks consistently decreased. This validates the hypothesis that diversity in memory storage is more important than raw memory size. Additionally, the MRL was shown to significantly expedite the training process, allowing the model to converge faster than a standard DNC.

Conclusion

The Distributed Associative Memory network demonstrates that quadratic complexity is not a prerequisite for deep relational reasoning. By leveraging distributed representations and a self-supervised rehearsal mechanism, the DAM-MR model achieves superior performance on complex reasoning benchmarks while maintaining linear efficiency. Future work will focus on extending these capabilities to even longer context lengths and investigating the specific coordination dynamics between individual memory blocks.

Summary updated

BibTeX
@inproceedings{park2021distributed,
  author = {Taewon Park and Inchul Choi and Minho Lee},
  title = {{Distributed Memory based Self-Supervised Differentiable Neural Computer}},
  booktitle = {Neural Networks},
  year = {2021},
  url = {https://openreview.net/forum?id=1ZqQKptsFF},
}
Figure