Interweaving Memories of a Siamese Large Language Model

Xin Song, Zhikai Xue, Guoxiu He, Jiawei Liu, Wei Lu

Abstract

Parameter-efficient fine-tuning (PEFT) methods optimize large language models (LLMs) by modifying or introducing a small number of parameters to enhance alignment with downstream tasks. However, they can result in catastrophic forgetting, where LLMs prioritize new knowledge at the expense of comprehensive world knowledge. A promising approach to mitigate this issue is to recall prior memories based on the original knowledge. To this end, we propose a model-agnostic PEFT framework, IMSM, which Interweaves Memories of a Siamese Large Language Model. Specifically, our siamese LLM is equipped with an existing PEFT method. Given an incoming query, it generates two distinct memories based on the pre-trained and fine-tuned parameters. IMSM then incorporates an interweaving mechanism that regulates the contributions of both original and enhanced memories when generating the next token. This framework is theoretically applicable to all open-source LLMs and existing PEFT methods. We conduct extensive experiments across various benchmark datasets, evaluating the performance of popular open-source LLMs using the proposed IMSM, in comparison to both classical and leading PEFT methods. Our findings indicate that IMSM maintains comparable time and space efficiency to backbone PEFT methods while significantly improving performance and effectively mitigating catastrophic forgetting.

Introduction: The Challenge of Catastrophic Forgetting

Fine-tuning Large Language Models (LLMs) often presents a trade-off: gaining specialized task performance at the cost of general "world knowledge"—a phenomenon known as catastrophic forgetting. When using Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA to adapt an LLM to a new domain, even small weight modifications can cause a distribution shift that disrupts the model’s original reasoning. Existing solutions often involve replaying old pre-training data, which is computationally expensive and requires access to datasets that may no longer be available.

Comparison between Vanilla PEFT and the proposed IMSM framework. Vanilla PEFT often loses general knowledge, while IMSM preserves both domain and general knowledge using a Siamese architecture.

The IMSM Framework

The authors introduce IMSM, a model-agnostic framework that employs a "Siamese" architecture to preserve original knowledge while learning new tasks.

Siamese Architecture and Parallel Memories

The framework treats the LLM as two entities sharing the same structure: a frozen Original LLM representing prior knowledge and an Enhanced LLM equipped with PEFT modules (e.g., LoRA, AdaLoRA). For any input, the model generates two sets of hidden states, termed "memories."

Query-Aware Gating and Interweaving

The core of IMSM is a learnable, query-aware gating mechanism. Instead of simple summation, the gate analyzes the specific query to decide, token-by-token, the relative contribution of the original versus the enhanced memory. The final token is generated from a fused hidden state, effectively interweaving the model's past and present knowledge.

Detailed architecture of the IMSM framework, highlighting the parallel memory generation and the interweaving mechanism with query-aware gating.

Experimental Evaluation

The researchers evaluated IMSM across diverse tasks to test both learning and forgetting:

  • Downstream Alignment: MRPC (paraphrasing), CoLA (grammar), ROPES (reading comprehension), and GSM8K (mathematical reasoning).
  • General Knowledge Preservation: WebQ, FreebaseQA, and MultiRC.
  • Backbone Models: ChatGLM3-6B, Qwen1.5-4B, Llama2-7B, and Llama3-8B.

Key Results

  • Performance Gains: IMSM consistently outperformed vanilla PEFT. For ChatGLM3, IMSM improved the average score across four tasks from 73.10% (standard LoRA) to 75.29%.
  • Mitigating Forgetting: On the MultiRC benchmark, standard LoRA caused ChatGLM3's accuracy to drop from 71.74% to 66.67%. IMSM maintained it at 71.47%, nearly eliminating the forgetting penalty. In some cases, it even improved performance on non-target datasets compared to the original model.
  • Efficiency: Inference speed remains high; for Llama3, IMSM achieved ~21 tokens/sec compared to LoRA's ~23 tokens/sec.

Conclusion and Future Work

IMSM demonstrates that treating hidden states as memories and dynamically balancing them can effectively mitigate catastrophic forgetting in PEFT. While currently focused on final-layer fusion, future work may explore multi-layer memory interweaving and more complex sequential fine-tuning scenarios.

Code available at: https://github.com/ECNU-Text-Computing/IMSM

Summary updated

Abstract

Parameter-efficient fine-tuning (PEFT) methods optimize large language models (LLMs) by modifying or introducing a small number of parameters to enhance alignment with downstream tasks. However, they can result in catastrophic forgetting, where LLMs prioritize new knowledge at the expense of comprehensive world knowledge. A promising approach to mitigate this issue is to recall prior memories based on the original knowledge. To this end, we propose a model-agnostic PEFT framework, IMSM, which Interweaves Memories of a Siamese Large Language Model. Specifically, our siamese LLM is equipped with an existing PEFT method. Given an incoming query, it generates two distinct memories based on the pre-trained and fine-tuned parameters. IMSM then incorporates an interweaving mechanism that regulates the contributions of both original and enhanced memories when generating the next token. This framework is theoretically applicable to all open-source LLMs and existing PEFT methods. We conduct extensive experiments across various benchmark datasets, evaluating the performance of popular open-source LLMs using the proposed IMSM, in comparison to both classical and leading PEFT methods. Our findings indicate that IMSM maintains comparable time and space efficiency to backbone PEFT methods while significantly improving performance and effectively mitigating catastrophic forgetting.
BibTeX
@inproceedings{song2025interweaving,
  author = {Xin Song and Zhikai Xue and Guoxiu He and Jiawei Liu and Wei Lu},
  title = {{Interweaving Memories of a Siamese Large Language Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=poWxs5eQGP},
}
Figure