KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation

Xindi Luo, Zequn Sun, Jing Zhao, Zhe Zhao, Wei Hu

Abstract

Parameter-efficient finetuning (PEFT) is a key technique for adapting large language models (LLMs) to downstream tasks. In this paper, we study leveraging knowledge graph embeddings to improve the effectiveness of PEFT. We propose a knowledgeable adaptation method called KnowLA. It inserts an adaptation layer into an LLM to integrate the embeddings of entities appearing in the input text. The adaptation layer is trained in combination with LoRA on instruction data. Experiments on six benchmarks with two popular LLMs and three knowledge graphs demonstrate the effectiveness and robustness of KnowLA. We show that KnowLA can help activate the relevant parameterized knowledge in an LLM to answer a question without changing its parameters or input prompts.

Introduction

Large Language Models (LLMs) like Llama 2 have revolutionized natural language processing, yet they remain susceptible to factual inaccuracies and hallucinations. While structured Knowledge Graphs (KGs) offer a potential remedy through knowledge injection, traditional methods often require full-model finetuning—a process that is computationally prohibitive for models with billions of parameters. Furthermore, most existing research has focused on encoder-based architectures, leaving the effectiveness of knowledge injection in modern decoder-only LLMs under-explored.

To bridge this gap, we introduce KnowLA (Knowledgeable Adaptation), a parameter-efficient method that enhances LLMs by injecting external KG embeddings through a lightweight adaptation layer. KnowLA aligns structured knowledge with the LLM’s internal representations, not only providing external facts but also "activating" the model's own latent knowledge.

Methodology: The KnowLA Layer

The core of our approach is the KnowLA Layer, a specialized module inserted between the decoder layers of an LLM. In our experiments with Llama 2-7B, this layer is typically placed after the final transformer block. The KnowLA architecture operates through three primary stages:

  1. Entity Linking: The system identifies key tokens in the input text and maps them to entities in a Knowledge Graph (e.g., ConceptNet, Wikidata, or WordNet) using string matching and the Text-Rank algorithm.
  2. Knowledge Mapping: To resolve the semantic mismatch between KG embeddings (often trained via TransE) and LLM hidden states, KnowLA utilizes a trainable SwiGLU-based feed-forward network to project KG embeddings into the LLM's semantic space.
  3. Knowledge Fusion: A "knowledge sentinel" attention mechanism calculates the relevance of retrieved entities to the current context. This allows the model to dynamically weigh external knowledge against its internal representations.
Figure 1: The KnowLA architecture, illustrating the integration of the adaptation layer within the LLM and the knowledge injection process.

Throughout this process, the original LLM parameters and KG embeddings remain frozen. Only the KnowLA layer and standard LoRA adapters are trained, maintaining high efficiency with only about 0.55% additional trainable parameters.

Experimental Results

We evaluated KnowLA across six benchmarks, including CommonsenseQA, SIQA, and BIG-Bench Hard (BBH), using three distinct Knowledge Graphs: WordNet, ConceptNet, and Wikidata.

Figure 5: Performance comparison of KnowLA against Llama 2, Alpaca2 (LoRA), and RAG baselines (Contriever, KAPING) across multiple reasoning benchmarks.

As shown in the results, KnowLA consistently outperforms standard LoRA-based finetuning (Alpaca2) across almost all datasets. For example, on CommonsenseQA, accuracy rose from 56.92% (Alpaca2) to 58.39% (KnowLA + ConceptNet). Notably, KnowLA also surpasses Retrieval-Augmented Generation (RAG) baselines like KAPING and Contriever. This suggests that deeper integration of knowledge into the model's hidden states is more effective than simply appending facts to the input prompt.

Key Insights: Knowledge Activation

A significant finding of this work is that KnowLA does more than just inject new facts; it acts as a "key" to unlock the LLM's internal memory. Analysis of the hidden states revealed that KnowLA induces higher activity in the upper Feed-Forward Network (FFN) layers compared to standard LoRA. This "knowledge activation" allows the model to better utilize its own parameterized knowledge to answer questions. Furthermore, the structured nature of KG embeddings appears to improve general reasoning flow, as evidenced by performance gains even on tasks not directly related to the specific knowledge in the KG.

Conclusion and Future Work

KnowLA demonstrates that knowledgeable adaptation is a powerful and efficient way to enhance LLMs. While currently limited by the coverage of single Knowledge Graphs, future work will explore multi-source fusion and lifelong learning techniques to keep the model's knowledge base current and comprehensive.

Summary updated

Abstract

Parameter-efficient finetuning (PEFT) is a key technique for adapting large language models (LLMs) to downstream tasks. In this paper, we study leveraging knowledge graph embeddings to improve the effectiveness of PEFT. We propose a knowledgeable adaptation method called KnowLA. It inserts an adaptation layer into an LLM to integrate the embeddings of entities appearing in the input text. The adaptation layer is trained in combination with LoRA on instruction data. Experiments on six benchmarks with two popular LLMs and three knowledge graphs demonstrate the effectiveness and robustness of KnowLA. We show that KnowLA can help activate the relevant parameterized knowledge in an LLM to answer a question without changing its parameters or input prompts.
BibTeX
@inproceedings{luo2024knowla,
  author = {Xindi Luo and Zequn Sun and Jing Zhao and Zhe Zhao and Wei Hu},
  title = {{KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=i9aM2K3gYO},
}
Figure