HGAdapter: Hypergraph-based Adapters in Language Models for Code Summarization and Clone Detection

Guang Yang, Yujie ZhuEMNLP 2025

TL;DR

This paper introduces HGAdapter, a novel adapter module for pre-trained language models (PLMs) that captures high-order correlations in code beyond pairwise token relationships. By representing code structures as hypergraphs, HGAdapter significantly improves performance on code summarization and clone detection tasks, demonstrating the value of richer structural understanding.

What Problem Does This Paper Address?

Modern pre-trained language models (PLMs) have shown remarkable success in code-related tasks like summarization and clone detection. However, these models typically treat code as a sequence of tokens and primarily capture pairwise relationships between them, often through self-attention mechanisms. This overlooks the inherent, complex, high-order structural information present in code. For instance, multiple tokens might form a meaningful unit (like a function call or a complex expression) that is more than the sum of its parts. Existing PLMs struggle to effectively encode these multi-entity relationships, leaving room for improvement in understanding and generating code.

Methods

The core innovation is the HGAdapter, a lightweight module designed to be inserted into existing PLMs. It works in conjunction with a tokens and hyperedges generator.

  1. Tokens and Hyperedges Generator: This component goes beyond standard tokenization. It parses code into an Abstract Syntax Tree (AST) and identifies three types of high-order correlations:
  2. AST family correlation: Tokens that share a common parent in the AST (e.g., operands and operator in an expression).
  3. Lexical correlation: Tokens that form a single semantic unit, like a camelCase or underscore-separated variable/function name that might be split by the tokenizer.
  4. Line correlation: Tokens that reside on the same line of code, representing a cohesive unit of programming logic. These correlations are represented as hyperedges, where a hyperedge can connect any number of tokens.

  5. HGAdapter Module: This module is inspired by Hypergraph Neural Networks (HGNNs) and adapter tuning. It's inserted between PLM layers, with the PLM's parameters frozen during fine-tuning, and only the adapter's parameters are trained. The HGAdapter uses a simplified attention mechanism to aggregate information from tokens to hyperedges and vice-versa, effectively encoding the high-order structural information captured by the hyperedges. This allows the PLM to leverage these richer representations.

Datasets

The paper evaluates HGAdapter on two main types of tasks using established public datasets:

  • Code Summarization:
    • CodeSearchNet: A large dataset containing code snippets and their corresponding natural language descriptions across six programming languages (Ruby, JavaScript, Java, Go, PHP, Python). The version used was filtered and organized by CodeXGLUE.
  • Code Clone Detection:
    • BigCloneBench: A widely used dataset for Java code clone detection, containing pairs of code snippets to determine functional equivalence. This dataset was also provided by CodeXGLUE.

Key Insights and Ideas

  • Beyond Pairwise: The Power of High-Order Correlations: The central idea is that code's structure is richer than simple token sequences. Capturing relationships involving more than two tokens (like AST families, split identifiers, or entire lines) significantly boosts performance.
  • Hypergraphs as a Natural Fit: Hypergraphs are an intuitive and powerful framework for representing these multi-entity relationships, making them ideal for encoding complex code structures.
  • Adapter Efficiency: HGAdapter is designed as a lightweight module, meaning it adds minimal parameters to the PLM, making it efficient for fine-tuning without requiring extensive computational resources.
  • Lexical Correlations are Particularly Impactful: Ablation studies suggest that lexical correlations (e.g., reconstructing split identifiers) have a particularly significant impact on performance, highlighting the importance of preserving semantic units.
  • Encoder-Based Models Benefit More: HGAdapter shows more substantial improvements on encoder-based PLMs (like BERT variants) compared to decoder-based models, likely because encoder models already excel at extracting contextual information, and HGAdapter further refines this by adding structural insights.

Key Results

HGAdapter consistently improved performance across various PLMs and tasks:

  • Code Summarization: HGAdapter achieved significant overall improvements, with gains of up to 2.33 BLEU-4 points over full fine-tuned models and 2.31 points over standard adapter-tuned models on the CodeSearchNet dataset. It also outperformed structural adapters by up to 1.68 points.
  • Code Clone Detection: HGAdapter improved F1 scores by up to 1.28 points over full fine-tuned models and 1.31 points over standard adapter-tuned models on BigCloneBench. It also outperformed structural adapters by up to 1.01 points in F1 score.
  • Parameter Efficiency: HGAdapter adds only 3%-11% additional parameters compared to standard adapters, demonstrating its efficiency.

Limitations and Future Work

  • Scalability to Larger PLMs: The impact of HGAdapter on extremely large-scale PLMs needs further exploration.
  • Increased Latency: While parameter-efficient, HGAdapter does introduce additional computational steps, potentially increasing training time and inference latency.
  • Task Applicability: HGAdapter is currently designed for tasks requiring complete code input and is not directly applicable to tasks like code generation that might involve partial or incremental code.
  • Future Directions: The authors suggest exploring more high-order correlations, integrating more advanced parameter-efficient fine-tuning methods with hypergraphs, and applying the approach to a wider range of code-related tasks. They also propose investigating how to integrate HGAdapter with other PEFT methods or explore correlations in natural language.

Summary updated

BibTeX
@inproceedings{yang2025hgadapter,
  author = {Guang Yang and Yujie Zhu},
  title = {{HGAdapter: Hypergraph-based Adapters in Language Models for Code Summarization and Clone Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure