Heterogeneous Dependency Graph-Guided Attentionfor Patent Representation Learning

Yongmin Yoo, Qiongkai Xu, Zhangkai Wu, Longbing CaoarXiv 2026 · arXiv

TL;DR

Standard language models treat patent claims as flat text sequences, completely ignoring the complex legal and technical hierarchies that connect them. This paper introduces PHAGE, a framework that injects a heterogeneous claim dependency graph directly into Transformer self-attention during training. Crucially, these graph-specific components are stripped away at inference, delivering state-of-the-art patent representation performance with zero computational overhead at deployment.

What Problem Does This Paper Address?

In patent analysis, tasks like classification and prior-art retrieval heavily rely on Pre-trained Language Models (PLMs). However, patents possess a unique structural characteristic: claims form a strict, directed dependency hierarchy where dependent claims inherit and narrow the scope of antecedent claims.

Standard PLMs treat text as a linear sequence of tokens. This flat encoding conflates adjacent but completely unrelated claims while failing to capture explicit connections between distant claims.

Injecting this structure into Transformers introduces two major hurdles:

  1. Heterogeneous Reliability: Claim relationships aren't uniform. They include highly reliable explicit legal citations as well as noisier, inferred technical term flows. Treating them equally can introduce harmful noise.
  2. Granularity Mismatch: Patent dependencies exist at the claim level, but Transformer self-attention operates at the token level.

Methods

The authors propose the Patent Heterogeneous Attention-Guided Graph Encoder (PHAGE), which solves these challenges through a three-stage pipeline:

Figure 2: Overview of the PHAGE Framework Pipeline. The framework consists of three stages: (1) constructing a Claim Dependency Graph (CDG), (2) encoding claims via a relation-guided structure-aware attention mechanism, and (3) refining representations using a contrastive fine-tuning objective.
  1. Heterogeneous Claim Dependency Graph (CDG) Construction: PHAGE uses a deterministic pipeline to extract four distinct relation types between claims: explicit legal citations ($r_{cite}$), technical term flows ($r_{term}$), functional couplings ($r_{func}$), and combined signals ($r_{both}$).
  2. Graph-Guided Attention: To bridge the claim-to-token gap, PHAGE projects the claim graph into token-level attention. It applies a hard connectivity mask ($C$) that restricts cross-claim attention exclusively to valid CDG edges. It also introduces layer-specific, learnable relation-aware biases ($B^{(l)}$) to modulate attention strength based on the relationship type.
  3. Dual-Granularity Contrastive Fine-Tuning: The model optimizes a joint objective function. At the macro level, it uses International Patent Classification (IPC) taxonomy to align document embeddings. At the micro level, it uses a claim-level contrastive loss to pull structurally connected claims closer together in the vector space.

The Clever Twist: At inference time, the connectivity mask and relation biases are completely omitted. The model reverts to a standard, vanilla BERT forward pass. This means you reap all the structural benefits during training without paying any graph-construction or graph-computation tax at runtime.

Datasets

  • Training Corpus: 131,653 USPTO patents (filed 2013–2017) sourced from the Harvard USPTO Patent Dataset, spanning 610 unique IPC subclasses.
  • Evaluation Benchmarks:
  • CLEF-IP & USPTO-70k: Used to evaluate patent classification and clustering performance.
  • DAPFAM: A cross-domain retrieval benchmark consisting of 1,247 query families and 45,336 target patents.

Key Insights and Ideas

  • Structure as a Training Scaffold: PHAGE proves that you don't need to run heavy Graph Neural Networks (GNNs) or maintain graph indices at deployment. By using the graph purely to guide attention weights during fine-tuning, the language model internalizes the structural inductive bias directly into its standard parameters.
  • The Power of Implicit Edges: While explicit legal citations are highly reliable, implicit relations (like functional coupling) actually make up the vast majority (60.3%) of the extracted graph edges. Ablation studies revealed that removing these implicit technical relations significantly hurts downstream performance, proving they provide crucial semantic context.
  • Depth-Dependent Attention Specialization: The learned relation biases revealed an intriguing pattern: lower layers of the Transformer naturally assign higher biases to cross-claim relations, whereas upper layers increasingly favor intra-claim token interactions.

Key Results

PHAGE consistently outperformed domain-adapted, citation-aware, and post-hoc graph baselines across all downstream tasks under a frozen-encoder evaluation protocol.

Patent Classification

On the CLEF-IP and USPTO-70k datasets, PHAGE achieved substantial gains in both Micro-F1 and Macro-F1. Notably, it outperformed its own backbone (BERT-for-Patents) by 9.7 points in Macro-F1 on CLEF-IP, demonstrating a much better capability to represent rare patent categories.

Patent Classification Results Table

Patent Retrieval

On the cross-domain DAPFAM benchmark, PHAGE achieved an average NDCG@100 of 0.498, easily outpacing strong citation-aware models like QaECTER (0.379) and more than doubling the performance of vanilla BERT-for-Patents (0.228).

Limitations and Future Work

  • English-Language Bias: The technical term flow component ($r_{term}$) relies on English grammar rules (specifically tracking transitions from indefinite articles like a/an to definite articles like the/said). Adapting PHAGE to languages without article systems (e.g., Chinese, Japanese, Korean) will require language-specific extraction rules.
  • Context Truncation: Because the underlying BERT backbone is limited to a 512-token window, longer patents suffer from truncation. Claims and dependency edges that fall outside this window do not contribute structural guidance during training.
  • Inference Trade-offs: While graph-free inference is highly efficient, the authors note that they did not explicitly compare it against graph-conditioned inference using the exact same checkpoint. Quantifying the potential performance drop of discarding the graph at runtime remains an open question for future work.

Summary updated

Abstract

Pre-trained language models advance patent classification and retrieval by encoding claims as flat token sequences, but they overlook the dependency hierarchy among claims. Incorporating this hierarchy into self-attention poses two challenges. First, claim dependencies include relation types with different levels of reliability, so treating them uniformly may allow noisy technical relations to interfere with more reliable legal citations. Second, claim dependencies are defined at the claim level, whereas Transformer attention operates over tokens, making direct structural injection nontrivial. We propose the Patent Heterogeneous Attention-Guided Graph Encoder (PHAGE), which constructs a typed claim graph that distinguishes legal citations from technical relations. PHAGE projects this claim-level topology into token-level attention through a connectivity mask and learnable relation-aware biases, and fine-tunes the encoder using a dual-granularity contrastive objective that combines inter-patent taxonomy with intra-patent topology. At inference, the graph-specific attention components are removed, allowing representations to be generated through a standard encoder forward pass without CDG construction. Experiments on patent classification, retrieval, and clustering show that PHAGE consistently outperforms domain-adapted and citation-aware baselines, demonstrating the value of claim-level structural guidance for graph-free patent representation learning.
BibTeX
@misc{yoo2026heterogeneous,
  author = {Yongmin Yoo and Qiongkai Xu and Zhangkai Wu and Longbing Cao},
  title = {{Heterogeneous Dependency Graph-Guided Attentionfor Patent Representation Learning}},
  year = {2026},
  eprint = {2605.10073},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.10073},
}

Loading related content...

Figure