TL;DR
Standard language models treat patent claims as flat text sequences, completely ignoring the complex legal and technical hierarchies that connect them. This paper introduces PHAGE, a framework that injects a heterogeneous claim dependency graph directly into Transformer self-attention during training. Crucially, these graph-specific components are stripped away at inference, delivering state-of-the-art patent representation performance with zero computational overhead at deployment.
What Problem Does This Paper Address?
In patent analysis, tasks like classification and prior-art retrieval heavily rely on Pre-trained Language Models (PLMs). However, patents possess a unique structural characteristic: claims form a strict, directed dependency hierarchy where dependent claims inherit and narrow the scope of antecedent claims.
Standard PLMs treat text as a linear sequence of tokens. This flat encoding conflates adjacent but completely unrelated claims while failing to capture explicit connections between distant claims.
Injecting this structure into Transformers introduces two major hurdles:
- Heterogeneous Reliability: Claim relationships aren't uniform. They include highly reliable explicit legal citations as well as noisier, inferred technical term flows. Treating them equally can introduce harmful noise.
- Granularity Mismatch: Patent dependencies exist at the claim level, but Transformer self-attention operates at the token level.
Methods
The authors propose the Patent Heterogeneous Attention-Guided Graph Encoder (PHAGE), which solves these challenges through a three-stage pipeline:
- Heterogeneous Claim Dependency Graph (CDG) Construction: PHAGE uses a deterministic pipeline to extract four distinct relation types between claims: explicit legal citations ($r_{cite}$), technical term flows ($r_{term}$), functional couplings ($r_{func}$), and combined signals ($r_{both}$).
- Graph-Guided Attention: To bridge the claim-to-token gap, PHAGE projects the claim graph into token-level attention. It applies a hard connectivity mask ($C$) that restricts cross-claim attention exclusively to valid CDG edges. It also introduces layer-specific, learnable relation-aware biases ($B^{(l)}$) to modulate attention strength based on the relationship type.
- Dual-Granularity Contrastive Fine-Tuning: The model optimizes a joint objective function. At the macro level, it uses International Patent Classification (IPC) taxonomy to align document embeddings. At the micro level, it uses a claim-level contrastive loss to pull structurally connected claims closer together in the vector space.
The Clever Twist: At inference time, the connectivity mask and relation biases are completely omitted. The model reverts to a standard, vanilla BERT forward pass. This means you reap all the structural benefits during training without paying any graph-construction or graph-computation tax at runtime.
Datasets
- Training Corpus: 131,653 USPTO patents (filed 2013–2017) sourced from the Harvard USPTO Patent Dataset, spanning 610 unique IPC subclasses.
- Evaluation Benchmarks:
- CLEF-IP & USPTO-70k: Used to evaluate patent classification and clustering performance.
- DAPFAM: A cross-domain retrieval benchmark consisting of 1,247 query families and 45,336 target patents.
Key Insights and Ideas
- Structure as a Training Scaffold: PHAGE proves that you don't need to run heavy Graph Neural Networks (GNNs) or maintain graph indices at deployment. By using the graph purely to guide attention weights during fine-tuning, the language model internalizes the structural inductive bias directly into its standard parameters.
- The Power of Implicit Edges: While explicit legal citations are highly reliable, implicit relations (like functional coupling) actually make up the vast majority (60.3%) of the extracted graph edges. Ablation studies revealed that removing these implicit technical relations significantly hurts downstream performance, proving they provide crucial semantic context.
- Depth-Dependent Attention Specialization: The learned relation biases revealed an intriguing pattern: lower layers of the Transformer naturally assign higher biases to cross-claim relations, whereas upper layers increasingly favor intra-claim token interactions.
Key Results
PHAGE consistently outperformed domain-adapted, citation-aware, and post-hoc graph baselines across all downstream tasks under a frozen-encoder evaluation protocol.
Patent Classification
On the CLEF-IP and USPTO-70k datasets, PHAGE achieved substantial gains in both Micro-F1 and Macro-F1. Notably, it outperformed its own backbone (BERT-for-Patents) by 9.7 points in Macro-F1 on CLEF-IP, demonstrating a much better capability to represent rare patent categories.
Patent Retrieval
On the cross-domain DAPFAM benchmark, PHAGE achieved an average NDCG@100 of 0.498, easily outpacing strong citation-aware models like QaECTER (0.379) and more than doubling the performance of vanilla BERT-for-Patents (0.228).
Limitations and Future Work
- English-Language Bias: The technical term flow component ($r_{term}$) relies on English grammar rules (specifically tracking transitions from indefinite articles like a/an to definite articles like the/said). Adapting PHAGE to languages without article systems (e.g., Chinese, Japanese, Korean) will require language-specific extraction rules.
- Context Truncation: Because the underlying BERT backbone is limited to a 512-token window, longer patents suffer from truncation. Claims and dependency edges that fall outside this window do not contribute structural guidance during training.
- Inference Trade-offs: While graph-free inference is highly efficient, the authors note that they did not explicitly compare it against graph-conditioned inference using the exact same checkpoint. Quantifying the potential performance drop of discarding the graph at runtime remains an open question for future work.
Summary updated