Introduction
Patent classification is a critical task where mislabeling can lead to significant legal and economic consequences. While Large Language Models (LLMs) can generate natural language rationales to explain their predictions, these explanations often suffer from logical inconsistencies or "hallucinations." Standard knowledge distillation techniques typically assume teacher-generated rationales are ground truth, inadvertently passing these errors to smaller student models.
To address this, researchers have introduced Self-Filtered Distillation (SFD). This framework treats LLM-generated rationales as "trust signals" rather than absolute facts. By implementing unsupervised metrics to evaluate reasoning quality, SFD filters out unreliable data, significantly improving the performance and reliability of distilled student models.
The SFD Framework
The SFD framework operates in three distinct steps to ensure that only high-quality reasoning is used to train the student model.
1. Rationale Generation
A teacher LLM generates both a predicted label and a set of candidate rationales for a given input patent document. This step provides the raw material for the distillation process.
2. Reasoning Quality Evaluation
The framework calculates a Combined Trust Score (CTS) based on three unsupervised "trust indicators":
- Self-Consistency (SC): Measures the stability of reasoning across multiple runs. High variance in explanations for the same input leads to a lower score.
- Class Entailment Alignment (CEA): Anchors the reasoning in domain-specific knowledge by comparing rationales against official Cooperative Patent Classification (CPC) definitions.
- LLM Agreement Scoring (LAS): Utilizes an independent "judge" LLM to verify the plausibility of the rationale-label pair, mitigating overconfidence in the teacher model.
3. Trust-aware Training Control
The CTS is used to guide the training of the student model through two mechanisms:
- Hard Filtering: Rationales with a CTS below a predefined threshold ((\tau)) are discarded entirely to prevent the student from learning "garbage" logic.
- Soft Weighting: The loss function is weighted by the trust score, ensuring the student model learns more from highly reliable examples.
Experimental Results
The researchers evaluated SFD using the USPTO-2M dataset, focusing on the complex CPC subclass hierarchy. The framework was tested across various student model architectures, including Qwen, Phi, Llama, and Falcon.
As shown in the performance comparison, SFD consistently outperforms traditional distillation methods. For instance, using the Falcon3-3B model as a student, the framework achieved a 38.7% relative improvement in Macro-F1 score over the strongest baseline. Furthermore, the study found that a high filtering threshold ((\tau > 0.9)) yielded the best results, suggesting that training on a smaller, high-trust dataset is more effective than using a larger, noisier one.
Key Insights and Conclusion
A primary takeaway from this work is that LLM rationales are highly effective as diagnostic tools for the training process. The Class Entailment Alignment (CEA) metric proved particularly vital, highlighting that specialized domains require grounding in formal taxonomies rather than relying solely on general LLM reasoning.
While SFD shows great promise, the authors note limitations such as the computational overhead of generating multiple rationales and the current focus on English-language patents. Future work aims to expand this to multilingual corpora and further integrate human expert validation into the trust-scoring loop. Ultimately, SFD provides a robust path toward building smaller, more reliable models for high-stakes technical domains.
Summary updated