Zero-Shot Learning for Materials Science Texts: Leveraging Duck Typing Principles

Xin Zhang, Peiliang Zhang, Jingling Yuan, Lin LiAAAI 2025 · OpenReview

TL;DR

This paper introduces MatDuck, a novel zero-shot learning approach for materials science text mining (MSTM) that leverages the "duck typing" principle from programming. MatDuck effectively handles the ambiguity of material science descriptors by generating task-specific class definitions and reframing tasks into a code-like format for Large Language Models (LLMs), achieving significant performance improvements without task-specific fine-tuning.

What Problem Does This Paper Address?

Extracting valuable information from materials science literature is crucial for scientific advancement. This process often relies on "descriptors" – labels for properties, structures, or behaviors. However, these descriptors can be highly ambiguous, with the same term having different meanings depending on the specific task (e.g., "Material" could refer to a synthesis component or a participant in a fuel cell experiment). This ambiguity makes it challenging for models trained on one task to generalize to others, requiring costly retraining or fine-tuning. The paper addresses this gap by proposing a method that can perform MSTM tasks effectively in a zero-shot setting, meaning without needing task-specific labeled data.

Methods

MatDuck is built upon two core principles inspired by "duck typing" (if it walks like a duck and quacks like a duck, it's a duck):

  1. Principle I: Descriptor Meaning through Characteristics: If a descriptor's meaning is unclear, infer it from its characteristic properties. MatDuck achieves this by using LLMs to generate diverse "class definitions" for each descriptor. These definitions include not just the descriptor name but also its associated properties and annotations, effectively capturing its nuanced meaning. This is done using a Chain-of-Thought approach for more precise definitions.
  2. Principle II: Task-Specific Adaptation: The generated class definitions are then adapted to the specific task. MatDuck calculates a "fitness degree" for each class definition based on how well its inferred properties and characteristics align with the actual text of the task. This ensures that the most relevant meaning of the descriptor for that particular task is selected.

Finally, MatDuck reframes MSTM tasks into a code-style format for in-context learning (ICL). This leverages LLMs' strong proficiency in understanding code and object-oriented structures, allowing them to better interpret and perform the tasks without explicit fine-tuning.

Datasets

The paper evaluates MatDuck on eight benchmark datasets across seven MSTM tasks, including:

  • Named Entity Recognition (NER)
  • Relation Extraction (RE)
  • Event Attribute Extraction (EAE)
  • Paragraph Classification (PC)
  • Synthesis Action Retrieval (SAR)
  • Sentence Classification (SC)
  • Slot Filling (SF)

The specific datasets are not explicitly named as newly introduced benchmarks but are described as public materials science text benchmark datasets.

Key Insights and Ideas

  • Duck Typing for NLP: The paper ingeniously applies the "duck typing" concept from programming to natural language processing, specifically for handling ambiguous terms in a specialized domain. This provides a novel framework for descriptor disambiguation.
  • LLMs as Knowledge Elicitors: MatDuck demonstrates that LLMs possess significant embedded material knowledge that can be effectively "evoked" through clever prompting and structured output generation (class definitions).
  • Code-Style ICL is Powerful: Reframing NLP tasks into a code-like structure significantly enhances LLMs' zero-shot performance, suggesting that their code understanding capabilities are a valuable, underutilized resource for text-based tasks.
  • Descriptor Ambiguity is a Major Hurdle: The high descriptor misuse rates (e.g., 43.9% in NER) highlighted in the paper underscore the critical need for methods that can robustly handle semantic variations in specialized scientific text.

Key Results

  • MatDuck significantly improves zero-shot MSTM performance, achieving an average improvement of 11.3% across seven tasks and eight benchmark datasets.
  • When using LLaMA3-8B as a backbone, MatDuck outperforms the best commercial LLMs (GPT-4 and Claude3.5) by 7.2% and 4.5% respectively in zero-shot settings.
  • Ablation studies show that both the inclusion of task-specific properties (Principle I) and task adaptation (Principle II) are crucial for MatDuck's effectiveness, with performance dropping significantly when these components are removed.
  • The number of properties in descriptor class definitions positively correlates with performance, with peak performance observed at 4-6 properties.

Limitations and Future Work

  • Overabundance of Properties: While more properties generally improve performance, too many can lead to diminishing returns or negative effects due to overly long contexts.
  • Future Work: The authors suggest further exploring material knowledge and capabilities within LLMs to advance MSTM and the broader field of materials science. They also highlight the potential for MatDuck to be a user-friendly, plug-and-play solution for researchers without extensive NLP backgrounds.

Summary updated

Abstract

Materials science text mining (MSTM), involving tasks like property extraction and synthesis action retrieval, is pivotal for advancing research by deriving critical insights from scientific literature. Descriptors, serving as essential task labels, often vary in meaning depending on researchers' usage purposes across different mining tasks. (e.g., 'Material' can refer to both synthesis components and participants in fuel cell experiment). This meaning difference makes it difficult for existing methods, fine-tuned to specific task, to handle the same descriptors in other tasks. To overcome above limitation, we propose MatDuck, a simple and effective approach for Zero-Shot MSTM by evoking material knowledge within Large Language Models (LLMs). Specifically, inspired by the Duck Typing principles in programming languages, we present a ClassDefinition-Style Descriptor generation method that evokes task-specific characteristics to address usage variation. Subsequently, we introduce code-style in-context learning for zero-shot tasks, reframing them into code to leverage LLMs' proficiency in code understanding. Extensive experiments on eight benchmark datasets demonstrate that MatDuck, as a plug-and-play approach, significantly improves the Zero-Shot MSTM performance of LLMs by an average of 11.3% across seven tasks.
BibTeX
@inproceedings{zhang2025zero,
  author = {Xin Zhang and Peiliang Zhang and Jingling Yuan and Lin Li},
  title = {{Zero-Shot Learning for Materials Science Texts: Leveraging Duck Typing Principles}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=sDpw37F1uo},
}
Figure