Astronomical Knowledge Entity Extraction in Astrophysics Journal Articles via Large Language Models

Wujun Shao, Yaohua Hu, Pengli Ji, Xiaoran Yan, Dongwei Fan, Rui Zhang, Mi, Linying, Chen, Lang, Zhang, RuiarXiv 2023 · DOI · arXiv

TL;DR

This paper introduces Prompt-KEE, a specialized prompting strategy designed to extract celestial object identifiers and telescope names from dense astrophysical literature using Large Language Models (LLMs). The study demonstrates that high-end models like GPT-4 and Claude 2 can effectively automate this complex extraction task, significantly reducing the manual effort required to build astronomical knowledge graphs and literature retrieval systems.

What Problem Does This Paper Address?

The field of astronomy is experiencing an information explosion, with a massive volume of papers published across repositories like ADS and ArXiv. To make this data searchable and linkable, researchers need to extract "knowledge entities"—specifically celestial object identifiers (e.g., LAMOST J151003.74+305407.3) and telescope names (e.g., Zwicky Transient Facility).

Traditional methods for this task, known as Named Entity Recognition (NER), rely on:

  1. Rule-based systems: Brittle and difficult to maintain as naming conventions evolve.
  2. Dictionary-based methods: Limited by the scope of the pre-defined list.
  3. Supervised Machine Learning: Requires massive amounts of manually labeled training data, which is expensive and time-consuming to produce for niche scientific domains.

The authors explore whether pre-trained LLMs can solve these issues through "zero-shot" or "few-shot" prompting, bypassing the need for specialized training data.

Methods

The authors propose Prompt-KEE, a two-stage conversation framework. The strategy breaks a prompt down into five key elements:

  1. Task Descriptions: Assigning the LLM the role of an "experienced astronomer."
  2. Entity Definitions: Providing formal definitions (sourced from Wikipedia) to ground the model’s understanding of what constitutes a "telescope" or "celestial object."
  3. Task Emphasis: Specific constraints and rules, such as "do not create entities not in the text" and "prioritize abbreviated forms."
  4. Task Examples: Providing a few "gold standard" input-output pairs (few-shot learning).
  5. Second Conversation: A self-verification step where the model is asked to re-examine its initial output to correct omissions or errors.

The researchers tested eight different combinations of these elements across four models: GPT-4, Claude 2, GPT-3.5, and Llama-2-70B.

Datasets

The study utilized a newly constructed dataset based on 30 astrophysics journal articles from authoritative sources like ApJ, A&A, and MNRAS. Because LLMs have varying "context windows" (the amount of text they can process at once), the authors created two versions of the data:

  • Full Texts: Used for GPT-4 and Claude 2, which support long-context processing.
  • Paragraph Collections: The articles were segmented into individual paragraphs to accommodate the smaller context windows of Llama-2 and GPT-3.5.

The ground truth for evaluation was manually annotated by the authors.

Key Insights and Ideas

  • The Power of "Task Emphasis": One of the most significant findings was that adding specific constraints (Task Emphasis) improved performance more consistently than simply adding more examples. It helped reduce "hallucinations" (making up non-existent stars).
  • Telescopes are "Easier" than Stars: All models performed better at identifying telescopes than celestial identifiers. The authors attribute this to the fact that telescope names often appear with distinctive keywords (like "survey" or "observatory") and belong to a smaller, more stable vocabulary. Celestial identifiers, by contrast, are often strings of coordinates and letters that vary wildly by project.
  • The "Example Bias": In smaller models like Llama-2, providing examples sometimes decreased precision. The models became biased toward the specific formats shown in the examples, occasionally ignoring valid entities in the text that looked different.
  • Precision vs. Recall Trade-off: Processing full texts led to higher precision (fewer false positives), likely because the model had the full context of the paper. Processing paragraph-by-paragraph led to higher recall (finding more entities), as the model could focus its "attention" on a smaller slice of text at a time.

Key Results

  • GPT-4 was the top performer. On full-text extraction using the "All" prompt (combining all five elements), it achieved an F1-score of 0.767 for celestial objects and 0.816 for telescopes.
  • Claude 2 followed closely, showing a particular strength in identifying telescope names with high recall.
  • Self-Correction Works: The "Second Conversation" step consistently improved the F1-score across models, proving that LLMs are capable of catching their own mistakes when prompted to double-check.
  • Llama-2-70B struggled significantly with precision (often below 0.15), though it maintained a respectable recall, suggesting it "guesses" too often.

Limitations and Future Work

  • Structured Data: The study explicitly ignored tables and figures. In astrophysics, a significant portion of object identifiers are found within tables, which remain a challenge for current LLMs to parse accurately.
  • Entity Diversity: The work focused only on two entity types. Future work could expand to redshifts, chemical abundances, or galaxy classifications.
  • Cost and Latency: While LLMs save on human annotation time, the API costs for processing thousands of full-text papers via GPT-4 are non-trivial.
  • Boundary Recognition: LLMs still occasionally struggle with where an entity name starts and ends, especially with complex coordinate-based identifiers.

Summary updated

Abstract

Astronomical knowledge entities, such as celestial object identifiers, are crucial for literature retrieval and knowledge graph construction, and other research and applications in the field of astronomy. Traditional methods of extracting knowledge entities from texts face challenges like high manual effort, poor generalization, and costly maintenance. Consequently, there is a pressing need for improved methods to efficiently extract them. This study explores the potential of pre-trained Large Language Models (LLMs) to perform astronomical knowledge entity extraction (KEE) task from astrophysical journal articles using prompts. We propose a prompting strategy called Prompt-KEE, which includes five prompt elements, and design eight combination prompts based on them. Celestial object identifier and telescope name, two most typical astronomical knowledge entities, are selected to be experimental object. And we introduce four currently representative LLMs, namely Llama-2-70B, GPT-3.5, GPT-4, and Claude 2. To accommodate their token limitations, we construct two datasets: the full texts and paragraph collections of 30 articles. Leveraging the eight prompts, we test on full texts with GPT-4 and Claude 2, on paragraph collections with all LLMs. The experimental results demonstrated that pre-trained LLMs have the significant potential to perform KEE tasks in astrophysics journal articles, but there are differences in their performance. Furthermore, we analyze some important factors that influence the performance of LLMs in entity extraction and provide insights for future KEE tasks in astrophysical articles using LLMs.
BibTeX
@misc{shao2023astronomical,
  author = {Wujun Shao and Yaohua Hu and Pengli Ji and Xiaoran Yan and Dongwei Fan and Rui Zhang and Mi, Linying and Chen, Lang and Zhang, Rui},
  title = {{Astronomical Knowledge Entity Extraction in Astrophysics Journal Articles via Large Language Models}},
  year = {2023},
  eprint = {2310.17892},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.17892},
}
Figure