TL;DR
This paper introduces Prompt-KEE, a specialized prompting strategy designed to extract celestial object identifiers and telescope names from dense astrophysical literature using Large Language Models (LLMs). The study demonstrates that high-end models like GPT-4 and Claude 2 can effectively automate this complex extraction task, significantly reducing the manual effort required to build astronomical knowledge graphs and literature retrieval systems.
What Problem Does This Paper Address?
The field of astronomy is experiencing an information explosion, with a massive volume of papers published across repositories like ADS and ArXiv. To make this data searchable and linkable, researchers need to extract "knowledge entities"—specifically celestial object identifiers (e.g., LAMOST J151003.74+305407.3) and telescope names (e.g., Zwicky Transient Facility).
Traditional methods for this task, known as Named Entity Recognition (NER), rely on:
- Rule-based systems: Brittle and difficult to maintain as naming conventions evolve.
- Dictionary-based methods: Limited by the scope of the pre-defined list.
- Supervised Machine Learning: Requires massive amounts of manually labeled training data, which is expensive and time-consuming to produce for niche scientific domains.
The authors explore whether pre-trained LLMs can solve these issues through "zero-shot" or "few-shot" prompting, bypassing the need for specialized training data.
Methods
The authors propose Prompt-KEE, a two-stage conversation framework. The strategy breaks a prompt down into five key elements:
- Task Descriptions: Assigning the LLM the role of an "experienced astronomer."
- Entity Definitions: Providing formal definitions (sourced from Wikipedia) to ground the model’s understanding of what constitutes a "telescope" or "celestial object."
- Task Emphasis: Specific constraints and rules, such as "do not create entities not in the text" and "prioritize abbreviated forms."
- Task Examples: Providing a few "gold standard" input-output pairs (few-shot learning).
- Second Conversation: A self-verification step where the model is asked to re-examine its initial output to correct omissions or errors.
The researchers tested eight different combinations of these elements across four models: GPT-4, Claude 2, GPT-3.5, and Llama-2-70B.
Datasets
The study utilized a newly constructed dataset based on 30 astrophysics journal articles from authoritative sources like ApJ, A&A, and MNRAS. Because LLMs have varying "context windows" (the amount of text they can process at once), the authors created two versions of the data:
- Full Texts: Used for GPT-4 and Claude 2, which support long-context processing.
- Paragraph Collections: The articles were segmented into individual paragraphs to accommodate the smaller context windows of Llama-2 and GPT-3.5.
The ground truth for evaluation was manually annotated by the authors.
Key Insights and Ideas
- The Power of "Task Emphasis": One of the most significant findings was that adding specific constraints (Task Emphasis) improved performance more consistently than simply adding more examples. It helped reduce "hallucinations" (making up non-existent stars).
- Telescopes are "Easier" than Stars: All models performed better at identifying telescopes than celestial identifiers. The authors attribute this to the fact that telescope names often appear with distinctive keywords (like "survey" or "observatory") and belong to a smaller, more stable vocabulary. Celestial identifiers, by contrast, are often strings of coordinates and letters that vary wildly by project.
- The "Example Bias": In smaller models like Llama-2, providing examples sometimes decreased precision. The models became biased toward the specific formats shown in the examples, occasionally ignoring valid entities in the text that looked different.
- Precision vs. Recall Trade-off: Processing full texts led to higher precision (fewer false positives), likely because the model had the full context of the paper. Processing paragraph-by-paragraph led to higher recall (finding more entities), as the model could focus its "attention" on a smaller slice of text at a time.
Key Results
- GPT-4 was the top performer. On full-text extraction using the "All" prompt (combining all five elements), it achieved an F1-score of 0.767 for celestial objects and 0.816 for telescopes.
- Claude 2 followed closely, showing a particular strength in identifying telescope names with high recall.
- Self-Correction Works: The "Second Conversation" step consistently improved the F1-score across models, proving that LLMs are capable of catching their own mistakes when prompted to double-check.
- Llama-2-70B struggled significantly with precision (often below 0.15), though it maintained a respectable recall, suggesting it "guesses" too often.
Limitations and Future Work
- Structured Data: The study explicitly ignored tables and figures. In astrophysics, a significant portion of object identifiers are found within tables, which remain a challenge for current LLMs to parse accurately.
- Entity Diversity: The work focused only on two entity types. Future work could expand to redshifts, chemical abundances, or galaxy classifications.
- Cost and Latency: While LLMs save on human annotation time, the API costs for processing thousands of full-text papers via GPT-4 are non-trivial.
- Boundary Recognition: LLMs still occasionally struggle with where an entity name starts and ends, especially with complex coordinate-based identifiers.
Summary updated