TL;DR
Researchers explored whether Large Language Models (LLMs) like ChatGPT can automate the extraction of specialized mathematical concepts from research abstracts. While ChatGPT serves as a useful "decent helper" for preliminary indexing, it currently fails to match the precision of human experts, particularly in distinguishing between domain-specific concepts and general academic "meta-language."
What Problem Does This Paper Address?
Automatic Term Extraction (ATE)—the process of identifying key phrases and concepts in a text—is a foundational task for building knowledge graphs and searchable indices. However, mathematics presents a unique "final boss" challenge for ATE.
Mathematical text is dense with LaTeX formatting, symbols, and a highly specialized vernacular. Furthermore, there is a subtle distinction between mathematical concepts (e.g., "sheaf," "abelian group") and mathematical practice terms (e.g., "conjecture," "proof," "theorem"). While the latter are common in math papers, they shouldn't necessarily appear in a subject index. The authors set out to see if LLMs could navigate these nuances in the field of Category Theory, a notoriously abstract branch of pure mathematics.
Methods
The authors developed a multi-stage pipeline to compare human performance against ChatGPT:
- Human Annotation Guidelines: To minimize subjectivity, the team created strict rules (e.g., use singular forms, treat multi-word concepts as "black boxes," and exclude meta-concepts like "corollary").
- Annotation Tool: They repurposed a tool from Natural Language Inference research to allow humans to highlight and edit concepts within sentences efficiently.
- Prompt Engineering: The team tested three levels of prompting for ChatGPT:
- Initial: Simple task description with a few examples.
- Specialized: Added instructions to avoid plural forms and names.
- Detailed: Included "Chain of Thought" style reasoning, explaining why certain terms were chosen or rejected.
- Evaluation: They used the Jaccard similarity index—a ratio of the intersection of two sets over their union—to measure agreement between different annotators (human vs. human and human vs. machine).
Datasets
The study utilized a corpus of 755 abstracts from the journal Theory and Applications of Categories (TAC), totaling roughly 3,200 sentences.
- Pilot Set: 100 sentences used for initial human/LLM comparison.
- Experiment Set: 436 sentences where one human and ChatGPT were compared, with a second human acting as an adjudicator for disagreements.
- nLab Dataset: A larger-scale test involving 55,000 sentences from the nLab wiki, intended for future community-driven verification.
Key Insights and Ideas
- The "Implicit" Gap: Humans are excellent at inferring concepts that aren't explicitly written. For example, if a paper mentions "non-Artinian rings," a human knows "Artinian ring" is the core concept. LLMs often miss these implicit connections.
- Human Disagreement is High: Even with strict guidelines, human experts only achieved a Jaccard agreement of 0.75 to 0.8. This highlights how subjective "what makes a concept" truly is.
- LLMs are "Noisy": ChatGPT tends to over-extract. It frequently identifies general academic terms (e.g., "previous work," "motivation," "issue") as mathematical concepts, despite instructions to the contrary.
- The "Black Box" Strategy: The authors found it best to treat complex terms as single units. Instead of breaking down "enriched accessible categories," they kept the full span while also listing sensible sub-spans like "accessible category."
Key Results
The quantitative results showed a clear performance gap. While human annotators agreed with each other at a relatively high rate, ChatGPT struggled to align with the expert consensus.
In the pilot study, the Jaccard similarity between ChatGPT and the union of human experts was only 0.45, rising slightly to 0.5 after manual filtering. In contrast, human-to-human agreement hovered around 0.75–0.79. In the second experiment, manual adjudication and filtering of ChatGPT's output improved its similarity to humans from 0.531 to 0.631, suggesting that LLMs are best used as a first-pass tool that still requires human oversight.
Limitations and Future Work
The authors acknowledge that their results are tied to Category Theory; results might differ in more "crystallized" fields like undergraduate calculus where terminology is more standardized. Furthermore, the study relied solely on ChatGPT; other LLMs might yield different results.
Future work aims to scale this to the 55,000-sentence nLab corpus and involve the broader mathematical community via a "living experiment" website. The ultimate goal is to move toward the formalization of mathematics by building a knowledge graph rooted in "mathematical English"—the actual lingua franca of researchers.
Summary updated