Fine-tuning Language Models for Triple Extraction with Data Augmentation

Yujia Zhang, Tyler Sadler, Mohammad Reza Taesiri, Wenjie Xu, Marek ReformatACL 2024

TL;DR

This paper explores how to fine-tune large language models (LLMs) for the task of triple extraction, a crucial step in building knowledge graphs. By creatively augmenting existing datasets and developing tailored prompts, the researchers demonstrate that even smaller, fine-tuned models can achieve performance comparable to or exceeding state-of-the-art models like GPT-4.

What Problem Does This Paper Address?

Knowledge graphs (KGs) are powerful tools for organizing and understanding information, but their construction relies heavily on extracting structured data (triples: subject, predicate, object) from vast amounts of unstructured text. This process is challenging due to the complexity of natural language, implicit references, and the sheer volume of data. While large language models (LLMs) like ChatGPT and GPT-4 show promise in understanding text, effectively and efficiently extracting high-quality triples from them remains an open problem. Existing methods often struggle with consistency, coverage, and the specific format required for KG construction, particularly when aiming for a standardized ontology like DBpedia.

Methods

The core of this work lies in data augmentation and prompt engineering to fine-tune LLMs for triple extraction. Instead of relying solely on existing datasets, the authors developed a multi-stage process to create richer training data. This involved:

  • Text2triples: Generating triples from text.
  • Explanations: Using GPT-3.5 to explain the triple extraction process, helping models understand why certain triples are formed.
  • Triples2text: Generating text from triples, reinforcing the relationship between structured and unstructured data.
  • Reflection: Using a pre-trained Vicuna model to review and correct extracted triples, further refining the data.

They also experimented with different prompt formats to guide the LLMs, ensuring consistency between training and inference. The fine-tuning itself was performed using Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA, allowing them to train models with billions of parameters efficiently.

Datasets

The primary dataset used for training and evaluation was WebNLG. The authors created several augmented versions of this dataset:

  • WebNLG-combined: Combines Text2triples, Explanations, and Triples2text.
  • WebNLG-combined-with-reflections: Adds the Reflection data to WebNLG-combined.
  • WebNLG-reflections-updated-instructions: Uses updated instructions with the reflection data.

They also conducted experiments on other datasets for comparison and to assess model generalization:

  • KELM: A large-scale synthetic corpus.
  • DocRED: A document-level relation extraction dataset.
  • FewREL: A few-shot relation classification dataset.
  • SKE: A Chinese dataset for triple extraction.

Key Insights and Ideas

  • Data Augmentation is Key: The study strongly suggests that the quality and diversity of training data, achieved through creative augmentation strategies, are more critical than simply using massive amounts of raw text. The "Reflection" and "Explanation" steps significantly improved model performance.
  • Smaller Models Can Shine: Even models with 7 billion parameters, when fine-tuned with the right data and prompts, can outperform larger, general-purpose models like GPT-4 on specific tasks like triple extraction. This highlights the power of domain-specific fine-tuning.
  • Prompt Engineering Matters: The way instructions are phrased and the context provided to the LLM significantly impacts its ability to perform triple extraction accurately.
  • Ontology Consistency is Crucial: For KG construction, ensuring triples adhere to a specific ontology (like DBpedia) is vital. The authors emphasized the importance of this consistency in their data preparation.
  • Multilingualism Requires Exposure: Models trained on data predominantly in one language struggle with tasks involving other languages, as seen with the SKE dataset and Orca-mini-3.

Key Results

The fine-tuned 7b models, particularly Orca-mini-3-7b and Mistral-t5-7b, achieved the best performance on the WebNLG dataset, often surpassing GPT-4 in terms of F1 scores across various evaluation criteria (Type, Partial, Exact, Strict). The performance improvements were most pronounced in the "Exact" and "Strict" evaluation metrics, indicating a higher degree of accuracy in generating identical triples. The authors found that the WebNLG-reflections-updated-instructions dataset yielded the best results, demonstrating the effectiveness of their comprehensive augmentation approach.

Limitations and Future Work

The authors acknowledge several limitations:

  • Hallucinations and Looping: Fine-tuned models occasionally "hallucinate" incorrect information (e.g., wrong birthplaces) and can get stuck in repetitive loops, generating output until the token limit is reached.
  • Dataset Specificity: While models performed well on WebNLG, their performance on other datasets like DocRED was poor, suggesting that models can be overly specialized and struggle with different data formats or relation types.
  • Quality of Other Datasets: The quality of reference triples in datasets like DocRED and FewRel was noted as being poor, with limited relations and incoherent structures, hindering effective training.

Future work could focus on mitigating these hallucination and looping issues, developing more robust methods for handling diverse ontologies and relation types, and exploring even more advanced data augmentation and prompt engineering techniques.

Summary updated

BibTeX
@inproceedings{zhang2024fine,
  author = {Yujia Zhang and Tyler Sadler and Mohammad Reza Taesiri and Wenjie Xu and Marek Reformat},
  title = {{Fine-tuning Language Models for Triple Extraction with Data Augmentation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure