TL;DR
This paper tackles the critical issue of "hallucination" in Large Language Models (LLMs) when generating structured outputs like workflows. They propose a Retrieval-Augmented Generation (RAG) approach that significantly reduces these hallucinations, making LLM-generated workflows more trustworthy and deployable in enterprise settings. Notably, their method allows for smaller, more resource-efficient LLMs without sacrificing performance.
What Problem Does This Paper Address?
Generative AI (GenAI) models, particularly Large Language Models (LLMs), have shown remarkable capabilities in understanding and generating human-like text. However, a significant drawback is their tendency to "hallucinate" – producing outputs that are factually incorrect, nonsensical, or invent information that doesn't exist. This is a major barrier to adopting GenAI in real-world applications, especially in enterprise environments where accuracy and reliability are paramount. The paper focuses on structured output tasks, specifically generating workflows represented as JSON documents. Hallucinations in this context mean generating non-existent steps, properties, or table names, rendering the workflow unusable or incorrect. While fine-tuning LLMs can help, it's often prohibitively expensive, and a more efficient solution is needed.
Methods
The core of the paper's solution is Retrieval-Augmented Generation (RAG). Instead of solely relying on the LLM's internal knowledge, their RAG system augments the LLM's prompt with relevant information retrieved from a knowledge base.
Here's a breakdown of their approach:
- Retriever Training: They train a specialized retriever model to map natural language queries to structured JSON objects representing workflow steps and table names. This is crucial because the domain of workflow steps can be vast and highly customizable. They use a siamese transformer encoder and train it with a contrastive loss to align the semantic spaces of natural language queries and structured JSON data. This fine-tuning is key to improving retrieval accuracy in their specific domain.
- LLM Training (RAG Fashion): The retrieved JSON objects (steps and table names) are then prepended to the user's natural language query as part of the LLM's prompt. This provides the LLM with concrete, relevant information, guiding it to generate a more accurate and less hallucinatory structured output. They opt for separate training of the retriever and LLM for simplicity and modularity.
- Structured Output Generation: The LLM then generates the workflow in JSON format using greedy decoding, leveraging the augmented prompt.
Datasets
The authors created their own datasets from internal deployments of their enterprise platform.
- Training & Evaluation Datasets: They extracted approximately 4,000 deployed workflows and had annotators write natural language requirements for them. An additional 1,000 samples were generated deterministically for simple workflows. A "Human Eval" split was created from expert users simulating interactions to ensure realistic evaluation.
- Out-of-Domain (OOD) Datasets: To assess generalization, they created five OOD splits from different enterprise deployments (e.g., HR, finance), which contained workflows with varying degrees of difference from their training data (up to 70% of steps not present in the training set).
Key Insights and Ideas
- RAG is Crucial for Structured Outputs: The paper strongly demonstrates that RAG is not just for factual question answering but is highly effective in reducing hallucinations in structured output generation tasks like workflow creation.
- Fine-tuned Retriever is Key: Simply using off-the-shelf retrievers wasn't enough. Fine-tuning a retriever on domain-specific data significantly improved its ability to map natural language to the correct JSON steps and table names.
- Smaller LLMs + RAG = Efficiency: A major finding is that a RAG-augmented smaller LLM (e.g., 3B parameters) can achieve comparable or better results in terms of hallucination reduction than a much larger LLM without RAG. This has significant implications for deployment costs and resource constraints.
- Decoupling Retriever and LLM is Beneficial: Separately training and deploying the retriever and LLM offers modularity, allows for independent optimization, and enables the retriever to be reused across different use cases.
Key Results
- Hallucination Reduction: Using RAG, the percentage of hallucinated steps and tables on the "Human Eval" split decreased significantly. For StarCoderBase LLMs, hallucinated steps dropped from up to 21% to less than 7.5%, and hallucinated tables from up to 21% to less than 4.5%.
- Performance with Smaller Models: A RAG fine-tuned 3B StarCoderBase model was competitive with a 15.5B version without RAG on key metrics like Trigger EM and Bag of Steps, while maintaining low hallucination rates.
- Out-of-Domain Performance: The RAG approach maintained performance across various OOD splits, showing good generalization capabilities. On average, OOD metrics were similar to in-domain results.
- Retriever Performance: The fine-tuned all-mpnet-base-v2 retriever achieved a Recall@15 of 0.743 for steps and 0.766 for tables on the "Human Eval" split, outperforming off-the-shelf models.
Limitations and Future Work
- Retriever Errors: The paper acknowledges that errors can still occur due to issues with the retriever (failing to suggest crucial steps) or the LLM (not utilizing suggestions effectively). They suggest decomposing queries for more precise retrieval.
- LLM Understanding of Logic: For complex logic steps (like IF, TRY, FOREACH), the LLM sometimes struggles to fully grasp the semantics, leading to incorrect structures. Synthetic data generation is proposed as a solution.
- Response Time: While acceptable, they suggest further optimizations like changing the output format to YAML, using speculative decoding, or streaming outputs.
- Joint Training: For scientific purposes, they suggest exploring joint training of the retriever and LLM to potentially improve synergy.
- Generalization to More Domains: While OOD performance was good, further work could explore broader domain generalization.
Summary updated