TL;DR
Evaluating natural language generation often requires pinpointing factual errors and hallucinations at the individual word or phrase level. factgenie is an open-source, lightweight framework designed to collect, manage, and visualize span-based text annotations from both human crowdworkers and large language models (LLMs) within a unified web interface.
What Problem Does This Paper Address?
While modern LLMs produce fluent, human-like text, they frequently generate subtle factual inaccuracies, unsupported statements, or irrelevant details. Identifying these issues requires fine-grained, word-level span annotations rather than coarse, text-level quality scores.
Existing annotation tools suffer from notable limitations:
- Many are tied to specific tasks, such as machine translation or retrieval-augmented generation (RAG).
- Most tools support either human annotators (e.g., doccano, Appraise) or LLMs (e.g., LLMeBench), but rarely both in a consistent format.
- Systems that combine both modalities often relegate humans strictly to post-editing roles or require data to be pre-annotated externally before visualization.
Researchers lacked a lightweight, customizable tool capable of running span-level annotation campaigns across both humans and LLMs using a shared evaluation format.
Methods
factgenie pairs a Python/Flask backend with an HTML/Bootstrap 5.3 and jQuery frontend. Its design focuses on rapid prototyping and easy self-hosting:
- Span Annotation & Visualization: The interface uses the YPet library for highlighting text spans. To help annotators ground their judgements, the tool renders structured input data (such as tables, JSON, or RDF graphs) side-by-side with generated text using TinyHTML and Highcharts.JS.
- Data Loaders: Researchers can extend a base
Datasetclass to parse custom input types, with ready-made loaders already available for tabular and structured formats. - Human Annotation Workflow: Researchers set up campaign parameters (error categories, example quotas), write instructions, host the platform on a public URL, and direct crowdsourced workers to the interface.
- LLM Annotation Workflow: Provides a lightweight wrapper supporting both local models via Ollama and cloud models via the OpenAI API. System messages and prompts are configured to force the model to output span annotations in a structured JSON schema identical to the human annotation format.
Datasets
Not discussed in the paper. As a system demonstration, the paper does not release a new benchmark dataset or train models. The authors illustrate the tool's capabilities using an example task: evaluating model-generated weather forecasts against structured meteorological data.
Key Insights and Ideas
- Unified Schema for Humans and LLMs: By constraining LLM API responses into structured JSON spans matching human annotation outputs, researchers can directly compare, benchmark, and interchange human and model-based evaluations.
- Rapid Iteration and Debugging: Researchers can preview inputs, run small-scale test prompts on LLMs, and immediately inspect visual highlights on the text. This tight feedback loop makes it easy to debug prompts or clarify instructions before launching costly crowdsourcing campaigns.
- Grounding Through Custom Input Rendering: Factual evaluation requires validating text against raw source data. factgenie emphasizes custom, visual rendering of complex source data (e.g., interactive weather charts) rather than displaying only raw text or key-value pairs.
Key Results
Because this paper introduces a software framework rather than an empirical study, it does not report quantitative performance metrics (such as F1 scores or speedups). The primary result is the open-source software release, demonstrating end-to-end support for custom span categories (e.g., incorrect fact, fact not checkable, misleading fact), data visualization, and dual-modality annotation collection.
Limitations and Future Work
The authors note several current constraints and future directions:
- Campaign Configuration: Setting up campaigns currently relies on editing configuration files; the team is actively developing an in-browser configuration UI.
- Integration Breadth: While the platform currently supports Ollama and OpenAI APIs, future releases aim to add built-in integrations for more model providers, data formats, and crowdsourcing platforms.
Summary updated