Abstract
Supervised machine learning has become the cornerstone of today's data-driven society, increasing the need for labeled data. However, the process of acquiring labels is often expensive and tedious. One possible remedy is to use active learning (AL)—a special family of machine learning algorithms designed to reduce labeling costs. Although AL has been successful in practice, a number of practical challenges hinder its effectiveness and are often overlooked in existing AL annotation tools. To address these challenges, we developed ALANNO, an open-source annotation system for NLP tasks equipped with features to make AL effective in real-world annotation projects. ALANNO facilitates annotation management in a multi-annotator setup and supports a variety of AL methods and underlying models, which are easily configurable and extensible.
Introduction
While Active Learning (AL) is a powerful theoretical framework for reducing labeling costs by selecting the most informative data points, a significant gap exists between AL research and its practical application. Most research assumes idealized conditions: a single annotator, perfectly balanced datasets, and infinite budgets. In reality, annotation projects are logistically complex, involving multiple annotators, class imbalances, and the difficult "stopping problem"—knowing when additional labeling yields diminishing returns.
ALANNO addresses these hurdles by introducing "for mortals" features designed for real-world NLP projects. It provides a robust framework for multi-annotator management, performance forecasting, and unbiased evaluation.
System Architecture and Workflow
ALANNO is designed to support the full lifecycle of an annotation project, from initial data ingestion to final model evaluation. The system manages the interaction between project managers, who configure the AL parameters, and annotators, who provide the labels.
Technical Solutions for Practical Challenges
Balanced Multi-Annotator Assignment
Coordinating multiple annotators to ensure high-quality "gold" labels is logistically difficult. ALANNO uses Quasi-Monte Carlo (QMC) sampling, specifically Sobol sequences, to distribute tasks. This mathematical approach ensures that annotator pairs and triplets appear with uniform frequency, providing a stable foundation for calculating inter-annotator agreement and reducing the variance inherent in standard random assignment.
Performance Forecasting and the Stopping Problem
Practitioners often struggle to decide when to stop labeling. ALANNO implements Bayesian performance forecasting (via the Pyro library) to estimate the future F1 score of the model. By visualizing the expected performance curve, managers can make data-driven decisions about whether to continue labeling or if the model has reached a plateau.
Handling Class Imbalance and Evaluation Bias
To combat class imbalance, ALANNO supports Guided Learning (Active Search), allowing users to use BM25 keyword searches to find and label minority class instances manually. Furthermore, to ensure trustworthy metrics, the system automatically reserves a random, representative test set before the AL process begins. This prevents the sampling bias inherent in AL from inflating performance evaluations.
Key Results and Insights
- Reduced Assignment Variance: In simulations with 1,000 instances and 10 annotators, the QMC method reduced the variance of annotator pair frequencies from 132.66 (standard random) to just 0.75.
- Accurate Forecasting: Case studies on the SST (Stanford Sentiment Treebank) and SUBJ (Subjectivity) datasets demonstrated that the Bayesian forecast closely tracks actual performance gains, successfully predicting the point of diminishing returns.
- Extensibility: The system supports a wide range of models, from traditional SVMs to modern Transformers, alongside diverse acquisition functions like Core-set and BADGE.
Conclusion
ALANNO bridges the gap between theoretical AL and practical annotation needs. By providing tools for workload balancing, performance forecasting, and unbiased evaluation, it turns AL into a manageable and predictable process for real-world NLP applications. The system is open-source under the Apache 2.0 license, offering a full-stack alternative to commercial annotation platforms.
Summary updated