Introduction
Few-shot text classification—the task of training a model on a limited number of examples per category—is a pervasive challenge in real-world Natural Language Processing (NLP). This difficulty is amplified when models must distinguish between a "multitude of classes" (e.g., dozens or hundreds of categories) that are semantically similar, such as specific customer service intents.
Existing approaches generally fall into two categories: prompting Large Language Models (LLMs) or fine-tuning smaller encoder models. LLMs often struggle with the long context required to define many classes and suffer from high latency and cost. Conversely, current fine-tuning tools like SetFit, while effective, can be computationally slow and may fail to capture the fine-grained differences necessary to separate highly similar classes. FastFit is introduced as a high-performance, "pip-installable" alternative designed to bridge this gap by optimizing for speed, accuracy, and ease of use.
Methodology
FastFit departs from traditional classification heads by treating the problem as a similarity task within a shared embedding space. Its architecture incorporates three primary innovations:
- Batch Contrastive Learning: Unlike the triplet loss used in methods like SetFit, FastFit employs a batch contrastive loss. This approach is significantly more efficient as every example within a training batch serves as a negative sample for others, allowing the model to learn class boundaries much faster.
- Token-Level Similarity: Rather than representing a sentence as a single vector (e.g., via the [CLS] token), FastFit utilizes a strategy inspired by ColBERT. It calculates similarity based on the sum of maximum cosine similarities between individual tokens. This granular approach allows the model to detect subtle linguistic nuances that a single sentence-level embedding might miss.
- Smart Augmentation: The training pipeline automatically incorporates class names as training examples and utilizes dropout-based repetitions to enhance model robustness.
Experimental Setup
The researchers evaluated FastFit against several challenging benchmarks characterized by many overlapping categories:
- English Benchmarks: CLINC150 (150 classes), BANKING77 (77 classes), and HWU64 (64 classes).
- Multilingual Benchmarks: The Amazon MASSIVE dataset, covering 60 classes across six languages (English, German, Japanese, Spanish, French, and Chinese).
Key Results
Efficiency and Speed
One of FastFit's primary advantages is its training efficiency. Training typically completes within 35 to 70 seconds, representing a 3-20x speedup over SetFit.
Classification Accuracy
FastFit consistently outperforms both smaller fine-tuned baselines and massive LLMs. On average, FastFit-Large outperformed SetFit by 2.1% and standard classifiers by 3.4%. Notably, specialized models like FastFit-Large (355M parameters) can significantly outperform models as large as Llama-2-70B in these specific tasks, as LLMs often struggle to manage the extensive context required for 150 different class definitions.
Discussion and Insights
The success of FastFit highlights two critical insights for NLP practitioners. First, negatives are free: by utilizing batch contrastive learning, computational bottlenecks are transformed into advantages, where every batch example helps refine the boundaries of every other class. Second, granularity wins: the transition to token-level similarity provides the necessary resolution to distinguish between similar intents (e.g., "cancel my card" vs. "lost my card") that might otherwise appear identical in a sentence-level embedding.
Limitations and Future Work
While FastFit is highly effective, the authors note that token-level similarity introduces some computational overhead compared to simple vector comparisons. Future work may explore domain-specific pre-training to further boost performance or the integration of approximate nearest neighbor (ANN) search to maintain efficiency when scaling to thousands of classes.
Summary updated