Abstract
LLMEmbed introduces a paradigm shift in text classification, moving from slow, generative "prompt-based reasoning" to efficient "embedding-based transfer learning." By utilizing lightweight Large Language Models (LLMs) like LLaMA2-7B as frozen feature extractors rather than reasoning engines, the framework achieves state-of-the-art accuracy while requiring only 1.5% of the runtime and 1.8% of the electricity compared to sophisticated prompting methods.
Introduction
The prevailing trend in Natural Language Processing (NLP) is to treat classification tasks as generation problems. Researchers often employ "Chain-of-Thought" (CoT) or "Clue and Reasoning Prompting" (CARP) to guide LLMs through multi-step reasoning before outputting a label. However, this approach faces significant hurdles:
- Efficiency Bottlenecks: Serial reasoning is computationally expensive, leading to high latency and energy costs.
- Model Scale Dependency: Lightweight open-source models (e.g., 7B parameters) often struggle with hallucinations or fail to follow complex prompts compared to massive models like GPT-4.
- Prompt Engineering Overhead: High performance often depends on brittle, manually tuned prompts to align inputs with expected outputs.
LLMEmbed addresses these by asking whether the "knowledge" within a lightweight LLM can be harnessed more effectively without forcing it to act as a human-like reasoner.
Methodology
LLMEmbed treats the LLM as a sophisticated encoder within a transfer learning framework. Key technical components include:
- Multi-Depth Feature Extraction: Rather than relying solely on the final hidden state, LLMEmbed extracts and averages token embeddings from multiple transformer blocks (specifically the last five layers of LLaMA2-7B) to capture a broader semantic spectrum.
- Cross-Model Fusion: The framework concatenates embeddings from generative models (LLaMA2) with traditional discriminative models (BERT or RoBERTa).
- Co-occurrence Pooling: A power-normalized co-occurrence pooling strategy aligns different embedding spaces into a unified representation, capturing high-order relationships between different embedding vectors.
- Frozen Backbone: The LLM remains frozen, and only a lightweight classifier head is trained. This prevents "catastrophic forgetting" and ensures high-speed processing through parallelism.
Evaluation and Results
The authors evaluated LLMEmbed on five benchmarks: SST-2 and MR (sentiment analysis), AGNews (topic classification), and R8/R52 (news categorization).
Performance and Efficiency
LLMEmbed achieved an average accuracy of 96.20% on the LLaMA2-7B backbone, significantly outperforming the prompt-based CARP method (81.14%). Remarkably, this performance is comparable to GPT-3 (175B) using sophisticated prompting, despite LLMEmbed using only 4% of the parameters.
Beyond accuracy, the efficiency gains are transformative. Compared to prompt-based methods on the same backbone, LLMEmbed required just 1.5% of the runtime and 1.8% of the electricity. For a typical workload, the energy cost for LLMEmbed was approximately $0.025, compared to nearly $200 in API costs for a GPT-3 prompt-based approach.
Discussion
The results suggest that for lightweight models, the latent semantic space is more robust than the ability to generate coherent reasoning steps. The "genuine function" of a localized LLM in classification lies in its role as a high-dimensional feature extractor. Furthermore, the synergy between generative (LLaMA2) and discriminative (RoBERTa) models indicates they capture complementary semantic features.
Limitations and Future Work
While efficient, LLMEmbed currently faces challenges with very long-text classification without fine-tuning the backbone. Additionally, moving away from reasoning prompts reduces "human-readable" explainability. Future work aims to bridge this gap by combining the efficiency of embeddings with the interpretability of reasoning prompts.
Summary updated