Support Vector Generation: Kernelizing Large Language Models for Efficient Zero‑Shot NLP

Shohei OhsawaNeurIPS 2025 · OpenReview

Modern Pre-trained Language Models (PLMs) are incredibly powerful at zero-shot tasks, but their deployment is often hindered by two major blockers: opacity and computational cost. With billions of parameters, understanding why a model made a specific prediction is nearly impossible, and running these autoregressive models typically requires expensive, high-end GPUs.

To address this, researchers have introduced Support Vector Generation (SVG), a novel framework that transforms frozen, "black-box" Large Language Models into efficient, interpretable classifiers without requiring any fine-tuning or GPU acceleration.

Comparison of learning paradigms, highlighting the SVG approach where a Transformer acts as a teacher to a Kernel Machine.

The SVG Methodology: A Teacher-Student Approach

The SVG algorithm operates as a two-stage process that treats a massive LLM as a "teacher" for a non-parametric, lightweight "student" kernel machine.

  1. The Language Kernel: The framework defines a positive-definite kernel based on the negative log-likelihood of the frozen PLM. This embeds sentences into a Reproducing Kernel Hilbert Space (RKHS) where the dimensionality is completely independent of the LLM's massive parameter count.
  2. Metropolis-Hastings Sampling: Instead of training on a fixed dataset, SVG uses a Metropolis-Hastings (MCMC) sampler to query the PLM. It probes the model to synthesize candidate sentences that are likely to lie near the decision boundary—the areas where the model is most uncertain.
  3. SVM Optimization: SVG then solves a dual Support Vector Machine (SVM) problem restricted to these generated candidate sentences. It identifies a small set of "support vectors" (the most informative sentences) and assigns them weights.

At test time, the classifier bypasses the need to run a full forward pass of the massive LLM. Instead, it performs a weighted vote over a tiny set (at most 32) of these natural-language support vectors.

Key Insights: Natural Language as Support Vectors

A major theoretical insight of SVG is that its generalization bound depends on the number of support vectors, not the size of the LLM. This decouples model size from mathematical complexity, allowing you to use a trillion-parameter model to build a robust, mathematically "simple" classifier.

Unlike traditional SVMs where support vectors are abstract points in a high-dimensional space, SVG's support vectors are actual, readable sentences (e.g., "This movie gave me such a warm feeling"). This provides a faithful rationale for every decision. The framework essentially reinterprets machine learning as a communication game: the LLM communicates its internal knowledge to a kernel machine using natural language prompts, bypassing gradient-based fine-tuning entirely.

Visualization of the SVG decision boundary and the corresponding natural language support vectors with their learned weights for the SST-2 task.

Performance and Efficiency

Evaluated on the GLUE benchmark (including SST-2, CoLA, QQP, MRPC, RTE, QNLI, and MNLI) using GPT-3 family models as teachers, SVG demonstrated remarkable results:

  • High Accuracy: SVG matched or outperformed standard prompting-based zero-shot baselines. On the SST-2 sentiment task, SVG achieved 91.7% accuracy, significantly higher than the 83.6% achieved by standard prompting.
  • Extreme Efficiency: The entire training phase (sampling + SVM optimization) completes in under three minutes per task on a single CPU (3 GHz, 1 GB RAM). No GPUs are required for training the student model.
  • Clear Interpretability: Decisions are condensed into a weighted combination of roughly 32 sentences. The dual coefficients provide exact influence scores, showing precisely which "support sentences" drove a specific classification.

Limitations and Future Directions

While highly effective for zero- and few-shot settings, SVG has limitations. Its success relies heavily on the quality of the sentences generated by the PLM; if the teacher model is biased or produces low-quality text, the student kernel machine will inherit those flaws. Additionally, kernel methods have a computational complexity of $O(n^2)$ relative to the number of samples, meaning they may lag behind standard Transformers ($O(nm^2)$) as training data scales up significantly.

Future work may explore more advanced sampling strategies, such as active learning, and utilize approximations like Random Fourier Features to help the kernel approach scale efficiently to "many-shot" scenarios.

Summary updated

Abstract

We introduce Support Vector Generation (SVG), a kernel-based framework that converts a frozen language model into an interpretable, training-free classifier for zero- and few-shot learning. SVG operates by combining Metropolis–Hastings sampling with support vector machine optimization in the reproducing kernel Hilbert space (RKHS) induced by the language model's embedding. Each classification decision is based on a weighted combination of at most 32 natural-language sentences, which serve as explicit support vectors and provide faithful rationales. Our theoretical analysis proves that SVG minimizes the empirical hinge loss over the span of the supports and admits a generalization bound independent of the language model size. Experiments on the GLUE benchmark show that SVG matches or surpasses prompting-based zero-shot baselines in accuracy across multiple tasks—without any fine-tuning or GPU acceleration. Notably, our CPU-only implementation completes training in under three minutes per task, and maintains competitive inference speed. These results suggest that SVG offers a viable path toward efficient, interpretable NLP systems under compute constraints.
BibTeX
@inproceedings{ohsawa2025support,
  author = {Shohei Ohsawa},
  title = {{Support Vector Generation: Kernelizing Large Language Models for Efficient Zero‑Shot NLP}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=upU88pUpzX},
}
Figure