Adapting Classification Techniques to Open Ended Generation

713 concepts615 papers1 proposal

When we study how Large Language Models (LLMs) learn in-context or adapt to new tasks, we usually turn to classification benchmarks, such as sentiment analysis or multiple-choice question answering. Classification provides a clean, bounded sandbox: the output space is finite, labels are discrete, and success is binary.

However, the true power of LLMs lies in open-ended generation—tasks like abstractive summarization, long-form reasoning, dialogue, and code synthesis. A major challenge in the field is that the prompting, tuning, and mechanistic frameworks built for discrete classification tasks do not seamlessly generalize to the high-dimensional, unconstrained space of open-ended generation.

The Geometry of the Output Space

Why do classification-optimized techniques struggle when the output space opens up? The answer lies in how models organize information internally.

In a classification task, the model's job is to map an input to a tightly clustered region of the embedding space corresponding to a specific label. Andrey Kirsanov et al. explore these internal mechanics in The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Models, showing how few-shot examples reorganize internal representations. Similarly, Haolin Yang et al. demonstrate in Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning that in-context learning often operates as a two-stage geometric process, separating hidden states by label and aligning them with output vectors.

Classification Space:
[Input] ---> (Manifold Compression) ---> [Discrete Label Region (A/B/C)]

Generative Space:
[Input] ---> (Continuous Trajectory) ---> [Token 1] ---> [Token 2] ---> [Token N] (Unconstrained)

But what happens when there are no discrete labels to align with? In open-ended generation, the model must navigate a continuous, high-dimensional trajectory where each token depends on the history of all previously generated tokens. The clean phase transitions and manifold compressions observed in multiple-choice settings may dissolve when the model is unconstrained by fixed output categories, as noted by Diego Doimo in The Representation Landscape of Few-Shot Learning and Fine-Tuning in Large Language Models.

The Breakdown of Selection and Compression Heuristics

Because generative spaces lack discrete boundaries, popular methods for optimizing prompts and demonstrations face significant friction.

Consider the task of selecting the best "support examples" for in-context learning (ICL). In classification, we can evaluate an example's utility based on how well it helps draw a decision boundary. In Finding Support Examples for In-Context Learning, Xiaonan Li et al. introduce frameworks to optimize these demonstrations, yet the authors note that the mechanisms defining a good support example may differ fundamentally when the output space is open-ended rather than categorical.

Similarly, methods that compress demonstrations into compact activation vectors to save compute—such as the framework proposed by Zhuowei Li in Implicit In-context Learning—have primarily proven effective for classification. Compressing a rich, multi-step generative prompt into a single vector risks discarding the high-dimensional structural capacity required for complex sequence generation.

The Failure of Discrete Feedback and Steering Loops

Another major area of divergence is how we guide and correct models at inference time. Many state-of-the-art frameworks rely on feedback loops or voting mechanisms. For example, Hongling Xu et al. show in Improving In-Context Learning with Prediction Feedback for Sentiment Analysis that showing a model its own classification errors drastically improves performance.

But in a generative task like summarization, "error correction" is rarely binary. An output is not simply right or wrong; it can be factually accurate but stylistically poor, or coherent but slightly off-topic. Evaluating the quality of open-ended responses is significantly more complex, making simple iterative filtering or discrete feedback loops difficult to apply.

This limitation extends to internal interventions like steering vectors. While steering vectors can easily nudge a model toward a specific multiple-choice answer, applying them to free-form generation introduces a steep "quality cliff." As Joschka Braun et al. discover in Beyond Multiple Choice: Evaluating Steering Vectors for Adaptive Free-Form Summarization, high-intensity internal steering frequently degrades the structural coherence and topical focus of free-form text, requiring a delicate balance with traditional prompting to remain effective.

Rethinking Generalization Across Task Types

To bridge the gap between classification and generation, researchers are beginning to explore how high-level inductive biases transfer across task types. The core challenge is moving away from methods that rely on discrete label boundaries and toward frameworks that accommodate high-dimensional, structured outputs.

Whether designing parameter-efficient fine-tuning recipes, internal knowledge consolidation mechanisms, or automated prompt optimization strategies, validating these techniques solely on classification benchmarks creates a false sense of security. True generative task generalization requires architectures and prompting paradigms designed from the ground up to handle the unconstrained, continuous nature of human language.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generative_task_generalization_52eaa607,
  title = {generative task generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-task-generalization/rdr_52eaa60723174469a00bd1ebd5c28e5c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_generative_task_generalization_52eaa607_v1,
  title = {generative task generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-task-generalization/rdr_52eaa60723174469a00bd1ebd5c28e5c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake