Compress to Think, Decompress to Speak: Dual-Mode Reasoning in Transformers

Angeliki Giannou, Liu Yang, Kangwook Lee, Robert D Nowak, Dimitris PapailiopoulosOpenReview 2026 · Submission to ICLR 2026 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Latent reasoning has emerged as an alternative to reasoning with natural language and involves feeding back the last layer's hidden state representation (soft token) to the input of the transformer. This idea is promising, since soft tokens have increased expressive capacity compared to tokens from the vocabulary (hard tokens). Existing works on training transformers with soft tokens often suffer from performance loss and do not allow for sampling of different reasoning traces. We propose a training paradigm for transformers that uses soft tokens, in which the model learns to operate in two modes; one that processes the soft tokens (latent thinking mode) and one that decompresses the soft tokens into few reasoning steps with hard tokens from the vocabulary (local decoding mode). We focus on logical and math reasoning tasks, and fine-tune pretrained models of different size. Our method achieves similar or better performance, compared to supervised fine-tuning with chain-of-thought data across all tasks; while it requires reduced KV cache and allows sampling different reasoning traces at inference.
BibTeX
@misc{giannou2026compress,
  author = {Angeliki Giannou and Liu Yang and Kangwook Lee and Robert D Nowak and Dimitris Papailiopoulos},
  title = {{Compress to Think, Decompress to Speak: Dual-Mode Reasoning in Transformers}},
  year = {2026},
  url = {https://openreview.net/forum?id=c9FF7JR8BM},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2026},
}

Loading related content...

Figure