Neural networks are built on the calculus of small changes. To train a model, we calculate a gradient—a direction that tells us how to slightly nudge a parameter to reduce error. This works beautifully for continuous data like image pixels, where a value of 0.5 can be nudged to 0.501.
However, modern AI is increasingly built on discrete tokens: words in a sentence, categories in a table, or patches in a Vision Transformer. As Tom Roth and colleagues note in their survey on Adversarial Attacks, there is no "intermediate state" between the word cat and the word dog. This discreteness creates a "non-differentiable bottleneck" that prevents us from using standard gradient-based tools to optimize prompts, find adversarial vulnerabilities, or prune model architectures.
A growing body of research aims to bridge this gap, creating "differentiable bridges" that allow continuous gradients to navigate discrete landscapes.
The Embedded World as a Continuous Proxy
The most common strategy for bypassing the discreteness bottleneck is to move the optimization problem from the "hard" token space into the "soft" embedding space.
In Bridge the Gap Between CV and NLP!, Lifan Yuan and Wei Wei introduce T-PGD, a framework that treats text like an image. Instead of trying to swap words directly, they perturb the continuous word embeddings. To ensure these perturbations remain linguistically valid, they use a Masked Language Model (MLM) to "decode" the perturbed vectors back into the nearest valid tokens.
Similarly, Alexey Zaytsev and his team explore Traversing an Embedded World to find adversarial sequences. By treating the discrete vocabulary as a set of points in a high-dimensional continuous space, researchers can use first-order optimization to find "paths" between categories that would otherwise be invisible to the model's loss function.
Softening the Discrete Choice
While embedding-space optimization works for attacks, it often fails when the model itself needs to make a discrete decision during training—such as deciding which tokens to merge for efficiency.
Dong Hoon Lee and Seunghoon Hong address this in Learning to Merge Tokens via Decoupled Embedding. Traditionally, token merging is a discrete "hard" operation. Their method, DTEM, uses a lightweight module to predict merging policies and employs a "soft merging" process during training. This allows the merging policy to be optimized via gradients, enabling the model to learn which information is redundant rather than relying on fixed heuristics.
This "softening" approach is also gaining traction in generative modeling. Yuanzhi Zhu proposes Soft-Di[M]O, which replaces discrete image tokens with soft embeddings. This makes the entire generation pipeline differentiable, allowing fast, one-step generators to be refined using reward-based feedback or adversarial training—techniques that were previously reserved for continuous models.
Smoothing the Jagged Landscape
Even with continuous proxies, the optimization landscape of a discrete model is often "jagged." A small step in embedding space might lead to a token that is semantically unrelated, causing the gradient's prediction to fail. Qiongxiu Li and Xiaolin Hu point out in Faster-GCG that first-order Taylor approximations (the basis of gradients) often break down when token embeddings are sparsely distributed.
To combat this, researchers are turning to Homotopy and Relaxation:
- Functional Homotopy: Divyam Anshumaan and Yudong Chen developed Functional Homotopy (FH), which starts by optimizing a "misaligned" (and thus smoother) version of the model and gradually "hardens" the problem back to the original discrete state.
- Convex Relaxation: In DISCO, Anish Acharya and Sujay Sanghavi transform the "needle-in-a-haystack" discrete decoding problem into a continuous optimization task using convex relaxation, providing theoretical guarantees for convergence that greedy search methods lack.
System-Level Differentiability: Prompts as Weights
The ultimate goal of this research direction is to treat entire pipelines of Large Language Models (LLMs) as differentiable neural networks.
In Deep Language Networks (DLNs), Alessandro Sordoni and his team treat prompts as the "weights" of a network. Because prompts are discrete text, they use variational inference to "backpropagate" through the language, allowing them to jointly optimize instructions across multiple stacked LLMs.
This mirrors the work of Christopher Potts and Omar Khattab on BetterTogether, which alternates between optimizing discrete prompt templates and fine-tuning model weights. By treating the discrete prompt and the continuous weights as two parts of a single optimization problem, they allow models to "teach themselves" more effectively than either method could alone.
By finding clever ways to "soften" the hard edges of tokens, these researchers are turning the discrete nature of language from a barrier into a steerable, optimizable feature of modern AI.