Backward Problem Synthesis for Deep Learning

38 concepts30 papers1 proposal

Many of the most critical problems in science, mathematics, and engineering exhibit a profound computational asymmetry: finding a solution is incredibly difficult, but verifying a solution—or generating a problem from a solution—is computationally trivial.

Consider symbolic integration. If you are handed a complex mathematical function and asked to find its integral, you face a challenging search problem that requires deep heuristic knowledge. However, if you start with a target solution and differentiate it, the process is purely algorithmic, deterministic, and cheap.

When training deep learning models to solve these hard tasks, researchers traditionally face a severe data bottleneck. Generating a dataset by sampling random problems and solving them using classical algorithms is often computationally intractable, especially for NP-hard or doubly exponential problems. Backward problem synthesis flips this workflow entirely. By starting with a known solution or target state and working backward to construct a valid problem instance, researchers can generate virtually unlimited, perfectly labeled datasets to train neural networks on otherwise intractable tasks.

The Core Mechanism: Exploiting Mathematical Asymmetry

To build intuition for why backward synthesis works, let us look at the domain of symbolic mathematics. In their foundational work, Guillaume Lample et al. demonstrated this principle in Deep Learning For Symbolic Mathematics. Instead of trying to solve randomly generated integration problems to create training data, the authors generated random functions $F(x)$, computed their derivatives $f(x)$ via standard computer algebra rules, and used the resulting pair $(f(x), F(x))$ as a training instance. The model learns to map the problem back to the solution, bypassing the need to ever run a traditional integration solver during data generation.

Traditional Workflow (Forward Data Generation):
Random Problem ───[ Highly Expensive Solver ]───> Solution (Bottlenecked)

Backward Synthesis Workflow:
Random Solution ───[ Cheap Invertible Operation ]───> Valid Problem (Scalable)

This concept of building problem instances from known mathematical structures appears across various symbolic domains. For instance, when training models to perform symbolic regression on recurrent sequences, Stéphane d'Ascoli et al. in Deep symbolic regression for recurrence prediction constructed random mathematical expression trees first, then sampled initial conditions to generate corresponding numerical sequences. By starting with the formula, they guaranteed that every generated sequence possessed a well-defined symbolic ground truth.

The power of backward synthesis becomes even more apparent when dealing with problems of extreme algorithmic complexity, such as computing Gröbner bases in algebraic geometry, which carries a notorious doubly exponential time complexity.

To train Transformers to handle these systems, Hiroshi Kera et al. developed a backward generation strategy in Learning to compute Gröbner bases and Gröbner basis computation via learning. Instead of choosing random polynomial equations and attempting to compute their bases, the authors started with a known target basis or canonical form and applied a sequence of random, invertible algebraic transformations to obfuscate it into a highly complex input system. This approach, also utilized by Hiroshi Kera et al. in Computational Algebra with Attention: Transformer Oracles for Border Basis Algorithms and Mohamed Malhou et al. in HATSolver: Learning Gröbner Bases with Hierarchical Attention Transformers, ensures that the generated problems are mathematically guaranteed to be solvable and structurally meaningful, completely avoiding the exponential cost of forward computation during dataset preparation.

A parallel challenge exists in control theory: finding global Lyapunov functions to prove the stability of dynamical systems. In Global Lyapunov functions: a long-standing open problem in mathematics, with symbolic transformers, Alberto Alfarano et al. inverted the problem by sampling a valid candidate Lyapunov function first, and then mathematically deriving a corresponding stable dynamical system. This allowed them to train symbolic transformers on massive synthetic datasets, enabling the models to discover stability proofs for complex systems that defy traditional algorithmic solvers. Similarly, Nico Pelleriti et al. in Neural Sum-of-Squares: Certifying the Nonnegativity of Polynomials with Transformers used reverse sampling from compact solutions to construct complex polynomial instances with ground-truth sparse structures for non-negativity certification.

Bypassing Expert Scarcity in NP-Hard Optimization

In combinatorial optimization and engineering design, expert data is often proprietary or incredibly scarce. Backward synthesis circumvents this by generating optimal solutions directly and calculating the problem parameters that satisfy them.

In semiconductor design, chip placement is an NP-hard problem traditionally solved by slow heuristics. Vint Lee et al. tackled this in Chip Placement with Diffusion Models by reversing the design pipeline. They first generated random, valid spatial layouts of components on a canvas and then synthesized the logical netlists that would naturally be optimized by those specific placements. This allowed them to pre-train diffusion models on a massive corpus of synthetic training pairs where the optimal ground truth was known by construction.

Traditional Chip Design:
Logical Netlist ───[ Slow Optimization Heuristics ]───> Optimal Placement

Backward Synthesis:
Random Valid Placement ───[ Procedural Synthesis ]───> Corresponding Netlist

This strategy of "inverse demonstration" is highly effective across other optimization domains:

Forward Simulation as a Data Engine for Physical Systems

When dealing with continuous physical systems, directly computing an inverse mapping—such as finding the exact inputs required to achieve a specific target state—is often mathematically intractable due to non-linearities. However, the forward mapping (simulating the system given an input) is typically straightforward and highly parallelizable. Backward problem synthesis exploits this by sampling inputs, running the cheap forward simulation, and treating the resulting output-input pairs as training data for an inverse model.

For instance, in real-time neural engineering, Yuki Minai et al. developed OMiSO: Adaptive optimization of state-dependent brain stimulation to shape neural population states. To train an inverse model that maps a desired brain state back to the optimal electrical stimulation parameters, the authors generated a large synthetic dataset by passing random stimulation parameters through a pre-trained forward predictive model. This bypassed the mathematical complexity of directly inverting non-linear neural networks in real time.

Similarly, Sören Arlt et al. in Meta-Designing Quantum Experiments with Language Models sampled random procedural descriptions of quantum setups and simulated their physical outcomes, creating a dataset that allows language models to map desired quantum states back to generating code.

In the domain of partial differential equations (PDEs), Rafael Bischof leveraged the Method of Manufactured Solutions in HyPINO: Multi-Physics Neural Operators via HyperPINNs and the Method of Manufactured Solutions. By defining analytical solutions first and deriving the corresponding PDEs that satisfy them, the author built large-scale supervised datasets without the massive computational burden of traditional numerical PDE solvers. This paradigm of using forward simulation to accelerate inverse problem data generation is also seen in kinetic theory simulations by Mohsen Sadr et al. in Gaussian Process Regression for Maximum Entropy Distribution and in electrode array analysis by Xinying Wang et al. in HDK: Toward High-Performance Deep-Learning-Based Kirchhoff Analysis.

Extending to Textual and Logical Constraints

The core intuition of backward synthesis is not restricted to numerical or symbolic equations; it applies equally well to textual reasoning. When prompting large language models (LLMs) to generate mathematical word problems, a common failure mode is that the generated questions are logically inconsistent or unsolvable.

To solve this, Zimu Lu et al. introduced the "question back-translation" pipeline in MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs. Instead of asking an LLM to write a question and then solve it, the pipeline trains a model to perform the inverse task: taking a complex, logically sound mathematical solution and generating a coherent question that leads to it.

Forward Text Generation (Error-Prone):
Write a Question ───> Attempt to Solve ───> Logical Errors / Unsolvable

Backward Text Generation (MathGenie):
Rigorous Solution Path ───[ Back-Translation ]───> Coherent, Solvable Question

Because the solution path already contains the necessary logical flow and constraints, the synthesized questions are guaranteed to be solvable and high-quality, providing a highly effective data engine for scaling up mathematical reasoning capabilities.

Conclusion

Backward problem synthesis shifts the bottleneck of machine learning data generation from computation to creative inversion. By identifying domains where the solution-to-problem mapping is cheaper than the problem-to-solution mapping, researchers can bypass NP-hard complexities, proprietary data scarcity, and simulation bottlenecks. Whether dealing with algebraic geometry, chip layout design, or textual reasoning, this paradigm provides a robust framework for training deep learning models to navigate highly complex, structured environments.

Key Foundational Work

Idiap Abstract Text Summarization System for German Text Summarization Task (2019) — contributed 1 concept elements to this direction.
Simulated Data for Linear Regression with Structured and Sparse Penalties: Introducing pylearn-simulate (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_backward_problem_synthesis_22393159,
  title = {backward problem synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/backward-problem-synthesis/rdr_22393159adfc4a3b8e180650d38b4cc8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_backward_problem_synthesis_22393159_v1,
  title = {backward problem synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/backward-problem-synthesis/rdr_22393159adfc4a3b8e180650d38b4cc8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake