Robustness Assessment of Mathematical Reasoning in the Presence of Missing and Contradictory Conditions

Shi-Yu Tian, Zhi Zhou, Lin-Han Jia, Lan-Zhe Guo, Yu-Feng Li

Abstract

Large language models (LLMs) have demonstrated impressive performance on reasoning tasks, which can be further improved through few-shot prompting techniques. However, the current evaluation primarily focuses on carefully constructed benchmarks and neglects the consideration of real-world reasoning problems that present missing and contradictory conditions, known as ill-defined problems. Our observations suggest that existing few-shot prompting techniques are ineffective in such scenarios, often providing overconfident answers or hallucination. To further study this problem, we develop a benchmark called Problems with Missing and Contradictory conditions (PMC) and introduce two novel metrics to evaluate the performance of few-shot prompting methods in these scenarios. Our analysis using the PMC benchmark reveals a trade-off dilemma between the performance of mathematical reasoning for well-defined problems and the ability to recognize ill-defined problems. To address the challenges posed by PMC, we propose a novel few-shot prompting method called SMT-LIB Prompting (SLP), which utilizes the SMT-LIB language to model the problems instead of solving them directly. Subsequently, a double-check solving strategy checks the satisfiability and uniqueness of the solution and provides final feedback. Extensive experiments demonstrate the superiority of our SLP approach compared to existing few-shot prompting methods when dealing with problems with missing and contradictory conditions. We will open-source our benchmark and code to facilitate future research.

Introduction

While current Large Language Models (LLMs) exhibit remarkable mathematical capabilities, they often suffer from a significant "people-pleasing" bias. When presented with unsolvable or ill-defined problems, models frequently hallucinate answers rather than identifying the underlying issues. This behavior is particularly problematic in real-world educational or technical settings where user input may be incomplete or contradictory.

Examples of well-defined and ill-defined mathematical problems and typical LLM failure modes.

To address this, we introduce the PMC (Problems with Missing and Contradictory conditions) benchmark, consisting of over 5,000 ill-defined math problems. Our research identifies a critical trade-off: prompting a model to be skeptical often degrades its performance on well-defined, solvable problems. We aim to bridge this gap, enabling models to function as both high-performing solvers and reliable "BS detectors."

Methodology: VCSEARCH

We propose VCSEARCH (Variable-Constraint Search), a training-free framework that integrates the linguistic flexibility of LLMs with the logical rigor of Satisfiability Modulo Theories (SMT) solvers. Unlike traditional methods that attempt to solve problems directly, VCSEARCH models them formally.

The VCSEARCH framework workflow, illustrating the iterative variable-constraint search and verification process.

The framework operates in two main phases:

  1. Anchored Initialization: The LLM generates an initial formal draft in SMT-Lib language. A symbolic solver (Z3) performs an initial check for satisfiability.
  2. Variable-Constraint Dynamic Search: The system iteratively refines the problem model. It selects a "head variable," explores related constraints using the LLM's knowledge, and employs an "LLM-Judge" to verify if the refined constraints improve the model's accuracy.

By utilizing the Z3 solver as a "truth engine," the system can definitively categorize problems as UNSAT (contradictory), Multi (missing information), or Ans (well-defined).

The PMC Benchmark

The PMC dataset was constructed by mutating "seed" problems from standard benchmarks like GSM8k and SVAMP. We employed an LLM-based mutation strategy to create:

  • Missing-type: Essential information is removed.
  • Contra-type: Conflicting information is injected.

The resulting 5,000+ problems were cross-verified by multiple LLMs and human annotators to ensure they are truly unsolvable.

Results and Discussion

Our experiments demonstrate that VCSEARCH significantly outperforms existing methods:

  • Enhanced Rejection: Accuracy in identifying unsolvable problems improved by at least 12% across models like Qwen2.5 and DeepSeek.
  • Improved Modeling: On well-defined problems, VCSEARCH outperformed traditional formal prompting (SatLM) by an average of 14.95%, indicating that iterative search leads to better mathematical understanding.
  • Scalability: VCSEARCH maintained high reliability even on smaller models (1.5B to 3B parameters), allowing them to rival the rejection capabilities of much larger counterparts.

Key Insights

  • The Rejection Trade-off: Standard Chain-of-Thought prompting forces a binary choice between solving and rejecting. VCSEARCH mitigates this by using formal solvers as a "safety rail."
  • Complexity of Contradictions: "Contra-type" problems remain significantly harder for models to detect than "Missing-type" problems.

Conclusion and Future Work

While VCSEARCH provides a robust hallucination filter, it currently faces a computational trade-off, requiring approximately 79.8 seconds per problem. Future work will focus on optimizing the search process through parallelization and pruning strategies. Additionally, we aim to extend the framework's scope beyond arithmetic to more complex mathematical domains like geometry.

Summary updated

Abstract

Large language models (LLMs) have demonstrated impressive performance on reasoning tasks, which can be further improved through few-shot prompting techniques. However, the current evaluation primarily focuses on carefully constructed benchmarks and neglects the consideration of real-world reasoning problems that present missing and contradictory conditions, known as ill-defined problems. Our observations suggest that existing few-shot prompting techniques are ineffective in such scenarios, often providing overconfident answers or hallucination. To further study this problem, we develop a benchmark called Problems with Missing and Contradictory conditions (PMC) and introduce two novel metrics to evaluate the performance of few-shot prompting methods in these scenarios. Our analysis using the PMC benchmark reveals a trade-off dilemma between the performance of mathematical reasoning for well-defined problems and the ability to recognize ill-defined problems. To address the challenges posed by PMC, we propose a novel few-shot prompting method called SMT-LIB Prompting (SLP), which utilizes the SMT-LIB language to model the problems instead of solving them directly. Subsequently, a double-check solving strategy checks the satisfiability and uniqueness of the solution and provides final feedback. Extensive experiments demonstrate the superiority of our SLP approach compared to existing few-shot prompting methods when dealing with problems with missing and contradictory conditions. We will open-source our benchmark and code to facilitate future research.
BibTeX
@inproceedings{tian2025robustness,
  author = {Shi-Yu Tian and Zhi Zhou and Lin-Han Jia and Lan-Zhe Guo and Yu-Feng Li},
  title = {{Robustness Assessment of Mathematical Reasoning in the Presence of Missing and Contradictory Conditions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=z8PyS02Twn},
}
Figure