SelfzCoT: a Self-Prompt Zero-shot CoT from Semantic-level to Code-level for a Better Utilization of LLMs

IokTong Lei, Zhidong DengarXiv 2023 · DOI · arXiv

Abstract

Large Language Models (LLMs) often struggle with multi-step reasoning tasks. While Zero-shot Chain of Thought (CoT) and Program of Thought (PoT) offer improvements, they face limitations in logical robustness and semantic flexibility, respectively. This paper introduces Hint of Pseudo Code (HoPC), a novel zero-shot prompting technique that bridges the gap between natural language reasoning and structured programming. By guiding models to decompose problems into sub-questions and solve them using pseudo-code, HoPC enhances logical precision without requiring external code execution. Evaluations across arithmetic and common-sense benchmarks demonstrate significant performance gains, particularly for models like Llama3-8b and GPT-3.5-Turbo.

Introduction

The reasoning capabilities of Large Language Models (LLMs) are a focal point of current AI research. Despite their power, LLMs frequently fail on complex, multi-step problems. Existing prompting strategies like Zero-shot Chain of Thought (CoT), which uses the prompt "Let’s think step by step," often lack the necessary logical structure for intricate tasks. Conversely, Program of Thought (PoT) requires the model to generate executable code, which necessitates an external interpreter and limits the model's ability to handle "semantic" reasoning—problems that involve common sense or natural language nuances that are difficult to formalize in strict code.

Comparison of prompting techniques: Direct Prompt, Zero-shot CoT, Zero-shot PoT, and the proposed HoPC.

Methodology: Hint of Pseudo Code (HoPC)

To address these limitations, the authors propose Hint of Pseudo Code (HoPC). This method utilizes a single zero-shot prompt to guide the LLM through a three-stage "hint chain":

  1. Problem Decomposition: The model breaks the primary question into smaller, sequential sub-questions, creating a structured roadmap for reasoning.
  2. Semantic Pseudo-Code Reasoning: The model generates high-level, language-agnostic pseudo-code. This step enforces logical flow and reduces ambiguity by using programming-like structures (e.g., loops, conditionals) expressed in natural language.
  3. Answer Extraction: The model interprets its own pseudo-code to derive the final answer in a specified format.

Unlike PoT, HoPC does not require an external interpreter; the LLM serves as both the "coder" and the "executor," leveraging its internal logic to follow the structured plan.

Experimental Setup

The effectiveness of HoPC was evaluated on five diverse reasoning benchmarks:

  • Arithmetic: GSM8K, AQUA-RAT, SVAMP, and ADDSUB.
  • Common-sense Reasoning: StrategyQA (requiring implicit multi-hop reasoning).

Tests were conducted using several state-of-the-art models, including GPT-3.5-Turbo, Qwen2.5-7b, and Llama3-8b.

Results and Analysis

HoPC consistently outperformed standard Zero-shot CoT across all benchmarks. The improvements were particularly pronounced for the Llama3-8b model, where accuracy on the AQUA algebra dataset jumped from 37.4% to 74.4%. On StrategyQA, HoPC enabled GPT-3.5-Turbo to achieve 82.9% accuracy, compared to 59.2% with standard CoT.

A comparative example showing how Zero-shot CoT, PoT, and HoPC handle a complex word problem. HoPC correctly identifies the logic where CoT fails.

The "pseudo-code" approach acts as a crucial middle ground. It provides the structural rigor needed to prevent the model from deviating from the logical path—a common failure mode in CoT—while maintaining the flexibility to handle non-mathematical, semantic reasoning that PoT cannot easily accommodate. Furthermore, the decomposition into sub-questions enhances interpretability, allowing researchers to pinpoint exactly where a reasoning chain might break down.

Conclusion and Limitations

HoPC represents a significant step forward in zero-shot prompting by combining the strengths of natural language and structured logic. While its performance remains dependent on the base model's capabilities, it offers a robust, interpreter-free method for boosting LLM reasoning. Future work will explore more adaptive hint chains and the scalability of this approach to even larger model architectures. The authors also note that while pseudo-code improves transparency, it does not entirely solve the "black box" nature of LLM decision-making.

Summary updated

BibTeX
@misc{lei2023selfzcot,
  author = {IokTong Lei and Zhidong Deng},
  title = {{SelfzCoT: a Self-Prompt Zero-shot CoT from Semantic-level to Code-level for a Better Utilization of LLMs}},
  year = {2023},
  eprint = {2305.11461},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.11461},
}
Figure