TL;DR
This paper introduces Marco-01, a large reasoning model designed to tackle open-ended problems beyond standard math or coding tasks. By combining Chain-of-Thought fine-tuning with Monte Carlo Tree Search and novel reasoning strategies, Marco-01 aims to improve generalization to domains lacking clear evaluation metrics.
What Problem Does This Paper Address?
The field of large reasoning models (LRMs) has seen significant progress, particularly with models excelling in domains with well-defined answers like mathematics and coding. However, a key challenge remains: how to effectively equip these models to handle "open-ended" problems where solutions are not easily quantifiable and clear reward signals are absent. This paper addresses this gap by proposing Marco-01, which aims to generalize LRM capabilities to these more complex, real-world scenarios, inspired by OpenAI's o1 model.
Methods
Marco-01 is built upon several key techniques:
- Chain-of-Thought (CoT) Fine-Tuning: The model is fine-tuned using CoT data, which encourages step-by-step reasoning. This includes both existing open-source CoT datasets and synthetically generated data.
- Monte Carlo Tree Search (MCTS): To explore a wider range of potential reasoning paths, MCTS is integrated. Nodes in the search tree represent reasoning states, and actions are generated by the LLM. Confidence scores derived from the LLM's output probabilities guide the search towards more promising solutions.
- Reasoning Action Strategy: The paper explores different granularities for actions within MCTS. Instead of just using full reasoning steps, they introduce "mini-steps" (e.g., 32 or 64 tokens) to allow for finer-grained exploration of reasoning paths.
- Reflection Mechanism: A simple yet effective "reflection after thinking" prompt is added to encourage the model to self-critique and reevaluate its reasoning steps, leading to improved accuracy on challenging problems.
Datasets
The following datasets were used for training and evaluation:
- Open-O1 CoT Dataset (Filtered): A curated version of the Open-O1 project's CoT dataset, filtered for quality.
- Marco-01 CoT Dataset (Synthetic): A dataset generated using MCTS to create complex reasoning pathways.
- Marco Instruction Dataset: Data focused on instruction following.
- MGSM (Math and Science Generalization Benchmark): Used for evaluating reasoning capabilities in both English (MGSM-En) and Chinese (MGSM-Zh).
Key Insights and Ideas
- MCTS for Open-Endedness: Integrating MCTS significantly expands the solution space, which is crucial for open-ended problems where multiple valid approaches might exist.
- Granularity Matters: The choice of action granularity in MCTS (steps vs. mini-steps) can impact performance. Finer granularity (mini-steps) can help navigate complex reasoning paths, but the optimal choice might be problem-dependent.
- Self-Reflection Boosts Accuracy: A simple "rethink from scratch" prompt can significantly improve performance, especially on difficult problems, by enabling the model to act as its own critic.
- Translation of Nuance: Marco-01 shows promise in translating colloquial and slang expressions, demonstrating a deeper understanding of cultural context and nuance than standard translation tools.
Key Results
- MGSM Performance: Marco-01-CoT showed improvements over the base Qwen2-7B-Instruct model on MGSM-En. MCTS-enhanced versions (Marco-01-MCTS) further improved accuracy, with the "step as Action" strategy performing best on MGSM-En and "mini-step as Action (32)" on MGSM-Zh. For example, Marco-01-MCTS (step) achieved 90.40% accuracy on MGSM-En, compared to 84.00% for the base model.
- Test@N Evaluation: MCTS demonstrated an advantage in Test@1 accuracy, suggesting its ability to find correct solutions with fewer attempts.
- Translation Quality: Case studies showed Marco-01 providing more accurate and natural translations of colloquial Chinese phrases compared to Google Translate.
Limitations and Future Work
- Randomness in MCTS: The use of confidence scores as rewards in MCTS can introduce randomness.
- Action Strategy Nuance: Definitive conclusions on the superiority of action strategies (step vs. mini-step) could not be drawn, suggesting that the reward signal's accuracy is key.
- Future Work: The authors plan to refine the reward signal using Outcome Reward Modeling (ORM) and Process Reward Modeling (PRM). They also intend to explore reinforcement learning techniques for further fine-tuning decision-making processes and enhancing the model's ability to tackle complex real-world tasks.
Summary updated