TL;DR
This paper introduces PGPSNet, a novel multi-modal neural solver for geometry problem solving (GPS) that effectively fuses textual problem descriptions with diagram information. By converting diagrams into structured textual clauses and employing a specialized pre-training strategy, PGPSNet achieves state-of-the-art performance on existing benchmarks and a new large-scale dataset, PGPS9K.
What Problem Does This Paper Address?
Geometry Problem Solving (GPS) is a challenging AI task that requires understanding both natural language descriptions and visual geometric diagrams. Existing neural solvers struggle to effectively leverage the rich structural and semantic information present in diagrams, often leading to a performance gap compared to symbolic solvers. The paper highlights that standard image feature extraction methods fail to capture the fine-grained details and complex spatial relationships within geometry diagrams, hindering multi-modal fusion and geometric reasoning.
Methods
The core of PGPSNet lies in its innovative approach to representing and fusing multi-modal information:
- Textual Clause Representation: Diagrams are converted into basic textual clauses, comprising "structural clauses" (describing connections between geometric primitives) and "semantic clauses" (describing relationships between geometric and non-geometric primitives). This transforms visual information into a more structured, text-based format.
- Structural and Semantic Pre-training: A language model is pre-trained using a Masked Language Modeling (MLM) task on these textual clauses and the problem text. This pre-training incorporates class and section tags to imbue the model with a deeper understanding of geometric structure and semantics.
- Multi-Modal Fusion: A CNN encoder processes the diagram for coarse-grained visual features, which are then combined with the text embeddings (enhanced by pre-training). This fused representation is fed into a bidirectional GRU encoder for context aggregation.
- Self-Limited GRU Decoder: A novel decoder is proposed to generate the solution program sequentially. It limits the search space of output tokens by only considering those present in the problem text, improving efficiency and performance.
- Data Augmentation: Five strategies are employed to increase the diversity and robustness of the training data, including token replacement, clause shuffling, and diagram flipping.
Datasets
- Geometry3K: An existing dataset used for evaluation.
- PGPS9K: A newly introduced large-scale dataset comprising 9,022 geometry problems. It is notable for its fine-grained diagram annotations and interpretable, theorem-based solution programs, making it suitable for training and evaluating neural solvers.
Key Insights and Ideas
- Textualizing Diagrams is Key: Converting geometric diagrams into structured textual clauses is a highly effective way to bridge the gap between visual and textual modalities for GPS. This approach preserves fine-grained information that standard image features miss.
- Pre-training Matters for Geometry: Applying structural and semantic pre-training specifically tailored to geometric textual clauses significantly boosts the model's ability to understand and reason about geometry, leading to substantial performance gains.
- Interpretable Solution Programs: The paper emphasizes the importance of generating interpretable, theorem-based solution programs, which not only aids in understanding the model's reasoning but also simplifies the learning process.
- Self-Limited Decoding for Efficiency: The proposed self-limited GRU decoder offers a more efficient and effective way to generate solution programs compared to complex tree decoders, especially for the structured output required in GPS.
Key Results
- PGPSNet achieves state-of-the-art performance on the Geometry3K dataset, outperforming existing neural solvers and achieving comparable results to well-designed symbolic solvers.
- On the newly introduced PGPS9K dataset, PGPSNet demonstrates significant improvements, particularly in "Completion" and "Choice" metrics.
- Ablation studies confirm the crucial contributions of each component: data augmentation, the self-limited decoder, structural clauses, and especially the structural and semantic pre-training. The pre-training alone led to a substantial performance gain.
Limitations and Future Work
- Performance Gap with Human Experts: The paper acknowledges that a significant performance gap still exists between PGPSNet and human experts, indicating room for improvement in handling more complex geometric reasoning.
- Complex Geometric Reasoning: While PGPSNet shows promise, it still struggles with highly complex geometric problems that require combining multiple conditions or advanced reasoning steps, as illustrated in the case studies.
- Future Directions: The authors suggest that more training samples or more elaborate modal fusion techniques could further enhance the model's capabilities. The potential for improvement with more advanced pre-training strategies and larger datasets is also implied.
Summary updated