TL;DR
GeoILP is a massive new synthetic dataset designed to push Inductive Logic Programming (ILP) out of its "small-scale" comfort zone. By leveraging plane geometry problems—ranging from textbook exercises to International Mathematical Olympiad (IMO) level—it provides a benchmark that is currently unsolvable by state-of-the-art systems, challenging the community to develop more scalable, "expert-free" rule induction methods.
What Problem Does This Paper Address?
Inductive Logic Programming (ILP) is the "interpretable" branch of machine learning. Instead of relying on black-box weights, it learns human-readable logical rules (e.g., "If X is the father of Y and Y is the father of Z, then X is the grandfather of Z").
However, ILP has a severe scaling problem. Most current systems rely heavily on hand-crafted language bias—essentially, a human expert must tell the AI exactly what kind of rules to look for, such as restricting the search to rules with only two variables or forbidding recursion. This is the symbolic equivalent of old-school feature engineering. Furthermore, existing ILP benchmarks are tiny, often featuring hypotheses with fewer than 10 rules.
The authors argue that for ILP to be useful in modern, large-scale AI applications, it must move toward automatic language bias discovery and handle much larger, more complex rule sets.
Methods
The authors created GeoILP using a "reverse" deductive reasoning approach to generate logically sound tasks.
- Deduction Engine: They built a symbolic engine based on 63 expert-defined plane geometry rules.
- Task Synthesis: The engine takes a set of geometric premises (e.g., "Point A is the midpoint of BC") and deduces all possible conclusions using forward chaining.
- Traceback: To create a specific task, they pick a conclusion (the "target") and trace the logic backward to find the minimal set of premises needed to prove it. These premises become the Background Knowledge (BK), and the target becomes the Positive Example.
- Multi-modal Data: Uniquely, they also used the diagram builder from Google DeepMind’s AlphaGeometry to generate images for each task. This allows researchers to test "neuro-symbolic" systems that must perceive a geometric image and induce logical rules simultaneously.
Datasets
The paper introduces GeoILP, which consists of 835 single-tasks and 207 multi-tasks categorized from "basic" to "complex" levels.
The scale of GeoILP is unprecedented in the field. While previous standard datasets like $\partial$ILP featured simple predicates and tiny hypotheses, GeoILP introduces massive complexity:
- Arity: Features predicates with an arity of up to 8 (e.g., equal ratios or angles involving multiple points).
- Rule Count: Hypotheses can exceed 3,500 rules.
- Language Biases: It specifically targets challenging biases like mutual recursion (where two rules define each other), predicate invention, and argument symmetry.
Key Insights and Ideas
- Geometry as the Perfect Sandbox: Plane geometry is an ideal domain for ILP because it is a rigorous formal system that naturally requires complex logical structures that are rare in simpler datasets.
- The "Expert-Free" Paradigm: By providing a dataset where the "correct" language bias is unknown to the solver, the authors are forcing the development of ILP systems that can explore the hypothesis space autonomously, much like how deep learning replaced manual feature extraction.
- The Perception-Reasoning Gap: By including image-form data, the authors highlight a major bottleneck in AI: the ability to transform raw sensory input (pixels) into symbolic knowledge and then reason over that knowledge to find general rules.
Key Results
The most striking result of the paper is that GeoILP is currently unreachable by existing methods. * Popper (a state-of-the-art symbolic ILP system) was unable to solve even the "basic" level tasks within a 24-hour window. * Difflog (a leading neuro-symbolic system) threw out-of-memory errors on a server with 500GB of RAM when attempting the basic level.
These results prove that GeoILP successfully identifies a massive gap between current ILP capabilities and the requirements of large-scale, real-world reasoning.
Limitations and Future Work
The authors acknowledge a few key limitations:
- Domain Specificity: While geometry is complex, the dataset is still confined to one mathematical domain. The rules induced might not immediately generalize to non-geometric logic tasks.
- Open-World Assumption: The negative examples are "noisy" because the rule set is technically incomplete for the entire field of plane geometry; some "false" examples might actually be true but are simply unprovable by the current engine.
The authors suggest that the next step for the community is developing more memory-efficient neural-symbolic models and search algorithms that can navigate the combinatorial explosion of the GeoILP hypothesis space.
Summary updated