For decades, the "Digital Aristotle"—an AI capable of reading a textbook and passing a science exam—has been a benchmark for artificial intelligence. While modern Large Language Models (LLMs) have finally begun to pass these exams, researchers like Peter Clark note that these successes often rely on text-heavy assessments that bypass the visual and spatial reasoning central to actual scientific practice.
The emerging consensus across recent research is that "knowing science" is not merely a matter of linguistic fluency or factual recall. Instead, true scientific intelligence requires multimodal reasoning: the ability to synthesize textual descriptions with visual data like diagrams, charts, and molecular structures.
The Perception-Reasoning Gap
Current Vision-Language Models (VLMs) often suffer from a "perception-reasoning gap." As demonstrated in MaCBench, a model might excel at basic perception—identifying a beaker or reading a value from a table—yet fail significantly when asked to perform spatial reasoning or multi-step logical inference based on that visual data.
In physics, for instance, a model must do more than recognize a circuit diagram; it must map those visual features to physical variables to solve a mathematical problem. Benchmarks like SciBench and UGPhysics show that even advanced models like GPT-4 struggle with this integration, often failing to understand the physical implications of visual representations like force vectors or wave functions.
Beyond Textual Tables and Charts
In specialized domains like finance and e-commerce, information is rarely presented as pure text. It is frequently embedded in "multimodal tables" that interleave text with icons, sparklines, and maps.
Standard benchmarks often flatten these structures into text, but MMTABQA and MMTBENCH reveal that this approach fails to capture how humans actually interpret data. For example, verifying a claim against a chart requires a model to jointly process textual labels and geometric structures (like bar heights or line slopes). Models like ChartBERT, introduced in ChartFC, attempt to solve this by using spatial "structural embeddings" to bridge the gap between natural language and visual evidence.
Domain-Specific Visual Semantics
General-purpose models often miss the specific "grammar" of scientific visualizations. In chemistry, a molecular diagram isn't just an image; it's a precise semantic map of atoms and bonds. Junxian Li and colleagues developed ChemVLM to address this, showing that models specifically engineered to bridge the gap between chemical visual data and textual reasoning can outperform generalist models on tasks like reaction scheme interpretation.
Similarly, in theoretical physics, models often fail to grasp the underlying connectivity rules of specialized diagrams. Sirui Lu argues that an "AI Physicist" must move beyond recognizing Feynman diagrams to understanding the rigorous mathematical derivations they represent. Without this deep multimodal synthesis, models remain "passive guessers" rather than active scientific collaborators.
From One-Shot Answers to Agentic Discovery
The research direction is shifting from evaluating "one-shot" question answering to assessing autonomous "DeepResearch" agents. These agents, such as the one proposed in SR-Scientist, use code interpreters and iterative loops to propose and refine hypotheses.
However, a critical bottleneck remains: visual feedback. A human scientist refines a hypothesis by looking at a residual plot or a diagnostic signal. Tianyu Fan and others emphasize that until evaluation frameworks like DEEPRESEARCH-REPORTEVAL can holistically measure an agent's ability to generate and interpret these visual elements, we cannot fully trust AI to drive the discovery lifecycle.
The Path Forward: Decoupling and Mastery
To move forward, researchers are developing frameworks that "decouple" different capabilities. FinEval-KR and FinCDM suggest that we must distinguish between a model's domain knowledge (e.g., knowing a tax law) and its reasoning ability (e.g., applying that law to a complex financial chart).
By moving from "score flattening" to detailed "mastery profiles," the field is beginning to identify exactly where the multimodal link breaks—whether it's a failure of visual perception, a lack of domain-specific logic, or an inability to synthesize the two into a cohesive scientific workflow.