Proposition from the Perspective of Chinese Language: A Chinese Proposition Classification Evaluation Benchmark

Conghui Niu, Mengyang Hu, Lin Bo, Xiaoli He, Dong Yu, Pengyuan Liu

Abstract

Existing propositions often rely on logical constants for classification. Compared with Western languages that lean towards hypotaxis such as English, Chinese often relies on semantic or logical understanding rather than logical connectives in daily expressions, exhibiting the characteristics of parataxis. However, existing research has rarely paid attention to this issue. And accurately classifying these propositions is crucial for natural language understanding and reasoning. In this paper, we put forward the concepts of explicit and implicit propositions and propose a comprehensive multi-level proposition classification system based on linguistics and logic. Correspondingly, we create a large-scale Chinese proposition dataset PEACE from multiple domains, covering all categories related to propositions. To evaluate the Chinese proposition classification ability of existing models and explore their limitations, We conduct evaluations on PEACE using several different methods including the Rule-based method, SVM, BERT, RoBERTA, and ChatGPT. Results show the importance of properly modeling the semantic features of propositions. BERT has relatively good proposition classification capability, but lacks cross-domain transferability. ChatGPT performs poorly, but its classification ability can be improved by providing more proposition information. Many issues are still far from being resolved and require further study.

Introduction: The Challenge of Parataxis

In formal logic, "logical constants"—words like if, then, and, and all—serve as the scaffolding for understanding argument structure. Western languages like English are predominantly "hypotactic," relying heavily on these explicit connectives. In contrast, Chinese is "paratactic." In natural Chinese discourse, logical relationships are frequently left unsaid, woven instead into the semantic fabric of the sentence. For example, while English might require "If it rains, the ground is wet," a Chinese speaker might simply say "下雨,地湿" (Rain, ground wet).

Humans intuitively bridge these gaps, but machines often falter without explicit markers. This distinction between explicit and implicit propositions represents a significant hurdle for natural language understanding (NLU) in non-Western languages. A new study from researchers at Beijing Language and Culture University addresses this by introducing PEACE (Proposition Evaluation and Analysis for Chinese Expressions), a benchmark designed to move beyond keyword-matching toward true semantic logical understanding.

The PEACE Dataset

The PEACE dataset is a large-scale resource comprising over 45,000 annotated sentences. It is distinguished by its scale, its coverage of four distinct domains—Comprehensive/Encyclopedia (C&E), Finance, Law, and Medicine—and its focus on implicit logic. Unlike previous datasets, PEACE specifically includes "implicit" propositions where logical constants are omitted, reflecting the natural distribution of the Chinese language. Notably, in the "EI" (Encyclopedia Implicit) subset, nearly 100% of categorical propositions are implicit, highlighting the inadequacy of traditional rule-based approaches.

A Multi-Level Classification Framework

The researchers propose a hierarchical system based on both linguistics and formal logic to categorize these complex expressions.

The multi-level proposition classification system.

The system operates across four levels:

  1. Modal Proposition Recognition (MPR): Distinguishing between non-modal (factual) and modal (expressing possibility, obligation, or ability) statements.
  2. Non-modal Classification (NPC): Categorizing facts into categorical, conjunctive, disjunctive, or hypothetical types.
  3. Categorical Classification (CPC): Breaking down simple statements into the four Aristotelian types (Universal/Particular Affirmative/Negative).
  4. Modal Classification (MPC): Sorting modal statements into Possible, Deontic (permission/obligation), or Dynamic (ability/willingness).

Experimental Results and Model Performance

The framework was tested using traditional methods (Rule-based and SVM), Pre-trained Language Models (PLMs like BERT and RoBERTa), and Large Language Models (LLMs like ChatGPT).

BERT’s Semantic Edge

BERT and RoBERTa significantly outperformed rule-based systems, achieving high in-domain F1-scores (up to 0.95 on Categorical tasks). This suggests that these models capture underlying semantic logic rather than merely memorizing keywords. However, a significant "implicit logic gap" remains: on the NPC task, BERT’s F1-score plummeted from 0.91 on explicit data to 0.48 on implicit data. Furthermore, models trained on general data showed sharp performance drops when applied to specialized fields like Medicine or Law, indicating that logical expression is highly domain-dependent.

The ChatGPT Learning Curve

ChatGPT initially struggled in zero-shot settings (ChatGPT-0), likely because its internal definition of a "proposition" is tied to formal, explicit logic. However, its performance improved steadily as more examples were provided through few-shot prompting.

ChatGPT performance across different tasks and domains with increasing few-shot examples.

As shown in the figure above, ChatGPT's F1-score on the MPR task rose from 0.38 (zero-shot) to 0.44 with five examples (ChatGPT-5). While this allows it to eventually outperform rule-based methods, it still trails behind fine-tuned BERT models.

Conclusion

The PEACE benchmark reveals that while modern AI models have made strides in understanding Chinese logic, they remain fragile when faced with implicit structures and domain shifts. The "parataxis" of Chinese requires models to move beyond surface-level keyword detection toward deeper semantic modeling. Future research must focus on improving "logical migration"—the ability to recognize logical structures across diverse and specialized contexts.

Paper Citation: Niu, C., et al. (2023). Proposition from the Perspective of Chinese Language: A Chinese Proposition Classification Evaluation Benchmark. arXiv:2309.09602v1 [cs.CL].

Summary updated

Abstract

Existing propositions often rely on logical constants for classification. Compared with Western languages that lean towards hypotaxis such as English, Chinese often relies on semantic or logical understanding rather than logical connectives in daily expressions, exhibiting the characteristics of parataxis. However, existing research has rarely paid attention to this issue. And accurately classifying these propositions is crucial for natural language understanding and reasoning. In this paper, we put forward the concepts of explicit and implicit propositions and propose a comprehensive multi-level proposition classification system based on linguistics and logic. Correspondingly, we create a large-scale Chinese proposition dataset PEACE from multiple domains, covering all categories related to propositions. To evaluate the Chinese proposition classification ability of existing models and explore their limitations, We conduct evaluations on PEACE using several different methods including the Rule-based method, SVM, BERT, RoBERTA, and ChatGPT. Results show the importance of properly modeling the semantic features of propositions. BERT has relatively good proposition classification capability, but lacks cross-domain transferability. ChatGPT performs poorly, but its classification ability can be improved by providing more proposition information. Many issues are still far from being resolved and require further study.
BibTeX
@misc{niu2023proposition,
  author = {Conghui Niu and Mengyang Hu and Lin Bo and Xiaoli He and Dong Yu and Pengyuan Liu},
  title = {{Proposition from the Perspective of Chinese Language: A Chinese Proposition Classification Evaluation Benchmark}},
  year = {2023},
  eprint = {2309.09602},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.09602},
}
Figure