Findings of the First Workshop on Simulating Conversational Intelligence in Chat

Yvette Graham, Mohammed Rameez Qureshi, Haider Khalid, Gerasimos Lampouras, Ignacio Iacobacci, Qun Liu

Introduction

The SCI-CHAT 2024 workshop was established to address the persistent "evaluation crisis" in open-domain dialogue systems. While standard NLP metrics like BLEU or ROUGE are effective for tasks with narrow reference targets, they fail significantly in open-domain chat where a single prompt can elicit thousands of valid, creative responses. This paper details a robust, live human evaluation framework designed to measure "conversational intelligence"—the ability to maintain topical depth, posit arguments, and reason over multiple turns.

Methodology

To ensure reliable results, the workshop organizers moved away from automated metrics toward a controlled crowdsourcing approach using Direct Assessment (DA).

Steered Conversation and Direct Assessment

To prevent interactions from devolving into shallow pleasantries, human judges were assigned specific topics derived from Freakonomics podcast titles. Following each multi-turn interaction, judges rated the models on a 0–100 scale across nine distinct criteria: Overall, Intelligent, Interesting, Informative, Fluent, Credible, Inconsistent, Incoherent, and Repetitive.

Quality Control and Normalization

A significant challenge in crowdsourced evaluation is the presence of "noisy" or low-effort data. To mitigate this, the organizers employed a "bad model" trick: a control model known to produce poor responses was included in the evaluation pool. Workers who could not statistically distinguish between a legitimate system and the control model (using a significance test with $p < 0.05$) were discarded.

Distribution of worker correlation scores (r) showing the separation between those who passed and failed quality control.

As shown in the quality control analysis, only approximately 17.6% of workers passed these rigorous checks. Finally, raw scores were converted to z-scores to account for individual judge biases, and Mann-Whitney U tests were used to determine statistical significance between model rankings.

Results and Analysis

The evaluation revealed several key insights into the current state of conversational AI. While models have become highly proficient at sounding human, there remains a gap in their ability to provide consistently logical or informative content.

Model Performance

ANON-A, a sequence-to-sequence model with 400 million parameters fine-tuned on Reddit data, emerged as the outright winner in the "Overall" category. However, the results also demonstrated a "clustering" effect, where several systems (including SARCEMP and EMO-GEN BART) were statistically tied for second place.

Table of standardized scores across nine evaluation criteria for participating models.

The Fluency-Intelligence Gap

A notable trend across all participants was that models consistently scored higher on "Fluency" than on "Intelligence" or "Informativeness." This suggests that while Large Language Models (LLMs) are excellent at maintaining grammatical and syntactic coherence, they still struggle with the deeper requirements of conversational intelligence, such as logical consistency and factual depth.

Discussion and Limitations

While the workshop proved that live human evaluation is feasible and can produce meaningful rankings within a short timeframe, several limitations remain. The evaluation topics were sampled from the same Freakonomics dataset used to train the baseline models, potentially introducing an in-distribution bias. Future iterations of the workshop will focus on out-of-distribution topics to better stress-test the models' reasoning capabilities.

Conclusion

The SCI-CHAT 2024 findings demonstrate that a rigorous human-in-the-loop framework can overcome the limitations of automated metrics. By combining steered conversations with strict quality control, the community can move closer to a standardized benchmark for measuring true conversational intelligence in open-domain agents.

Summary updated

BibTeX
@inproceedings{graham2024findings,
  author = {Yvette Graham and Mohammed Rameez Qureshi and Haider Khalid and Gerasimos Lampouras and Ignacio Iacobacci and Qun Liu},
  title = {{Findings of the First Workshop on Simulating Conversational Intelligence in Chat}},
  booktitle = {EACL},
  year = {2024},
}
Figure