WinoFlexi: A Crowdsourcing Platform for the Development of Winograd Schemas

Nicos Isaak, Loizos MichaelAustralasian Conference on Artificial Intelligence 2019 · OpenReview

Abstract

The Winograd Schema Challenge, the task of resolving pronouns in certain carefully-structured sentences, has received considerable interest in the past few years as an alternative to the Turing Test. Systems developed to tackle this challenge have typically been evaluated on a small set of hand-crafted collections of sentences, since the development of new sentences by individuals is itself a rather challenging task, requiring care and creativity. In this paper we approach the problem of developing Winograd schemas via the introduction of WinoFlexi, a flexible online crowdsourcing system. Our empirical evaluation of the system’s performance suggests that WinoFlexi allows crowdworkers to develop Winograd schemas of quality similar to that of most typical existing collections.
BibTeX
@inproceedings{isaak2019winoflexi,
  author = {Nicos Isaak and Loizos Michael},
  title = {{WinoFlexi: A Crowdsourcing Platform for the Development of Winograd Schemas}},
  booktitle = {Australasian Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=Bf3aG3LbWfi8},
}

Loading related content...

Figure