about
The Defeat of the Winograd Schema Challenge (arxiv.org)
1 point by YeGoblynQueenne on Mar 1, 2022 | hide | past | pdf | discuss on HN

In plain words: The Winograd Schema Challenge is a set of twin sentences where a pronoun must be resolved using everyday commonsense. A review of its history finds that by 2019 AI systems passed over 90% of them, showing such puzzle tests are weak stand-ins for intelligence.

Abstract

The Winograd Schema Challenge - a set of twin sentences involving pronoun reference disambiguation that seem to require the use of commonsense knowledge - was proposed by Hector Levesque in 2011. By 2019, a number of AI systems, based on large pre-trained transformer-based language models and fine-tuned on these kinds of problems, achieved better than 90% accuracy. In this paper, we review the history of the Winograd Schema Challenge and discuss the lasting contributions of the flurry of research that has taken place on the WSC in the last decade. We discuss the significance of various datasets developed for WSC, and the research community's deeper understanding of the role of surrogate tasks in assessing the intelligence of an AI system.

Vid Kocijan, Ernest Davis, Thomas Lukasiewicz, Gary Marcus, Leora Morgenstern
arXiv:2201.02387 · cs.CL · submitted Jan 7, 2022 · updated Jan 23, 2023
abstract · pdf · html

add comment on HN