about
Making a Miracl: Multilingual Information Retrieval Across a Continuum of Langs (arxiv.org)
1 point by caprock on Dec 13, 2022 | hide | past | pdf | discuss on HN

In plain words: A new search test collection in 18 languages pairs Wikipedia articles with queries and native-speaker judgments, so systems can be scored on answering questions in the same language. It holds over 700,000 judgments for about 77,000 queries, covering languages usually left out of such tests.

Abstract · Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages

MIRACL (Multilingual Information Retrieval Across a Continuum of Languages) is a multilingual dataset we have built for the WSDM 2023 Cup challenge that focuses on ad hoc retrieval across 18 different languages, which collectively encompass over three billion native speakers around the world. These languages have diverse typologies, originate from many different language families, and are associated with varying amounts of available resources -- including what researchers typically characterize as high-resource as well as low-resource languages. Our dataset is designed to support the creation and evaluation of models for monolingual retrieval, where the queries and the corpora are in the same language. In total, we have gathered over 700k high-quality relevance judgments for around 77k queries over Wikipedia in these 18 languages, where all assessments have been performed by native speakers hired by our team. Our goal is to spur research that will improve retrieval across a continuum of languages, thus enhancing information access capabilities for diverse populations around the world, particularly those that have been traditionally underserved. This overview paper describes the dataset and baselines that we share with the community. The MIRACL website is live at http://miracl.ai/.

Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, Jimmy Lin
arXiv:2210.09984 · cs.IR, cs.CL · submitted Oct 18, 2022
abstract · pdf · html

add comment on HN