about
Ian Soboroff: Don't Use LLMs to Make Relevance Judgments (arxiv.org)
1 point by danielsgriffin on Sep 24, 2024 | hide | past | pdf | 2 comments on HN

In plain words: Search test collections rely on trained human contractors who read documents and mark which ones truly answer a query. This keynote argues large language models should not replace that human labeling, saying the expensive human process is still the right one.

Abstract · Don't Use LLMs to Make Relevance Judgments

Making the relevance judgments for a TREC-style test collection can be complex and expensive. A typical TREC track usually involves a team of six contractors working for 2-4 weeks. Those contractors need to be trained and monitored. Software has to be written to support recording relevance judgments correctly and efficiently. The recent advent of large language models that produce astoundingly human-like flowing text output in response to a natural language prompt has inspired IR researchers to wonder how those models might be used in the relevance judgment collection process. At the ACM SIGIR 2024 conference, a workshop ``LLM4Eval'' provided a venue for this work, and featured a data challenge activity where participants reproduced TREC deep learning track judgments, as was done by Thomas et al (arXiv:2408.08896, arXiv:2309.10621). I was asked to give a keynote at the workshop, and this paper presents that keynote in article form. The bottom-line-up-front message is, don't use LLMs to create relevance judgments for TREC-style evaluations.

Ian Soboroff
arXiv:2409.15133 · cs.IR · submitted Sep 23, 2024 · updated Mar 26, 2025
abstract · pdf · html

add comment on HN

> Information retrieval is a field founded on inescapable uncertainty. We cannot fully model the meaning of the documents, because language is fundamentally ambiguous. We cannot fully comprehend the user’s notion of relevance, because that notion is defined with respect to the entire cognitive state of the person, which changes as they use the system. When we receive a query from a user, it is a poor representation of the user’s need or goal, and systems are forced to guess where to look. Industry systems make extensive use of behavioral observations, which are uncertain in their implications. So even though we have many tools, we cannot actually perform matching of documents to information needs with absolute certainty