about
TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks (2023) (arxiv.org)
2 points by locknitpicker 182 days ago | hide | past | pdf | discuss on HN

In plain words: A classification system sorts prompts by style and level of detail, so studies on hard tasks can report which kind of prompt they used. Since answers change a lot with wording, this shared standard lets different studies compare results fairly instead of mixing unlike prompts.

Abstract · TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks

While LLMs have shown great success in understanding and generating text in traditional conversational settings, their potential for performing ill-defined complex tasks is largely under-studied. Indeed, we are yet to conduct comprehensive benchmarking studies with multiple LLMs that are exclusively focused on a complex task. However, conducting such benchmarking studies is challenging because of the large variations in LLMs' performance when different prompt types/styles are used and different degrees of detail are provided in the prompts. To address this issue, the paper proposes a general taxonomy that can be used to design prompts with specific properties in order to perform a wide range of complex tasks. This taxonomy will allow future benchmarking studies to report the specific categories of prompts used as part of the study, enabling meaningful comparisons across different studies. Also, by establishing a common standard through this taxonomy, researchers will be able to draw more accurate conclusions about LLMs' performance on a specific complex task.

Shubhra Kanti Karmaker Santu, Dongji Feng
arXiv:2305.11430 · cs.AI, cs.CL, cs.IR, cs.LG · submitted May 19, 2023 · updated Oct 24, 2023
abstract · pdf · html · Accepted to EMNLP 2023

add comment on HN