about
The Legal Embedding Benchmark (MLEB) (arxiv.org)
1 point by fzliu 257 days ago | hide | past | pdf | discuss on HN

In plain words: An open benchmark tests how well systems find legal information, using ten hand-labeled sets of cases, laws and contracts across several countries, for search, sorting and question answering. It is the largest legal retrieval test yet, with most sets newly built to fill gaps.

Abstract · The Massive Legal Embedding Benchmark (MLEB)

We present the Massive Legal Embedding Benchmark (MLEB), the largest, most diverse, and most comprehensive open-source benchmark for legal information retrieval to date. MLEB consists of ten expert-annotated datasets spanning multiple jurisdictions (the US, UK, EU, Australia, Ireland, and Singapore), document types (cases, legislation, regulatory guidance, contracts, and literature), and task types (search, zero-shot classification, and question answering). Seven of the datasets in MLEB were newly constructed in order to fill domain and jurisdictional gaps in the open-source legal information retrieval landscape. We document our methodology in building MLEB and creating the new constituent datasets, and release our code, results, and data openly to assist with reproducible evaluations.

Umar Butler, Abdur-Rahman Butler, Adrian Lucas Malec
arXiv:2510.19365 · cs.CL, cs.AI, cs.IR · submitted Oct 22, 2025
abstract · pdf · html · 15 pages, 2 figures

add comment on HN