about
Grounded in Context: Retrieval-Based Method for Hallucination Detection (arxiv.org)
1 point by AsDivyansh on May 20, 2025 | hide | past | pdf | discuss on HN

In plain words: It checks long AI answers for made-up claims by pulling the most relevant source passages and letting a small text-matching model judge whether each claim is supported. It scored 0.83 at spotting unsupported claims, matching systems trained on that test set and beating other small-model tools.

Abstract · ORION Grounded in Context: Retrieval-Based Method for Hallucination Detection

Despite advancements in grounded content generation, production Large Language Models (LLMs) based applications still suffer from hallucinated answers. We present "Grounded in Context" - a member of Deepchecks' ORION (Output Reasoning-based InspectiON) family of lightweight evaluation models. It is our framework for hallucination detection, designed for production-scale long-context data and tailored to diverse use cases, including summarization, data extraction, and RAG. Inspired by RAG architecture, our method integrates retrieval and Natural Language Inference (NLI) models to predict factual consistency between premises and hypotheses using an encoder-based model with only a 512-token context window. Our framework identifies unsupported claims with an F1 score of 0.83 in RAGTruth's response-level classification task, matching methods that trained on the dataset, and outperforming all comparable frameworks using similar-sized models.

Assaf Gerner, Netta Madvil, Nadav Barak, Alex Zaikman, Jonatan Liberman, Liron Hamra, Rotem Brazilay, Shay Tsadok, Yaron Friedman, Neal Harow, Noam Bressler, Shir Chorev, et al.
arXiv:2504.15771 · cs.LG · submitted Apr 22, 2025 · updated May 22, 2025
abstract · pdf · html

add comment on HN