about
Facts About Building Retrieval Augmented Generation-Based Chatbots (arxiv.org)
1 point by belter on Jul 11, 2024 | hide | past | pdf | discuss on HN

In plain words: FACTS names five things to get right in company chatbots that answer from your documents: fresh content, setup, cost, testing, and security, plus 15 places to tune the pipeline. From three NVIDIA chatbots, it measures how big and small models trade accuracy against speed.

Abstract · FACTS About Building Retrieval Augmented Generation-based Chatbots

Enterprise chatbots, powered by generative AI, are emerging as key applications to enhance employee productivity. Retrieval Augmented Generation (RAG), Large Language Models (LLMs), and orchestration frameworks like Langchain and Llamaindex are crucial for building these chatbots. However, creating effective enterprise chatbots is challenging and requires meticulous RAG pipeline engineering. This includes fine-tuning embeddings and LLMs, extracting documents from vector databases, rephrasing queries, reranking results, designing prompts, honoring document access controls, providing concise responses, including references, safeguarding personal information, and building orchestration agents. We present a framework for building RAG-based chatbots based on our experience with three NVIDIA chatbots: for IT/HR benefits, financial earnings, and general content. Our contributions are three-fold: introducing the FACTS framework (Freshness, Architectures, Cost, Testing, Security), presenting fifteen RAG pipeline control points, and providing empirical results on accuracy-latency tradeoffs between large and small LLMs. To the best of our knowledge, this is the first paper of its kind that provides a holistic view of the factors as well as solutions for building secure enterprise-grade chatbots."

Rama Akkiraju, Anbang Xu, Deepak Bora, Tan Yu, Lu An, Vishal Seth, Aaditya Shukla, Pritam Gundecha, Hridhay Mehta, Ashwin Jha, Prithvi Raj, Abhinav Balasubramanian, et al.
arXiv:2407.07858 · cs.LG, cs.CL · submitted Jul 10, 2024
abstract · pdf · html · 8 pages, 6 figures, 2 tables, Preprint submission to ACM CIKM 2024

add comment on HN