about
Unifying Multimodal Retrieval via Document Screenshot Embedding (arxiv.org)
2 points by fzliu on Sep 7, 2024 | hide | past | pdf | discuss on HN

In plain words: Instead of pulling text out of documents, this system feeds page screenshots to an AI model that reads images and turns a page into a searchable vector, keeping text, images, and layout. On Wikipedia search it beat keyword matching by 17 points in top-1 accuracy.

Abstract

In the real world, documents are organized in different formats and varied modalities. Traditional retrieval pipelines require tailored document parsing techniques and content extraction modules to prepare input for indexing. This process is tedious, prone to errors, and has information loss. To this end, we propose Document Screenshot Embedding (DSE), a novel retrieval paradigm that regards document screenshots as a unified input format, which does not require any content extraction preprocess and preserves all the information in a document (e.g., text, image and layout). DSE leverages a large vision-language model to directly encode document screenshots into dense representations for retrieval. To evaluate our method, we first craft the dataset of Wiki-SS, a 1.3M Wikipedia web page screenshots as the corpus to answer the questions from the Natural Questions dataset. In such a text-intensive document retrieval setting, DSE shows competitive effectiveness compared to other text retrieval methods relying on parsing. For example, DSE outperforms BM25 by 17 points in top-1 retrieval accuracy. Additionally, in a mixed-modality task of slide retrieval, DSE significantly outperforms OCR text retrieval methods by over 15 points in nDCG@10. These experiments show that DSE is an effective document retrieval paradigm for diverse types of documents. Model checkpoints, code, and Wiki-SS collection will be released.

Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, Jimmy Lin
arXiv:2406.11251 · cs.IR · submitted Jun 17, 2024 · updated Dec 2, 2024
abstract · pdf · html · EMNLP2024 main

add comment on HN