about
Novel Prompt Injection Threats to Application-Integrated Large Language Models (arxiv.org)
1 point by rntn on Mar 9, 2023 | hide | past | pdf | discuss on HN

In plain words: Attackers hide instructions inside data an AI assistant reads, like web pages or emails, so the app obeys them without the user typing anything. The trick worked on real tools like Bing Chat, letting attackers steal data and control which actions the app takes.

Abstract · Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

Large Language Models (LLMs) are increasingly being integrated into various applications. The functionalities of recent LLMs can be flexibly modulated via natural language prompts. This renders them susceptible to targeted adversarial prompting, e.g., Prompt Injection (PI) attacks enable attackers to override original instructions and employed controls. So far, it was assumed that the user is directly prompting the LLM. But, what if it is not the user prompting? We argue that LLM-Integrated Applications blur the line between data and instructions. We reveal new attack vectors, using Indirect Prompt Injection, that enable adversaries to remotely (without a direct interface) exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts and vulnerabilities, including data theft, worming, information ecosystem contamination, and other novel security risks. We demonstrate our attacks' practical viability against both real-world systems, such as Bing's GPT-4 powered Chat and code-completion engines, and synthetic applications built on GPT-4. We show how processing retrieved prompts can act as arbitrary code execution, manipulate the application's functionality, and control how and if other APIs are called. Despite the increasing integration and reliance on LLMs, effective mitigations of these emerging threats are currently lacking. By raising awareness of these vulnerabilities and providing key insights into their implications, we aim to promote the safe and responsible deployment of these powerful models and the development of robust defenses that protect users and systems from potential attacks.

Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, Mario Fritz
arXiv:2302.12173 · cs.CR, cs.AI, cs.CL, cs.CY · submitted Feb 23, 2023 · updated May 5, 2023
abstract · pdf · html

add comment on HN
Also discussed: May 2023 (43 points, 20 comments) · May 2023 (3 points, 2 comments) · Feb 2023 (2 points, 1 comment) · Feb 2023 (6 points, 1 comment) · Feb 2023 (8 points, 2 comments)