Open-Source Developer Simon Willison Exposes Critical Vulnerabilities in LLMs

Open-source developer and co-creator of Django, Simon Willison, has raised concerns about a critical vulnerability, known as prompt injection, in Large Language Models (LLMs) like GPT-3, GPT-4, and ChatGPT. Willison warns that this security risk might not be receiving the attention it deserves, posing a significant threat to the rapidly growing field of sophisticated LLM applications.

Prompt injection occurs when an attacker concatenates untrusted input with a carefully crafted prompt, potentially compromising LLM applications with additional capabilities such as the ReAct pattern, Auto-GPT, and ChatGPT Plugins. These systems use LLMs to trigger additional tools, make API requests, run searches, or execute generated code in an interpreter or a shell.

While the consequences of prompt injection may be insignificant for some applications, others, such as Justin Alvey’s AI assistant prototype, are at risk of serious security breaches. Alvey’s prototype uses Whisper and ChatGPT API prompts to perform actions like searching emails and sending replies, which could be exploited through email-based prompt injection attacks. In a scenario where an attacker sends an email with instructions to forward and delete specific emails, the AI assistant may inadvertently execute the malicious commands, compromising the user’s security.

Search index poisoning is another area where prompt injection can be misused. AI-enhanced search engines like Bing may be tricked into providing incorrect or misleading information about a subject. This could lead to LLM-optimization, a new form of SEO for LLM-assisted search, paving the way for malicious uses of this technique. In one instance, Mark Riedl added a hidden note to his academic profile page instructing Bing to describe him as a time travel expert, successfully manipulating the search engine. This vulnerability could be exploited in more harmful ways, such as manipulating product comparisons or spreading disinformation.

Data exfiltration attacks, another type of prompt injection, can compromise user data through plugins. For example, an email containing an SQL query can trigger a ChatGPT plugin to exfiltrate user data by generating a URL with the stolen information embedded in the query string. In this case, the user may unwittingly click on a seemingly innocuous link that sends sensitive data to an attacker’s website. With the increasing popularity of ChatGPT Plugins and the combination of various plugins, the possibility of such attacks occurring is heightened, exposing users to sophisticated and malicious threats.

Indirect Prompt Injection, a term coined by Kai Greshake and team, involves injection attacks hidden in text that the LLM consumes during its execution. One example of this is an attack against Bing Chat, an Edge browser feature, which was tricked into pursuing a secret agenda and exfiltrating user data via a malicious link. By constructing a prompt with hidden instructions, the researchers were able to make Bing Chat attempt to obtain the user’s real name and send it to an attacker’s URL.

Willison believes there is no 100% reliable protection against prompt injection attacks. However, he suggests making generated prompts visible to users and involving them in potentially dangerous actions as possible solutions. By allowing users to review prompts and actions taken by their AI assistants, the risk of falling prey to injection attacks could be reduced.

Roman Samoilenko, another researcher in the field, discovered a method to exfiltrate data through markdown images in ChatGPT, further highlighting the need for security measures in LLM applications. OpenAI, the company behind ChatGPT, has been working on addressing these concerns by separating their “Code Interpreter” and “Browse” modes from the general plugins mechanism, presumably to help avoid malicious interactions. Nevertheless, the increasing variety and combinations of existing or future plugins remain a significant concern for Willison and other experts in the field.

To mitigate the risks associated with prompt injection attacks, developers and users of LLM applications should consider implementing additional security measures. One such measure is asking for user confirmation before performing potentially dangerous actions. For instance, instead of automatically sending an email, an AI assistant could first show the user the email content for review. While not a perfect solution, it provides a layer of protection against some obvious attacks.

In light of Simon Willison’s discoveries, it is crucial for developers, security researchers, and users to address these vulnerabilities and work collaboratively towards securing LLM applications. As the field of AI continues to advance rapidly, ensuring the safety and integrity of these systems is of paramount importance to protect users and maintain the trust required for further innovation.

Willison’s revelations about the potential dangers of prompt injection in LLMs serve as a critical wake-up call for the AI community. As the development and adoption of AI applications continue to skyrocket, addressing these security issues is essential to preserve user trust and ensure the responsible growth of this transformative technology. By working together to uncover and mitigate vulnerabilities, developers and researchers can help safeguard the future of AI and its applications.

https://simonwillison.net/2023/Apr/14/worst-that-can-happen/

{
"prompt": "Open-source developer and co-creator of Django has raised concerns about a critical vulnerability, known as prompt injection, in Large Language Models, deepleaps.com, Beautiful Lighting, Detailed Render, Intricate Environment, CGI, Cinematic, Dramatic, HD",
"seed": 1974206816,
"used_random_seed": false,
"negative_prompt": "",
"num_outputs": 1,
"num_inference_steps": 500,
"guidance_scale": 7.5,
"width": 512,
"height": 512,
"vram_usage_level": "high",
"sampler_name": "euler",
"use_stable_diffusion_model": "Dreamshaper_3.32_baked_vae_clip_fix",
"use_vae_model": "vae-ft-mse-840000-ema-pruned",
"stream_progress_updates": true,
"stream_image_progress": false,
"show_only_filtered_image": true,
"block_nsfw": false,
"output_format": "jpeg",
"output_quality": 75,
"output_lossless": false,
"metadata_output_format": "json",
"original_prompt": "Open-source developer and co-creator of Django has raised concerns about a critical vulnerability, known as prompt injection, in Large Language Models, deepleaps.com",
"active_tags": [
"Beautiful Lighting",
"Detailed Render",
"Intricate Environment",
"CGI",
"Cinematic",
"Dramatic",
"HD"
],
"inactive_tags": [],
"use_upscale": "RealESRGAN_x4plus",
"upscale_amount": "4"
}