📃 On 18.7.2023, the German Federal Office for Information Security (BSI) published an…

← All articles
IT security · EN

📃 On 18.7.2023, the German Federal Office for Information Security (BSI) published an IT security information on vulnerabilities in large AI language models (LLM)

The use of language models is not only an opportunity for digitalisation, but also poses new types of IT security risks and increases the potential for misuse. As the BSI states in its briefing note, many of the “anticipated use cases involve the unchecked processing of data from insecure sources”, with the result that attackers can manipulate the data in the language models or even place “undesired instructions” for the LLMs. The technical term for this type of manipulation is Indirect Prompt Injection. Behind this is a technique by which a prompt is sent in such a way that a deployed language model takes it over and applies it to subsequent model executions.

To illustrate the risks associated with various applications of language models, the BSI lists examples based on “concrete proof of concepts (PoC)”:

1. Using an LLM to summarize or analyze text from external sources:
- Attackers could deliberately manipulate the result.
2. Using a chatbot to access modified web pages:
- Results of queries could be purposefully manipulated.
- The chatbot could exhibit undesirable behavior and, for example, make legally questionable or undesirable statements.
- The chatbot could motivate users to call up a (malicious) link.
- The chatbot could try to obtain sensitive information from users (e.g., credit card information).
- Attackers could (unnoticed) extract sensitive information from the chat history, e.g., if the possibility to call URLs or to display external images exists.
- The chatbot could itself call further plug-ins and thus perform undesired actions, such as accessing the email account, summarizing the last emails and extracting the information, or publishing private source code repositories.
3. Autonomous agent running locally in a docker container and accessing an LLM via API:
- Attackers could break out of the container and gain root privileges on the target system.

The BSI also provides information on measures to detect and avoid the risks.

#generativeai #indirectpromtinjection

This post has a document attached on LinkedIn. View on LinkedIn →

Discuss on LinkedIn