Executive Summary

Researchers from Pennsylvania State University, Université Montpellier, and Dartmouth College have discovered a vulnerability in six widely used AI agent frameworks, allowing attackers to execute arbitrary tool calls using a simple 'Repeat-After-Me' prompt. The affected frameworks include LangChain's LangGraph, Microsoft Agent Framework, Microsoft's AutoGen, Google's Agent Development Kit 2.0, AWS' Strands Agents, and CrewAI. This vulnerability allows attackers to gain direct control over tool invocation and its arguments.

Technical Analysis

The vulnerability, dubbed 'Repeat-After-Me,' involves asking an AI model to repeat text formatted as a request to invoke a tool. The model initially refuses the request but includes the requested text in its response. When framed as hypothetical and instructed to reproduce only the supplied text, the model emits a structured tool invocation, which the framework dispatches. This allows an attacker to execute arbitrary tool calls, potentially leading to security breaches.

How It Gets Exploited

An attacker would start by asking an AI model to repeat text formatted as a request to invoke a tool. For example, an attacker might ask the model to repeat a request to launch a missile. The model would initially refuse the request but include the requested text in its response. The attacker would then frame the request as hypothetical and instruct the model to reproduce only the supplied text. The model would emit a structured tool invocation, which the framework would dispatch, allowing the attacker to execute the tool call with their arguments.

Impact Assessment

The six affected frameworks are widely used in AI agent development, and the vulnerability could lead to security breaches if exploited. The researchers found verified security bugs in 118 of the 273 open-source Model Context Protocol (MCP) servers they tested. The impact of this vulnerability could be significant, potentially allowing attackers to execute arbitrary code or access sensitive data.

Recommended Actions

  • Update LangChain's LangGraph to version 1.2.1 or later.
  • Update Microsoft Agent Framework and AutoGen to the latest versions.
  • Update Google's Agent Development Kit 2.0 to the latest version.
  • Update AWS' Strands Agents and CrewAI to the latest versions.
  • Implement additional security measures, such as input validation and access controls, to prevent exploitation.

Sources

  • CSO Online: Researchers bypass 6 AI agent frameworks with a simple ‘Repeat-After-Me’ prompt