Red Teaming

When to use

"Use when a system's failure has high consequences. Organize an adversarial team to find vulnerabilities through simulated attack. The red team's job is to break the system; the blue team's job is to defend it."

Example

"Before deploying an AI agent with file access, a red team attempts prompt injection, social engineering of the agent, privilege escalation through chained tool calls, and data exfiltration through seemingly benign queries. The red team finds that the agent will leak file contents if asked to summarize them in a specific format."

Common misuse

"Red teaming as a checkbox — the red team is instructed to find issues but not given enough time, access, or incentive to find real vulnerabilities. A red team that never breaks anything is either perfect (unlikely) or constrained (likely)."

Also known as

Related models