Red teaming is systematic, deliberately adversarial testing of a model with the goal of finding its weaknesses before real attackers or users do. The term comes from the military and from cybersecurity; in an AI context it means the structured search for inputs that push a model into breaking its alignment – into dangerous instructions, leaking sensitive data, discriminatory outputs or convincing hallucinations. One distinguishes manual red teaming, where experts from various fields exploit domain knowledge, language variants and contextual tricks, from automated red teaming, where attack prompts are generated by another model and their success is scored by a classifier, allowing orders of magnitude more scenarios to be explored. With agents, the whole tool chain is tested as well, especially resilience against indirect prompt injection. The cases found are not used only for patches: they flow back into training as fine-tuning data and into a regression test suite, so that the same weakness does not resurface in the next version. Red teaming is now a standard part of shipping a model, and its results are usually published in system cards.
It is like a bank hiring a burglar to try to crack its own vault. Not because it trusts him, but because it is better to hear about a weak hinge from someone who will report it than from someone who walks off with the gold. The burglar does not only try the door. He tries a bogus phone call to reception, the cleaner, a different language version of the security policy, and the fire escape. Every gap he finds, the bank closes and adds to a checklist it reviews after every refurbishment.