OpenAI Details GPT-Red: An Internal Automated Red-Teaming Model That Beat Human Red-Teamers 84% To 13% On Prompt Injection
Evolving story · 3 updatesOpenAI GPT-Red safety systemTimeline →OpenAI's GPT-Red model outperformed human red-teamers in a prompt injection test, with a success rate of 84% compared to 13%. The model was trained using self-play reinforcement learning against a population of defender LLMs.
- GPT-Red outperformed human red-teamers in a prompt injection test
- The model was trained using self-play reinforcement learning
- GPT-Red discovered a novel attack class called 'Fake Chain-of-Thought'
- The model helped reduce the failure rate of GPT-5.6 Sol by a factor of six
OpenAI has developed an internal automated red-teaming model called GPT-Red, which has demonstrated impressive capabilities in identifying vulnerabilities in language models. The model was trained using self-play reinforcement learning, where it played against a population of defender LLMs. This approach allowed GPT-Red to learn and improve its attack strategies.
The results of the test were striking, with GPT-Red outperforming human red-teamers by a significant margin. The model's success rate of 84% compared to the human team's 13% highlights the potential of automated red-teaming in AI security testing.
One of the key findings of the test was the discovery of a novel attack class called "Fake Chain-of-Thought". This attack exploits the way language models process and generate text, and it has significant implications for the development of more secure AI systems.
OpenAI has also reported that GPT-Red has helped to reduce the failure rate of its GPT-5.6 Sol model by a factor of six on a direct injection benchmark. However, the company acknowledges that there is still work to be done, particularly in addressing multi-turn and image-based attacks.
The development of GPT-Red is an important step forward in the field of AI security, and it has significant implications for the development of more secure and robust language models.
Improved AI security testing
More secure language models
Potential for increased investment in AI security
Advancements in AI security
- red-teaming
- The practice of testing a system's defenses by simulating an attack
AI ResearchKimi K3, and what we can still learn from the pelican benchmark
SECNAV Hung Cao unveils plan for Navy to ‘out-learn and out-fight any adversary’ with AI - DefenseScoop
AI ResearchKimi's open model K3 nears GPT-5.6 Sol and Fable 5 while signaling the end of super cheap Chinese AI
How artificial intelligence is changing the mental health space - KTEN
Google DeepMind Is Preparing For AI's Biggest Risk. Biosecurity Is Becoming A New Battleground. - International Business Times
Agentic Commerce Is Coming—Will the Legal System Be Ready? - Consumer Finance Monitor
The emergence of agentic commerce, which involves AI-driven transactions, is likely to challenge the legal system. Experts are questioning whether the current legal framework is prepared to handle the implications of this new technology.
This AI Can Now Log Into Websites for You. But Should You Let It? - inc.com
A new AI-powered login tool can automatically log into websites, but experts question its safety and usability.
Nobody is Immune From the Nvidia GPU Crunch - The Information
Nvidia's GPU shortage is impacting various tech companies, causing delays and disruptions in the industry.
About 300 Netflix Programs Used Generative AI This Year, Company Reveals - Variety
Netflix reveals that about 300 of its programs have utilized generative AI this year, highlighting the company's growing reliance on the technology.
AI-powered real estate services promise homebuyers thousands in savings, but experts warn against replacing agents - News4JAX
AI-powered real estate services promise homebuyers thousands in savings, but experts warn against replacing human agents.
AI Has Upended Chip Stocks. This Chart Shows How Much. - Barron's
A recent chart shows how AI has significantly affected the performance of chip stocks, highlighting a notable shift in the market.