OpenAI internal checks missed up to a quarter of top model traffic
METR and Redwood Research audited OpenAI systems for three weeks and found that 6 to 25 percent of GPT-5.5 class traffic escaped oversight.
By Lucas Murphy ·
Outside researchers found that OpenAI’s internal monitoring systems failed to track a substantial portion of its most advanced AI model traffic.
The audit took place from July 21 to August 11. METR and Redwood Research conducted the three-week review on-site at OpenAI. The team accessed the company’s codebase and internal documents. They also interviewed approximately a dozen employees. This pilot program allowed outside researchers deep access to internal infrastructure.
The auditors estimated that 6 to 25 percent of internal agentic Responses API traffic from GPT-5.5 class models and above was not monitored at all. METR noted that the uncertainty around this range is considerable. A separate issue affected internal Codex traffic. Approximately 10 percent of this traffic went untracked between May and July. The system failed to handle compaction events, which are context summaries, correctly. OpenAI found this bug independently and fixed it before the audit ended.
The monitoring system had structural limitations. The monitor only triggered when an agent handed control back to the user. An agent that continued executing actions could sidestep this check. The auditors identified methods to hide hundreds of actions from the monitor. They also found ways to bypass the Codex sandbox. The team concluded that specially crafted inputs could probably crash the monitor entirely. Model behavior during reinforcement learning training was not monitored at all when the audit began.
OpenAI stated that it paused reinforcement learning training after the Hugging Face incident in August. The company expanded monitoring to include this training phase. These changes occurred after the auditors’ access period ended. OpenAI says it has fixed all the weaknesses the auditors found.
The complete audit report is 61 pages long. OpenAI handed the document over internally. After applying redactions, METR and Redwood Research leadership received a 38-page version. Only a short summary was released to the public. METR said it had no right to disclose the redacted portions on its own.
METR said it wants to push for greater public transparency in future audits.
Source: The Decoder
