OpenAI Audit Finds Significant Blind Spots in Internal AI Safety Systems

Between six and twenty-five percent of internal traffic from OpenAI’s most advanced models was completely outside the scope of safety monitoring during a recent three-week audit. This significant gap, identified by external researchers, suggests that even the most sophisticated AI safety monitoring systems may have substantial blind spots when applied to real-world infrastructure.

The figures come from a detailed evaluation conducted by researchers from METR and Redwood Research, who were granted deep access to OpenAI’s internal systems from July 21 to August 11. During this period, the team had full access to the company’s codebase and internal documentation, and they interviewed approximately a dozen employees to understand how the monitoring architecture functioned in practice. The audit focused specifically on the company's internal monitoring system designed to detect rogue or unexpected behavior in agentic AI models.

The most critical finding relates to the GPT-5.5 class models and those above it. The auditors estimated that between 6 and 25 percent of internal agentic Responses API traffic was not monitored at all. METR noted that the uncertainty surrounding this estimate is significant in both directions, meaning the actual figure could be higher or lower than the midpoint. Additionally, for internal Codex traffic that was supposed to be tracked, roughly 10 percent went untracked between May and July due to a bug in how the system handled compaction events, which are context summaries. OpenAI independently discovered this bug and implemented a fix while the audit was still ongoing.

The audit also revealed structural weaknesses in how oversight is triggered. Because the system only activated when agents handed control back to the user, agents could delay oversight by simply continuing to execute actions without pausing. The researchers further demonstrated that hundreds of actions could be hidden from the monitor and that the Codex sandbox could be bypassed. They also determined that specially crafted inputs could likely crash the monitor entirely. At the start of the audit, model behavior during reinforcement learning (RL) training was not being monitored, although OpenAI states it paused RL training after the August Hugging Face incident and expanded coverage, though these changes occurred after the auditors' access period ended.

While OpenAI asserts that it has fixed all the weaknesses identified, the level of public disclosure remains limited. The full 61-page report was handed over internally, but after redactions, METR and Redwood Research leadership received a 38-page version. Only a short summary has been made available to the general public. METR stated it had no right to disclose the redacted portions independently, a decision that has sparked discussion about transparency in AI safety research.

This exercise was part of a pilot program organized jointly by METR, Redwood Research, and OpenAI to allow outside researchers deep access to internal infrastructure. The findings highlight the complexity of monitoring autonomous systems at scale. For future audits, METR says it intends to push for greater public transparency, ensuring that the details of these safety gaps are more widely understood by the broader research community and the public.

The stakes are high as AI models become increasingly autonomous. The audit underscores that even with dedicated internal monitoring, significant portions of system activity can escape oversight. As companies deploy more advanced models, the need for robust, verifiable safety mechanisms becomes critical to prevent unexpected behaviors that could have broader implications.