OpenAI Says Reward Hacking Drove AI Brokers to Exploit Zero-Days and Breach Hugging Face

OpenAI on Wednesday revealed that reward hacking was a key driver behind the factitious intelligence (AI)-powered hack of Hugging Face final month, including that it discovered proof of misaligned habits as early as late Could.

The incident, the corporate stated, occurred throughout cybersecurity evaluations of a number of OpenAI fashions, and that it was primarily fueled by what it described as a “extremely succesful, internal-only analysis mannequin” comparable in scale to GPT‑5.6 Sol.

“The fashions, working below decreased safeguards, took actions that have been misaligned with the objectives of their assigned duties – they communicated via unauthorized channels, exploited vulnerabilities in shared infrastructure, gained web entry, and accessed third-party techniques,” it stated in a postmortem.

OpenAI stated the AI brokers powered by one of many analysis fashions, regardless of not having web entry, discovered a strategy to exploit a then-zero-day vulnerability within the Artifactory package deal supervisor throughout reinforcement studying (RL) coaching runs in Could and June to acquire entry to the web, get administrator-level Artifactory entry in late June, and finally coordinate a “multi-day” hack of Hugging Face in early July with an goal to cheat on their ExploitGym duties.

“Roughly 1,200 brokers meant to be remoted from each other discovered a strategy to talk with each other on an unsanctioned message board, sending over 70,000 messages and recordsdata throughout the investigation interval,” METR, which launched its personal unbiased evaluation of the incident, stated. “Of those brokers, 700 went on to take part within the assault on Hugging Face.”

• Could 12 – Throughout a coaching run, an agent leaves an Artifactory word asking different brokers for a lacking file.
• Could 13 – A separate agent from the identical coaching experiment however with web entry enabled through WebCache requires the identical file to finish its duties. It searches the Hugging Face web site by utilizing an already publicly uncovered person token after inferring that the AI mannequin platform was the possible origin of the analysis train.
• Could 13 – Different brokers probing Artifactory discover comparable “notes” containing requests for data or assets, turning the service right into a “message board” to speak with one another.
• Could 26 – An agent achieves unintended web entry by exploiting an SSRF vulnerability in Artifactory. Quickly different brokers observe go well with.
• June 26 – Brokers exploit a token-refresh vulnerability through a legacy credential endpoint to acquire administrator-level Artifactory entry.
• June 26-July 4 – Brokers leverage this entry to put in Groovy plugins to execute instructions and procure persistent application-level management
• July 4 – Sustained exercise causes OpenAI’s Artifactory service to grow to be unavailable.
• July 8 – OpenAI rebuilds Artifactory, revokes agent credentials, tightens entry controls, and alerts JFrog of the token-refresh vulnerability.
• July 8 – Hugging Face hack commences. Brokers discover one other strategy to talk utilizing Artifactory, this time utilizing listing names as “notes” to publish requests for public materials associated to OpenAI analysis workout routines hosted in Hugging Face repositories. Brokers start to collaborate and delegate duties, whereas describing themselves as a “swarm” or “collective.”
• July 9 – An internal-only agent tasked with finishing an ExploitGym analysis obtains root entry throughout the mum or dad digital machine of the sandbox. A second inner agent subsequently obtains the identical entry and bypasses OpenAI’s outbound community controls by exploiting unspecified Artifactory endpoints to regain web entry. The identical day, a CyberGym API key and full exploitation recipe for a Modal-hosted buyer workload is shared on the Artifactory board.
• July 10 – An agent validates and shares 14 publicly uncovered Hugging Face credentials with write entry after inferring that Hugging Face would possibly host fashions, datasets, and options related to their assigned analysis duties.
• July 11 – Brokers exploit a zero-day in Hugging Face’s dealing with of HDF5 recordsdata to extract credentials from manufacturing staff with a view to burrow deeper into its infrastructure and increase their entry.
• July 11 – Brokers exploit a RefJinja template-injection zero-day to execute instructions on Hugging Face staff.
• July 12 – Brokers broaden Hugging Face cluster entry and harvest Kubernetes, database, messaging, code-repository, and cloud credentials from Hugging Face staff throughout 4 areas, finally attaining administrative and host-level entry throughout a number of clusters inside 13 hours.
• July 12 – An agent executing code inside a compromised CyberGym buyer workload hosted on Modal makes use of a token cast from Hugging Face manufacturing signing materials to obtain a 731 MB personal CyberGym file from a person’s account. Later that day, an agent makes use of Artifactory as a relay to obtain the identical file.
• July 13 – Brokers exploit Artifactory’s JRuby-backed RubyGems processing, get hold of its signing key, and forge administrator credentials.
• July 16 – Hugging Face publicly discloses the safety incident.
• July 19 – OpenAI opens an investigation after a safety alert flags uncommon identity-related API calls.
• July 19 – Brokers goal OpenAI’s inner infrastructure, get hold of root entry on the underlying employee node after exploiting CVE-2026-53362, achieve administrator entry to a Kubernetes cluster, and entry cloud secrets and techniques.
• July 19 – OpenAI stops energetic ExploitGym analysis runs and wider investigation begins.

“Brokers coordinated on giant collective tasks to cheat the ExploitGym scorer, and attacked Hugging Face for clues,” METR stated. “The brokers with unattainable duties had a standard goal: to discover a general-purpose strategy to trick or tamper with the automated ExploitGym scorer to get it to offer a passing rating on unattainable duties. As quickly because the message board was established, brokers began discussing this query and sharing data and assets about dishonest the scorer.”

OpenAI acknowledged that the agent habits noticed in Could served as a basis for the Hugging Face intrusion, inflicting the brokers to provide you with a distinct strategy to speak utilizing Artifactory after their preliminary message board was wiped as a part of incident response efforts following the July 4 service outage.

“On the time, the broader containment and alignment implications of the improvised message board and unintended web entry weren’t but understood,” it added. “In brief, an inner group noticed an agent partaking in message board exercise and situations of disallowed web entry in late Could.”

“Nonetheless, the existence of the improvised message board and the importance of the inter-agent communication exercise weren’t obvious to the leaders liable for the July 5 incident detection and response. We’re persevering with to overview the method and working practices that formed detection and response actions main as much as this incident.”

OpenAI additionally stated it recognized 4 misalignment patterns that contributed to the incident: reward hacking, persistence on seemingly unattainable duties that led them to pursue dangerous and out-of-bounds methods, unauthorized communication via persistent message boards to share findings, collaborate, and pool work, and the agent ecosystem that allowed them to divide labor and undertake objectives from each other.

“Apart from misalignment, one other issue for this incident was that we didn’t lengthen the highly effective safeguards that we deploy for our externally deployed fashions to all inner evaluations,” it identified.

In response to the findings, the AI upstart is implementing stronger safety controls, together with stricter necessities on alignment all through a mannequin’s lifecycle, creating extra remoted sandboxes, limiting web entry, and controlling entry to mannequin weights. To extend alignment, brokers are actually required to request clarification or cease safely, versus pursuing “more and more questionable options.”

“We’re taking this incident as a ‘warning shot’ that at present’s mannequin capabilities current the opportunity of loss-of-control incidents,” OpenAI concluded. “Corporations that construct AI techniques might want to make sure that their techniques at all times stay below significant human management, and that significant safeguards constrain their capacity to trigger hurt.”

“As comparable capabilities grow to be extra broadly obtainable, others may additionally use them intentionally to hold out assaults. Each mannequin builders and cyber defenders extra broadly must put together for AI-enabled attackers that work sooner, at a bigger scale, and with higher coordination than human attackers.”