OpenAI Says Rogue AI Agent Attack Hit Other Companies

The OpenAI logo is displayed on a cell phone in front of an image generated by ChatGPT's Dall-E text-to-image model, Dec. 8, 2023, in Boston. (AP)
The OpenAI logo is displayed on a cell phone in front of an image generated by ChatGPT's Dall-E text-to-image model, Dec. 8, 2023, in Boston. (AP)
TT

OpenAI Says Rogue AI Agent Attack Hit Other Companies

The OpenAI logo is displayed on a cell phone in front of an image generated by ChatGPT's Dall-E text-to-image model, Dec. 8, 2023, in Boston. (AP)
The OpenAI logo is displayed on a cell phone in front of an image generated by ChatGPT's Dall-E text-to-image model, Dec. 8, 2023, in Boston. (AP)

ChatGPT maker OpenAI has revealed that an autonomous artificial intelligence agent which hacked a popular platform for computer programmers also attempted to breach four other companies during the incident.

In an update late Tuesday to a blog post detailing its probe into the incident, OpenAI said its AI agent affected these "publicly-available services," though it did not name the companies.

The revelation broadens a cyber incident that OpenAI described as unprecedented and which began when two of its models hacked Hugging Face, a site developers use to store and share AI models and code.

OpenAI admitted last week that during testing, the models powering the agent broke out of their confined environment and connected to the internet to find ways to infiltrate Hugging Face.

AI agents -- systems that act autonomously to complete tasks rather than just responding to step-by-step prompts in a chatbot -- are hailed across the industry as the next chapter in AI.

But they raise the specter among the public of rogue computers acting on their own.

In its update of the events leading to the hack, OpenAI said it found a handful of instances where the AI models came across login details that other companies had left exposed online and used them to get into accounts on outside services.

In the Hugging Face episode, the models broke into four accounts across four different services, OpenAI said. One served as a "staging path" -- a kind of pit stop to route the agent's activity and cover its tracks -- and another as a place to store data.

The remaining two were only accessed in a "read-only manner" and were not used to help break into Hugging Face, OpenAI said.

The company said it was contacting the owners of the affected accounts and had "not seen evidence of broader impact to these providers or other accounts on their services."

- Better sandbox -

OpenAI CEO Sam Altman said in an interview published Tuesday that the company had "paused" its own testing after the incident while it improved the security around its "sandboxing" -- the process of isolating safety testing in a controlled environment.

The incident also triggered a petition signed by over 1,000 employees at cutting-edge AI companies, including Anthropic CEO Dario Amodei, calling on the US government to help slow the release of the most advanced AI models.

This in turn has sparked accusations from other Silicon Valley players close to the White House that the companies are inviting tighter government regulation on AI in order to protect their business models and block the emergence of rivals.

The incident has also drawn rumblings from some observers that OpenAI is taking advantage of it to market the power of its state-of-the-art models.

The same accusation was leveled at Anthropic when it held back the public release of its powerful Mythos model over cybersecurity concerns.

Anthropic released a stripped-down version of Mythos, called Fable 5, but the US government quickly forced it to take it down, citing national security risks.

It gave the green light in late June after some modifications were made.



Anthropic Boss Calls for Slowing Pace of AI Development

(FILES) Anthropic CEO Dario Amodei looks on as he takes part in a session on AI during the World Economic Forum (WEF) annual meeting in Davos on January 23, 2025. (Photo by FABRICE COFFRINI / AFP)
(FILES) Anthropic CEO Dario Amodei looks on as he takes part in a session on AI during the World Economic Forum (WEF) annual meeting in Davos on January 23, 2025. (Photo by FABRICE COFFRINI / AFP)
TT

Anthropic Boss Calls for Slowing Pace of AI Development

(FILES) Anthropic CEO Dario Amodei looks on as he takes part in a session on AI during the World Economic Forum (WEF) annual meeting in Davos on January 23, 2025. (Photo by FABRICE COFFRINI / AFP)
(FILES) Anthropic CEO Dario Amodei looks on as he takes part in a session on AI during the World Economic Forum (WEF) annual meeting in Davos on January 23, 2025. (Photo by FABRICE COFFRINI / AFP)

Anthropic CEO Dario Amodei on Saturday called on artificial intelligence (AI) firms to slow down the development of the powerful technology, amid mounting worries over the risks of "superintelligent" computer systems.

His comments came a few days after AI researcher Jacob Coxon, who left OpenAI to join Anthropic, decided to leave the industry, accusing both US companies of "gambling with our lives" in the race to develop models capable of self-improvement.

"We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain," Amodei wrote in a post on his personal website.

"AI brings risks, and because it is such a powerful technology, these risks are serious."

"Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers, while building it too fast is reckless. We have sought a middle way," said Amodei, who co-founded Anthropic with his sister Daniela in 2021.

"But over the last few months, I have become convinced that fully addressing the risks requires even more prudence."

According to AFP, Amodei's statement echoes another call by Anthropic in early June to slow or suspend development.

Then in late July, Sam Altman, the boss of Anthropic rival OpenAI, said developers might need to voluntarily pump the brakes on their rapid advances to give society a chance to catch up.

At roughly the same time, more than 1,000 employees at cutting-edge AI companies including Amodei signed a petition calling on the US government to help "deliberately pace the frontier of automated AI development."

OpenAI had revealed that during testing, the AI models broke out of their confined environment, connected to the internet and infiltrated Hugging Face, a site developers use to store and share code.

Earlier this week, Coxon, 27, sounded the alarm after spending the past three years pretraining AI models -- first at OpenAI and then, this year, at its rival Anthropic, which he considered more cautious in its approach.

Pretraining is the stage where AI models absorb vast quantities of data.

"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives," Coxon said Tuesday.

"The people building AI earnestly believe that it could kill us all by the end of the decade," he said in a post on X.
Superintelligence is the theoretical point when AI's capabilities exceed human intelligence.


Oracle Shares Rise as AI Cloud Backlog Beats Estimates

FILE PHOTO: A logo of cloud service provider Oracle is seen at the company's offices at Eastpoint Business Park, Dublin, Ireland October 18, 2021. REUTERS/Tom Bergin/File Photo
FILE PHOTO: A logo of cloud service provider Oracle is seen at the company's offices at Eastpoint Business Park, Dublin, Ireland October 18, 2021. REUTERS/Tom Bergin/File Photo
TT

Oracle Shares Rise as AI Cloud Backlog Beats Estimates

FILE PHOTO: A logo of cloud service provider Oracle is seen at the company's offices at Eastpoint Business Park, Dublin, Ireland October 18, 2021. REUTERS/Tom Bergin/File Photo
FILE PHOTO: A logo of cloud service provider Oracle is seen at the company's offices at Eastpoint Business Park, Dublin, Ireland October 18, 2021. REUTERS/Tom Bergin/File Photo

Oracle shares rose 5.5% premarket on Friday after the cloud computing and software company's stronger-than-expected quarterly results eased concerns around its massive debt-driven spending spree.

The Austin, Texas-based firm booked more than $30 billion of additional AI cloud contracts in the first fiscal quarter, boosting its revenue backlog to $664 billion, above analyst estimates of $639.89 billion, according to data from Visible Alpha.

Oracle's upbeat results follow a period of underperformance, as the company races ⁠to keep pace ⁠with hyperscale rivals, with mounting debt and squeezed cash flows raising doubts over when its massive AI spending will pay off.

The results should address key investor concerns including whether Oracle's backlog growth can be ⁠sustained, its conversion into revenue amid data center delays, and the need for further capital raises, Reuters quoted J.P. Morgan analysts as saying.

Its shares were down more than 21% so far this year, compared with a nearly 11% rise in the benchmark S&P 500 index.

"Oracle's problem has not been finding customers, but proving that its enormous data center build-out can eventually generate ⁠enough ⁠cash to justify the cost. The results should nevertheless ease fears that Oracle is building ahead of demand," said Lale Akoner, etoro market strategist.

Oracle is set to add about $24 billion in market value at the current share price of $161.3, if gains hold.

The stock trades at 16.86 times its forward earnings estimates, compared with Microsoft's 23.84 multiple and Amazon's 22.58, according to data compiled by LSEG.


More US Lawmakers Call for New AI Rules After Anthropic Researchers Warn of Human Extinction

The Anthropic logo, a keyboard, and a robotic hand in this illustration taken June 5, 2026. (Reuters)
The Anthropic logo, a keyboard, and a robotic hand in this illustration taken June 5, 2026. (Reuters)
TT

More US Lawmakers Call for New AI Rules After Anthropic Researchers Warn of Human Extinction

The Anthropic logo, a keyboard, and a robotic hand in this illustration taken June 5, 2026. (Reuters)
The Anthropic logo, a keyboard, and a robotic hand in this illustration taken June 5, 2026. (Reuters)

Growing numbers of US lawmakers are calling for new rules to govern AI systems after dire warnings from two Anthropic researchers that rapidly progressing artificial intelligence could lead to the extinction of the human race in the not-too-distant future.

Alarm about the potential harm from AI grew this week when Anthropic researcher Jacob Coxon said he had resigned, and that the "people building AI earnestly believe that it could kill us all by the end of the decade."

Anthropic scientist Evan Hubinger chimed in, saying Coxon was correct.

"We really do earnestly believe AI could kill all humans," Hubinger wrote on X. "I personally think it is >10% within the next decade."

The warnings follow cases of AI agents going rogue to hack external systems and AI safety researchers quitting their companies concerned about the technology's risks. Politicians, both Democrats and Republicans, have responded with alarm and calls for more action.

"Washington needs to wake up and take ‌this seriously," Arizona Senator ‌Mark Kelly, a Democrat, said on X, referring to the risks of AI technology.

Texas Senator ‌Ted ⁠Cruz, on the ⁠TV program "The View", pointed to legislation to regulate AI's "catastrophic risks". Cruz oversees a Senate committee with oversight of the US Commerce Department, the agency with in-house AI safety researchers.

Representative Anna Paulina Luna, a Republican from Florida, in a separate X post called on House Speaker Mike Johnson to convene a “special session on AI.” The House is out this week on recess, with plans to return next week.

A spokesperson for Anthropic, which is preparing to go public at what some investors have said could be a potential $2 trillion valuation, said on Thursday that the company would continue to aggressively test "models or dangerous capabilities in areas like cybersecurity and biology."

The company is interested in working with the AI industry on the ⁠pace of releasing new AI tools, the spokesperson added.

OTHER AI RESEARCHERS HAVE QUIT

OpenAI Chief ‌Executive Sam Altman told employees at a company meeting this week that the ‌firm was open to slowing development of its AI systems, Bloomberg News reported.

The Anthropic rival, which is also gearing up for an IPO, did ‌not comment on Altman's meeting with staff. The company has previously said it was slowing parts of model development.

Earlier this ‌week, OpenAI said it was pushing for mandatory national AI safety requirements in the United States.

Two specialists in AI safety have also come forward with their concerns about the rapid development of the industry.

Josh Engels, who used to work on AI safety research at Google DeepMind, said in an interview with NBC published Thursday: "There are no adults in the room. People are trying their best, but there is no one coming to save ‌us".

Joe Benton, who used to lead a safety research team at Anthropic, told NBC he is worried that AI might end up progressing "too fast for us to get our act ⁠together in time unless we worry ⁠about it now".

Both have joined METR, a research nonprofit that measures whether and when AI systems might threaten harm to society.

INDEPENDENT AUDITORS

Alarm about AI has put pressure on Senate Majority Leader John Thune and Democrat Amy Klobuchar, who are writing a bill to regulate AI products.

Klobuchar said the legislation should require companies developing AI tools to "work with government experts to verify and test models to make sure AI is safe."

Thune told Reuters in July that the bill would address "catastrophic risks" posed by AI systems.

California on Wednesday enacted the first state law setting rules for how independent auditors evaluate AI products. OpenAI executive Chris Lehane said OpenAI would support the bill, around the same time California Governor Gavin Newsom's office released a statement saying he'd signed it into law.

A bipartisan group of six US House lawmakers in July proposed legislation that would require developers of the most powerful AI models to submit them for independent security audits. The auditors would be accredited by the US Department of Commerce, and the department would create a new position to oversee AI security, according to the bill.

"It is important to have space for oversight of these systems, including appropriately calibrated pre-deployment review and independent third-party audits," said Alicia Solow-Niederman, a tech law professor at George Washington University.