AI Experts Ready ‘Humanity’s Last Exam’ to Stump Powerful Tech

Figurines with computers and smartphones are seen in front of the words "Artificial Intelligence AI" in this illustration taken, February 19, 2024. (Reuters)
Figurines with computers and smartphones are seen in front of the words "Artificial Intelligence AI" in this illustration taken, February 19, 2024. (Reuters)
TT
20

AI Experts Ready ‘Humanity’s Last Exam’ to Stump Powerful Tech

Figurines with computers and smartphones are seen in front of the words "Artificial Intelligence AI" in this illustration taken, February 19, 2024. (Reuters)
Figurines with computers and smartphones are seen in front of the words "Artificial Intelligence AI" in this illustration taken, February 19, 2024. (Reuters)

A team of technology experts issued a global call on Monday seeking the toughest questions to pose to artificial intelligence systems, which increasingly have handled popular benchmark tests like child's play.

Dubbed "Humanity's Last Exam," the project seeks to determine when expert-level AI has arrived. It aims to stay relevant even as capabilities advance in future years, according to the organizers, a non-profit called the Center for AI Safety (CAIS) and the startup Scale AI.

The call comes days after the maker of ChatGPT previewed a new model, known as OpenAI o1, which "destroyed the most popular reasoning benchmarks," said Dan Hendrycks, executive director of CAIS and an advisor to Elon Musk's xAI startup.

Hendrycks co-authored two 2021 papers that proposed tests of AI systems that are now widely used, one quizzing them on undergraduate-level knowledge of topics like US history, the other probing models' ability to reason through competition-level math. The undergraduate-style test has more downloads from the online AI hub Hugging Face than any such dataset.

At the time of those papers, AI was giving almost random answers to questions on the exams. "They're now crushed," Hendrycks told Reuters.

As one example, the Claude models from the AI lab Anthropic have gone from scoring about 77% on the undergraduate-level test in 2023, to nearly 89% a year later, according to a prominent capabilities leaderboard.

These common benchmarks have less meaning as a result.

AI has appeared to score poorly on lesser-used tests involving plan formulation and visual pattern-recognition puzzles, according to Stanford University’s AI Index Report from April. OpenAI o1 scored around 21% on one version of the pattern-recognition ARC-AGI test, for instance, the ARC organizers said on Friday.

Some AI researchers argue that results like this show planning and abstract reasoning to be better measures of intelligence, though Hendrycks said the visual aspect of ARC makes it less suited to assessing language models. "Humanity’s Last Exam" will require abstract reasoning, he said.

Answers from common benchmarks may also have ended up in data used to train AI systems, industry observers have said. Hendrycks said some questions on "Humanity's Last Exam" will remain private to make sure AI systems' answers are not from memorization.

The exam will include at least 1,000 crowd-sourced questions due November 1 that are hard for non-experts to answer. These will undergo peer review, with winning submissions offered co-authorship and up to $5,000 prizes sponsored by Scale AI.

"We desperately need harder tests for expert-level models to measure the rapid progress of AI," said Alexandr Wang, Scale's CEO.

One restriction: the organizers want no questions about weapons, which some say would be too dangerous for AI to study.



Musk’s Social Media Firm X Bought by His AI Company, Valued at $33 Billion

 xAI and X logos are seen in this illustration taken, March 28, 2025. (Reuters)
xAI and X logos are seen in this illustration taken, March 28, 2025. (Reuters)
TT
20

Musk’s Social Media Firm X Bought by His AI Company, Valued at $33 Billion

 xAI and X logos are seen in this illustration taken, March 28, 2025. (Reuters)
xAI and X logos are seen in this illustration taken, March 28, 2025. (Reuters)

Elon Musk's xAI has acquired X in a deal that values the social media platform at $33 billion and allows the value of his artificial intelligence firm to be shared with his co-investors in the company formerly known as Twitter.

The deal could also help xAI's ability to train its chatbot known as Grok.

"xAI and X's futures are intertwined," Musk, who also heads automaker Tesla and SpaceX, wrote in a post on X: "Today, we officially take the step to combine the data, models, compute, distribution and talent."

He said the combination values "xAI at $80 billion and X at $33 billion ($45B less $12B debt)".

Representatives for X and xAI did not immediately respond to requests for comment. Much of the deal's specifics remain unclear, such as how X's leaders would be integrated in the new firm or whether there would be regulatory scrutiny.

Musk, the world's wealthiest man, is also a close ally of US President Donald Trump and heads the Department of Government Efficiency.

D.A. Davidson analyst Gil Luria said the price tag for X of $45 billion when debt was included was not a coincidence. "It is $1 billion higher than the take-private transaction for Twitter in 2022."

An investor in xAI who declined to be identified said they were not surprised by the deal, viewing it as Musk consolidating his leadership and management at his own companies.

Musk did not ask investors for approval but told them that the two companies had been collaborating closely and the deal would drive deeper integration with Grok, the investor said.

OPENAI RIVALRY

Musk's xAI startup was launched less than two years ago and recently raised $10 billion in a funding round that valued the company at $75 billion, according to a media report.

It competes with the likes of Microsoft-backed OpenAI as well as with Chinese startup DeepSeek.

In February, Musk, 53, made a $97.4 billion bid with a consortium for OpenAI, which was rejected and he has sued to prevent the ChatGPT maker from converting from a non-profit to a for-profit business. A judge this month denied Musk's request for a preliminary injunction that would prevent the changeover.

As competition in AI intensifies, xAI has been ramping up its data center capacity to train more advanced models, and its supercomputer cluster in Memphis, Tennessee, called "Colossus," is touted as the largest in the world.

xAI introduced Grok-3, the latest iteration of its chatbot, in February.

The X platform could serve to further distribute xAI products, while also providing a real-time feed of users' musings, screenshots and other data.

After buying Twitter, Musk gutted the company's workforce, prompting advertisers to flee the platform and a rapid decline in revenue. Recently, brands have been returning to X as Musk's influence in the Trump administration grows.

The seven banks that extended $13 billion in loans to Musk to buy X kept the debt on their books for two years until they were able to sell it all at once last month, according to a source familiar with the transactions.

This was made possible after a surge in investor interest for exposure to AI companies along with X's improved operating performance over the previous two quarters, among other factors, according to two people familiar with the matter.

After the merger, investors who bought the debt from the banks will profit, said Espen Robak, founder of Pluris Valuation Advisors, which specializes in illiquid assets. "For sure the debt is worth more now, if not fully paid off."

Separately, a US judge on Friday rejected a bid by Musk to dismiss a lawsuit claiming he had defrauded former Twitter shareholders by waiting too long to disclose his initial investment in the company.