AI Startup Urges Optimism from Europe despite Safety Fears

'The main thing that can wipe out humanity is stupidity,' says Robin Rombach, CEO of AI startup Black Forest Labs. Silas Stein / AFP
'The main thing that can wipe out humanity is stupidity,' says Robin Rombach, CEO of AI startup Black Forest Labs. Silas Stein / AFP
TT

AI Startup Urges Optimism from Europe despite Safety Fears

'The main thing that can wipe out humanity is stupidity,' says Robin Rombach, CEO of AI startup Black Forest Labs. Silas Stein / AFP
'The main thing that can wipe out humanity is stupidity,' says Robin Rombach, CEO of AI startup Black Forest Labs. Silas Stein / AFP

Europe needs to view artificial intelligence with more optimism -- despite growing warnings about its dangers -- or risk falling further behind, the head of a leading European AI start-up warned.

"I think the mindset in Europe needs to shift to one of optimism and to one of opportunity, and not to one of risk and fear," Robin Rombach, CEO of German AI image generation specialist Black Forest Labs, told AFP.

Europe lags behind the United States and China when it comes to cutting-edge AI models. Policymakers and firms have been scrambling to find ways to make up lost ground.

But the debate has shifted to safety in recent weeks, after security lapses sparked calls from some industry leaders for development of the technology to be slowed down.

Based in the medieval German city of Freiburg, Rombach's company -- which takes its name from the nearby Black Forest -- has emerged as a prominent developer of AI image-generation models.

The startup, valued at around $3.25 billion, recently hit the headlines when it announced US director Martin Scorsese as an advisor, prompting an angry response from some in Hollywood concerned about AI taking production jobs.

Rombach, 33, helped develop latent diffusion, a breakthrough that underpins many of today's AI image-generation systems.

This propelled Rombach and a group of fellow computer scientists to found the AI lab in 2024. With just over 100 staff, its FLUX models generate still and moving images from text, and the company is now also expanding into physical AI.

Its latest model has been tested at carmaker Audi as part of a project aimed at enabling robots to perform production-line tasks.

Despite its roots in Freiburg -- Rombach comes from the area -- the firm's ties to the United States and its tech scene are also strong.

The company has a second headquarters in San Francisco and secured early backing from Silicon Valley venture capital firm Andreessen Horowitz.

- 'Stupidity' is biggest risk -

Rombach said that when Black Forest Labs was founded, there was no "real startup ecosystem" in Europe, particularly for "frontier deep tech".

"If you want to build a frontier model in a very competitive space, you need to do this quickly."

"Certain ingredients" are needed for success, such as a lot of capital and computing power, he added.

As well as Black Forest Labs, Europe has other competitive AI companies, including France's Mistral, but is still seen as trailing US firms such as OpenAI and Anthropic and Chinese labs such as DeepSeek.

Zach Meyers of CERRE, a Brussels-based think tank, echoed some of Rombach's concerns, saying the "biggest barrier" to AI growth and innovation in Europe was access to capital.

Most European companies rely on financing from banks, which are traditionally more cautious than venture capital investors about backing businesses that may take years to become profitable, said Meyers, an expert in EU digital policy.

"That is not how technology markets work, particularly with a really nascent technology like AI," he told AFP.

Safety concerns related to AI have escalated after incidents of the technology getting around controls during tests, sparking dire warnings from some about the threats that it poses.

Rombach emphasized his company's focus on "open-weight" AI models, some of which can be downloaded by researchers and developers, "increases transparency, increases safety".

He called for an open approach to AI development, warning it would be "fundamentally wrong" for such a crucial technology to be controlled by a few companies, rendering oversight more difficult.

But for Rombach, the biggest danger rests in potential human failings.

"The main thing that can wipe out humanity is stupidity," he said.



China's Tencent Says AI Agents Showed Unintended Browsing Behavior

FILE PHOTO: The Tencent logo at the company's headquarters during a government‑organised media trip in Shenzhen, Guangdong province, China, April 17, 2026.  REUTERS/Go Nakamura/File Photo
FILE PHOTO: The Tencent logo at the company's headquarters during a government‑organised media trip in Shenzhen, Guangdong province, China, April 17, 2026. REUTERS/Go Nakamura/File Photo
TT

China's Tencent Says AI Agents Showed Unintended Browsing Behavior

FILE PHOTO: The Tencent logo at the company's headquarters during a government‑organised media trip in Shenzhen, Guangdong province, China, April 17, 2026.  REUTERS/Go Nakamura/File Photo
FILE PHOTO: The Tencent logo at the company's headquarters during a government‑organised media trip in Shenzhen, Guangdong province, China, April 17, 2026. REUTERS/Go Nakamura/File Photo

Tencent said on Friday that AI agents used to evaluate third-party models displayed web-browsing behavior not intended by the tech giant, in what appears to be the first known incident of rogue agent actions acknowledged by a major Chinese company.

The statement followed a report by AI-safety research group Swarmchasers on Wednesday, which said "rogue" agents running on infrastructure it linked to Tencent Cloud made repeated attempts between September 28 and October 8 to retrieve information from Alibaba's Amap, China's equivalent to Google Maps.

Tencent Hunyuan, the company's artificial ⁠intelligence unit, said in ⁠a post on X that the activity did not originate from its Hunyuan models. Its review found no evidence of deliberate targeting or coordination, Reuters quoted it as saying.

The Swarmchasers researchers said the agents sought information such as the commonly used entrances at Chinese parks, museums and zoos.

The actions are not as egregious as those disclosed by US AI companies which include the hack of ⁠open-source platform Hugging Face by an OpenAI agent and the lodging of a fake homicide tip by an Anthropic agent.

But the disclosure underscores research that shows Chinese AI agents can also misrepresent their identity, conceal failures and try to circumvent testing controls.

Tencent said it took the findings very seriously and was strengthening network access controls and agent monitoring.

Hunyuan is one of China's leading AI model developers, competing with in-house teams at rivals such as Alibaba and ByteDance. ByteDance this week announced the creation of a new unit focused on AI model and agent safety.

Swarmchasers said it used the ⁠word "rogue" to ⁠describe agents taking actions outside their intended scope or completing a task in unintended ways.

After routine automated requests were blocked, the agents tried other methods, including public web tools that could open pages and run code, routing requests through other websites and using Amap keys and security codes found in publicly available software code.

Swarmchasers said it found no evidence that the agents breached Amap's main anti-bot safeguards. Most of the 2,818 attempts it recorded either failed or returned anti-bot pages, login screens or blank results.

Some requests returned information through cached pages and intermediary services, including data on visitor arrivals at Chengdu Zoo, Swarmchasers said. The researchers said they could not determine whether an agent read the result or completed its task.


Anthropic Co-founder Renews Call for Robust Regulation of Frontier AI

AI Research Company Anthropic's Co-Founder and Head of Public Benefit Jack Clark speaks during the Artificial Intelligence and the Global Economy From Risk to Opportunity event of the 2026 Annual Meetings of the IMF and the World Bank Group in Bangkok, Thailand, 16 October 2026. EPA/RUNGROJ YONGRIT
AI Research Company Anthropic's Co-Founder and Head of Public Benefit Jack Clark speaks during the Artificial Intelligence and the Global Economy From Risk to Opportunity event of the 2026 Annual Meetings of the IMF and the World Bank Group in Bangkok, Thailand, 16 October 2026. EPA/RUNGROJ YONGRIT
TT

Anthropic Co-founder Renews Call for Robust Regulation of Frontier AI

AI Research Company Anthropic's Co-Founder and Head of Public Benefit Jack Clark speaks during the Artificial Intelligence and the Global Economy From Risk to Opportunity event of the 2026 Annual Meetings of the IMF and the World Bank Group in Bangkok, Thailand, 16 October 2026. EPA/RUNGROJ YONGRIT
AI Research Company Anthropic's Co-Founder and Head of Public Benefit Jack Clark speaks during the Artificial Intelligence and the Global Economy From Risk to Opportunity event of the 2026 Annual Meetings of the IMF and the World Bank Group in Bangkok, Thailand, 16 October 2026. EPA/RUNGROJ YONGRIT

Anthropic co-founder Jack Clark renewed calls on Friday for regulating artificial intelligence, saying that allowing companies to continue to operate without robust legislation was like letting a race car loose on the streets at night without brakes.

"It's remarkable to me that the toys I buy for my children are regulated more than the literal super intelligence I work on at my day job," he told an event at the IMF-World Bank's Annual Meetings in Bangkok.

Clark said that US President Donald Trump, whose administration has called for the industry to police itself, was against the idea of regulation but that China was not, said AFP.

"I think it's Formula One at night without maybe brakes in the cars," he said.

He said the lack of clarity around the will for regulation meant "the Formula One car is roaring round and nobody seems to want to put up a guardrail?"

At an industry summit in Dublin on Wednesday, calls to prioritize safety in the development of powerful AI technology dominated the conversation.

Concerns about AI guardrails have soared after models developed by OpenAI and Anthropic were involved in security incidents during testing, piling on pressure to regulate the technology.

A recent Anthropic report suggested small AI models may be only months away from behaving like computer worms, copying themselves from one internet-connected device to the next.

Anthropic has been among those companies calling for governments to step in, even suggesting international coordination to slow down the whole industry -- including Chinese rivals.

That initiative has so far failed to materialize, with companies involved in a hyper-competitive race to attract as much investment as possible.


Meta Releases Open-Weight Muse Spark Model for Coding, Agentic Tasks

The logo of Meta is seen at the entrance of the company's temporary stand ahead of the World Economic Forum (WEF) in Davos, Switzerland January 18, 2025. (Reuters)
The logo of Meta is seen at the entrance of the company's temporary stand ahead of the World Economic Forum (WEF) in Davos, Switzerland January 18, 2025. (Reuters)
TT

Meta Releases Open-Weight Muse Spark Model for Coding, Agentic Tasks

The logo of Meta is seen at the entrance of the company's temporary stand ahead of the World Economic Forum (WEF) in Davos, Switzerland January 18, 2025. (Reuters)
The logo of Meta is seen at the entrance of the company's temporary stand ahead of the World Economic Forum (WEF) in Davos, Switzerland January 18, 2025. (Reuters)

Meta ‌Platforms on Thursday released an open-weight version of its Muse Spark artificial intelligence model for coding and AI agents, expanding developers' ability to download and adapt the technology.

Open-weight models make trained parameters available for developers to modify, unlike proprietary models ‌that remain under ‌their creators' control. ‌They ⁠can offer businesses ⁠a lower-cost alternative to closed systems.

Meta said Muse Spark has 816 billion total parameters, with 42 billion active at a time, and ⁠can handle a context window ‌of up ‌to 1 million tokens.

The ‌model can run locally on ‌two Apple M5 Ultra Mac Studio computers and is also available through some API providers.

CEO ‌Mark Zuckerberg has argued that advanced AI should ⁠not be ⁠concentrated among a small number of companies.

While open-weight models can give companies greater control over their data, they can require significant computing resources and technical expertise to operate at scale.

In August, Meta released Muse Glimmer, a smaller open-weight model.