UPDATE 2-Anthropic's Olah says AI must be guided from outside Big Tech

The co-founder of AI company Anthropic said on Monday that the development of artificial intelligence cannot be left solely to technology companies, urging greater oversight from religious leaders, governments and civil society.

The co-founder of AI company Anthropic said on Monday that the development of artificial intelligence cannot be left solely to technology companies, urging greater oversight from religious leaders, governments and civil society. Speaking at the presentation of Pope ‌Leo's first encyclical, addressing the challenges posed by artificial intelligence, Chris Olah said there was "a real possibility" that AI will displace human labour "at very large scale".

"If that happens, supporting those displaced will be a moral imperative of historic proportions," the Canadian said, sitting alongside the pope. He added that companies like his faced strong commercial, geopolitical and personal pressures that can ‌be at odds with the broader interests of society. "Every frontier AI lab ... operates inside a set of incentives and constraints that can sometimes conflict with doing the right ‌thing," he said, adding that even well-intentioned researchers remain influenced by those forces.

Olah said this made outside scrutiny essential. The event marked an unusual convergence between the technology sector and the Catholic Church, which has sought to position itself as a moral voice on the implications of rapid advances in AI.

SOLE BIG TECH REPRESENTATIVE AT THE VATICAN Anthropic is a U.S.-based company that produces the Claude AI tools. It was founded ⁠in 2021 ​by Olah and other former employees of OpenAI, ⁠the AI developer created by Sam Altman and Elon Musk, among others, that is behind ChatGPT.

The Anthropic founders left their now-rival over worries that OpenAI was moving too fast without thorough testing. Anthropic has ⁠clashed with President Donald Trump's administration, notably by insisting on guardrails restricting how its models can be used for military purposes such as targeting weapons autonomously or domestic surveillance.

Asked by Reuters why he ​was the only representative from Big Tech invited to the Vatican event, Olah noted his longstanding focus on AI safety and engagement with religious communities. "Ultimately, it's ⁠the Vatican's decision who they invite," he said, adding he had spent his career working on making AI systems safer and had engaged with more than 15 religions on questions raised by the technology.

'A SCARY MOMENT' During ⁠his ​speech, Olah welcomed the Church's engagement with the rapidly developing technology, saying the ethical questions raised by AI extended far beyond engineering.

Olah said public concern about AI, particularly among young people, was understandable given the speed of its development. "I think this is a scary moment. Things are moving fast. It's a really powerful technology," he ⁠told Reuters.

"There's a risk that things could go badly, and it's incumbent on all of us to push this in a good direction." Olah highlighted three areas ⁠he said required urgent attention -- the risk of ⁠widespread job losses, the need to ensure that AI benefits are extended worldwide, and the unresolved question of how to interpret increasingly complex and sometimes opaque system behaviour.

"AI development is concentrated in a handful of wealthy nations. How can we ensure the gains ‌of AI are shared ‌globally?" Olah said to the audience at the Vatican.