Most Sunday nights, Dario Amodei heads over to his younger sister Daniela’s house to play their favourite video game, Final Fantasy VII Remake, set in a dystopian world where the goal is to stop an all-powerful corporation from plundering the planet’s resources.
Then, on Mondays, they show up at the headquarters of Anthropic, the $60 billion rival to OpenAI they co-founded, to develop artificial intelligence (AI), which they believe will soon replace swathes of human work and, in the process, probably transform their start-up into one of the megacorporations of tomorrow.
Amodei, 42, has predicted that “superintelligence”, which he defines as AIs that are more capable than Nobel prizewinners in most fields, could arrive as soon as next year.
“AI is going to be better than all of us at everything,” he said. If he’s right, we will have to figure out a new way to orient society around a reality where no human will ever be smarter than a machine. It will be a transition not unlike the industrial revolution, he said, when machines supplanted human brawn — and reordered the western world in the process.
“We probably, at some point, need to work out another new way to do things, and we have to do it very quickly.” Part of the solution, he argued, will probably include some form of universal basic income: government hand-outs to underemployed humans. “It’s only the beginning of the solution,” he said. “It’s not the end of the solution, because a job isn’t only money, it’s a way of structuring society.”
Anthropic, based in a glass and steel tower in San Francisco that once served as the headquarters of the business software giant Slack, has set out its stall as the yin to OpenAI’s yang. The Amodei siblings grew up in San Francisco and took diverging paths. Dario was a computational biologist who got drawn out of academia and into AI by his conviction that machines could help humans crack the most difficult problems in biology.
Daniela dabbled in politics as a Congressional staffer before joining Stripe, the payments giant, and then running safety at OpenAI, where Dario led the development of the language model that powered ChatGPT. In 2020, they, and five other senior OpenAI leaders, all left together. Why?
The Sunday Times interviewed several of Anthropic’s founding team. They were all studiously vague about the exact reason for their mass exit, but what is clear is that they disagreed with how Sam Altman, OpenAI’s billionaire boss, was running the company, and in particular how he was, in their eyes, straying from the original mission of developing AI safely, for the good of humanity.
“I tried for a very long time to point out concerns, to say, ‘This isn’t the way we should do things. This is how I think we should do things,’” Dario said. “At the end of the day, that’s just not that effective.”
Anthropic, which originally billed itself as an “AI Safety Lab”, has made astounding progress in just over four years. Its popular chatbot, Claude, is as good or better than ChatGPT across a number of industry benchmarks. The company has reeled in billions in funding from the likes of Amazon and Google, and is close to sealing a $3.5 billion financing round that would value it at $61.5 billion. Its ranks have tripled in a year to more than 1,000 people. “Two-thirds of the company didn’t work here a year ago,” Daniela said.
Exact sales figures are hard to come by because Anthropic is private, but it is losing billions. It is estimated to have brought in at least $400 million last year and is targeting a ten-fold jump this year to $4 billion as people, businesses and governments integrate Claude into their operations. The company offers everything from a free basic service to an $18-a-month “pro” tier to “pay-as-you-go” plans for companies that build apps atop their models.
Jack Clark, a Brit and Anthropic’s head of policy, predicted that tools such as its “computer use” agent, which can take control of a screen cursor and autonomously complete tasks, will mark another step change.
“We’re kind of gearing up for this next year to be a year where you have ‘Oh shit’ moments,” he said. “It feels like ChatGPT happened, and then people were like, ‘OK.’ But the frog’s been boiling for a couple of years with no one really noticing and what you need is another breakthrough in user experience. Stuff like this computer use thing would be an example that might trigger it.”
Before Anthropic even had a product to sell, it hired a safety team to game out the worst-possible outcomes — AI being used to launch cyberattacks, to build a dirty bomb, to enslave humanity — and how to avoid them. That safety focus meant that Anthropic was seen by many as overly obsessed with “doomerism”: the notion that AI was inevitably going to go terribly wrong for all of us.
That perception was not helped by Anthropic’s association with, and support by, many people in the “effective altruism” (EA) community. This movement shares a cultish, hyper-rationalist worldview that is focused on doing maximal good in the world, as measured in cold, hard data, such as number of lives saved.
EAs became particularly focused on the existential dangers of AI. The most famous EA, Sam Bankman-Fried, invested $500 million in Anthropic in the early days. Daniela’s husband, Holden Karnofsky, co-founded Open Philanthropy, an EA investment group. “I would not use that term [EA] for myself,” Daniela said. “I think there are a lot of organisations who do really cool work in that area.”
Yet coupled with that safety obsession was a conviction, starting with Dario, that superintelligent AI was going to arrive far faster than virtually anyone thought. That belief revolved around “scaling laws”: a theory that AI model improvement is directly correlated to the data and computing power you feed them. The more they get, the better they will get. Even a few years ago, the theory was hotly debated.
As Anthropic, Google, OpenAI, Elon Musk’s Grok and even China’s DeepSeek leapfrog each other almost weekly with new models, that conviction has become much more widely held. Dario said: “I’ve been watching that curve for ten years, and every year there’s a point where it looks like it’s going to slow down. You train the model and it looks like it’s levelled off because you weren’t using the right kind of data. There’s problems at every stage. And at every stage so far, it’s kind of cleared. We always find a way around it.”
Those dual convictions — that scaling laws would soon deliver superintelligence, and that therefore safety was critical — led to the birth of Anthropic. The question is: where does it all lead? Clark said he was “unnerved” by the sluggishness of governments’ response.
“The next time we go for a big growth in public attention, which we are expecting will happen this year, it will look increasingly ridiculous to people that this is like a complete Wild West.”
It is impossible to avoid the apparent contradiction that safety-obsessed Anthropic is also working so hard to bring forward that unsettling future as fast as possible.
Dario, however, has an answer. He reckons that if Anthropic builds its AI in a more thoughtful, better way, it will create a “race to the top”. For example, Claude is imbued with what he has dubbed “Constitutional AI”, a mix of rules from everything from the UN Declaration of Human Rights to Apple’s terms of service, that together serve as a guidebook for the bot’s behaviour. A second AI monitors Claude to ensure it is keeping to the constitution rules.
Anthropic has also published a “Responsible Scaling Policy” that provides a safety framework, including shutting down training if models exhibit an unacceptable risk for “certain catastrophic behaviours”.
Yet Anthropic is a for-profit corporation with billions of dollars of investor capital at stake that is locked in a race for supremacy of a powerful technology. Surely, the pressure to cut some safety corners must be immense? Clark disagreed. Pumping out cutting-edge models that the market will pay for is, itself, a clear message that their way is working.
“People are like, ‘How do we help with this amazing policy mission?’ My response is: be a successful company. Because it’s extremely hard to have legitimacy in policy if you’re an unsuccessful company.”
It’s not all doomerism. Dario Amodei recently penned a 10,000-word essay called Machines of Loving Grace, in which he focused on a simple idea: “What if it all goes right?”
He paints a fantastical future in which, very soon, we will have access to superintelligent AIs, or as Amodei dubs them, “a country of geniuses in a data centre”. What might we do with them? Compress a century of scientific progress into a decade. Double our lifespan. Cure nearly all infectious diseases. Solve Alzheimer’s and cancer.
Amodei explained: “There’s nothing about these biological problems that’s beyond humans’ ability to understand. There’s just more facts, right? Machines should be able to do a better job than humans of sorting through this complexity.” He added: “It’s a country of geniuses versus a clump of cells. I think AI can out-think a bunch of cells.”
Ethan Mollick, a business professor at Pennsylvania University and author of the book Co-Intelligence, said we are still in the foothills of adopting AI — and the disruption that it will unleash.
“Adoption of this technology is historically huge and more is coming. Every controlled study we do shows large-scale effects on performance from using AI systems at the individual level. It’s good at medicine. It’s good in education. There’s a lot of stuff.”
He added, however, that inertia is a powerful force, so that even if “superintelligence” arrived tomorrow, it would still take years and years to percolate through the economy.
The reality is, of course, no one knows. Indeed, few are as conversant in the unpredictability of technology as Mike Krieger. Before he joined Anthropic as chief product officer last year, Krieger co-founded Instagram. His photo-sharing app became a social media force used by more than a billion people and has been implicated in an array of social ills, from teenage depression to the loneliness epidemic.
What lessons, I ask, did he draw from the Instagram rollercoaster? “Understanding that there are going to be outliers in usage early, and that they can teach you something about where things get to eventually,” he said. “I remember looking at ‘time spent’ charts on Instagram when the average, at the time, was 15 minutes a day. But the outliers were 60 minutes.”
Today, people spend more than two hours a day on social media. Yet, two decades into the social media era, executives and regulators are still struggling to catch up with a technology that has become, for many, the centre of their lives.
With AI, the effects are potentially more profound, and the adoption curve is far steeper. Amodei said: “There’s danger at every turn.”