Anthropic Is Trying to Win the AI Race Without Losing Its Soul

Anthropic Chief Executive Officer Dario Amodei received a message on Slack one day in mid- February: Senior members of his company’s safety team were concerned that, without the right safeguards, the artificial intelligence model they were about to release to the public could be used to help create bioweapons.

Some content could not be imported from the original document. View content ↗

This startling revelation came at a time when pressure was already ratcheting up on Amodei. The model in question, Claude 3.7 Sonnet, was only days away from release, as Anthropic PBC sprinted to keep pace with competitors who were rushing their own models to market. At the same time, the 42-year-old Amodei, a bespectacled ringlet-haired man who’d spent the early years of his career in academic labs carefully extracting the eyeballs of dead salamanders, was in the process of closing a multibillion-dollar investment round valuing Anthropic at more than $60 billion.

It was hardly an opportune time to tap the brakes, but that’s effectively what Amodei had promised to do when he helped start Anthropic four years earlier. More than most other leaders in the AI industry, Amodei has argued that the technology he’s building comes with significant risks.

At the time, a group of Anthropic staffers known as the “frontier red team” were at a security conference in Santa Cruz, California. They holed up in a hotel room to deal with the issue, along with outside experts from biosecurity consulting company Gryphon Scientific LLC who were also attending the event. With Amodei participating via Google Meet, the group ran the model through a series of tests. The Anthropic staffers told him what most bosses in this situation would presumably want to hear. They’d pull an all-nighter or two to assess the issue and stick to the release schedule. “They were saying, ‘We can stay up all night. We can get this done in time. We can get no sleep and do this in 72 hours,’” Amodei says, at Anthropic’s headquarters in San Francisco. “I’m like, ‘If you do that, you will not do a good job,’” he says. He told them to take the time to test the model more rigorously.

Anticipating a moment like this, Anthropic had built a framework called the Responsible Scaling Policy, loosely modeled after the US government’s biosafety lab standards, to determine how to handle the risks associated with increasingly advanced AI. As long as its models stayed below a certain level, which it called AI Safety Level 2, it was business as usual. An ASL-2 system might have the ability to give instructions on how to build a biological weapon, but not reliably useful ones or in any more detail than what’s available through a search engine. An ASL-3 system could significantly help a user—particularly one with some basic technical knowledge—actually create or deploy such a weapon.

The models Anthropic had released to that point were all at ASL-2 or lower. If one were to reach ASL-3, Anthropic’s internal guidelines require that the company increase its safeguards. These actions could include hardening defenses so malicious actors couldn’t steal the code or trick the system into giving away dangerous information. Until Anthropic implemented these advanced measures, it would have to take interim measures, such as intentionally weakening the model, blocking certain responses or not releasing it at all.

After almost a week of work, the team determined that the model wasn’t as powerful as Amodei’s staff feared after all. Anthropic released it a little later than expected and—so far at least—it hasn’t led to the collapse of human civilization.

Amodei says the brief delay was “painful,” given the competitive pressures. But if Anthropic succeeds in building technology as powerful as it says it plans to, there are even more uncomfortable decisions ahead. Amodei is convinced that AI is going to transform the world, by creating a “country of geniuses in a data center.” On the bright side, this AI could cure cancer, but it might also cause most of the world’s population to lose their livelihoods. Also, the technology that will cause this massive reordering of society is coming as soon as next year, according to Amodei, and almost certainly not after 2030. “It’s almost an abdication of our moral responsibility not to try to describe in clear terms and as often as possible exactly what is happening,” he says.

Anthropic was founded to usher in this transformation in the most responsible way possible. But customers are also beginning to pay real money for access to its technology: As of April, Anthropic was on track to generate $2 billion in annual revenue, double the rate from four months earlier. Anthropic says it’s not profitable now because of the enormous cost of training AI systems; Amodei has said it could eventually cost as much as $100 billion to train a cutting-edge model.

Customers are almost certainly going to keep wanting more powerful AI. Anthropic fully expects to hit ASL-3 soon, perhaps imminently, and has already begun beefing up its safeguards in anticipation. In recent years the company has hired at least a half-dozen prominent researchers from OpenAI, some of whom have criticized their previous employers for moving away from their own stated commitments to safety. Presumably they won’t stand by quietly if Anthropic tries to sidestep its own commitments when the time comes.

When that reckoning might arrive remains a matter of debate. Like Google, Meta Platforms Inc. and OpenAI, Anthropic has fallen behind on its projected timelines for releasing new versions of its most costly, powerful line of AI models. Skeptics question whether all the talk about the dangers of AI is intended to make the technology appear more powerful than it actually is.

People who worry about AI safety, meanwhile, say the market pressure to build as quickly as possible could push companies into irresponsible decisions. Those investors didn’t give Anthropic $14 billion, after all, to lose to OpenAI, DeepSeek or Meta, and deciding to ignore commercial incentives stopped being an option once it took all that money. “You can’t really fight the market head-on,” Amodei acknowledges. But he also says he can create what he describes as a “race to the top,” where Anthropic pulls the entire AI industry along by demonstrating how to build world-changing AI without destroying the world in the process.

Amodei is a native San Franciscan who’s never considered himself a tech person. He grew up in the Mission District before tech money transformed it. The Amodeis were working class. His late father, who’d grown up an orphan in Italy, worked as a leather craftsman before chronic health issues forced him to stop when Amodei was a child; he passed away when Amodei was a teenager. Amodei’s mother was a library project manager.

Amodei’s sister, Daniela Amodei—also a co-founder of Anthropic and the company’s president—remembers her brother being an unusually gifted child, particularly in math and science. As a toddler, he would declare “counting days” and count as high as he could. “It’d be like a whole day,” she says. “What kind of 3-year-old has that attention span?” He started taking classes at the University of California at Berkeley while still in high school, before studying physics at the California Institute of Technology for two years then transferring to Stanford University.

In college, Daniela remembers Dario first getting interested in AI after reading Ray Kurzweil’s The Singularity Is Near: When Humans Transcend Biology, which predicted that AI would reach human intelligence by 2029 and that people would merge with machines by 2045.

Amodei got his undergraduate degree in 2006, then switched focus to the neurological and biological applications of physics. For his graduate work, he moved east to pursue a doctorate in biophysics at Princeton University. His research involved studying the neural structures found in the ganglion cells of amphibians, which is how he found himself slicing up salamanders to examine their retinas. “I wasn’t thrilled by the animal rights implications of that,” says Amodei, who’s been a vegetarian since childhood. (He makes an exception for shrimp and other invertebrates). But, he adds, “I was a scientist. I wanted to solve the problems of biology, human health.”

The thing that actually got to Amodei wasn’t the ethics of laboratory life, but the pace. While he was struggling through the drudgery of his day job, Amodei saw things moving much faster in another attempt to uncover the essence of intelligence: the development of artificial neural networks. So-called deep learning had fallen out of fashion among computer scientists, but the field was starting to pick up around 2012, and Amodei was impressed by the advances researchers were making using the technology to improve computer vision. “I was like, ‘Wow, this really works,’” Amodei remembers thinking. In 2014, Andrew Ng, a professor in the computer science department at Stanford, where Amodei had done postdoctoral research, recruited him to work on AI at a unit he was running for the Chinese tech company Baidu Inc. He jumped at the opportunity.

Amodei ended up spending a year at Baidu, then another year at Google Brain, an AI-focused research team within Google, where he started thinking about the ethical considerations of AI’s rapid progress. In 2016 he published a well-regarded paper called Concrete Problems in AI Safety outlining the five key areas where AI could cause unintended and harmful behavior.

Although Google was probably the best place a rising AI researcher could work in the early 2010s, a new nonprofit lab called OpenAI seemed to align better with Amodei’s interests. He joined in 2016 as the safety research lead. He lived in a shared house in San Francisco’s Glen Park neighborhood with several roommates, including three other OpenAI colleagues who would later become co-founders of Anthropic. One of them was Daniela. At the time, both Amodei siblings ran in social circles affiliated with effective altruism, a philosophy that emphasizes rational thinking as the most efficient way to improve the world, which was popular among people interested in AI safety. (The movement fell out of favor after one of its most prominent leaders, the crypto mogul Sam Bankman-Fried—whose company was an Anthropic investor before selling its stake during bankruptcy proceedings—was convicted of defrauding his investors.)

Amodei’s most substantial research contribution at OpenAI was developing the concept of “scaling laws,” the idea that you can make fundamental improvements to a neural network simply by increasing the model size and adding more data and computing power. For much of the history of computer science, the assumption was that such breakthroughs would come primarily by designing ever-better algorithms. By helping to pioneer the bigger-is-better strategy, Amodei played a key role in the rise of the large language models that dominate the current AI boom. This earned him a place of prominence within OpenAI and the broader industry.

Amodei’s feelings of responsibility about AI weighed on him, and over time he soured on OpenAI. In 2020 he and six OpenAI colleagues left to start Anthropic, promising to build a more responsible AI lab. It was easy for the gang to get along: They’d already worked with one another, three of them had lived together, and two were related.

The defection remains a subject of intrigue within Silicon Valley, which loves messy startup drama, especially when the companies involved are some of the most valuable unicorns of all time. Amodei remains vague about the subject but talks about losing confidence in OpenAI’s leadership. “I don’t think there was any one specific turning point. It was just a realization over many years that we wanted to operate in a different way,” he says. “We wanted to work with people we trusted.” At the time, Anthropic’s prospects seemed iffy at best, given OpenAI’s access to immense capital and its head start in building the actual models. “The doctrine a few years ago was Anthropic would not be able to scale, because it wouldn’t be able to raise the money,” says Eric Schmidt, former Google CEO and an early investor in the startup.

Yet Anthropic has developed into a serious rival, with comparable tech and a growing roster of paying customers in finance, pharmaceuticals, software development and other industries. (It also makes a publicly available AI-powered chatbot, Claude, but is less focused than OpenAI on the consumer market.)

Schmidt remembers making a 2018 visit to Amodei and his partner, Camilla Clark—now his wife—in the starter apartment they lived in, near the freeway in San Francisco. Amodei was still at OpenAI then, but Schmidt was impressed and ended up investing in Anthropic later. Schmidt was dubious of Amodei’s plan to run Anthropic as a public benefit corporation, a type of for-profit organization dedicated to pursuing a public mission. When Schmidt urged Amodei to establish it as a traditional startup, Amodei refused.

Such debates were common in Anthropic’s early days. “There was a lot of discussion around ‘Are we just going to be philanthropically funded? Are we primarily focused on purely doing research on safety? How much funding do you need?’” says Jared Kaplan, a friend of Amodei’s from graduate school who became Anthropic’s co-founder and chief science officer. “There were some folks who were thinking we should be a nonprofit. I think Dario and I both kind of thought that that was probably not a good idea. We should kind of keep our options open.”

Anthropic now seems well-positioned to be among the handful of winners to emerge from the current AI boom, says Hemant Taneja, CEO of investment firm General Catalyst, which backed Anthropic in its most recent funding round. “This is a company that probably has the right things going for it to be one of those that’s going to matter in the end,” he says. “But I have never written a check from GC this big, with this much uncertainty. I will tell you that.”

Even as it barrels ahead, Anthropic has cultivated a reputation for taking issues such as safety and responsibility more seriously than the company from which it has sprung. OpenAI’s brief 2023 ouster of Sam Altman, whom its board accused of being “not consistently candid,” has been followed by persistent questions about the company’s integrity and commitment to its initial mission.

Amodei generally avoids direct criticism of his former employer, but he and his company aren’t above taking some thinly veiled shots. Anthropic has paid for billboards around San Francisco with taglines that read, “AI that you can trust” and “The one without all the drama.” Unlike other tech executives (including Altman), Amodei has made little attempt to ingratiate himself with the Trump administration, saying his message is the same now as it was when Joe Biden was president. He refers to “a number of players” in the industry who, by contrast, “say whatever” to the party in power in an attempt to curry political favor. “You can tell that it’s very unprincipled,” he says.

This January, Amodei made his first trip to the World Economic Forum in Davos, Switzerland, where he put on a pinstriped suit and engaged in some thought leadership and high-stakes dealmaking. On the same day that he gave a talk touching on DeepSeek and AI-powered health care at Bloomberg House Davos, he spent the better part of an hour huddling with AIG CEO Peter Zaffino. Anthropic walked away with a multiyear contract to help analyze customer data during the insurance underwriting process. (AIG says the deal came out of an 18-month pilot during which Anthropic helped speed up that work by 8 to 10 times.) Zaffino says he picked Anthropic because of its specific focus on trustworthiness and accuracy in citing specific data sources, especially in the highly regulated industry of insurance. Zaffino says he was impressed at what a quick study Amodei was. “For whatever Dario lacks in business experience, the algorithm in his brain moves really fast,” Zaffino says. “He is able to apply what he’s learning and what we’re talking about in terms of what the business objective is.”

When the day’s work was done in Davos, he skipped the evening parties, retreating instead to his hotel room to write an essay about how DeepSeek highlighted the need for stronger export controls on semiconductors. Anthropic’s recent fundraising round has made him a billionaire, and he often travels with a security detail. But he also still lives in a rental house in a suburb south of San Francisco, raising chickens in his yard. He has committed to donating “the vast majority” of his wealth to charitable causes.

When he hosts a Bloomberg Businessweek reporter at Anthropic’s office in March, the pinstripes are nowhere to be seen; Amodei is noticeably content to be back in what Clark calls his “cozies,” stretchy gray sweatpants and a similarly comfortable-looking, also gray T-shirt. The outfit, he says, “helps me think.”

Amodei’s way of thinking, rooted in his years in academia, drives the culture at Anthropic. Every two weeks, Anthropic employees—they call themselves Ants—gather to listen to Amodei deliver roughly hourlong lectures known internally as Dario vision quests. Accompanying documents are distributed in advance, for employees to read before the meeting. Under Amodei’s leadership, Anthropic also researches subjects that aren’t immediately monetizable, such as mechanistic interpretability (the study of how opaque algorithms make decisions) and AI welfare (the ethics of interacting with computers if they ever do achieve sentience).

Through this track record, Anthropic has cultivated a reputation for being genuinely serious about responsible AI development at a time when others in tech can appear to be just paying lip service to the idea—or, in some cases, expressing hostility to the suggestion that an ethical framework is even a reasonable goal if it slows down development. “Dario and the whole team deserve credit and confidence for acting in good faith on safety,” says Matthew Yglesias, a prominent writer on economics and policy with whom Amodei has consulted on his writing. “But it’s not clear if that changes the structural situation. If you’re racing, it’s hard to be safe, even if you’re acting in perfect good faith.”

Anthropic aims to build machines that can do almost anything, but there’s one thing its AI already does particularly well: write computer code. The company recently released an app for coders, Claude Code, and its tech also powers popular independent coding apps including Cursor. Anthropic’s own February economic index report showed that 37% of all job-related interactions about Claude were for coding, the highest of any category. (Arts and media came in second at around 10%.) Amodei says automated coding has probably been the fastest-growing part of its business in recent months.

AI-generated coding doesn’t have the same emotional resonance as computer-generated music or painting. Unlike with a song, consumers don’t much care if the code underlying the app they’re using comes from a real person. Coders themselves have also largely accepted that AI is part of the job: A survey by GitHub Inc. last year of 2,000 technical staff found that almost all, 97%, had used coding tools at some point in their work.

But looming job losses in the field also feel less hypothetical than AI safety issues such as computers making dirty bombs. Anthropic has found 79% of programmers who use Claude Code do so to “automate” rather than “augment” tasks. (The economic index report itself was built in part by Claude Code, according to Anthropic’s head of policy and co-founder–and former Bloomberg News reporter–Jack Clark.)

This is the area where Amodei is already confronting the harsh realities of his company’s work. In a March 10 talk in Washington, DC, hosted by the Council on Foreign Relations, Amodei predicted that AI could be writing almost all computer code within a year. A clip of the comments went viral, sparking a mixture of fear and skepticism. Amodei says the remark was taken out of context. At the event he also said humans will still be involved in the overall coding process, such as specifying what kind of app to make or how it should integrate with other systems. “At the end of the day, these models are going to be better than all of us at everything,” he says. “We have to deal with that. At a societal level, everyone’s got to deal with that. My goal is not to create an underclass in the period before then.”

Awkwardly for Anthropic, the affected workers in this technological shift could be the people in its offices today building those tools of automation. “The vibe is, ‘Oh, it’s getting real,’” Clark says. Amodei made the subject the focus of a recent vision quest talk, in which he told employees that Anthropic’s technology is leading to substantial changes in the way the company organizes its work. “We may slow down our hiring because of Claude, and we’ll do that because we don’t want to fire anyone because of Claude,” he says, recounting what he told his staff. He added that the company will help coders adapt to their evolving roles.

In an internal memo that accompanied the meeting, Amodei wrote that there was a 70% chance that sometime this year, AI’s ability to “perform key technical tasks” such as writing code, debugging, and proposing and managing experiments will go from a “helpful tool” to “something absolutely indispensable” that does the majority of these technical tasks, doubling Anthropic’s execution speed.

“The majority of the contribution to AI progress will come from AI itself,” Amodei wrote, with the caveat that humans will still play a very central role, “probably for a while due to comparative advantage.” The role of humans could be gradually whittled away, though, until AI begins to create new AI in a kind of recursive loop.

This ability, if it’s indeed developed, would send Anthropic’s models shooting up its danger scale. At ASL-4, an AI would have “the ability to fully automate the work of an entry-level, remote-only researcher at Anthropic.” There’s an ASL-5, when the AI has the ability to improve itself with increased acceleration.

In Machines of Loving Grace, a widely read essay Amodei first published internally and then publicly on his personal blog last October, he laid out what the endgame looks like if everything goes right with AI. Drawing on his expertise in biology, he says AI will speed up scientific discoveries at 10 times the current rate, helping cure almost all infectious diseases, most cancers and Alzheimer’s, and ultimately doubling the human life span. (Anthropic now prints pocket-size bound copies of the Machines of Loving Grace essay to give to employees.)

The tone shifts in the part about AI’s relationship to work and meaning. He says this issue is particularly difficult, because it is “fuzzier and harder to predict in advance.” Amodei anticipates that AI could eventually replace most human labor, leaving people to live off a universal basic income or other redistribution method unless they find some yet-to-be-determined way to continue to be economically valuable.

“We will likely have to fight to get a good outcome here: Exploitative or dystopian directions are clearly also possible and have to be prevented,” Amodei wrote. “Much more could be written about these questions, and I hope to do so at some later time.”
Read more: Microsoft’s CEO on How AI Will Remake Every Company, Including His