Anthropic CEO Dario Amodei’s AI Safety Philosophy and Founding Journey

Author: The New York Times

Compiled by: Jiahuan, ChainCatcher

On a night in 2007, Dario Amodei, then a PhD student at Princeton University, hosted a visiting friend. Having just finished reading Ray Kurzweil’s 2005 book The Singularity Is Near, he eagerly shared the book’s ideas with his friend.

Kurzweil predicted in his book that artificial intelligence would eventually surpass the human brain, leading to a moment known as the "singularity." Dario, twirling his hair around his finger, told his friend Colin Heckman that this moment could arrive within their lifetimes.

“He was obsessed,” said Heckman, who attended the same high school in San Francisco as Dario. “He said AI had to be created the right way, or we were done.”

At 24, Dario was a physics PhD student in a phase of absorbing vast amounts of information. He was immersed in reading relatively niche blogs such as Accelerating Future and Overcoming Bias, where authors discussed the possibility of AI leading to human extinction.

He also delved into GiveWell’s online forum, an organization dedicated to maximizing the impact of charitable donations, where intellectuals discuss how to earn the most money for charity.

We interviewed more than twenty of Dario’s classmates, friends, teachers, and colleagues. Their accounts reveal that these books, blogs, and online communities continue to influence him today and offer insight into how the 43-year-old CEO of Anthropic, which is expected to go public this month and could become one of the largest public listings globally, views AI.

His friends and colleagues say that the core of Dario’s worldview consists of two interwoven ideas, both of which firmly believe that mathematical reasoning can transform the future of humanity.

One is "rationalism," which emphasizes viewing the world in a calm, rational manner. Advocates of this view, such as self-taught AI researcher Eliezer Yudkowsky, believe that AI will become powerful enough to ultimately destroy society.

Another is "effective altruism," which developed alongside rationalism and seeks to maximize social welfare through precise mathematical calculations.

His friends and colleagues say that at Dario’s, these two currents merged into a firm conviction: AI could be extremely dangerous, and only he and like-minded technical experts deserved to be entrusted with developing it. This drove him to strive to create AI before those who don’t care about human safety, and fueled his desire to control the technology.

In a September article calling for a slowdown in AI development, Dario wrote: "Only by building this technology the right way can we reap its benefits. It is worth taking extra care to get it right, as long as we can make good use of the time we gain."

These ideas became the foundation of Anthropic. In 2021, Dario co-founded the company with his sister Daniela Amodei and five others. All of them had previously been involved with the rationalist and effective altruism communities. As the company grew, their views gained increasing attention, and Dario has consistently and actively advocated for them publicly.

Caption: Daniela Amodei, co-founder of Anthropic, at the company’s San Francisco office in 2023. She is close to her brother, Dario.

Dario also participated in discussions on geopolitical issues. He stated that the U.S. should restrict the sale of chips to China to weaken China’s competitive capabilities in AI. He called for regulation of AI to prevent the technology from spiraling out of control. In the dispute with the U.S. Department of Defense, he also stated that AI should not be used for mass surveillance of Americans or for autonomous weapons.

“He truly believes we’re facing enormous existential risks,” said Nathan Avery. Avery ran cross-country with Dario in high school and had many deep conversations with him over the years, though they’ve grown apart since then. “He felt he needed to put himself in a position where he could take action.”

Dario is a billionaire whose wealth is expected to grow further as Anthropic prepares for its public listing. He declined the interview through company representatives.

His ideas have earned him supporters, particularly among some professionals in the AI safety field, who believe that Dario is developing AI responsibly.

But this also made him enemies in Silicon Valley and Washington, with critics including Meta’s Mark Zuckerberg and NVIDIA’s Jensen Huang, who accused him of being a “doomsayer” trying to centralize power in the AI field. In September, the U.S. Department of Defense posted on social media: “Americanism, not effective altruism.”

Dario seems to have realized that his ideas angered some people, so he attempted to publicly distinguish his beliefs from rationalism and effective altruism. In 2023, on The New York Times podcast Hard Fork, he stated that he does not consider himself an effective altruist.

“I agree with some of these ideas,” he said, but he opposes “dividing people into different tribes.”

Dario and his sister Daniela grew up in the Mission District of San Francisco. Their father, Riccardo Amodei, was a leather craftsman, and their mother, Elena Engel, managed the renovation of a public library. Divorce records show that the two separated in 1991.

While in eighth grade at Lorton Alternative School, Dario was already standout. In 1997, he and a classmate were named "most likely to succeed," and in their yearbook photo, both held U.S. dollar bills.

While attending Lowell High School, San Francisco’s top public high school, Dario eagerly studied and also took classes at the University of California, Berkeley. Classmates say he pondered ideas such as utilitarianism, which holds that people should act to create the greatest happiness for the greatest number of lives. He was also concerned that the Y2K problem—where some computer programs recorded years using only two digits and might misinterpret “00” as 1900—could cause global computer systems to fail.

Richard Shapiro, a former physics teacher at Lowell High School, said that Dario often felt he didn’t perform well on exams.

“I would yell at him: ‘Dario, relax!’” Shapiro said.

In 2001, Dario enrolled at the California Institute of Technology in Pasadena, but he frequently returned to San Francisco to visit friends and family. On one such visit, he went to a sushi restaurant with Avery and Heckman, and the three discussed utilitarianism. Dario was a vegetarian, but he believed it was acceptable to eat crabs because they are invertebrates.

During the meal, Heckman posed a thought experiment: If a person keeps killing chickens, under what circumstances would it make sense to kill that person to prevent more chickens from being killed?

Avery recalled that Dario let out a light laugh and said that, from a utilitarian perspective, when the number of chickens reaches a certain threshold, their lives may be more important than a single human life.

Friends say Dario once identified as an atheist, yet he held strong convictions. In 2003, he drove Avery and others to San Francisco to participate in a protest against the U.S. invasion of Iraq.

In his second year, Dario transferred to Stanford University, at the heart of Silicon Valley. This transfer also brought him closer to his sister, who was attending high school nearby at the time.

In 2023, he spoke about his sister on a podcast: “We’ve always wanted to save the world together.”

After graduating from Stanford University in 2006, Dario entered Princeton University to pursue a Ph.D. in physics, ultimately focusing on applying physics to biology and computational neuroscience. That year, his father, who had hepatitis C, passed away.

Dario devoted himself fully to retinal cell research and began developing an interest in technologies that could accelerate scientific progress.

Jade Wang, who previously dated Dario and co-founded a scientific discussion group with him, said the two often discussed this question: whether scientists who have spent their careers studying a single disease could have a greater impact by shifting their focus to developing technologies or methods for the entire field.

She said that Dario mentioned his father, who passed away before the drug that could treat most hepatitis C patients became available.

Caption: Ray Kurzweil, author of The Singularity Is Near. Dario is interested in his writings on artificial intelligence.

At that time, Dario had already begun to resonate with the writings of Kurzweil and rationalist leader Yudkowsky. He was particularly intrigued by Kurzweil’s idea that advanced technology could help humanity transcend its natural limitations—a concept known as “transhumanism.”

In 2007, during Heckman’s visit to Princeton, Dario told him that the singularity would arrive before 2040, and that humans might one day merge their minds with machines.

Heckman recalled that Dario said at the time: “Over time, we will become the new superintelligent AI, or a hybrid of AI and humans.”

Dario was so fascinated by this idea that he encouraged his friends and sister to attend the annual conference hosted by Yudkowsky’s Singularity Institute, which is now known as the Machine Intelligence Research Institute.

In a 2008 email seen in The New York Times, Dario told a friend that Yudkowsky was "one of the most profound thinkers I've ever met" and "explicitly trying to bring about a human-benefiting singularity."

Yudkowsky stated in an email to The New York Times that Dario “is not currently seeking or taking my advice, and I don’t recall him ever doing so.”

Caption: Self-taught AI researcher Eliezer Yudkowsky. Dario began to resonate with his writings. Yudkowsky believes that AI will become powerful enough to destroy society.

Around the same time, Dario also began corresponding with Elon Musk, the founder of the rocket company SpaceX. According to two people familiar with their communications, Dario told friends that he admired the tech magnate and considered working for him, but ultimately did not join the space startup.

In 2008, Dario encountered GiveWell while reading the blog of economist Tyler Cowen, who described the organization as offering “interesting recommendations” about charitable organizations.

Dario became active on GiveWell’s email list and forums, discussing how to conduct charity and scientific work in an effective altruism manner. An email seen by The New York Times showed that in 2010, he wrote that he was “considering entering the finance industry so that I could donate more than I do now.”

In the same year, Dario discussed his decision to donate $10,000 in a blog post on GiveWell, choosing between two organizations: the Stop TB Partnership, which focuses on preventing tuberculosis in adults, and VillageReach, which helps vaccinate children in African villages.

Dario wrote that the "cost-effectiveness" of the "Partnership to Combat Tuberculosis" is higher because the organization serves adults, and "adults have deeper, more meaningful experiences than infants." He added, "I believe the death of an adult may be two to three times more severe than the death of an infant."

But after further comparing how the two organizations executed the project, Dario changed his mind and concluded that VillageReach’s work was more effective.

In 2011, he began postdoctoral research at Stanford University studying cancer cells. Friends say that in his personal life, he tried taking anti-aging supplements, adhered to a strict swimming regimen, and became interested in calorie restriction. Some studies suggest that calorie restriction can extend the lifespan of laboratory mice.

“He often talks about the finiteness of time and how time is the only non-renewable resource,” says Avery. He adds that Dario is “almost harsh when it comes to time.”

Two people who knew him said that in 2014, Dario was diagnosed with testicular cancer, which further deepened this feeling. He underwent chemotherapy and eventually made a full recovery.

By then, he had developed a close relationship with Holden Karnofsky, one of GiveWell’s founders. GiveWell began collaborating with Good Ventures, the charitable foundation of Facebook co-founder Dustin Moskovitz and his wife Cari Tuna, which funds causes related to effective altruism.

Dario is providing advice on life sciences funding for the joint project GiveWell Labs.

Caption: One of GiveWell’s founders, Holden Karnofsky (right), photographed in 2012. Dario was active on GiveWell’s online forum.

At the time, Karnofsky was dating Daniela. The two moved into a shared house in San Francisco, where later residents included Dario, theoretical physicist Jared Kaplan, and college dropout Christopher Olah. Dario had previously tutored Olah in mathematics.

Three individuals who visited the residence said that the roommates discussed Yudkowsky’s articles, exploring rationalism and effective altruism. They also attended the annual countercultural gathering in the Nevada desert, Burning Man, sometimes wearing animal costumes.

Kaplan and Olah later co-founded Anthropic with the Amodei siblings. In 2025, Karnofsky also began working for the company.

In the early 2010s, advances in machine learning spurred tech companies to increase their investment in AI again. Machine learning enables computers to learn how to perform tasks by analyzing data.

Baidu, one of China’s internet giants, is among the companies involved. Baidu opened a laboratory in Sunnyvale, California, and hired Andrew Ng, a pioneer in AI from Google, as its chief scientist.

Dario joined Baidu in 2014 and quickly became known for pulling all-nighters at the office and speaking his mind openly.

Caption: Andrew Ng, an AI pioneer who previously worked at Google and Baidu. Dario joined Baidu to conduct AI research.

Gregory Diamos, a researcher who worked with Dario at Baidu, said: "Dario would passionately argue whenever he had the chance."

Even in the early stages of this wave of AI development, Dario was already concerned about the potential dangers of the technology. Diamos recalls that in 2015, Dario told his colleagues that AI could soon reach or surpass human brain capabilities, and therefore companies developing AI should not ignore the possibility of losing control of the technology.

In the same year, Dario joined Google's AI lab, Google Brain, focusing on AI safety. In 2016, he and five other researchers, including his roommate Olah, published the paper "Concrete Problems in AI Safety," outlining potential anomalous behaviors that AI could exhibit if developed improperly.

In 2016, Dario joined a new AI project: the nonprofit organization OpenAI. He led OpenAI’s safety team, as well as another team researching “scaling laws,” which describe how AI model performance may improve as model size, training data, and computational resources increase.

Seven individuals who worked with Dario said that during his time at OpenAI, his conviction about the right way to develop AI grew stronger. He spoke more frequently about the power and potential risks of AI, and took action to embed his ideas into the development of the technology.

For example, three days after Donald Trump was elected President of the United States in November 2016, Dario wrote in OpenAI’s internal communication app that he was concerned the United States was sliding toward authoritarianism.

According to the New York Times, he stated that he “could not in good conscience research technologies with significant military applications in an authoritarian state.”

Internal messages show that Dario also wrote that investor Peter Thiel, for donating $1.25 million to Trump’s campaign, is “morally culpable” and that OpenAI should no longer accept his money. (Thiel’s venture capital firm, Founders Fund, later invested in Anthropic.)

Dario also helped OpenAI establish a partnership with funders who shared its vision. Three people familiar with the arrangement said that in 2017, he facilitated a connection between OpenAI and the charity operated by Karnofsky, where he served as a technical advisor.

Karnofsky’s organization, now called Coefficient Giving, donated $30 million to OpenAI. Karnofsky has also joined OpenAI’s board.

Caption: Dario at OpenAI in 2017, and Daniela at OpenAI in 2019.

Later that year, Karnofsky married Daniela. According to the couple’s wedding website, guests received plush toys, and those who matched up the “most romantic pairings” among different characters were rewarded.

Dario attended the wedding wearing a panda headpiece, as the dress code was “funny formal.” In a photo from The New York Times, he is seen in this outfit alongside Olah, wearing a giraffe headpiece, and his then-girlfriend, Cami Clark. He married Clark in 2022.

In 2018, Daniela joined OpenAI and eventually became Vice President of Safety and Policy. Some employees soon began to question the growing influence of the siblings.

Three people familiar with the discussions said that in December of the same year, OpenAI researcher Joshua Achiam told CEO Sam Altman that appointing Dario as the company’s chief ethics officer was unwise. Achiam pointed to the Amodei siblings’ growing power and stated that oversight of AI should be shared by individuals with differing perspectives.

When Achiam insisted that an external expert should be appointed as Chief Ethics Officer, Dario confronted him directly. According to insiders, Dario raised his voice, stating that no one was more suited for the role than himself.

In 2018, the Amodei siblings took over a company project called GPT. GPT stands for "Generative Pre-trained Transformer."

Six individuals who previously worked at the company said they explored a security technology designed to teach AI tasks through human instructions.

This approach later became a defining characteristic of Dario’s career: bringing AI closer to potential dangers and then testing it with safety technologies.

OpenAI President Greg Brockman wanted to be involved in the GPT project, but was resisted by the Amodei siblings. Two people familiar with the company said that Dario repeatedly told Altman that he would leave if Brockman was not kept away from the work.

A person familiar with Altman’s thinking said that Altman found these ultimatums excessive and therefore rejected the siblings’ demands.

In December 2020, the Amodei siblings and other colleagues left OpenAI. Four weeks later, they founded Anthropic.

Caption: In May this year, the Amodei siblings appeared on stage with the company's Chief Product Officer, Ami Vora (left), at the Anthropic Developer Conference in San Francisco.