On September 8, a 27-year-old British researcher named Jacob Coxon sat down on a park bench in San Francisco’s Alamo Square and drafted his resignation from Anthropic. The people building AI, he wrote, believe it “could kill us all by the end of the decade.” In less than two days, his post had been viewed by more than 150 million people.
Since then, things have gotten weird. Anthropic’s head of alignment endorsed Coxon, putting the odds of human extinction above 10%. Later, an OpenAI staffer went with 70% in three years. Dario Amodei then published a 3,800-word essay calling for an AI slowdown. Sam Altman agreed. Obama weighed in. Tech stocks tumbled. The president of the United States posted about it five times in one day, naming himself the “Hoax Buster,” and declaring that the only guardrail AI needs is a “STRONG AND SMART (High IQ!) PRESIDENT.”
I’ve spent the past two weeks discussing this topic with everyone, from engineers and computer scientists to policymakers and philosophers. I’ve talked about it on my own podcast and on national television. After hours of rumination, I’ve finally reached my conclusion:
This is the dumbest conversation of the year.
Groundless
First, let me clarify what I’m not saying. I’m not saying AI safety concerns are stupid. Far from it. Misaligned AI could be used to destroy critical infrastructure, destabilize financial systems, escalate military conflicts, and even initiate biological warfare. AI safety is an incredibly important topic and one that I take very seriously.
What I am calling stupid, however, is the way in which this conversation is being had. Unlike productive conversations, which are grounded in data, evidence, and historical context, this one has been grounded in … nothing. It started with a guy’s opinion, and the world piled on with more opinions. But, similar to a Daily Mail gossip tabloid, facts never played much of a role.
We could begin with Jacob Coxon’s post, which, alarming as it may have been, didn’t actually break any news. He told us he was worried about AI, that others were too, and that he thought we might all be dead in the next three years. This was an interesting hot take, but nothing more. Despite that, we all pretended as if something huge had happened. In reality, a guy had said an opinion online.
The conversation got heated after current Anthropic researcher Evan Hubinger said the probability of human extinction was 10% or higher. How did he come up with these odds? What variables went into his calculation? None, of course. It was an opinion with a random number attached. The element of probability made it sound like a statistical computation, but it wasn’t rooted in evidence. It was just another opinion.
Empty Spiral
From that point forward, it was an empty spiral of ever-hotter takes. Some accused Jacob Coxon of being a political plant, sent by the Democrats to put a stop to Anthropic and OpenAI. Others accused Anthropic and OpenAI of hiring him to help them establish a duopoly with regulatory capture. Both takeaways were completely contradictory — and completely lacking in evidence — but that didn’t stop people from repeating these claims with certainty.
The conspiracies continued. Former AI Czar David Sacks soon accused China of being behind the whole thing. (He also claimed he had evidence, but never presented any.) Later, Elon Musk said the extinction claims were a “setup” to support Anthropic’s IPO, despite previously agreeing with those same claims just days earlier. Trump’s position? All of the above. It was a hoax, manufactured by China, Anthropic, OpenAI, and the Democrats. How could that make sense? Don’t worry about it.
Eventually it wasn’t clear whose side anyone was on. Baseless accusations of conspiracy were flung at every camp: the left, the right, the doomers, the accelerationists — everyone was culpable. The AI extinction debate soon felt like the Spiderman meme. So many fingers were pointed in so many directions that none of us knew what we were arguing about anymore.
Part Of The Problem
The AI peanut gallery was largely of our own making, as social media has made us allergic to evidence and addicted to takes. Scholarly articles backed by peer-reviewed research usually come into contact with only a handful of people. Banger tweets, meanwhile, reach millions. This is the nature of media today — it no longer rewards factual rigor.
But it’s also the fault of the AI leaders themselves. For years, they have flip-flopped on their own beliefs, making it impossible for us to know when they’re lying and when they’re telling the truth. Sam Altman once said AI would end us, and later that doomerism was over the top. Does Elon Musk believe AI will destroy civilization? Depends on when you ask him.
Your God, Your Problem
In addition to evidence, what’s fundamentally lacking from the AI conversation is accountability. When Boeing’s planes started falling out of the sky, the CEO didn’t publish a 3,800-word think piece asking the industry to fly a little slower. The planes were grounded, the company paid $2.5 billion to the government, and the CEO lost his job. If Anthropic truly believes it’s building something that could kill us, they should do something about it. But for some reason they think they’re not responsible for their own products. For some reason, they think we are.
Companies have refused to take accountability before, but that’s where law enforcement comes in. When seven tobacco CEOs swore to Congress in 1994 that nicotine wasn’t addictive, the government didn’t respond with a quote-tweet. It went and got the documents, including a 1963 memo from Brown & Williamson’s general counsel: “Nicotine is addictive. We are, then, in the business of selling nicotine, an addictive drug.” Four years later, the industry signed a $206 billion settlement. Same with Purdue Pharma ($8.3 billion, two guilty pleas) and Volkswagen ($4.3 billion, one executive in prison). We didn’t debate what they “really” believed. We simply read their emails.
The solution to the accountability problem is simple: we must subpoena Anthropic and OpenAI immediately. Dario Amodei, Sam Altman, Jacob Coxon, Evan Hubinger, and any other relevant personnel must be brought in front of Congress for questioning. They must offer up evidence of their claims, including how they determined their technology has a 10-70% chance of making us extinct.
If it turns out they are right, then we must obviously do something dramatic (and probably shut the labs down.) But if it turns out they were lying or exaggerating, then we ought to discuss other remedies — remedies that would address the public deception and securities fraud they might have engaged in when they propagated these claims.
The point is, someone needs to be held accountable. It’s no use shouting more takes online.
Red Tide Rising
What does China think of all this? Does it make them more confident to see us losing our minds over a couple of tweets? Answer: Probably.
Meanwhile, the people of China feel good about this technology. 84% of Chinese people say they’re excited about AI (the highest of any country surveyed), and 72% say they trust it. In America, those numbers are 38% and 32%, respectively. Americans feel worse about AI than almost any other nation. Considering the nature of our conversations, you can understand why.
China is also doing what any normal government would do with a new technology: They’re regulating it. Beijing has implemented strict labeling rules, model registration requirements, and algorithmic filing laws. Their protections are far stricter than America’s, and it shows.
Perhaps we ought to take a page out of China’s book. Perhaps we ought to skip past the AI industry’s intellectual vagaries and start actually investigating them. Maybe once we actually know what’s going on, we can start to create regulation that makes sense. Until then, though, it’s more going in circles. Hope you brought your Dramamine.
See you next week,
Ed
P.S. On Friday at 11am ET, I’m going live with Bending Spoons CEO Luca Ferrari, the man who bought AOL, Airtable, and Vimeo, for our first-ever Founder Series livestream. It’s for Prof G+ paid subscribers only. Upgrade here.
The “Y2K anyone?” comment sent me down a small rabbit hole.
New technology has a funny way of making us leap from “this could change a lot” to “this will change everything, probably for the worse.”
In the 1920s, a Princeton dean blamed the automobile for declining “moral standards.” In 1926, an adult-education guide was asking whether the telephone would “break up home life” and make people lazy. By 1993, the LA Times was already describing “cellular hysteria” around alleged health risks from mobile phones.
The fears weren’t always irrational, and the technologies weren’t consequence-free. What followed was adaptation: better evidence, regulation, standards, infrastructure, and behavioral adjustment. Even Y2K is complicated—the rhetoric got apocalyptic, but the remediation effort was very real.
Which is why I see Ed’s argument. We seem to have a long history of turning technological uncertainty into civilizational certainty.
AI may be different in important ways. Our reaction to it feels very familiar.
The new accelerant is the distribution system: social media lets every extreme prediction travel farther and faster than the underlying evidence.
I agree with you, mostly. But "Companies have refused to take accountability before, but that’s where law enforcement comes in." is of course not true, at least not anymore. The 2008/09 financial devastation that impacted ordinary people over the whole world, led to exactly one arrest, if I am not mistaken. The other, hundreds (thousands?) of culprits have since long enjoyed their million dollar bonuses again. Same goes for social media executives, who made millions and sometimes billions, knowingly exploiting young peoples' insecurities and social angst for their own profit.... I have seen nobody go to prison yet. (The recently touted fines are just pocket money for these companies). And these are only 2 examples. I actually think that exactly the fact that accountability is hard to find these days, helps to get people freaked out of their wits. Because if nobody is held accountable anymore, how can we trust them to really reign in their worst tendencies?