\n<\/p>\n
The AI industry seems to be having its loudest debate yet about whether its technology poses an existential threat to humanity.<\/p>\n
The current discussion began after AI researcher Jacob Coxon said that he\u2019s resigned from Anthropic<\/a> because he\u2019s worried that the leading AI companies are \u201cgambling with our lives.\u201d Then Anthropic\u2019s alignment leaned chimed in with a post declaring<\/a>, \u201cWe really do earnestly believe AI could kill all humans!\u201d adding that he personally thinks the chance is \u201c>10% within the next decade.\u201d<\/p>\n On the latest episode of TechCrunch\u2019s Equity podcast<\/a>, Kirsten Korosec, Sean O\u2019Kane, and I discussed the latest apocalyptic warnings. I tried to articulate why I\u2019m skeptical of many AI doomer narratives, while Kirsten asked if this was \u201cjust a weird way of flexing to show how far advanced their company\u2019s AI model is,\u201d particularly as these companies prepare to go public.<\/p>\n And Sean wondered how these concerns might show up in Anthropic\u2019s S-1 filing for its IPO: \u201cAre there junior lawyers right now who are going through and having to rewrite that entire section of the S-1 filing to say, \u2018It\u2019s a officially Anthropic\u2019s position that there\u2019s a more than 10% chance that we could develop something that would eradicate all of humanity and that would be materially bad for our business\u2019?\u201d<\/p>\n Keep reading for a preview of our conversation, edited for length and clarity. (Note: We recorded this episode before Anthropic CEO Dario Amodei published his plan for more cautious AI development<\/a>.)<\/p>\n Sean O\u2019Kane:<\/strong> I\u2019m hard-pressed to think of something that blew up so fast. Not only did this warning shot come out from this young researcher who has also worked at OpenAI, but also was immediately shared on X by the alignment lead at Anthropic \u2014 who, in what might go down as one of the best misplaced exclamation marks ever, shared Coxon\u2019s post and and thread and said, \u201cWe really do earnestly believe AI could kill all humans!\u201d Exclamation mark!\u00a0<\/p>\n What a weird vibe. That was just a ton of accelerant on an already fraught post or series of posts. Coming after the Hugging Face hack from OpenAI\u2019s internal model, plus just the increased capabilities we\u2019ve seen with the latest models released by Anthropic and and now OpenAI with with Astra a few weeks ago, I think this was just perfectly timed to be a powder keg type of thing for this young researcher to say.<\/p>\n Anthony Ha:<\/strong> Just to disagree with you, I do think that if you believe that AI could destroy all humanity, that does deserve an exclamation point. I would argue that that is a perfectly well used exclamation point!<\/p>\n My issue with that tweet was more the \u201cwe.\u201d Who is the \u201cwe\u201d here? To what extent can we talk about sort of the AI community or AI research community as a monolith? And the greater than 10% chance \u2014 that\u2019s just a made up number, that doesn\u2019t mean anything. Sometimes [there is] this habit in both the tech industry and other places to just throw out these percentages, they\u2019re not based on anything or calculated based on anything. [In retrospect, I realize the tweet was probably referencing the concept of P(doom)<\/a>, but I still think it\u2019s silly.]<\/p>\n One thing I will say about Coxon\u2019s statement and decision is \u2014 there\u2019s this recurring theme on Equity, when someone like Sam Altman or Dario Amodei is doing this doomer narrative, there\u2019s always this element of: Well, then, why are you doing what you\u2019re doing? If you actually believe that [AI could destroy humanity], you would not continue doing this.\u00a0<\/p>\n [Whereas] this is actually somebody putting his professional trajectory where his mouth is. He\u2019s actually saying, \u201cI believe this is really, really, really bad, and I don\u2019t want to keep working on it.\u201d And so, props for having the courage to do that, if nothing else.<\/p>\n Kirsten Korosec:<\/strong> Yeah, I put him in a separate camp than everyone else saying that and talking about the dangers.<\/p>\n I\u2019m going to put my speculative hat on, because I want to ask both of you a question, which is: Is it possible that every single time we see the increasing number of blog posts about yet another incident in which one of their AI agents breaks through unintentionally, or they talk about how humanity is at risk, is this a weird way of flexing to show how far advanced their company\u2019s AI model is?<\/p>\n I mean, that sounds very cynical, but it does achieve that purpose. Which is: If these AI models weren\u2019t advanced and weren\u2019t capable and weren\u2019t breaking through, we wouldn\u2019t have to worry about these things, right? It\u2019s like a very weird way to brag about the capabilities of the models that you\u2019ve created within your own company.<\/p>\n Anthony:<\/strong> I\u2019ve definitely wondered about this. I don\u2019t think it\u2019s completely cynical, in the sense that I don\u2019t think it\u2019s all just a very conscious marketing ploy across the board. I think that when a lot of these people \u2014 whether the researchers or CEOs \u2014 talk about it, they do have real concern.<\/p>\n But of course, it does align with [their] business interests in a lot of ways, to say, \u201cWow, we\u2019ve built the most deadly software that\u2019s ever been made.\u201d I don\u2019t want to get too psychoanalytic here, but others have pointed out that there is this temptation on a personal level of: Of course, you want to believe that the thing you\u2019re working on is the most important and most dangerous thing in the world.<\/p>\n Sean: <\/strong>The thing that sticks out in my mind when I think about that question is, there\u2019s certainly an element that makes it seem like, \u201cOkay, we\u2019re doing this thing that\u2019s so capable, and that\u2019s good for us in some way, even if it looks bad in a lot of different lights.\u201d<\/p>\n I think what\u2019s different about some of these most recent examples is, it really gives you the feeling that these companies don\u2019t have a handle on this stuff in certain ways, especially with the OpenAI stuff.<\/p>\n We keep seeing more and more reporting about other internal agents that have accessed different wikis on the web<\/a> and are leaving messages for each other, and in a way that doesn\u2019t seem like it\u2019s being handled in a competent way from OpenAI. I would imagine there would be just a bit more polish on the story being told, if it was wholly about getting people to believe that, \u201cOh my gosh, they\u2019ve made something so incredibly capable.\u201d<\/p>\n The other thing that I think is really fascinating about this, in particular, [is] we\u2019re what, a few weeks at most out from seeing Anthropic\u2019s S-1 filing for its IPO, and just a couple more weeks or month or two away from a potential IPO.<\/p>\n And the idea that you\u2019re going to come out and say these things in this clear language ahead of an IPO \u2014 I\u2019m very interested in what that means for that process. How much of this kind of stuff had they already written into the S-1 and the risk factors inside that document? Are there junior lawyers right now who are going through and having to rewrite that entire section of the S-1 filing to say, \u201cIt\u2019s a officially Anthropic\u2019s position that there\u2019s a more than 10% chance that we could develop something that would eradicate all of humanity and that would be materially bad for our business\u201d?<\/p>\n Kirsten:<\/strong> You\u2019re assuming that it\u2019s not in there already.<\/p>\n Sean:<\/strong> That\u2019s what I\u2019m saying, though: Is it in there already and being reworded? Or is this something that\u2019s a true scramble? There has to have been language in there. It\u2019s one of the reasons I\u2019m so eager to read this document in a way that goes even further, in some ways, than the SpaceX [S-1], because I\u2019m sure that there\u2019s probably stuff specific to these ideas that will be interesting to see.<\/p>\n Kirsten:<\/strong> Here\u2019s the thing: In a traditional investment environment, one might believe that language like this would hurt the valuation of a company, because it\u2019s suddenly dangerous. But we don\u2019t live in normal times.<\/p>\n And so again, back to my point, it could end up being a weird beneficial flex for the company on the valuation side. It\u2019s not the same as the whole rage-baiting trend that we saw last year, but it\u2019s in that same, let\u2019s say, universe, in which the strength, capability, even elements of danger of something, equals high valuation. So I guess we\u2019ll see in a few weeks.<\/p>\n Putting that aside for a minute, what is being done about it? And can we control this? Tthe U.S. executive director of a nonprofit called ControlAI, Connor Leahy, he was on the show this week<\/a>, talking about this. So what are you paying attention to in terms of how to control the dangerous aspects of AI, or are we throwing up our hands and watching it all unfold?<\/p>\n Anthony:<\/strong> I myself do not necessarily have a great answer to this, but I have been thinking about some aspects of this debate and maybe why I respond the way I do.\u00a0<\/p>\n To echo one of Sean\u2019s points, I do think that part of what this speaks to is the extent to which these major AI companies are feeling like they\u2019re not really in control of these models anymore. That\u2019s definitely not great. That is something that we should all be worried about.\u00a0<\/p>\n I do think that part of the reason I\u2019m skeptical of the doomer narrative or resistant to the doomer narrative is because it reaches this level of hysteria of, \u201cWow, this could destroy humanity in the next 10 years.\u201d It is a little bit of a distraction from the more immediate harms that AI can have, whether that\u2019s labor-related, whether that\u2019s environment- and climate-related.\u00a0<\/p>\n Ideally, I think we should be able to discuss all of these things, and have regulatory and other kinds of safeguards against all of these things [including AI\u2019s existential threat]. But once you start using phrases like AGI and superintelligence, that just sucks up all the oxygen in the room in a way that is not very helpful.<\/p>\n<\/div>\n When you purchase through links in our articles, we may earn a small commission<\/a>. This doesn\u2019t affect our editorial independence.<\/em><\/p>\n