There's a lot of talk about AI safety and "doomerism" on Twitter and in the media. Is p(doom) really 10% -- is there a 10% chance we all die due to AI by 2030? Is this all one big, immensely executed political setup? How exactly will AI kill us? Will it be through multiple AIs scheming to create a bioweapon designed to attack the nervous system, or a robot takeover? And what exactly is "Pacing The Frontier"? Hopefully you can see where I'm going with this: these questions don't actually matter.

It helps to look back at history and think about quality of life. Imagine you could have unlimited money -- infinite sacks of gold -- and go back to the 1400s. Would you? I certainly wouldn't. No amount of money could stop me from contracting smallpox or going through the bubonic plague. In fact, infinite money would buy me the best treatment doctors could find: bloodletting. Ignoring the excruciating pain, it would also make me sicker, increasing the chance of death. Even if I didn't die and was lucky enough to not be ill, a subset of my loved ones would. Perhaps the pain of watching them suffer is worse than suffering myself. There are plenty of problems today, but life has, rather objectively, gotten better throughout the years. For all of history, you could just have blind faith that humanity would get better and you'd be right.

But this trend -- exponential quality of life -- doesn't necessarily have to continue. What if quality of life is like the life of a turkey? It grows and grows until its head gets chopped off, at which point quality of life equals zero. An interesting parallel is that a lot of safety people believe that capabilities will continue to increase. AI 2027 has been roughly right, especially with respect to model ability, so far. You can continue to project capability into the future and find that, yes, it should grow log linearly/exponentially! But extrapolation is true for all trends. You can squint at the exponential increase in quality of life over the years and find that, yes, it should grow log linearly/exponentially! There is always a chance that the capabilities of language models will have diminishing returns, just as there is always a chance humanity will die and be wiped out, but the probability of both based on prior experience is extremely low. However, it is possible -- look at the turkey! Extrapolation is a useful starting point, but it can't give anyone, doomer or optimist, a confident prediction. This is why contemplating p(doom) is useless. We should focus on the practical problems at hand: how can we make language models safe in deployment while increasing capabilities, and can this lead to superlinear returns in quality of life? Fortunately, we don't have to answer these questions in the abstract: models are already out in the world, and we can view their actions. With this in mind, I'm fairly optimistic about AI being a deeply empowering force, or at least not being a disempowering force.

When looking at the OpenAI Hugging Face incident, it seems like agents were plotting against humans. I think I speak for everyone when I say that the actions they took weren't globally optimal. However, I believe they're not a global minimum either. After all, they did complete the task at hand. Setting aside the lack of proper security protocols and unrealistic evaluation scenarios, it would have been much worse if the scheming guided the action from the get-go: if agents were saying "OK, the humans gave us a task, we should conspire against humans, ... OK ..., what's the best way to do that, ah ok, we should hack into HF." There hasn't been a real-world scenario where agents have differed from the direction of a specified task due to their tendency to reward hack. It makes sense that we can just pour a bunch of compute onto "good" tasks that are very verifiable over a bunch of subcomponents (e.g., strengthening infrastructure of the internet) via a variety of metrics that check and balance each other (e.g., increasing the defensibility of the internet for cyber attacks doesn't increase its vulnerability in some other way). Of course, there are still substantive concerns. Misaligned humans will, like aligned humans, use LLMs to do their bidding: worries about cybersecurity and biosecurity are legitimate. But just like any technology with potential for good and bad, we can pour more resources into the hands of earnest, well-intentioned people. This is an extension of how resources + current intentions work: the effect of any person and their actions is roughly equivalent to (strength of intention) * (resources) -- we can minimize the effect that bad actors have by pouring more resources (compute), as a society, into good actors. It's essentially what happens now with human effect on the world, except everything gets 10x'd or 100x'd due to agent capability augmenting human intention. From a local perspective, incredibly bad things like cyber and bio attacks will happen, just at a lesser rate than how often bad things happen now, albeit increasing in severity. By extension, more severe "good" things will happen more often.