Novo Serbia only needs three instruments to successfully plan an economy, an Input-Output table. a firing squad and that other thing.
Better cry Sy
“Neither is it likely that we get a sudden explosive takeoff in AI capabilities. Frankly, I see the path that AI development has taken as ruling that out. Training AI, both pre and post, takes enormous quantities of resources. The old threat model of a self-improving program racing ahead in hours to complete world domination is at odds with the facts that creating new models has taken exponentially increasing quantities of compute.”
Most of your claims in this post are downstream of this, making me think of it as the linchpin of your argument. And yet, I just don’t think you have properly argued for it at all.
Large and expensive pretraining runs are becoming less and less important as antecedents of the AI models that get constructed. Companies are increasingly able to extract ever-more capable models from the same pretraining base, as a consequence of the RL revolution in LLMs that first gave rise to the o1-series in late 2024. It used to be you had a large pretraining run for GPT-2, then for GPT-3, then for GPT-4. Now several models get released from the same base.
Moreover, algorithmic progress and hardware efficiency have both gone up tremendously. Extracting “capabilities” (through whichever tightly correlated proxy you select) is an engineering task that AI labs and developers get increasingly better at over time.
Finally, and most importantly, AI is increasingly being used in AI research itself. Currently, Anthropic allows more than 26% of its research projects to be led (not just worked on, but led) by internal Claude versions. These already speed up production by a factor of at least 1.5x, as METR rigorously showed through their analyses. And this is a compounding effect: the better models get, the more attention is paid to their AI research capabilities, the more of a contribution to AI research they have, and so on.
It is not guaranteed that we will have a hard takeoff. But we have nonetheless seen the equivalent thereof in several domains already.
In Oct 2025, Karpathy went on the Dwarkesh Patel podcast and claimed not only that AGI is still a decade away, but that autonomous coding agents that can be trusted as project leads were a decade away. Less than two full months (not years, months!) later, he was talking about how capability improvement had been so fast that he never wrote any code by hand anymore, and was instead just coordinating teams of agents. Mind you, this was with Dec 2025 coding agents, not the Astra’s and Opus 5.5’s of today, which are significantly better.
Similarly, with mathematics, AI went from not being able to autonomously prove an interesting open research problem until the end of 2025 (arguably the first such instance of it was on Jan 4 2026, with Erdős problem 728) to being robustly superhuman at frontier mathematical research by early October of this year. The first major open problem solved fully autonomously by AI was the Erdős unit distance conjecture, on May 20. By Oct 6, OpenAI had dropped 722 manuscripts in 372 problem families, some of which were among the greatest and hardest math problems in the world (experts thought many of them would never get solved in their lifetimes), and most of them were solved within 3 hours by what insiders expect to be a future publicly available ChatGPT model.
Will the same happen with AI research? I do not believe it is guaranteed. But I even more strongly believe that offhanded dismissals of the possibility of rapid liftoff happening very soon are deeply mistaken
The pro-pause faction lacks credibility if it is just a reaction against change--in fact, one of it's best arguments is that, because the good AI future could be so great, it's worth sacrificing a little time now to dramatically increase our chances of actually getting there.
We won't be sure that alignment worked (until its too late to fix it if it goes wrong, at least under the current paradigm)--but we can be more or less confident. In your linguistic analogy, if I move to Shanghai I will never be certain that I understand the people around me exactly right, but it would still be a good idea to study Chinese first.
“Neither is it likely that we get a sudden explosive takeoff in AI capabilities. Frankly, I see the path that AI development has taken as ruling that out. Training AI, both pre and post, takes enormous quantities of resources. The old threat model of a self-improving program racing ahead in hours to complete world domination is at odds with the facts that creating new models has taken exponentially increasing quantities of compute.”
Most of your claims in this post are downstream of this, making me think of it as the linchpin of your argument. And yet, I just don’t think you have properly argued for it at all.
Large and expensive pretraining runs are becoming less and less important as antecedents of the AI models that get constructed. Companies are increasingly able to extract ever-more capable models from the same pretraining base, as a consequence of the RL revolution in LLMs that first gave rise to the o1-series in late 2024. It used to be you had a large pretraining run for GPT-2, then for GPT-3, then for GPT-4. Now several models get released from the same base.
Moreover, algorithmic progress and hardware efficiency have both gone up tremendously. Extracting “capabilities” (through whichever tightly correlated proxy you select) is an engineering task that AI labs and developers get increasingly better at over time.
Finally, and most importantly, AI is increasingly being used in AI research itself. Currently, Anthropic allows more than 26% of its research projects to be led (not just worked on, but led) by internal Claude versions. These already speed up production by a factor of at least 1.5x, as METR rigorously showed through their analyses. And this is a compounding effect: the better models get, the more attention is paid to their AI research capabilities, the more of a contribution to AI research they have, and so on.
It is not guaranteed that we will have a hard takeoff. But we have nonetheless seen the equivalent thereof in several domains already.
In Oct 2025, Karpathy went on the Dwarkesh Patel podcast and claimed not only that AGI is still a decade away, but that autonomous coding agents that can be trusted as project leads were a decade away. Less than two full months (not years, months!) later, he was talking about how capability improvement had been so fast that he never wrote any code by hand anymore, and was instead just coordinating teams of agents. Mind you, this was with Dec 2025 coding agents, not the Astra’s and Opus 5.5’s of today, which are significantly better.
Similarly, with mathematics, AI went from not being able to autonomously prove an interesting open research problem until the end of 2025 (arguably the first such instance of it was on Jan 4 2026, with Erdős problem 728) to being robustly superhuman at frontier mathematical research by early October of this year. The first major open problem solved fully autonomously by AI was the Erdős unit distance conjecture, on May 20. By Oct 6, OpenAI had dropped 722 manuscripts in 372 problem families, some of which were among the greatest and hardest math problems in the world (experts thought many of them would never get solved in their lifetimes), and most of them were solved within 3 hours by what insiders expect to be a future publicly available ChatGPT model.
Will the same happen with AI research? I do not believe it is guaranteed. But I even more strongly believe that offhanded dismissals of the possibility of rapid liftoff happening very soon are deeply mistaken
The pro-pause faction lacks credibility if it is just a reaction against change--in fact, one of it's best arguments is that, because the good AI future could be so great, it's worth sacrificing a little time now to dramatically increase our chances of actually getting there.
We won't be sure that alignment worked (until its too late to fix it if it goes wrong, at least under the current paradigm)--but we can be more or less confident. In your linguistic analogy, if I move to Shanghai I will never be certain that I understand the people around me exactly right, but it would still be a good idea to study Chinese first.