This is a good post and I think people should take seriously the points it raises and think more about them. I would just like to make additional comments on the observation that LLM intelligence seems very different from human intelligence, because it's something I have been thinking about a lot as a result of using LLMs and one reason I think AGI may be further away than what many people assume. One of the main things I use LLMs for, besides coding, is to help me formalize my intuitions about a phenomenon into a model. I also use them a lot to learn stuff by asking them questions and iterating from their answers until I'm satisfied that I understand everything I wanted to know. And they're extraordinarily useful for that, but at the same time, I think it has shown me they still have limitations. What I have found is that, in a sense that is not easy to explain, LLMs still lack understanding. They're very good at helping me find mathematical tools to model the mechanisms I have in mind and working through the derivations, but they're still limited on the deeper conceptual issues. At some level, they just go through the motions, but still lack the ability to take a step back, think about what they're doing and why they're doing it. For instance, they are now able to solve complicated mathematical problems, but my sense is that they still don't really understand, at a deeper level, why those problems are interesting in the first place. Similarly, if you are trying to model something, they are incredibly useful if you have put them on the right tracks by framing the phenomenon you're trying to model in the right way, but if you haven't they are still not very good at pointing out that you are not thinking about the issue in the right way. If the basic approach you are using to model the phenomenon of interest was misguided in the first place, they will help you work out the technical aspects of that misguided framework by deriving mathematical results within that framework, but often they won't understand that it's just the wrong way to think about the whole thing and will actually help you build castles in the air instead of pointing out that you're not thinking about the issue in the right way. For instance, a few months ago I wanted to understand crude oil demand better, so I started working on a microfounded model for it. My initial approach was to think about oil users as having a menu of possible discrete uses of crude, each of them with a reservation price. Demand was generated by summing the quantities associated with oil uses whose reservation price exceeded the oil price. GPT helped me create a formal model based on this idea and derive various technical results within that model, but eventually I just realized that it was a fundamentally flawed way to think about the microfoundations of oil demand and that a better way to think about it was that, instead of considering a menu of possible discrete oil uses, individual crude processors choose continuous production levels and product mixes so as to maximize profit subject to fixed short-run production constraints. Once I made that conceptual switch, GPT was again incredibly useful to work out the details of the model and derive various results within it, but the point is that the switch had to come from me. GPT never pointed out that it was the wrong way to think about the microfoundations of oil demand, even though I think it clearly was. That's just an example but I keep running into similar issues, which I see as showing that, in a sense that is difficult to spell out precisely, LLMs still lack understanding and are still limited on deeper conceptual issues. It's hard to tell to what extent they are getting better on that sort of things, because it's such a difficult to measure thing, but my sense is that progress has been more limited than on many other things. The way in which I explain why it seems harder to improve their capabilities on this kind of conceptual tasks is that it's difficult to define a reward function for that kind of things, because the success criteria for them are hard to spell out, in part because people often disagree about what's the right way to frame an issue conceptually. Anyway, I'm still working through those issues and my ideas on the topic remain very disorganized, but it's something I have been thinking about based on my experience with using LLMs and I think it's related to some of the points Andrew makes below. He makes other points that are more relevant to the economic impact of LLMs or lack thereof that I think are also valid, but I wanted to talk about the points that pertain to the debate about AGI and how close we are to it.