In short: reality is a messy place.
I think we overrated intelligence, or misread what it is.
Here’s how I’d define it. We hold a thin scatter of facts about the world, and intelligence or reasoning is whatever fills the space between them. It extrapolates between things we already know, and LLMs are incredible at that. In studying our words, they learned the structure of our thoughts. We will soon have it in unlimited supply, at 200 IQ, for close to nothing.
Where the space between the facts behaves well, this is close to godlike. Real estate law isn’t hard. We made it up, it’s internally consistent, and a model holding every statute and ruling should beat any lawyer for free. Coding, math and most administrative work are similarly benign. What makes them easy is that they have relatively smooth solution spaces and are tractably verifiable.
Most of what matters doesn’t behave like that. The universe is mostly the emergent behavior of complex systems. Stir cream into coffee. Watch a storm build out of nothing but atmospheric temperature differences and water vapor. The local rules are simple and the tornado is not. In systems like these, no amount of reasoning delivers the answer, because there’s no shortcut hiding in the gaps. You have to run the thing. And running it has a ceiling of its own, since small errors compound, which is why no supercomputer will ever give you a clean two-week forecast. The smallest object that can perfectly simulate the universe is the universe. Human systems are similarly complex. We can’t even predict our own behaviors reliably, let alone the emergent properties of large groups of people acting over time.
That, I suspect, is why the [AI] explosion never came. The takeoff story assumed the limit was thinking. It isn’t. The limit is contact with reality. A smarter reasoner fills the gaps between known facts in simple areas faster and better, but it doesn’t produce new facts.
Clifford Sosin – Maybe Intelligence Ain’t All That (HT Noah Smith’s What will more intelligence actually do for us?)
Harold Abelson said it like this in 1986:
A large airliner is an extremely complex system, and the aeronautical engineers who design that are dealing with immense complexity. But there’s a difference between that kind of complexity and what we deal with in computer science, and that is that computer science in some sense isn’t real.
You see, when an engineer is designing a physical system that’s made out of real parts, the engineers who worry about that have to address problems of tolerance and approximation and noise in the system. So for example, as an electrical engineer I can go off and easily build a one-stage amplifier, or a two-stage amplifier, and I can imagine cascading a lot of them to build a million-stage amplifier but it’s ridiculous to build such a thing because long before the millionth stage the thermal noise in those components way at the beginning is going to get amplified and make the whole thing meaningless.
Computer science deals with idealised components. So we know as much as want about these little program and data pieces that we’re fitting things together. So we don’t have to worry about tolerance. And that means that in building a large program, there’s not that much difference between what I can build and what I can imagine, because the parts are these abstract entities that I know as much as I want, I know about them as precisely as I’d like.
Harold Abelson – Structure and Interpretation of Computer Programs, Lecture 1 (full quote here)