In all my recent conversations with peers about AI, this idea – we’re all letting the agents cook and we don’t understand, but we’re cool with it – has come up. There’s a novelty factor that makes it easy to overlook what’s happening to our brains. Maybe it’s ok. I don’t really understand how plumbing and electricity get made, but I like the world we have because of them.
But the real threat isn’t either of those things. It’s quieter, and more boring, and therefore more dangerous. The real threat is a slow, comfortable drift toward not understanding what you’re doing. Not a dramatic collapse. Not Skynet. Just a generation of researchers who can produce results but can’t produce understanding. Who know what buttons to press but not why those buttons exist.
