brettworks

“nobody—not even the people who train AI models for a living—knows exactly how large language models work” (226).

“There was no end. Nobody would ever agree on what AGI was, or when we’d reached it. There were too many conflicting ideas and moving goalposts, and no universally accepted definition of the thing they were all sprinting toward, which meant they could never stop” (410).

“[Philosopher David Chalmers] now predicted that humans would become ‘part AI,’ and would draw on machine superintelligence to help them realize their dreams. ‘You could imagine doing a little simple drawing and giving it to Michelangelo, and saying Hey, Michelangelo, can you make a Sistine Chapel out of this simple drawing? And then you go and look at the thing Michelangelo did, and you see, Wow, it is my drawing, but every little place I look there is more depth and more stories. I think that is what the part AI could feel like’ he said” (435)

“Large language models (or whatever we’re calling the class of chatting, reasoning, agentic-coding contraptions that include ChatGPT, Claude, and Gemini) were also trained to do a thing: predict the next token. But in the course of doing that thing, they learned to do other things. Poetry, cookie recipes, JavaScript, multivariable calculus—it was all in there, inside the vast compendium of human knowledge and creativity that the researchers chucked into their models—and it all got swept up into the next-token prediction machines. The bigger they made these machines, the more general they got. When they ran out of data, they bolted reinforcement learning loops on top of the machines, taught them to think step-by-step, threw them at new tasks, and saw that their circle of competence kept expanding. If you doubt that this method of creating generality via next-token prediction—this inexplicable, cosmic accident—has managed to produce something like human intelligence, well, that’s your right. I won’t argue. But I can tell you that if you took your laptop in a time machine back to 2017 and gave any of the ten thousand brightest minds in computer science a chance to fiddle around with any of the 2026 models for even five minutes, they would have erupted in hysterical disbelief” (449).

“My bar is this: I believe AGI is a generally capable AI system that does useful things for me, and that is at least as intelligent, truthful, and reliable as a random stranger picked from a crowd” (450).

“Scaling up next-token prediction created a new type of alien mind whose intelligence mirrored ours in some ways and diverged wildly from it in others. As a result, we reap the benefits of their cognitive diversity, and we have the chance to learn from them, as they’ve learned from us” (451).

Kevin Roose, The AGI Chronicles (2026)