Press Space to continue
Language models plan ahead while writing one token at a time
Press Space to continue
My biggest takeaways from @AnthropicAI interpretability researcher @mlpowered: 1. Predicting the next token requires understanding the universe. Language models got better than humans at predicting the next token around 2019. Once a model masters the easy tokens (sentence structure, then basic facts), the only ones left are hard, like the final paragraph of a PhD thesis. To finish “The capital of the state that contains Dallas is . . .,” a model has to know that Dallas is in Texas and that…
Lenny Rachitsky@lennysan6d ago
"To make them is not to understand them." Emmanuel Ameisen (@mlpowered) has spent the last 3 years doing interpretability research at @AnthropicAI, looking inside LLMs like Claude to figure out what they're actually doing. As he puts it, models are "grown," not built. We discuss: 🔸 His discovery that models plan many tokens ahead 🔸 Why anthropomorphizing AI can actually be the right approach 🔸 What the "J-space" reveals about how models think 🔸 How interpretability connects to AI alignment and safety Listen now 👇 https://youtu.be/JHo-aiePjoM