Highly recommended read on looped and depth-grown models. There is a lot of interest in looped and depth-grown language models for reasoning. The question here is what reusing layers does to in-context learning. At 3B parameters, both grown and looped models gain more from later context than a dense model of the same depth. They score higher on GSM8K at every number of demonstrations and predict unseen formal languages more accurately. In controlled tasks, they pick up in-context learning…