MTS@MTSliveAug 25

Jerry Tworek argues the Transformer architecture itself is now the bottleneck to faster AI progress: "Do the planes we have today look like planes we had five years after the first flight? Do the computers we have today look like the first computers? They have the same fundamentals, but the technology undergoes rapid change, especially rapid change in the early days of the field." "What I believe is true, that at this moment, the bottleneck to faster progress in AI technology is the architecture itself, is the transformer. We've been riding this for many years. It's extremely successful." "But I think it's very likely there are better architectures that we can pour a lot of compute in, and they can give us something that transformer has a hard time doing." "The North Star is about models that learn at test time, models that learn from user interaction, from user data. We think the way to achieve that is through new architectures." @MillionInt @CoreAutoAI