Lisan al Gaib@scaling0115h ago

Ad - Meta provided free access to Muse Wonder models. The tests and conclusions are my own. This is a massive update for LisanBench: - I finally bit the bullet and tested multiple reasoning levels. We now have some sweet sweet data. - Muse Wonder Full and Muse Wonder perform extremely well on LisanBench and have crazy scaling and very even spacing between reasoning efforts. Just based on scores both Muse models with max reasoning effort rank #1 and #2. However, they are still slightly less efficient than Anthropic's Opus 5.5 - DeepSeek-V4.1-Flash 1017 is also very impressive and on the perf-cost frontier together with Haiku 5.5. - Notably Gemini 4 Argon low and medium, as well as Sonnet 5.5 low and medium produce no reasoning tokens and respond instantly, but still score high. Gemini 4 Argon low takes the #1 spot on the non-thinking leaderboard (it's a dirty looper). - Interestingly GPT-6.1-Sol, and GPT-6 models perform poorly as they are not good at following the edit-distance one rule. Argon high shows similar behavior. For GPT models I think the reason could be related to their very terse reasoning style. But since I haven't seen Gemini 4 Argon's raw reasoning I'm not super confident in that theory. Another explanation could be that it's related to dirty loopers. - When scoring the models without the edit-distance one rule (all the other rules still active) the ranking changes completely: 1. GPT 6 Astra medium: 56,140 (+659%) 2. Gemini 4 Argon high: 51,312 (+579%) 3. GPT 6.1 Sol medium: 49,887 (+731%) - The worst offenders in terms of cheating are Gemini 4 Argon and DeepSeek V4.1 Flash 1017. Gemini exposes an entirely new bug in the scorer using non-ASCII characters and DeepSeek has multiple longer chains with up to 64 consecutive two-letter words. When I deliberately RL-hillclimbed LisanBench I saw the same behavior of collapsing word lengths. Anthropic models also seem very optimized for this game, but in less obvious ways. They keep reusing the same long chains across multiple different starting words. - The minimum cost of reaching 2000, 5000 and 10000 LisanBench points has declined by 22-25% per month or halving every 2.4-2.8 months