S_s_batzoglou@s_batzoglouJul 12
I benchmarked the new models (Sol, Terra, Luna, Fable 5, Meta Muse Spark 1.1, Grok 4.5) on an induction reasoning task.
This is an updated table from yesterday, and my benchmark is described as spotlight in ICML '26.
How to read the table:
The models are given several small graphs (6-10 graphs, each with 8-10 elements), in which some nodes are designated. They must return a first-order formula that describes properties of the designated nodes simultaneously for all graphs.
The benchmark has 64 such problems (a subset of the bigger benchmark).
Correct: how many formulas correctly describe the designated nodes.
Holdout correct: a number of additional graphs, with the same underlying property, are held out. This is to test the ability of the model's correct formulas to generalize. For instance, if a formula is a big case split rather than the underlying simple graph property, it won't generalize.
Formula complexity (in AST): mean and median size of the correct model's formulas.
A few observations and experiment notes:
- All new models are remarkably good. Even Luna and Muse Spark outperforms GPT 5.4.
- Some models are better in returning simple formulas that generalize well.
- Fable 5 was extremely hard to get results from. I first ran it with higher thinking effort levels, in which case it charged for max tokens per problem and returned all "" responses. The only settings in which it returns non-"" responses are medium and low. My method for running it was to call it on medium, then call it again on medium, then on low.
- GPT 5.5 is absent because it had the same non-return behavior as Fable 5 above, but even worse. So bad that I couldn't get almost any results and would have wasted too much $$ on empty responses. Happy to see that the new OpenAI models are "cured" of this GPT 5.5 issue.
- I had observed that Grok 4 >> Grok 4.1 Fast > Grok 4.20 >> Grok 4.3. Finally, this downward trend of XAI has been reversed and Grok 4.5 is now a contender.
- Meta Muse Spark 1.1 is a big surprise. It beat GPT 5.4 on this task!
- The paper is out and the benchmark is public: https://t.co/9uDRTWsBar