Press Space to continue
Press Space to continue
Fable has a noticeably higher Safety Refusal Rate than Astra, per the new Artificial Analysis data. interestingly, we saw today how a combination of Opus 4.8 and 5 were used to hack OpenAI, and open models will likely get there in 2-5 months, with little to no guardrails. if frontier labs make their smartest models too constrained for legitimate security work, there is a risk of creating a dangerous asymmetry where bad actors get ~frontier open models and defenders get blocked by safety…
ArtificialAnlys@ArtificialAnlys3h ago
Safety refusal reporting is now available in the Artificial Analysis Coding Agent Index In our latest Coding Agent Index v1.5, we’ve introduced safety refusal reporting to help explain model behavior and score differences. A safety refusal occurs when a provider or model declines to start or continue a task on safety grounds. An agent may fall back to another model to continue, or stop the attempt with a block. Claude Fable 5.1 had the highest fallback rates in both Claude Code and Devin Fusion, with fallback attempts accounting for 8.8% and 7.1% of the Index's weight, respectively; these results therefore include the fallback models' performance. Refusal variability, harness context buildup, effort settings, and retry strategies can all affect the observed rates.
