Progress in machine learning is bottlenecked by having good evals, interpretability is no exception. We've made WorkspaceBench: a range of scenarios where we know what the model should be thinking about, to see if your interp tool can find it J-Lens is great but only outputs a single token. I think a good multi-token J-Lens should do well on WorkspaceBench!

camila_blank@camila_blank7h ago

Astra can do crazy things without chain-of-thought. This is scary, but interpretability can help! We made WorkspaceBench, an eval that tells you how well your interp tool surfaces the contents of the global workspace, i.e. the important intermediate variables in a forward pass. https://t.co/fTYZNXb9o2