I'm very worried about changes to AI architectures that result in AIs thinking in opaque activations instead of in chain of thought (aka "neuralese" architectures). Based on limited public evidence, it seems like Astra was a concerning step in this direction. Unfortunately, there was insufficient public information to have a well-informed public scientific discussion about whether the changes to architectures and training methods that went into Astra were a good trade-off between…

redwood_ai@redwood_ai2h ago

Some architectures could weaken CoT monitorability, or remove the CoT altogether. We've written a proposal for how companies could be transparent about no-CoT reasoning abilities, other monitorability evidence, and policies for preserving monitorability. https://t.co/32UC0LllOp