Human beings whose emotional centres are damaged, even if their intelligence is still intact, have terrible decision-making skills. Whatever role emotions are playing in humans, it's necessary for agency. Ilya speculates that the equivalent for AIs is something to do with value functions - and that it might not emerge through pre-training alone.