"A Fix for Softmax and Cross-Entropy" Turns out two of the most fundamental functions in deep learning have numerical bugs that can actually destabilize training. This paper proposes simple fixes to softmax and cross-entropy that prevent tiny gradients from incorrectly rounding to zero, or remaining nonzero when they should vanish, as these errors can cause sudden loss spikes and interfere with knowledge distillation. On small-scale grokking benchmarks, the fixes eliminate Slingshot loss…