How big of a problem is benchmark contamination? This paper trains models from scratch with controlled amounts of test leakage, then studies how further training and sampling change its effects. A 344M model trained with 100 copies of the MATH test set scores 98.96% on the original questions, but 2.80% on reworded versions and 2.51% when the numbers change. Even keeping the original questions, changing from greedy decoding to temperature 1 drops accuracy to 17.58%. Further pretraining on…