“Score Centering Stabilizes Off-Policy Reinforcement Learning” Small mismatches between the model generating rollouts and the model being trained can compound over RL and eventually collapse training. And this mismatch creates a small directional bias in every policy gradient update, repeatedly pulling the model toward its slightly different sampler. So this paper, score centering, estimates this unwanted pull and subtracts it from every update, keeping the useful reward signal while…