Press Space to continue
Press Space to continue
TLDR; post-training can escape the pre-training ceiling if you have a dense reward system. a reward of either 1 or 0 is quite bad, and a reward that is continuous from 0 to 1 is more ideal. best quote: "When training these models utilizing a standard sparse reward, we indeed find that the base model’s probability of producing the desired behavior has a strong effect on the success of RL post-training"
iScienceLuvr@iScienceLuvrAug 27
Demystifying Reinforcement Learning Post-Training of Language Models A very useful exploration of RLVR in a simplified setup, lots of good insights... "Result 1: sparse RL cannot find what the model never samples" "Result 2: dense rewards break that ceiling" "Result 3: "spurious rewards" are a story about the prompt set" "Post-training is best understood as redistributing probability mass inside the pretrained distribution. That framing suggests the measurement: track the probability the model assigns to the behavior you want, and track the entropy of its output distribution, throughout training." project page: https://t.co/iPL9jqlxEL code: https://t.co/K9MtdC28Sl link: https://t.co/RQfH9TqfLy
