MillionInt@MillionInt1d ago

Alignment is not as hard to solve as many claim, but it is in the end an algorithmic problem which a lot of ML community mostly stopped working on. Formulation of alignment stated by three (or four) laws of robotics can take us very far, so we roughly know the objective. The tricky part is, how do we take gradient with respect to alignment? We have two algorithms right now at our disposal: pretraining and RL. Pretraining takes gradient with respect to next token prediction - that's def not the alignment objective. We could create RL environments that embody the alignment objective, but: - those are expensive to create, so often cheaper, hackable proxies are used in practice - RL as an objective needs successful and unsuccessful rollouts to happen to take the gradient step. We DO NOT want to harm any humans in the process of aligning our modes - this is a pretty big problem. Therefore there are two solutions forward for the alignment problem: - either we RL models in a simulated environments with simulated alignments and decreasing the likelihood of harming simulated humans, which will never be perfect - or we create a new algorithm that can teach our models to not harm humans without harming any humans in the process Science is the process how we solve the hardest problems ahead and that is one of them