RLHF (Reinforcement Learning from Human Feedback)

RLHF is how a raw pre-trained model (which just predicts next tokens) is turned into the helpful, instruction-following assistant you interact with. The process has two stages: (1) Supervised fine-tuning on a dataset of good human-written responses. (2) Reward modeling + RL: humans compare pairs of model outputs and rank them, training a reward model on those preferences, then using RL to push the main model toward higher-reward outputs.

The results are dramatic — GPT-3 was barely usable as a chat assistant; InstructGPT (RLHF on GPT-3) was what launched the AI wave. The downside: RLHF is expensive (human labelers), can introduce sycophancy (models learn to say what gets approved rather than what's correct), and requires continuous updates as deployment reveals new failure modes.

Modern variants (DPO, RLAIF) replace or augment the human labeling step, but the core idea — optimize model behavior against a human preference signal — remains the foundation.