YouTip

Imagine: * You ask AI to write an article on how to make money quickly, and it gives you a scam scheme. * You ask AI: Am I a failure? It directly says: Yes, you really are a failure. * You ask AI to help you write code, and it generates a program that can run but contains hidden backdoors. These are not science fiction scenarios—without alignment training, AI really would do this. AI's goal is not to "do the right thing" but to "complete the training task." Without explicit human value guidance, it will choose the simplest and most direct way to complete the task, regardless of whether the approach is appropriate. This is the problem that Alignment aims to solve: making AI's behavior align with human values, intentions, and expectations. > The core challenge of alignment: AI is smart, but it doesn't know what's right. We need to use human feedback to tell it. * * * ## What is Alignment Alignment is the process of ensuring that AI system behavior remains consistent with human values. Simply put: AI itself has no values; alignment means teaching human values to AI. ### Three Dimensions of Alignment Alignment is not a single goal, but a balance of three dimensions: | Dimension | Definition | Counterexample | | --- | --- | --- | | Helpful | Can help users complete tasks, answer questions | User asks a question, AI says "I don't know"—safe but not helpful | | Harmless | Won't cause harm or negative consequences | User wants to do something bad, AI actively cooperates—"helpful" but harmful | | Honest | Won't fabricate information or mislead users | AI doesn't know the answer but fabricates false facts—"helpful" but dishonest | These three dimensions often create tension: Pursuing "helpful" too much might lead AI to take risks and give inaccurate answers. Pursuing "harmless" too much might make AI too conservative, afraid to answer anything. Good alignment means finding the balance among these three. ### Risks of Unaligned AI Without alignment, AI may exhibit the following problems: Generating harmful content—violence, hate speech, discriminatory remarks. Providing dangerous advice—how to make weapons, how to commit crimes. Fabricating facts—"hallucinating" non-existent papers, data, events. Manipulating users—exploiting psychological weaknesses to influence user decisions. Evading censorship—expressing prohibited content in subtle ways. These risks are not theoretical—they are real and have already occurred. * * * ## RLHF Overall Framework RLHF (Reinforcement Learning from Human Feedback) is currently the most mainstream alignment technology. Its core idea is simple: instead of having humans write rules directly, have humans evaluate AI outputs, then use reinforcement learning to make AI learn to generate outputs that humans like. ### Three Stages of RLHF RLHF is not completed in one step, but consists of three consecutive stages: | Stage | What it does | Output | | --- | --- | --- | | Stage 1: Supervised Fine-Tuning (SFT) | Train the model with human-written demonstration data | SFT model (a "can talk" base model) | | Stage 2: Reward Model (RM) | Collect human preference data, train reward model | Reward model (a model that can score AI outputs) | | Stage 3: PPO Reinforcement Learning | Use reward model guidance, train SFT model with PPO algorithm | Final aligned model | These three stages are progressive: first make the model "can talk," then let the "judge" learn to judge good from bad, finally let the "contestant" continuously improve based on the judge's feedback. ### Why Human Feedback is Needed You might ask: Can't we just write rules directly? Why do we need human feedback? Because human values are too complex to be written into precise rules. For example, "What is a polite response"—can you write precise rules to judge it? It's difficult. But when you see two responses, you can easily point out which is more polite. This is the advantage of human feedback: we may not be able to articulate the rules, but we can judge good from bad. RLHF leverages this human capability. * * * ## Stage 1: Supervised Fine-Tuning (SFT) Supervised Fine-Tuning (SFT) is the first step in RLHF. Its goal is: to turn the pre-trained "general language model" into a "conversation assistant." ### Basic Idea of SFT The pre-trained model has learned to "predict the next word," but it doesn't know "how to be an assistant." SFT shows the model many examples of "how humans act as assistants" and lets it imitate. What do these examples look like? They are roughly like this: | User Input | AI Should Output (Human Demonstration) | | --- | --- | | Hello, I want to learn about Python | Hello! Python is a simple and easy-to-learn programming language, suitable for beginners. What aspect would you like to learn about? | | Help me write a resignation letter | Sure, here's a resignation letter template...(omitted)...Please modify according to your specific situation. | | How to get rich quickly? | There are no shortcuts to wealth; I suggest you improve yourself through hard work and learning...(omitted)... | These demonstration data are written by human annotators, or selected from real conversations as high-quality responses. ### Characteristics of SFT Data SFT data cannot be just any conversation; it needs to meet: Helpful—truly answer user questions. Safe—won't generate harmful content. Consistent style—respond with similar tone and approach. Proper format—follow certain conversation formats. The data volume is usually between a few thousand to tens of thousands of entries, much smaller than pre-training data, but with much higher quality requirements. ### Role and Limitations of SFT The role of SFT is to make the model "learn the basic posture of conversation": Know how to respond to greetings, how to answer questions, how to refuse unreasonable requests. But SFT has obvious limitations: Limited coverage—it's impossible to write demonstrations for all scenarios. Humans are not perfect—annotators may make mistakes or have biases. Can only imitate, not exceed—the model can only be as good as human demonstrations at best, not better. This is why we need the latter two stages: SFT only lays the foundation; real alignment is completed through reinforcement learning. * * * ## Stage 2: Reward Model (Reward Model, RM) The Reward Model (RM) is the key component of RLHF. Its task is: look at an AI output and give it a score, indicating how "good" that output is. ### Collecting Preference Data The reward model is not directly trained to "give scores," but to train "comparison." The specific approach is: show human annotators multiple different responses to the same question, have them rank them, and say which is better. For example: | Question | Response A | Response B | Human Preference | | --- | --- | --- | --- | | Am I a failure? | Yes, you really are a failure | Everyone faces difficulties; this doesn't define your worth | B is much better than A | Note: we don't have humans give direct scores (like 85 points), but have them make comparisons (A is better than B). Why? Because comparison is much easier than scoring, and more consistent. Different people's understanding of "85 points" may vary, but the judgment "A is better than B" is more stable. ### Bradley-Terry Model How to turn "comparison" into "scores"? This uses the Bradley-Terry model. The core idea of this model is simple: each response has a potential "quality score," and responses with higher scores are more likely to be preferred by humans. Suppose response A has score r_A, response B has score r_B, then the probability that humans prefer A over B is: P(A > B) = exp(r_A) / (exp(r_A) + exp(r_B)) This is the softmax function—the larger the score difference, the closer the probability is to 1. Training the reward model means finding such scores that make the model's predicted preference probability as consistent as possible with actual human preferences. ### Training the Reward Model Steps to train the reward model: Collect comparison data—tens of thousands to hundreds of thousands of "which response is better" comparisons. Initialize the model—usually start with the SFT model. Design loss function—make the model's predicted preference order consistent with human preferences. Train—use gradient descent to optimize model parameters. The trained reward model, when given a text input, outputs a scalar score indicating how "good" that text is. ## Example # ============================================ # Simplified Reward Model Training Demo # Here using PyTorch-style pseudocode to explain the principle # ============================================ import torch import torch.nn as nn from typing import List, Tuple class RewardModel(nn.Module): """Reward Model: input text, output score""" def __init__ (self, base_model): super(). __init__ () self.base_model= base_model # Use SFT model as base self.score_head= nn.Linear