Reinforcement Learning Terminology for LLM Readers: How a Single Score Is Distributed to Each Token
The article uses DeepSeek-R1-Zero's pass@1 on AIME 2024 rising from 15.6% to 71.0% as an example to show that a rule-based judge gives only a single score for the entire response, whereas training needs to adjust the probability of each token. The author uses the sampling example "Is OpenAI open?" to explain in turn policy, state, action, reward, return, value, Q, advantage, baseline, TD error, GAE, clipping, and KL, and points out that PPO uses a critic to learn V, while GRPO uses the average score within a group to replace V.
The article uses DeepSeek-R1-Zero's pass@1 on AIME 2024 rising from 15.6% to 71.0% as an example to show that a rule-based judge gives only a single score for the entire response, whereas training needs to adjust the probability of each token. The author uses the sampling example "Is OpenAI open?" to explain in turn policy, state, action, reward, return, value, Q, advantage, baseline, TD error, GAE, clipping, and KL, and points out that PPO uses a critic to learn V, while GRPO uses the average s
Matched fieldsBody