{"id":58994,"date":"2025-08-16T19:45:01","date_gmt":"2025-08-16T11:45:01","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/58994.html"},"modified":"2025-08-16T19:45:01","modified_gmt":"2025-08-16T11:45:01","slug":"grpo%ef%bc%88group-relative-policy-optimization%ef%bc%89%e5%85%ac%e5%bc%8f%e9%80%9f%e8%a7%88","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/58994.html","title":{"rendered":"GRPO\uff08Group Relative Policy Optimization\uff09\u516c\u5f0f\u901f\u89c8"},"content":{"rendered":"

GRPO（Group Relative Policy Optimization）\u516c\u5f0f\u901f\u89c8<\/h2>\n

\u628a 1600 \u884c\u6e90\u7801\u6d53\u7f29\u6210\u4e00\u9875\u53ef\u6284\u8fdb\u8bba\u6587\u7684\u516c\u5f0f\u8868\u3002<\/p>\n

\n

1 \u7ec4\u5185\u5f52\u4e00\u5316\u4f18\u52bf（Group-Relative Advantage）<\/h3>\n\n\u7b26\u53f7\u542b\u4e49\u4ee3\u7801\u53d8\u91cf<\/tr>\n\n\n\n\n\n
q<\/td>\nprompt<\/td>\nprompt<\/td>\n<\/tr>\n
G<\/td>\n\u7ec4\u5927\u5c0f<\/td>\nnum_generations<\/td>\n<\/tr>\n
o_i<\/td>\n\u7b2c i \u6761 completion<\/td>\ncompletions[i]<\/td>\n<\/tr>\n
r(q,o_i)<\/td>\n\u5956\u52b1<\/td>\nrewards[i]<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

\u03bcr(q)=1G\u2211i=1Gr(q,oi)\u03c3r(q)=stdi=1..G\u2009r(q,oi)Aq,oi=r(q,oi)\u2212\u03bcr(q)\u03c3r(q)+\u03b5(\u03b5=1\u00d710\u22124)
\n\\\\boxed{
\n\\\\begin{aligned}
\n\\\\mu_r(q) &= \\\\frac{1}{G}\\\\sum_{i=1}^{G} r(q, o_i) \\\\\\\\
\n\\\\sigma_r(q) &= \\\\text{std}_{i=1..G}\\\\, r(q, o_i) \\\\\\\\
\nA_{q,o_i} &= \\\\frac{r(q, o_i) – \\\\mu_r(q)}{\\\\sigma_r(q) + \\\\varepsilon} \\\\quad (\\\\varepsilon = 1\\\\times10^{-4})
\n\\\\end{aligned}
\n}
\n<\/span><\/span><\/span><\/span>\u03bc<\/span><\/span>r<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span>q<\/span>)<\/span><\/span><\/span><\/span>\u03c3<\/span><\/span>r<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span>q<\/span>)<\/span><\/span><\/span><\/span>A<\/span><\/span>q<\/span>,<\/span>o<\/span><\/span>i<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span><\/span><\/span>G<\/span><\/span><\/span><\/span><\/span><\/span><\/span>1<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>i<\/span>=<\/span>1<\/span><\/span><\/span><\/span><\/span>\u2211<\/span><\/span><\/span><\/span>G<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>r<\/span>(<\/span>q<\/span>,<\/span><\/span>o<\/span><\/span>i<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span>std<\/span><\/span><\/span>i<\/span>=<\/span>1..<\/span>G<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>r<\/span>(<\/span>q<\/span>,<\/span><\/span>o<\/span><\/span>i<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span><\/span><\/span>\u03c3<\/span><\/span>r<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span>q<\/span>)<\/span><\/span>+<\/span><\/span>\u03b5<\/span><\/span><\/span><\/span><\/span><\/span><\/span>r<\/span>(<\/span>q<\/span>,<\/span><\/span>o<\/span><\/span>i<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span>\u2212<\/span><\/span>\u03bc<\/span><\/span>r<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span>q<\/span>)<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span>\u03b5<\/span><\/span>=<\/span><\/span>1<\/span><\/span>\u00d7<\/span><\/span>1<\/span>0<\/span><\/span>\u2212<\/span>4<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

\n

2 \u7b56\u7565\u88c1\u526a\u76ee\u6807（Token-Level）<\/h3>\n

ri,t(\u03b8)=\u03c0\u03b8(oi,t\u2223q,oi,<t)\u03c0\u03b8old(oi,t\u2223q,oi,<t)Lclip(\u03b8)=\u2211i=1G\u2211t=1\u2223oi\u2223min\u2061\u2009\u2063(ri,t(\u03b8)\u2009Aq,oi,\u2005\u200aclip(ri,t(\u03b8),\u20091\u2212\u03b5low,\u20091+\u03b5high)\u2009Aq,oi)
\n\\\\boxed{
\n\\\\begin{aligned}
\nr_{i,t}(\\\\theta) &= \\\\frac{\\\\pi_\\\\theta(o_{i,t}\\\\mid q, o_{i,<t})}{\\\\pi_{\\\\theta_{\\\\text{old}}}(o_{i,t}\\\\mid q, o_{i,<t})} \\\\\\\\[6pt]
\nL_{\\\\text{clip}}(\\\\theta) &= \\\\sum_{i=1}^{G}\\\\sum_{t=1}^{|o_i|} \\\\min\\\\!\\\\Bigl(
\n r_{i,t}(\\\\theta)\\\\,A_{q,o_i},\\\\;
\n \\\\text{clip}(r_{i,t}(\\\\theta),\\\\,1-\\\\varepsilon_{\\\\text{low}},\\\\,1+\\\\varepsilon_{\\\\text{high}})\\\\,A_{q,o_i}
\n\\\\Bigr)
\n\\\\end{aligned}
\n}
\n<\/span><\/span><\/span><\/span>r<\/span><\/span>i<\/span>,<\/span>t<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span>\u03b8<\/span>)<\/span><\/span><\/span><\/span>L<\/span><\/span>clip<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span>\u03b8<\/span>)<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span><\/span><\/span>\u03c0<\/span><\/span>\u03b8<\/span><\/span>old<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span>o<\/span><\/span>i<\/span>,<\/span>t<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u2223<\/span><\/span>q<\/span>,<\/span><\/span>o<\/span><\/span>i<\/span>,<\/span><<\/span>t<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u03c0<\/span><\/span>\u03b8<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span>o<\/span><\/span>i<\/span>,<\/span>t<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u2223<\/span><\/span>q<\/span>,<\/span><\/span>o<\/span><\/span>i<\/span>,<\/span><<\/span>t<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span><\/span>i<\/span>=<\/span>1<\/span><\/span><\/span><\/span><\/span>\u2211<\/span><\/span><\/span><\/span>G<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>t<\/span>=<\/span>1<\/span><\/span><\/span><\/span><\/span>\u2211<\/span><\/span><\/span><\/span>\u2223<\/span>o<\/span><\/span>i<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u2223<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>min<\/span><\/span>(<\/span><\/span>r<\/span><\/span>i<\/span>,<\/span>t<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span>\u03b8<\/span>)<\/span><\/span>A<\/span><\/span>q<\/span>,<\/span>o<\/span><\/span>i<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span><\/span><\/span>clip<\/span><\/span>(<\/span>r<\/span><\/span>i<\/span>,<\/span>t<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span>\u03b8<\/span>)<\/span>,<\/span><\/span><\/span>1<\/span><\/span>\u2212<\/span><\/span>\u03b5<\/span><\/span>low<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span><\/span><\/span>1<\/span><\/span>+<\/span><\/span>\u03b5<\/span><\/span>high<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span>A<\/span><\/span>q<\/span>,<\/span>o<\/span><\/span>i<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

\u4ee3\u7801\u5bf9\u5e94：per_token_logps, old_per_token_logps, coef_1, coef_2<\/p>\n

\n

3 KL \u6b63\u5219\u9879（\u53ef\u9009，\u03b2>0 \u65f6\u542f\u7528<\/h3>\n

KLreg=\u03b2\u2005\u200aDKL\u2009\u2063[\u03c0\u03b8\u2009\u2225\u2009\u03c0ref]\\\\boxed{
\n\\\\mathrm{KL}_{\\\\mathrm{reg}} = \\\\beta \\\\; D_{\\\\mathrm{KL}}\\\\!\\\\bigl[\\\\pi_{\\\\theta}\\\\,\\\\|\\\\,\\\\pi_{\\\\mathrm{ref}}\\\\bigr]}<\/span><\/span><\/span>KL<\/span><\/span><\/span>reg<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span>\u03b2<\/span><\/span>D<\/span><\/span>KL<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>[<\/span><\/span>\u03c0<\/span><\/span>\u03b8<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u2225<\/span><\/span>\u03c0<\/span><\/span>ref<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>]<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

\u4ee3\u7801\u5bf9\u5e94：per_token_kl, beta<\/p>\n

\n

4 \u6700\u7ec8\u635f\u5931（Token-Level）<\/h3>\n

| GRPO | Ltotal=\u2212Lclip\u2211t1+KLreg\\\\displaystyle L_{\\\\text{total}} = -\\\\frac{L_{\\\\text{clip}}}{\\\\sum_t 1} + \\\\text{KL}_{\\\\text{reg}}<\/span><\/span>L<\/span><\/span>total<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span><\/span><\/span>\u2212<\/span><\/span><\/span>\u2211<\/span><\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>1<\/span><\/span><\/span><\/span><\/span><\/span><\/span>L<\/span><\/span>clip<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>+<\/span><\/span><\/span><\/span>KL<\/span><\/span><\/span>reg<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>
\n| BNPO | Ltotal=\u2212Lclip\u2211t1+KLreg\\\\displaystyle L_{\\\\text{total}} = -\\\\frac{L_{\\\\text{clip}}}{\\\\sum_t 1} + \\\\text{KL}_{\\\\text{reg}}<\/span><\/span>L<\/span><\/span>total<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span><\/span><\/span>\u2212<\/span><\/span><\/span>\u2211<\/span><\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>1<\/span><\/span><\/span><\/span><\/span><\/span><\/span>L<\/span><\/span>clip<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>+<\/span><\/span><\/span><\/span>KL<\/span><\/span><\/span>reg<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>
\n| DR-GRPO | Ltotal=\u2212LclipB\u22c5T+KLreg\\\\displaystyle L_{\\\\text{total}} = -\\\\frac{L_{\\\\text{clip}}}{B\\\\cdot T} + \\\\text{KL}_{\\\\text{reg}}<\/span><\/span>L<\/span><\/span>total<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span><\/span><\/span>\u2212<\/span><\/span><\/span>B<\/span><\/span>\u22c5<\/span><\/span>T<\/span><\/span><\/span><\/span><\/span><\/span><\/span>L<\/span><\/span>clip<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>+<\/span><\/span><\/span><\/span>KL<\/span><\/span><\/span>reg<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

    \n
  • \u4ee3\u7801\u7531 loss_type \u53c2\u6570\u5207\u6362\u3002<\/li>\n<\/ul>\n
    \n

    **BNPO vs GRPO：\u4e00\u53e5\u8bdd\u901f\u8bb0 **<\/p>\n

    BNPO = GRPO \u7684\u201c\u5956\u52b1\u5f52\u4e00\u5316\u5916\u6302\u201d
    \n\u4e8c\u8005\u5171\u7528\u540c\u4e00\u5957\u201c\u7ec4\u5185\u76f8\u5bf9\u4f18\u52bf + KL + clip\u201d\u6846\u67b6，\u53ea\u662f BNPO \u628a\u9759\u6001\u5747\u503c-\u65b9\u5dee\u6362\u6210\u4e86\u52a8\u6001 Beta \u5f52\u4e00\u5316\u3002<\/p>\n

    \n

    \u2705 \u6838\u5fc3\u5dee\u522b\u8868<\/h4>\n\n\u7ef4\u5ea6GRPOBNPO<\/tr>\n\n\n\n\n\n\n
    \u5f52\u4e00\u5316\u65b9\u5f0f<\/td>\n\u7ec4\u5185\u5747\u503c-\u65b9\u5dee（\u9759\u6001）<\/td>\nBeta \u5206\u5e03\u81ea\u9002\u5e94<\/td>\n<\/tr>\n
    \u5956\u52b1\u5047\u8bbe<\/td>\n\u4efb\u4f55\u6570\u503c<\/td>\n\u4e8c\u503c\u5956\u52b1 Bernoulli<\/td>\n<\/tr>\n
    \u57fa\u7ebf\u66f4\u65b0<\/td>\n\u6bcf\u6b21 batch \u91cd\u7b97 \u03bc, \u03c3<\/td>\n\u5b9e\u65f6\u66f4\u65b0 \u03b1, \u03b2 \u53c2\u6570<\/td>\n<\/tr>\n
    \u68af\u5ea6\u65b9\u5dee<\/td>\n\u56fa\u5b9a<\/td>\n\u968f\u7b56\u7565\u52a8\u6001\u51cf\u5c0f<\/td>\n<\/tr>\n
    \u662f\u5426 GRPO \u7684\u8d85\u96c6<\/td>\n\u2717<\/td>\n\u662f：GRPO \u662f \u03b2 \u56fa\u5b9a\u65f6\u7684\u7279\u4f8b<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    \n

    \u2705 \u516c\u5f0f\u5bf9\u7167（\u4e00\u53e5\u8bdd\u770b\u61c2）<\/h4>\n\n\u7b97\u6cd5\u4f18\u52bf\u51fd\u6570<\/tr>\n\n\n\n
    GRPO<\/td>\nAGRPO=r\u2212\u03bcr\u03c3r+\u03b5A_{\\\\text{GRPO}} = \\\\frac{r – \\\\mu_r}{\\\\sigma_r+\\\\varepsilon}<\/span><\/span>A<\/span><\/span>GRPO<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span><\/span><\/span><\/span><\/span>\u03c3<\/span><\/span>r<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>+<\/span>\u03b5<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>r<\/span>\u2212<\/span>\u03bc<\/span><\/span>r<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/td>\n<\/tr>\n
    BNPO<\/td>\nABNPO=r\u2212\u03bc^\u03bc^(1\u2212\u03bc^)+\u03b4A_{\\\\text{BNPO}} = \\\\frac{r – \\\\hat{\\\\mu}}{\\\\sqrt{\\\\hat{\\\\mu}(1-\\\\hat{\\\\mu})+\\\\delta}}<\/span><\/span>A<\/span><\/span>BNPO<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u03bc<\/span><\/span><\/span>^<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span>1<\/span>\u2212<\/span><\/span>\u03bc<\/span><\/span><\/span>^<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span>+<\/span>\u03b4<\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>r<\/span>\u2212<\/span><\/span>\u03bc<\/span><\/span><\/span>^<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> \u5176\u4e2d \u03bc^\u223cBeta(\u03b1,\u03b2)\\\\hat{\\\\mu}\\\\sim\\\\text{Beta}(\\\\alpha,\\\\beta)<\/span><\/span><\/span>\u03bc<\/span><\/span><\/span>^<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u223c<\/span><\/span><\/span><\/span>Beta<\/span><\/span>(<\/span>\u03b1<\/span>,<\/span><\/span>\u03b2<\/span>)<\/span><\/span><\/span><\/span><\/span>，\u53c2\u6570 \u03b1,\u03b2 \u7528\u6700\u8fd1 N \u6b65\u7684\u5956\u52b1\u5728\u7ebf\u4f30\u8ba1\u3002<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    \n

    \u2705 \u4f7f\u7528\u573a\u666f<\/h4>\n
      \n
    • GRPO：\u901a\u7528\u3001\u7b80\u5355\u3001\u5feb\u901f\u5b9e\u73b0\u3002<\/li>\n
    • BNPO：\n
        \n
      • \u5956\u52b1\u53ea\u6709 0\/1（\u6b63\u786e\/\u9519\u8bef）<\/li>\n
      • \u8bad\u7ec3\u521d\u671f\u5956\u52b1\u5206\u5e03\u6f02\u79fb\u5927（\u5982\u6570\u5b66\u63a8\u7406\u4efb\u52a1）<\/li>\n
      • \u9700\u8981\u66f4\u4f4e\u68af\u5ea6\u65b9\u5dee\u3001\u66f4\u9ad8\u7a33\u5b9a\u6027<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
        \n

        \u2705 \u7ed3\u8bba<\/h4>\n
          \n
        • BNPO \u4e0d\u63a8\u7ffb GRPO，\u53ea\u662f\u628a\u201c\u9759\u6001\u5747\u503c\u57fa\u7ebf\u201d\u5347\u7ea7\u4e3a\u201c\u52a8\u6001 Beta \u57fa\u7ebf\u201d\u3002<\/li>\n
        • \u5728 TRL \u4e2d\u53ea\u9700\u628a loss_type='bnpo' \u5373\u53ef\u542f\u7528，\u5176\u4f59\u6d41\u7a0b（\u91c7\u6837\u3001clip\u3001KL）\u5b8c\u5168\u4e00\u81f4\u3002<\/li>\n<\/ul>\n
          \n

          Huggingface TRL\u4e2d\u662f\u600e\u4e48\u5b9e\u73b0\u7684<\/h3>\n

          \u8ba1\u7b97reward<\/p>\n

          def<\/span> _calculate_rewards<\/span>(<\/span>self,<\/span> inputs,<\/span> prompts,<\/span> completions,<\/span> completion_ids_list)<\/span>:<\/span>
          \n device =<\/span> self.<\/span>accelerator.<\/span>device
          \n rewards_per_func =<\/span> torch.<\/span>zeros(<\/span>len<\/span>(<\/span>prompts)<\/span>,<\/span> len<\/span>(<\/span>self.<\/span>reward_funcs)<\/span>,<\/span> device=<\/span>device)<\/span><\/p>\n

          # Repeat all input columns (but "prompt", "completion", and "completion_ids") to match the num of generations<\/span>
          \n keys =<\/span> [<\/span>key for<\/span> key in<\/span> inputs[<\/span>0<\/span>]<\/span> if<\/span> key not<\/span> in<\/span> [<\/span>"prompt"<\/span>,<\/span> "completion"<\/span>,<\/span> "completion_ids"<\/span>]<\/span>]<\/span>
          \n reward_kwargs =<\/span> {<\/span>key:<\/span> [<\/span>example[<\/span>key]<\/span> for<\/span> example in<\/span> inputs]<\/span> for<\/span> key in<\/span> keys}<\/span><\/p>\n

          # This allows for dynamic reward shaping based on training progress.<\/span>
          \n reward_kwargs[<\/span>"trainer_state"<\/span>]<\/span> =<\/span> self.<\/span>state<\/p>\n

          for<\/span> i,<\/span> (<\/span>reward_func,<\/span> reward_processing_class,<\/span> reward_func_name)<\/span> in<\/span> enumerate<\/span>(<\/span>
          \n zip<\/span>(<\/span>self.<\/span>reward_funcs,<\/span> self.<\/span>reward_processing_classes,<\/span> self.<\/span>reward_func_names)<\/span>
          \n )<\/span>:<\/span>
          \n with<\/span> profiling_context(<\/span>self,<\/span> reward_func_name)<\/span>:<\/span>
          \n if<\/span> isinstance<\/span>(<\/span>reward_func,<\/span> nn.<\/span>Module)<\/span>:<\/span> # Module (no PretrainedModel) for compat with compiled models<\/span>
          \n if<\/span> is_conversational(<\/span>inputs[<\/span>0<\/span>]<\/span>)<\/span>:<\/span>
          \n messages =<\/span> [<\/span>{<\/span>"messages"<\/span>:<\/span> p +<\/span> c}<\/span> for<\/span> p,<\/span> c in<\/span> zip<\/span>(<\/span>prompts,<\/span> completions)<\/span>]<\/span>
          \n texts =<\/span> [<\/span>apply_chat_template(<\/span>x,<\/span> reward_processing_class)<\/span>[<\/span>"text"<\/span>]<\/span> for<\/span> x in<\/span> messages]<\/span>
          \n else<\/span>:<\/span>
          \n texts =<\/span> [<\/span>p +<\/span> c for<\/span> p,<\/span> c in<\/span> zip<\/span>(<\/span>prompts,<\/span> completions)<\/span>]<\/span>
          \n reward_inputs =<\/span> reward_processing_class(<\/span>
          \n text=<\/span>texts,<\/span> return_tensors=<\/span>"pt"<\/span>,<\/span> padding=<\/span>True<\/span>,<\/span> padding_side=<\/span>"right"<\/span>,<\/span> add_special_tokens=<\/span>False<\/span>
          \n )<\/span>
          \n reward_inputs =<\/span> super<\/span>(<\/span>)<\/span>.<\/span>_prepare_inputs(<\/span>reward_inputs)<\/span>
          \n with<\/span> torch.<\/span>inference_mode(<\/span>)<\/span>:<\/span>
          \n rewards_per_func[<\/span>:<\/span>,<\/span> i]<\/span> =<\/span> reward_func(<\/span>**<\/span>reward_inputs)<\/span>.<\/span>logits[<\/span>:<\/span>,<\/span> 0<\/span>]<\/span> # Shape (B*G,)<\/span>
          \n else<\/span>:<\/span>
          \n output_reward_func =<\/span> reward_func(<\/span>
          \n prompts=<\/span>prompts,<\/span> completions=<\/span>completions,<\/span> completion_ids=<\/span>completion_ids_list,<\/span> **<\/span>reward_kwargs
          \n )<\/span>
          \n # Convert None values to NaN<\/span>
          \n output_reward_func =<\/span> [<\/span>reward if<\/span> reward is<\/span> not<\/span> None<\/span> else<\/span> torch.<\/span>nan for<\/span> reward in<\/span> output_reward_func]<\/span><\/p>\n

          rewards_per_func[<\/span>:<\/span>,<\/span> i]<\/span> =<\/span> torch.<\/span>tensor(<\/span>output_reward_func,<\/span> dtype=<\/span>torch.<\/span>float32,<\/span> device=<\/span>device)<\/span><\/p>\n

          # If all reward functions return None for a given row, issue a detailed warning<\/span>
          \n if<\/span> torch.<\/span>isnan(<\/span>rewards_per_func)<\/span>.<\/span>all<\/span>(<\/span>dim=<\/span>1<\/span>)<\/span>.<\/span>any<\/span>(<\/span>)<\/span>:<\/span>
          \n nan_row_idx =<\/span> torch.<\/span>isnan(<\/span>rewards_per_func)<\/span>.<\/span>all<\/span>(<\/span>dim=<\/span>1<\/span>)<\/span>.<\/span>nonzero(<\/span>as_tuple=<\/span>True<\/span>)<\/span>[<\/span>0<\/span>]<\/span>[<\/span>0<\/span>]<\/span>
          \n row_reward_kwargs =<\/span> {<\/span>key:<\/span> value[<\/span>nan_row_idx]<\/span> for<\/span> key,<\/span> value in<\/span> reward_kwargs.<\/span>items(<\/span>)<\/span>}<\/span>
          \n row_reward_kwargs[<\/span>"prompt"<\/span>]<\/span> =<\/span> prompts[<\/span>nan_row_idx]<\/span>
          \n row_reward_kwargs[<\/span>"completion"<\/span>]<\/span> =<\/span> completions[<\/span>nan_row_idx]<\/span>
          \n warnings.<\/span>warn(<\/span>
          \n f"All reward functions returned None for the following kwargs: <\/span>{<\/span>row_reward_kwargs}<\/span><\/span>. "<\/span><\/span>
          \n "Please ensure that at least one reward function returns a valid reward."<\/span>
          \n )<\/span><\/p>\n

          # Gather the reward per function: this part is crucial, because the rewards are normalized per group and the<\/span>
          \n # completions may be distributed across processes<\/span>
          \n rewards_per_func =<\/span> gather(<\/span>rewards_per_func)<\/span>
          \n return<\/span> rewards_per_func<\/p>\n

          _calculate_rewards \u628a\u201c\u591a\u6761 completion\u201d\u5582\u7ed9\u201c\u591a\u4e2a\u5956\u52b1\u51fd\u6570\u201d，\u8fd4\u56de\u4e00\u5f20 (B\u00d7G, F) \u7684\u5956\u52b1\u77e9\u9635，\u5e76\u8de8\u8fdb\u7a0b\u540c\u6b65，\u4e3a\u540e\u7eed GRPO \u7ec4\u5185\u5f52\u4e00\u5316\u505a\u51c6\u5907\u3002<\/p>\n

          \n

          \u2705 \u8f93\u5165\u8f93\u51fa<\/h4>\n\n\u53c2\u6570\u5f62\u72b6 \/ \u542b\u4e49<\/tr>\n\n\n\n\n\n
          prompts<\/td>\nList[str] \u6216 List[Messages]，\u957f\u5ea6 = B\u00d7G<\/td>\n<\/tr>\n
          completions<\/td>\nList[str] \u6216 List[Messages]，\u957f\u5ea6 = B\u00d7G<\/td>\n<\/tr>\n
          completion_ids_list<\/td>\nList[List[int]]，token id，\u957f\u5ea6 = B\u00d7G<\/td>\n<\/tr>\n
          \u8fd4\u56de\u503c<\/td>\nTensor \u5f62\u72b6 (B\u00d7G, F)，F = \u5956\u52b1\u51fd\u6570\u4e2a\u6570<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
          \n

          \u2705 \u6838\u5fc3\u6b65\u9aa4<\/h4>\n
        • \n

          \u521d\u59cb\u5316\u5bb9\u5668
          \nrewards_per_func = zeros(B\u00d7G, F) \u5148\u5360\u597d\u4f4d\u7f6e\u3002<\/p>\n<\/li>\n

        • \n

          \u628a\u989d\u5916\u5217\u6253\u5305\u6210 kwargs
          \n\u4efb\u4f55 inputs[0] \u91cc\u9664 "prompt"\/"completion"\/"completion_ids" \u4ee5\u5916\u7684\u5b57\u6bb5\u5168\u90e8\u6309\u884c\u91cd\u590d，\u4f9b\u81ea\u5b9a\u4e49\u5956\u52b1\u51fd\u6570\u4f7f\u7528\u3002<\/p>\n<\/li>\n

        • \n

          \u904d\u5386 F \u4e2a\u5956\u52b1\u51fd\u6570<\/p>\n

            \n
          • \u5982\u679c\u662f\u6a21\u578b（nn.Module）：\n
              \n
            • \u6784\u9020 prompt+completion \u7684\u6587\u672c \u2192 \u8d70 tokenizer \u2192 \u524d\u5411 \u2192 \u53d6 logits[:, 0] \u4f5c\u4e3a\u6807\u91cf\u5956\u52b1\u3002<\/li>\n<\/ul>\n<\/li>\n
            • \u5982\u679c\u662f\u51fd\u6570（Callable）：\n
                \n
              • \u76f4\u63a5\u8c03\u7528，\u5141\u8bb8\u8fd4\u56de None \u2192 \u8f6c\u6210 NaN \u5360\u4f4d\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n
              • \n

                \u8de8\u8fdb\u7a0b\u540c\u6b65
                \ngather(rewards_per_func) \u8ba9 \u6240\u6709 GPU \u62ff\u5230 \u5168\u5c40 (N\u00d7G, F) \u5956\u52b1\u77e9\u9635，\u4fdd\u8bc1\u540e\u7eed\u7ec4\u5185\u5f52\u4e00\u5316\u4e00\u81f4\u3002<\/p>\n<\/li>\n

              • \n

                \u5f02\u5e38\u68c0\u6d4b
                \n\u5982\u679c\u67d0\u4e00\u884c\u5168\u662f NaN，\u6253\u5370\u8be6\u7ec6 warning，\u65b9\u4fbf\u6392\u67e5\u5956\u52b1\u51fd\u6570\u6f0f\u8fd4\u56de\u503c\u3002<\/p>\n<\/li>\n

                \n

                \u2705 \u603b\u7ed3<\/h4>\n

                \u201c\u628a B\u00d7G \u6761 completion \u5582\u7ed9 F \u4e2a\u5956\u52b1\u51fd\u6570，\u8de8\u8fdb\u7a0b\u6536\u96c6\u7ed3\u679c，\u751f\u6210 (B\u00d7G, F) \u7684\u5956\u52b1\u5f20\u91cf，\u4f9b GRPO \u505a\u7ec4\u5185\u5f52\u4e00\u5316\u3002\u201d<\/p>\n

                _generate_and_score_completions<\/p>\n

                def<\/span> _generate_and_score_completions<\/span>(<\/span>
                \n self,<\/span> inputs:<\/span> list<\/span>[<\/span>dict<\/span>[<\/span>str<\/span>,<\/span> Union[<\/span>torch.<\/span>Tensor,<\/span> Any]<\/span>]<\/span>]<\/span>
                \n )<\/span> –<\/span>><\/span> dict<\/span>[<\/span>str<\/span>,<\/span> Union[<\/span>torch.<\/span>Tensor,<\/span> Any]<\/span>]<\/span>:<\/span>
                \n device =<\/span> self.<\/span>accelerator.<\/span>device
                \n mode =<\/span> "train"<\/span> if<\/span> self.<\/span>model.<\/span>training else<\/span> "eval"<\/span><\/p>\n

                prompts =<\/span> [<\/span>x[<\/span>"prompt"<\/span>]<\/span> for<\/span> x in<\/span> inputs]<\/span><\/p>\n

                # We don't yet support visual reward models\/function, so we keep a copy of the original text-only prompts for<\/span>
                \n # later use in the reward computation. If images are present, we insert {"type": "image"} as required by the<\/span>
                \n # VLM chat template.<\/span>
                \n original_prompts =<\/span> copy.<\/span>deepcopy(<\/span>prompts)<\/span><\/p>\n

                # If the prompts are conversational and the inputs contain images, we need to convert the prompts from<\/span>
                \n # [{"role": "user", "content": "What color is the sky?"}] to<\/span>
                \n # [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What color is the sky?"}]}]<\/span>
                \n kwargs =<\/span> {<\/span>}<\/span>
                \n has_images =<\/span> "image"<\/span> in<\/span> inputs[<\/span>0<\/span>]<\/span>
                \n if<\/span> has_images:<\/span>
                \n images =<\/span> [<\/span>example.<\/span>get(<\/span>"image"<\/span>)<\/span> for<\/span> example in<\/span> inputs]<\/span>
                \n kwargs =<\/span> {<\/span>"images"<\/span>:<\/span> [<\/span>[<\/span>img]<\/span> for<\/span> img in<\/span> images]<\/span>}<\/span>
                \n for<\/span> prompt in<\/span> prompts:<\/span>
                \n if<\/span> isinstance<\/span>(<\/span>prompt,<\/span> list<\/span>)<\/span>:<\/span>
                \n for<\/span> message in<\/span> prompt:<\/span>
                \n if<\/span> not<\/span> isinstance<\/span>(<\/span>message,<\/span> dict<\/span>)<\/span>:<\/span>
                \n continue<\/span>
                \n content =<\/span> message.<\/span>get(<\/span>"content"<\/span>)<\/span>
                \n role =<\/span> message.<\/span>get(<\/span>"role"<\/span>)<\/span>
                \n if<\/span> isinstance<\/span>(<\/span>content,<\/span> str<\/span>)<\/span>:<\/span>
                \n if<\/span> role ==<\/span> "user"<\/span>:<\/span>
                \n message[<\/span>"content"<\/span>]<\/span> =<\/span> [<\/span>{<\/span>"type"<\/span>:<\/span> "image"<\/span>}<\/span>,<\/span> {<\/span>"type"<\/span>:<\/span> "text"<\/span>,<\/span> "text"<\/span>:<\/span> content}<\/span>]<\/span>
                \n elif<\/span> role ==<\/span> "system"<\/span>:<\/span>
                \n message[<\/span>"content"<\/span>]<\/span> =<\/span> [<\/span>{<\/span>"type"<\/span>:<\/span> "text"<\/span>,<\/span> "text"<\/span>:<\/span> content}<\/span>]<\/span><\/p>\n

                prompts_text =<\/span> [<\/span>maybe_apply_chat_template(<\/span>example,<\/span> self.<\/span>processing_class)<\/span>[<\/span>"prompt"<\/span>]<\/span> for<\/span> example in<\/span> inputs]<\/span><\/p>\n

                prompt_inputs =<\/span> self.<\/span>processing_class(<\/span>
                \n text=<\/span>prompts_text,<\/span>
                \n return_tensors=<\/span>"pt"<\/span>,<\/span>
                \n padding=<\/span>True<\/span>,<\/span>
                \n padding_side=<\/span>"left"<\/span>,<\/span>
                \n add_special_tokens=<\/span>False<\/span>,<\/span>
                \n **<\/span>kwargs,<\/span>
                \n )<\/span>
                \n prompt_inputs =<\/span> super<\/span>(<\/span>)<\/span>.<\/span>_prepare_inputs(<\/span>prompt_inputs)<\/span>
                \n prompt_ids,<\/span> prompt_mask =<\/span> prompt_inputs[<\/span>"input_ids"<\/span>]<\/span>,<\/span> prompt_inputs[<\/span>"attention_mask"<\/span>]<\/span><\/p>\n

                if<\/span> self.<\/span>max_prompt_length is<\/span> not<\/span> None<\/span>:<\/span>
                \n # If max_prompt_length is set, we trim the prompt to keep only the last `max_prompt_length` tokens.<\/span>
                \n # Then we decode those tokens back into text. We manually remove leading pad tokens from the decoded text,<\/span>
                \n # because we can't use `skip_special_tokens=True` (some special tokens are still needed for generation).<\/span>
                \n protected =<\/span> [<\/span>self.<\/span>image_token_id,<\/span> self.<\/span>vision_start_token_id,<\/span> self.<\/span>vision_end_token_id]<\/span>
                \n protected =<\/span> [<\/span>token for<\/span> token in<\/span> protected if<\/span> token is<\/span> not<\/span> None<\/span>]<\/span>
                \n prompt_ids,<\/span> prompt_mask =<\/span> truncate_with_protected_tokens(<\/span>
                \n prompt_ids,<\/span> prompt_mask,<\/span> self.<\/span>max_prompt_length,<\/span> protected
                \n )<\/span><\/p>\n

                prompts_text =<\/span> self.<\/span>processing_class.<\/span>batch_decode(<\/span>
                \n prompt_ids,<\/span> skip_special_tokens=<\/span>False<\/span>,<\/span> clean_up_tokenization_spaces=<\/span>False<\/span>
                \n )<\/span>
                \n prompts_text =<\/span> [<\/span>re.<\/span>sub(<\/span>rf"^(<\/span>{<\/span>re.<\/span>escape(<\/span>self.<\/span>pad_token)<\/span>}<\/span><\/span>)+"<\/span><\/span>,<\/span> ""<\/span>,<\/span> text)<\/span> for<\/span> text in<\/span> prompts_text]<\/span><\/p>\n

                # The chat template sometimes inserts a single image token into the prompt text. However, when this text is<\/span>
                \n # later tokenized, the single image token string is expanded into multiple image token IDs, depending on the<\/span>
                \n # image size. Since we're detokenizing here, we may see repeated image tokens in the decoded text. We<\/span>
                \n # collapse them back into a single token string to match the original chat template in case it originally<\/span>
                \n # applies it. Otherwise, it assumes that the chat template uses only vision_start_token_id to indicate images<\/span>
                \n # (e.g. Gemma 3) and removes all image_token instances and vision_end_token_id as well, leaving only<\/span>
                \n # the vision_start_token_id (e.g. <start_of_image>).<\/span>
                \n if<\/span> self.<\/span>image_token is<\/span> not<\/span> None<\/span>:<\/span>
                \n escaped_img_token =<\/span> re.<\/span>escape(<\/span>self.<\/span>image_token)<\/span>
                \n # Search for the image token in the chat template<\/span>
                \n if<\/span> re.<\/span>search(<\/span>escaped_img_token,<\/span> self.<\/span>processing_class.<\/span>chat_template)<\/span>:<\/span>
                \n prompts_text =<\/span> [<\/span>
                \n re.<\/span>sub(<\/span>rf"(<\/span>{<\/span>escaped_img_token}<\/span><\/span>)+"<\/span><\/span>,<\/span> self.<\/span>image_token,<\/span> text)<\/span> for<\/span> text in<\/span> prompts_text
                \n ]<\/span>
                \n else<\/span>:<\/span>
                \n # If the chat template doesn't use the image token, we remove all instances of it + vision_end_token_id<\/span>
                \n if<\/span> self.<\/span>vision_end_token_id is<\/span> not<\/span> None<\/span>:<\/span>
                \n escaped_eoi_token =<\/span> re.<\/span>escape(<\/span>
                \n self.<\/span>processing_class.<\/span>tokenizer.<\/span>decode(<\/span>[<\/span>self.<\/span>vision_end_token_id]<\/span>)<\/span>
                \n )<\/span>
                \n prompts_text =<\/span> [<\/span>
                \n re.<\/span>sub(<\/span>rf"(<\/span>{<\/span>escaped_img_token}<\/span><\/span>)+<\/span>{<\/span>escaped_eoi_token}<\/span><\/span>"<\/span><\/span>,<\/span> ""<\/span>,<\/span> text)<\/span> for<\/span> text in<\/span> prompts_text
                \n ]<\/span>
                \n else<\/span>:<\/span>
                \n # If vision_end_token_id is None, just remove the image tokens<\/span>
                \n prompts_text =<\/span> [<\/span>re.<\/span>sub(<\/span>rf"(<\/span>{<\/span>escaped_img_token}<\/span><\/span>)+"<\/span><\/span>,<\/span> ""<\/span>,<\/span> text)<\/span> for<\/span> text in<\/span> prompts_text]<\/span><\/p>\n

                # Generate completions using either vLLM or regular generation<\/span>
                \n if<\/span> self.<\/span>use_vllm:<\/span>
                \n # First, update the vLLM weights if needed<\/span>
                \n if<\/span> self.<\/span>state.<\/span>global_step !=<\/span> self.<\/span>_last_loaded_step:<\/span>
                \n self.<\/span>_move_model_to_vllm(<\/span>)<\/span>
                \n self.<\/span>_last_loaded_step =<\/span> self.<\/span>state.<\/span>global_step<\/p>\n

                # Generate completions using vLLM: gather all prompts and use them in a single call in the main process<\/span>
                \n if<\/span> self.<\/span>vllm_mode ==<\/span> "server"<\/span>:<\/span>
                \n all_prompts_text =<\/span> gather_object(<\/span>prompts_text)<\/span>
                \n if<\/span> has_images:<\/span>
                \n all_images =<\/span> gather_object(<\/span>images)<\/span><\/p>\n

                if<\/span> self.<\/span>accelerator.<\/span>is_main_process:<\/span>
                \n # Since 'prompts' contains 'num_generations' duplicates, we first take unique prompts, and generate<\/span>
                \n # num_generations outputs for each one. This is faster than generating outputs for each duplicate<\/span>
                \n # prompt individually.<\/span>
                \n ordered_set_of_prompts =<\/span> all_prompts_text[<\/span>:<\/span>:<\/span> self.<\/span>num_generations]<\/span><\/p>\n

                if<\/span> has_images:<\/span>
                \n ordered_set_of_images =<\/span> all_images[<\/span>:<\/span>:<\/span> self.<\/span>num_generations]<\/span>
                \n else<\/span>:<\/span>
                \n ordered_set_of_images =<\/span> None<\/span><\/p>\n

                with<\/span> profiling_context(<\/span>self,<\/span> "vLLM.generate"<\/span>)<\/span>:<\/span>
                \n completion_ids =<\/span> self.<\/span>vllm_client.<\/span>generate(<\/span>
                \n prompts=<\/span>ordered_set_of_prompts,<\/span>
                \n images=<\/span>ordered_set_of_images,<\/span>
                \n n=<\/span>self.<\/span>num_generations,<\/span>
                \n repetition_penalty=<\/span>self.<\/span>repetition_penalty,<\/span>
                \n temperature=<\/span>self.<\/span>temperature,<\/span>
                \n top_p=<\/span>self.<\/span>top_p,<\/span>
                \n top_k=<\/span>–<\/span>1<\/span> if<\/span> self.<\/span>top_k is<\/span> None<\/span> else<\/span> self.<\/span>top_k,<\/span>
                \n min_p=<\/span>0.0<\/span> if<\/span> self.<\/span>min_p is<\/span> None<\/span> else<\/span> self.<\/span>min_p,<\/span>
                \n max_tokens=<\/span>self.<\/span>max_completion_length,<\/span>
                \n guided_decoding_regex=<\/span>self.<\/span>guided_decoding_regex,<\/span>
                \n generation_kwargs=<\/span>self.<\/span>args.<\/span>generation_kwargs,<\/span>
                \n )<\/span>
                \n else<\/span>:<\/span>
                \n completion_ids =<\/span> [<\/span>None<\/span>]<\/span> *<\/span> len<\/span>(<\/span>all_prompts_text)<\/span>
                \n # Broadcast the completions from the main process to all processes, ensuring each process receives its<\/span>
                \n # corresponding slice.<\/span>
                \n completion_ids =<\/span> broadcast_object_list(<\/span>completion_ids,<\/span> from_process=<\/span>0<\/span>)<\/span>
                \n process_slice =<\/span> slice<\/span>(<\/span>
                \n self.<\/span>accelerator.<\/span>process_index *<\/span> len<\/span>(<\/span>prompts)<\/span>,<\/span>
                \n (<\/span>self.<\/span>accelerator.<\/span>process_index +<\/span> 1<\/span>)<\/span> *<\/span> len<\/span>(<\/span>prompts)<\/span>,<\/span>
                \n )<\/span>
                \n completion_ids =<\/span> completion_ids[<\/span>process_slice]<\/span><\/p>\n

                # Generate completions using colocated vLLM instances: each device holds vLLM copy and work on their own batch of prompts<\/span>
                \n elif<\/span> self.<\/span>vllm_mode ==<\/span> "colocate"<\/span>:<\/span>
                \n if<\/span> self.<\/span>guided_decoding_regex:<\/span>
                \n guided_decoding =<\/span> GuidedDecodingParams(<\/span>regex=<\/span>self.<\/span>guided_decoding_regex)<\/span>
                \n else<\/span>:<\/span>
                \n guided_decoding =<\/span> None<\/span><\/p>\n

                generation_kwargs =<\/span> {<\/span>
                \n "n"<\/span>:<\/span> 1<\/span>,<\/span> # vLLM on each GPU generates only 1 in colocate mode<\/span>
                \n "repetition_penalty"<\/span>:<\/span> self.<\/span>repetition_penalty,<\/span>
                \n "temperature"<\/span>:<\/span> self.<\/span>temperature,<\/span>
                \n "top_p"<\/span>:<\/span> self.<\/span>top_p,<\/span>
                \n "top_k"<\/span>:<\/span> –<\/span>1<\/span> if<\/span> self.<\/span>top_k is<\/span> None<\/span> else<\/span> self.<\/span>top_k,<\/span>
                \n "min_p"<\/span>:<\/span> 0.0<\/span> if<\/span> self.<\/span>min_p is<\/span> None<\/span> else<\/span> self.<\/span>min_p,<\/span>
                \n "max_tokens"<\/span>:<\/span> self.<\/span>max_completion_length,<\/span>
                \n "guided_decoding"<\/span>:<\/span> guided_decoding,<\/span>
                \n }<\/span>
                \n if<\/span> self.<\/span>args.<\/span>generation_kwargs is<\/span> not<\/span> None<\/span>:<\/span>
                \n generation_kwargs.<\/span>update(<\/span>self.<\/span>args.<\/span>generation_kwargs)<\/span>
                \n sampling_params =<\/span> SamplingParams(<\/span>**<\/span>generation_kwargs)<\/span><\/p>\n

                if<\/span> self.<\/span>vllm_tensor_parallel_size ><\/span> 1<\/span>:<\/span>
                \n # Gather prompts from all ranks in the TP group and flatten.<\/span>
                \n # Each rank starts with its own prompts; after gathering, all ranks see the full group set.<\/span>
                \n orig_size =<\/span> len<\/span>(<\/span>prompts_text)<\/span>
                \n gathered_prompts =<\/span> [<\/span>None<\/span> for<\/span> _ in<\/span> range<\/span>(<\/span>self.<\/span>vllm_tensor_parallel_size)<\/span>]<\/span>
                \n torch.<\/span>distributed.<\/span>all_gather_object(<\/span>gathered_prompts,<\/span> prompts_text,<\/span> group=<\/span>self.<\/span>tp_group)<\/span>
                \n all_prompts_text =<\/span> [<\/span>p for<\/span> sublist in<\/span> gathered_prompts for<\/span> p in<\/span> sublist]<\/span><\/p>\n

                if<\/span> has_images:<\/span>
                \n gathered_images =<\/span> [<\/span>None<\/span> for<\/span> _ in<\/span> range<\/span>(<\/span>self.<\/span>vllm_tensor_parallel_size)<\/span>]<\/span>
                \n torch.<\/span>distributed.<\/span>all_gather_object(<\/span>gathered_images,<\/span> images,<\/span> group=<\/span>self.<\/span>tp_group)<\/span>
                \n all_images =<\/span> [<\/span>img for<\/span> sublist in<\/span> gathered_images for<\/span> img in<\/span> sublist]<\/span>
                \n else<\/span>:<\/span>
                \n all_images =<\/span> None<\/span>
                \n else<\/span>:<\/span>
                \n all_prompts_text =<\/span> prompts_text
                \n all_images =<\/span> images if<\/span> has_images else<\/span> None<\/span><\/p>\n

                if<\/span> has_images and<\/span> all_images:<\/span>
                \n vllm_inputs =<\/span> [<\/span>]<\/span>
                \n for<\/span> prompt,<\/span> image in<\/span> zip<\/span>(<\/span>all_prompts_text,<\/span> all_images)<\/span>:<\/span>
                \n if<\/span> image is<\/span> not<\/span> None<\/span>:<\/span>
                \n vllm_inputs.<\/span>append(<\/span>{<\/span>"prompt"<\/span>:<\/span> prompt,<\/span> "multi_modal_data"<\/span>:<\/span> {<\/span>"image"<\/span>:<\/span> image}<\/span>}<\/span>)<\/span>
                \n else<\/span>:<\/span>
                \n vllm_inputs.<\/span>append(<\/span>prompt)<\/span>
                \n else<\/span>:<\/span>
                \n vllm_inputs =<\/span> all_prompts_text<\/p>\n

                with<\/span> profiling_context(<\/span>self,<\/span> "vLLM.generate"<\/span>)<\/span>:<\/span>
                \n all_outputs =<\/span> self.<\/span>llm.<\/span>generate(<\/span>vllm_inputs,<\/span> sampling_params=<\/span>sampling_params,<\/span> use_tqdm=<\/span>False<\/span>)<\/span><\/p>\n

                completion_ids =<\/span> [<\/span>output.<\/span>token_ids for<\/span> outputs in<\/span> all_outputs for<\/span> output in<\/span> outputs.<\/span>outputs]<\/span><\/p>\n

                if<\/span> self.<\/span>vllm_tensor_parallel_size ><\/span> 1<\/span>:<\/span>
                \n # Slice completions for this rank within its TP group.<\/span>
                \n # Each rank generates all outputs \u2014 we keep only our share.<\/span>
                \n local_rank_in_group =<\/span> torch.<\/span>distributed.<\/span>get_rank(<\/span>group=<\/span>self.<\/span>tp_group)<\/span>
                \n tp_slice =<\/span> slice<\/span>(<\/span>local_rank_in_group *<\/span> orig_size,<\/span> (<\/span>local_rank_in_group +<\/span> 1<\/span>)<\/span> *<\/span> orig_size)<\/span>
                \n completion_ids =<\/span> completion_ids[<\/span>tp_slice]<\/span><\/p>\n

                # Pad the completions, and concatenate them with the prompts<\/span>
                \n completion_ids =<\/span> [<\/span>torch.<\/span>tensor(<\/span>ids,<\/span> device=<\/span>device)<\/span> for<\/span> ids in<\/span> completion_ids]<\/span>
                \n completion_ids =<\/span> pad(<\/span>completion_ids,<\/span> padding_value=<\/span>self.<\/span>pad_token_id)<\/span>
                \n prompt_completion_ids =<\/span> torch.<\/span>cat(<\/span>[<\/span>prompt_ids,<\/span> completion_ids]<\/span>,<\/span> dim=<\/span>1<\/span>)<\/span><\/p>\n

                elif<\/span> self.<\/span>use_transformers_paged:<\/span>
                \n # Re-process inputs for paged generation if needed<\/span>
                \n # Note: images are already validated and preprocessed above<\/span>
                \n paged_prompt_inputs =<\/span> self.<\/span>processing_class(<\/span>text=<\/span>prompts_text,<\/span> **<\/span>kwargs)<\/span>
                \n previous_attn =<\/span> self.<\/span>model_wrapped.<\/span>config.<\/span>_attn_implementation<\/p>\n

                if<\/span> is_flash_attn_2_available(<\/span>)<\/span>:<\/span>
                \n self.<\/span>model_wrapped.<\/span>config.<\/span>_attn_implementation =<\/span> "paged_attention"<\/span>
                \n else<\/span>:<\/span>
                \n self.<\/span>model_wrapped.<\/span>config.<\/span>_attn_implementation =<\/span> "sdpa_paged"<\/span>
                \n with<\/span> (<\/span>
                \n profiling_context(<\/span>self,<\/span> "transformers.generate_batch"<\/span>)<\/span>,<\/span>
                \n unwrap_model_for_generation(<\/span>
                \n self.<\/span>model_wrapped,<\/span> self.<\/span>accelerator,<\/span> gather_deepspeed3_params=<\/span>self.<\/span>args.<\/span>ds3_gather_for_generation
                \n )<\/span> as<\/span> unwrapped_model,<\/span>
                \n torch.<\/span>no_grad(<\/span>)<\/span>,<\/span>
                \n FSDP.<\/span>summon_full_params(<\/span>self.<\/span>model_wrapped,<\/span> recurse=<\/span>False<\/span>)<\/span> if<\/span> self.<\/span>is_fsdp_enabled else<\/span> nullcontext(<\/span>)<\/span>,<\/span>
                \n )<\/span>:<\/span>
                \n # Cast to the appropriate dtype based on training configuration<\/span>
                \n if<\/span> self.<\/span>args.<\/span>bf16:<\/span>
                \n unwrapped_model.<\/span>to(<\/span>torch.<\/span>bfloat16)<\/span>
                \n elif<\/span> self.<\/span>args.<\/span>fp16:<\/span>
                \n unwrapped_model.<\/span>to(<\/span>torch.<\/span>float16)<\/span>
                \n with<\/span> torch.<\/span>inference_mode(<\/span>)<\/span>:<\/span>
                \n all_outputs =<\/span> unwrapped_model.<\/span>generate_batch(<\/span>
                \n paged_prompt_inputs.<\/span>input_ids,<\/span> generation_config=<\/span>self.<\/span>generation_config,<\/span> progress_bar=<\/span>False<\/span>
                \n )<\/span>
                \n completion_ids =<\/span> [<\/span>output.<\/span>generated_tokens for<\/span> output in<\/span> all_outputs.<\/span>values(<\/span>)<\/span>]<\/span>
                \n completion_ids =<\/span> [<\/span>torch.<\/span>tensor(<\/span>ids,<\/span> device=<\/span>device)<\/span> for<\/span> ids in<\/span> completion_ids]<\/span>
                \n completion_ids =<\/span> pad(<\/span>completion_ids,<\/span> padding_value=<\/span>self.<\/span>pad_token_id,<\/span> padding_side=<\/span>"right"<\/span>)<\/span>
                \n prompt_ids =<\/span> [<\/span>torch.<\/span>tensor(<\/span>ids,<\/span> device=<\/span>device)<\/span> for<\/span> ids in<\/span> paged_prompt_inputs.<\/span>input_ids]<\/span>
                \n prompt_ids =<\/span> pad(<\/span>prompt_ids,<\/span> padding_value=<\/span>self.<\/span>pad_token_id,<\/span> padding_side=<\/span>"left"<\/span>)<\/span>
                \n prompt_completion_ids =<\/span> torch.<\/span>cat(<\/span>[<\/span>prompt_ids,<\/span> completion_ids]<\/span>,<\/span> dim=<\/span>1<\/span>)<\/span>
                \n # Restore the original attention implementation, training mode<\/span>
                \n self.<\/span>model_wrapped.<\/span>config.<\/span>_attn_implementation =<\/span> previous_attn
                \n else<\/span>:<\/span>
                \n # Regular generation path<\/span>
                \n with<\/span> (<\/span>
                \n profiling_context(<\/span>self,<\/span> "transformers.generate"<\/span>)<\/span>,<\/span>
                \n unwrap_model_for_generation(<\/span>
                \n self.<\/span>model_wrapped,<\/span> self.<\/span>accelerator,<\/span> gather_deepspeed3_params=<\/span>self.<\/span>args.<\/span>ds3_gather_for_generation
                \n )<\/span> as<\/span> unwrapped_model,<\/span>
                \n torch.<\/span>no_grad(<\/span>)<\/span>,<\/span>
                \n FSDP.<\/span>summon_full_params(<\/span>self.<\/span>model_wrapped,<\/span> recurse=<\/span>False<\/span>)<\/span> if<\/span> self.<\/span>is_fsdp_enabled else<\/span> nullcontext(<\/span>)<\/span>,<\/span>
                \n )<\/span>:<\/span>
                \n prompt_inputs[<\/span>"input_ids"<\/span>]<\/span>,<\/span> prompt_inputs[<\/span>"attention_mask"<\/span>]<\/span> =<\/span> prompt_ids,<\/span> prompt_mask
                \n prompt_completion_ids =<\/span> unwrapped_model.<\/span>generate(<\/span>
                \n **<\/span>prompt_inputs,<\/span> generation_config=<\/span>self.<\/span>generation_config,<\/span> disable_compile=<\/span>True<\/span>
                \n )<\/span>
                \n # Compute prompt length and extract completion ids<\/span>
                \n prompt_length =<\/span> prompt_ids.<\/span>size(<\/span>1<\/span>)<\/span>
                \n prompt_ids =<\/span> prompt_completion_ids[<\/span>:<\/span>,<\/span> :<\/span>prompt_length]<\/span>
                \n completion_ids =<\/span> prompt_completion_ids[<\/span>:<\/span>,<\/span> prompt_length:<\/span>]<\/span><\/p>\n

                # Mask everything after the first EOS token<\/span>
                \n is_eos =<\/span> completion_ids ==<\/span> self.<\/span>eos_token_id
                \n eos_idx =<\/span> torch.<\/span>full(<\/span>(<\/span>is_eos.<\/span>size(<\/span>0<\/span>)<\/span>,<\/span>)<\/span>,<\/span> is_eos.<\/span>size(<\/span>1<\/span>)<\/span>,<\/span> dtype=<\/span>torch.<\/span>long<\/span>,<\/span> device=<\/span>device)<\/span>
                \n eos_idx[<\/span>is_eos.<\/span>any<\/span>(<\/span>dim=<\/span>1<\/span>)<\/span>]<\/span> =<\/span> is_eos.<\/span>int<\/span>(<\/span>)<\/span>.<\/span>argmax(<\/span>dim=<\/span>1<\/span>)<\/span>[<\/span>is_eos.<\/span>any<\/span>(<\/span>dim=<\/span>1<\/span>)<\/span>]<\/span>
                \n sequence_indices =<\/span> torch.<\/span>arange(<\/span>is_eos.<\/span>size(<\/span>1<\/span>)<\/span>,<\/span> device=<\/span>device)<\/span>.<\/span>expand(<\/span>is_eos.<\/span>size(<\/span>0<\/span>)<\/span>,<\/span> –<\/span>1<\/span>)<\/span>
                \n completion_mask =<\/span> (<\/span>sequence_indices <=<\/span> eos_idx.<\/span>unsqueeze(<\/span>1<\/span>)<\/span>)<\/span>.<\/span>int<\/span>(<\/span>)<\/span><\/p>\n

                # Convert tensor to a list of lists of token IDs. This will be passed to the reward function, avoiding the need<\/span>
                \n # to re-tokenize completions if the reward is computed from tokens.<\/span>
                \n completion_ids_list =<\/span> [<\/span>
                \n [<\/span>id<\/span>.<\/span>item(<\/span>)<\/span> for<\/span> id<\/span>,<\/span> m in<\/span> zip<\/span>(<\/span>row,<\/span> mask_row)<\/span> if<\/span> m]<\/span> for<\/span> row,<\/span> mask_row in<\/span> zip<\/span>(<\/span>completion_ids,<\/span> completion_mask)<\/span>
                \n ]<\/span><\/p>\n

                # Sum along sequence dimension (dim=1) to get completion length per sequence, used for logging<\/span>
                \n completion_lengths =<\/span> completion_mask.<\/span>sum<\/span>(<\/span>1<\/span>)<\/span><\/p>\n

                # If mask_truncated_completions is enabled, zero out truncated completions in completion_mask<\/span>
                \n if<\/span> self.<\/span>mask_truncated_completions:<\/span>
                \n truncated_completions =<\/span> ~<\/span>is_eos.<\/span>any<\/span>(<\/span>dim=<\/span>1<\/span>)<\/span>
                \n completion_mask =<\/span> completion_mask *<\/span> (<\/span>~<\/span>truncated_completions)<\/span>.<\/span>unsqueeze(<\/span>1<\/span>)<\/span>.<\/span>int<\/span>(<\/span>)<\/span><\/p>\n

                # Concatenate prompt_mask with completion_mask for logit computation<\/span>
                \n attention_mask =<\/span> torch.<\/span>cat(<\/span>[<\/span>prompt_mask,<\/span> completion_mask]<\/span>,<\/span> dim=<\/span>1<\/span>)<\/span> # (B, P+C)<\/span><\/p>\n

                logits_to_keep =<\/span> completion_ids.<\/span>size(<\/span>1<\/span>)<\/span> # we only need to compute the logits for the completion tokens<\/span>
                \n batch_size =<\/span> self.<\/span>args.<\/span>per_device_train_batch_size if<\/span> mode ==<\/span> "train"<\/span> else<\/span> self.<\/span>args.<\/span>per_device_eval_batch_size<\/p>\n

                with<\/span> torch.<\/span>no_grad(<\/span>)<\/span>:<\/span>
                \n # If the generation and optimization steps are misaligned\u2014i.e., if generation does not occur at the end of<\/span>
                \n # a full optimizer step (when gradient_accumulation_steps is not a multiple of generate_every)\u2014then the<\/span>
                \n # samples may come from an earlier version of the model. In that case, we need to track old_per_token_logps<\/span>
                \n # for importance sampling. If the steps are aligned, importance sampling isn't necessary and we set<\/span>
                \n # old_per_token_logps to None.<\/span>
                \n generate_every =<\/span> self.<\/span>args.<\/span>steps_per_generation *<\/span> self.<\/span>num_iterations # generation frequency<\/span>
                \n if<\/span> self.<\/span>args.<\/span>gradient_accumulation_steps %<\/span> generate_every !=<\/span> 0<\/span>:<\/span>
                \n old_per_token_logps,<\/span> _ =<\/span> self.<\/span>_get_per_token_logps_and_entropies(<\/span>
                \n self.<\/span>model,<\/span>
                \n prompt_completion_ids,<\/span>
                \n attention_mask,<\/span>
                \n logits_to_keep,<\/span>
                \n batch_size,<\/span>
                \n pixel_values=<\/span>prompt_inputs.<\/span>get(<\/span>"pixel_values"<\/span>)<\/span>,<\/span>
                \n image_grid_thw=<\/span>prompt_inputs.<\/span>get(<\/span>"image_grid_thw"<\/span>)<\/span>,<\/span>
                \n pixel_attention_mask=<\/span>prompt_inputs.<\/span>get(<\/span>"pixel_attention_mask"<\/span>)<\/span>,<\/span>
                \n image_sizes=<\/span>prompt_inputs.<\/span>get(<\/span>"image_sizes"<\/span>)<\/span>,<\/span>
                \n )<\/span>
                \n else<\/span>:<\/span>
                \n old_per_token_logps =<\/span> None<\/span><\/p>\n

                # Compute the per-token log probabilities for the reference model<\/span>
                \n if<\/span> self.<\/span>beta !=<\/span> 0.0<\/span>:<\/span>
                \n if<\/span> self.<\/span>ref_model is<\/span> not<\/span> None<\/span>:<\/span>
                \n ref_per_token_logps,<\/span> _ =<\/span> self.<\/span>_get_per_token_logps_and_entropies(<\/span>
                \n self.<\/span>ref_model,<\/span>
                \n prompt_completion_ids,<\/span>
                \n attention_mask,<\/span>
                \n logits_to_keep,<\/span>
                \n batch_size=<\/span>batch_size,<\/span>
                \n pixel_values=<\/span>prompt_inputs.<\/span>get(<\/span>"pixel_values"<\/span>)<\/span>,<\/span>
                \n image_grid_thw=<\/span>prompt_inputs.<\/span>get(<\/span>"image_grid_thw"<\/span>)<\/span>,<\/span>
                \n pixel_attention_mask=<\/span>prompt_inputs.<\/span>get(<\/span>"pixel_attention_mask"<\/span>)<\/span>,<\/span>
                \n image_sizes=<\/span>prompt_inputs.<\/span>get(<\/span>"image_sizes"<\/span>)<\/span>,<\/span>
                \n )<\/span>
                \n else<\/span>:<\/span>
                \n with<\/span> self.<\/span>accelerator.<\/span>unwrap_model(<\/span>self.<\/span>model)<\/span>.<\/span>disable_adapter(<\/span>)<\/span>:<\/span>
                \n ref_per_token_logps,<\/span> _ =<\/span> self.<\/span>_get_per_token_logps_and_entropies(<\/span>
                \n self.<\/span>model,<\/span>
                \n prompt_completion_ids,<\/span>
                \n attention_mask,<\/span>
                \n logits_to_keep,<\/span>
                \n batch_size=<\/span>batch_size,<\/span>
                \n pixel_values=<\/span>prompt_inputs.<\/span>get(<\/span>"pixel_values"<\/span>)<\/span>,<\/span>
                \n image_grid_thw=<\/span>prompt_inputs.<\/span>get(<\/span>"image_grid_thw"<\/span>)<\/span>,<\/span>
                \n pixel_attention_mask=<\/span>prompt_inputs.<\/span>get(<\/span>"pixel_attention_mask"<\/span>)<\/span>,<\/span>
                \n image_sizes=<\/span>prompt_inputs.<\/span>get(<\/span>"image_sizes"<\/span>)<\/span>,<\/span>
                \n )<\/span>
                \n else<\/span>:<\/span>
                \n ref_per_token_logps =<\/span> None<\/span><\/p>\n

                # Decode the generated completions<\/span>
                \n completions_text =<\/span> self.<\/span>processing_class.<\/span>batch_decode(<\/span>completion_ids,<\/span> skip_special_tokens=<\/span>True<\/span>)<\/span>
                \n if<\/span> is_conversational(<\/span>inputs[<\/span>0<\/span>]<\/span>)<\/span>:<\/span>
                \n completions =<\/span> [<\/span>]<\/span>
                \n for<\/span> prompt,<\/span> completion in<\/span> zip<\/span>(<\/span>prompts,<\/span> completions_text)<\/span>:<\/span>
                \n bootstrap =<\/span> prompt.<\/span>pop(<\/span>)<\/span>[<\/span>"content"<\/span>]<\/span> if<\/span> prompt[<\/span>–<\/span>1<\/span>]<\/span>[<\/span>"role"<\/span>]<\/span> ==<\/span> "assistant"<\/span> else<\/span> ""<\/span>
                \n completions.<\/span>append(<\/span>[<\/span>{<\/span>"role"<\/span>:<\/span> "assistant"<\/span>,<\/span> "content"<\/span>:<\/span> bootstrap +<\/span> completion}<\/span>]<\/span>)<\/span>
                \n else<\/span>:<\/span>
                \n completions =<\/span> completions_text<\/p>\n

                # Calculate rewards for each reward function. rewards_per_func aggregates rewards across all processes. This is<\/span>
                \n # important because rewards will be normalized per group, and completions are distributed. We will later slice<\/span>
                \n # rewards_per_func to extract each process's subset.<\/span>
                \n rewards_per_func =<\/span> self.<\/span>_calculate_rewards(<\/span>inputs,<\/span> original_prompts,<\/span> completions,<\/span> completion_ids_list)<\/span><\/p>\n

                # Apply weights to each reward function's output and sum<\/span>
                \n rewards =<\/span> (<\/span>rewards_per_func *<\/span> self.<\/span>reward_weights.<\/span>to(<\/span>device)<\/span>.<\/span>unsqueeze(<\/span>0<\/span>)<\/span>)<\/span>.<\/span>nansum(<\/span>dim=<\/span>1<\/span>)<\/span><\/p>\n

                # Compute grouped-wise rewards<\/span>
                \n mean_grouped_rewards =<\/span> rewards.<\/span>view(<\/span>–<\/span>1<\/span>,<\/span> self.<\/span>num_generations)<\/span>.<\/span>mean(<\/span>dim=<\/span>1<\/span>)<\/span>
                \n std_grouped_rewards =<\/span> rewards.<\/span>view(<\/span>–<\/span>1<\/span>,<\/span> self.<\/span>num_generations)<\/span>.<\/span>std(<\/span>dim=<\/span>1<\/span>)<\/span>
                \n is_std_zero =<\/span> torch.<\/span>isclose(<\/span>std_grouped_rewards,<\/span> torch.<\/span>zeros_like(<\/span>std_grouped_rewards)<\/span>)<\/span><\/p>\n

                # Normalize the rewards to compute the advantages<\/span>
                \n mean_grouped_rewards =<\/span> mean_grouped_rewards.<\/span>repeat_interleave(<\/span>self.<\/span>num_generations,<\/span> dim=<\/span>0<\/span>)<\/span>
                \n std_grouped_rewards =<\/span> std_grouped_rewards.<\/span>repeat_interleave(<\/span>self.<\/span>num_generations,<\/span> dim=<\/span>0<\/span>)<\/span>
                \n advantages =<\/span> rewards –<\/span> mean_grouped_rewards
                \n if<\/span> self.<\/span>scale_rewards:<\/span>
                \n advantages =<\/span> advantages \/<\/span> (<\/span>std_grouped_rewards +<\/span> 1e-4<\/span>)<\/span><\/p>\n

                # Slice to keep only the local part of the data<\/span>
                \n process_slice =<\/span> slice<\/span>(<\/span>
                \n self.<\/span>accelerator.<\/span>process_index *<\/span> len<\/span>(<\/span>prompts)<\/span>,<\/span>
                \n (<\/span>self.<\/span>accelerator.<\/span>process_index +<\/span> 1<\/span>)<\/span> *<\/span> len<\/span>(<\/span>prompts)<\/span>,<\/span>
                \n )<\/span>
                \n all_process_advantages =<\/span> advantages.<\/span>clone(<\/span>)<\/span> # keep the aggregated advantages for logging<\/span>
                \n advantages =<\/span> advantages[<\/span>process_slice]<\/span><\/p>\n

                # Log the metrics<\/span>
                \n if<\/span> mode ==<\/span> "train"<\/span>:<\/span>
                \n self.<\/span>state.<\/span>num_input_tokens_seen +=<\/span> self.<\/span>accelerator.<\/span>gather(<\/span>attention_mask.<\/span>sum<\/span>(<\/span>)<\/span>)<\/span>.<\/span>sum<\/span>(<\/span>)<\/span>.<\/span>item(<\/span>)<\/span>
                \n self.<\/span>_metrics[<\/span>mode]<\/span>[<\/span>"num_tokens"<\/span>]<\/span> =<\/span> [<\/span>self.<\/span>state.<\/span>num_input_tokens_seen]<\/span><\/p>\n

                # Log completion lengths, mean, min, max<\/span>
                \n agg_completion_lengths =<\/span> self.<\/span>accelerator.<\/span>gather(<\/span>completion_lengths)<\/span>
                \n self.<\/span>_metrics[<\/span>mode]<\/span>[<\/span>"completions\/mean_length"<\/span>]<\/span>.<\/span>append(<\/span>agg_completion_lengths.<\/span>float<\/span>(<\/span>)<\/span>.<\/span>mean(<\/span>)<\/span>.<\/span>item(<\/span>)<\/span>)<\/span>
                \n self.<\/span>_metrics[<\/span>mode]<\/span>[<\/span>"completions\/min_length"<\/span>]<\/span>.<\/span>append(<\/span>agg_completion_lengths.<\/span>float<\/span>(<\/span>)<\/span>.<\/span>min<\/span>(<\/span>)<\/span>.<\/span>item(<\/span>)<\/span>)<\/span>
                \n self.<\/span>_metrics[<\/span>mode]<\/span>[<\/span>"completions\/max_length"<\/span>]<\/span>.<\/span>append(<\/span>agg_completion_lengths.<\/span>float<\/span>(<\/span>)<\/span>.<\/span>max<\/span>(<\/span>)<\/span>.<\/span>item(<\/span>)<\/span>)<\/span><\/p>\n

                # Identify sequences that terminated with EOS and log their lengths<\/span>
                \n agg_terminated_with_eos =<\/span> self.<\/span>accelerator.<\/span>gather(<\/span>is_eos.<\/span>any<\/span>(<\/span>dim=<\/span>1<\/span>)<\/span>)<\/span>
                \n term_completion_lengths =<\/span> agg_completion_lengths[<\/span>agg_terminated_with_eos]<\/span>
                \n clipped_completions_ratio =<\/span> 1<\/span> –<\/span> len<\/span>(<\/span>term_completion_lengths)<\/span> \/<\/span> len<\/span>(<\/span>agg_completion_lengths)<\/span>
                \n self.<\/span>_metrics[<\/span>mode]<\/span>[<\/span>"completions\/clipped_ratio"<\/span>]<\/span>.<\/span>append(<\/span>clipped_completions_ratio)<\/span>
                \n if<\/span> len<\/span>(<\/span>term_completion_lengths)<\/span> ==<\/span> 0<\/span>:<\/span> # edge case where no terminated sequences are found<\/span>
                \n term_completion_lengths =<\/span> torch.<\/span>zeros(<\/span>1<\/span>,<\/span> device=<\/span>device)<\/span>
                \n self.<\/span>_metrics[<\/span>mode]<\/span>[<\/span>"completions\/mean_terminated_length"<\/span>]<\/span>.<\/span>append(<\/span>term_completion_lengths.<\/span>float<\/span>(<\/span>)<\/span>.<\/span>mean(<\/span>)<\/span>.<\/span>item(<\/span>)<\/span>)<\/span>
                \n self.<\/span>_metrics[<\/span>mode]<\/span>[<\/span>"completions\/min_terminated_length"<\/span>]<\/span>.<\/span>append(<\/span>term_completion_lengths.<\/span>float<\/span>(<\/span>)<\/span>.<\/span>min<\/span>(<\/span>)<\/span>.<\/span>item(<\/span>)<\/span>)<\/span>
                \n self.<\/span>_metrics[<\/span>mode]<\/span>[<\/span>"completions\/max_terminated_length"<\/span>]<\/span>.<\/span>append(<\/span>term_completion_lengths.<\/span>float<\/span>(<\/span>)<\/span>.<\/span>max<\/span>(<\/span>)<\/span>.<\/span>item(<\/span>)<\/span>)<\/span><\/p>\n

                # Calculate mean reward per function, but only for samples where the function was applied (non-NaN values)<\/span>
                \n for<\/span> i,<\/span> reward_func_name in<\/span> enumerate<\/span>(<\/span>self.<\/span>reward_func_names)<\/span>:<\/span>
                \n mean_rewards =<\/span> torch.<\/span>nanmean(<\/span>rewards_per_func[<\/span>:<\/span>,<\/span> i]<\/span>)<\/span>.<\/span>item(<\/span>)<\/span>
                \n self.<\/span>_metrics[<\/span>mode]<\/span>[<\/span>f"rewards\/<\/span>{<\/span>reward_func_name}<\/span><\/span>\/mean"<\/span><\/span>]<\/span>.<\/span>append(<\/span>mean_rewards)<\/span>
                \n std_rewards =<\/span> nanstd(<\/span>rewards_per_func[<\/span>:<\/span>,<\/span> i]<\/span>)<\/span>.<\/span>item(<\/span>)<\/span>
                \n self.<\/span>_metrics[<\/span>mode]<\/span>[<\/span>f"rewards\/<\/span>{<\/span>reward_func_name}<\/span><\/span>\/std"<\/span><\/span>]<\/span>.<\/span>append(<\/span>std_rewards)<\/span>
                \n self.<\/span>_metrics[<\/span>mode]<\/span>[<\/span>"reward"<\/span>]<\/span>.<\/span>append(<\/span>mean_grouped_rewards.<\/span>mean(<\/span>)<\/span>.<\/span>item(<\/span>)<\/span>)<\/span>
                \n self.<\/span>_metrics[<\/span>mode]<\/span>[<\/span>"reward_std"<\/span>]<\/span>.<\/span>append(<\/span>std_grouped_rewards.<\/span>mean(<\/span>)<\/span>.<\/span>item(<\/span>)<\/span>)<\/span>
                \n self.<\/span>_metrics[<\/span>mode]<\/span>[<\/span>"frac_reward_zero_std"<\/span>]<\/span>.<\/span>append(<\/span>is_std_zero.<\/span>float<\/span>(<\/span>)<\/span>.<\/span>mean(<\/span>)<\/span>.<\/span>item(<\/span>)<\/span>)<\/span><\/p>\n

                # Log prompt and completion texts<\/span>
                \n self.<\/span>_logs[<\/span>"prompt"<\/span>]<\/span>.<\/span>extend(<\/span>gather_object(<\/span>prompts_text)<\/span>)<\/span>
                \n self.<\/span>_logs[<\/span>"completion"<\/span>]<\/span>.<\/span>extend(<\/span>gather_object(<\/span>completions_text)<\/span>)<\/span>
                \n for<\/span> i,<\/span> name in<\/span> enumerate<\/span>(<\/span>self.<\/span>reward_func_names)<\/span>:<\/span>
                \n self.<\/span>_logs[<\/span>"rewards"<\/span>]<\/span>[<\/span>name]<\/span>.<\/span>extend(<\/span>rewards_per_func[<\/span>:<\/span>,<\/span> i]<\/span>.<\/span>tolist(<\/span>)<\/span>)<\/span>
                \n self.<\/span>_logs[<\/span>"advantages"<\/span>]<\/span>.<\/span>extend(<\/span>all_process_advantages.<\/span>tolist(<\/span>)<\/span>)<\/span><\/p>\n

                if<\/span> has_images:<\/span>
                \n self.<\/span>_logs[<\/span>"image"<\/span>]<\/span>.<\/span>extend(<\/span>gather_object(<\/span>images)<\/span>)<\/span><\/p>\n

                output =<\/span> {<\/span>
                \n "prompt_ids"<\/span>:<\/span> prompt_ids,<\/span>
                \n "prompt_mask"<\/span>:<\/span> prompt_mask,<\/span>
                \n "completion_ids"<\/span>:<\/span> completion_ids,<\/span>
                \n "completion_mask"<\/span>:<\/span> completion_mask,<\/span>
                \n "advantages"<\/span>:<\/span> advantages,<\/span>
                \n }<\/span>
                \n if<\/span> old_per_token_logps is<\/span> not<\/span> None<\/span>:<\/span>
                \n output[<\/span>"old_per_token_logps"<\/span>]<\/span> =<\/span> old_per_token_logps
                \n if<\/span> ref_per_token_logps is<\/span> not<\/span> None<\/span>:<\/span>
                \n output[<\/span>"ref_per_token_logps"<\/span>]<\/span> =<\/span> ref_per_token_logps
                \n if<\/span> "pixel_values"<\/span> in<\/span> prompt_inputs:<\/span>
                \n output[<\/span>"pixel_values"<\/span>]<\/span> =<\/span> prompt_inputs[<\/span>"pixel_values"<\/span>]<\/span>
                \n if<\/span> "image_grid_thw"<\/span> in<\/span> prompt_inputs:<\/span>
                \n output[<\/span>"image_grid_thw"<\/span>]<\/span> =<\/span> prompt_inputs[<\/span>"image_grid_thw"<\/span>]<\/span>
                \n if<\/span> "pixel_attention_mask"<\/span> in<\/span> prompt_inputs:<\/span>
                \n output[<\/span>"pixel_attention_mask"<\/span>]<\/span> =<\/span> prompt_inputs[<\/span>"pixel_attention_mask"<\/span>]<\/span>
                \n if<\/span> "image_sizes"<\/span> in<\/span> prompt_inputs:<\/span>
                \n output[<\/span>"image_sizes"<\/span>]<\/span> =<\/span> prompt_inputs[<\/span>"image_sizes"<\/span>]<\/span>
                \n return<\/span> output<\/p>\n

                _generate_and_score_completions \u662f GRPOTrainer \u7684\u201c\u5fc3\u810f\u201d\u2014\u2014
                \n\u4e00\u6b21\u6027\u5b8c\u6210 prompt \u5904\u7406 \u2192 \u591a\u540e\u7aef\u751f\u6210 \u2192 \u5956\u52b1\u6253\u5206 \u2192 \u7ec4\u5185\u5f52\u4e00\u5316 \u2192 \u8f93\u51fa\u8bad\u7ec3\u6240\u9700\u5168\u90e8\u5f20\u91cf\u3002<\/p>\n

                \n

                \u2705 \u4e00\u53e5\u8bdd\u603b\u7ed3<\/h4>\n

                \u201c\u628a\u4e00\u6279 prompt \u53d8\u6210 B\u00d7G \u6761 completion，\u5956\u52b1\u6253\u5206\u540e\u7b97\u7ec4\u5185\u4f18\u52bf，\u6253\u5305\u6210\u53ef\u76f4\u63a5\u5582\u7ed9\u635f\u5931\u51fd\u6570\u7684\u8bad\u7ec3\u5b57\u5178\u3002\u201d<\/p>\n

                \n

                \u2705 \u6838\u5fc3\u6d41\u7a0b（8 \u6b65\u901f\u8bb0）<\/h4>\n\n\u6b65\u9aa4\u5173\u952e\u52a8\u4f5c\u4ee3\u7801\/\u53d8\u91cf<\/tr>\n\n\n\n\n\n\n\n\n\n
                1\ufe0f\u20e3 \u8f93\u5165\u51c6\u5907<\/td>\n\u63d0\u53d6 prompt\u3001\u5904\u7406\u56fe\u6587<\/td>\nprompts, has_images<\/td>\n<\/tr>\n
                2\ufe0f\u20e3 token \u5316<\/td>\n\u5de6\u586b\u5145\u3001\u622a\u65ad\u3001\u4fdd\u62a4\u7279\u6b8a token<\/td>\ntruncate_with_protected_tokens<\/td>\n<\/tr>\n
                3\ufe0f\u20e3 \u751f\u6210<\/td>\nvLLM \/ transformers \/ paged-attention \u4e09\u9009\u4e00<\/td>\ncompletion_ids<\/td>\n<\/tr>\n
                4\ufe0f\u20e3 \u540e\u5904\u7406<\/td>\n\u622a\u65ad EOS\u3001\u751f\u6210 mask<\/td>\ncompletion_mask, completion_lengths<\/td>\n<\/tr>\n
                5\ufe0f\u20e3 \u5956\u52b1\u6253\u5206<\/td>\n\u8c03\u7528 _calculate_rewards<\/td>\nrewards_per_func<\/td>\n<\/tr>\n
                6\ufe0f\u20e3 \u52a0\u6743\u6c42\u548c<\/td>\n\u591a\u5956\u52b1\u51fd\u6570\u52a0\u6743 \u2192 \u5355\u6761\u5956\u52b1<\/td>\nrewards<\/td>\n<\/tr>\n
                7\ufe0f\u20e3 \u7ec4\u5185\u5f52\u4e00\u5316<\/td>\n\u5747\u503c-\u65b9\u5dee \u2192 \u4f18\u52bf<\/td>\nadvantages<\/td>\n<\/tr>\n
                8\ufe0f\u20e3 \u8de8\u8fdb\u7a0b\u540c\u6b65<\/td>\ngather & slice \u4fdd\u8bc1\u5206\u5e03\u5f0f\u4e00\u81f4<\/td>\ngather, process_slice<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
                \n

                \u2705 \u8f93\u51fa\u5b57\u5178（\u53ef\u76f4\u63a5\u5582\u635f\u5931）<\/h4>\n

                {<\/span>
                \n "prompt_ids"<\/span> :<\/span> Tensor,<\/span> # (B\u00d7G, P)<\/span>
                \n "prompt_mask"<\/span> :<\/span> Tensor,<\/span> # (B\u00d7G, P)<\/span>
                \n "completion_ids"<\/span> :<\/span> Tensor,<\/span> # (B\u00d7G, C)<\/span>
                \n "completion_mask"<\/span> :<\/span> Tensor,<\/span> # (B\u00d7G, C)<\/span>
                \n "advantages"<\/span> :<\/span> Tensor,<\/span> # (B\u00d7G,) \u7ec4\u5185\u5f52\u4e00\u5316\u4f18\u52bf<\/span>
                \n "old_per_token_logps"<\/span> :<\/span> Tensor,<\/span> # \u53ef\u9009，\u91cd\u8981\u6027\u91c7\u6837<\/span>
                \n "ref_per_token_logps"<\/span> :<\/span> Tensor,<\/span> # \u53ef\u9009，KL \u8ba1\u7b97<\/span>
                \n .<\/span>.<\/span>.<\/span> # \u56fe\u50cf\u76f8\u5173\u5b57\u6bb5（\u82e5\u591a\u6a21\u6001）<\/span>
                \n}<\/span><\/p>\n

                \n

                \u2705 \u518d\u603b\u7ed3<\/h4>\n

                \u53ea\u8981\u8c03\u7528\u4e00\u6b21 _generate_and_score_completions，\u5c31\u80fd\u628a\u201cprompt\u201d\u53d8\u6210\u201c\u5e26\u4f18\u52bf\u7684\u8bad\u7ec3\u6837\u672c\u201d\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"

                \u6587\u7ae0\u6d4f\u89c8\u9605\u8bfb145\u6b21\u3002GRPO\uff08Group Relative Policy Optimization\uff09\u516c\u5f0f\u6458\u8981\uff1a GRPO\u91c7\u7528\u7ec4\u5185\u5f52\u4e00\u5316\u4f18\u52bf\u8ba1\u7b97\uff0c\u901a\u8fc7\u7ec4\u5185\u5747\u503c\u548c\u6807\u51c6\u5dee\u5bf9\u5956\u52b1\u8fdb\u884c\u6807\u51c6\u5316\u5904\u7406\u3002\u7b56\u7565\u4f18\u5316\u91c7\u7528token\u7ea7\u522b\u7684\u88c1\u526a\u76ee\u6807\u51fd\u6570\uff08\u516c\u5f0f2\uff09\uff0c\u7ed3\u5408KL\u6b63\u5219\u9879\uff08\u516c\u5f0f3\uff09\u5f62\u6210\u6700\u7ec8\u635f\u5931\u51fd\u6570\uff08\u516c\u5f0f4\uff09\u3002\u4e0eBNPO\u76f8\u6bd4\uff0cGRPO\u4f7f\u7528\u9759\u6001\u7684\u7ec4\u5185\u5747\u503c-\u65b9\u5dee\u5f52\u4e00\u5316\uff0c\u9002\u7528\u4e8e\u901a\u7528\u573a\u666f\uff1b\u800cBNPO\u91c7\u7528\u52a8\u6001Beta\u5206\u5e03\u81ea\u9002\u5e94\uff0c\u66f4\u9002\u5408\u4e8c\u503c\u5956\u52b1\u4efb\u52a1\u3002\u5728HuggingFace TRL\u5b9e\u73b0\u4e2d\uff0c\u5956\u52b1\u8ba1\u7b97\u901a\u8fc7\u591a\u4e2a\u5956\u52b1\u51fd\u6570\u5e76\u884c\u5904\u7406\uff0c\u652f\u6301\u52a8\u6001\u5956\u52b1\u8c03\u6574\u3002<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[6241,752],"topic":[],"class_list":["post-58994","post","type-post","status-publish","format-standard","hentry","category-server","tag-rlhf","tag-752"],"yoast_head":"\nGRPO\uff08Group Relative Policy Optimization\uff09\u516c\u5f0f\u901f\u89c8 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.wsisp.com\/helps\/58994.html\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"GRPO\uff08Group Relative Policy Optimization\uff09\u516c\u5f0f\u901f\u89c8 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"og:description\" content=\"\u6587\u7ae0\u6d4f\u89c8\u9605\u8bfb145\u6b21\u3002GRPO\uff08Group Relative Policy Optimization\uff09\u516c\u5f0f\u6458\u8981\uff1a GRPO\u91c7\u7528\u7ec4\u5185\u5f52\u4e00\u5316\u4f18\u52bf\u8ba1\u7b97\uff0c\u901a\u8fc7\u7ec4\u5185\u5747\u503c\u548c\u6807\u51c6\u5dee\u5bf9\u5956\u52b1\u8fdb\u884c\u6807\u51c6\u5316\u5904\u7406\u3002\u7b56\u7565\u4f18\u5316\u91c7\u7528token\u7ea7\u522b\u7684\u88c1\u526a\u76ee\u6807\u51fd\u6570\uff08\u516c\u5f0f2\uff09\uff0c\u7ed3\u5408KL\u6b63\u5219\u9879\uff08\u516c\u5f0f3\uff09\u5f62\u6210\u6700\u7ec8\u635f\u5931\u51fd\u6570\uff08\u516c\u5f0f4\uff09\u3002\u4e0eBNPO\u76f8\u6bd4\uff0cGRPO\u4f7f\u7528\u9759\u6001\u7684\u7ec4\u5185\u5747\u503c-\u65b9\u5dee\u5f52\u4e00\u5316\uff0c\u9002\u7528\u4e8e\u901a\u7528\u573a\u666f\uff1b\u800cBNPO\u91c7\u7528\u52a8\u6001Beta\u5206\u5e03\u81ea\u9002\u5e94\uff0c\u66f4\u9002\u5408\u4e8c\u503c\u5956\u52b1\u4efb\u52a1\u3002\u5728HuggingFace TRL\u5b9e\u73b0\u4e2d\uff0c\u5956\u52b1\u8ba1\u7b97\u901a\u8fc7\u591a\u4e2a\u5956\u52b1\u51fd\u6570\u5e76\u884c\u5904\u7406\uff0c\u652f\u6301\u52a8\u6001\u5956\u52b1\u8c03\u6574\u3002\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.wsisp.com\/helps\/58994.html\" \/>\n<meta property=\"og:site_name\" content=\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"article:published_time\" content=\"2025-08-16T11:45:01+00:00\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"21 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/58994.html\",\"url\":\"https:\/\/www.wsisp.com\/helps\/58994.html\",\"name\":\"GRPO\uff08Group Relative Policy Optimization\uff09\u516c\u5f0f\u901f\u89c8 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"isPartOf\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\"},\"datePublished\":\"2025-08-16T11:45:01+00:00\",\"dateModified\":\"2025-08-16T11:45:01+00:00\",\"author\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/58994.html#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.wsisp.com\/helps\/58994.html\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/58994.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.wsisp.com\/helps\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"GRPO\uff08Group Relative Policy Optimization\uff09\u516c\u5f0f\u901f\u89c8\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\",\"url\":\"https:\/\/www.wsisp.com\/helps\/\",\"name\":\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"description\":\"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"contentUrl\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"caption\":\"admin\"},\"sameAs\":[\"http:\/\/wp.wsisp.com\"],\"url\":\"https:\/\/www.wsisp.com\/helps\/author\/admin\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"GRPO\uff08Group Relative Policy Optimization\uff09\u516c\u5f0f\u901f\u89c8 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.wsisp.com\/helps\/58994.html","og_locale":"zh_CN","og_type":"article","og_title":"GRPO\uff08Group Relative Policy Optimization\uff09\u516c\u5f0f\u901f\u89c8 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","og_description":"\u6587\u7ae0\u6d4f\u89c8\u9605\u8bfb145\u6b21\u3002GRPO\uff08Group Relative Policy Optimization\uff09\u516c\u5f0f\u6458\u8981\uff1a GRPO\u91c7\u7528\u7ec4\u5185\u5f52\u4e00\u5316\u4f18\u52bf\u8ba1\u7b97\uff0c\u901a\u8fc7\u7ec4\u5185\u5747\u503c\u548c\u6807\u51c6\u5dee\u5bf9\u5956\u52b1\u8fdb\u884c\u6807\u51c6\u5316\u5904\u7406\u3002\u7b56\u7565\u4f18\u5316\u91c7\u7528token\u7ea7\u522b\u7684\u88c1\u526a\u76ee\u6807\u51fd\u6570\uff08\u516c\u5f0f2\uff09\uff0c\u7ed3\u5408KL\u6b63\u5219\u9879\uff08\u516c\u5f0f3\uff09\u5f62\u6210\u6700\u7ec8\u635f\u5931\u51fd\u6570\uff08\u516c\u5f0f4\uff09\u3002\u4e0eBNPO\u76f8\u6bd4\uff0cGRPO\u4f7f\u7528\u9759\u6001\u7684\u7ec4\u5185\u5747\u503c-\u65b9\u5dee\u5f52\u4e00\u5316\uff0c\u9002\u7528\u4e8e\u901a\u7528\u573a\u666f\uff1b\u800cBNPO\u91c7\u7528\u52a8\u6001Beta\u5206\u5e03\u81ea\u9002\u5e94\uff0c\u66f4\u9002\u5408\u4e8c\u503c\u5956\u52b1\u4efb\u52a1\u3002\u5728HuggingFace TRL\u5b9e\u73b0\u4e2d\uff0c\u5956\u52b1\u8ba1\u7b97\u901a\u8fc7\u591a\u4e2a\u5956\u52b1\u51fd\u6570\u5e76\u884c\u5904\u7406\uff0c\u652f\u6301\u52a8\u6001\u5956\u52b1\u8c03\u6574\u3002","og_url":"https:\/\/www.wsisp.com\/helps\/58994.html","og_site_name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","article_published_time":"2025-08-16T11:45:01+00:00","author":"admin","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"admin","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"21 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.wsisp.com\/helps\/58994.html","url":"https:\/\/www.wsisp.com\/helps\/58994.html","name":"GRPO\uff08Group Relative Policy Optimization\uff09\u516c\u5f0f\u901f\u89c8 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","isPartOf":{"@id":"https:\/\/www.wsisp.com\/helps\/#website"},"datePublished":"2025-08-16T11:45:01+00:00","dateModified":"2025-08-16T11:45:01+00:00","author":{"@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41"},"breadcrumb":{"@id":"https:\/\/www.wsisp.com\/helps\/58994.html#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.wsisp.com\/helps\/58994.html"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.wsisp.com\/helps\/58994.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.wsisp.com\/helps"},{"@type":"ListItem","position":2,"name":"GRPO\uff08Group Relative Policy Optimization\uff09\u516c\u5f0f\u901f\u89c8"}]},{"@type":"WebSite","@id":"https:\/\/www.wsisp.com\/helps\/#website","url":"https:\/\/www.wsisp.com\/helps\/","name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","description":"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41","name":"admin","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/","url":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","contentUrl":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","caption":"admin"},"sameAs":["http:\/\/wp.wsisp.com"],"url":"https:\/\/www.wsisp.com\/helps\/author\/admin"}]}},"_links":{"self":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/58994","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/comments?post=58994"}],"version-history":[{"count":0,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/58994\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media?parent=58994"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/categories?post=58994"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/tags?post=58994"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/topic?post=58994"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}