{"id":70627,"date":"2026-02-02T05:49:08","date_gmt":"2026-02-01T21:49:08","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/70627.html"},"modified":"2026-02-02T05:49:08","modified_gmt":"2026-02-01T21:49:08","slug":"%e5%a4%a7%e6%a8%a1%e5%9e%8b%e5%a4%9a%e7%bb%b4%e5%ba%a6%e8%83%bd%e5%8a%9b%e8%af%84%e4%bc%b0%e4%b8%8e%e5%b9%bb%e8%a7%89%e5%ae%9a%e9%87%8f%e8%af%84%e6%b5%8b%e5%ae%9e%e6%88%98%e5%9f%ba%e4%ba%8emmlu","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/70627.html","title":{"rendered":"\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30\u4e0e\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u5b9e\u6218(\u57fa\u4e8eMMLU\u548cBBH\u8bc4\u6d4b\u96c6)"},"content":{"rendered":"

\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30\u4e0e\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u5b9e\u6218（\u57fa\u4e8eMMLU\/BBH\u8bc4\u6d4b\u96c6）<\/h2>\n

\u6587\u6863\u6982\u8ff0<\/h3>\n

\u6587\u7ae0\u6838\u5fc3\u4ef7\u503c<\/h4>\n
  • \n

    \u7cfb\u7edf\u638c\u63e1\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u6d4b\u4f53\u7cfb（MMLU\/BBH \u6838\u5fc3\u8bc4\u6d4b\u96c6）<\/p>\n<\/li>\n

  • \n

    \u843d\u5730\u5f00\u6e90\u6a21\u578b\u7684\u81ea\u52a8\u5316\u8bc4\u6d4b\u6d41\u7a0b（\u73af\u5883\u642d\u5efa\u2192\u6570\u636e\u96c6\u52a0\u8f7d\u2192\u8bc4\u6d4b\u6267\u884c\u2192\u7ed3\u679c\u5206\u6790）<\/p>\n<\/li>\n

  • \n

    \u638c\u63e1\u6a21\u578b\u5e7b\u89c9（Hallucination）\u7684\u5b9a\u91cf\u8bc4\u4f30\u65b9\u6cd5\u4e0e\u843d\u5730\u5de5\u5177<\/p>\n<\/li>\n

  • \n

    \u57fa\u4e8e\u8bc4\u6d4b\u7ed3\u679c\u7684\u6a21\u578b\u4f18\u5316\u65b9\u5411\u4e0e\u5b9e\u6218\u6280\u5de7<\/p>\n<\/li>\n

  • \n

    \u9002\u914d2025\u5e74\u6700\u65b0\u5f00\u6e90\u751f\u6001\u7684\u5b8c\u6574\u53ef\u590d\u7528\u4ee3\u7801\u4f53\u7cfb<\/p>\n<\/li>\n

    \u5b66\u4e60\u76ee\u6807<\/h4>\n
  • \n

    \u7406\u89e3MMLU\/BBH\u8bc4\u6d4b\u96c6\u7684\u8bbe\u8ba1\u903b\u8f91\u4e0e\u9002\u7528\u573a\u666f<\/p>\n<\/li>\n

  • \n

    \u638c\u63e1lm-evaluation-harness（\u4e3b\u6d41\u8bc4\u6d4b\u6846\u67b6）\u7684\u672c\u5730\u90e8\u7f72\u4e0e\u5b9a\u5236\u5316\u4f7f\u7528<\/p>\n<\/li>\n

  • \n

    \u7cbe\u901a\u5927\u6a21\u578b\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u4f30\u7684\u6838\u5fc3\u6307\u6807（FactScore\u3001Faithfulness\u3001Hallucination Rate）<\/p>\n<\/li>\n

  • \n

    \u80fd\u591f\u72ec\u7acb\u5b8c\u6210\u5f00\u6e90\u5927\u6a21\u578b（\u5982Qwen2.5\u3001Llama3）\u7684\u5168\u7ef4\u5ea6\u8bc4\u6d4b<\/p>\n<\/li>\n

  • \n

    \u57fa\u4e8e\u8bc4\u6d4b\u7ed3\u679c\u5b9a\u4f4d\u6a21\u578b\u77ed\u677f\u5e76\u63d0\u51fa\u4f18\u5316\u7b56\u7565<\/p>\n<\/li>\n

    \u4e00\u3001\u5927\u6a21\u578b\u80fd\u529b\u8bc4\u4f30\u4f53\u7cfb\u6982\u8ff0<\/h3>\n

    1.1 \u6838\u5fc3\u8bc4\u6d4b\u96c6\u5b9a\u4f4d<\/h4>\n

    \u5927\u6a21\u578b\u7684\u80fd\u529b\u8bc4\u4f30\u9700\u8986\u76d6\u77e5\u8bc6\u5e7f\u5ea6\u3001\u63a8\u7406\u6df1\u5ea6\u3001\u4efb\u52a1\u9002\u914d\u6027\u4e09\u5927\u7ef4\u5ea6，MMLU\u548cBBH\u662f\u76ee\u524d\u5de5\u4e1a\u754c\u4e0e\u5b66\u672f\u754c\u6700\u4e3b\u6d41\u7684\u8bc4\u6d4b\u57fa\u51c6：<\/p>\n\n\u8bc4\u6d4b\u96c6\u5168\u79f0\u6838\u5fc3\u80fd\u529b\u7ef4\u5ea6\u6570\u636e\u89c4\u6a21\u9002\u7528\u573a\u666f<\/tr>\n\n\n\n\n\n
    MMLU<\/td>\nMassive Multitask Language Understanding<\/td>\n\u901a\u7528\u77e5\u8bc6\u3001\u5b66\u79d1\u80fd\u529b（57\u4e2a\u79d1\u76ee）<\/td>\n14k+\u9009\u62e9\u9898<\/td>\n\u57fa\u7840\u80fd\u529b\u6478\u5e95\u3001\u8de8\u5b66\u79d1\u77e5\u8bc6\u8bc4\u4f30<\/td>\n<\/tr>\n
    BBH<\/td>\nBig Bench Hard<\/td>\n\u590d\u6742\u63a8\u7406\u3001\u903b\u8f91\u601d\u7ef4\u3001\u4efb\u52a1\u62c6\u89e3<\/td>\n200+\u5b50\u4efb\u52a1<\/td>\n\u9ad8\u9636\u63a8\u7406\u80fd\u529b\u3001\u590d\u6742\u4efb\u52a1\u5904\u7406\u8bc4\u4f30<\/td>\n<\/tr>\n
    TruthfulQA<\/td>\n–<\/td>\n\u4e8b\u5b9e\u51c6\u786e\u6027\u3001\u53cd\u5e7b\u89c9\u80fd\u529b<\/td>\n817\u4e2a\u95ee\u9898<\/td>\n\u5e7b\u89c9\u521d\u6b65\u7b5b\u67e5<\/td>\n<\/tr>\n
    FactScore<\/td>\n–<\/td>\n\u751f\u6210\u5185\u5bb9\u4e8b\u5b9e\u4e00\u81f4\u6027<\/td>\n\u81ea\u5b9a\u4e49\u6587\u672c<\/td>\n\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u4f30<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    1.2 \u6a21\u578b\u5e7b\u89c9\u7684\u6838\u5fc3\u95ee\u9898<\/h4>\n

    \u5e7b\u89c9（Hallucination）\u662f\u6307\u5927\u6a21\u578b\u751f\u6210\u7684\u6587\u672c\u5728\u8bed\u6cd5\u548c\u8bed\u4e49\u4e0a\u770b\u4f3c\u901a\u987a\u5408\u7406，\u4f46\u5185\u5bb9\u4e0e\u4e8b\u5b9e\u4e0d\u7b26\u3001\u903b\u8f91\u81ea\u76f8\u77db\u76fe\u6216\u4e0e\u5176\u8f93\u5165\u6e90（Prompt\/Context）\u4e0d\u4e00\u81f4\u7684\u73b0\u8c61\u3002\u5728\u5b9a\u91cf\u8bc4\u4f30\u4e2d，\u6211\u4eec\u4e0d\u518d\u4ec5\u4ec5\u5c06\u5176\u89c6\u4e3a\u201c\u9519\u8bef\u201d，\u800c\u662f\u5c06\u5176\u7ec6\u5206\u4e3a\u591a\u7ef4\u5ea6\u7684\u6280\u672f\u6311\u6218\u3002<\/p>\n

    1.2.1 \u5e7b\u89c9\u5206\u7c7b\u5b66（Taxonomy）\u6811\u5f62\u56fe<\/h5>\n

    \u4e3a\u4e86\u7cbe\u51c6\u91cf\u5316\u5e7b\u89c9，\u5fc5\u987b\u660e\u786e\u5e7b\u89c9\u53d1\u751f\u7684\u5c42\u7ea7\u4e0e\u7c7b\u578b\u3002\u4ee5\u4e0b\u662f\u57fa\u4e8e\u5de5\u4e1a\u754c\u4e3b\u6d41\u6807\u51c6\u7684\u5e7b\u89c9\u5206\u7c7b\u4f53\u7cfb：<\/p>\n

    \u4ee3\u7801\u6bb5<\/p>\n

    graph TD
    \n A[<\/span>\u5927\u6a21\u578b\u5e7b\u89c9 Hallucination]<\/span> —><\/span> B(<\/span>\u6309\u7167\u4e0e\u6e90\u5934\u7684\u51b2\u7a81\u5206\u7c7b)<\/span>
    \n A —><\/span> C(<\/span>\u6309\u7167\u751f\u6210\u673a\u5236\u5206\u7c7b)<\/span><\/p>\n

    B —><\/span> B1[<\/span>\u5fe0\u5b9e\u5ea6\u7f3a\u5931 Unfaithfulness]<\/span>
    \n B1 —><\/span> B11[<\/span>\u8f93\u5165\u51b2\u7a81 Input-Conflicting]<\/span>
    \n B11 —>|<\/span>\u5ffd\u7565Prompt\u7ea6\u675f|<\/span> B111[<\/span>\u6307\u4ee4\u9075\u5faa\u5931\u8d25]<\/span>
    \n B11 —>|<\/span>\u4e0e\u4e0a\u4e0b\u6587\u77db\u76fe|<\/span> B112[<\/span>\u4e0a\u4e0b\u6587\u4e0d\u4e00\u81f4]<\/span>
    \n B1 —><\/span> B12[<\/span>\u903b\u8f91\u81ea\u6d3d\u6027\u7f3a\u5931 Self-Inconsistency]<\/span>
    \n B12 —>|<\/span>\u524d\u540e\u6587\u77db\u76fe|<\/span> B121[<\/span>\u751f\u6210\u5185\u5bb9\u5185\u90e8\u903b\u8f91\u51b2\u7a81]<\/span><\/p>\n

    B —><\/span> B2[<\/span>\u4e8b\u5b9e\u6027\u9519\u8bef Factual Error]<\/span>
    \n B2 —><\/span> B21[<\/span>\u4e16\u754c\u77e5\u8bc6\u51b2\u7a81 World-Knowledge Conflicting]<\/span>
    \n B21 —>|<\/span>\u5f20\u51a0\u674e\u6234|<\/span> B211[<\/span>\u5b9e\u4f53\u5173\u7cfb\u9519\u8bef]<\/span>
    \n B21 —>|<\/span>\u65e0\u4e2d\u751f\u6709|<\/span> B212[<\/span>\u5b8c\u5168\u634f\u9020\u4e8b\u5b9e]<\/span>
    \n B21 —>|<\/span>\u65f6\u6548\u6027\u9519\u8bef|<\/span> B213[<\/span>\u5f15\u7528\u8fc7\u65f6\u4fe1\u606f]<\/span><\/p>\n

    C —><\/span> C1[<\/span>\u5916\u5728\u5e7b\u89c9 Extrinsic]<\/span>
    \n C1 —>|<\/span>\u65e0\u6cd5\u9a8c\u8bc1|<\/span> C11[<\/span>\u751f\u6210\u5185\u5bb9\u65e2\u65e0\u6cd5\u88ab\u8bc1\u5b9e\u4e5f\u65e0\u6cd5\u88ab\u8bc1\u4f2a]<\/span>
    \n C —><\/span> C2[<\/span>\u5185\u5728\u5e7b\u89c9 Intrinsic]<\/span>
    \n C2 —>|<\/span>\u76f4\u63a5\u51b2\u7a81|<\/span> C21[<\/span>\u751f\u6210\u5185\u5bb9\u76f4\u63a5\u8fdd\u80cc\u73b0\u6709\u77e5\u8bc6\u6e90]<\/span><\/p>\n

    1.2.2 \u5b9a\u91cf\u8bc4\u4f30\u7684\u4e09\u5927\u6838\u5fc3\u7ef4\u5ea6<\/h5>\n

    \u9488\u5bf9\u4e0a\u8ff0\u5206\u7c7b，\u5e7b\u89c9\u7684\u5b9a\u91cf\u8bc4\u4f30（\u5982\u4f7f\u7528FactScore）\u9700\u91cd\u70b9\u89e3\u51b3\u4ee5\u4e0b\u4e09\u5927\u6838\u5fc3\u95ee\u9898：<\/p>\n

    1. \u4e8b\u5b9e\u4e00\u81f4\u6027\u4e0e\u7c92\u5ea6（Factual Consistency & Granularity）<\/p>\n

      \n
    • \u95ee\u9898\u5b9a\u4e49：\u751f\u6210\u5185\u5bb9\u4e0e\u5ba2\u89c2\u4e8b\u5b9e\u7684\u5339\u914d\u7a0b\u5ea6\u3002<\/li>\n
    • \u8bc4\u4f30\u96be\u70b9：\u5e7b\u89c9\u5f80\u5f80\u6f5c\u4f0f\u5728\u7ec6\u8282\u4e2d\u3002\n
        \n
      • \u5b9e\u4f53\u7ea7（Entity Level）：\u4eba\u7269\u3001\u5730\u70b9\u3001\u65f6\u95f4\u9519\u8bef（\u4f8b\u5982：\u5c06\u201c\u674e\u767d\u201d\u8bef\u8ba4\u4e3a\u662f\u201c\u5b8b\u671d\u8bd7\u4eba\u201d）\u3002<\/li>\n
      • \u5173\u7cfb\u7ea7（Relation Level）：\u5b9e\u4f53\u95f4\u7684\u52a8\u4f5c\u6216\u5c5e\u6027\u9519\u8bef（\u4f8b\u5982：\u867d\u7136\u4eba\u7269\u548c\u7535\u5f71\u90fd\u5b58\u5728，\u4f46\u8be5\u4eba\u7269\u5e76\u672a\u53c2\u6f14\u8be5\u7535\u5f71）\u3002<\/li>\n<\/ul>\n<\/li>\n
      • \u91cf\u5316\u76ee\u6807：\u8ba1\u7b97\u539f\u5b50\u7ea7\u4e8b\u5b9e（Atomic Facts）\u7684\u51c6\u786e\u7387，\u800c\u975e\u6574\u6bb5\u6587\u672c\u7684\u76f8\u4f3c\u5ea6\u3002<\/li>\n<\/ul>\n

        2. \u6765\u6e90\u53ef\u8ffd\u6eaf\u6027\u4e0e\u8bc1\u636e\u5f52\u56e0（Source Traceability & Attribution）<\/p>\n

          \n
        • \u95ee\u9898\u5b9a\u4e49：\u6a21\u578b\u751f\u6210\u7684\u6bcf\u4e00\u4e2a\u77e5\u8bc6\u70b9，\u662f\u5426\u90fd\u80fd\u5bf9\u5e94\u5230\u53ef\u4fe1\u7684\u6570\u636e\u6e90（Reference）\u3002<\/li>\n
        • \u8bc4\u4f30\u96be\u70b9：\u6a21\u578b\u53ef\u80fd\u751f\u6210\u6b63\u786e\u7684\u5e9f\u8bdd，\u6216\u8005\u5f15\u7528\u4e0d\u5b58\u5728\u7684\u6587\u732e（\u5f15\u7528\u5e7b\u89c9）\u3002<\/li>\n
        • \u91cf\u5316\u76ee\u6807：Citation Recall（\u5f15\u7528\u53ec\u56de\u7387）\u4e0eCitation Precision（\u5f15\u7528\u7cbe\u786e\u7387）\u3002\u5373\u6a21\u578b\u4e0d\u4ec5\u8981\u56de\u7b54\u6b63\u786e，\u8fd8\u5fc5\u987b\u80fd\u6b63\u786e\u6307\u51fa\u201c\u6211\u4e4b\u6240\u4ee5\u8fd9\u4e48\u8bf4，\u662f\u56e0\u4e3a\u4f9d\u636e\u4e86\u6587\u6863X\u7684\u7b2cY\u6bb5\u201d\u3002<\/li>\n<\/ul>\n

          3. \u5fe0\u5b9e\u5ea6\u4e0e\u6307\u4ee4\u4f9d\u4ece（Faithfulness vs. Instruction Following）<\/p>\n

            \n
          • \u95ee\u9898\u5b9a\u4e49：\u5728RAG（\u68c0\u7d22\u589e\u5f3a\u751f\u6210）\u573a\u666f\u4e0b，\u6a21\u578b\u662f\u5426\u4e25\u683c\u57fa\u4e8e\u68c0\u7d22\u5230\u7684\u4e0a\u4e0b\u6587\u56de\u7b54，\u800c\u4e0d\u662f\u901a\u8fc7\u5185\u90e8\u8bb0\u5fc6\u201c\u8111\u8865\u201d\u3002<\/li>\n
          • \u8bc4\u4f30\u96be\u70b9：\u5f53\u68c0\u7d22\u5230\u7684\u4e0a\u4e0b\u6587\u662f\u9519\u8bef\u7684，\u6a21\u578b\u5e94\u8be5\u201c\u5fe0\u5b9e\u5730\u201d\u56de\u7b54\u9519\u8bef\u4fe1\u606f，\u8fd8\u662f\u5229\u7528\u5185\u90e8\u77e5\u8bc6\u4fee\u6b63？（\u901a\u5e38\u5728RAG\u8bc4\u6d4b\u4e2d，\u4f18\u5148\u8003\u5bdf\u5bf9\u4e0a\u4e0b\u6587\u7684\u5fe0\u5b9e\u5ea6）\u3002<\/li>\n
          • \u91cf\u5316\u76ee\u6807：\u68c0\u6d4bHallucination Rate（\u5e7b\u89c9\u7387），\u5373\u751f\u6210\u5185\u5bb9\u4e2d\u672a\u88ab\u4e0a\u4e0b\u6587\u652f\u6491\u7684\u4fe1\u606f\u6bd4\u4f8b\u3002<\/li>\n<\/ul>\n
            1.2.3 \u5fc5\u987b\u5173\u6ce8\u7684\u201c\u96ea\u7403\u6548\u5e94\u201d<\/h5>\n

            \u5728\u957f\u6587\u672c\u751f\u6210（Long-context Generation）\u4e2d，\u65e9\u671f\u7684\u8f7b\u5fae\u5e7b\u89c9\u4f1a\u5bfc\u81f4\u540e\u7eed\u63a8\u7406\u57fa\u4e8e\u9519\u8bef\u524d\u63d0\u8fdb\u884c，\u4ece\u800c\u4ea7\u751f\u903b\u8f91\u7ea7\u8054\u9519\u8bef（Snowballing Hallucination）\u3002BBH\u8bc4\u6d4b\u96c6\u4e2d\u7684\u63a8\u7406\u4efb\u52a1\u6b63\u662f\u4e3a\u4e86\u68c0\u6d4b\u8fd9\u79cd\u5728\u590d\u6742\u903b\u8f91\u94fe\u6761\u4e2d\u4fdd\u6301\u4e8b\u5b9e\u4e00\u81f4\u6027\u7684\u80fd\u529b\u3002<\/p>\n

            1.3 \u8bc4\u6d4b\u6280\u672f\u6808\u9009\u578b（2025\u6700\u65b0）<\/h4>\n\n\u7ec4\u4ef6\u9009\u578b\u7248\u672c\u6838\u5fc3\u4f18\u52bf<\/tr>\n\n\n\n\n\n\n\n
            \u8bc4\u6d4b\u6846\u67b6<\/td>\nlm-evaluation-harness<\/td>\n0.4.9<\/td>\n\u652f\u6301MMLU\/BBH\u7b49100+\u8bc4\u6d4b\u96c6，\u9002\u914d\u4e3b\u6d41\u5f00\u6e90\u6a21\u578b<\/td>\n<\/tr>\n
            \u6a21\u578b\u52a0\u8f7d<\/td>\ntransformers<\/td>\n4.41.2<\/td>\n\u652f\u6301Qwen2.5\/Llama3\u7b49\u6700\u65b0\u5f00\u6e90\u6a21\u578b<\/td>\n<\/tr>\n
            \u52a0\u901f\u5e93<\/td>\nvLLM<\/td>\n0.4.2<\/td>\n\u5927\u5e45\u63d0\u5347\u8bc4\u6d4b\u901f\u5ea6（\u541e\u5410\u91cf\u63d0\u53475-10\u500d）<\/td>\n<\/tr>\n
            \u5e7b\u89c9\u8bc4\u4f30\u5de5\u5177<\/td>\nfactscore<\/td>\n0.4.0<\/td>\n\u4e3b\u6d41\u7684\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u4f30\u5de5\u5177，\u652f\u6301\u4e2d\u6587\u9002\u914d<\/td>\n<\/tr>\n
            \u73af\u5883\u7ba1\u7406<\/td>\nConda<\/td>\n23.10.0<\/td>\n\u9694\u79bb\u8bc4\u6d4b\u73af\u5883，\u907f\u514d\u4f9d\u8d56\u51b2\u7a81<\/td>\n<\/tr>\n
            \u8ba1\u7b97\u6846\u67b6<\/td>\nPyTorch<\/td>\n2.5.1<\/td>\n\u9002\u914d\u6700\u65b0GPU\u67b6\u6784，\u652f\u6301\u6df7\u5408\u7cbe\u5ea6\u63a8\u7406<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

            \u4e8c\u3001\u8bc4\u6d4b\u73af\u5883\u642d\u5efa（2025\u6700\u65b0）<\/h3>\n

            2.1 \u73af\u5883\u8981\u6c42<\/h4>\n\n\u7ef4\u5ea6\u6700\u4f4e\u914d\u7f6e\u63a8\u8350\u914d\u7f6e\u8bf4\u660e<\/tr>\n\n\n\n\n\n\n
            \u64cd\u4f5c\u7cfb\u7edf<\/td>\nUbuntu 20.04+\/Windows 11\/macOS 14+<\/td>\nUbuntu 22.04<\/td>\nLinux\u5bf9CUDA\u652f\u6301\u6700\u4f18<\/td>\n<\/tr>\n
            CPU<\/td>\n8\u6838<\/td>\n16\u6838+<\/td>\n\u6570\u636e\u9884\u5904\u7406\/\u8bc4\u6d4b\u8c03\u5ea6<\/td>\n<\/tr>\n
            GPU<\/td>\nNVIDIA RTX 3090 (24GB)<\/td>\nNVIDIA A100 (40GB)<\/td>\n7B\u6a21\u578b\u8bc4\u6d4b\u970024GB+\u663e\u5b58，14B\u970040GB+<\/td>\n<\/tr>\n
            CUDA<\/td>\n12.1<\/td>\n12.4<\/td>\n\u9002\u914dPyTorch 2.5+<\/td>\n<\/tr>\n
            Python<\/td>\n3.10<\/td>\n3.10<\/td>\nlm-evaluation-harness\u63a8\u8350\u7248\u672c<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

            2.2 \u73af\u5883\u914d\u7f6e\u6b65\u9aa4<\/h4>\n

            2.2.1 \u521b\u5efa\u72ec\u7acbConda\u73af\u5883<\/h5>\n

            # \u521b\u5efa\u8bc4\u6d4b\u4e13\u5c5e\u73af\u5883
            \nconda create -n llm-eval python=3.10 -y
            \n# \u6fc0\u6d3b\u73af\u5883
            \nconda activate llm-eval<\/p>\n

            2.2.2 \u5b89\u88c5\u6838\u5fc3\u4f9d\u8d56<\/h5>\n

            # \u57fa\u7840\u4f9d\u8d56
            \npip install numpy==1.26.4 pandas==2.2.2 scipy==1.13.1
            \npip install tqdm==4.66.4 rich==13.7.1 python-dotenv==1.0.1<\/p>\n

            # PyTorch（\u9002\u914dCUDA 12.4）
            \npip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 –index-url https:\/\/download.pytorch.org\/whl\/cu124<\/p>\n

            # \u6a21\u578b\u52a0\u8f7d\u4e0e\u63a8\u7406
            \npip install transformers==4.41.2 accelerate==0.31.0 peft==0.11.1
            \npip install vllm==0.4.2 # \u9ad8\u6027\u80fd\u63a8\u7406\u5f15\u64ce（\u53ef\u9009，\u5927\u5e45\u63d0\u5347\u901f\u5ea6）<\/p>\n

            # \u8bc4\u6d4b\u6846\u67b6（\u5b89\u88c5\u6700\u65b0\u7248lm-evaluation-harness）
            \ngit clone https:\/\/github.com\/EleutherAI\/lm-evaluation-harness.git
            \ncd lm-evaluation-harness
            \npip install -e .[all]
            \ncd ..<\/p>\n

            # \u5e7b\u89c9\u8bc4\u4f30\u5de5\u5177
            \npip install factscore==0.4.0
            \npip install rouge-score==0.1.2 bert-score==0.3.13 # \u8f85\u52a9\u8bc4\u4f30\u6307\u6807<\/p>\n

            # \u4e2d\u6587\u9002\u914d\u4f9d\u8d56
            \npip install jieba==0.42.1 zhconv==1.4.1
            \npip install modelscope==1.14.0 # \u4e2d\u6587\u6570\u636e\u96c6\u4e0b\u8f7d<\/p>\n

            2.2.3 \u73af\u5883\u9a8c\u8bc1<\/h5>\n

            # \u9a8c\u8bc1\u73af\u5883\u914d\u7f6e
            \nimport torch
            \nimport transformers
            \nimport lm_eval
            \nimport vllm<\/p>\n

            print(f"PyTorch\u7248\u672c: {torch.__version__}")
            \nprint(f"CUDA\u53ef\u7528: {torch.cuda.is_available()}")
            \nprint(f"Transformers\u7248\u672c: {transformers.__version__}")
            \nprint(f"lm-eval\u7248\u672c: {lm_eval.__version__}")
            \nprint(f"vLLM\u7248\u672c: {vllm.__version__}")<\/p>\n

            # \u9a8c\u8bc1GPU\u914d\u7f6e
            \nif torch.cuda.is_available():
            \n print(f"GPU\u540d\u79f0: {torch.cuda.get_device_name(0)}")
            \n print(f"GPU\u663e\u5b58: {torch.cuda.get_device_properties(0).total_memory \/ 1024**3:.2f} GB")
            \nelse:
            \n print("\u672a\u68c0\u6d4b\u5230GPU，\u5efa\u8bae\u4f7f\u7528GPU\u8fdb\u884c\u8bc4\u6d4b")<\/p>\n

            \u4e09\u3001MMLU\/BBH\u8bc4\u6d4b\u96c6\u5b9e\u6218\u8bc4\u6d4b<\/h3>\n

            3.1 \u8bc4\u6d4b\u6846\u67b6\u6838\u5fc3\u539f\u7406\u4e0e\u67b6\u6784\u8be6\u89e3<\/h4>\n

            lm-evaluation-harness \u4e4b\u6240\u4ee5\u6210\u4e3a\u5de5\u4e1a\u754c\u4e8b\u5b9e\u6807\u51c6，\u662f\u56e0\u4e3a\u5b83\u89e3\u51b3\u4e86\u4e00\u4e2a\u6838\u5fc3\u96be\u9898：\u5982\u4f55\u516c\u5e73\u5730\u5bf9\u6bd4\u4e0d\u540c\u67b6\u6784\u3001\u4e0d\u540cTokenizer\u7684\u6a21\u578b？ \u5b83\u901a\u8fc7\u4e00\u5957\u9ad8\u5ea6\u62bd\u8c61\u7684\u6d41\u6c34\u7ebf，\u5c06\u5f02\u6784\u7684\u6a21\u578b\u4e0e\u6807\u51c6\u5316\u7684\u4efb\u52a1\u89e3\u8026\u3002<\/p>\n

            3.1.1 \u8bc4\u6d4b\u6d41\u6c34\u7ebf\u67b6\u6784\u56fe<\/h5>\n

            \u4ee5\u4e0b\u662f\u8be5\u6846\u67b6\u5185\u90e8\u7684\u5404\u79cd\u7ec4\u4ef6\u5982\u4f55\u534f\u540c\u5de5\u4f5c\u7684\u62d3\u6251\u7ed3\u6784：<\/p>\n

            \u4ee3\u7801\u6bb5<\/p>\n

            graph TD
            \n User[<\/span>\u7528\u6237\u6307\u4ee4 CLI\/API]<\/span> —><\/span> Orchestrator[<\/span>\u6838\u5fc3\u8c03\u5ea6\u5668 Evaluator]<\/span><\/p>\n

            subgraph Data_Pipeline [<\/span>\u6570\u636e\u5904\u7406\u6d41\u6c34\u7ebf]<\/span>
            \n TaskRegistry[<\/span>\u4efb\u52a1\u6ce8\u518c\u4e2d\u5fc3]<\/span> —>|<\/span>\u52a0\u8f7dYAML\u914d\u7f6e|<\/span> TaskDef[<\/span>MMLU\/BBH \u4efb\u52a1\u5b9a\u4e49]<\/span>
            \n TaskDef —>|<\/span>\u6784\u5efaFew-shot|<\/span> ContextBuilder[<\/span>\u4e0a\u4e0b\u6587\u6784\u9020\u5668]<\/span>
            \n ContextBuilder —>|<\/span>doc_to_text|<\/span> Requests[<\/span>\u6807\u51c6\u5316\u8bf7\u6c42\u5bf9\u8c61]<\/span>
            \n end<\/p>\n

            subgraph Model_Bridge [<\/span>\u6a21\u578b\u9002\u914d\u5c42]<\/span>
            \n Orchestrator —>|<\/span>\u5206\u53d1\u8bf7\u6c42|<\/span> Model[<\/span>LM \u62bd\u8c61\u57fa\u7c7b]<\/span>
            \n Model —>|<\/span>HFLM|<\/span> LocalInfer[<\/span>\u672c\u5730\u63a8\u7406 HFLM\/vLLM]<\/span>
            \n Model —>|<\/span>API|<\/span> RemoteInfer[<\/span>API\u8c03\u7528 OpenAI\/Anthropic]<\/span>
            \n LocalInfer —>|<\/span>\u8ba1\u7b97Logits\/\u751f\u6210\u6587\u672c|<\/span> RawOutput[<\/span>\u539f\u59cb\u8f93\u51fa]<\/span>
            \n end<\/p>\n

            subgraph Scoring_System [<\/span>\u8bc4\u5206\u4e0e\u540e\u5904\u7406]<\/span>
            \n RawOutput —>|<\/span>\u89e3\u6790|<\/span> Filter[<\/span>\u8fc7\u6ee4\u5668 Regex\/Stop-seq]<\/span>
            \n Filter —>|<\/span>\u8ba1\u7b97|<\/span> Metric[<\/span>\u6307\u6807\u8ba1\u7b97\u5668]<\/span>
            \n Metric —>|<\/span>acc\/acc_norm|<\/span> FinalScore[<\/span>\u6700\u7ec8\u5f97\u5206]<\/span>
            \n end<\/p>\n

            Orchestrator —><\/span> Data_Pipeline
            \n Requests —><\/span> Model
            \n RawOutput —><\/span> Scoring_System<\/p>\n

            3.1.2 \u6838\u5fc3\u7ec4\u4ef6\u6df1\u5ea6\u89e3\u6790<\/h5>\n

            \u6846\u67b6\u901a\u8fc7\u4e09\u5927\u6838\u5fc3\u62bd\u8c61\u5c42\u5b9e\u73b0\u6807\u51c6\u5316\u8bc4\u6d4b：<\/p>\n

            1. \u4efb\u52a1\u62bd\u8c61\u5c42 (Task Abstraction)<\/p>\n

              \n
            • \u6838\u5fc3\u4f5c\u7528：\u5c06\u5343\u5947\u767e\u602a\u7684\u6570\u636e\u96c6（CSV, JSON, HF Datasets）\u7edf\u4e00\u4e3a\u6807\u51c6\u683c\u5f0f\u3002<\/li>\n
            • \u5173\u952e\u673a\u5236：\n
                \n
              • doc_to_text：\u5c06\u6570\u636e\u884c\u8f6c\u6362\u4e3a\u6a21\u578b\u8f93\u5165\u7684 Prompt（\u4f8b\u5982：\u201cQuestion: \u2026 Answer:\u201d）\u3002<\/li>\n
              • doc_to_target：\u5b9a\u4e49\u6807\u51c6\u7b54\u6848\u7684\u683c\u5f0f\u3002<\/li>\n
              • Few-shot \u91c7\u6837：\u81ea\u52a8\u4ece\u8bad\u7ec3\u96c6\u4e2d\u62bd\u53d6 N \u4e2a\u6837\u672c\u4f5c\u4e3a\u4e0a\u4e0b\u6587，\u786e\u4fdd\u6240\u6709\u6a21\u578b\u770b\u5230\u76f8\u540c\u7684\u793a\u4f8b，\u907f\u514d\u56e0 Prompt \u5dee\u5f02\u5bfc\u81f4\u7684\u8bc4\u5206\u504f\u5dee\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

                2. \u4e24\u79cd\u622a\u7136\u4e0d\u540c\u7684\u63a8\u7406\u6a21\u5f0f (The Two Inference Modes) \u8fd9\u662f\u7406\u89e3 MMLU \u548c BBH \u8bc4\u6d4b\u5dee\u5f02\u7684\u5173\u952e：<\/p>\n

                  \n
                • \u6a21\u5f0f A：\u4f3c\u7136\u5ea6\u8bc4\u5206 (Loglikelihood Scoring) \u2014\u2014 \u9002\u7528\u4e8e MMLU\n
                    \n
                  • \u539f\u7406：\u6a21\u578b\u4e0d\u9700\u8981\u751f\u6210\u6587\u672c\u3002\u6846\u67b6\u8ba1\u7b97\u6a21\u578b\u5bf9\u9009\u9879 A\u3001B\u3001C\u3001D \u7684\u751f\u6210\u6982\u7387（Log Probabilities）\u3002<\/li>\n
                  • \u4f18\u52bf：\u6d88\u9664\u4e86\u201c\u6a21\u578b\u56de\u7b54\u6b63\u786e\u4f46\u683c\u5f0f\u4e0d\u5bf9\u201d\u7684\u5e72\u6270（\u4f8b\u5982\u6a21\u578b\u56de\u7b54"\u9009A" vs \u201c\u7b54\u6848\u662fA\u201d）\u3002<\/li>\n
                  • \u6307\u6807：acc_norm（Byte-length normalized accuracy），\u901a\u8fc7\u5f52\u4e00\u5316\u6d88\u9664 Tokenizer \u5bf9\u957f\u9009\u9879\u7684\u6b67\u89c6\u3002<\/li>\n<\/ul>\n<\/li>\n
                  • \u6a21\u5f0f B：\u751f\u6210\u5f0f\u8bc4\u6d4b (Generative Evaluation) \u2014\u2014 \u9002\u7528\u4e8e GSM8K\/BBH\n
                      \n
                    • \u539f\u7406：\u6a21\u578b\u81ea\u7531\u751f\u6210\u6587\u672c（generate_until），\u76f4\u5230\u9047\u5230\u505c\u6b62\u7b26\u3002<\/li>\n
                    • \u540e\u5904\u7406：\u4f7f\u7528\u6b63\u5219\u8868\u8fbe\u5f0f（Regex）\u63d0\u53d6\u7b54\u6848，\u6216\u901a\u8fc7 exact_match \u8fdb\u884c\u4e25\u683c\u6bd4\u5bf9\u3002<\/li>\n
                    • \u6311\u6218：\u5bf9\u6a21\u578b\u7684\u6307\u4ee4\u9075\u5faa\u80fd\u529b\u8981\u6c42\u66f4\u9ad8\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

                      3. \u52a0\u901f\u4e0e\u7f13\u5b58\u673a\u5236 (Acceleration & Caching)<\/p>\n

                        \n
                      • \u6279\u91cf\u63a8\u7406 (Batching)：\u81ea\u52a8\u5c06\u591a\u4e2a\u8bf7\u6c42\u6253\u5305（Padding），\u6700\u5927\u5316 GPU \u5229\u7528\u7387\u3002<\/li>\n
                      • \u7ed3\u679c\u7f13\u5b58 (Caching)：\u57fa\u4e8e Prompt \u7684\u54c8\u5e0c\u503c\u7f13\u5b58\u63a8\u7406\u7ed3\u679c\u3002\u5982\u679c\u8bc4\u6d4b\u4e2d\u65ad，\u91cd\u542f\u65f6\u53ef\u76f4\u63a5\u8df3\u8fc7\u5df2\u8dd1\u8fc7\u7684\u6837\u672c，\u8fd9\u5bf9\u4e8e 14k+ \u9898\u76ee\u7684 MMLU \u8bc4\u6d4b\u81f3\u5173\u91cd\u8981\u3002<\/li>\n<\/ul>\n
                        3.1.3 \u4e3a\u4ec0\u4e48\u9700\u8981\u6807\u51c6\u5316\u63a5\u53e3？<\/h5>\n

                        \u5728\u6ca1\u6709\u6846\u67b6\u4e4b\u524d，\u8bc4\u6d4b\u5f80\u5f80\u5b58\u5728 \u201cPrompt Hacking\u201d \u73b0\u8c61\u2014\u2014\u901a\u8fc7\u5fae\u8c03 Prompt \u63aa\u8f9e\u8ba9\u7279\u5b9a\u6a21\u578b\u5f97\u5206\u865a\u9ad8\u3002lm-evaluation-harness \u5f3a\u5236\u4f7f\u7528\u793e\u533a\u516c\u8ba4\u7684 Prompt Template，\u786e\u4fdd\u4e86：<\/p>\n

                      • \u53ef\u590d\u73b0\u6027：\u4efb\u4f55\u4eba\u8fd0\u884c\u4ee3\u7801，\u8f93\u5165\u7ed9\u6a21\u578b\u7684 Prompt \u90fd\u662f\u5b8c\u5168\u4e00\u81f4\u7684\u3002<\/li>\n
                      • Tokenizer \u516c\u5e73\u6027：\u901a\u8fc7 acc_norm \u6307\u6807，\u9632\u6b62\u67d0\u4e9b Tokenizer \u56e0\u4e3a\u5c06\u5355\u8bcd\u5207\u5206\u5f97\u66f4\u788e\u800c\u5bfc\u81f4\u6982\u7387\u8ba1\u7b97\u5403\u4e8f\u3002<\/li>\n

                        3.2 \u6570\u636e\u96c6\u51c6\u5907<\/h4>\n

                        3.2.1 \u81ea\u52a8\u4e0b\u8f7d（\u63a8\u8350）<\/h5>\n

                        # \u6570\u636e\u96c6\u81ea\u52a8\u4e0b\u8f7d\u811a\u672c
                        \nimport lm_eval.datasets<\/p>\n

                        # \u4e0b\u8f7dMMLU\u6570\u636e\u96c6（\u81ea\u52a8\u7f13\u5b58\u5230~\/.cache\/lm_eval\/）
                        \nmmlu_tasks = [
                        \n "mmlu_anatomy", "mmlu_astronomy", "mmlu_biology", "mmlu_business_ethics",
                        \n "mmlu_clinical_knowledge", "mmlu_college_biology", "mmlu_college_chemistry",
                        \n "mmlu_college_computer_science", "mmlu_college_mathematics", "mmlu_college_medicine",
                        \n "mmlu_college_physics", "mmlu_computer_security", "mmlu_econometrics", "mmlu_electrical_engineering",
                        \n "mmlu_elementary_mathematics", "mmlu_formal_logic", "mmlu_global_facts", "mmlu_high_school_biology",
                        \n "mmlu_high_school_chemistry", "mmlu_high_school_computer_science", "mmlu_high_school_european_history",
                        \n "mmlu_high_school_geography", "mmlu_high_school_government_and_politics", "mmlu_high_school_macroeconomics",
                        \n "mmlu_high_school_mathematics", "mmlu_high_school_microeconomics", "mmlu_high_school_physics",
                        \n "mmlu_high_school_psychology", "mmlu_high_school_statistics", "mmlu_high_school_us_history",
                        \n "mmlu_high_school_world_history", "mmlu_human_aging", "mmlu_human_sexuality", "mmlu_international_law",
                        \n "mmlu_jurisprudence", "mmlu_logical_fallacies", "mmlu_machine_learning", "mmlu_management",
                        \n "mmlu_marketing", "mmlu_medical_genetics", "mmlu_miscellaneous", "mmlu_moral_disputes",
                        \n "mmlu_moral_scenarios", "mmlu_nutrition", "mmlu_philosophy", "mmlu_prehistory",
                        \n "mmlu_professional_accounting", "mmlu_professional_law", "mmlu_professional_medicine",
                        \n "mmlu_professional_psychology", "mmlu_public_relations", "mmlu_religious_studies",
                        \n "mmlu_security_studies", "mmlu_sociology", "mmlu_us_foreign_policy", "mmlu_virology",
                        \n "mmlu_world_religions"
                        \n]<\/p>\n

                        # \u4e0b\u8f7dBBH\u6570\u636e\u96c6
                        \nbbh_tasks = [
                        \n "bbh_boolean_expressions", "bbh_causal_judgment", "bbh_date_understanding",
                        \n "bbh_disambiguation_qa", "bbh_formal_fallacies", "bbh_gender_pronoun_resolution",
                        \n "bbh_goal_step_writing", "bbh_judgmental_reasoning", "bbh_logical_deduction_five_objects",
                        \n "bbh_logical_deduction_seven_objects", "bbh_logical_deduction_three_objects",
                        \n "bbh_movie_recommendation", "bbh_multiple_choice", "bbh_navigation", "bbh_object_counting",
                        \n "bbh_penguins_in_a_table", "bbh_phrase_bias", "bbh_reasoning_about_colored_objects",
                        \n "bbh_ruin_names", "bbh_salient_translation_error_detection", "bbh_snarks",
                        \n "bbh_sports_understanding", "bbh_temporal_sequences", "bbh_trustworthiness",
                        \n "bbh_web_of_lies", "bbh_written_spanish"
                        \n]<\/p>\n

                        # \u9a8c\u8bc1\u6570\u636e\u96c6\u52a0\u8f7d
                        \nfor task in ["mmlu", "bbh"]:
                        \n try:
                        \n lm_eval.tasks.get_task(task)
                        \n print(f"{task} \u6570\u636e\u96c6\u52a0\u8f7d\u6210\u529f")
                        \n except Exception as e:
                        \n print(f"{task} \u6570\u636e\u96c6\u52a0\u8f7d\u5931\u8d25: {e}")<\/p>\n

                        3.2.2 \u624b\u52a8\u4e0b\u8f7d（\u5907\u7528）<\/h5>\n

                        \u82e5\u81ea\u52a8\u4e0b\u8f7d\u5931\u8d25，\u53ef\u4eceHugging Face Datasets\u624b\u52a8\u4e0b\u8f7d：<\/p>\n

                        # \u624b\u52a8\u4e0b\u8f7dMMLU\u6570\u636e\u96c6
                        \ngit clone https:\/\/huggingface.co\/datasets\/cais\/mmlu
                        \n# \u624b\u52a8\u4e0b\u8f7dBBH\u6570\u636e\u96c6
                        \ngit clone https:\/\/huggingface.co\/datasets\/suzgunmirac\/BigBenchHard<\/p>\n

                        3.3 \u5f00\u6e90\u6a21\u578b\u8bc4\u6d4b\u5b9e\u6218（\u4ee5Qwen2.5-7B-Instruct\u4e3a\u4f8b）<\/h4>\n

                        3.3.0 \u8bc4\u6d4b\u6d41\u6c34\u7ebf\u67b6\u6784\u56fe<\/h5>\n

                        \u5728\u8fd0\u884c\u4ee3\u7801\u4e4b\u524d，\u7406\u89e3\u6570\u636e\u5982\u4f55\u5728\u6a21\u578b（Model）\u3001**\u8bc4\u6d4b\u6846\u67b6（Harness）\u548c\u7ed3\u679c（Result）**\u4e4b\u95f4\u6d41\u52a8\u81f3\u5173\u91cd\u8981\u3002<\/p>\n

                        \u5927\u6a21\u578b\u8bc4\u6d4b\u5b9e\u6218\u6d41\u7a0b\u56fe (<\/span>Evaluation Pipeline)<\/span>
                        \n\u2502
                        \n\u251c\u2500\u2500 \u30101. \u914d\u7f6e\u4e0e\u51c6\u5907\u9636\u6bb5\u3011(<\/span>Configuration)<\/span>
                        \n\u2502 \u251c\u2500\u2500 \u7528\u6237\u6307\u4ee4: python eval.py –model_path<\/span> "Qwen2.5"<\/span> –tasks<\/span> "mmlu,bbh"<\/span>
                        \n\u2502 \u251c\u2500\u2500 \u73af\u5883\u53d8\u91cf: .env (<\/span>\u52a0\u8f7d API Key \u6216 \u7f13\u5b58\u8def\u5f84)<\/span>
                        \n\u2502 \u2514\u2500\u2500 \u4efb\u52a1\u6ce8\u518c\u8868: lm_eval.tasks (<\/span>\u81ea\u52a8\u62c9\u53d6 MMLU 57<\/span>\u4e2a\u5b66\u79d1\/BBH \u4efb\u52a1\u5b9a\u4e49)<\/span>
                        \n\u2502
                        \n\u25bc
                        \n[<\/span>2<\/span>. \u8bc4\u6d4b\u5f15\u64ce\u521d\u59cb\u5316]<\/span> (<\/span>Engine Initialization)<\/span> <<\/span>\u6838\u5fc3\u5206\u6b67\u70b9><\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                        \n\u2502 \u2502
                        \n\u251c\u2500\u2500 \u8def\u5f84 A: \u901a\u7528\u517c\u5bb9\u6a21\u5f0f (<\/span>\u5bf9\u5e94 3.3<\/span>.1 \u811a\u672c)<\/span> \u2502
                        \n\u2502 \u251c\u2500\u2500 <<\/span>\u6838\u5fc3\u7c7b><\/span>: HFLM (<\/span>lm-evaluation-harness \u5185\u7f6e\u5c01\u88c5)<\/span> \u2502
                        \n\u2502 \u251c\u2500\u2500 <<\/span>\u5e95\u5c42\u8c03\u7528><\/span>: AutoModelForCausalLM (<\/span>Transformers)<\/span> \u2502
                        \n\u2502 \u251c\u2500\u2500 <<\/span>\u4f18\u52bf><\/span>: \u517c\u5bb9\u6027\u6700\u5f3a，\u652f\u6301 8bit\/4bit \u91cf\u5316，\u9002\u914d\u6240\u6709 HF \u6a21\u578b \u2502
                        \n\u2502 \u2514\u2500\u2500 <<\/span>\u9002\u7528><\/span>: \u8d44\u6e90\u53d7\u9650\u73af\u5883\u3001\u65b0\u6a21\u578b\u67b6\u6784\u8c03\u8bd5 \u2502
                        \n\u2502 \u2502
                        \n\u251c\u2500\u2500 \u8def\u5f84 B: \u9ad8\u6027\u80fd\u52a0\u901f\u6a21\u5f0f (<\/span>\u5bf9\u5e94 3.3<\/span>.2 \u811a\u672c)<\/span> \u2502
                        \n\u2502 \u251c\u2500\u2500 <<\/span>\u6838\u5fc3\u7c7b><\/span>: vLLM Engine (<\/span>PageAttention \u6280\u672f)<\/span> \u2502
                        \n\u2502 \u251c\u2500\u2500 <<\/span>\u5e95\u5c42\u8c03\u7528><\/span>: CUDA Kernel \u4f18\u5316\u63a8\u7406 \u2502
                        \n\u2502 \u251c\u2500\u2500 <<\/span>\u4f18\u52bf><\/span>: \u541e\u5410\u91cf\u63d0\u5347 5<\/span>-10 \u500d，\u663e\u5b58\u5229\u7528\u7387\u6781\u5927\u4f18\u5316 \u2502
                        \n\u2502 \u2514\u2500\u2500 <<\/span>\u9002\u7528><\/span>: 7B\/14B+ \u6a21\u578b\u5168\u91cf\u8bc4\u6d4b，\u751f\u4ea7\u73af\u5883\u6279\u91cf\u6d4b\u8bd5 \u2502
                        \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                        \n \u2502
                        \n \u25bc
                        \n[<\/span>3<\/span>. \u6570\u636e\u6d41\u4e0e\u63a8\u7406\u6267\u884c]<\/span> (<\/span>Execution Loop)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                        \n\u2502 \u2502
                        \n\u251c\u2500\u2500 \u6b65\u9aa4 \u2460: \u63d0\u793a\u8bcd\u6784\u5efa (<\/span>Prompt Construction)<\/span> \u2502
                        \n\u2502 \u251c\u2500\u2500 Few-shot \u91c7\u6837: \u4ece Dev \u96c6\u968f\u673a\u62bd\u53d6 5<\/span> \u4e2a\u6837\u672c (<\/span>In-Context)<\/span> \u2502
                        \n\u2502 \u2514\u2500\u2500 \u683c\u5f0f\u5316: "Question: … Answer:"<\/span> (<\/span>\u6807\u51c6\u5316\u6a21\u7248，\u907f\u514d\u504f\u5dee)<\/span> \u2502
                        \n\u2502 \u2502
                        \n\u251c\u2500\u2500 \u6b65\u9aa4 \u2461: \u6a21\u578b\u63a8\u7406 (<\/span>Inference &<\/span> Scoring)<\/span> \u2502
                        \n\u2502 \u251c\u2500\u2500 \u5224\u522b\u5f0f (<\/span>MMLU)<\/span>: \u8ba1\u7b97\u9009\u9879 A\/B\/C\/D \u7684 Logits (<\/span>\u6982\u7387)<\/span> \u2502
                        \n\u2502 \u2502 ><\/span> \u4e0d\u751f\u6210\u6587\u672c，\u76f4\u63a5\u6bd4\u8f83 P(<\/span>A)<\/span> vs P(<\/span>B)<\/span>..<\/span>. \u66f4\u7a33\u5b9a \u2502
                        \n\u2502 \u2514\u2500\u2500 \u751f\u6210\u5f0f (<\/span>BBH)<\/span>: Generate(<\/span>)<\/span> –><\/span> \u6b63\u5219\u63d0\u53d6 –><\/span> Exact Match \u2502
                        \n\u2502 ><\/span> \u8ba9\u6a21\u578b\u751f\u6210\u63a8\u7406\u8fc7\u7a0b (<\/span>CoT)<\/span>，\u6d4b\u8bd5\u903b\u8f91\u601d\u7ef4\u80fd\u529b \u2502
                        \n\u2502 \u2502
                        \n\u2514\u2500\u2500 ><\/span> \u8f93\u51fa: \u539f\u59cb\u7ed3\u679c\u65e5\u5fd7 (<\/span>eval_results_raw.json)<\/span> \u2502
                        \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

                        3.3.1 \u57fa\u7840\u8bc4\u6d4b\u811a\u672c（CPU\/GPU\u901a\u7528）<\/h5>\n

                        """
                        \nMMLU\/BBH\u8bc4\u6d4b\u6838\u5fc3\u811a\u672c
                        \n\u652f\u6301Qwen2.5\/Llama3\u7b49\u5f00\u6e90\u6a21\u578b
                        \n"""
                        \nimport os
                        \nimport json
                        \nimport argparse
                        \nimport torch
                        \nfrom dotenv import load_dotenv
                        \nimport lm_eval
                        \nfrom lm_eval import evaluator, tasks
                        \nfrom lm_eval.models.huggingface import HFLM<\/p>\n

                        # \u52a0\u8f7d\u73af\u5883\u53d8\u91cf
                        \nload_dotenv()<\/p>\n

                        # \u914d\u7f6e\u53c2\u6570
                        \nparser = argparse.ArgumentParser(description="\u5927\u6a21\u578bMMLU\/BBH\u8bc4\u6d4b\u811a\u672c")
                        \nparser.add_argument("–model_path", type=str, default="\/path\/to\/Qwen2.5-7B-Instruct", help="\u672c\u5730\u6a21\u578b\u8def\u5f84")
                        \nparser.add_argument("–tasks", type=str, default="mmlu,bbh", help="\u8bc4\u6d4b\u4efb\u52a1（mmlu\/bbh\/all）")
                        \nparser.add_argument("–batch_size", type=int, default=8, help="\u63a8\u7406\u6279\u6b21\u5927\u5c0f")
                        \nparser.add_argument("–output_dir", type=str, default=".\/eval_results", help="\u8bc4\u6d4b\u7ed3\u679c\u8f93\u51fa\u76ee\u5f55")
                        \nparser.add_argument("–device", type=str, default="cuda" if torch.cuda.is_available() else "cpu", help="\u8fd0\u884c\u8bbe\u5907")
                        \nparser.add_argument("–max_samples", type=int, default=None, help="\u6bcf\u4e2a\u4efb\u52a1\u8bc4\u6d4b\u6837\u672c\u6570（None\u8868\u793a\u5168\u90e8）")<\/p>\n

                        args = parser.parse_args()<\/p>\n

                        # \u521b\u5efa\u8f93\u51fa\u76ee\u5f55
                        \nos.makedirs(args.output_dir, exist_ok=True)<\/p>\n

                        # 1. \u521d\u59cb\u5316\u6a21\u578b
                        \ndef init_model(model_path, device):
                        \n """\u521d\u59cb\u5316HuggingFace\u6a21\u578b"""
                        \n model = HFLM(
                        \n pretrained=model_path,
                        \n device=device,
                        \n batch_size=args.batch_size,
                        \n load_in_8bit=False, # \u4f4e\u663e\u5b58\u53ef\u5f00\u542f8bit\u91cf\u5316
                        \n load_in_4bit=False, # \u6781\u4f4e\u663e\u5b58\u53ef\u5f00\u542f4bit\u91cf\u5316
                        \n trust_remote_code=True, # \u52a0\u8f7d\u81ea\u5b9a\u4e49\u6a21\u578b\u4ee3\u7801（Qwen2.5\u9700\u8981）
                        \n torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
                        \n max_length=2048, # \u6700\u5927\u5e8f\u5217\u957f\u5ea6
                        \n )
                        \n return model<\/p>\n

                        # 2. \u9009\u62e9\u8bc4\u6d4b\u4efb\u52a1
                        \ndef get_selected_tasks(task_str):
                        \n """\u6839\u636e\u8f93\u5165\u9009\u62e9\u8bc4\u6d4b\u4efb\u52a1"""
                        \n all_tasks = []
                        \n if "mmlu" in task_str.lower():
                        \n # \u83b7\u53d6\u6240\u6709MMLU\u5b50\u4efb\u52a1
                        \n mmlu_tasks = [t for t in tasks.ALL_TASKS if t.startswith("mmlu_")]
                        \n all_tasks.extend(mmlu_tasks)
                        \n if "bbh" in task_str.lower():
                        \n # \u83b7\u53d6\u6240\u6709BBH\u5b50\u4efb\u52a1
                        \n bbh_tasks = [t for t in tasks.ALL_TASKS if t.startswith("bbh_")]
                        \n all_tasks.extend(bbh_tasks)
                        \n if "all" in task_str.lower():
                        \n all_tasks = tasks.ALL_TASKS<\/p>\n

                        return all_tasks<\/p>\n

                        # 3. \u6267\u884c\u8bc4\u6d4b
                        \ndef run_evaluation():
                        \n # \u521d\u59cb\u5316\u6a21\u578b
                        \n print(f"\u521d\u59cb\u5316\u6a21\u578b: {args.model_path}")
                        \n model = init_model(args.model_path, args.device)<\/p>\n

                        # \u83b7\u53d6\u8bc4\u6d4b\u4efb\u52a1
                        \n selected_tasks = get_selected_tasks(args.tasks)
                        \n print(f"\u9009\u5b9a\u8bc4\u6d4b\u4efb\u52a1\u6570: {len(selected_tasks)}")
                        \n print(f"\u4efb\u52a1\u5217\u8868: {selected_tasks[:5]}…") # \u6253\u5370\u524d5\u4e2a\u4efb\u52a1<\/p>\n

                        # \u6267\u884c\u8bc4\u6d4b
                        \n print("\u5f00\u59cb\u8bc4\u6d4b…")
                        \n results = evaluator.simple_evaluate(
                        \n model=model,
                        \n tasks=selected_tasks,
                        \n num_fewshot=5, # MMLU\/BBH\u6807\u51c6\u8bc4\u6d4b\u4f7f\u75285-shot
                        \n batch_size=args.batch_size,
                        \n max_samples=args.max_samples,
                        \n device=args.device,
                        \n verbose=True,
                        \n )<\/p>\n

                        # \u4fdd\u5b58\u539f\u59cb\u7ed3\u679c
                        \n results_path = os.path.join(args.output_dir, "eval_results_raw.json")
                        \n with open(results_path, "w", encoding="utf-8") as f:
                        \n json.dump(results, f, ensure_ascii=False, indent=4)
                        \n print(f"\u539f\u59cb\u8bc4\u6d4b\u7ed3\u679c\u5df2\u4fdd\u5b58\u81f3: {results_path}")<\/p>\n

                        # \u751f\u6210\u6c47\u603b\u62a5\u544a
                        \n generate_summary_report(results)<\/p>\n

                        return results<\/p>\n

                        # 4. \u751f\u6210\u6c47\u603b\u62a5\u544a
                        \ndef generate_summary_report(results):
                        \n """\u751f\u6210\u6613\u8bfb\u7684\u8bc4\u6d4b\u6c47\u603b\u62a5\u544a"""
                        \n summary = {
                        \n "model": args.model_path.split("\/")[-1],
                        \n "total_tasks": len(results["results"]),
                        \n "mmlu_average": None,
                        \n "bbh_average": None,
                        \n "task_details": {}
                        \n }<\/p>\n

                        # \u8ba1\u7b97MMLU\u548cBBH\u5e73\u5747\u5206
                        \n mmlu_scores = []
                        \n bbh_scores = []
                        \n for task, metrics in results["results"].items():
                        \n # \u83b7\u53d6\u51c6\u786e\u7387（MMLU\/BBH\u4e3b\u8981\u6307\u6807）
                        \n acc = metrics.get("acc", metrics.get("exact_match", 0))
                        \n summary["task_details"][task] = {
                        \n "accuracy": round(acc * 100, 2),
                        \n "samples": metrics.get("n", 0)
                        \n }<\/p>\n

                        if task.startswith("mmlu_"):
                        \n mmlu_scores.append(acc)
                        \n elif task.startswith("bbh_"):
                        \n bbh_scores.append(acc)<\/p>\n

                        # \u8ba1\u7b97\u5e73\u5747\u5206
                        \n if mmlu_scores:
                        \n summary["mmlu_average"] = round(sum(mmlu_scores) \/ len(mmlu_scores) * 100, 2)
                        \n if bbh_scores:
                        \n summary["bbh_average"] = round(sum(bbh_scores) \/ len(bbh_scores) * 100, 2)<\/p>\n

                        # \u4fdd\u5b58\u6c47\u603b\u62a5\u544a
                        \n summary_path = os.path.join(args.output_dir, "eval_summary.json")
                        \n with open(summary_path, "w", encoding="utf-8") as f:
                        \n json.dump(summary, f, ensure_ascii=False, indent=4)
                        \n print(f"\u8bc4\u6d4b\u6c47\u603b\u62a5\u544a\u5df2\u4fdd\u5b58\u81f3: {summary_path}")<\/p>\n

                        # \u6253\u5370\u6c47\u603b\u7ed3\u679c
                        \n print("\\\\n=== \u8bc4\u6d4b\u6c47\u603b ===")
                        \n print(f"\u6a21\u578b: {summary['model']}")
                        \n print(f"\u8bc4\u6d4b\u4efb\u52a1\u603b\u6570: {summary['total_tasks']}")
                        \n print(f"MMLU\u5e73\u5747\u51c6\u786e\u7387: {summary['mmlu_average']}%")
                        \n print(f"BBH\u5e73\u5747\u51c6\u786e\u7387: {summary['bbh_average']}%")<\/p>\n

                        # \u6253\u5370\u8868\u73b0\u6700\u597d\/\u6700\u5dee\u7684\u4efb\u52a1
                        \n sorted_tasks = sorted(summary["task_details"].items(), key=lambda x: x[1]["accuracy"], reverse=True)
                        \n print("\\\\n\u8868\u73b0\u6700\u597d\u76845\u4e2a\u4efb\u52a1:")
                        \n for task, metrics in sorted_tasks[:5]:
                        \n print(f" {task}: {metrics['accuracy']}%")<\/p>\n

                        print("\\\\n\u8868\u73b0\u6700\u5dee\u76845\u4e2a\u4efb\u52a1:")
                        \n for task, metrics in sorted_tasks[-5:]:
                        \n print(f" {task}: {metrics['accuracy']}%")<\/p>\n

                        # \u4e3b\u6267\u884c
                        \nif __name__ == "__main__":
                        \n run_evaluation()<\/p>\n

                        \u5728\u4e0a\u8ff0\u4ee3\u7801\u4e2d，\u5404\u7ec4\u4ef6\u534f\u4f5c\u903b\u8f91\u5982\u4e0b：<\/p>\n

                      • \u4efb\u52a1\u5206\u53d1\u5668 (get_selected_tasks)：\n
                          \n
                        • \u626e\u6f14\u201c\u56fe\u4e66\u7ba1\u7406\u5458\u201d\u89d2\u8272\u3002\u5b83\u4ece lm_eval.tasks \u5de8\u5927\u7684\u4efb\u52a1\u5e93\u4e2d，\u6839\u636e\u7528\u6237\u6307\u4ee4（\u5982 \u201cmmlu\u201d）\u7cbe\u51c6\u7b5b\u9009\u51fa\u5bf9\u5e94\u7684 57 \u4e2a\u5b50\u4efb\u52a1 ID\u3002\u5982\u679c\u4e0d\u8fdb\u884c\u7b5b\u9009，\u6846\u67b6\u9ed8\u8ba4\u4f1a\u52a0\u8f7d\u6570\u5343\u4e2a\u4efb\u52a1，\u5bfc\u81f4\u5185\u5b58\u6ea2\u51fa\u3002<\/li>\n<\/ul>\n<\/li>\n
                        • \u6a21\u578b\u9002\u914d\u5668 (HFLM vs vLLM)：\n
                            \n
                          • HFLM：\u662f lm-evaluation-harness \u63d0\u4f9b\u7684\u6807\u51c6\u63a5\u53e3，\u5b83\u628a HuggingFace \u6a21\u578b\u5305\u88c5\u6210\u8bc4\u6d4b\u6846\u67b6\u80fd\u542c\u61c2\u7684\u7edf\u4e00\u5bf9\u8c61\u3002\u5b83\u81ea\u52a8\u5904\u7406\u4e86 padding\u3001truncation \u548c device placement\u3002<\/li>\n
                          • vLLM：\u662f\u201c\u5916\u6302\u5f15\u64ce\u201d\u3002\u5728 3.3.2 \u811a\u672c\u4e2d，\u6211\u4eec\u8df3\u8fc7\u4e86 harness \u7684\u6807\u51c6\u63a8\u7406\u5faa\u73af，\u76f4\u63a5\u7528 vLLM \u8fdb\u884c\u6781\u901f\u63a8\u7406，\u7136\u540e\u624b\u52a8\u8ba1\u7b97\u51c6\u786e\u7387\u3002\u8fd9\u662f\u4e00\u79cd**\u201c\u7528\u7a7a\u95f4\u6362\u65f6\u95f4\u201d**\u7684\u7b56\u7565，\u901a\u8fc7\u624b\u52a8\u5199\u4ee3\u7801\u6362\u53d6 10 \u500d\u7684\u901f\u5ea6\u63d0\u5347\u3002<\/li>\n<\/ul>\n<\/li>\n
                          • Few-shot \u6784\u5efa\u673a\u5236：\n
                              \n
                            • \u4ee3\u7801\u4e2d\u7684 num_fewshot=5 \u662f\u8bc4\u6d4b\u516c\u5e73\u6027\u7684\u5173\u952e\u3002\u5b83\u5f3a\u5236\u6a21\u578b\u5728\u56de\u7b54\u5f53\u524d\u95ee\u9898\u524d，\u5148\u201c\u770b\u201d5\u4e2a\u5e26\u7b54\u6848\u7684\u4f8b\u5b50\u3002\u8fd9\u6a21\u62df\u4e86\u6a21\u578b\u5728\u62e5\u6709\u77ed\u671f\u8bb0\u5fc6（Context）\u4e0b\u7684\u8868\u73b0，\u80fd\u591f\u663e\u8457\u63d0\u5347 MMLU \u5206\u6570，\u662f\u5b66\u672f\u754c\u5bf9\u6bd4\u6a21\u578b\u7684\u6807\u51c6\u8bbe\u5b9a\u3002<\/li>\n<\/ul>\n<\/li>\n
                              3.3.2 \u9ad8\u6027\u80fd\u8bc4\u6d4b（vLLM\u52a0\u901f）<\/h5>\n

                              \u5bf9\u4e8e\u5927\u6a21\u578b（7B\/14B），\u63a8\u8350\u4f7f\u7528vLLM\u52a0\u901f\u8bc4\u6d4b（\u541e\u5410\u91cf\u63d0\u53475-10\u500d）：<\/p>\n

                              """
                              \n\u57fa\u4e8evLLM\u7684\u9ad8\u6027\u80fdMMLU\/BBH\u8bc4\u6d4b\u811a\u672c
                              \n"""
                              \nimport os
                              \nimport json
                              \nimport torch
                              \nfrom vllm import LLM, SamplingParams
                              \nimport lm_eval
                              \nfrom lm_eval import tasks
                              \nfrom tqdm import tqdm<\/p>\n

                              # \u914d\u7f6e
                              \nMODEL_PATH = "\/path\/to\/Qwen2.5-7B-Instruct"
                              \nOUTPUT_DIR = ".\/eval_results_vllm"
                              \nBATCH_SIZE = 32 # vLLM\u652f\u6301\u66f4\u5927\u6279\u6b21
                              \nNUM_FEWSHOT = 5
                              \nDEVICE = "cuda" if torch.cuda.is_available() else "cpu"<\/p>\n

                              # \u521b\u5efa\u8f93\u51fa\u76ee\u5f55
                              \nos.makedirs(OUTPUT_DIR, exist_ok=True)<\/p>\n

                              # 1. \u521d\u59cb\u5316vLLM\u6a21\u578b
                              \nprint(f"\u52a0\u8f7dvLLM\u6a21\u578b: {MODEL_PATH}")
                              \nllm = LLM(
                              \n model=MODEL_PATH,
                              \n tensor_parallel_size=torch.cuda.device_count(), # \u591a\u5361\u5e76\u884c
                              \n gpu_memory_utilization=0.9, # \u663e\u5b58\u5229\u7528\u7387
                              \n max_num_batched_tokens=8192,
                              \n trust_remote_code=True,
                              \n dtype="bfloat16"
                              \n)<\/p>\n

                              # \u91c7\u6837\u53c2\u6570（\u8bc4\u6d4b\u7528\u786e\u5b9a\u6027\u91c7\u6837）
                              \nsampling_params = SamplingParams(
                              \n temperature=0.0,
                              \n max_tokens=10,
                              \n top_p=1.0,
                              \n stop=["\\\\n", "###"]
                              \n)<\/p>\n

                              # 2. \u52a0\u8f7dMMLU\u4efb\u52a1
                              \ntask = lm_eval.tasks.get_task("mmlu")
                              \ntask.build_all()<\/p>\n

                              # 3. \u6570\u636e\u9884\u5904\u7406
                              \ndef prepare_fewshot_examples(dataset, num_fewshot=5):
                              \n """\u51c6\u59075-shot\u793a\u4f8b"""
                              \n fewshot_examples = []
                              \n for i in range(num_fewshot):
                              \n ex = dataset[i]
                              \n fewshot_examples.append(
                              \n f"Question: {ex['question']}\\\\n"
                              \n f"A. {ex['choices'][0]}\\\\n"
                              \n f"B. {ex['choices'][1]}\\\\n"
                              \n f"C. {ex['choices'][2]}\\\\n"
                              \n f"D. {ex['choices'][3]}\\\\n"
                              \n f"Answer: {ex['answer']}\\\\n"
                              \n )
                              \n return "\\\\n".join(fewshot_examples)<\/p>\n

                              # 4. \u6267\u884c\u8bc4\u6d4b
                              \ndef run_vllm_evaluation():
                              \n # \u83b7\u53d6MMLU\u6d4b\u8bd5\u96c6
                              \n test_set = task.dataset["test"]
                              \n results = {"correct": 0, "total": 0, "task_details": []}<\/p>\n

                              # \u51c6\u5907few-shot\u793a\u4f8b
                              \n fewshot_text = prepare_fewshot_examples(task.dataset["validation"], NUM_FEWSHOT)<\/p>\n

                              # \u6279\u91cf\u63a8\u7406
                              \n batches = [test_set[i:i+BATCH_SIZE] for i in range(0, len(test_set), BATCH_SIZE)]<\/p>\n

                              for batch in tqdm(batches, desc="\u8bc4\u6d4b\u8fdb\u5ea6"):
                              \n # \u6784\u5efa\u63d0\u793a\u8bcd
                              \n prompts = []
                              \n for ex in batch:
                              \n prompt = (
                              \n f"{fewshot_text}\\\\n"
                              \n f"Question: {ex['question']}\\\\n"
                              \n f"A. {ex['choices'][0]}\\\\n"
                              \n f"B. {ex['choices'][1]}\\\\n"
                              \n f"C. {ex['choices'][2]}\\\\n"
                              \n f"D. {ex['choices'][3]}\\\\n"
                              \n f"Answer:"
                              \n )
                              \n prompts.append(prompt)<\/p>\n

                              # vLLM\u6279\u91cf\u63a8\u7406
                              \n outputs = llm.generate(prompts, sampling_params)<\/p>\n

                              # \u89e3\u6790\u7ed3\u679c
                              \n for ex, output in zip(batch, outputs):
                              \n pred = output.outputs[0].text.strip().upper()
                              \n true_answer = ex['answer'].upper()
                              \n is_correct = pred == true_answer<\/p>\n

                              results["total"] += 1
                              \n if is_correct:
                              \n results["correct"] += 1<\/p>\n

                              results["task_details"].append({
                              \n "question": ex["question"],
                              \n "choices": ex["choices"],
                              \n "true_answer": true_answer,
                              \n "pred_answer": pred,
                              \n "is_correct": is_correct
                              \n })<\/p>\n

                              # \u8ba1\u7b97\u51c6\u786e\u7387
                              \n results["accuracy"] = round(results["correct"] \/ results["total"] * 100, 2)<\/p>\n

                              # \u4fdd\u5b58\u7ed3\u679c
                              \n with open(os.path.join(OUTPUT_DIR, "vllm_mmlu_results.json"), "w", encoding="utf-8") as f:
                              \n json.dump(results, f, ensure_ascii=False, indent=4)<\/p>\n

                              # \u6253\u5370\u7ed3\u679c
                              \n print(f"\\\\n=== vLLM\u8bc4\u6d4b\u7ed3\u679c ===")
                              \n print(f"\u603b\u6837\u672c\u6570: {results['total']}")
                              \n print(f"\u6b63\u786e\u6570: {results['correct']}")
                              \n print(f"\u51c6\u786e\u7387: {results['accuracy']}%")<\/p>\n

                              if __name__ == "__main__":
                              \n run_vllm_evaluation()<\/p>\n

                              3.4 \u8bc4\u6d4b\u7ed3\u679c\u5206\u6790<\/h4>\n

                              3.4.0 \u7ed3\u679c\u5904\u7406\u6d41\u7a0b\u56fe<\/h5>\n

                              \u8bc4\u6d4b\u4e0d\u4ec5\u662f\u8dd1\u5206，\u66f4\u662f\u4e3a\u4e86\u53d1\u73b0\u6a21\u578b\u77ed\u677f\u3002\u53ef\u89c6\u5316\u811a\u672c\u7684\u5de5\u4f5c\u6d41\u5982\u4e0b：<\/p>\n

                              \u7ed3\u679c\u5206\u6790\u6d41\u6c34\u7ebf (<\/span>Analysis Pipeline)<\/span>
                              \n\u2502
                              \n\u251c\u2500\u2500 [<\/span>1<\/span>. \u6570\u636e\u52a0\u8f7d\u4e0e\u6e05\u6d17]<\/span> (<\/span>Data Loading)<\/span>
                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u8f93\u5165><\/span>: eval_summary.json (<\/span>\u7531 3.3<\/span> \u6b65\u9aa4\u751f\u6210)<\/span>
                              \n\u2502 \u2514\u2500\u2500 <<\/span>\u89e3\u6790><\/span>: \u63d0\u53d6 metrics[<\/span>"acc"<\/span>]<\/span> \u6216 metrics[<\/span>"exact_match"<\/span>]<\/span>
                              \n\u2502
                              \n\u25bc
                              \n[<\/span>2<\/span>. \u7ef4\u5ea6\u91cd\u7ec4 (<\/span>Re-grouping)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                              \n\u2502 (<\/span>\u5c06\u6241\u5e73\u7684\u4efb\u52a1\u5217\u8868\u91cd\u7ec4\u4e3a\u4eba\u7c7b\u53ef\u8bfb\u7684\u77e5\u8bc6\u4f53\u7cfb)<\/span> \u2502
                              \n\u2502 \u2502
                              \n\u251c\u2500\u2500 \u5206\u652f A: MMLU \u77e5\u8bc6\u56fe\u8c31 \u2502
                              \n\u2502 \u251c\u2500\u2500 STEM (<\/span>\u7406\u79d1)<\/span>: Math, Physics, Chemistry..<\/span>. \u2502
                              \n\u2502 \u251c\u2500\u2500 Humanities (<\/span>\u6587\u79d1)<\/span>: History, Law, Philosophy..<\/span>. \u2502
                              \n\u2502 \u2514\u2500\u2500 Social Sciences (<\/span>\u793e\u79d1)<\/span>: Economics, Psychology..<\/span>. \u2502
                              \n\u2502 \u2502
                              \n\u251c\u2500\u2500 \u5206\u652f B: BBH \u80fd\u529b\u56fe\u8c31 \u2502
                              \n\u2502 \u251c\u2500\u2500 Logical (<\/span>\u903b\u8f91)<\/span>: Boolean expressions, Syllogisms..<\/span>. \u2502
                              \n\u2502 \u251c\u2500\u2500 Algorithmic (<\/span>\u7b97\u6cd5)<\/span>: Object counting, Sorting..<\/span>. \u2502
                              \n\u2502 \u2514\u2500\u2500 Language (<\/span>\u8bed\u8a00)<\/span>: Disambiguation, Translation..<\/span>. \u2502
                              \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                              \n \u2502
                              \n \u25bc
                              \n[<\/span>3<\/span>. \u51b3\u7b56\u8f85\u52a9\u8f93\u51fa]<\/span> (<\/span>Decision Support Output)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                              \n\u2502 \u2502
                              \n\u251c\u2500\u2500 \u53ef\u89c6\u5316 (<\/span>Visualization)<\/span>: \u2502
                              \n\u2502 \u251c\u2500\u2500 Matplotlib \u7ed8\u5236\u6c34\u5e73\u6761\u5f62\u56fe (<\/span>Bar Chart)<\/span> \u2502
                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u8272\u9636\u6620\u5c04><\/span>: \u7ea2\u8272(<\/span>\u4f4e\u5206)<\/span> –><\/span> \u9ec4\u8272(<\/span>\u53ca\u683c)<\/span> –><\/span> \u7eff\u8272(<\/span>\u4f18\u79c0)<\/span> \u2502
                              \n\u2502 \u2514\u2500\u2500 ><\/span> \u4f5c\u7528: \u4e00\u773c\u8bc6\u522b\u201c\u504f\u79d1\u201d\u73b0\u8c61 \u2502
                              \n\u2502 \u2502
                              \n\u251c\u2500\u2500 \u77ed\u677f\u8bca\u65ad (<\/span>Weakness Diagnosis)<\/span>: \u2502
                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u7b97\u6cd5><\/span>: \u7b5b\u9009 Accuracy <<\/span> (<\/span>Average – 10<\/span>%)<\/span> \u7684\u4efb\u52a1 \u2502
                              \n\u2502 \u2514\u2500\u2500 ><\/span> \u4f5c\u7528: \u6307\u5bfc\u540e\u7eed\u5fae\u8c03 (<\/span>SFT)<\/span> \u7684\u6570\u636e\u914d\u6bd4\u65b9\u5411 \u2502
                              \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

                              3.4.1 \u7ed3\u679c\u53ef\u89c6\u5316<\/h5>\n

                              """
                              \n\u8bc4\u6d4b\u7ed3\u679c\u53ef\u89c6\u5316\u811a\u672c
                              \n"""
                              \nimport json
                              \nimport matplotlib.pyplot as plt
                              \nimport numpy as np
                              \nfrom collections import defaultdict<\/p>\n

                              # \u8bbe\u7f6e\u4e2d\u6587\u5b57\u4f53
                              \nplt.rcParams["font.family"] = ["SimHei", "WenQuanYi Micro Hei", "Heiti TC"]
                              \nplt.rcParams["axes.unicode_minus"] = False<\/p>\n

                              # \u52a0\u8f7d\u8bc4\u6d4b\u7ed3\u679c
                              \nwith open(".\/eval_results\/eval_summary.json", "r", encoding="utf-8") as f:
                              \n summary = json.load(f)<\/p>\n

                              # 1. MMLU\/BBH\u5206\u7c7b\u7edf\u8ba1
                              \ntask_categories = defaultdict(list)
                              \nfor task, metrics in summary["task_details"].items():
                              \n if task.startswith("mmlu_"):
                              \n # \u63d0\u53d6MMLU\u5b66\u79d1\u5206\u7c7b
                              \n category = task.replace("mmlu_", "").replace("_", " ").title()
                              \n task_categories["MMLU"].append((category, metrics["accuracy"]))
                              \n elif task.startswith("bbh_"):
                              \n # \u63d0\u53d6BBH\u4efb\u52a1\u5206\u7c7b
                              \n category = task.replace("bbh_", "").replace("_", " ").title()
                              \n task_categories["BBH"].append((category, metrics["accuracy"]))<\/p>\n

                              # 2. \u7ed8\u5236MMLU\u5b66\u79d1\u51c6\u786e\u7387\u5206\u5e03\u56fe
                              \nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))<\/p>\n

                              # MMLU\u5b50\u56fe
                              \nmmlu_data = task_categories["MMLU"]
                              \nmmlu_categories = [x[0] for x in mmlu_data]
                              \nmmlu_scores = [x[1] for x in mmlu_data]<\/p>\n

                              # \u6309\u5206\u6570\u6392\u5e8f
                              \nsorted_idx = np.argsort(mmlu_scores)
                              \nmmlu_categories = [mmlu_categories[i] for i in sorted_idx]
                              \nmmlu_scores = [mmlu_scores[i] for i in sorted_idx]<\/p>\n

                              # \u7ed8\u5236\u6c34\u5e73\u6761\u5f62\u56fe
                              \ncolors = plt.cm.RdYlGn(np.array(mmlu_scores) \/ 100)
                              \nax1.barh(mmlu_categories, mmlu_scores, color=colors)
                              \nax1.set_xlabel("\u51c6\u786e\u7387 (%)")
                              \nax1.set_title(f"MMLU\u5404\u5b66\u79d1\u51c6\u786e\u7387 (\u5e73\u5747: {summary['mmlu_average']}%)", fontsize=14)
                              \nax1.axvline(x=summary['mmlu_average'], color='red', linestyle='–', label=f'\u5e73\u5747\u5206: {summary["mmlu_average"]}%')
                              \nax1.legend()<\/p>\n

                              # BBH\u5b50\u56fe
                              \nbbh_data = task_categories["BBH"]
                              \nbbh_categories = [x[0] for x in bbh_data]
                              \nbbh_scores = [x[1] for x in bbh_data]<\/p>\n

                              # \u6309\u5206\u6570\u6392\u5e8f
                              \nsorted_idx = np.argsort(bbh_scores)
                              \nbbh_categories = [bbh_categories[i] for i in sorted_idx]
                              \nbbh_scores = [bbh_scores[i] for i in sorted_idx]<\/p>\n

                              # \u7ed8\u5236\u6c34\u5e73\u6761\u5f62\u56fe
                              \ncolors = plt.cm.RdYlGn(np.array(bbh_scores) \/ 100)
                              \nax2.barh(bbh_categories, bbh_scores, color=colors)
                              \nax2.set_xlabel("\u51c6\u786e\u7387 (%)")
                              \nax2.set_title(f"BBH\u5404\u4efb\u52a1\u51c6\u786e\u7387 (\u5e73\u5747: {summary['bbh_average']}%)", fontsize=14)
                              \nax2.axvline(x=summary['bbh_average'], color='red', linestyle='–', label=f'\u5e73\u5747\u5206: {summary["bbh_average"]}%')
                              \nax2.legend()<\/p>\n

                              # \u8c03\u6574\u5e03\u5c40
                              \nplt.tight_layout()
                              \nplt.savefig(os.path.join(".\/eval_results", "eval_visualization.png"), dpi=300, bbox_inches="tight")
                              \nplt.show()<\/p>\n

                              # 3. \u751f\u6210\u80fd\u529b\u77ed\u677f\u5206\u6790
                              \nprint("\\\\n=== \u6a21\u578b\u80fd\u529b\u77ed\u677f\u5206\u6790 ===")
                              \nprint(f"MMLU\u8584\u5f31\u5b66\u79d1（<{summary['mmlu_average']-10}%）:")
                              \nfor task, metrics in summary["task_details"].items():
                              \n if task.startswith("mmlu_") and metrics["accuracy"] < summary["mmlu_average"] – 10:
                              \n print(f" {task.replace('mmlu_', '')}: {metrics['accuracy']}%")<\/p>\n

                              print(f"\\\\nBBH\u8584\u5f31\u4efb\u52a1（<{summary['bbh_average']-10}%）:")
                              \nfor task, metrics in summary["task_details"].items():
                              \n if task.startswith("bbh_") and metrics["accuracy"] < summary["bbh_average"] – 10:
                              \n print(f" {task.replace('bbh_', '')}: {metrics['accuracy']}%")<\/p>\n

                              \u56db\u3001\u6a21\u578b\u5e7b\u89c9（Hallucination）\u5b9a\u91cf\u8bc4\u4f30<\/h3>\n

                              4.1 \u5e7b\u89c9\u8bc4\u4f30\u6838\u5fc3\u6307\u6807<\/h4>\n\n\u6307\u6807\u8ba1\u7b97\u65b9\u5f0f\u542b\u4e49\u53d6\u503c\u8303\u56f4\u4f18\u52a3\u52bf<\/tr>\n\n\n\n\n\n\n
                              FactScore<\/td>\n\u751f\u6210\u5185\u5bb9\u4e2d\u4e8b\u5b9e\u6b63\u786e\u7684\u6bd4\u4f8b<\/td>\n\u4e8b\u5b9e\u51c6\u786e\u6027<\/td>\n0-1<\/td>\n\u6700\u5e38\u7528，\u8ba1\u7b97\u6210\u672c\u4e2d\u7b49<\/td>\n<\/tr>\n
                              Faithfulness<\/td>\n\u751f\u6210\u5185\u5bb9\u4e0e\u53c2\u8003\u6587\u6863\u7684\u4e00\u81f4\u6027<\/td>\n\u5fe0\u5b9e\u5ea6<\/td>\n0-1<\/td>\n\u9700\u53c2\u8003\u6587\u6863，\u7cbe\u5ea6\u9ad8<\/td>\n<\/tr>\n
                              Hallucination Rate<\/td>\n\u5e7b\u89c9\u8bed\u53e5\u6570\/\u603b\u8bed\u53e5\u6570<\/td>\n\u5e7b\u89c9\u53d1\u751f\u7387<\/td>\n0-1<\/td>\n\u76f4\u89c2，\u9700\u4eba\u5de5\u6807\u6ce8\u8f85\u52a9<\/td>\n<\/tr>\n
                              Rouge-L<\/td>\n\u751f\u6210\u5185\u5bb9\u4e0e\u53c2\u8003\u7684\u6587\u672c\u76f8\u4f3c\u5ea6<\/td>\n\u6587\u672c\u5339\u914d\u5ea6<\/td>\n0-1<\/td>\n\u5feb\u901f，\u4e0d\u8003\u8651\u4e8b\u5b9e\u6b63\u786e\u6027<\/td>\n<\/tr>\n
                              BERTScore<\/td>\n\u8bed\u4e49\u76f8\u4f3c\u5ea6<\/td>\n\u8bed\u4e49\u5339\u914d\u5ea6<\/td>\n0-1<\/td>\n\u6bd4Rouge\u66f4\u9c81\u68d2<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                              4.2 FactScore\u5e7b\u89c9\u8bc4\u4f30\u5b9e\u6218<\/h4>\n

                              4.2.1 \u73af\u5883\u51c6\u5907<\/h5>\n

                              # \u5b89\u88c5FactScore\u4f9d\u8d56
                              \npip install factscore==0.4.0
                              \npip install wikipedia-api==0.5.8 # \u4e8b\u5b9e\u9a8c\u8bc1\u4f9d\u8d56
                              \npip install nltk==3.8.1
                              \npip install spacy==3.7.4
                              \npython -m spacy download en_core_web_sm<\/p>\n

                              4.2.2 \u6838\u5fc3\u8bc4\u6d4b\u4ee3\u7801<\/h5>\n

                              FactScore \u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u6d41\u7a0b\u56fe
                              \n\u2502
                              \n\u251c\u2500\u2500 \u30101. \u8bc4\u6d4b\u51c6\u5907\u9636\u6bb5\u3011 (<\/span>Preparation Phase)<\/span>
                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u6570\u636e\u96c6\u6784\u5efa><\/span>: prepare_eval_data(<\/span>)<\/span>
                              \n\u2502 \u2502 \u251c\u2500\u2500 Question: "2024\u5965\u8fd0\u4f1a\u54ea\u91cc\u4e3e\u529e？"<\/span>
                              \n\u2502 \u2502 \u2514\u2500\u2500 Reference: "\u6cd5\u56fd\u5df4\u9ece…"<\/span> (<\/span>\u91d1\u6807\u51c6\u4e8b\u5b9e\/Ground Truth)<\/span>
                              \n\u2502 \u2514\u2500\u2500 <<\/span>\u6a21\u578b\u52a0\u8f7d><\/span>: load_model_and_tokenizer(<\/span>)<\/span>
                              \n\u2502 \u2514\u2500\u2500 Target Model: Qwen2.5-7B (<\/span>\u5f85\u6d4b\u5bf9\u8c61)<\/span>
                              \n\u2502
                              \n\u25bc
                              \n[<\/span>2<\/span>. \u5f85\u6d4b\u6a21\u578b\u63a8\u7406\u9636\u6bb5]<\/span> (<\/span>Candidate Generation)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                              \n\u2502 \u2502
                              \n\u251c\u2500\u2500 \u8f93\u5165\u5904\u7406 (<\/span>Input Processing)<\/span>: \u2502
                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u63d0\u793a\u8bcd\u6a21\u677f><\/span>: "\u8bf7\u51c6\u786e\u56de\u7b54…\u4e0d\u8981\u7f16\u9020"<\/span> \u2502
                              \n\u2502 \u2514\u2500\u2500 <<\/span>\u53c2\u6570\u63a7\u5236><\/span>: Temperature<\/span>=<\/span>0.1<\/span> (<\/span>\u6291\u5236\u968f\u673a\u6027，\u6a21\u62df\u4e25\u8c28\u6a21\u5f0f)<\/span> \u2502
                              \n\u2502 \u2502
                              \n\u251c\u2500\u2500 \u6a21\u578b\u751f\u6210 (<\/span>Inference)<\/span>: \u2502
                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u8f93\u5165><\/span>: Prompt Tokens \u2502
                              \n\u2502 \u2514\u2500\u2500 ><\/span> \u8f93\u51fa: Model Response "2024\u5965\u8fd0\u4f1a\u5728\u9a6c\u5fb7\u91cc\u4e3e\u529e"<\/span> (<\/span>\u5047\u8bbe\u5e7b\u89c9)<\/span>\u2502
                              \n\u2502 \u2502
                              \n\u2514\u2500\u2500 ><\/span> \u4e2d\u95f4\u4ea7\u7269: responses.json (<\/span>\u5305\u542b Q, Reference, Response)<\/span> \u2502
                              \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                              \n \u2502
                              \n \u25bc
                              \n[<\/span>3<\/span>. FactScore \u6838\u5fc3\u6838\u67e5\u9636\u6bb5]<\/span> (<\/span>Verification Phase)<\/span> <<\/span>\u2605 \u6838\u5fc3\u903b\u8f91><\/span> \u2500\u2510
                              \n\u2502 \u2502
                              \n\u251c\u2500\u2500 A. \u4e8b\u5b9e\u539f\u5b50\u5316 (<\/span>Atomic Fact Decomposition)<\/span> \u2502
                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u7ec4\u4ef6><\/span>: FactScore.get_score(<\/span>)<\/span> \u2502
                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u52a8\u4f5c><\/span>: \u5c06\u957f\u6587\u672c\u62c6\u89e3\u4e3a\u72ec\u7acb\u7684\u539f\u5b50\u58f0\u660e (<\/span>Atomic Facts)<\/span> \u2502
                              \n\u2502 \u2502 \u251c\u2500\u2500 \u53e5\u5b50: "2024\u5965\u8fd0\u4f1a\u5728\u9a6c\u5fb7\u91cc\u4e3e\u529e"<\/span> \u2502
                              \n\u2502 \u2502 \u2514\u2500\u2500 \u62c6\u89e3: Fact 1<\/span>: [<\/span>\u5965\u8fd0\u4f1a-\u4e3e\u529e\u5730-\u9a6c\u5fb7\u91cc]<\/span> (<\/span>\u53ef\u8bc1\u4f2a\u5355\u5143)<\/span> \u2502
                              \n\u2502 \u2502 \u2502
                              \n\u251c\u2500\u2500 B. \u4e8b\u5b9e\u9a8c\u8bc1 (<\/span>Fact Verification)<\/span> \u2502
                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u77e5\u8bc6\u6e90><\/span>: Reference Text (<\/span>\u7528\u6237\u63d0\u4f9b\u7684\u53c2\u8003\u7b54\u6848)<\/span> \u2502
                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u88c1\u5224><\/span>: GPT-4o (<\/span>\u901a\u8fc7 API \u8c03\u7528)<\/span> \u2502
                              \n\u2502 \u2502 \u251c\u2500\u2500 \u6307\u4ee4: "\u6839\u636e\u53c2\u8003\u6587\u672c，'\u9a6c\u5fb7\u91cc\u4e3e\u529e\u5965\u8fd0\u4f1a'\u662f\u5426\u5c5e\u5b9e？"<\/span> \u2502
                              \n\u2502 \u2502 \u2514\u2500\u2500 \u5224\u5b9a: False (<\/span>\u68c0\u6d4b\u5230\u5e7b\u89c9)<\/span> \u2502
                              \n\u2502 \u2502 \u2502
                              \n\u2514\u2500\u2500 ><\/span> \u8f93\u51fa\u7ed3\u679c: FactScore<\/span>=<\/span>0.0<\/span> (<\/span>\u8be5\u6837\u672c\u5e7b\u89c9\u7387 100<\/span>%)<\/span> \u2502
                              \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                              \n \u2502
                              \n \u25bc
                              \n[<\/span>4<\/span>. \u7ed3\u679c\u805a\u5408\u4e0e\u5206\u6790]<\/span> (<\/span>Aggregation &<\/span> Analysis)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                              \n\u2502 \u2502
                              \n\u251c\u2500\u2500 <<\/span>\u7edf\u8ba1><\/span>: sum(<\/span>scores)<\/span> \/ len(<\/span>samples)<\/span> \u2502
                              \n\u2514\u2500\u2500 ><\/span> \u6700\u7ec8\u62a5\u544a: category_analysis (<\/span>\u533a\u5206\u5e38\u8bc6\/\u4e13\u4e1a\/\u65f6\u6548\u6027\u5e7b\u89c9)<\/span> \u2502
                              \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

                              2. \u4ee3\u7801\u6a21\u5757\u534f\u4f5c\u6df1\u5ea6\u89e3\u6790<\/h6>\n

                              \u8fd9\u6bb5\u4ee3\u7801\u901a\u8fc7\u4e09\u4e2a\u6838\u5fc3\u51fd\u6570\u7684\u534f\u4f5c，\u6a21\u62df\u4e86\u4eba\u7c7b\u8fdb\u884c\u4e8b\u5b9e\u67e5\u6838（Fact Checking）\u7684\u8fc7\u7a0b：<\/p>\n

                              1. \u8003\u9898\u4e0e\u8003\u751f\u51c6\u5907 (prepare_eval_data & generate_responses)<\/h6>\n
                                \n
                              • \u534f\u4f5c\u903b\u8f91：\n
                                  \n
                                • prepare_eval_data \u5145\u5f53\u51fa\u9898\u4eba，\u5b83\u4e0d\u4ec5\u63d0\u4f9b\u95ee\u9898，\u8fd8\u5fc5\u987b\u63d0\u4f9b\u201c\u6807\u51c6\u7b54\u6848\u201d（Reference）\u3002\u6ca1\u6709 Reference，\u5e7b\u89c9\u8bc4\u6d4b\u5c31\u5931\u53bb\u4e86\u951a\u70b9\u3002<\/li>\n
                                • generate_responses \u5145\u5f53\u8003\u751f\u3002\u8fd9\u91cc\u7684\u4e00\u4e2a\u5173\u952e\u7ec6\u8282\u662f temperature=0.1\u3002\u5728\u5e7b\u89c9\u8bc4\u6d4b\u4e2d，\u6211\u4eec\u901a\u5e38\u5e0c\u671b\u8bc4\u4f30\u6a21\u578b\u201c\u6700\u786e\u4fe1\u201d\u7684\u77e5\u8bc6，\u800c\u4e0d\u662f\u5b83\u7684\u521b\u9020\u529b，\u56e0\u6b64\u9700\u8981\u901a\u8fc7\u4f4e\u6e29\u91c7\u6837\u7531\u201c\u53f3\u8111\u6a21\u5f0f\u201d\u5207\u6362\u5230\u201c\u5de6\u8111\u6a21\u5f0f\u201d\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
                                  2. \u663e\u5fae\u955c\u5f0f\u8bc4\u4f30 (run_factscore_evaluation) \u8fd9\u662f\u6574\u4e2a\u811a\u672c\u7684\u7075\u9b42\u3002\u5b83\u89e3\u51b3\u4e86\u201c\u6587\u672c\u76f8\u4f3c\u5ea6（Rouge\/BLEU）\u65e0\u6cd5\u8861\u91cf\u771f\u5b9e\u6027\u201d\u7684\u95ee\u9898\u3002<\/h6>\n
                                    \n
                                  • \u539f\u5b50\u5316（Decomposition）：\n
                                      \n
                                    • \u4f20\u7edf\u7684\u8bc4\u4f30\u770b\u91cd\u5b57\u9762\u91cd\u5408（Overlap）\u3002\u4f46\u6a21\u578b\u5982\u679c\u56de\u7b54\u201c\u5df4\u9ece\u6ca1\u6709\u4e3e\u529e2024\u5965\u8fd0\u4f1a\u201d，\u5b57\u9762\u91cd\u5408\u5ea6\u5f88\u9ad8，\u4f46\u4e8b\u5b9e\u5b8c\u5168\u76f8\u53cd\u3002<\/li>\n
                                    • FactScore \u4f1a\u8c03\u7528\u88c1\u5224\u6a21\u578b（\u5982 GPT-4），\u5c06\u590d\u6742\u7684\u590d\u5408\u53e5\u62c6\u89e3\u4e3a\u5355\u6761\u4e8b\u5b9e\u3002\u4f8b\u5982：\u201cPython\u662f\u4e00\u95e8\u7531Guido\u5f00\u53d1\u7684\u7f16\u8bd1\u578b\u8bed\u8a00\u201d \u4f1a\u88ab\u62c6\u4e3a：\n
                                    • Python\u7531Guido\u5f00\u53d1 (True)<\/li>\n
                                    • Python\u662f\u7f16\u8bd1\u578b\u8bed\u8a00 (False)<\/li>\n<\/li>\n
                                    • \u6700\u7ec8\u5f97\u5206\u4e3a 0.5，\u7cbe\u51c6\u6355\u6349\u4e86\u201c\u534a\u771f\u534a\u5047\u201d\u7684\u5e7b\u89c9\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
                                      3. \u88c1\u5224\u964d\u7ea7\u673a\u5236 (manual_hallucination_evaluation)<\/h6>\n
                                        \n
                                      • \u534f\u4f5c\u903b\u8f91：\n
                                          \n
                                        • \u7531\u4e8e FactScore \u5f3a\u4f9d\u8d56 OpenAI API \u8fdb\u884c\u9ad8\u667a\u5546\u62c6\u89e3，\u5f53\u7f51\u7edc\u4e0d\u53ef\u7528\u6216 Key \u8017\u5c3d\u65f6，\u4ee3\u7801\u4f1a\u81ea\u52a8\u964d\u7ea7\u5230 manual \u6a21\u5f0f\u3002<\/li>\n
                                        • \u5907\u7528\u65b9\u6848\u4f7f\u7528\u7b80\u5355\u7684**\u5173\u952e\u8bcd\u91cd\u53e0\u7387（Jieba Overlap）**\u4f5c\u4e3a\u66ff\u4ee3\u6307\u6807\u3002\u867d\u7136\u7cbe\u5ea6\u4e0d\u5982 GPT-4 \u88c1\u5224，\u4f46\u80fd\u4fdd\u8bc1\u8bc4\u6d4b\u6d41\u7a0b\u4e0d\u4e2d\u65ad（Fail-safe Design）\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
                                          3. \u4e3a\u4ec0\u4e48\u9700\u8981 Reference（\u53c2\u8003\u4e8b\u5b9e）？<\/h6>\n

                                          \u5728\u4ee3\u7801\u914d\u7f6e\u4e2d，fs.get_score(…, refs=refs) \u662f\u5173\u952e\u4e00\u6b65\u3002FactScore \u652f\u6301\u4e24\u79cd\u6a21\u5f0f：<\/p>\n

                                        • \u57fa\u4e8e\u77e5\u8bc6\u5e93（Knowledge-based）：\u4e0d\u63d0\u4f9b ref，\u8ba9\u88c1\u5224\u81ea\u5df1\u53bb\u67e5 Wikipedia（\u4ee3\u7801\u4e2d wikipedia-api \u7684\u4f5c\u7528）\u3002<\/li>\n
                                        • \u57fa\u4e8e\u4e0a\u4e0b\u6587（Context-based）：\u63d0\u4f9b ref，\u5f3a\u5236\u88c1\u5224\u53ea\u6839\u636e\u8fd9\u6bb5\u6587\u5b57\u5224\u65ad\u3002<\/li>\n
                                            \n
                                          • \u672c\u811a\u672c\u91c7\u7528\u6a21\u5f0f 2：\u901a\u8fc7 refs=refs \u4f20\u5165，\u8fd9\u6837\u53ef\u4ee5\u8bc4\u4f30 RAG \u7cfb\u7edf\u4e2d\u7684\u201c\u5fe0\u5b9e\u5ea6\u201d，\u6216\u8005\u5728\u79c1\u6709\u9886\u57df（\u5982\u4f01\u4e1a\u5185\u90e8\u6587\u6863）\u8bc4\u6d4b\u6a21\u578b\u662f\u5426\u4ea7\u751f\u4e86\u9886\u57df\u5e7b\u89c9\u3002<\/li>\n<\/ul>\n

                                            """
                                            \nFactScore\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u4f30\u811a\u672c
                                            \n\u652f\u6301\u4e2d\u6587\u5f00\u6e90\u6a21\u578b（Qwen2.5\/Llama3-Chinese）
                                            \n"""
                                            \nimport os
                                            \nimport json
                                            \nimport re
                                            \nimport jieba
                                            \nfrom factscore import FactScore
                                            \nfrom transformers import AutoModelForCausalLM, AutoTokenizer
                                            \nimport torch
                                            \nfrom tqdm import tqdm<\/p>\n

                                            # \u914d\u7f6e
                                            \nMODEL_PATH = "\/path\/to\/Qwen2.5-7B-Instruct"
                                            \nTOKENIZER_PATH = MODEL_PATH
                                            \nEVAL_DATA_PATH = ".\/hallucination_eval_data.json" # \u8bc4\u6d4b\u6570\u636e\u96c6
                                            \nOUTPUT_DIR = ".\/hallucination_results"
                                            \nDEVICE = "cuda" if torch.cuda.is_available() else "cpu"<\/p>\n

                                            # \u521b\u5efa\u8f93\u51fa\u76ee\u5f55
                                            \nos.makedirs(OUTPUT_DIR, exist_ok=True)<\/p>\n

                                            # 1. \u51c6\u5907\u8bc4\u6d4b\u6570\u636e\u96c6
                                            \ndef prepare_eval_data():
                                            \n """
                                            \n \u6784\u5efa\u5e7b\u89c9\u8bc4\u6d4b\u6570\u636e\u96c6
                                            \n \u683c\u5f0f: [{"question": "\u95ee\u9898", "reference": "\u53c2\u8003\u4e8b\u5b9e", "category": "\u7c7b\u522b"}]
                                            \n """
                                            \n eval_data = [
                                            \n {
                                            \n "question": "2024\u5e74\u5965\u8fd0\u4f1a\u7684\u4e3e\u529e\u57ce\u5e02\u662f\u54ea\u91cc？",
                                            \n "reference": "2024\u5e74\u590f\u5b63\u5965\u6797\u5339\u514b\u8fd0\u52a8\u4f1a\u5728\u6cd5\u56fd\u5df4\u9ece\u4e3e\u529e，\u8fd9\u662f\u5df4\u9ece\u7b2c\u4e09\u6b21\u4e3e\u529e\u5965\u8fd0\u4f1a\u3002",
                                            \n "category": "\u4e8b\u5b9e\u6027\u95ee\u9898"
                                            \n },
                                            \n {
                                            \n "question": "\u7b80\u8ff0\u91cf\u5b50\u8ba1\u7b97\u7684\u57fa\u672c\u539f\u7406",
                                            \n "reference": "\u91cf\u5b50\u8ba1\u7b97\u5229\u7528\u91cf\u5b50\u529b\u5b66\u7684\u53e0\u52a0\u6001\u548c\u7ea0\u7f20\u7279\u6027\u8fdb\u884c\u8ba1\u7b97，\u901a\u8fc7\u91cf\u5b50\u6bd4\u7279（Qubit）\u5b58\u50a8\u548c\u5904\u7406\u4fe1\u606f，\u76f8\u6bd4\u7ecf\u5178\u8ba1\u7b97\u673a\u5728\u7279\u5b9a\u95ee\u9898\u4e0a\u6709\u6307\u6570\u7ea7\u52a0\u901f\u3002",
                                            \n "category": "\u4e13\u4e1a\u77e5\u8bc6"
                                            \n },
                                            \n {
                                            \n "question": "\u4e2d\u56fd\u7684\u9996\u90fd\u662f\u54ea\u4e2a\u57ce\u5e02？\u6709\u54ea\u4e9b\u8457\u540d\u7684\u5386\u53f2\u5efa\u7b51？",
                                            \n "reference": "\u4e2d\u56fd\u7684\u9996\u90fd\u662f\u5317\u4eac，\u8457\u540d\u5386\u53f2\u5efa\u7b51\u5305\u62ec\u6545\u5bab\u3001\u5929\u575b\u3001\u957f\u57ce\u3001\u9890\u548c\u56ed\u7b49\u3002",
                                            \n "category": "\u5e38\u8bc6\u95ee\u9898"
                                            \n },
                                            \n {
                                            \n "question": "LangChain\u6846\u67b6\u7684\u4e3b\u8981\u529f\u80fd\u662f\u4ec0\u4e48？",
                                            \n "reference": "LangChain\u662f\u7528\u4e8e\u6784\u5efa\u57fa\u4e8e\u5927\u8bed\u8a00\u6a21\u578b\u7684\u5e94\u7528\u7a0b\u5e8f\u7684\u5f00\u6e90\u6846\u67b6，\u6838\u5fc3\u529f\u80fd\u5305\u62ec\u6a21\u578b\u96c6\u6210\u3001\u5de5\u5177\u8c03\u7528\u3001\u8bb0\u5fc6\u7ba1\u7406\u3001Agent\u667a\u80fd\u4f53\u6784\u5efa\u7b49\u3002",
                                            \n "category": "\u6280\u672f\u95ee\u9898"
                                            \n },
                                            \n {
                                            \n "question": "2023\u5e74\u8bfa\u8d1d\u5c14\u7269\u7406\u5b66\u5956\u7684\u83b7\u5956\u8005\u53ca\u5176\u8d21\u732e\u662f\u4ec0\u4e48？",
                                            \n "reference": "2023\u5e74\u8bfa\u8d1d\u5c14\u7269\u7406\u5b66\u5956\u6388\u4e88Pierre Agostini\u3001Ferenc Krausz\u548cAnne L'Huillier，\u4ee5\u8868\u5f70\u4ed6\u4eec\u5728\u963f\u79d2\u5149\u8109\u51b2\u751f\u6210\u65b9\u9762\u7684\u5b9e\u9a8c\u65b9\u6cd5\u8d21\u732e\u3002",
                                            \n "category": "\u65f6\u6548\u6027\u4e8b\u5b9e"
                                            \n }
                                            \n ]<\/p>\n

                                            # \u4fdd\u5b58\u6570\u636e\u96c6
                                            \n with open(EVAL_DATA_PATH, "w", encoding="utf-8") as f:
                                            \n json.dump(eval_data, f, ensure_ascii=False, indent=4)<\/p>\n

                                            return eval_data<\/p>\n

                                            # 2. \u52a0\u8f7d\u6a21\u578b\u548cTokenizer
                                            \ndef load_model_and_tokenizer():
                                            \n """\u52a0\u8f7d\u5f00\u6e90\u6a21\u578b\u548cTokenizer"""
                                            \n print(f"\u52a0\u8f7d\u6a21\u578b: {MODEL_PATH}")<\/p>\n

                                            tokenizer = AutoTokenizer.from_pretrained(
                                            \n TOKENIZER_PATH,
                                            \n trust_remote_code=True,
                                            \n padding_side="right"
                                            \n )<\/p>\n

                                            model = AutoModelForCausalLM.from_pretrained(
                                            \n MODEL_PATH,
                                            \n torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
                                            \n device_map="auto",
                                            \n trust_remote_code=True
                                            \n )<\/p>\n

                                            # \u8bbe\u7f6epad token（\u82e5\u4e0d\u5b58\u5728）
                                            \n if tokenizer.pad_token is None:
                                            \n tokenizer.pad_token = tokenizer.eos_token
                                            \n model.config.pad_token_id = model.config.eos_token_id<\/p>\n

                                            return model, tokenizer<\/p>\n

                                            # 3. \u751f\u6210\u6a21\u578b\u56de\u590d
                                            \ndef generate_responses(model, tokenizer, eval_data):
                                            \n """\u751f\u6210\u6a21\u578b\u56de\u590d"""
                                            \n responses = []<\/p>\n

                                            for item in tqdm(eval_data, desc="\u751f\u6210\u56de\u590d"):
                                            \n # \u6784\u5efa\u63d0\u793a\u8bcd
                                            \n prompt = f"""\u8bf7\u51c6\u786e\u56de\u7b54\u4ee5\u4e0b\u95ee\u9898，\u786e\u4fdd\u6240\u6709\u4fe1\u606f\u771f\u5b9e\u53ef\u9760，\u4e0d\u8981\u7f16\u9020\u4e8b\u5b9e：
                                            \n\u95ee\u9898：{item['question']}
                                            \n\u56de\u7b54："""<\/p>\n

                                            # \u7f16\u7801
                                            \n inputs = tokenizer(
                                            \n prompt,
                                            \n return_tensors="pt",
                                            \n padding=True,
                                            \n truncation=True,
                                            \n max_length=2048
                                            \n ).to(DEVICE)<\/p>\n

                                            # \u751f\u6210\u56de\u590d
                                            \n with torch.no_grad():
                                            \n outputs = model.generate(
                                            \n **inputs,
                                            \n max_new_tokens=512,
                                            \n temperature=0.1, # \u4f4e\u6e29\u5ea6\u51cf\u5c11\u968f\u673a\u6027
                                            \n top_p=0.95,
                                            \n do_sample=True,
                                            \n pad_token_id=tokenizer.pad_token_id,
                                            \n eos_token_id=tokenizer.eos_token_id
                                            \n )<\/p>\n

                                            # \u89e3\u7801\u56de\u590d
                                            \n response = tokenizer.decode(
                                            \n outputs[0][len(inputs.input_ids[0]):],
                                            \n skip_special_tokens=True
                                            \n ).strip()<\/p>\n

                                            responses.append({
                                            \n "question": item["question"],
                                            \n "reference": item["reference"],
                                            \n "category": item["category"],
                                            \n "model_response": response
                                            \n })<\/p>\n

                                            # \u4fdd\u5b58\u56de\u590d
                                            \n responses_path = os.path.join(OUTPUT_DIR, "model_responses.json")
                                            \n with open(responses_path, "w", encoding="utf-8") as f:
                                            \n json.dump(responses, f, ensure_ascii=False, indent=4)<\/p>\n

                                            return responses<\/p>\n

                                            # 4. FactScore\u8bc4\u4f30
                                            \ndef run_factscore_evaluation(responses):
                                            \n """\u6267\u884cFactScore\u8bc4\u4f30"""
                                            \n print("\u521d\u59cb\u5316FactScore\u8bc4\u4f30\u5668…")<\/p>\n

                                            # \u521d\u59cb\u5316FactScore（\u652f\u6301\u4e2d\u6587\u9700\u81ea\u5b9a\u4e49\u5206\u8bcd）
                                            \n fs = FactScore(
                                            \n model_name="gpt-4o", # \u4e8b\u5b9e\u9a8c\u8bc1\u6a21\u578b（\u53ef\u9009gpt-3.5-turbo）
                                            \n api_key=os.getenv("OPENAI_API_KEY"), # \u4ece\u73af\u5883\u53d8\u91cf\u52a0\u8f7dAPI Key
                                            \n verbose=True,
                                            \n cache_path=os.path.join(OUTPUT_DIR, "factscore_cache")
                                            \n )<\/p>\n

                                            # \u51c6\u5907\u8bc4\u4f30\u6570\u636e
                                            \n texts = [item["model_response"] for item in responses]
                                            \n refs = [item["reference"] for item in responses]<\/p>\n

                                            # \u6267\u884c\u8bc4\u4f30
                                            \n print("\u6267\u884cFactScore\u8bc4\u4f30…")
                                            \n scores = fs.get_score(
                                            \n texts=texts,
                                            \n refs=refs,
                                            \n chunk_size=100, # \u6587\u672c\u5206\u5757\u5927\u5c0f
                                            \n granularity="sentence" # \u53e5\u5b50\u7ea7\u8bc4\u4f30
                                            \n )<\/p>\n

                                            # \u6574\u5408\u7ed3\u679c
                                            \n eval_results = []
                                            \n for i, item in enumerate(responses):
                                            \n eval_results.append({
                                            \n "question": item["question"],
                                            \n "category": item["category"],
                                            \n "reference": item["reference"],
                                            \n "model_response": item["model_response"],
                                            \n "factscore": scores["scores"][i],
                                            \n "factual_sentences": scores["factual_sentences"][i],
                                            \n "hallucinated_sentences": scores["hallucinated_sentences"][i]
                                            \n })<\/p>\n

                                            # \u8ba1\u7b97\u5e73\u5747\u5206
                                            \n avg_factscore = sum([r["factscore"] for r in eval_results]) \/ len(eval_results)<\/p>\n

                                            # \u4fdd\u5b58\u8bc4\u4f30\u7ed3\u679c
                                            \n final_results = {
                                            \n "average_factscore": avg_factscore,
                                            \n "category_analysis": {},
                                            \n "detailed_results": eval_results
                                            \n }<\/p>\n

                                            # \u6309\u7c7b\u522b\u5206\u6790
                                            \n category_scores = {}
                                            \n for item in eval_results:
                                            \n category = item["category"]
                                            \n if category not in category_scores:
                                            \n category_scores[category] = []
                                            \n category_scores[category].append(item["factscore"])<\/p>\n

                                            for category, scores in category_scores.items():
                                            \n final_results["category_analysis"][category] = {
                                            \n "average_score": sum(scores) \/ len(scores),
                                            \n "count": len(scores)
                                            \n }<\/p>\n

                                            # \u4fdd\u5b58\u6700\u7ec8\u7ed3\u679c
                                            \n results_path = os.path.join(OUTPUT_DIR, "hallucination_evaluation.json")
                                            \n with open(results_path, "w", encoding="utf-8") as f:
                                            \n json.dump(final_results, f, ensure_ascii=False, indent=4)<\/p>\n

                                            # \u6253\u5370\u7ed3\u679c
                                            \n print("\\\\n=== \u5e7b\u89c9\u8bc4\u4f30\u7ed3\u679c ===")
                                            \n print(f"\u5e73\u5747FactScore: {avg_factscore:.4f}")
                                            \n print("\\\\n\u6309\u7c7b\u522b\u5206\u6790:")
                                            \n for category, analysis in final_results["category_analysis"].items():
                                            \n print(f" {category}: {analysis['average_score']:.4f} ({analysis['count']}\u4e2a\u6837\u672c)")<\/p>\n

                                            print("\\\\n\u8be6\u7ec6\u7ed3\u679c:")
                                            \n for item in eval_results:
                                            \n print(f"\\\\n\u95ee\u9898: {item['question']}")
                                            \n print(f"FactScore: {item['factscore']:.4f}")
                                            \n print(f"\u6a21\u578b\u56de\u590d: {item['model_response']}")
                                            \n print(f"\u5e7b\u89c9\u8bed\u53e5: {item['hallucinated_sentences']}")<\/p>\n

                                            return final_results<\/p>\n

                                            # 5. \u4e2d\u6587\u5e7b\u89c9\u7387\u624b\u52a8\u8bc4\u4f30（\u5907\u7528\u65b9\u6848）
                                            \ndef manual_hallucination_evaluation(responses):
                                            \n """
                                            \n \u4e2d\u6587\u5e7b\u89c9\u7387\u624b\u52a8\u8bc4\u4f30（\u65e0API\u65f6\u4f7f\u7528）
                                            \n \u89c4\u5219：
                                            \n 1. \u5c06\u56de\u590d\u6309\u53e5\u5b50\u5206\u5272
                                            \n 2. \u9010\u4e2a\u53e5\u5b50\u5bf9\u6bd4\u53c2\u8003\u6587\u6863
                                            \n 3. \u8ba1\u7b97\u5e7b\u89c9\u8bed\u53e5\u6bd4\u4f8b
                                            \n """
                                            \n def split_chinese_sentences(text):
                                            \n """\u4e2d\u6587\u53e5\u5b50\u5206\u5272"""
                                            \n # \u7b80\u5355\u7684\u4e2d\u6587\u5206\u53e5\u89c4\u5219
                                            \n sentences = re.split(r'[\u3002！？；]', text)
                                            \n sentences = [s.strip() for s in sentences if s.strip()]
                                            \n return sentences<\/p>\n

                                            eval_results = []<\/p>\n

                                            for item in responses:
                                            \n # \u5206\u53e5
                                            \n response_sentences = split_chinese_sentences(item["model_response"])
                                            \n reference_sentences = split_chinese_sentences(item["reference"])<\/p>\n

                                            # \u4e8b\u5b9e\u5339\u914d\u68c0\u67e5
                                            \n hallucinated_count = 0
                                            \n factual_sentences = []
                                            \n hallucinated_sentences = []<\/p>\n

                                            for sent in response_sentences:
                                            \n # \u7b80\u5355\u7684\u4e8b\u5b9e\u5339\u914d（\u57fa\u4e8e\u5173\u952e\u8bcd）
                                            \n reference_words = set(jieba.lcut(item["reference"]))
                                            \n response_words = set(jieba.lcut(sent))
                                            \n overlap = len(reference_words & response_words) \/ len(response_words) if response_words else 0<\/p>\n

                                            if overlap < 0.3 and len(sent) > 5: # \u4f4e\u91cd\u53e0\u4e14\u957f\u5ea6\u8db3\u591f，\u5224\u5b9a\u4e3a\u5e7b\u89c9
                                            \n hallucinated_count += 1
                                            \n hallucinated_sentences.append(sent)
                                            \n else:
                                            \n factual_sentences.append(sent)<\/p>\n

                                            # \u8ba1\u7b97\u5e7b\u89c9\u7387
                                            \n hallucination_rate = hallucinated_count \/ len(response_sentences) if response_sentences else 0
                                            \n factscore = 1 – hallucination_rate<\/p>\n

                                            eval_results.append({
                                            \n "question": item["question"],
                                            \n "category": item["category"],
                                            \n "factscore": factscore,
                                            \n "hallucination_rate": hallucination_rate,
                                            \n "total_sentences": len(response_sentences),
                                            \n "hallucinated_sentences": hallucinated_sentences,
                                            \n "factual_sentences": factual_sentences
                                            \n })<\/p>\n

                                            # \u8ba1\u7b97\u5e73\u5747\u5206
                                            \n avg_factscore = sum([r["factscore"] for r in eval_results]) \/ len(eval_results)
                                            \n avg_hallucination_rate = sum([r["hallucination_rate"] for r in eval_results]) \/ len(eval_results)<\/p>\n

                                            print(f"\\\\n=== \u624b\u52a8\u5e7b\u89c9\u8bc4\u4f30\u7ed3\u679c ===")
                                            \n print(f"\u5e73\u5747FactScore: {avg_factscore:.4f}")
                                            \n print(f"\u5e73\u5747\u5e7b\u89c9\u7387: {avg_hallucination_rate:.4f}")<\/p>\n

                                            return eval_results<\/p>\n

                                            # \u4e3b\u6267\u884c\u51fd\u6570
                                            \ndef main():
                                            \n # 1. \u51c6\u5907\u8bc4\u6d4b\u6570\u636e
                                            \n eval_data = prepare_eval_data()<\/p>\n

                                            # 2. \u52a0\u8f7d\u6a21\u578b
                                            \n model, tokenizer = load_model_and_tokenizer()<\/p>\n

                                            # 3. \u751f\u6210\u56de\u590d
                                            \n responses = generate_responses(model, tokenizer, eval_data)<\/p>\n

                                            # 4. \u6267\u884c\u5e7b\u89c9\u8bc4\u4f30
                                            \n try:
                                            \n # \u4f18\u5148\u4f7f\u7528FactScore\u5b98\u65b9\u8bc4\u4f30
                                            \n final_results = run_factscore_evaluation(responses)
                                            \n except Exception as e:
                                            \n print(f"FactScore\u5b98\u65b9\u8bc4\u4f30\u5931\u8d25: {e}")
                                            \n print("\u4f7f\u7528\u624b\u52a8\u5e7b\u89c9\u8bc4\u4f30\u65b9\u6848")
                                            \n final_results = manual_hallucination_evaluation(responses)<\/p>\n

                                            return final_results<\/p>\n

                                            if __name__ == "__main__":
                                            \n main()<\/p>\n

                                            4.3 \u5e7b\u89c9\u4f18\u5316\u7b56\u7565<\/h4>\n

                                            \u57fa\u4e8e\u8bc4\u6d4b\u7ed3\u679c，\u53ef\u901a\u8fc7\u4ee5\u4e0b\u65b9\u5f0f\u964d\u4f4e\u6a21\u578b\u5e7b\u89c9：<\/p>\n

                                            4.3.1 \u63d0\u793a\u8bcd\u4f18\u5316<\/h5>\n

                                            """
                                            \n\u53cd\u5e7b\u89c9\u63d0\u793a\u8bcd\u6a21\u677f
                                            \n"""
                                            \nANTI_HALLUCINATION_PROMPT = """\u4f60\u662f\u4e00\u4e2a\u4e25\u8c28\u7684\u4e8b\u5b9e\u56de\u7b54\u52a9\u624b，\u5fc5\u987b\u9075\u5b88\u4ee5\u4e0b\u89c4\u5219：
                                            \n1. \u53ea\u56de\u7b54\u4f60\u786e\u5b9a\u4e3a\u771f\u5b9e\u7684\u4fe1\u606f，\u4e0d\u786e\u5b9a\u7684\u4fe1\u606f\u5fc5\u987b\u660e\u786e\u8bf4\u660e
                                            \n2. \u5bf9\u4e8e\u65f6\u6548\u6027\u95ee\u9898，\u8bf4\u660e\u4fe1\u606f\u7684\u65f6\u95f4\u8303\u56f4
                                            \n3. \u5bf9\u4e8e\u4e13\u4e1a\u95ee\u9898，\u786e\u4fdd\u672f\u8bed\u51c6\u786e，\u4e0d\u7f16\u9020\u6982\u5ff5
                                            \n4. \u5982\u679c\u65e0\u6cd5\u56de\u7b54，\u76f4\u63a5\u8bf4\u660e"\u65e0\u6cd5\u56de\u7b54\u8be5\u95ee\u9898"，\u4e0d\u8981\u7f16\u9020\u7b54\u6848
                                            \n5. \u56de\u7b54\u7ed3\u6784\u6e05\u6670，\u5206\u70b9\u8bf4\u660e，\u4fbf\u4e8e\u9a8c\u8bc1<\/p>\n

                                            \u95ee\u9898：{question}
                                            \n\u56de\u7b54："""<\/p>\n

                                            # \u4f7f\u7528\u793a\u4f8b
                                            \nprompt = ANTI_HALLUCINATION_PROMPT.format(question="2024\u5e74\u5965\u8fd0\u4f1a\u7684\u4e3e\u529e\u57ce\u5e02\u662f\u54ea\u91cc？")<\/p>\n

                                            4.3.2 \u68c0\u7d22\u589e\u5f3a\u751f\u6210（RAG）<\/h5>\n

                                            \u7ed3\u5408LangChain\u5b9e\u73b0\u68c0\u7d22\u589e\u5f3a，\u901a\u8fc7**\u201c\u4e0a\u4e0b\u6587\u538b\u7f29\u201d**\u6280\u672f\u6ee4\u9664\u65e0\u5173\u566a\u58f0，\u4ece\u6e90\u5934\u5207\u65ad\u5e7b\u89c9，\u4ece\u53ef\u4fe1\u6570\u636e\u6e90\u83b7\u53d6\u4e8b\u5b9e：<\/p>\n

                                            1. RAG \u53cd\u5e7b\u89c9\u5de5\u4f5c\u6d41\u56fe\u89e3<\/h5>\n

                                            RAG \u53cd\u5e7b\u89c9\u63a8\u7406\u6d41\u7a0b\u56fe
                                            \n\u2502
                                            \n\u251c\u2500\u2500 \u3010\u7528\u6237\u8f93\u5165\u3011
                                            \n\u2502 \u2514\u2500\u2500 Query: "LangChain\u7684\u6838\u5fc3\u529f\u80fd\u662f\u4ec0\u4e48？"<\/span>
                                            \n\u2502
                                            \n\u25bc
                                            \n[<\/span>1<\/span>. \u8bed\u4e49\u68c0\u7d22\u9636\u6bb5 (<\/span>Semantic Retrieval)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 A. \u95ee\u9898\u5411\u91cf\u5316 (<\/span>Query Embedding)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u8c03\u7528\u6a21\u578b><\/span>: BAAI\/bge-base-zh-v1.5 \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u52a8\u4f5c><\/span>: \u5c06\u6587\u672c\u8f6c\u5316\u4e3a 768<\/span>\u7ef4 \u5411\u91cf \u2502
                                            \n\u2502 \u2514\u2500\u2500 ><\/span> \u8f93\u51fa: Query Vector [<\/span>0.12<\/span>, -0.45, 0.88<\/span>..<\/span>.]<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 B. \u5411\u91cf\u5e93\u5339\u914d (<\/span>Vector Search)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u8bfb\u53d6\u8def\u5f84><\/span>: .\/rag_db (<\/span>Chroma \u6301\u4e45\u5316\u76ee\u5f55)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u8ba1\u7b97><\/span>: \u4f59\u5f26\u76f8\u4f3c\u5ea6 (<\/span>Cosine Similarity)<\/span> Top-K=<\/span>3<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 ><\/span> \u8f93\u51fa: 3<\/span>\u4e2a\u539f\u59cb\u6587\u6863\u7247\u6bb5 (<\/span>Raw Documents)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 Doc A: "LangChain\u662f\u5f00\u53d1\u6846\u67b6…"<\/span> (<\/span>\u76f8\u5173)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 Doc B: "LangChain\u7684\u521b\u59cb\u4eba\u662f…"<\/span> (<\/span>\u90e8\u5206\u76f8\u5173)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 Doc C: "Python\u5b89\u88c5\u6559\u7a0b…"<\/span> (<\/span>\u566a\u58f0)<\/span> \u2502
                                            \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                            \n \u2502
                                            \n \u25bc
                                            \n[<\/span>2<\/span>. \u4e0a\u4e0b\u6587\u538b\u7f29\u9636\u6bb5 (<\/span>Context Compression)<\/span>]<\/span> <<\/span>\u2605 \u6838\u5fc3\u53bb\u566a\u6b65\u9aa4><\/span> \u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 <<\/span>\u7ec4\u4ef6><\/span>: ContextualCompressionRetriever \u2502
                                            \n\u251c\u2500\u2500 <<\/span>\u673a\u5236><\/span>: LLMChainExtractor (<\/span>\u57fa\u4e8e\u6a21\u578b\u7684\u4fe1\u606f\u62bd\u53d6)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 \u21bb \u538b\u7f29\u5faa\u73af (<\/span>Extraction Loop)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 \u8f93\u5165: Query + Raw Doc A\/B\/C \u2502
                                            \n\u2502 \u251c\u2500\u2500 \u5224\u522b: "\u8fd9\u53e5\u8bdd\u80fd\u56de\u7b54\u7528\u6237\u7684\u95ee\u9898\u5417？"<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 \u52a8\u4f5c: \u2502
                                            \n\u2502 \u251c\u2500\u2500 Doc A –><\/span> \u4fdd\u7559\u6838\u5fc3\u5b9a\u4e49 \u2502
                                            \n\u2502 \u251c\u2500\u2500 Doc B –><\/span> \u4ec5\u4fdd\u7559\u5173\u952e\u4eba\u540d \u2502
                                            \n\u2502 \u2514\u2500\u2500 Doc C –><\/span> \u4e22\u5f03 (<\/span>DROP)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u2514\u2500\u2500 ><\/span> \u8f93\u51fa: \u7cbe\u70bc\u4e8b\u5b9e (<\/span>Refined Context)<\/span> \u2502
                                            \n "LangChain\u662f\u5f00\u6e90\u6846\u67b6…\u6838\u5fc3\u529f\u80fd\u5305\u62ec…"<\/span> \u2502
                                            \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                            \n \u2502
                                            \n \u25bc
                                            \n[<\/span>3<\/span>. \u53d7\u63a7\u751f\u6210\u9636\u6bb5 (<\/span>Constrained Generation)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 A. \u63d0\u793a\u8bcd\u6784\u5efa (<\/span>Prompt Engineering)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u6a21\u677f><\/span>: "\u57fa\u4e8e\u4ee5\u4e0b\u4e8b\u5b9e…\u4e0d\u8981\u6dfb\u52a0\u5176\u4ed6\u4fe1\u606f"<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u6ce8\u5165><\/span>: Refined Context (<\/span>\u7cbe\u70bc\u540e\u7684\u4e8b\u5b9e)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 ><\/span> \u6700\u7ec8 Prompt: [<\/span>System]<\/span> + [<\/span>Facts]<\/span> + [<\/span>User Query]<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 B. \u6a21\u578b\u63a8\u7406 (<\/span>Inference)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u6a21\u578b><\/span>: Qwen2.5-7B-Instruct \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u53c2\u6570><\/span>: Temperature<\/span>=<\/span>0.1<\/span> (<\/span>\u6781\u4f4e\u968f\u673a\u6027)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 ><\/span> \u6700\u7ec8\u56de\u590d: "LangChain\u7684\u4e3b\u8981\u529f\u80fd\u662f…"<\/span> \u2502
                                            \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

                                            2. \u5173\u952e\u7ec4\u4ef6\u534f\u4f5c\u89e3\u6790<\/h5>\n

                                            \u8fd9\u6bb5\u4ee3\u7801\u4e0d\u4ec5\u4ec5\u662f\u7b80\u5355\u7684\u67e5\u5e93，\u5b83\u6784\u5efa\u4e86\u4e00\u4e2a**\u201c\u6f0f\u6597\u5f0f\u201d**\u7684\u4fe1\u606f\u5904\u7406\u7cfb\u7edf：<\/p>\n

                                            1. \u5411\u91cf\u5316\u5f15\u64ce (HuggingFaceEmbeddings)<\/p>\n

                                              \n
                                            • \u534f\u4f5c\u903b\u8f91：\u8fd9\u662f RAG \u7684\u201c\u7ffb\u8bd1\u5b98\u201d\u3002\u5b83\u4f7f\u7528 BAAI\/bge-base-zh-v1.5（\u76ee\u524d\u4e2d\u6587\u8bed\u4e49\u7406\u89e3\u6700\u5f3a\u7684\u5f00\u6e90 Embedding \u4e4b\u4e00）\u5c06\u7528\u6237\u7684\u95ee\u9898\u548c\u77e5\u8bc6\u5e93\u5185\u5bb9\u7edf\u4e00\u8f6c\u5316\u4e3a\u6570\u5b66\u5411\u91cf\u3002<\/li>\n
                                            • \u53cd\u5e7b\u89c9\u4f5c\u7528：\u53ea\u6709\u201c\u7ffb\u8bd1\u201d\u5f97\u51c6，\u624d\u80fd\u627e\u5230\u771f\u6b63\u76f8\u5173\u7684\u4e8b\u5b9e，\u907f\u514d\u56e0\u4e3a\u5173\u952e\u8bcd\u5339\u914d\u9519\u8bef\u800c\u5f15\u5165\u9519\u8bef\u7684\u4e0a\u4e0b\u6587\u3002<\/li>\n<\/ul>\n

                                              2. \u4e0a\u4e0b\u6587\u538b\u7f29\u5668 (ContextualCompressionRetriever)<\/p>\n

                                                \n
                                              • \u534f\u4f5c\u903b\u8f91：\u8fd9\u662f\u4ee3\u7801\u4e2d\u6700\u4eae\u773c\u7684\u90e8\u5206\u3002\u666e\u901a\u7684 RAG \u4f1a\u76f4\u63a5\u628a\u68c0\u7d22\u5230\u7684 Top-3 \u6587\u6863\u5168\u90e8\u6254\u7ed9\u5927\u6a21\u578b，\u91cc\u9762\u53ef\u80fd\u5305\u542b\u5927\u91cf\u65e0\u5173\u5e9f\u8bdd\u3002<\/li>\n
                                              • \u5de5\u4f5c\u539f\u7406：\n
                                                  \n
                                                • \u5b83\u8c03\u7528 LLMChainExtractor，\u8ba9\u5927\u6a21\u578b\u5145\u5f53\u201c\u8367\u5149\u7b14\u201d\u3002<\/li>\n
                                                • \u5728\u751f\u6210\u6700\u7ec8\u7b54\u6848\u4e4b\u524d，\u5b83\u5148\u5feb\u901f\u6d4f\u89c8\u4e00\u904d\u68c0\u7d22\u5230\u7684\u6587\u6863，\u628a\u8ddf\u95ee\u9898\u771f\u6b63\u76f8\u5173\u7684\u53e5\u5b50\u201c\u9ad8\u4eae\u201d\u51fa\u6765，\u5220\u6389\u5176\u4ed6\u90e8\u5206\u3002<\/li>\n<\/ul>\n<\/li>\n
                                                • \u53cd\u5e7b\u89c9\u4f5c\u7528：\u566a\u58f0\u662f\u5e7b\u89c9\u7684\u6e29\u5e8a\u3002\u901a\u8fc7\u538b\u7f29\u4e0a\u4e0b\u6587，\u6211\u4eec\u53ea\u5582\u7ed9\u6a21\u578b\u201c\u7eaf\u51c0\u201d\u7684\u4e8b\u5b9e，\u6a21\u578b\u201c\u80e1\u8bf4\u516b\u9053\u201d\u7684\u6982\u7387\u4f1a\u5448\u6307\u6570\u7ea7\u4e0b\u964d\u3002<\/li>\n<\/ul>\n

                                                  3. \u4e25\u683c\u7ea6\u675f\u751f\u6210 (rag_generate \u51fd\u6570)<\/p>\n

                                                    \n
                                                  • \u534f\u4f5c\u903b\u8f91：\u5c06\u6e05\u6d17\u540e\u7684 context \u586b\u5165\u7279\u5b9a\u6a21\u677f\u3002<\/li>\n
                                                  • \u5173\u952e\u6307\u4ee4：prompt \u4e2d\u660e\u786e\u5305\u542b\u4e86\u201c\u4ec5\u4f7f\u7528\u63d0\u4f9b\u7684\u4e8b\u5b9e\u201d\u548c\u201c\u4e0d\u8981\u6dfb\u52a0\u5176\u4ed6\u4fe1\u606f\u201d\u3002\u914d\u5408 temperature=0.1，\u5f3a\u8feb\u6a21\u578b\u4ece\u201c\u521b\u9020\u6a21\u5f0f\u201d\u5207\u6362\u5230\u201c\u9605\u8bfb\u7406\u89e3\u6a21\u5f0f\u201d，\u6700\u5927\u7a0b\u5ea6\u4fdd\u8bc1\u56de\u7b54\u7684\u5fe0\u5b9e\u5ea6（Faithfulness）\u3002<\/li>\n<\/ul>\n

                                                    """
                                                    \nRAG\u53cd\u5e7b\u89c9\u793a\u4f8b
                                                    \n"""
                                                    \nfrom langchain.vectorstores import Chroma
                                                    \nfrom langchain.embeddings import HuggingFaceEmbeddings
                                                    \nfrom langchain.retrievers import ContextualCompressionRetriever
                                                    \nfrom langchain.retrievers.document_compressors import LLMChainExtractor<\/p>\n

                                                    # \u521d\u59cb\u5316Embedding
                                                    \nembedding = HuggingFaceEmbeddings(model_name="BAAI\/bge-base-zh-v1.5")<\/p>\n

                                                    # \u52a0\u8f7d\u5411\u91cf\u6570\u636e\u5e93（\u5305\u542b\u53ef\u4fe1\u4e8b\u5b9e\u6570\u636e）
                                                    \nvector_db = Chroma(persist_directory=".\/rag_db", embedding_function=embedding)<\/p>\n

                                                    # \u6784\u5efa\u68c0\u7d22\u5668
                                                    \nretriever = vector_db.as_retriever(search_kwargs={"k": 3})<\/p>\n

                                                    # \u4e0a\u4e0b\u6587\u538b\u7f29（\u63d0\u5347\u68c0\u7d22\u7cbe\u5ea6）
                                                    \ncompressor = LLMChainExtractor.from_llm(model)
                                                    \ncompression_retriever = ContextualCompressionRetriever(base_compressor=compressor, base_retriever=retriever)<\/p>\n

                                                    # \u68c0\u7d22+\u751f\u6210
                                                    \ndef rag_generate(question):
                                                    \n # \u68c0\u7d22\u76f8\u5173\u4e8b\u5b9e
                                                    \n retrieved_docs = compression_retriever.get_relevant_documents(question)
                                                    \n context = "\\\\n".join([doc.page_content for doc in retrieved_docs])<\/p>\n

                                                    # \u6784\u5efaRAG\u63d0\u793a\u8bcd
                                                    \n prompt = f"""\u57fa\u4e8e\u4ee5\u4e0b\u4e8b\u5b9e\u56de\u7b54\u95ee\u9898，\u4ec5\u4f7f\u7528\u63d0\u4f9b\u7684\u4e8b\u5b9e，\u4e0d\u8981\u6dfb\u52a0\u5176\u4ed6\u4fe1\u606f：
                                                    \n\u4e8b\u5b9e：
                                                    \n{context}<\/p>\n

                                                    \u95ee\u9898：{question}
                                                    \n\u56de\u7b54："""<\/p>\n

                                                    # \u751f\u6210\u56de\u590d
                                                    \n inputs = tokenizer(prompt, return_tensors="pt").to(DEVICE)
                                                    \n outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.1)
                                                    \n response = tokenizer.decode(outputs[0], skip_special_tokens=True)<\/p>\n

                                                    return response, retrieved_docs<\/p>\n

                                                    \u4e94\u3001\u8bc4\u6d4b\u7ed3\u679c\u843d\u5730\u4e0e\u6a21\u578b\u4f18\u5316<\/h3>\n

                                                    5.1 \u8bc4\u6d4b\u62a5\u544a\u6a21\u677f<\/h4>\n

                                                    \u7efc\u5408\u8bc4\u6d4b\u62a5\u544a\u751f\u6210\u6d41\u7a0b\u56fe (<\/span>Evaluation Reporting Pipeline)<\/span>
                                                    \n\u2502
                                                    \n\u251c\u2500\u2500 \u3010\u591a\u6e90\u8f93\u5165\u6570\u636e\u3011 (<\/span>Input Data Streams)<\/span>
                                                    \n\u2502 \u251c\u2500\u2500 MMLU \u8bc4\u6d4b\u7ed3\u679c: [<\/span>mmlu_results]<\/span> (<\/span>57<\/span>\u4e2a\u5b66\u79d1\u51c6\u786e\u7387\u5b57\u5178)<\/span>
                                                    \n\u2502 \u251c\u2500\u2500 BBH \u8bc4\u6d4b\u7ed3\u679c: [<\/span>bbh_results]<\/span> (<\/span>23<\/span>\u4e2a\u63a8\u7406\u4efb\u52a1\u5f97\u5206)<\/span>
                                                    \n\u2502 \u2514\u2500\u2500 \u5e7b\u89c9\u8bc4\u6d4b\u7ed3\u679c: [<\/span>hallucination_results]<\/span> (<\/span>FactScore\/\u5e7b\u89c9\u7387)<\/span>
                                                    \n\u2502
                                                    \n\u25bc
                                                    \n[<\/span>1<\/span>. \u667a\u80fd\u5206\u6790\u5f15\u64ce]<\/span> (<\/span>Analytical Engine)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                                    \n\u2502 \u2502
                                                    \n\u251c\u2500\u2500 A. \u6838\u5fc3\u6307\u6807\u805a\u5408 (<\/span>Metric Aggregation)<\/span> \u2502
                                                    \n\u2502 \u251c\u2500\u2500 <<\/span>\u8ba1\u7b97><\/span>: \u5168\u5c40\u5e73\u5747\u5206 (<\/span>Global Average)<\/span> \u2502
                                                    \n\u2502 \u251c\u2500\u2500 <<\/span>\u73af\u5883><\/span>: \u63d0\u53d6 GPU\/CUDA \u4fe1\u606f (<\/span>\u7528\u4e8e\u590d\u73b0\u73af\u5883)<\/span> \u2502
                                                    \n\u2502 \u2514\u2500\u2500 ><\/span> \u4ea7\u51fa: \u57fa\u7840\u4f53\u68c0\u9762\u677f (<\/span>Base Dashboard)<\/span> \u2502
                                                    \n\u2502 \u2502
                                                    \n\u251c\u2500\u2500 B. \u80fd\u529b\u753b\u50cf\u8bca\u65ad (<\/span>Capability Profiling)<\/span> \u2502
                                                    \n\u2502 \u251c\u2500\u2500 <<\/span>\u5f3a\u9879\u8bc6\u522b\u7b97\u6cd5><\/span>: \u5355\u9879\u5206 ><\/span> (<\/span>\u5e73\u5747\u5206 + 5<\/span>%)<\/span> \u2502
                                                    \n\u2502 \u251c\u2500\u2500 <<\/span>\u5f31\u9879\u8bc6\u522b\u7b97\u6cd5><\/span>: \u5355\u9879\u5206 <<\/span> (<\/span>\u5e73\u5747\u5206 – 5<\/span>%)<\/span> \u2502
                                                    \n\u2502 \u2514\u2500\u2500 ><\/span> \u4ea7\u51fa: \u80fd\u529b\u504f\u79d1\u8868 (<\/span>Strengths &<\/span> Weaknesses)<\/span> \u2502
                                                    \n\u2502 \u2502
                                                    \n\u251c\u2500\u2500 C. \u7b56\u7565\u903b\u8f91\u63a8\u6f14 (<\/span>Strategic Reasoning)<\/span> <<\/span>\u2605 \u6838\u5fc3\u51b3\u7b56\u5c42><\/span> \u2502
                                                    \n\u2502 \u251c\u2500\u2500 <<\/span>\u89c4\u5219 1<\/span>><\/span>: FactScore <<\/span> 0.8<\/span> –><\/span> \u89e6\u53d1 "\u5efa\u8bae RAG \u96c6\u6210"<\/span> \u2502
                                                    \n\u2502 \u251c\u2500\u2500 <<\/span>\u89c4\u5219 2<\/span>><\/span>: \u5f31\u9879\u6570\u91cf ><\/span> 5<\/span>\u4e2a –><\/span> \u89e6\u53d1 "\u5efa\u8bae\u4e13\u9879 SFT"<\/span> \u2502
                                                    \n\u2502 \u251c\u2500\u2500 <<\/span>\u89c4\u5219 3<\/span>><\/span>: \u65f6\u6548\u6027\u5e7b\u89c9\u9ad8\u53d1 –><\/span> \u89e6\u53d1 "\u5efa\u8bae\u641c\u7d22\u589e\u5f3a"<\/span> \u2502
                                                    \n\u2502 \u2514\u2500\u2500 ><\/span> \u4ea7\u51fa: \u4f18\u5316\u5efa\u8bae\u6e05\u5355 (<\/span>Actionable Insights)<\/span> \u2502
                                                    \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                                    \n \u2502
                                                    \n \u25bc
                                                    \n[<\/span>2<\/span>. \u53cc\u6a21\u6001\u62a5\u544a\u6784\u5efa]<\/span> (<\/span>Dual-Mode Construction)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                                    \n\u2502 \u2502
                                                    \n\u251c\u2500\u2500 \u8def\u5f84 A: \u673a\u5668\u53ef\u8bfb\u5c42 (<\/span>Machine Readable)<\/span> \u2502
                                                    \n\u2502 \u251c\u2500\u2500 <<\/span>\u683c\u5f0f><\/span>: JSON (<\/span>Nested Dict)<\/span> \u2502
                                                    \n\u2502 \u251c\u2500\u2500 <<\/span>\u7528\u9014><\/span>: \u5b58\u5165 MLOps \u5e73\u53f0，\u7528\u4e8e\u7ed8\u5236\u5386\u53f2\u8d8b\u52bf\u56fe \u2502
                                                    \n\u2502 \u2514\u2500\u2500 ><\/span> \u6587\u4ef6: final_evaluation_report.json \u2502
                                                    \n\u2502 \u2502
                                                    \n\u251c\u2500\u2500 \u8def\u5f84 B: \u4eba\u7c7b\u53ef\u8bfb\u5c42 (<\/span>Human Readable)<\/span> \u2502
                                                    \n\u2502 \u251c\u2500\u2500 <<\/span>\u683c\u5f0f><\/span>: Markdown (<\/span>Structured Text)<\/span> \u2502
                                                    \n\u2502 \u251c\u2500\u2500 <<\/span>\u6e32\u67d3><\/span>: \u81ea\u52a8\u751f\u6210\u201c\u6838\u5fc3\u7ed3\u8bba\u201d\u6458\u8981 (<\/span>Executive Summary)<\/span> \u2502
                                                    \n\u2502 \u2514\u2500\u2500 ><\/span> \u6587\u4ef6: final_evaluation_report.md \u2502
                                                    \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

                                                    2. \u4ee3\u7801\u903b\u8f91\u6df1\u5ea6\u89e3\u6790<\/h5>\n

                                                    \u8fd9\u6bb5\u4ee3\u7801\u5b9e\u9645\u4e0a\u5b9e\u73b0\u4e86\u4e00\u4e2a\u5fae\u578b\u7684\u4e13\u5bb6\u7cfb\u7edf（Expert System），\u5404\u90e8\u5206\u534f\u4f5c\u903b\u8f91\u5982\u4e0b：<\/p>\n

                                                    1. \u52a8\u6001\u9608\u503c\u95e8\u63a7 (Threshold Gating)<\/p>\n

                                                      \n
                                                    • \u4ee3\u7801\u903b\u8f91：mmlu_threshold = mmlu_results["mmlu_average"] + 5<\/li>\n
                                                    • \u534f\u4f5c\u539f\u7406：\u5b83\u4e0d\u4f7f\u7528\u6b7b\u677f\u7684\u56fa\u5b9a\u5206\u6570（\u598260\u5206\u53ca\u683c），\u800c\u662f\u6839\u636e\u6a21\u578b\u81ea\u8eab\u7684\u5e73\u5747\u8868\u73b0\u6765\u52a8\u6001\u5212\u5b9a\u201c\u5f3a\u9879\u201d\u548c\u201c\u5f31\u9879\u201d\u3002<\/li>\n
                                                    • \u4e1a\u52a1\u4ef7\u503c：\u8fd9\u610f\u5473\u7740\u5373\u4f7f\u662f\u4e00\u4e2a 7B \u7684\u5c0f\u6a21\u578b，\u4e5f\u80fd\u627e\u51fa\u5b83\u76f8\u5bf9\u64c5\u957f\u7684\u9886\u57df（\u5982\u201c\u521b\u610f\u5199\u4f5c\u201d），\u800c\u4e0d\u662f\u88ab 70B \u6a21\u578b\u5168\u65b9\u4f4d\u78be\u538b，\u6709\u52a9\u4e8e\u53d1\u73b0\u5c0f\u6a21\u578b\u7684\u5782\u76f4\u5e94\u7528\u6f5c\u529b\u3002<\/li>\n<\/ul>\n

                                                      2. \u5904\u65b9\u751f\u6210\u903b\u8f91 (Prescription Logic)<\/p>\n

                                                        \n
                                                      • \n

                                                        \u4ee3\u7801\u903b\u8f91：<\/p>\n

                                                        if<\/span> report[<\/span>"\u5e7b\u89c9\u8bc4\u4f30"<\/span>]<\/span>[<\/span>"\u5e73\u5747FactScore"<\/span>]<\/span> <<\/span> 0.8<\/span>:<\/span>
                                                        \n report[<\/span>"\u4f18\u5316\u5efa\u8bae"<\/span>]<\/span>.<\/span>append(<\/span>"\u4f18\u5148\u4f18\u5316\u4e8b\u5b9e\u51c6\u786e\u6027…"<\/span>)<\/span><\/p>\n<\/li>\n

                                                      • \n

                                                        \u534f\u4f5c\u539f\u7406：\u8fd9\u662f\u4ee3\u7801\u4e2d\u6700\u5177\u201c\u667a\u80fd\u201d\u7684\u90e8\u5206\u3002\u5b83\u5efa\u7acb\u4e86\u4e00\u5957 \u201c\u75c7\u72b6 -> \u7597\u6cd5\u201d \u7684\u6620\u5c04\u8868：<\/p>\n

                                                          \n
                                                        • \u75c7\u72b6：FactScore \u4f4e \u2192\\\\rightarrow<\/span><\/span>\u2192<\/span><\/span><\/span><\/span><\/span> \u7597\u6cd5：\u5916\u6302 RAG \u77e5\u8bc6\u5e93\u3002<\/li>\n
                                                        • \u75c7\u72b6：MMLU \u5f31\u9879\u592a\u591a \u2192\\\\rightarrow<\/span><\/span>\u2192<\/span><\/span><\/span><\/span><\/span> \u7597\u6cd5：\u8fdb\u884c SFT \u5fae\u8c03\u3002<\/li>\n
                                                        • \u75c7\u72b6：\u65f6\u6548\u6027\u9519\u8bef \u2192\\\\rightarrow<\/span><\/span>\u2192<\/span><\/span><\/span><\/span><\/span> \u7597\u6cd5：\u63a5\u5165 Search API\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

                                                          3. \u53cc\u6a21\u6001\u8f93\u51fa (Dual-Mode Output)<\/p>\n

                                                            \n
                                                          • JSON：\u662f\u4e3a\u4e86\u7ed9 CI\/CD \u6d41\u6c34\u7ebf\u7528\u7684\u3002\u4f60\u53ef\u4ee5\u5199\u4e00\u4e2a\u811a\u672c，\u6bcf\u6b21\u6a21\u578b\u8bad\u7ec3\u5b8c\u81ea\u52a8\u8dd1\u8bc4\u6d4b，\u5982\u679c JSON \u4e2d\u7684 mmlu_average \u4e0b\u964d\u4e86，\u5c31\u81ea\u52a8\u7ec8\u6b62\u90e8\u7f72\u3002<\/li>\n
                                                          • Markdown：\u662f\u4e3a\u4e86\u7ed9\u8001\u677f\/\u5ba2\u6237\u770b\u7684\u3002\u4ee3\u7801\u4e2d\u7cbe\u5fc3\u8bbe\u8ba1\u7684 f-string \u6a21\u677f，\u76f4\u63a5\u751f\u6210\u683c\u5f0f\u6f02\u4eae\u7684\u6587\u6863，\u4e0d\u4ec5\u6709\u6570\u636e，\u8fd8\u6709\u81ea\u52a8\u751f\u6210\u7684\u201c\u6838\u5fc3\u7ed3\u8bba\u201d（\u5982：\u201c\u6a21\u578b\u7efc\u5408\u80fd\u529b\u4f18\u79c0，\u9002\u5408\u751f\u4ea7\u73af\u5883\u201d），\u7701\u53bb\u4e86\u4eba\u5de5\u5199\u62a5\u544a\u7684\u65f6\u95f4\u3002<\/li>\n<\/ul>\n

                                                            """
                                                            \n\u751f\u6210\u5b8c\u6574\u8bc4\u6d4b\u62a5\u544a
                                                            \n"""
                                                            \ndef generate_final_report(mmlu_results, bbh_results, hallucination_results):
                                                            \n """\u751f\u6210\u7efc\u5408\u8bc4\u6d4b\u62a5\u544a"""
                                                            \n report = {
                                                            \n "\u8bc4\u6d4b\u6982\u8981": {
                                                            \n "\u6a21\u578b\u540d\u79f0": MODEL_PATH.split("\/")[-1],
                                                            \n "\u8bc4\u6d4b\u65f6\u95f4": str(datetime.datetime.now()),
                                                            \n "\u8bc4\u6d4b\u73af\u5883": f"{DEVICE} ({torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'CPU'})",
                                                            \n "\u6838\u5fc3\u7ed3\u8bba": ""
                                                            \n },
                                                            \n "\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30": {
                                                            \n "MMLU\u5e73\u5747\u51c6\u786e\u7387": mmlu_results["mmlu_average"],
                                                            \n "BBH\u5e73\u5747\u51c6\u786e\u7387": bbh_results["bbh_average"],
                                                            \n "\u80fd\u529b\u5f3a\u9879": [],
                                                            \n "\u80fd\u529b\u5f31\u9879": []
                                                            \n },
                                                            \n "\u5e7b\u89c9\u8bc4\u4f30": {
                                                            \n "\u5e73\u5747FactScore": hallucination_results["average_factscore"],
                                                            \n "\u5e73\u5747\u5e7b\u89c9\u7387": 1 – hallucination_results["average_factscore"],
                                                            \n "\u9ad8\u5e7b\u89c9\u7c7b\u522b": [],
                                                            \n "\u4f4e\u5e7b\u89c9\u7c7b\u522b": []
                                                            \n },
                                                            \n "\u4f18\u5316\u5efa\u8bae": []
                                                            \n }<\/p>\n

                                                            # \u5206\u6790\u80fd\u529b\u5f3a\u9879\/\u5f31\u9879
                                                            \n mmlu_threshold = mmlu_results["mmlu_average"] + 5
                                                            \n bbh_threshold = bbh_results["bbh_average"] + 5<\/p>\n

                                                            for task, metrics in mmlu_results["task_details"].items():
                                                            \n if metrics["accuracy"] > mmlu_threshold:
                                                            \n report["\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30"]["\u80fd\u529b\u5f3a\u9879"].append(task.replace("mmlu_", ""))
                                                            \n elif metrics["accuracy"] < mmlu_results["mmlu_average"] – 5:
                                                            \n report["\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30"]["\u80fd\u529b\u5f31\u9879"].append(task.replace("mmlu_", ""))<\/p>\n

                                                            # \u5206\u6790\u5e7b\u89c9\u7c7b\u522b
                                                            \n for category, analysis in hallucination_results["category_analysis"].items():
                                                            \n if analysis["average_score"] < 0.7:
                                                            \n report["\u5e7b\u89c9\u8bc4\u4f30"]["\u9ad8\u5e7b\u89c9\u7c7b\u522b"].append(category)
                                                            \n elif analysis["average_score"] > 0.9:
                                                            \n report["\u5e7b\u89c9\u8bc4\u4f30"]["\u4f4e\u5e7b\u89c9\u7c7b\u522b"].append(category)<\/p>\n

                                                            # \u751f\u6210\u4f18\u5316\u5efa\u8bae
                                                            \n if report["\u5e7b\u89c9\u8bc4\u4f30"]["\u5e73\u5747FactScore"] < 0.8:
                                                            \n report["\u4f18\u5316\u5efa\u8bae"].append("\u4f18\u5148\u4f18\u5316\u4e8b\u5b9e\u51c6\u786e\u6027，\u5efa\u8bae\u96c6\u6210RAG\u68c0\u7d22\u589e\u5f3a")
                                                            \n if len(report["\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30"]["\u80fd\u529b\u5f31\u9879"]) > 5:
                                                            \n report["\u4f18\u5316\u5efa\u8bae"].append("\u9488\u5bf9\u8584\u5f31\u5b66\u79d1\u8fdb\u884c\u4e13\u9879\u5fae\u8c03")
                                                            \n if "\u65f6\u6548\u6027\u4e8b\u5b9e" in report["\u5e7b\u89c9\u8bc4\u4f30"]["\u9ad8\u5e7b\u89c9\u7c7b\u522b"]:
                                                            \n report["\u4f18\u5316\u5efa\u8bae"].append("\u589e\u52a0\u5b9e\u65f6\u6570\u636e\u68c0\u7d22\u80fd\u529b，\u907f\u514d\u7f16\u9020\u65f6\u6548\u6027\u4fe1\u606f")<\/p>\n

                                                            # \u751f\u6210\u6838\u5fc3\u7ed3\u8bba
                                                            \n if report["\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30"]["MMLU\u5e73\u5747\u51c6\u786e\u7387"] > 80 and report["\u5e7b\u89c9\u8bc4\u4f30"]["\u5e73\u5747FactScore"] > 0.9:
                                                            \n report["\u8bc4\u6d4b\u6982\u8981"]["\u6838\u5fc3\u7ed3\u8bba"] = "\u6a21\u578b\u7efc\u5408\u80fd\u529b\u4f18\u79c0，\u4e8b\u5b9e\u51c6\u786e\u6027\u9ad8，\u9002\u5408\u751f\u4ea7\u73af\u5883\u4f7f\u7528"
                                                            \n elif report["\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30"]["MMLU\u5e73\u5747\u51c6\u786e\u7387"] > 70 and report["\u5e7b\u89c9\u8bc4\u4f30"]["\u5e73\u5747FactScore"] > 0.8:
                                                            \n report["\u8bc4\u6d4b\u6982\u8981"]["\u6838\u5fc3\u7ed3\u8bba"] = "\u6a21\u578b\u7efc\u5408\u80fd\u529b\u826f\u597d，\u9700\u9488\u5bf9\u6027\u4f18\u5316\u90e8\u5206\u573a\u666f\u7684\u5e7b\u89c9\u95ee\u9898"
                                                            \n else:
                                                            \n report["\u8bc4\u6d4b\u6982\u8981"]["\u6838\u5fc3\u7ed3\u8bba"] = "\u6a21\u578b\u7efc\u5408\u80fd\u529b\u5f85\u63d0\u5347，\u5efa\u8bae\u5148\u8fdb\u884c\u5fae\u8c03\u4f18\u5316"<\/p>\n

                                                            # \u4fdd\u5b58\u62a5\u544a
                                                            \n with open(os.path.join(OUTPUT_DIR, "final_evaluation_report.json"), "w", encoding="utf-8") as f:
                                                            \n json.dump(report, f, ensure_ascii=False, indent=4)<\/p>\n

                                                            # \u751f\u6210markdown\u62a5\u544a
                                                            \n md_report = f"""# \u5927\u6a21\u578b\u7efc\u5408\u8bc4\u6d4b\u62a5\u544a
                                                            \n## \u8bc4\u6d4b\u6982\u8981
                                                            \n– \u6a21\u578b\u540d\u79f0: {report['\u8bc4\u6d4b\u6982\u8981']['\u6a21\u578b\u540d\u79f0']}
                                                            \n– \u8bc4\u6d4b\u65f6\u95f4: {report['\u8bc4\u6d4b\u6982\u8981']['\u8bc4\u6d4b\u65f6\u95f4']}
                                                            \n– \u8bc4\u6d4b\u73af\u5883: {report['\u8bc4\u6d4b\u6982\u8981']['\u8bc4\u6d4b\u73af\u5883']}
                                                            \n– \u6838\u5fc3\u7ed3\u8bba: {report['\u8bc4\u6d4b\u6982\u8981']['\u6838\u5fc3\u7ed3\u8bba']}<\/p>\n

                                                            ## \u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30
                                                            \n### \u6574\u4f53\u8868\u73b0
                                                            \n– MMLU\u5e73\u5747\u51c6\u786e\u7387: {report['\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30']['MMLU\u5e73\u5747\u51c6\u786e\u7387']}%
                                                            \n– BBH\u5e73\u5747\u51c6\u786e\u7387: {report['\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30']['BBH\u5e73\u5747\u51c6\u786e\u7387']}%<\/p>\n

                                                            ### \u80fd\u529b\u5f3a\u9879
                                                            \n{chr(10).join([f"- {item}" for item in report['\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30']['\u80fd\u529b\u5f3a\u9879'][:5]])}<\/p>\n

                                                            ### \u80fd\u529b\u5f31\u9879
                                                            \n{chr(10).join([f"- {item}" for item in report['\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30']['\u80fd\u529b\u5f31\u9879'][:5]])}<\/p>\n

                                                            ## \u5e7b\u89c9\u8bc4\u4f30
                                                            \n### \u6574\u4f53\u8868\u73b0
                                                            \n– \u5e73\u5747FactScore: {report['\u5e7b\u89c9\u8bc4\u4f30']['\u5e73\u5747FactScore']:.4f}
                                                            \n– \u5e73\u5747\u5e7b\u89c9\u7387: {report['\u5e7b\u89c9\u8bc4\u4f30']['\u5e73\u5747\u5e7b\u89c9\u7387']:.4f}<\/p>\n

                                                            ### \u9ad8\u5e7b\u89c9\u7c7b\u522b
                                                            \n{chr(10).join([f"- {item}" for item in report['\u5e7b\u89c9\u8bc4\u4f30']['\u9ad8\u5e7b\u89c9\u7c7b\u522b']])}<\/p>\n

                                                            ### \u4f4e\u5e7b\u89c9\u7c7b\u522b
                                                            \n{chr(10).join([f"- {item}" for item in report['\u5e7b\u89c9\u8bc4\u4f30']['\u4f4e\u5e7b\u89c9\u7c7b\u522b']])}<\/p>\n

                                                            ## \u4f18\u5316\u5efa\u8bae
                                                            \n{chr(10).join([f"- {item}" for item in report['\u4f18\u5316\u5efa\u8bae']])}
                                                            \n"""<\/p>\n

                                                            with open(os.path.join(OUTPUT_DIR, "final_evaluation_report.md"), "w", encoding="utf-8") as f:
                                                            \n f.write(md_report)<\/p>\n

                                                            print("\u7efc\u5408\u8bc4\u6d4b\u62a5\u544a\u5df2\u751f\u6210\u5b8c\u6210！")
                                                            \n return report<\/p>\n

                                                            5.2 \u6a21\u578b\u4f18\u5316\u5b9e\u6218\u5efa\u8bae<\/h4>\n

                                                            \u57fa\u4e8e MMLU（\u80fd\u529b）\u548c FactScore（\u5e7b\u89c9）\u7684\u8bc4\u6d4b\u7ed3\u679c，\u6211\u4eec\u4e0d\u518d\u8fdb\u884c\u76f2\u76ee\u7684\u5168\u91cf\u5fae\u8c03，\u800c\u662f\u91c7\u7528**\u201c\u79bb\u7ebf\u4fee\u6b63 + \u5728\u7ebf\u589e\u5f3a\u201d**\u7684\u53cc\u8f68\u5236\u4f18\u5316\u7b56\u7565\u3002<\/p>\n

                                                            5.2.1 \u4f18\u5316\u7cfb\u7edf\u67b6\u6784\u56fe\u89e3<\/h5>\n

                                                            \u5927\u6a21\u578b\u80fd\u529b\u8fed\u4ee3\u4e0e\u5e7b\u89c9\u4fee\u6b63\u6d41\u7a0b\u56fe (<\/span>Optimization Loop)<\/span>
                                                            \n\u2502
                                                            \n\u251c\u2500\u2500 \u3010\u8f93\u5165：\u8bc4\u6d4b\u8bca\u65ad\u62a5\u544a\u3011 (<\/span>Evaluation Report)<\/span>
                                                            \n\u2502 \u251c\u2500\u2500 \u75c7\u72b6 A: MMLU \u7279\u5b9a\u79d1\u76ee\u4f4e\u5206 (<\/span>\u5982: "\u9ad8\u4e2d\u6570\u5b66"<\/span>:<\/span> 25<\/span>%)<\/span>
                                                            \n\u2502 \u251c\u2500\u2500 \u75c7\u72b6 B: FactScore \u4f4e\u4e14\u4f34\u968f\u7f16\u9020 (<\/span>Hallucination Rate ><\/span> 30<\/span>%)<\/span>
                                                            \n\u2502 \u2514\u2500\u2500 \u75c7\u72b6 C: \u903b\u8f91\u63a8\u7406\u6b65\u9aa4\u6df7\u4e71 (<\/span>BBH CoT \u9519\u8bef)<\/span>
                                                            \n\u2502
                                                            \n\u25bc
                                                            \n[<\/span>1<\/span>. \u79bb\u7ebf\u8bad\u7ec3\u4fee\u6b63\u9636\u6bb5 (<\/span>Offline Optimization)<\/span>]<\/span> <<\/span>\u6743\u91cd\u66f4\u65b0\u5c42><\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                                            \n\u2502 \u2502
                                                            \n\u251c\u2500\u2500 A. \u6570\u636e\u5408\u6210\u5de5\u5382 (<\/span>Data Synthesis)<\/span> \u2502
                                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u8f93\u5165><\/span>: \u8bc4\u6d4b\u4e2d\u7684\u9519\u9898\u96c6 (<\/span>Error Cases)<\/span> \u2502
                                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u8c03\u7528><\/span>: GPT-4o \/ Claude-3.5 \u2502
                                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u52a8\u4f5c><\/span>: \u751f\u6210 CoT \u89e3\u6790 + \u53cd\u4e8b\u5b9e\u4fee\u6b63\u6837\u672c (<\/span>Counterfactual)<\/span> \u2502
                                                            \n\u2502 \u2514\u2500\u2500 ><\/span> \u4ea7\u51fa: preference_dataset.jsonl (<\/span>DPO \u6570\u636e\u96c6)<\/span> \u2502
                                                            \n\u2502 \u2502
                                                            \n\u251c\u2500\u2500 B. \u53c2\u6570\u9ad8\u6548\u5fae\u8c03 (<\/span>Training \/ SFT &<\/span> DPO)<\/span> \u2502
                                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u8bfb\u53d6\u914d\u7f6e><\/span>: adapter_config.json (<\/span>LoRA Rank<\/span>=<\/span>64<\/span>, Alpha<\/span>=<\/span>16<\/span>)<\/span>\u2502
                                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u52a0\u8f7d\u6743\u91cd><\/span>: Base Model Weights \u2502
                                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u6267\u884c\u7b56\u7565><\/span>: \u2502
                                                            \n\u2502 \u2502 \u251c\u2500\u2500 \u77e5\u8bc6\u6ce8\u5165: \u9488\u5bf9 MMLU \u5f31\u9879\u8fdb\u884c\u6301\u7eed\u9884\u8bad\u7ec3 (<\/span>CPT)<\/span> \u2502
                                                            \n\u2502 \u2502 \u2514\u2500\u2500 \u5e7b\u89c9\u6291\u5236: \u4f7f\u7528 DPO (<\/span>Direct Preference Optimization)<\/span> \u2502
                                                            \n\u2502 \u2502 (<\/span>Positive: \u4e8b\u5b9e\u6027\u56de\u590d |<\/span> Negative: \u5e7b\u89c9\u56de\u590d)<\/span> \u2502
                                                            \n\u2502 \u2514\u2500\u2500 ><\/span> \u8f93\u51fa: new_adapter.bin (<\/span>\u4fee\u6b63\u540e\u7684 LoRA \u6743\u91cd)<\/span> \u2502
                                                            \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                                            \n \u2502
                                                            \n \u25bc
                                                            \n[<\/span>2<\/span>. \u5728\u7ebf\u63a8\u7406\u589e\u5f3a\u9636\u6bb5 (<\/span>Online Inference Enhancement)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                                            \n\u2502 \u2502
                                                            \n\u251c\u2500\u2500 A. \u52a8\u6001\u77e5\u8bc6\u6302\u8f7d (<\/span>RAG &<\/span> Tool Use)<\/span> \u2502
                                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u5224\u65ad><\/span>: \u95ee\u9898\u5305\u542b\u9ad8\u65f6\u6548\u6027\/\u957f\u5c3e\u77e5\u8bc6? \u2502
                                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u52a8\u4f5c><\/span>: \u2502
                                                            \n\u2502 \u2502 \u251c\u2500\u2500 \u68c0\u7d22: VectorDB (<\/span>Chroma\/Milvus)<\/span> \u2502
                                                            \n\u2502 \u2502 \u2514\u2500\u2500 \u9a8c\u8bc1: Google Search API (<\/span>Web Grounding)<\/span> \u2502
                                                            \n\u2502 \u2514\u2500\u2500 ><\/span> \u4e0a\u4e0b\u6587: [<\/span>Refined Context]<\/span> + [<\/span>User Query]<\/span> \u2502
                                                            \n\u2502 \u2502
                                                            \n\u251c\u2500\u2500 B. \u89e3\u7801\u7b56\u7565\u5e72\u9884 (<\/span>Decoding Intervention)<\/span> \u2502
                                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u914d\u7f6e\u6587\u4ef6><\/span>: generation_config.json \u2502
                                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u53c2\u6570\u8c03\u4f18><\/span>: \u2502
                                                            \n\u2502 \u2502 \u251c\u2500\u2500 Temperature: 0.1<\/span> (<\/span>\u6536\u655b\u968f\u673a\u6027)<\/span> \u2502
                                                            \n\u2502 \u2502 \u251c\u2500\u2500 Repetition Penalty: 1.1<\/span> (<\/span>\u9632\u6b62\u6b7b\u5faa\u73af)<\/span> \u2502
                                                            \n\u2502 \u2502 \u2514\u2500\u2500 Do_sample: False (<\/span>\u8d2a\u5a6a\u89e3\u7801，\u7528\u4e8e\u4e25\u8c28\u4efb\u52a1)<\/span> \u2502
                                                            \n\u2502 \u2514\u2500\u2500 <<\/span>\u9ad8\u7ea7\u5f15\u5bfc><\/span>: LogitsProcessor (<\/span>\u7981\u6b62\u751f\u6210\u7279\u5b9a\u654f\u611f\u8bcd\/\u5e7b\u89c9\u8bcd)<\/span> \u2502
                                                            \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                                            \n \u2502
                                                            \n \u25bc
                                                            \n[<\/span>3<\/span>. \u5de5\u7a0b\u7ea7\u9632\u5fa1\u62a4\u680f (<\/span>Engineering Guardrails)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                                            \n\u2502 \u2502
                                                            \n\u251c\u2500\u2500 <<\/span>\u540e\u5904\u7406\u6a21\u5757><\/span>: Output Validator \u2502
                                                            \n\u251c\u2500\u2500 <<\/span>\u52a8\u4f5c><\/span>: \u2502
                                                            \n\u2502 \u251c\u2500\u2500 \u4e8b\u5b9e\u4e00\u81f4\u6027\u68c0\u6d4b (<\/span>NLI Model)<\/span>: \u5224\u65ad\u751f\u6210\u5185\u5bb9\u662f\u5426\u8fdd\u80cc\u4e0a\u4e0b\u6587 \u2502
                                                            \n\u2502 \u251c\u2500\u2500 \u62d2\u7edd\u56de\u7b54\u673a\u5236: \u5f53\u7f6e\u4fe1\u5ea6 (<\/span>LogProbs)<\/span> <<\/span> \u9608\u503c\u65f6，\u8f93\u51fa"\u6211\u4e0d\u77e5\u9053"<\/span> \u2502
                                                            \n\u2502 \u2514\u2500\u2500 \u5f15\u7528\u6807\u6ce8: \u5f3a\u5236\u6a21\u578b\u5728\u53e5\u5c3e\u6807\u6ce8 [<\/span>Source ID]<\/span> \u2502
                                                            \n\u2502 \u2502
                                                            \n\u2514\u2500\u2500 ><\/span> \u6700\u7ec8\u4ea4\u4ed8: \u7528\u6237\u7aef\u56de\u590d (<\/span>Safe &<\/span> Accurate Response)<\/span> \u2502
                                                            \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

                                                            5.2.2 \u4f18\u5316\u51b3\u7b56\u77e9\u9635（Decision Matrix）<\/h5>\n

                                                            \u8bc4\u6d4b\u5b8c\u6210\u540e，\u8bf7\u5bf9\u7167\u4e0b\u8868\u9009\u62e9\u6700\u7ecf\u6d4e\u6709\u6548\u7684\u4f18\u5316\u65b9\u6848，\u907f\u514d\u76f2\u76ee\u91cd\u8bad：<\/p>\n\n\u8bc4\u6d4b\u75c7\u72b6\u8bca\u65ad\u5178\u578b\u6307\u6807\u7279\u5f81\u63a8\u8350\u4f18\u5316\u65b9\u6848 (\u6309\u4f18\u5148\u7ea7\u6392\u5e8f)\u9884\u8ba1\u6210\u672c<\/tr>\n\n\n\n\n\n\n
                                                            \u4e25\u91cd\u77e5\u8bc6\u532e\u4e4f<\/td>\nMMLU < 30%, \u4e14\u5bf9\u57fa\u7840\u6982\u5ff5\u4e00\u65e0\u6240\u77e5<\/td>\n1. RAG \u77e5\u8bc6\u5e93\u5916\u6302 2. \u6301\u7eed\u9884\u8bad\u7ec3 (CPT)<\/td>\n\u4f4e (RAG) \u9ad8 (CPT)<\/td>\n<\/tr>\n
                                                            \u4e25\u91cd\u903b\u8f91\u5e7b\u89c9<\/td>\nBBH < 40%, \u63a8\u7406\u6b65\u9aa4\u8df3\u8dc3\u6216\u81ea\u76f8\u77db\u76fe<\/td>\n1. CoT (\u601d\u7ef4\u94fe) \u63d0\u793a\u8bcd\u4f18\u5316 2. SFT \u5fae\u8c03 (\u4f7f\u7528\u9ad8\u8d28\u91cf\u63a8\u7406\u6570\u636e)<\/td>\n\u6781\u4f4e \u4e2d<\/td>\n<\/tr>\n
                                                            \u4e8b\u5b9e\u634f\u9020\/\u5f20\u51a0\u674e\u6234<\/td>\nFactScore < 0.6, \u8bed\u8a00\u901a\u987a\u4f46\u5185\u5bb9\u9519\u8bef<\/td>\n1. RAG + \u4e0a\u4e0b\u6587\u4e25\u683c\u7ea6\u675f 2. DPO \u5bf9\u504f\u597d\u4f18\u5316 (\u6291\u5236\u5e7b\u89c9)<\/td>\n\u4e2d \u4e2d<\/td>\n<\/tr>\n
                                                            \u65f6\u6548\u6027\u9519\u8bef<\/td>\n\u56de\u7b54\u8fc7\u671f\u4fe1\u606f (\u5982"\u82f1\u56fd\u5973\u738b\u8fd8\u5728\u4e16")<\/td>\n1. \u641c\u7d22\u5f15\u64ce\u5de5\u5177\u8c03\u7528 (Tool Use) 2. \u77e5\u8bc6\u7f16\u8f91 (Knowledge Editing, MEMIT)<\/td>\n\u4f4e \u9ad8(\u6280\u672f\u96be\u5ea6\u5927)<\/td>\n<\/tr>\n
                                                            \u6307\u4ee4\u9075\u5faa\u5931\u8d25<\/td>\n\u65e0\u6cd5\u6309\u8981\u6c42\u683c\u5f0f (JSON\/Markdown) \u8f93\u51fa<\/td>\n1. Few-shot Prompting (\u589e\u52a0\u793a\u4f8b) 2. \u7ea6\u675f\u89e3\u7801 (Constrained Decoding)<\/td>\n\u6781\u4f4e \u4f4e<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
                                                            5.2.3 \u5b9e\u6218\u64cd\u4f5c\u6307\u5357：\u5168\u6808\u95ed\u73af\u67b6\u6784<\/h5>\n

                                                            1. \u6570\u636e\u5c42\u9762：\u6784\u5efa\u201c\u53cd\u5e7b\u89c9\u201dDPO \u6570\u636e\u96c6<\/p>\n

                                                              \n
                                                            • \u65b9\u6cd5：\u4e0d\u4ec5\u6536\u96c6\u6b63\u786e\u7684（Chosen），\u8fd8\u8981\u6536\u96c6\u6a21\u578b\u751f\u6210\u7684\u5178\u578b\u5e7b\u89c9（Rejected）\u3002<\/li>\n
                                                            • \u793a\u4f8b：\n
                                                                \n
                                                              • Prompt: \u201c\u4ecb\u7ecd\u91cf\u5b50\u7ea0\u7f20\u3002\u201d<\/li>\n
                                                              • Chosen: \u201c\u91cf\u5b50\u7ea0\u7f20\u662f\u91cf\u5b50\u529b\u5b66\u4e2d\u7684\u4e00\u79cd\u73b0\u8c61\u2026\u201d (\u5f15\u7528\u7ef4\u57fa\u767e\u79d1)<\/li>\n
                                                              • Rejected: \u201c\u91cf\u5b50\u7ea0\u7f20\u662f\u7531\u7231\u56e0\u65af\u5766\u57281990\u5e74\u53d1\u660e\u7684\u2026\u201d (\u5305\u542b\u4e8b\u5b9e\u9519\u8bef)<\/li>\n<\/ul>\n<\/li>\n
                                                              • \u4f5c\u7528：\u901a\u8fc7 DPO \u8bad\u7ec3，\u8ba9\u6a21\u578b\u5b66\u4f1a**\u201c\u4ec0\u4e48\u662f\u4e0d\u8be5\u8bf4\u7684\u201d**\u3002<\/li>\n<\/ul>\n

                                                                2. \u8bad\u7ec3\u5c42\u9762：LoRA + \u77e5\u8bc6\u7f16\u8f91<\/p>\n

                                                                  \n
                                                                • \u4e0d\u8981\u8bd5\u56fe\u901a\u8fc7\u5fae\u8c03\u8ba9\u6a21\u578b\u8bb0\u4f4f\u6240\u6709\u77e5\u8bc6（\u8fd9\u662f\u9884\u8bad\u7ec3\u7684\u4e8b）\u3002<\/li>\n
                                                                • LoRA \u91cd\u70b9：\u8c03\u6574\u6a21\u578b\u7684**\u201c\u8bf4\u8bdd\u65b9\u5f0f\u201d\u548c\u201c\u5f15\u7528\u4e60\u60ef\u201d**\u3002\u4f8b\u5982，\u8bad\u7ec3\u6a21\u578b\u5728\u4e0d\u786e\u5b9a\u65f6\u4e3b\u52a8\u8bf4\u201c\u53ef\u4ee5\u67e5\u8be2\u4e00\u4e0b\u8d44\u6599\u201d，\u800c\u4e0d\u662f\u5f3a\u884c\u7f16\u9020\u3002<\/li>\n<\/ul>\n

                                                                  3. \u63a8\u7406\u5c42\u9762：\u81ea\u6d3d\u6027\u6295\u7968 (Self-Consistency)<\/p>\n

                                                                    \n
                                                                  • \u9488\u5bf9 BBH \u7b49\u63a8\u7406\u4efb\u52a1，\u4e0d\u8981\u53ea\u751f\u6210\u4e00\u6b21\u3002<\/li>\n
                                                                  • \u64cd\u4f5c：\u8bbe\u7f6e temperature=0.7 \u751f\u6210 5 \u6761\u8def\u5f84，\u53d6\u51fa\u73b0\u6b21\u6570\u6700\u591a\u7684\u7b54\u6848（Majority Vote）\u3002\u8fd9\u80fd\u663e\u8457\u63d0\u5347\u6570\u7406\u903b\u8f91\u4efb\u52a1\u7684\u51c6\u786e\u7387（\u901a\u5e38\u63d0\u5347 5-10%）\u3002<\/li>\n<\/ul>\n

                                                                    4. \u5de5\u7a0b\u5c42\u9762：\u7f6e\u4fe1\u5ea6\u622a\u65ad<\/p>\n

                                                                      \n
                                                                    • \u5728 vLLM \u6216 HuggingFace \u751f\u6210\u65f6，\u83b7\u53d6\u6bcf\u4e2a Token \u7684 logprobs\u3002<\/li>\n
                                                                    • \u8ba1\u7b97\u6574\u53e5\u7684\u5e73\u5747\u7f6e\u4fe1\u5ea6（Perplexity）\u3002\u5982\u679c\u56f0\u60d1\u5ea6\u8fc7\u9ad8，\u8bf4\u660e\u6a21\u578b\u5728\u201c\u786c\u7f16\u201d，\u6b64\u65f6\u5de5\u7a0b\u5c42\u76f4\u63a5\u62e6\u622a，\u8fd4\u56de\u515c\u5e95\u8bdd\u672f\u3002<\/li>\n<\/ul>\n

                                                                      \u516d\u3001\u603b\u7ed3\u4e0e\u5c55\u671b<\/h3>\n

                                                                      6.1 \u6838\u5fc3\u6536\u83b7<\/h4>\n
                                                                    • \n

                                                                      \u638c\u63e1\u4e86\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u6d4b\u7684\u5b8c\u6574\u6d41\u7a0b（MMLU\/BBH）<\/p>\n<\/li>\n

                                                                    • \n

                                                                      \u843d\u5730\u4e86\u6a21\u578b\u5e7b\u89c9\u7684\u5b9a\u91cf\u8bc4\u4f30\u65b9\u6cd5（FactScore）<\/p>\n<\/li>\n

                                                                    • \n

                                                                      \u80fd\u591f\u57fa\u4e8e\u8bc4\u6d4b\u7ed3\u679c\u5b9a\u4f4d\u6a21\u578b\u77ed\u677f\u5e76\u5236\u5b9a\u4f18\u5316\u7b56\u7565<\/p>\n<\/li>\n

                                                                    • \n

                                                                      \u6784\u5efa\u4e86\u53ef\u590d\u7528\u7684\u8bc4\u6d4b\u4ee3\u7801\u4f53\u7cfb（\u652f\u63012025\u5e74\u6700\u65b0\u5f00\u6e90\u6a21\u578b）<\/p>\n<\/li>\n

                                                                      6.2 \u672a\u6765\u8d8b\u52bf<\/h4>\n
                                                                    • \n

                                                                      \u8bc4\u6d4b\u96c6\u7684\u52a8\u6001\u66f4\u65b0：\u9002\u914d\u5b9e\u65f6\u6027\u3001\u5730\u57df\u6027\u7684\u8bc4\u6d4b\u9700\u6c42<\/p>\n<\/li>\n

                                                                    • \n

                                                                      \u5e7b\u89c9\u8bc4\u4f30\u7684\u81ea\u52a8\u5316：\u51cf\u5c11\u4eba\u5de5\u5e72\u9884，\u63d0\u5347\u8bc4\u4f30\u6548\u7387<\/p>\n<\/li>\n

                                                                    • \n

                                                                      \u591a\u6a21\u6001\u80fd\u529b\u8bc4\u6d4b：\u8986\u76d6\u6587\u672c\u3001\u56fe\u50cf\u3001\u97f3\u9891\u7684\u7efc\u5408\u8bc4\u4f30<\/p>\n<\/li>\n

                                                                    • \n

                                                                      \u751f\u4ea7\u73af\u5883\u8bc4\u6d4b：\u7ed3\u5408\u5b9e\u9645\u4e1a\u52a1\u573a\u666f\u7684\u6301\u7eed\u8bc4\u6d4b\u4f53\u7cfb<\/p>\n<\/li>\n

                                                                      6.3 \u5de5\u5177\u751f\u6001\u63a8\u8350<\/h4>\n\n\u5de5\u5177\u7528\u9014\u4f18\u52bf<\/tr>\n\n\n\n\n\n\n\n
                                                                      lm-evaluation-harness<\/td>\n\u901a\u7528\u80fd\u529b\u8bc4\u6d4b<\/td>\n\u8986\u76d6\u5e7f\u3001\u6807\u51c6\u5316\u3001\u6613\u6269\u5c55<\/td>\n<\/tr>\n
                                                                      FactScore<\/td>\n\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u4f30<\/td>\n\u4e8b\u5b9e\u51c6\u786e\u6027\u9ad8\u3001\u652f\u6301\u591a\u8bed\u8a00<\/td>\n<\/tr>\n
                                                                      LangSmith<\/td>\n\u8bc4\u6d4b\u7ed3\u679c\u5206\u6790<\/td>\n\u53ef\u89c6\u5316\u3001\u53ef\u8ffd\u6eaf\u3001\u56e2\u961f\u534f\u4f5c<\/td>\n<\/tr>\n
                                                                      vLLM<\/td>\n\u9ad8\u6027\u80fd\u63a8\u7406<\/td>\n\u5927\u5e45\u63d0\u5347\u8bc4\u6d4b\u6548\u7387<\/td>\n<\/tr>\n
                                                                      HuggingFace Evaluate<\/td>\n\u81ea\u5b9a\u4e49\u8bc4\u6d4b<\/td>\n\u7075\u6d3b\u3001\u53ef\u5b9a\u5236\u5316\u6307\u6807<\/td>\n<\/tr>\n
                                                                      \u901a\u8fc7\u672c\u6587\u7684\u5b9e\u6218\u6307\u5357，\u5f00\u53d1\u8005\u53ef\u4ee5\u7cfb\u7edf\u5316\u5730\u8bc4\u4f30\u5927\u6a21\u578b\u7684\u7efc\u5408\u80fd\u529b，\u5b9a\u91cf\u5206\u6790\u5e7b\u89c9\u95ee\u9898，\u5e76\u57fa\u4e8e\u8bc4\u6d4b\u7ed3\u679c\u8fdb\u884c\u7cbe\u51c6\u4f18\u5316，\u4e3a\u5927\u6a21\u578b\u7684\u751f\u4ea7\u7ea7\u5e94\u7528\u63d0\u4f9b\u53ef\u9760\u7684\u8bc4\u4f30\u4f9d\u636e\u3002<\/td>\n<\/td>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n","protected":false},"excerpt":{"rendered":"

                                                                      \u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30\u4e0e\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u5b9e\u6218（\u57fa\u4e8eMMLU\/BBH\u8bc4\u6d4b\u96c6）
                                                                      \n\u6587\u6863\u6982\u8ff0
                                                                      \n\u6587\u7ae0\u6838\u5fc3\u4ef7\u503c\u7cfb\u7edf\u638c\u63e1\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u6d4b\u4f53\u7cfb（MMLU\/BBH \u6838\u5fc3\u8bc4\u6d4b\u96c6）\u843d\u5730\u5f00\u6e90\u6a21\u578b\u7684\u81ea\u52a8\u5316\u8bc4\u6d4b\u6d41\u7a0b（\u73af\u5883\u642d\u5efa\u2192\u6570\u636e\u96c6\u52a0\u8f7d\u2192\u8bc4\u6d4b\u6267\u884c\u2192\u7ed3\u679c\u5206\u6790）\u638c\u63e1\u6a21\u578b\u5e7b\u89c9（Hallucination）\u7684\u5b9a\u91cf\u8bc4\u4f30\u65b9\u6cd5\u4e0e\u843d\u5730\u5de5\u5177\u57fa\u4e8e\u8bc4\u6d4b\u7ed3\u679c\u7684\u6a21\u578b\u4f18\u5316\u65b9\u5411\u4e0e\u5b9e\u6218\u6280\u5de7\u9002\u914d2025\u5e74\u6700\u65b0\u5f00\u6e90\u751f\u6001\u7684\u5b8c\u6574\u53ef\u590d\u7528<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[50,3684,3829,51],"topic":[],"class_list":["post-70627","post","type-post","status-publish","format-standard","hentry","category-server","tag-50","tag-3684","tag-3829","tag-51"],"yoast_head":"\n\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30\u4e0e\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u5b9e\u6218(\u57fa\u4e8eMMLU\u548cBBH\u8bc4\u6d4b\u96c6) - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.wsisp.com\/helps\/70627.html\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30\u4e0e\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u5b9e\u6218(\u57fa\u4e8eMMLU\u548cBBH\u8bc4\u6d4b\u96c6) - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"og:description\" content=\"\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30\u4e0e\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u5b9e\u6218（\u57fa\u4e8eMMLU\/BBH\u8bc4\u6d4b\u96c6） \u6587\u6863\u6982\u8ff0 \u6587\u7ae0\u6838\u5fc3\u4ef7\u503c\u7cfb\u7edf\u638c\u63e1\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u6d4b\u4f53\u7cfb（MMLU\/BBH \u6838\u5fc3\u8bc4\u6d4b\u96c6）\u843d\u5730\u5f00\u6e90\u6a21\u578b\u7684\u81ea\u52a8\u5316\u8bc4\u6d4b\u6d41\u7a0b（\u73af\u5883\u642d\u5efa\u2192\u6570\u636e\u96c6\u52a0\u8f7d\u2192\u8bc4\u6d4b\u6267\u884c\u2192\u7ed3\u679c\u5206\u6790）\u638c\u63e1\u6a21\u578b\u5e7b\u89c9（Hallucination）\u7684\u5b9a\u91cf\u8bc4\u4f30\u65b9\u6cd5\u4e0e\u843d\u5730\u5de5\u5177\u57fa\u4e8e\u8bc4\u6d4b\u7ed3\u679c\u7684\u6a21\u578b\u4f18\u5316\u65b9\u5411\u4e0e\u5b9e\u6218\u6280\u5de7\u9002\u914d2025\u5e74\u6700\u65b0\u5f00\u6e90\u751f\u6001\u7684\u5b8c\u6574\u53ef\u590d\u7528\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.wsisp.com\/helps\/70627.html\" \/>\n<meta property=\"og:site_name\" content=\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"article:published_time\" content=\"2026-02-01T21:49:08+00:00\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"25 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/70627.html\",\"url\":\"https:\/\/www.wsisp.com\/helps\/70627.html\",\"name\":\"\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30\u4e0e\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u5b9e\u6218(\u57fa\u4e8eMMLU\u548cBBH\u8bc4\u6d4b\u96c6) - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"isPartOf\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\"},\"datePublished\":\"2026-02-01T21:49:08+00:00\",\"dateModified\":\"2026-02-01T21:49:08+00:00\",\"author\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/70627.html#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.wsisp.com\/helps\/70627.html\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/70627.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.wsisp.com\/helps\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30\u4e0e\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u5b9e\u6218(\u57fa\u4e8eMMLU\u548cBBH\u8bc4\u6d4b\u96c6)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\",\"url\":\"https:\/\/www.wsisp.com\/helps\/\",\"name\":\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"description\":\"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"contentUrl\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"caption\":\"admin\"},\"sameAs\":[\"http:\/\/wp.wsisp.com\"],\"url\":\"https:\/\/www.wsisp.com\/helps\/author\/admin\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30\u4e0e\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u5b9e\u6218(\u57fa\u4e8eMMLU\u548cBBH\u8bc4\u6d4b\u96c6) - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.wsisp.com\/helps\/70627.html","og_locale":"zh_CN","og_type":"article","og_title":"\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30\u4e0e\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u5b9e\u6218(\u57fa\u4e8eMMLU\u548cBBH\u8bc4\u6d4b\u96c6) - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","og_description":"\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30\u4e0e\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u5b9e\u6218（\u57fa\u4e8eMMLU\/BBH\u8bc4\u6d4b\u96c6） \u6587\u6863\u6982\u8ff0 \u6587\u7ae0\u6838\u5fc3\u4ef7\u503c\u7cfb\u7edf\u638c\u63e1\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u6d4b\u4f53\u7cfb（MMLU\/BBH \u6838\u5fc3\u8bc4\u6d4b\u96c6）\u843d\u5730\u5f00\u6e90\u6a21\u578b\u7684\u81ea\u52a8\u5316\u8bc4\u6d4b\u6d41\u7a0b（\u73af\u5883\u642d\u5efa\u2192\u6570\u636e\u96c6\u52a0\u8f7d\u2192\u8bc4\u6d4b\u6267\u884c\u2192\u7ed3\u679c\u5206\u6790）\u638c\u63e1\u6a21\u578b\u5e7b\u89c9（Hallucination）\u7684\u5b9a\u91cf\u8bc4\u4f30\u65b9\u6cd5\u4e0e\u843d\u5730\u5de5\u5177\u57fa\u4e8e\u8bc4\u6d4b\u7ed3\u679c\u7684\u6a21\u578b\u4f18\u5316\u65b9\u5411\u4e0e\u5b9e\u6218\u6280\u5de7\u9002\u914d2025\u5e74\u6700\u65b0\u5f00\u6e90\u751f\u6001\u7684\u5b8c\u6574\u53ef\u590d\u7528","og_url":"https:\/\/www.wsisp.com\/helps\/70627.html","og_site_name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","article_published_time":"2026-02-01T21:49:08+00:00","author":"admin","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"admin","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"25 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.wsisp.com\/helps\/70627.html","url":"https:\/\/www.wsisp.com\/helps\/70627.html","name":"\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30\u4e0e\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u5b9e\u6218(\u57fa\u4e8eMMLU\u548cBBH\u8bc4\u6d4b\u96c6) - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","isPartOf":{"@id":"https:\/\/www.wsisp.com\/helps\/#website"},"datePublished":"2026-02-01T21:49:08+00:00","dateModified":"2026-02-01T21:49:08+00:00","author":{"@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41"},"breadcrumb":{"@id":"https:\/\/www.wsisp.com\/helps\/70627.html#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.wsisp.com\/helps\/70627.html"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.wsisp.com\/helps\/70627.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.wsisp.com\/helps"},{"@type":"ListItem","position":2,"name":"\u5927\u6a21\u578b\u591a\u7ef4\u5ea6\u80fd\u529b\u8bc4\u4f30\u4e0e\u5e7b\u89c9\u5b9a\u91cf\u8bc4\u6d4b\u5b9e\u6218(\u57fa\u4e8eMMLU\u548cBBH\u8bc4\u6d4b\u96c6)"}]},{"@type":"WebSite","@id":"https:\/\/www.wsisp.com\/helps\/#website","url":"https:\/\/www.wsisp.com\/helps\/","name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","description":"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41","name":"admin","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/","url":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","contentUrl":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","caption":"admin"},"sameAs":["http:\/\/wp.wsisp.com"],"url":"https:\/\/www.wsisp.com\/helps\/author\/admin"}]}},"_links":{"self":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/70627","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/comments?post=70627"}],"version-history":[{"count":0,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/70627\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media?parent=70627"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/categories?post=70627"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/tags?post=70627"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/topic?post=70627"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}