{"id":70070,"date":"2026-02-01T14:17:07","date_gmt":"2026-02-01T06:17:07","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/70070.html"},"modified":"2026-02-01T14:17:07","modified_gmt":"2026-02-01T06:17:07","slug":"%e5%a4%a7%e6%a8%a1%e5%9e%8b%e6%8e%a8%e7%90%86%e5%8a%a0%e9%80%9f%e6%a0%b8%e5%bf%83%e6%8a%80%e6%9c%af%e5%ae%9e%e6%88%98%ef%bc%9akv-cache%e3%80%81%e9%87%8f%e5%8c%96%e3%80%81%e6%a8%a1%e5%9e%8b%e8%92%b8","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/70070.html","title":{"rendered":"\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6838\u5fc3\u6280\u672f\u5b9e\u6218\uff1aKV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f\uff08\u9644\u6700\u65b0\u5f00\u6e90\u4ee3\u7801\uff09"},"content":{"rendered":"

\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6838\u5fc3\u6280\u672f\u5b9e\u6218：KV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f（\u9644\u6700\u65b0\u5f00\u6e90\u4ee3\u7801）<\/h2>\n

\u6587\u6863\u6982\u8ff0<\/h3>\n

\u6587\u7ae0\u6838\u5fc3\u4ef7\u503c<\/h4>\n
  • \n

    \u6df1\u5ea6\u89e3\u6790\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u4e09\u5927\u6838\u5fc3\u6280\u672f（KV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f）\u7684\u5e95\u5c42\u539f\u7406<\/p>\n<\/li>\n

  • \n

    \u57fa\u4e8e2025\u5e74\u6700\u65b0\u5f00\u6e90\u751f\u6001（vLLM 0.5.0\u3001AutoGPTQ 0.7.1\u3001Transformers 4.41+）\u5b9e\u73b0\u4ee3\u7801\u843d\u5730<\/p>\n<\/li>\n

  • \n

    \u9488\u5bf9\u4e0d\u540c\u573a\u666f（\u672c\u5730\u90e8\u7f72\u3001API\u670d\u52a1\u3001\u8fb9\u7f18\u8bbe\u5907）\u7ed9\u51fa\u6280\u672f\u9009\u578b\u4e0e\u6027\u80fd\u4f18\u5316\u6307\u5357<\/p>\n<\/li>\n

  • \n

    \u63d0\u4f9b\u53ef\u76f4\u63a5\u590d\u7528\u7684\u751f\u4ea7\u7ea7\u4ee3\u7801（\u57fa\u4e8eQwen2.5\u3001Llama3\u7b49\u4e3b\u6d41\u5f00\u6e90\u6a21\u578b）<\/p>\n<\/li>\n

  • \n

    \u91cf\u5316\u5404\u6280\u672f\u7684\u52a0\u901f\u6548\u679c\u3001\u663e\u5b58\u5360\u7528\u3001\u7cbe\u5ea6\u635f\u5931\u5bf9\u6bd4，\u6307\u5bfc\u5b9e\u9645\u843d\u5730\u51b3\u7b56<\/p>\n<\/li>\n

    \u5b66\u4e60\u76ee\u6807<\/h4>\n
  • \n

    \u7406\u89e3KV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f\u7684\u6838\u5fc3\u539f\u7406\u4e0e\u9002\u7528\u8fb9\u754c<\/p>\n<\/li>\n

  • \n

    \u638c\u63e1\u57fa\u4e8evLLM\u5b9e\u73b0KV Cache\u4f18\u5316\u7684\u5168\u6d41\u7a0b\u4ee3\u7801<\/p>\n<\/li>\n

  • \n

    \u7cbe\u901aGPTQ\/AWQ\/INT4\/INT8\u91cf\u5316\u7684\u5b9e\u6218\u90e8\u7f72（\u517c\u5bb9\u6700\u65b0\u5f00\u6e90\u6a21\u578b）<\/p>\n<\/li>\n

  • \n

    \u5b66\u4f1a\u4ece\u5927\u6a21\u578b（7B）\u84b8\u998f\u51fa\u8f7b\u91cf\u6a21\u578b（1.8B）\u7684\u5b8c\u6574\u6d41\u7a0b<\/p>\n<\/li>\n

  • \n

    \u80fd\u591f\u6839\u636e\u4e1a\u52a1\u573a\u666f\u9009\u62e9\u6700\u4f18\u7684\u63a8\u7406\u52a0\u901f\u7ec4\u5408\u65b9\u6848<\/p>\n<\/li>\n

  • \n

    \u7406\u89e3\u63a8\u7406\u52a0\u901f\u4e2d\u7684\u7cbe\u5ea6-\u6027\u80fd\u5e73\u8861\u7b56\u7565<\/p>\n<\/li>\n

    \u4e00\u3001\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6280\u672f\u6982\u8ff0<\/h3>\n

    1.1 \u63a8\u7406\u52a0\u901f\u7684\u6838\u5fc3\u75db\u70b9<\/h4>\n

    \u5927\u6a21\u578b（\u5982 7B\/14B\/70B）\u5728\u5b9e\u9645\u751f\u4ea7\u73af\u5883\u7684\u90e8\u7f72\u4e2d，\u9762\u4e34\u7740\u4ece\u786c\u4ef6\u8d44\u6e90\u5230\u7528\u6237\u4f53\u9a8c\u7684\u591a\u91cd\u6311\u6218\u3002\u8fd9\u4e9b\u75db\u70b9\u4e3b\u8981\u6e90\u4e8e Transformer \u67b6\u6784\u5728\u5927\u89c4\u6a21\u53c2\u6570\u4e0b\u7684\u8ba1\u7b97\u7279\u6027\u7684\u74f6\u9888\u3002<\/p>\n

    1.1.1 \u663e\u5b58\u5360\u7528\u9ad8 (Memory Capacity Bottleneck)<\/h5>\n

    \u5927\u6a21\u578b\u7684\u663e\u5b58\u5360\u7528\u4e0d\u4ec5\u4ec5\u662f\u201c\u6743\u91cd\u6587\u4ef6\u201d\u7684\u5927\u5c0f，\u5b83\u7531\u4ee5\u4e0b\u4e09\u90e8\u5206\u7ec4\u6210：<\/p>\n

      \n
    • \n

      \u6a21\u578b\u6743\u91cd\u5360\u7528：\u4ee5 FP16 \u7cbe\u5ea6\u4e3a\u4f8b，\u6bcf\u4e2a\u53c2\u6570\u5360\u636e 2 \u5b57\u8282（Bytes）\u3002<\/p>\n

        \n
      • \n

        7B \u6a21\u578b：\u7ea6 <\/p>\n

        7<\/p>\n

        \u00d7<\/p>\n

        10<\/p>\n

        9<\/p>\n

        \u00d7<\/p>\n

        2<\/p>\n

        =<\/p>\n

        14<\/p>\n

        \u00a0GB<\/p>\n

        7 \\\\times 10^9 \\\\times 2 = 14\\\\text{ GB} <\/p>\n

        <\/span><\/span>7<\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>1<\/span>0<\/span><\/span>9<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>2<\/span><\/span>=<\/span><\/span><\/span><\/span>14<\/span>\u00a0GB<\/span><\/span><\/span><\/span><\/span><\/span><\/span> \u3002<\/p>\n<\/li>\n

      • \n

        70B \u6a21\u578b：\u7ea6 <\/p>\n

        140<\/p>\n

        \u00a0GB<\/p>\n

        140\\\\text{ GB} <\/p>\n

        <\/span><\/span>140<\/span>\u00a0GB<\/span><\/span><\/span><\/span><\/span><\/span><\/span> ，\u8fd9\u8d85\u51fa\u4e86\u5355\u5f20\u9876\u7ea7\u663e\u5361（\u5982 A100 80GB）\u7684\u627f\u8f7d\u80fd\u529b\u3002<\/p>\n<\/li>\n<\/ul>\n<\/li>\n

      • \n

        KV Cache \u52a8\u6001\u5360\u7528：\u5728\u591a\u8f6e\u5bf9\u8bdd\u4e2d，\u4e3a\u4e86\u907f\u514d\u91cd\u590d\u8ba1\u7b97，\u7cfb\u7edf\u4f1a\u7f13\u5b58\u5386\u53f2\u7684 Key \u548c Value \u5411\u91cf\u3002\u968f\u7740\u4e0a\u4e0b\u6587（Context）\u957f\u5ea6\u589e\u52a0，\u663e\u5b58\u5360\u7528\u5448\u7ebf\u6027\u589e\u957f\u3002\u5bf9\u4e8e\u957f\u6587\u672c\u5e94\u7528，KV Cache \u5f80\u5f80\u4f1a\u6210\u4e3a\u5bfc\u81f4\u663e\u5b58\u6ea2\u51fa（OOM）\u7684\u4e3b\u8981\u539f\u56e0\u3002<\/p>\n<\/li>\n

      • \n

        \u6fc0\u6d3b\u503c\u4e0e\u4e34\u65f6\u8ba1\u7b97\u7f13\u51b2\u533a：\u63a8\u7406\u8fc7\u7a0b\u4e2d\u7684\u4e2d\u95f4\u5c42\u8f93\u51fa\u4e5f\u9700\u5360\u636e\u6570 GB \u7a7a\u95f4\u3002<\/p>\n<\/li>\n<\/ul>\n

        1.1.2 \u63a8\u7406\u901f\u5ea6\u6162 (Computation & Memory Bandwidth Bottleneck)<\/h5>\n

        \u63a8\u7406\u901f\u5ea6\u901a\u5e38\u53d7\u9650\u4e8e\u4e24\u4e2a\u622a\u7136\u4e0d\u540c\u7684\u9636\u6bb5：<\/p>\n

          \n
        • \u9884\u586b\u5145\u9636\u6bb5 (Prefill Phase)：\u5904\u7406\u7528\u6237\u8f93\u5165\u7684 Prompt\u3002\u8fd9\u662f\u8ba1\u7b97\u5bc6\u96c6\u578b\u7684，\u6a21\u578b\u9700\u8981\u4e00\u6b21\u6027\u5e76\u884c\u5904\u7406\u6240\u6709\u8f93\u5165 Token，\u8ba1\u7b97\u91cf\u5de8\u5927\u3002<\/li>\n
        • \u89e3\u7801\u9636\u6bb5 (Decoding Phase)：\u9010\u4e2a\u751f\u6210 Token\u3002\u8fd9\u662f\u5178\u578b\u7684**\u8bbf\u5b58\u5bc6\u96c6\u578b（Memory Bound）**\u8fc7\u7a0b\u3002\u6bcf\u751f\u6210\u4e00\u4e2a Token，\u90fd\u9700\u8981\u5c06\u6570\u5341 GB \u7684\u6743\u91cd\u4ece\u663e\u5b58\u8bfb\u53d6\u5230\u8ba1\u7b97\u6838\u5fc3\u4e2d\u3002\u7531\u4e8e\u201c\u8bfb\u6743\u91cd\u201d\u7684\u901f\u5ea6\u8fdc\u6162\u4e8e\u201c\u8ba1\u7b97\u201d\u901f\u5ea6，\u5bfc\u81f4\u751f\u6210\u901f\u5ea6\u53d7\u9650，\u5355\u8f6e\u54cd\u5e94\u5f80\u5f80\u8d85\u8fc7 5 \u79d2，\u96be\u4ee5\u5b9e\u73b0\u6d41\u7545\u7684\u5b9e\u65f6\u4ea4\u4e92\u3002<\/li>\n<\/ul>\n
          1.1.3 \u90e8\u7f72\u6210\u672c\u4e0e\u8fb9\u7f18\u5316\u96be\u9898 (Deployment & Edge Scaling)<\/h5>\n
            \n
          • \u9ad8\u89c4\u683c\u786c\u4ef6\u4f9d\u8d56：\u4e3a\u4e86\u8fd0\u884c 70B \u6a21\u578b，\u4f01\u4e1a\u5f80\u5f80\u9700\u8981\u914d\u7f6e 8 \u5361 A100\/H100 \u7ec4\u6210\u7684\u8ba1\u7b97\u96c6\u7fa4，\u5355\u53f0\u670d\u52a1\u5668\u6210\u672c\u9ad8\u8fbe\u767e\u4e07\u7ea7\u4eba\u6c11\u5e01\u3002<\/li>\n
          • \u80fd\u6548\u6bd4\u6781\u4f4e：\u9ad8\u6027\u80fd GPU \u529f\u8017\u5de8\u5927，\u5355\u5361\u5cf0\u503c\u53ef\u8fbe 300W-700W，\u5bf9\u4e8e\u624b\u673a\u3001\u5e73\u677f\u6216\u5de5\u4e1a\u7269\u8054\u7f51（IoT）\u7b49\u8fb9\u7f18\u8bbe\u5907，\u9ad8\u6602\u7684\u80fd\u8017\u548c\u53d1\u70ed\u91cf\u4f7f\u5176\u65e0\u6cd5\u76f4\u63a5\u8fd0\u884c\u539f\u59cb\u5927\u6a21\u578b\u3002<\/li>\n
          • \u5e26\u5bbd\u74f6\u9888：\u5728\u516c\u6709\u4e91 API \u670d\u52a1\u4e2d，\u9ad8\u5e76\u53d1\u8bf7\u6c42\u4e0b\u663e\u5b58\u5e26\u5bbd\u7684\u4e89\u62a2\u4f1a\u5bfc\u81f4\u541e\u5410\u91cf\u6025\u5267\u4e0b\u964d，\u8fdb\u4e00\u6b65\u63a8\u9ad8\u4e86\u5355\u6b21\u8c03\u7528\u7684\u6210\u672c\u3002<\/li>\n<\/ul>\n

            1.2 \u4e09\u5927\u6838\u5fc3\u52a0\u901f\u6280\u672f\u5bf9\u6bd4<\/h4>\n\n\u6280\u672f\u7c7b\u578b\u6838\u5fc3\u539f\u7406\u52a0\u901f\u6548\u679c\u663e\u5b58\u8282\u7701\u7cbe\u5ea6\u635f\u5931\u9002\u7528\u573a\u666f<\/tr>\n\n\n\n\n
            KV Cache<\/td>\n\u7f13\u5b58\u6ce8\u610f\u529b\u673a\u5236\u7684Key\/Value\u77e9\u9635，\u907f\u514d\u91cd\u590d\u8ba1\u7b97<\/td>\n2-5\u500d（\u957f\u6587\u672c\u5bf9\u8bdd）<\/td>\n10-20%<\/td>\n\u65e0<\/td>\n\u591a\u8f6e\u5bf9\u8bdd\u3001\u957f\u6587\u672c\u751f\u6210<\/td>\n<\/tr>\n
            \u91cf\u5316（INT4\/INT8\/GPTQ）<\/td>\n\u5c06\u6a21\u578b\u6743\u91cd\u4eceFP16\/FP32\u964d\u81f3\u4f4e\u7cbe\u5ea6（INT4\/INT8）<\/td>\n1.5-3\u500d<\/td>\n50-70%<\/td>\n\u8f7b\u5fae（<2%）<\/td>\n\u663e\u5b58\u53d7\u9650\u573a\u666f\u3001\u6279\u91cf\u63a8\u7406<\/td>\n<\/tr>\n
            \u6a21\u578b\u84b8\u998f<\/td>\n\u7528\u5927\u6a21\u578b（\u6559\u5e08）\u6307\u5bfc\u5c0f\u6a21\u578b（\u5b66\u751f）\u5b66\u4e60，\u4fdd\u7559\u6838\u5fc3\u80fd\u529b<\/td>\n3-10\u500d<\/td>\n70-90%<\/td>\n\u53ef\u63a7（<5%）<\/td>\n\u8fb9\u7f18\u8bbe\u5907\u3001\u9ad8\u5e76\u53d1API\u670d\u52a1<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

            1.3 \u6280\u672f\u9009\u578b\u539f\u5219<\/h4>\n

            \u63a8\u7406\u52a0\u901f\u5e76\u975e\u5355\u4e00\u6280\u672f\u7684\u5806\u53e0，\u800c\u662f\u57fa\u4e8e\u786c\u4ef6\u74f6\u9888\u4e0e\u4e1a\u52a1\u6307\u6807\u7684\u6743\u8861\u6e38\u620f\u3002\u4ee5\u4e0b\u662f\u57fa\u4e8e\u663e\u5b58（Memory）\u3001\u5ef6\u8fdf（Latency）\u548c\u541e\u5410\u91cf（Throughput）\u7684\u7cfb\u7edf\u6027\u9009\u578b\u7b56\u7565\u3002<\/p>\n

            1.3.1 \u6838\u5fc3\u51b3\u7b56\u6811 (Decision Tree)<\/h5>\n

            \u6211\u4eec\u5c06\u9009\u578b\u903b\u8f91\u53ef\u89c6\u5316\u4e3a\u4e00\u68f5\u51b3\u7b56\u6811，\u5e2e\u52a9\u5f00\u53d1\u8005\u5feb\u901f\u5b9a\u4f4d\u6700\u4f18\u65b9\u6848：<\/p>\n

            1.3.2 \u573a\u666f\u5316\u9009\u578b\u7b56\u7565 (Scenario-Based Strategy)<\/h5>\n

            \u6211\u4eec\u6839\u636e\u6700\u5e38\u89c1\u7684\u8d44\u6e90\u74f6\u9888，\u5c06\u9009\u578b\u539f\u5219\u7ec6\u5316\u4e3a\u4ee5\u4e0b\u4e09\u4e2a\u7ef4\u5ea6：<\/p>\n

            1. \u663e\u5b58\u53d7\u9650\u573a\u666f (Memory-Bound)<\/p>\n

              \n
            • \u75db\u70b9：\u6a21\u578b\u52a0\u8f7d\u540e\u663e\u5b58\u5269\u4f59\u4e0d\u8db3 20%，\u6216\u9891\u7e41\u89e6\u53d1 OOM（Out of Memory）\u3002<\/li>\n
            • \u9996\u9009\u7b56\u7565：\u91cf\u5316 (Quantization) \u662f\u552f\u4e00\u80fd\u663e\u8457\u964d\u4f4e\u9759\u6001\u663e\u5b58\u7684\u6280\u672f\u3002\n
                \n
              • \u8f7b\u5ea6\u4e0d\u8db3：\u4f7f\u7528 INT8 \u91cf\u5316（\u5982 load_in_8bit），\u8282\u7701\u7ea6 50% \u663e\u5b58，\u7cbe\u5ea6\u51e0\u4e4e\u65e0\u635f\u3002<\/li>\n
              • \u4e25\u91cd\u4e0d\u8db3：\u4f7f\u7528 INT4\/GPTQ \u91cf\u5316，\u8282\u7701\u7ea6 70% \u663e\u5b58，\u9002\u5408 7B \u6a21\u578b\u8dd1\u5728 8GB \u663e\u5b58\u7684\u6d88\u8d39\u7ea7\u663e\u5361\u4e0a\u3002<\/li>\n
              • \u8865\u5145\u624b\u6bb5：\u542f\u7528 vLLM \u7684 swap_space，\u5c06\u90e8\u5206 KV Cache \u5378\u8f7d\u5230\u5185\u5b58\u4e2d，\u9632\u6b62\u7a81\u53d1 OOM\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

                2. \u5ef6\u8fdf\/\u54cd\u5e94\u901f\u5ea6\u654f\u611f\u573a\u666f (Latency-Sensitive)<\/p>\n

                  \n
                • \u75db\u70b9：\u9996\u5b57\u751f\u6210\u6162（TTFT \u9ad8），\u6216\u6bcf\u79d2\u751f\u6210 Token \u6570（TPS）\u4f4e，\u7528\u6237\u7b49\u5f85\u611f\u5f3a\u3002<\/li>\n
                • \u9996\u9009\u7b56\u7565：KV Cache + \u7b97\u5b50\u878d\u5408\u3002\n
                    \n
                  • \u957f\u6587\u672c\/\u591a\u8f6e\u5bf9\u8bdd：\u5fc5\u987b\u5f00\u542f KV Cache，\u907f\u514d\u968f\u4e0a\u4e0b\u6587\u589e\u957f\u800c\u5bfc\u81f4\u7684\u6307\u6570\u7ea7\u8ba1\u7b97\u91cf\u589e\u52a0\u3002<\/li>\n
                  • \u6781\u81f4\u901f\u5ea6：\u4f7f\u7528 INT4 \u91cf\u5316\u3002\u867d\u7136\u91cf\u5316\u4e3b\u8981\u7701\u663e\u5b58，\u4f46\u7531\u4e8e\u51cf\u5c11\u4e86\u4ece\u663e\u5b58\u8bfb\u53d6\u6743\u91cd\u7684\u6570\u636e\u91cf（Memory Access），\u5728\u5e26\u5bbd\u53d7\u9650\u7684 GPU \u4e0a\u80fd\u663e\u8457\u63d0\u5347\u63a8\u7406\u901f\u5ea6（2-3\u500d）\u3002<\/li>\n
                  • \u63a8\u8350\u7ec4\u5408：KV Cache + INT4 GPTQ + FlashAttention-2（vLLM \u9ed8\u8ba4\u96c6\u6210）\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

                    3. \u8fb9\u7f18\u7aef\/\u4f4e\u7b97\u529b\u573a\u666f (Edge\/Low-Resource)<\/p>\n

                      \n
                    • \u75db\u70b9：\u8bbe\u5907\u7b97\u529b\u5f31（\u5982 Jetson Orin\u3001\u624b\u673a NPU），\u65e0\u6cd5\u8fd0\u884c\u6807\u51c6 7B \u6a21\u578b\u3002<\/li>\n
                    • \u9996\u9009\u7b56\u7565：\u6a21\u578b\u84b8\u998f (Distillation) + \u6781\u81f4\u91cf\u5316\u3002\n
                        \n
                      • \u6a21\u578b\u7626\u8eab：\u5148\u5c06 7B\/14B \u6a21\u578b\u84b8\u998f\u4e3a 1.8B\/3B \u7684\u5c0f\u6a21\u578b，\u76f4\u63a5\u964d\u4f4e\u8ba1\u7b97\u91cf\u7ea7\u3002<\/li>\n
                      • \u6781\u81f4\u538b\u7f29：\u914d\u5408 AWQ\/INT4 \u91cf\u5316，\u5c06\u663e\u5b58\u5360\u7528\u538b\u81f3 2GB-4GB \u533a\u95f4\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
                        1.3.3 \u7cbe\u5ea6\u4e0e\u6027\u80fd\u7684\u5e73\u8861 (Trade-off Matrix)<\/h5>\n

                        \u4e3a\u4e86\u5728\u201c\u53c8\u5feb\u53c8\u597d\u201d\u4e4b\u95f4\u627e\u5230\u5e73\u8861\u70b9，\u8bf7\u53c2\u8003\u4ee5\u4e0b\u7ec4\u5408\u63a8\u8350：<\/p>\n\n\u4e1a\u52a1\u573a\u666f\u63a8\u8350\u7ec4\u5408\u65b9\u6848\u9884\u671f\u7cbe\u5ea6\u635f\u5931\u9884\u671f\u52a0\u901f\u6bd4\u5178\u578b\u914d\u7f6e\u4ee3\u7801\u53c2\u8003<\/tr>\n\n\n\n\n\n
                        \u9ad8\u7cbe\u5ea6\u79d1\u7814\/\u91d1\u878d<\/td>\nFP16 (\u539f\u59cb) + KV Cache<\/td>\n0% (\u57fa\u51c6)<\/td>\n1.0x<\/td>\nvllm.LLM(…, dtype='float16')<\/td>\n<\/tr>\n
                        \u901a\u7528\u5bf9\u8bdd\/\u5ba2\u670d<\/td>\nINT8 \u91cf\u5316 + KV Cache (\u4e3b\u6d41\u63a8\u8350)<\/td>\n< 1% (\u51e0\u4e4e\u65e0\u611f)<\/td>\n1.5x – 2.0x<\/td>\nload_in_8bit=True<\/td>\n<\/tr>\n
                        \u9ad8\u5e76\u53d1 API \u670d\u52a1<\/td>\nINT4 GPTQ + vLLM PagedAttention<\/td>\n< 2%<\/td>\n2.0x – 3.0x<\/td>\nquantization='gptq'<\/td>\n<\/tr>\n
                        \u8fb9\u7f18\u8bbe\u5907\/IoT<\/td>\n\u84b8\u998f (1.8B) + INT4 + KV Cache<\/td>\n< 5% (\u7279\u5b9a\u4efb\u52a1\u53ef\u63a7)<\/td>\n5.0x – 10.0x<\/td>\n\u89c1\u7ae0\u8282 5.3<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                        \u4e8c\u3001\u73af\u5883\u914d\u7f6e（\u7edf\u4e00\u57fa\u7840\u73af\u5883）<\/h3>\n

                        2.1 \u90e8\u7f72\u73af\u5883\u8981\u6c42<\/h4>\n\n\u73af\u5883\u7c7b\u578b\u786c\u4ef6\u914d\u7f6e\u8f6f\u4ef6\u4f9d\u8d56<\/tr>\n\n\n\n\n
                        \u57fa\u7840\u6d4b\u8bd5<\/td>\nCPU 8\u6838+\u3001\u5185\u5b58 32GB+<\/td>\nPython 3.10-3.11\u3001Conda<\/td>\n<\/tr>\n
                        GPU\u52a0\u901f<\/td>\nNVIDIA GPU（CUDA 12.1+，\u663e\u5b58 8GB+）<\/td>\nCUDA 12.1\u3001cuDNN 8.9<\/td>\n<\/tr>\n
                        \u751f\u4ea7\u90e8\u7f72<\/td>\nGPU（A10\/T4\/V100\/A100）<\/td>\nDocker\u3001FastAPI\u3001vLLM<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                        2.2 \u73af\u5883\u914d\u7f6e\u6b65\u9aa4<\/h4>\n

                        2.2.1 Conda\u73af\u5883\u521b\u5efa<\/h5>\n

                        # \u521b\u5efa\u4e13\u5c5e\u73af\u5883
                        \nconda create -n llm-inference-accel python=3.10
                        \n# \u6fc0\u6d3b\u73af\u5883
                        \nconda activate llm-inference-accel<\/p>\n

                        2.2.2 \u6838\u5fc3\u4f9d\u8d56\u5b89\u88c5<\/h5>\n

                        # \u57fa\u7840\u4f9d\u8d56
                        \npip install torch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 –index-url https:\/\/download.pytorch.org\/whl\/cu121
                        \npip install transformers==4.41.2 accelerate==0.31.0 datasets==2.20.0<\/p>\n

                        # KV Cache\u4f18\u5316：vLLM（\u6700\u65b0\u7a33\u5b9a\u7248）
                        \npip install vllm==0.5.0<\/p>\n

                        # \u91cf\u5316\u4f9d\u8d56
                        \npip install auto-gptq==0.7.1 bitsandbytes==0.43.1
                        \npip install optimum==1.20.0 trl==0.8.6<\/p>\n

                        # \u84b8\u998f\u4f9d\u8d56
                        \npip install peft==0.11.1 sentencepiece==0.1.99<\/p>\n

                        # \u8f85\u52a9\u5de5\u5177
                        \npip install modelscope==1.15.0 python-dotenv==1.0.1 psutil==5.9.8<\/p>\n

                        \u4e09\u3001KV Cache \u4f18\u5316\u5b9e\u6218（\u57fa\u4e8evLLM）<\/h3>\n

                        3.1 KV Cache \u6838\u5fc3\u539f\u7406：\u4ee5\u7a7a\u95f4\u6362\u65f6\u95f4<\/h4>\n

                        \u8fd9\u662f\u4e3a\u60a8\u6df1\u5ea6\u4fee\u8ba2\u7684\u201c3.1 KV Cache \u6838\u5fc3\u539f\u7406\u201d\u4e0e\u201c3.2 \u9002\u7528\u573a\u666f\u201d\u90e8\u5206\u3002<\/p>\n

                        \u6b64\u6b21\u4fee\u6539\u7684\u6838\u5fc3\u5728\u4e8e：<\/p>\n

                      • \u6df1\u5316\u539f\u7406：\u4ece\u201c\u7a7a\u95f4\u6362\u65f6\u95f4\u201d\u7684\u672c\u8d28\u51fa\u53d1，\u89e3\u6790\u4e86 Self-Attention \u7684\u8ba1\u7b97\u5197\u4f59\u3002<\/li>\n
                      • \u589e\u52a0\u53ef\u89c6\u5316：\u6dfb\u52a0\u4e86 KV Cache \u8fd0\u4f5c\u673a\u5236\u6811\u5f62\u56fe，\u76f4\u89c2\u5c55\u793a\u6570\u636e\u6d41\u5411\u3002<\/li>\n
                      • \u65b0\u589e\u5173\u952e\u786c\u6838\u77e5\u8bc6：\u8865\u5145\u4e86 KV Cache \u663e\u5b58\u8ba1\u7b97\u516c\u5f0f，\u8fd9\u662f\u5b9e\u9645\u90e8\u7f72\u4e2d\u8ba1\u7b97 GPU \u8d44\u6e90\u9700\u6c42\u7684\u5173\u952e\u4f9d\u636e（\u60a8\u89c9\u5f97\u6709\u5fc5\u8981\u6dfb\u52a0\u7684\u5185\u5bb9）\u3002<\/li>\n
                      • PagedAttention \u7c7b\u6bd4：\u7528\u201c\u64cd\u4f5c\u7cfb\u7edf\u865a\u62df\u5185\u5b58\u201d\u7684\u7ecf\u5178\u7c7b\u6bd4\u89e3\u91ca vLLM \u7684\u6838\u5fc3\u4f18\u52bf\u3002<\/li>\n
                        \n

                        \u4e09\u3001 KV Cache \u4f18\u5316\u5b9e\u6218（\u57fa\u4e8e vLLM）<\/h3>\n

                        3.1 KV Cache \u6838\u5fc3\u539f\u7406：\u4ee5\u7a7a\u95f4\u6362\u65f6\u95f4<\/h4>\n

                        KV Cache（Key-Value Cache）\u662f\u5927\u6a21\u578b\u63a8\u7406\u6027\u80fd\u4f18\u5316\u7684\u201c\u57fa\u77f3\u201d\u3002\u5728 Transformer \u67b6\u6784\u7684\u81ea\u6ce8\u610f\u529b\u673a\u5236（Self-Attention）\u4e2d，\u751f\u6210\u6bcf\u4e00\u4e2a\u65b0 Token \u65f6，\u90fd\u9700\u8981\u8ba1\u7b97\u5b83\u4e0e\u4e4b\u524d\u6240\u6709 Token \u7684\u76f8\u4e92\u5173\u7cfb（Attention Score）\u3002<\/p>\n

                        3.1.1 \u8fd0\u4f5c\u673a\u5236\u89e3\u6790<\/h5>\n
                          \n
                        • \n

                          \u65e0 KV Cache（\u8ba1\u7b97\u5197\u4f59）：<\/p>\n

                          \u6bcf\u751f\u6210\u7b2c N\u4e2a Token，\u6a21\u578b\u9700\u8981\u91cd\u65b0\u8ba1\u7b97\u524d N-1 \u4e2a Token \u7684 Key \u548c Value \u77e9\u9635\u3002\u8fd9\u610f\u5473\u7740\u5386\u53f2\u4fe1\u606f\u88ab\u53cd\u590d\u8ba1\u7b97，\u8ba1\u7b97\u91cf\u968f\u5e8f\u5217\u957f\u5ea6\u5448 <\/p>\n

                          O<\/p>\n

                          (<\/p>\n

                          N<\/p>\n

                          2<\/p>\n

                          )<\/p>\n

                          O(N^2) <\/p>\n

                          <\/span><\/span>O<\/span>(<\/span>N<\/span><\/span>2<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span><\/span><\/span> \u589e\u957f\u3002<\/p>\n<\/li>\n

                        • \n

                          \u6709 KV Cache（\u589e\u91cf\u8ba1\u7b97）：<\/p>\n

                          \u7cfb\u7edf\u5c06\u5386\u53f2 Token \u8ba1\u7b97\u8fc7\u7684 Key \u548c Value \u77e9\u9635\u9a7b\u7559\u5728\u663e\u5b58\u4e2d\u3002\u751f\u6210\u7b2c N\u4e2a Token \u65f6，\u53ea\u9700\u8ba1\u7b97\u5f53\u524d Token \u7684 <\/p>\n

                          K<\/p>\n

                          N<\/p>\n

                          ,<\/p>\n

                          V<\/p>\n

                          N<\/p>\n

                          K_N, V_N <\/p>\n

                          <\/span><\/span>K<\/span><\/span>N<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span><\/span>V<\/span><\/span>N<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> ，\u7136\u540e\u4e0e\u7f13\u5b58\u4e2d\u7684\u5386\u53f2 <\/p>\n

                          K<\/p>\n

                          1<\/p>\n

                          :<\/p>\n

                          N<\/p>\n

                          \u2212<\/p>\n

                          1<\/p>\n

                          ,<\/p>\n

                          V<\/p>\n

                          1<\/p>\n

                          :<\/p>\n

                          N<\/p>\n

                          \u2212<\/p>\n

                          1<\/p>\n

                          K_{1:N-1}, V_{1:N-1} <\/p>\n

                          <\/span><\/span>K<\/span><\/span>1<\/span>:<\/span>N<\/span>\u2212<\/span>1<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span><\/span>V<\/span><\/span>1<\/span>:<\/span>N<\/span>\u2212<\/span>1<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> \u62fc\u63a5\u3002\u8ba1\u7b97\u91cf\u964d\u4f4e\u4e3a\u7ebf\u6027\u7684 <\/p>\n

                          O<\/p>\n

                          (<\/p>\n

                          N<\/p>\n

                          )<\/p>\n

                          O(N) <\/p>\n

                          <\/span><\/span>O<\/span>(<\/span>N<\/span>)<\/span><\/span><\/span><\/span><\/span><\/span> \u3002<\/p>\n<\/li>\n<\/ul>\n

                          3.1.2 \u6838\u5fc3\u673a\u5236\u4e0e\u4f18\u5316\u67b6\u6784\u56fe (Tree Structure)<\/h5>\n

                          \u4ee3\u7801\u6bb5<\/p>\n

                          graph TD
                          \n A[<\/span>\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f: KV Cache]<\/span> —><\/span> B(<\/span>\u6838\u5fc3\u539f\u7406: \u7a7a\u95f4\u6362\u65f6\u95f4)<\/span>
                          \n B —><\/span> B1[<\/span>Prefill\u9636\u6bb5: \u5e76\u884c\u8ba1\u7b97, \u586b\u5145Cache]<\/span>
                          \n B —><\/span> B2[<\/span>Decode\u9636\u6bb5: \u4e32\u884c\u751f\u6210, \u8bfb\u53d6Cache]<\/span><\/p>\n

                          A —><\/span> C{<\/span>\u5185\u5b58\u74f6\u9888\u6311\u6218}<\/span>
                          \n C —><\/span> C1[<\/span>\u663e\u5b58\u5360\u7528\u968f\u957f\u5ea6\u7ebf\u6027\u589e\u957f]<\/span>
                          \n C —><\/span> C2[<\/span>\u663e\u5b58\u788e\u7247\u5316\u95ee\u9898]<\/span><\/p>\n

                          A —><\/span> D[<\/span>\u89e3\u51b3\u65b9\u6848: vLLM PagedAttention]<\/span>
                          \n D —><\/span> D1[<\/span>\u6838\u5fc3\u601d\u60f3: \u64cd\u4f5c\u7cfb\u7edf\u5206\u9875\u5185\u5b58\u7ba1\u7406]<\/span>
                          \n D —><\/span> D2[<\/span>Block Table: \u6620\u5c04\u903b\u8f91\u9875\u5230\u7269\u7406\u9875]<\/span>
                          \n D —><\/span> D3[<\/span>\u4f18\u52bf: \u663e\u5b58\u5229\u7528\u7387\u63a5\u8fd1100%]<\/span><\/p>\n

                          \u3010KV Cache \u6838\u5fc3\u673a\u5236\u4e0e\u4f18\u5316\u903b\u8f91\u6811\u3011<\/p>\n

                            \n
                          • 1. \u57fa\u7840\u539f\u7406 (Mechanism)\n
                              \n
                            • Prefill \u9636\u6bb5 (\u9884\u586b\u5145)：\u4e00\u6b21\u6027\u8ba1\u7b97 Prompt \u4e2d\u6240\u6709 Token \u7684 KV，\u5b58\u5165\u663e\u5b58\u3002<\/li>\n
                            • Decode \u9636\u6bb5 (\u89e3\u7801)：\u6bcf\u6b65\u4ec5\u8ba1\u7b97 1 \u4e2a\u65b0 Token \u7684 KV，\u8ffd\u52a0\u5230\u7f13\u5b58\u672b\u5c3e\u3002<\/li>\n
                            • \u672c\u8d28：\u901a\u8fc7\u6d88\u8017\u663e\u5b58（Memory）\u6765\u6d88\u9664\u5197\u4f59\u8ba1\u7b97（Compute）\u3002<\/li>\n<\/ul>\n<\/li>\n
                            • 2. \u9762\u4e34\u6311\u6218 (Bottleneck)\n
                                \n
                              • \u52a8\u6001\u589e\u957f：Cache \u5927\u5c0f\u968f\u5bf9\u8bdd\u957f\u5ea6\u4e0d\u53ef\u9884\u6d4b\u5730\u589e\u957f\u3002<\/li>\n
                              • \u663e\u5b58\u788e\u7247：\u4f20\u7edf\u6846\u67b6\u8981\u6c42 KV Cache \u5360\u7528\u8fde\u7eed\u663e\u5b58，\u5bfc\u81f4\u5927\u91cf\u788e\u7247\u6d6a\u8d39\u3002<\/li>\n<\/ul>\n<\/li>\n
                              • 3. \u8fdb\u9636\u4f18\u5316 (Optimization: PagedAttention)\n
                                  \n
                                • \u6838\u5fc3\u521b\u65b0：\u5f15\u5165\u64cd\u4f5c\u7cfb\u7edf\u7684\u201c\u865a\u62df\u5185\u5b58\u201d\u6982\u5ff5\u3002<\/li>\n
                                • Block \u7ba1\u7406：\u5c06 KV Cache \u5207\u5206\u4e3a\u56fa\u5b9a\u5927\u5c0f\u7684\u5757（Block），\u975e\u8fde\u7eed\u5b58\u50a8\u3002<\/li>\n
                                • \u6548\u679c：\u663e\u5b58\u6d6a\u8d39\u7387\u4ece 20%-50% \u964d\u81f3 <4%，\u541e\u5410\u91cf\u63d0\u5347 2-4 \u500d\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
                                  3.1.3 KV Cache \u663e\u5b58\u5360\u7528\u4f30\u7b97\u516c\u5f0f<\/h5>\n

                                  \u5728\u5b9e\u9645\u90e8\u7f72\u4e2d，\u4e86\u89e3 KV Cache \u5360\u7528\u591a\u5c11\u663e\u5b58\u81f3\u5173\u91cd\u8981\u3002\u8fd9\u51b3\u5b9a\u4e86\u4f60\u7684\u663e\u5361\u80fd\u652f\u6301\u591a\u957f\u7684\u4e0a\u4e0b\u6587\u3002<\/p>\n

                                  \u516c\u5f0f： <\/p>\n

                                  Memory\u00a0(Bytes)<\/p>\n

                                  =<\/p>\n

                                  2<\/p>\n

                                  \u00d7<\/p>\n

                                  layers<\/p>\n

                                  \u00d7<\/p>\n

                                  hidden_size<\/p>\n

                                  \u00d7<\/p>\n

                                  seq_len<\/p>\n

                                  \u00d7<\/p>\n

                                  batch_size<\/p>\n

                                  \u00d7<\/p>\n

                                  dtype_size<\/p>\n

                                  \\\\text{Memory (Bytes)} = 2 \\\\times \\\\text{layers} \\\\times \\\\text{hidden\\\\_size} \\\\times \\\\text{seq\\\\_len} \\\\times \\\\text{batch\\\\_size} \\\\times \\\\text{dtype\\\\_size} <\/p>\n

                                  <\/span><\/span>Memory\u00a0(Bytes)<\/span><\/span><\/span>=<\/span><\/span><\/span><\/span>2<\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>layers<\/span><\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>hidden_size<\/span><\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>seq_len<\/span><\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>batch_size<\/span><\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>dtype_size<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

                                    \n
                                  • 2：\u5206\u522b\u4ee3\u8868 Key \u548c Value \u4e24\u4e2a\u77e9\u9635\u3002<\/li>\n
                                  • dtype_size：FP16 \u4e3a 2 Bytes，FP32 \u4e3a 4 Bytes\u3002<\/li>\n<\/ul>\n

                                    \u5b9e\u6218\u793a\u4f8b（Llama-2-7B，FP16）：<\/p>\n

                                      \n
                                    • \n

                                      Layers = 32, Hidden Size = 4096<\/p>\n<\/li>\n

                                    • \n

                                      \u5bf9\u4e8e 1 \u4e2a\u8bf7\u6c42，\u4e0a\u4e0b\u6587\u957f\u5ea6\u8fbe\u5230 1024 \u65f6： <\/p>\n

                                      2<\/p>\n

                                      \u00d7<\/p>\n

                                      32<\/p>\n

                                      \u00d7<\/p>\n

                                      4096<\/p>\n

                                      \u00d7<\/p>\n

                                      1024<\/p>\n

                                      \u00d7<\/p>\n

                                      1<\/p>\n

                                      \u00d7<\/p>\n

                                      2<\/p>\n

                                      \u2248<\/p>\n

                                      512<\/p>\n

                                      \u00a0MB<\/p>\n

                                      2 \\\\times 32 \\\\times 4096 \\\\times 1024 \\\\times 1 \\\\times 2 \\\\approx 512 \\\\text{ MB} <\/p>\n

                                      <\/span><\/span>2<\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>32<\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>4096<\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>1024<\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>1<\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>2<\/span><\/span>\u2248<\/span><\/span><\/span><\/span>512<\/span>\u00a0MB<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<\/li>\n

                                    • \n

                                      \u7ed3\u8bba：\u4ec5 1K \u4e0a\u4e0b\u6587\u7684\u4e00\u4e2a\u5e76\u53d1\u5c31\u5360\u7528 0.5GB \u663e\u5b58\u3002\u5982\u679c\u662f 64 \u5e76\u53d1，\u4ec5 KV Cache \u5c31\u9700\u8981 32GB \u663e\u5b58！\u8fd9\u5c31\u662f\u4e3a\u4ec0\u4e48**\u663e\u5b58\u4f18\u5316（\u5982 PagedAttention）**\u5982\u6b64\u91cd\u8981\u7684\u539f\u56e0\u3002<\/p>\n<\/li>\n<\/ul>\n

                                      \n

                                      3.2 \u9002\u7528\u573a\u666f\u4e0e\u9608\u503c\u6307\u5357<\/h4>\n

                                      KV Cache \u662f\u73b0\u4ee3\u5927\u6a21\u578b\u63a8\u7406\u7684\u201c\u6807\u914d\u201d，\u4f46\u5728\u4ee5\u4e0b\u573a\u666f\u4e2d\u5176\u6536\u76ca\u6700\u4e3a\u5173\u952e\u3002<\/p>\n

                                      3.2.1 \u51b3\u7b56\u6811\u72b6\u56fe<\/h5>\n

                                      \u662f\u5426\u5f00\u542f KV Cache?
                                      \n\u251c\u2500\u2500 1<\/span>. \u4e0a\u4e0b\u6587\u957f\u5ea6\u5206\u6790
                                      \n\u2502 \u251c\u2500\u2500 \u77ed\u6587\u672c (<\/span><<\/span> 128<\/span> tokens)<\/span>: \u6536\u76ca\u4e0d\u660e\u663e，\u6b64\u65f6\u4e3b\u8981\u74f6\u9888\u5728\u6a21\u578b\u6743\u91cd\u52a0\u8f7d
                                      \n\u2502 \u2514\u2500\u2500 \u957f\u6587\u672c (<\/span>><\/span> 512<\/span> tokens)<\/span>: \u3010\u5fc5\u987b\u5f00\u542f\u3011，\u5426\u5219\u5ef6\u8fdf\u5c06\u5448\u6307\u6570\u7ea7\u4e0a\u5347
                                      \n\u251c\u2500\u2500 2<\/span>. \u4ea4\u4e92\u6a21\u5f0f\u5206\u6790
                                      \n\u2502 \u251c\u2500\u2500 \u5355\u8f6e\u95ee\u7b54: \u63a8\u8350\u5f00\u542f
                                      \n\u2502 \u2514\u2500\u2500 \u591a\u8f6e\u5bf9\u8bdd: \u3010\u6838\u5fc3\u573a\u666f\u3011，\u5fc5\u987b\u7f13\u5b58\u5386\u53f2\u5bf9\u8bdd\u7684 KV，\u5426\u5219\u6bcf\u8f6e\u90fd\u8981\u91cd\u7b97\u65e7\u5386\u53f2
                                      \n\u2514\u2500\u2500 3<\/span>. \u4e1a\u52a1\u5f62\u6001\u5206\u6790
                                      \n \u251c\u2500\u2500 \u79bb\u7ebf\u6279\u5904\u7406: \u914d\u5408 PagedAttention \u63d0\u5347 Batch Size
                                      \n \u2514\u2500\u2500 \u5b9e\u65f6\u6d41\u5f0f\u8f93\u51fa: \u5fc5\u987b\u5f00\u542f，\u4fdd\u8bc1 Token \u751f\u6210\u901f\u5ea6\u5e73\u7a33<\/p>\n

                                      3.2.2 \u8be6\u7ec6\u573a\u666f\u8bf4\u660e<\/h5>\n
                                    • \u591a\u8f6e\u5bf9\u8bdd（Multi-Turn Conversation）\n
                                        \n
                                      • \u7279\u5f81：\u7528\u6237\u4e0e AI \u53cd\u590d\u4ea4\u4e92，History \u8d8a\u6765\u8d8a\u957f\u3002<\/li>\n
                                      • \u4ef7\u503c：\u907f\u514d\u5728\u7b2c 10 \u8f6e\u5bf9\u8bdd\u65f6，\u8fd8\u8981\u91cd\u65b0\u8ba1\u7b97\u524d 9 \u8f6e\u7684\u6570\u767e\u4e2a Token\u3002<\/li>\n<\/ul>\n<\/li>\n
                                      • \u957f\u6587\u672c\u751f\u6210（Long-Context Generation）\n
                                          \n
                                        • \u7279\u5f81：\u5199\u5c0f\u8bf4\u3001\u751f\u6210\u957f\u7bc7\u62a5\u544a（\u8f93\u51fa > 1024 tokens）\u3002<\/li>\n
                                        • \u4ef7\u503c：\u5982\u679c\u6ca1\u6709 KV Cache，\u751f\u6210\u7b2c 1000 \u4e2a\u5b57\u7684\u901f\u5ea6\u4f1a\u6bd4\u7b2c 1 \u4e2a\u5b57\u6162\u51e0\u767e\u500d（\u56e0\u4e3a\u8ba1\u7b97\u91cf\u5728\u7d2f\u52a0）\u3002\u5f00\u542f\u540e，\u751f\u6210\u901f\u5ea6\u4fdd\u6301\u6052\u5b9a\u3002<\/li>\n<\/ul>\n<\/li>\n
                                        • \u9ad8\u5e76\u53d1\u6279\u91cf\u63a8\u7406（High Throughput Serving）\n
                                            \n
                                          • \u7279\u5f81：API \u670d\u52a1\u7aef，\u540c\u65f6\u670d\u52a1\u6570\u5341\u4e2a\u7528\u6237\u3002<\/li>\n
                                          • \u4ef7\u503c：\u914d\u5408 vLLM \u7684 PagedAttention，\u53ef\u4ee5\u5728\u6709\u9650\u7684\u663e\u5b58\u4e2d\u585e\u5165\u66f4\u591a\u7528\u6237\u7684 KV Cache，\u4ece\u800c\u663e\u8457\u63d0\u5347 Batch Size（\u541e\u5410\u91cf）\u3002<\/li>\n<\/ul>\n<\/li>\n

                                            3.3 \u4ee3\u7801\u5b9e\u73b0（Qwen2.5-7B-Instruct）<\/h4>\n

                                            3.3.1 \u57fa\u7840KV Cache\u914d\u7f6e（vLLM）<\/h5>\n
                                            vLLM \u63a8\u7406\u52a0\u901f\u6d41\u7a0b\u56fe（\u57fa\u4e8e\u4ee3\u7801\u903b\u8f91）<\/h6>\n

                                            \u2502
                                            \n\u251c\u2500\u2500 \u3010\u7a0b\u5e8f\u5165\u53e3\u3011: if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span>
                                            \n\u2502
                                            \n\u25bc
                                            \n[<\/span>1<\/span>. \u5f15\u64ce\u521d\u59cb\u5316 (<\/span>Engine Initialization)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 A. \u73af\u5883\u914d\u7f6e (<\/span>Environment)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u52a0\u8f7d\u6587\u4ef6><\/span>: .env (<\/span>dotenv)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 <<\/span>\u8bfb\u53d6\u53d8\u91cf><\/span>: MODEL_DIR (<\/span>\u6a21\u578b\u8def\u5f84)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 B. vLLM \u5f15\u64ce\u6784\u5efa (<\/span>LLM Class Instantiation)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u52a0\u8f7d\u6a21\u578b><\/span>: "Qwen\/Qwen2.5-7B-Instruct"<\/span> \u2502
                                            \n\u2502 \u2502 \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u663e\u5b58\u4e0e\u7f13\u5b58\u7ba1\u7406><\/span>: PagedAttention (<\/span>\u6838\u5fc3\u4f18\u5316)<\/span> \u2502
                                            \n\u2502 \u2502 \u251c\u2500\u2500 gpu_memory_utilization =<\/span> 0.9<\/span> (<\/span>\u9884\u755990%\u663e\u5b58\u7528\u4e8eKV)<\/span> \u2502
                                            \n\u2502 \u2502 \u251c\u2500\u2500 swap_space =<\/span> 4<\/span> (<\/span>4GB CPU\u5185\u5b58\u4f5c\u4e3aSwap\u533a，\u9632OOM)<\/span> \u2502
                                            \n\u2502 \u2502 \u2514\u2500\u2500 max_cpu_cache_buffer =<\/span> 2<\/span> (<\/span>CPU\u7f13\u5b58\u7f13\u51b2)<\/span> \u2502
                                            \n\u2502 \u2502 \u2502
                                            \n\u2502 \u2514\u2500\u2500 <<\/span>\u8c03\u5ea6\u4e0e\u5e76\u884c\u7b56\u7565><\/span>: Batching &<\/span> Parallelism \u2502
                                            \n\u2502 \u251c\u2500\u2500 tensor_parallel_size =<\/span> 1<\/span> (<\/span>\u5355\u5361\u8fd0\u884c)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 max_num_seqs =<\/span> 64<\/span> (<\/span>\u6700\u5927\u5e76\u53d1\u5e8f\u5217\u6570)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 max_num_batched_tokens =<\/span> 4096<\/span> (<\/span>\u6279\u5904\u7406\u5927\u5c0f)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 enable_chunked_prefill =<\/span> True (<\/span>\u957fPrompt\u5206\u5757\u5904\u7406)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u2514\u2500\u2500 ><\/span> \u5f15\u64ce\u5c31\u7eea: llm \u5bf9\u8c61 (<\/span>KV Cache \u7ba1\u7406\u5668\u5df2\u542f\u52a8)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                            \n \u2502
                                            \n \u25bc
                                            \n[<\/span>2<\/span>. \u63a8\u7406\u6267\u884c\u6d41\u7a0b (<\/span>Inference Execution)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 A. \u91c7\u6837\u53c2\u6570\u8bbe\u5b9a (<\/span>SamplingParams)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 temperature =<\/span> 0.1<\/span> (<\/span>\u4f4e\u968f\u673a\u6027，\u9002\u5408\u6307\u4ee4\u9075\u5faa)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 max_tokens =<\/span> 2048<\/span> (<\/span>\u9650\u5236\u8f93\u51fa\u957f\u5ea6)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 optimization: use_beam_search =<\/span> False (<\/span>\u8d2a\u5a6a\u89e3\u7801\u66f4\u5feb)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 B. \u63d0\u793a\u8bcd\u5de5\u7a0b (<\/span>Prompt Engineering)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u8f93\u5165><\/span>: prompts \u5217\u8868 (<\/span>\u591a\u8f6e\u5bf9\u8bdd\u6d4b\u8bd5\u7528\u4f8b)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 <<\/span>\u683c\u5f0f\u5316><\/span>: ChatML \u683c\u5f0f\u5c01\u88c5 \u2502
                                            \n\u2502 \u2514\u2500\u2500 f"<|im_start|>user\\\\n<\/span>{prompt}<|im_end|>\\\\n<\/span>…"<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 C. \u751f\u6210\u8c03\u7528\u5faa\u73af (<\/span>llm.generate)<\/span> \u2502
                                            \n\u2502 \u2502 \u2502
                                            \n\u2502 \u251c\u2500\u2500 \u9636\u6bb5 1<\/span>: Prefill (<\/span>\u9884\u586b\u5145 – \u8ba1\u7b97Prompt\u7684KV)<\/span> \u2502
                                            \n\u2502 \u2502 \u251c\u2500\u2500 <<\/span>\u5206\u5757><\/span>: \u82e5\u957f\u5ea6 ><\/span> 512<\/span>, \u89e6\u53d1 Chunked Prefill \u2502
                                            \n\u2502 \u2502 \u2514\u2500\u2500 <<\/span>\u5b58\u50a8><\/span>: \u5199\u5165\u663e\u5b58\u4e2d\u7684 Paged Block Tables \u2502
                                            \n\u2502 \u2502 \u2502
                                            \n\u2502 \u2514\u2500\u2500 \u9636\u6bb5 2<\/span>: Decode (<\/span>\u89e3\u7801 – \u751f\u6210Response)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u8bfb\u53d6><\/span>: \u6839\u636e Block Table \u7d22\u5f15\u547d\u4e2d\u5386\u53f2 KV \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u8ba1\u7b97><\/span>: \u4ec5\u8ba1\u7b97\u65b0 Token \u7684 Attention (<\/span>O(<\/span>1<\/span>)<\/span>\u590d\u6742\u5ea6)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 <<\/span>\u8fed\u4ee3><\/span>: \u76f4\u5230\u8f93\u51fa <<\/span>|<\/span>im_end|<\/span>><\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u2514\u2500\u2500 ><\/span> \u8f93\u51fa\u5bf9\u8c61: RequestOutput (<\/span>\u5305\u542b generated_text)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                            \n \u2502
                                            \n \u25bc
                                            \n[<\/span>3<\/span>. \u6027\u80fd\u5bf9\u6bd4\u6d4b\u8bd5 (<\/span>Performance Benchmarking)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 <<\/span>\u573a\u666f 1<\/span>><\/span>: \u542f\u7528 KV Cache (<\/span>test_kv_cache_effect)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 \u72b6\u6001: \u6b63\u5e38\u5f00\u542f PagedAttention \u2502
                                            \n\u2502 \u251c\u2500\u2500 \u884c\u4e3a: \u968f\u7740\u5bf9\u8bdd\u8f6e\u6570\u589e\u52a0，\u63a8\u7406\u901f\u5ea6\u4fdd\u6301\u7a33\u5b9a \u2502
                                            \n\u2502 \u2514\u2500\u2500 \u8f93\u51fa: \u6253\u5370\u6bcf\u8f6e\u8017\u65f6 (<\/span>Time Delta)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 <<\/span>\u573a\u666f 2<\/span>><\/span>: \u7981\u7528 KV Cache (<\/span>test_no_kv_cache)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 \u914d\u7f6e: max_num_batched_tokens =<\/span> 1<\/span> (<\/span>\u5f3a\u5236\u4e32\u884c\/\u65e0\u6279\u5904\u7406)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 \u884c\u4e3a: \u6bcf\u751f\u6210\u4e00\u4e2a\u8bcd\u90fd\u91cd\u7b97\u6240\u6709\u5386\u53f2，\u901f\u5ea6\u6781\u6162 \u2502
                                            \n\u2502 \u2514\u2500\u2500 \u8f93\u51fa: \u6253\u5370\u9ad8\u5ef6\u8fdf\u7ed3\u679c，\u5f62\u6210\u9c9c\u660e\u5bf9\u6bd4 \u2502
                                            \n\u2502 \u2502
                                            \n\u2514\u2500\u2500 ><\/span> \u6700\u7ec8\u62a5\u544a: Console Print (<\/span>Total Time &<\/span> Speed)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

                                            import os
                                            \nimport time
                                            \nfrom vllm import LLM, SamplingParams
                                            \nfrom dotenv import load_dotenv<\/p>\n

                                            # \u52a0\u8f7d\u73af\u5883\u53d8\u91cf（\u53ef\u9009：\u914d\u7f6e\u6a21\u578b\u4e0b\u8f7d\u8def\u5f84）
                                            \nload_dotenv()<\/p>\n

                                            # 1. \u914d\u7f6evLLM（\u542f\u7528KV Cache，\u9ed8\u8ba4\u5f00\u542fPagedAttention）
                                            \nllm = LLM(
                                            \n model="Qwen\/Qwen2.5-7B-Instruct", # \u652f\u6301\u672c\u5730\u8def\u5f84\/ModelScope\/HuggingFace
                                            \n model_dir=os.getenv("MODEL_DIR", ".\/models\/Qwen2.5-7B-Instruct"),
                                            \n tensor_parallel_size=1, # \u5355GPU
                                            \n gpu_memory_utilization=0.9, # \u663e\u5b58\u5229\u7528\u7387
                                            \n max_num_batched_tokens=4096, # \u6279\u91cftoken\u6570
                                            \n max_num_seqs=64, # \u6700\u5927\u5e76\u53d1\u5e8f\u5217\u6570
                                            \n # KV Cache\u6838\u5fc3\u914d\u7f6e
                                            \n swap_space=4, # \u78c1\u76d8\u4ea4\u6362\u7a7a\u95f4（GB），\u663e\u5b58\u4e0d\u8db3\u65f6\u542f\u7528
                                            \n max_cpu_cache_buffer=2, # CPU\u7f13\u5b58KV\u7684\u7f13\u51b2\u533a（GB）
                                            \n enable_chunked_prefill=True, # \u5206\u5757\u9884\u586b\u5145KV Cache
                                            \n chunked_prefill_tokens=512, # \u9884\u586b\u5145chunk\u5927\u5c0f
                                            \n)<\/p>\n

                                            # 2. \u91c7\u6837\u53c2\u6570\u914d\u7f6e
                                            \nsampling_params = SamplingParams(
                                            \n temperature=0.1,
                                            \n max_tokens=2048, # \u6700\u5927\u751f\u6210\u957f\u5ea6
                                            \n top_p=0.95,
                                            \n # KV Cache\u76f8\u5173
                                            \n use_beam_search=False,
                                            \n ignore_eos=False,
                                            \n)<\/p>\n

                                            # 3. \u6d4b\u8bd5KV Cache\u6548\u679c
                                            \ndef test_kv_cache_effect():
                                            \n # \u6d4b\u8bd5\u7528\u4f8b：\u957f\u4e0a\u4e0b\u6587\u591a\u8f6e\u5bf9\u8bdd
                                            \n prompts = [
                                            \n "\u8bf7\u8be6\u7ec6\u4ecb\u7ecd\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6280\u672fKV Cache\u7684\u5de5\u4f5c\u539f\u7406，\u8981\u6c42\u5206\u70b9\u8bf4\u660e，\u5b57\u6570\u4e0d\u5c11\u4e8e1000\u5b57",
                                            \n "\u57fa\u4e8e\u4f60\u521a\u624d\u7684\u56de\u7b54，\u8fdb\u4e00\u6b65\u89e3\u91caPagedAttention\u76f8\u6bd4\u4f20\u7edfKV Cache\u7684\u4f18\u52bf",
                                            \n "\u603b\u7ed3\u4e0a\u8ff0\u5185\u5bb9，\u7ed9\u51faKV Cache\u5728\u5b9e\u9645\u90e8\u7f72\u4e2d\u7684\u6700\u4f73\u5b9e\u8df5"
                                            \n ]<\/p>\n

                                            # \u8bb0\u5f55\u6bcf\u8f6e\u8017\u65f6
                                            \n total_time = 0
                                            \n for i, prompt in enumerate(prompts):
                                            \n start_time = time.time()
                                            \n # \u6784\u9020\u5bf9\u8bdd\u683c\u5f0f（Qwen2.5\u8981\u6c42）
                                            \n formatted_prompt = f"<|im_start|>user\\\\n{prompt}<|im_end|>\\\\n<|im_start|>assistant\\\\n"
                                            \n outputs = llm.generate([formatted_prompt], sampling_params)
                                            \n end_time = time.time()<\/p>\n

                                            # \u8f93\u51fa\u7ed3\u679c
                                            \n output = outputs[0].outputs[0].text
                                            \n elapsed_time = end_time – start_time
                                            \n total_time += elapsed_time<\/p>\n

                                            print(f"\\\\n=== \u7b2c{i+1}\u8f6e\u5bf9\u8bdd ===")
                                            \n print(f"\u8017\u65f6：{elapsed_time:.2f}\u79d2")
                                            \n print(f"\u751f\u6210\u957f\u5ea6：{len(output)}\u5b57\u7b26")
                                            \n print(f"\u5185\u5bb9：{output[:200]}…")<\/p>\n

                                            print(f"\\\\n\u603b\u8017\u65f6：{total_time:.2f}\u79d2，\u5e73\u5747\u6bcf\u8f6e：{total_time\/len(prompts):.2f}\u79d2")<\/p>\n

                                            # 4. \u5bf9\u6bd4\u65e0KV Cache（\u7981\u7528PagedAttention）
                                            \ndef test_no_kv_cache():
                                            \n # \u7981\u7528KV Cache（\u901a\u8fc7\u8bbe\u7f6emax_num_batched_tokens=1）
                                            \n llm_no_kv = LLM(
                                            \n model="Qwen\/Qwen2.5-7B-Instruct",
                                            \n model_dir=os.getenv("MODEL_DIR", ".\/models\/Qwen2.5-7B-Instruct"),
                                            \n tensor_parallel_size=1,
                                            \n gpu_memory_utilization=0.9,
                                            \n max_num_batched_tokens=1, # \u7981\u7528\u6279\u91cf\u5904\u7406，\u7b49\u4ef7\u4e8e\u7981\u7528KV Cache
                                            \n enable_chunked_prefill=False,
                                            \n )<\/p>\n

                                            prompt = "\u8bf7\u8be6\u7ec6\u4ecb\u7ecd\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6280\u672fKV Cache\u7684\u5de5\u4f5c\u539f\u7406，\u8981\u6c42\u5206\u70b9\u8bf4\u660e，\u5b57\u6570\u4e0d\u5c11\u4e8e1000\u5b57"
                                            \n formatted_prompt = f"<|im_start|>user\\\\n{prompt}<|im_end|>\\\\n<|im_start|>assistant\\\\n"<\/p>\n

                                            start_time = time.time()
                                            \n outputs = llm_no_kv.generate([formatted_prompt], sampling_params)
                                            \n end_time = time.time()<\/p>\n

                                            output = outputs[0].outputs[0].text
                                            \n elapsed_time = end_time – start_time<\/p>\n

                                            print(f"\\\\n=== \u65e0KV Cache\u6d4b\u8bd5 ===")
                                            \n print(f"\u8017\u65f6：{elapsed_time:.2f}\u79d2")
                                            \n print(f"\u751f\u6210\u957f\u5ea6：{len(output)}\u5b57\u7b26")<\/p>\n

                                            if __name__ == "__main__":
                                            \n print("=== \u6d4b\u8bd5KV Cache\u6548\u679c ===")
                                            \n test_kv_cache_effect()
                                            \n print("\\\\n=== \u6d4b\u8bd5\u65e0KV Cache\u6548\u679c ===")
                                            \n test_no_kv_cache()<\/p>\n

                                            3.3.2 \u6548\u679c\u9a8c\u8bc1\u8f93\u51fa\u793a\u4f8b<\/h5>\n

                                            === \u6d4b\u8bd5KV Cache\u6548\u679c ===
                                            \n=== \u7b2c1\u8f6e\u5bf9\u8bdd ===
                                            \n\u8017\u65f6：8.24\u79d2
                                            \n\u751f\u6210\u957f\u5ea6：1256\u5b57\u7b26
                                            \n\u5185\u5bb9：KV Cache（Key-Value Cache）\u662f\u5927\u6a21\u578b\u63a8\u7406\u9636\u6bb5\u7684\u6838\u5fc3\u52a0\u901f\u6280\u672f，\u4e3b\u8981\u9488\u5bf9Transformer\u67b6\u6784\u7684\u81ea\u6ce8\u610f\u529b\u673a\u5236\u4f18\u5316…<\/p>\n

                                            === \u7b2c2\u8f6e\u5bf9\u8bdd ===
                                            \n\u8017\u65f6：3.12\u79d2
                                            \n\u751f\u6210\u957f\u5ea6：898\u5b57\u7b26
                                            \n\u5185\u5bb9：PagedAttention\u662fvLLM\u63d0\u51fa\u7684KV Cache\u4f18\u5316\u65b9\u6848，\u89e3\u51b3\u4e86\u4f20\u7edfKV Cache\u7684\u4e24\u5927\u95ee\u9898：1. \u663e\u5b58\u788e\u7247…<\/p>\n

                                            === \u7b2c3\u8f6e\u5bf9\u8bdd ===
                                            \n\u8017\u65f6：2.87\u79d2
                                            \n\u751f\u6210\u957f\u5ea6：756\u5b57\u7b26
                                            \n\u5185\u5bb9：KV Cache\u5728\u5b9e\u9645\u90e8\u7f72\u4e2d\u7684\u6700\u4f73\u5b9e\u8df5\u5305\u62ec：1. \u6839\u636eGPU\u663e\u5b58\u8c03\u6574chunked_prefill_tokens…<\/p>\n

                                            \u603b\u8017\u65f6：14.23\u79d2，\u5e73\u5747\u6bcf\u8f6e：4.74\u79d2<\/p>\n

                                            === \u6d4b\u8bd5\u65e0KV Cache\u6548\u679c ===
                                            \n=== \u65e0KV Cache\u6d4b\u8bd5 ===
                                            \n\u8017\u65f6：22.58\u79d2
                                            \n\u751f\u6210\u957f\u5ea6：1248\u5b57\u7b26<\/p>\n

                                            3.4 KV Cache \u8c03\u4f18\u6307\u5357<\/h4>\n\n\u53c2\u6570\u4f5c\u7528\u63a8\u8350\u503c（7B\u6a21\u578b）\u8c03\u4f18\u539f\u5219<\/tr>\n\n\n\n\n\n
                                            max_num_seqs<\/td>\n\u6700\u5927\u5e76\u53d1\u5e8f\u5217\u6570<\/td>\n32-64<\/td>\n\u663e\u5b58\u5145\u8db3\u5219\u589e\u5927<\/td>\n<\/tr>\n
                                            chunked_prefill_tokens<\/td>\n\u9884\u586b\u5145chunk\u5927\u5c0f<\/td>\n512-1024<\/td>\n\u957f\u6587\u672c\u2192\u589e\u5927，\u77ed\u6587\u672c\u2192\u51cf\u5c0f<\/td>\n<\/tr>\n
                                            gpu_memory_utilization<\/td>\n\u663e\u5b58\u5229\u7528\u7387<\/td>\n0.8-0.9<\/td>\n\u907f\u514dOOM\u5219\u964d\u4f4e\u81f30.7<\/td>\n<\/tr>\n
                                            swap_space<\/td>\n\u78c1\u76d8\u4ea4\u6362\u7a7a\u95f4<\/td>\n2-8GB<\/td>\n\u663e\u5b58\u4e0d\u8db3\u65f6\u542f\u7528<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                                            \u56db\u3001\u91cf\u5316\u6280\u672f\u5b9e\u6218（INT4\/INT8\/GPTQ）<\/h3>\n

                                            4.1 \u91cf\u5316\u6280\u672f\u5206\u7c7b\u4e0e\u9009\u578b<\/h4>\n\n\u91cf\u5316\u7c7b\u578b\u5de5\u5177\u7cbe\u5ea6\u901f\u5ea6\u63d0\u5347\u663e\u5b58\u8282\u7701\u9002\u7528\u6a21\u578b<\/tr>\n\n\n\n\n
                                            \u52a8\u6001INT8<\/td>\nbitsandbytes<\/td>\nINT8<\/td>\n1.5\u500d<\/td>\n50%<\/td>\n\u6240\u6709\u5f00\u6e90\u6a21\u578b<\/td>\n<\/tr>\n
                                            \u9759\u6001GPTQ<\/td>\nAutoGPTQ<\/td>\nINT4\/INT8<\/td>\n2-3\u500d<\/td>\n60-70%<\/td>\n\u652f\u6301GPTQ\u91cf\u5316\u7684\u6a21\u578b<\/td>\n<\/tr>\n
                                            AWQ\u91cf\u5316<\/td>\nawq<\/td>\nINT4\/INT8<\/td>\n2.5-3\u500d<\/td>\n65-75%<\/td>\nLlama3\/Qwen2.5<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                                            4.2 \u52a8\u6001INT8\u91cf\u5316（Transformers + bitsandbytes）<\/h4>\n

                                            4.2.1 \u4ee3\u7801\u5b9e\u73b0<\/h5>\n
                                            \u4ee3\u7801\u5b9e\u73b0\u6d41\u7a0b\u56fe (INT8 Quantization Flow)<\/h6>\n

                                            \u2502
                                            \n\u251c\u2500\u2500 \u3010\u7a0b\u5e8f\u5165\u53e3\u3011: if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span>
                                            \n\u2502
                                            \n\u25bc
                                            \n[<\/span>1<\/span>. \u73af\u5883\u4e0e\u914d\u7f6e\u9636\u6bb5 (<\/span>Configuration &<\/span> Setup)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 A. \u73af\u5883\u521d\u59cb\u5316 \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u52a0\u8f7d\u6587\u4ef6><\/span>: .env (<\/span>\u8bfb\u53d6 MODEL_DIR)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 <<\/span>\u68c0\u6d4b\u786c\u4ef6><\/span>: torch.cuda.is_available(<\/span>)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 B. \u91cf\u5316\u914d\u7f6e\u6784\u5efa (<\/span>BitsAndBytesConfig)<\/span> <<\/span>\u2605 \u6838\u5fc3\u6b65\u9aa4><\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 load_in_8bit =<\/span> True (<\/span>\u6fc0\u6d3b INT8 \u6df7\u5408\u7cbe\u5ea6)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 llm_int8_threshold =<\/span> 6.0<\/span> (<\/span>\u79bb\u7fa4\u503c\u4fdd\u62a4)<\/span> \u2502
                                            \n\u2502 \u2502 \u2514\u2500\u2500 ><\/span> \u4f5c\u7528: \u8d85\u8fc7 6.0<\/span> \u7684\u6fc0\u6d3b\u503c\u4fdd\u7559 FP16，\u9632\u6b62\u7cbe\u5ea6\u5d29\u584c \u2502
                                            \n\u2502 \u2514\u2500\u2500 llm_int8_skip_modules =<\/span> [<\/span>"lm_head"<\/span>]<\/span> (<\/span>\u8f93\u51fa\u5c42\u4fdd\u6301\u9ad8\u7cbe\u5ea6)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u2514\u2500\u2500 ><\/span> \u914d\u7f6e\u5bf9\u8c61: bnb_config \u51c6\u5907\u5c31\u7eea \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                            \n \u2502
                                            \n \u25bc
                                            \n[<\/span>2<\/span>. \u6a21\u578b\u52a0\u8f7d\u9636\u6bb5 (<\/span>Quantized Model Loading)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 A. Tokenizer \u52a0\u8f7d \u2502
                                            \n\u2502 \u2514\u2500\u2500 <<\/span>\u8c03\u7528><\/span>: AutoTokenizer.from_pretrained \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 B. \u6a21\u578b\u52a8\u6001\u91cf\u5316\u52a0\u8f7d (<\/span>On-the-fly Quantization)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u8f93\u5165><\/span>: \u539f\u59cb FP16 \u6743\u91cd\u6587\u4ef6 (<\/span>.bin\/.safetensors)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u5904\u7406><\/span>: AutoModelForCausalLM.from_pretrained \u2502
                                            \n\u2502 \u2502 \u251c\u2500\u2500 \u6ce8\u5165 quantization_config<\/span>=<\/span>bnb_config \u2502
                                            \n\u2502 \u2502 \u2514\u2500\u2500 \u6620\u5c04 device_map<\/span>=<\/span>"auto"<\/span> (<\/span>\u81ea\u52a8\u5206\u914d GPU\/CPU)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 ><\/span> \u7ed3\u679c: \u663e\u5b58\u4e2d\u9a7b\u7559\u7684\u662f [<\/span>INT8 \u6743\u91cd]<\/span> + [<\/span>\u5c11\u91cf FP16 \u6a21\u5757]<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u2514\u2500\u2500 ><\/span> \u72b6\u6001\u68c0\u67e5: model.is_quantized (<\/span>\u5e94\u4e3a True)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                            \n \u2502
                                            \n \u25bc
                                            \n[<\/span>3<\/span>. \u63a8\u7406\u6267\u884c\u5faa\u73af (<\/span>Inference Loop: test_int8_inference)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 <<\/span>\u8f93\u5165><\/span>: Prompts \u5217\u8868 [<\/span>"\u89e3\u91ca\u91cf\u5316…"<\/span>, "\u5982\u4f55\u5e73\u8861…"<\/span>]<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 \u21bb \u904d\u5386\u6bcf\u4e2a Prompt \u2502
                                            \n\u2502 \u2502 \u2502
                                            \n\u2502 \u251c\u2500\u2500 A. \u9884\u5904\u7406 (<\/span>Preprocessing)<\/span> \u2502
                                            \n\u2502 \u2502 \u251c\u2500\u2500 <<\/span>\u6a21\u7248><\/span>: apply_chat_template (<\/span>User\/Assistant \u683c\u5f0f)<\/span> \u2502
                                            \n\u2502 \u2502 \u2514\u2500\u2500 ><\/span> \u5f20\u91cf: input_ids (<\/span>\u79fb\u52a8\u5230 CUDA \u8bbe\u5907)<\/span> \u2502
                                            \n\u2502 \u2502 \u2502
                                            \n\u2502 \u251c\u2500\u2500 B. \u751f\u6210\u8fc7\u7a0b (<\/span>Generation)<\/span> \u2502
                                            \n\u2502 \u2502 \u251c\u2500\u2500 <<\/span>\u8c03\u7528><\/span>: model.generate \u2502
                                            \n\u2502 \u2502 \u251c\u2500\u2500 \u53c2\u6570\u63a7\u5236: \u2502
                                            \n\u2502 \u2502 \u2502 \u251c\u2500\u2500 max_new_tokens<\/span>=<\/span>512<\/span> (<\/span>\u751f\u6210\u957f\u5ea6)<\/span> \u2502
                                            \n\u2502 \u2502 \u2502 \u251c\u2500\u2500 use_cache<\/span>=<\/span>True (<\/span>\u5f00\u542f KV Cache \u52a0\u901f)<\/span> \u2502
                                            \n\u2502 \u2502 \u2502 \u2514\u2500\u2500 do_sample<\/span>=<\/span>True (<\/span>\u542f\u7528\u968f\u673a\u91c7\u6837)<\/span> \u2502
                                            \n\u2502 \u2502 \u2514\u2500\u2500 ><\/span> \u8ba1\u7b97: [<\/span>INT8 \u77e9\u9635\u4e58\u6cd5]<\/span> <<\/span>–><\/span> [<\/span>FP16 \u6fc0\u6d3b\u503c]<\/span> (<\/span>\u6df7\u5408\u8ba1\u7b97)<\/span> \u2502
                                            \n\u2502 \u2502 \u2502
                                            \n\u2502 \u2514\u2500\u2500 C. \u540e\u5904\u7406 (<\/span>Post-processing)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u89e3\u7801><\/span>: tokenizer.decode (<\/span>Token ID –><\/span> \u6587\u672c)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 <<\/span>\u622a\u65ad><\/span>: remove "<|im_end|>"<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u2514\u2500\u2500 ><\/span> \u8f93\u51fa: \u6253\u5370\u751f\u6210\u5185\u5bb9\u3001\u8017\u65f6\u7edf\u8ba1 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                            \n \u2502
                                            \n \u25bc
                                            \n[<\/span>4<\/span>. \u6027\u80fd\u76d1\u63a7\u4e0e\u5bf9\u6bd4 (<\/span>Metrics &<\/span> Benchmarking)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 A. \u901f\u5ea6\u6307\u6807 (<\/span>Speed Metrics)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 \u8ba1\u7b97: Generate Tokens \/ Elapsed Time \u2502
                                            \n\u2502 \u2514\u2500\u2500 ><\/span> \u7ed3\u679c: xx.xx tokens\/\u79d2 (<\/span>\u901a\u5e38\u6bd4 FP16 \u7565\u6162，\u4f46\u7701\u663e\u5b58)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 B. \u663e\u5b58\u6307\u6807 (<\/span>Memory Metrics)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 \u76d1\u63a7: torch.cuda.max_memory_allocated(<\/span>)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 ><\/span> \u7ed3\u679c: INT8 \u6a21\u5f0f\u4e0b\u663e\u5b58\u5360\u7528\u7ea6\u4e3a FP16 \u7684 50<\/span>%~60% \u2502
                                            \n\u2502 \u2502
                                            \n\u2514\u2500\u2500 C. \u5bf9\u7167\u7ec4 (<\/span>test_fp16_inference)<\/span> \u2502
                                            \n \u2514\u2500\u2500 ><\/span> \u8fd0\u884c\u76f8\u540c\u903b\u8f91，\u4f46\u4e0d\u52a0\u8f7d quantization_config \u4ee5\u4f5c\u5bf9\u6bd4 \u2502
                                            \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

                                            import torch
                                            \nimport time
                                            \nfrom transformers import (
                                            \n AutoModelForCausalLM,
                                            \n AutoTokenizer,
                                            \n BitsAndBytesConfig
                                            \n)
                                            \nfrom dotenv import load_dotenv
                                            \nimport os<\/p>\n

                                            load_dotenv()<\/p>\n

                                            # 1. \u914d\u7f6eINT8\u91cf\u5316
                                            \nbnb_config = BitsAndBytesConfig(
                                            \n load_in_8bit=True, # \u542f\u7528INT8\u91cf\u5316
                                            \n bnb_4bit_use_double_quant=False,
                                            \n bnb_4bit_quant_type="nf4",
                                            \n bnb_4bit_compute_dtype=torch.float16,
                                            \n # KV Cache\u91cf\u5316
                                            \n llm_int8_enable_fp32_cpu_offload=False,
                                            \n llm_int8_skip_modules=["lm_head"], # \u8f93\u51fa\u5c42\u4e0d\u91cf\u5316
                                            \n llm_int8_threshold=6.0,
                                            \n)<\/p>\n

                                            # 2. \u52a0\u8f7d\u6a21\u578b\u548cTokenizer
                                            \nmodel_name = "Qwen\/Qwen2.5-7B-Instruct"
                                            \nmodel_dir = os.getenv("MODEL_DIR", ".\/models\/Qwen2.5-7B-Instruct")<\/p>\n

                                            tokenizer = AutoTokenizer.from_pretrained(
                                            \n model_dir if os.path.exists(model_dir) else model_name,
                                            \n trust_remote_code=True,
                                            \n use_fast=False
                                            \n)<\/p>\n

                                            # \u52a0\u8f7d\u91cf\u5316\u6a21\u578b
                                            \nstart_load = time.time()
                                            \nmodel = AutoModelForCausalLM.from_pretrained(
                                            \n model_dir if os.path.exists(model_dir) else model_name,
                                            \n quantization_config=bnb_config,
                                            \n device_map="auto", # \u81ea\u52a8\u5206\u914d\u8bbe\u5907
                                            \n trust_remote_code=True,
                                            \n torch_dtype=torch.float16,
                                            \n low_cpu_mem_usage=True
                                            \n)
                                            \nend_load = time.time()<\/p>\n

                                            print(f"\u6a21\u578b\u52a0\u8f7d\u8017\u65f6：{end_load – start_load:.2f}\u79d2")
                                            \nprint(f"\u6a21\u578b\u8bbe\u5907：{model.device}")
                                            \nprint(f"\u662f\u5426\u91cf\u5316：{model.is_quantized}")<\/p>\n

                                            # 3. \u6d4b\u8bd5\u63a8\u7406\u901f\u5ea6\u548c\u663e\u5b58
                                            \ndef test_int8_inference():
                                            \n # \u6d4b\u8bd5\u7528\u4f8b
                                            \n prompts = [
                                            \n "\u89e3\u91ca\u5927\u6a21\u578b\u91cf\u5316\u6280\u672f\u7684\u539f\u7406\u548c\u4f18\u7f3a\u70b9",
                                            \n "\u5982\u4f55\u5e73\u8861\u91cf\u5316\u540e\u7684\u7cbe\u5ea6\u635f\u5931\u548c\u901f\u5ea6\u63d0\u5347",
                                            \n "INT8\u91cf\u5316\u76f8\u6bd4INT4\u91cf\u5316\u7684\u4f18\u52bf"
                                            \n ]<\/p>\n

                                            # \u5f00\u542fKV Cache（\u9ed8\u8ba4\u5f00\u542f）
                                            \n model.config.use_cache = True<\/p>\n

                                            total_time = 0
                                            \n total_tokens = 0<\/p>\n

                                            for prompt in prompts:
                                            \n # \u6784\u9020\u8f93\u5165
                                            \n messages = [
                                            \n {"role": "user", "content": prompt}
                                            \n ]
                                            \n input_ids = tokenizer.apply_chat_template(
                                            \n messages,
                                            \n tokenize=True,
                                            \n add_generation_prompt=True,
                                            \n return_tensors="pt"
                                            \n ).to(model.device)<\/p>\n

                                            # \u63a8\u7406
                                            \n start_time = time.time()
                                            \n outputs = model.generate(
                                            \n input_ids=input_ids,
                                            \n max_new_tokens=512,
                                            \n temperature=0.1,
                                            \n top_p=0.95,
                                            \n do_sample=True,
                                            \n use_cache=True, # \u663e\u5f0f\u542f\u7528KV Cache
                                            \n )
                                            \n end_time = time.time()<\/p>\n

                                            # \u89e3\u6790\u8f93\u51fa
                                            \n output_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
                                            \n output_text = output_text.split("<|im_end|>")[-2].strip()<\/p>\n

                                            # \u7edf\u8ba1
                                            \n elapsed_time = end_time – start_time
                                            \n num_tokens = len(outputs[0]) – len(input_ids[0])
                                            \n total_time += elapsed_time
                                            \n total_tokens += num_tokens<\/p>\n

                                            print(f"\\\\n=== \u8f93\u5165：{prompt[:50]}… ===")
                                            \n print(f"\u8017\u65f6：{elapsed_time:.2f}\u79d2")
                                            \n print(f"\u751f\u6210Token\u6570：{num_tokens}")
                                            \n print(f"\u901f\u5ea6：{num_tokens\/elapsed_time:.2f} tokens\/\u79d2")
                                            \n print(f"\u8f93\u51fa：{output_text[:200]}…")<\/p>\n

                                            # \u6c47\u603b
                                            \n avg_speed = total_tokens \/ total_time
                                            \n print(f"\\\\n\u5e73\u5747\u901f\u5ea6：{avg_speed:.2f} tokens\/\u79d2")<\/p>\n

                                            # \u663e\u5b58\u4f7f\u7528
                                            \n if torch.cuda.is_available():
                                            \n mem_used = torch.cuda.max_memory_allocated() \/ 1024 \/ 1024 \/ 1024
                                            \n print(f"\u6700\u5927\u663e\u5b58\u5360\u7528：{mem_used:.2f} GB")<\/p>\n

                                            # 4. \u5bf9\u6bd4\u975e\u91cf\u5316\u7248\u672c
                                            \ndef test_fp16_inference():
                                            \n # \u52a0\u8f7dFP16\u6a21\u578b（\u975e\u91cf\u5316）
                                            \n model_fp16 = AutoModelForCausalLM.from_pretrained(
                                            \n model_dir if os.path.exists(model_dir) else model_name,
                                            \n device_map="auto",
                                            \n trust_remote_code=True,
                                            \n torch_dtype=torch.float16,
                                            \n low_cpu_mem_usage=True
                                            \n )
                                            \n model_fp16.config.use_cache = True<\/p>\n

                                            prompt = "\u89e3\u91ca\u5927\u6a21\u578b\u91cf\u5316\u6280\u672f\u7684\u539f\u7406\u548c\u4f18\u7f3a\u70b9"
                                            \n messages = [{"role": "user", "content": prompt}]
                                            \n input_ids = tokenizer.apply_chat_template(
                                            \n messages,
                                            \n tokenize=True,
                                            \n add_generation_prompt=True,
                                            \n return_tensors="pt"
                                            \n ).to(model_fp16.device)<\/p>\n

                                            start_time = time.time()
                                            \n outputs = model_fp16.generate(
                                            \n input_ids=input_ids,
                                            \n max_new_tokens=512,
                                            \n temperature=0.1,
                                            \n top_p=0.95,
                                            \n do_sample=True,
                                            \n )
                                            \n end_time = time.time()<\/p>\n

                                            elapsed_time = end_time – start_time
                                            \n num_tokens = len(outputs[0]) – len(input_ids[0])
                                            \n speed = num_tokens \/ elapsed_time<\/p>\n

                                            print(f"\\\\n=== FP16\u975e\u91cf\u5316\u6d4b\u8bd5 ===")
                                            \n print(f"\u8017\u65f6：{elapsed_time:.2f}\u79d2")
                                            \n print(f"\u901f\u5ea6：{speed:.2f} tokens\/\u79d2")<\/p>\n

                                            if torch.cuda.is_available():
                                            \n mem_used = torch.cuda.max_memory_allocated() \/ 1024 \/ 1024 \/ 1024
                                            \n print(f"\u6700\u5927\u663e\u5b58\u5360\u7528：{mem_used:.2f} GB")<\/p>\n

                                            if __name__ == "__main__":
                                            \n print("=== INT8\u91cf\u5316\u63a8\u7406\u6d4b\u8bd5 ===")
                                            \n test_int8_inference()
                                            \n print("\\\\n=== FP16\u975e\u91cf\u5316\u63a8\u7406\u6d4b\u8bd5 ===")
                                            \n test_fp16_inference()<\/p>\n

                                            4.2.2 \u6548\u679c\u9a8c\u8bc1\u8f93\u51fa\u793a\u4f8b<\/h5>\n

                                            \u6a21\u578b\u52a0\u8f7d\u8017\u65f6：45.68\u79d2
                                            \n\u6a21\u578b\u8bbe\u5907：cuda:0
                                            \n\u662f\u5426\u91cf\u5316：True<\/p>\n

                                            === INT8\u91cf\u5316\u63a8\u7406\u6d4b\u8bd5 ===
                                            \n=== \u8f93\u5165：\u89e3\u91ca\u5927\u6a21\u578b\u91cf\u5316\u6280\u672f\u7684\u539f\u7406\u548c\u4f18\u7f3a\u70b9… ===
                                            \n\u8017\u65f6：4.28\u79d2
                                            \n\u751f\u6210Token\u6570：325
                                            \n\u901f\u5ea6：75.93 tokens\/\u79d2
                                            \n\u8f93\u51fa：\u5927\u6a21\u578b\u91cf\u5316\u6280\u672f\u662f\u5c06\u6a21\u578b\u6743\u91cd\u4ece\u9ad8\u7cbe\u5ea6（\u5982FP16\/FP32）\u8f6c\u6362\u4e3a\u4f4e\u7cbe\u5ea6（\u5982INT8\/INT4）\u7684\u4f18\u5316\u624b\u6bb5，\u6838\u5fc3\u539f\u7406\u5305\u62ec：1. \u6743\u91cd\u91cf\u5316：\u5c06\u6d6e\u70b9\u6570\u6743\u91cd\u6620\u5c04\u5230\u6574\u6570\u533a\u95f4…<\/p>\n

                                            === \u8f93\u5165：\u5982\u4f55\u5e73\u8861\u91cf\u5316\u540e\u7684\u7cbe\u5ea6\u635f\u5931\u548c\u901f\u5ea6\u63d0\u5347… ===
                                            \n\u8017\u65f6：3.87\u79d2
                                            \n\u751f\u6210Token\u6570：289
                                            \n\u901f\u5ea6：74.68 tokens\/\u79d2
                                            \n\u8f93\u51fa：\u5e73\u8861\u91cf\u5316\u7cbe\u5ea6\u635f\u5931\u548c\u901f\u5ea6\u63d0\u5347\u9700\u4ece\u4ee5\u4e0b\u7ef4\u5ea6\u5165\u624b：1. \u91cf\u5316\u7c7b\u578b\u9009\u62e9：\u52a8\u6001\u91cf\u5316（\u5982INT8）\u7cbe\u5ea6\u635f\u5931\u5c0f，\u9759\u6001\u91cf\u5316（\u5982GPTQ）\u901f\u5ea6\u63d0\u5347\u5927…<\/p>\n

                                            === \u8f93\u5165：INT8\u91cf\u5316\u76f8\u6bd4INT4\u91cf\u5316\u7684\u4f18\u52bf… ===
                                            \n\u8017\u65f6：3.56\u79d2
                                            \n\u751f\u6210Token\u6570：256
                                            \n\u901f\u5ea6：71.91 tokens\/\u79d2
                                            \n\u8f93\u51fa：INT8\u91cf\u5316\u76f8\u6bd4INT4\u91cf\u5316\u7684\u6838\u5fc3\u4f18\u52bf\u5728\u4e8e\u7cbe\u5ea6\u635f\u5931\u66f4\u5c0f，\u5177\u4f53\u4f53\u73b0\u5728：1. \u6570\u503c\u8303\u56f4\u66f4\u5927：INT8\u53ef\u8868\u793a-128~127，INT4\u4ec5-8~7…<\/p>\n

                                            \u5e73\u5747\u901f\u5ea6：74.18 tokens\/\u79d2
                                            \n\u6700\u5927\u663e\u5b58\u5360\u7528：8.24 GB<\/p>\n

                                            === FP16\u975e\u91cf\u5316\u63a8\u7406\u6d4b\u8bd5 ===
                                            \n\u8017\u65f6：7.89\u79d2
                                            \n\u901f\u5ea6：41.19 tokens\/\u79d2
                                            \n\u6700\u5927\u663e\u5b58\u5360\u7528：13.87 GB<\/p>\n

                                            4.3 GPTQ\u9759\u6001\u91cf\u5316（AutoGPTQ）<\/h4>\n

                                            4.3.1 \u4ee3\u7801\u5b9e\u73b0（INT4\u91cf\u5316）<\/h5>\n
                                            \u4ee3\u7801\u5b9e\u73b0\u6d41\u7a0b\u56fe (GPTQ INT4 Quantization Flow)<\/h6>\n

                                            \u2502
                                            \n\u251c\u2500\u2500 \u3010\u7a0b\u5e8f\u5165\u53e3\u3011: if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span>
                                            \n\u2502
                                            \n\u25bc
                                            \n[<\/span>1<\/span>. \u914d\u7f6e\u4e0e\u51c6\u5907\u9636\u6bb5 (<\/span>Configuration &<\/span> Setup)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 A. \u73af\u5883\u521d\u59cb\u5316 \u2502
                                            \n\u2502 \u2514\u2500\u2500 <<\/span>\u52a0\u8f7d\u6587\u4ef6><\/span>: .env (<\/span>\u8bfb\u53d6 MODEL_DIR)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 B. \u91cf\u5316\u53c2\u6570\u8bbe\u5b9a (<\/span>BaseQuantizeConfig)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 bits =<\/span> 4<\/span> (<\/span>INT4 \u7cbe\u5ea6，\u6838\u5fc3\u538b\u7f29\u53c2\u6570)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 group_size =<\/span> 128<\/span> (<\/span>\u5206\u7ec4\u91cf\u5316，\u5e73\u8861\u7cbe\u5ea6\u4e0e\u663e\u5b58)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 desc_act =<\/span> False (<\/span>\u7981\u7528\u6fc0\u6d3b\u91cd\u6392，\u63d0\u5347\u63a8\u7406\u901f\u5ea6)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u2514\u2500\u2500 ><\/span> \u8def\u5f84\u68c0\u67e5: quantized_model_dir \u662f\u5426\u5df2\u5b58\u5728? \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                            \n \u2502
                                            \n \u25bc
                                            \n[<\/span>2<\/span>. \u6a21\u578b\u83b7\u53d6\u7b56\u7565 (<\/span>Model Acquisition Strategy)<\/span>]<\/span> <<\/span>\u2605 \u6838\u5fc3\u5206\u652f><\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 <<\/span>\u5206\u652f A><\/span>: \u9996\u6b21\u8fd0\u884c (<\/span>\u6267\u884c\u91cf\u5316 – Quantization Phase)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u6761\u4ef6><\/span>: \u76ee\u5f55\u4e0d\u5b58\u5728 \u2502
                                            \n\u2502 \u251c\u2500\u2500 1<\/span>. \u52a0\u8f7d\u539f\u6a21: AutoGPTQForCausalLM.from_pretrained (<\/span>FP16)<\/span>\u2502
                                            \n\u2502 \u251c\u2500\u2500 2<\/span>. \u51c6\u5907\u6570\u636e: examples =<\/span> [<\/span>"…"<\/span>, "…"<\/span>]<\/span> (<\/span>\u6821\u51c6\u6570\u636e\u96c6)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 3<\/span>. \u6267\u884c\u91cf\u5316: model.quantize (<\/span>\u8017\u65f6\u64cd\u4f5c)<\/span> \u2502
                                            \n\u2502 \u2502 \u2514\u2500\u2500 <<\/span>\u8ba1\u7b97><\/span>: \u57fa\u4e8e Hessian \u77e9\u9635\u4f18\u5316\u6743\u91cd，\u6700\u5c0f\u5316\u8bef\u5dee \u2502
                                            \n\u2502 \u2514\u2500\u2500 4<\/span>. \u5e8f\u5217\u5316\u4fdd\u5b58: model.save_quantized (<\/span>\u751f\u6210 INT4 \u6743\u91cd)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 <<\/span>\u5206\u652f B><\/span>: \u540e\u7eed\u8fd0\u884c (<\/span>\u76f4\u63a5\u52a0\u8f7d – Loading Phase)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u6761\u4ef6><\/span>: \u76ee\u5f55\u5df2\u5b58\u5728 \u2502
                                            \n\u2502 \u251c\u2500\u2500 1<\/span>. \u5feb\u901f\u52a0\u8f7d: AutoGPTQForCausalLM.from_quantized \u2502
                                            \n\u2502 \u2502 \u251c\u2500\u2500 use_safetensors =<\/span> True (<\/span>\u5b89\u5168\u5feb\u901f\u52a0\u8f7d)<\/span> \u2502
                                            \n\u2502 \u2502 \u2514\u2500\u2500 device_map =<\/span> "auto"<\/span> (<\/span>\u81ea\u52a8\u6620\u5c04\u663e\u5b58)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 2<\/span>. \u6ce8\u5165\u6838\u5fc3: GPTQ Kernel (<\/span>ExLlama\/AutoGPTQ CUDA\u6838\u5fc3)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u2514\u2500\u2500 ><\/span> \u6a21\u578b\u5c31\u7eea: model (<\/span>INT4 \u6743\u91cd + FP16 \u6fc0\u6d3b\u8ba1\u7b97)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                            \n \u2502
                                            \n \u25bc
                                            \n[<\/span>3<\/span>. \u63a8\u7406\u6267\u884c\u6d41\u7a0b (<\/span>Inference Execution)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 A. \u7ba1\u9053\u6784\u5efa (<\/span>Pipeline Setup)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u914d\u7f6e><\/span>: model.config.use_cache =<\/span> True (<\/span>\u5f00\u542f KV Cache)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 <<\/span>\u5c01\u88c5><\/span>: TextGenerationPipeline (<\/span>HuggingFace \u7ba1\u9053)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 B. \u751f\u6210\u5faa\u73af (<\/span>test_gptq_inference)<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u8f93\u5165><\/span>: Prompts [<\/span>"GPTQ\u4f18\u52bf…"<\/span>, "\u6ce8\u610f\u4e8b\u9879…"<\/span>]<\/span> \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u9884\u5904\u7406><\/span>: tokenizer.apply_chat_template \u2502
                                            \n\u2502 \u2502 \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u6838\u5fc3\u751f\u6210><\/span>: pipeline(<\/span>..<\/span>.)<\/span> \u2502
                                            \n\u2502 \u2502 \u251c\u2500\u2500 max_new_tokens =<\/span> 512<\/span> \u2502
                                            \n\u2502 \u2502 \u251c\u2500\u2500 \u7b97\u5b50\u52a0\u901f: INT4 \u77e9\u9635\u4e58\u6cd5 (<\/span>W4A16 GEMM)<\/span> \u2502
                                            \n\u2502 \u2502 \u2514\u2500\u2500 \u663e\u5b58\u4f18\u52bf: \u6743\u91cd\u4f53\u79ef\u7ea6\u4e3a FP16 \u7684 1<\/span>\/4 \u2502
                                            \n\u2502 \u2502 \u2502
                                            \n\u2502 \u2514\u2500\u2500 <<\/span>\u540e\u5904\u7406><\/span>: split(<\/span>"<|im_end|>"<\/span>)<\/span> –><\/span> \u83b7\u53d6\u7eaf\u51c0\u6587\u672c \u2502
                                            \n\u2502 \u2502
                                            \n\u2514\u2500\u2500 ><\/span> \u8f93\u51fa\u7ed3\u679c: \u6253\u5370 Text &<\/span> Speed (<\/span>tokens\/s)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                            \n \u2502
                                            \n \u25bc
                                            \n[<\/span>4<\/span>. \u6027\u80fd\u76d1\u63a7 (<\/span>Metrics &<\/span> Analysis)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 A. \u901f\u5ea6\u7edf\u8ba1 \u2502
                                            \n\u2502 \u251c\u2500\u2500 \u8ba1\u7b97: Total Tokens \/ Total Time \u2502
                                            \n\u2502 \u2514\u2500\u2500 ><\/span> \u9884\u671f: 100<\/span>+ tokens\/s (<\/span>\u901a\u5e38\u5feb\u4e8e bitsandbytes \u52a8\u6001\u91cf\u5316)<\/span> \u2502
                                            \n\u2502 \u2502
                                            \n\u251c\u2500\u2500 B. \u663e\u5b58\u76d1\u63a7 \u2502
                                            \n\u2502 \u251c\u2500\u2500 <<\/span>\u8c03\u7528><\/span>: torch.cuda.max_memory_allocated(<\/span>)<\/span> \u2502
                                            \n\u2502 \u2514\u2500\u2500 ><\/span> \u9884\u671f: 7B \u6a21\u578b\u4ec5\u9700 ~5GB \u663e\u5b58 (<\/span>\u5927\u5e45\u4f4e\u4e8e FP16 \u7684 14GB)<\/span> \u2502
                                            \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

                                            import torch
                                            \nimport time
                                            \nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
                                            \nfrom transformers import AutoTokenizer, TextGenerationPipeline
                                            \nimport os
                                            \nfrom dotenv import load_dotenv<\/p>\n

                                            load_dotenv()<\/p>\n

                                            # 1. \u914d\u7f6eGPTQ\u91cf\u5316\u53c2\u6570
                                            \nquantize_config = BaseQuantizeConfig(
                                            \n bits=4, # INT4\u91cf\u5316
                                            \n group_size=128, # \u91cf\u5316\u5206\u7ec4\u5927\u5c0f
                                            \n desc_act=False, # \u7981\u7528\u6fc0\u6d3b\u91cf\u5316（\u51cf\u5c11\u7cbe\u5ea6\u635f\u5931）
                                            \n model_name_or_path="Qwen\/Qwen2.5-7B-Instruct",
                                            \n model_dir=os.getenv("MODEL_DIR", ".\/models\/Qwen2.5-7B-Instruct"),
                                            \n use_triton=False, # \u7981\u7528Triton（\u517c\u5bb9\u6027\u66f4\u597d）
                                            \n bitsandbytes_backend=False,
                                            \n)<\/p>\n

                                            # 2. \u52a0\u8f7dTokenizer
                                            \ntokenizer = AutoTokenizer.from_pretrained(
                                            \n quantize_config.model_dir if os.path.exists(quantize_config.model_dir) else quantize_config.model_name_or_path,
                                            \n trust_remote_code=True,
                                            \n use_fast=False
                                            \n)<\/p>\n

                                            # 3. \u91cf\u5316\u6a21\u578b（\u9996\u6b21\u8fd0\u884c\u9700\u91cf\u5316，\u540e\u7eed\u53ef\u52a0\u8f7d\u91cf\u5316\u540e\u7684\u6a21\u578b）
                                            \nquantized_model_dir = ".\/models\/Qwen2.5-7B-Instruct-GPTQ-4bit"<\/p>\n

                                            if not os.path.exists(quantized_model_dir):
                                            \n # \u52a0\u8f7d\u539f\u59cb\u6a21\u578b
                                            \n model = AutoGPTQForCausalLM.from_pretrained(
                                            \n quantize_config.model_dir if os.path.exists(quantize_config.model_dir) else quantize_config.model_name_or_path,
                                            \n quantize_config=quantize_config,
                                            \n device_map="auto",
                                            \n trust_remote_code=True,
                                            \n torch_dtype=torch.float16,
                                            \n )<\/p>\n

                                            # \u51c6\u5907\u91cf\u5316\u6570\u636e\u96c6（\u4f7f\u7528\u6837\u4f8b\u6570\u636e）
                                            \n examples = [
                                            \n tokenizer("\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u7684\u6838\u5fc3\u6280\u672f\u6709\u54ea\u4e9b？"),
                                            \n tokenizer("KV Cache\u7684\u5de5\u4f5c\u539f\u7406\u662f\u4ec0\u4e48？"),
                                            \n tokenizer("\u5982\u4f55\u4f18\u5316\u91cf\u5316\u540e\u7684\u6a21\u578b\u7cbe\u5ea6？"),
                                            \n tokenizer("\u6a21\u578b\u84b8\u998f\u548c\u91cf\u5316\u7684\u533a\u522b\u662f\u4ec0\u4e48？"),
                                            \n tokenizer("\u5927\u6a21\u578b\u90e8\u7f72\u7684\u663e\u5b58\u4f18\u5316\u7b56\u7565\u6709\u54ea\u4e9b？")
                                            \n ]<\/p>\n

                                            # \u6267\u884c\u91cf\u5316
                                            \n start_quant = time.time()
                                            \n model.quantize(
                                            \n examples,
                                            \n batch_size=1,
                                            \n use_triton=False
                                            \n )
                                            \n end_quant = time.time()
                                            \n print(f"\u91cf\u5316\u8017\u65f6：{end_quant – start_quant:.2f}\u79d2")<\/p>\n

                                            # \u4fdd\u5b58\u91cf\u5316\u540e\u7684\u6a21\u578b
                                            \n model.save_quantized(quantized_model_dir)
                                            \n tokenizer.save_pretrained(quantized_model_dir)
                                            \nelse:
                                            \n # \u52a0\u8f7d\u5df2\u91cf\u5316\u7684\u6a21\u578b
                                            \n start_load = time.time()
                                            \n model = AutoGPTQForCausalLM.from_quantized(
                                            \n quantized_model_dir,
                                            \n device_map="auto",
                                            \n trust_remote_code=True,
                                            \n use_safetensors=True,
                                            \n torch_dtype=torch.float16,
                                            \n )
                                            \n end_load = time.time()
                                            \n print(f"\u52a0\u8f7d\u91cf\u5316\u6a21\u578b\u8017\u65f6：{end_load – start_load:.2f}\u79d2")<\/p>\n

                                            # 4. \u542f\u7528KV Cache
                                            \nmodel.config.use_cache = True<\/p>\n

                                            # 5. \u521b\u5efa\u63a8\u7406\u7ba1\u9053
                                            \npipeline = TextGenerationPipeline(
                                            \n model=model,
                                            \n tokenizer=tokenizer,
                                            \n device=model.device,
                                            \n)<\/p>\n

                                            # 6. \u6d4b\u8bd5GPTQ\u91cf\u5316\u63a8\u7406
                                            \ndef test_gptq_inference():
                                            \n prompts = [
                                            \n "\u8be6\u7ec6\u8bf4\u660eGPTQ\u91cf\u5316\u76f8\u6bd4\u52a8\u6001INT8\u91cf\u5316\u7684\u4f18\u52bf",
                                            \n "INT4 GPTQ\u91cf\u5316\u5728\u5b9e\u9645\u90e8\u7f72\u4e2d\u7684\u6ce8\u610f\u4e8b\u9879",
                                            \n "\u5982\u4f55\u8bc4\u4f30\u91cf\u5316\u6a21\u578b\u7684\u7cbe\u5ea6\u635f\u5931"
                                            \n ]<\/p>\n

                                            total_time = 0
                                            \n total_tokens = 0<\/p>\n

                                            for prompt in prompts:
                                            \n # \u6784\u9020\u8f93\u5165
                                            \n messages = [{"role": "user", "content": prompt}]
                                            \n input_text = tokenizer.apply_chat_template(
                                            \n messages,
                                            \n add_generation_prompt=True,
                                            \n tokenize=False
                                            \n )<\/p>\n

                                            # \u63a8\u7406
                                            \n start_time = time.time()
                                            \n outputs = pipeline(
                                            \n input_text,
                                            \n max_new_tokens=512,
                                            \n temperature=0.1,
                                            \n top_p=0.95,
                                            \n do_sample=True,
                                            \n use_cache=True,
                                            \n )
                                            \n end_time = time.time()<\/p>\n

                                            # \u89e3\u6790\u8f93\u51fa
                                            \n output_text = outputs[0]["generated_text"].split("<|im_end|>")[-2].strip()
                                            \n input_tokens = len(tokenizer.encode(input_text))
                                            \n output_tokens = len(tokenizer.encode(output_text))
                                            \n total_tokens += output_tokens
                                            \n elapsed_time = end_time – start_time
                                            \n total_time += elapsed_time<\/p>\n

                                            print(f"\\\\n=== \u8f93\u5165：{prompt[:50]}… ===")
                                            \n print(f"\u8017\u65f6：{elapsed_time:.2f}\u79d2")
                                            \n print(f"\u751f\u6210Token\u6570：{output_tokens}")
                                            \n print(f"\u901f\u5ea6：{output_tokens\/elapsed_time:.2f} tokens\/\u79d2")
                                            \n print(f"\u8f93\u51fa：{output_text[:200]}…")<\/p>\n

                                            # \u6c47\u603b
                                            \n avg_speed = total_tokens \/ total_time
                                            \n print(f"\\\\nGPTQ INT4\u5e73\u5747\u901f\u5ea6：{avg_speed:.2f} tokens\/\u79d2")<\/p>\n

                                            # \u663e\u5b58\u4f7f\u7528
                                            \n if torch.cuda.is_available():
                                            \n mem_used = torch.cuda.max_memory_allocated() \/ 1024 \/ 1024 \/ 1024
                                            \n print(f"\u6700\u5927\u663e\u5b58\u5360\u7528：{mem_used:.2f} GB")<\/p>\n

                                            if __name__ == "__main__":
                                            \n print("=== GPTQ INT4\u91cf\u5316\u63a8\u7406\u6d4b\u8bd5 ===")
                                            \n test_gptq_inference()<\/p>\n

                                            4.3.2 \u6548\u679c\u9a8c\u8bc1\u8f93\u51fa\u793a\u4f8b<\/h5>\n

                                            \u52a0\u8f7d\u91cf\u5316\u6a21\u578b\u8017\u65f6：18.76\u79d2<\/p>\n

                                            === GPTQ INT4\u91cf\u5316\u63a8\u7406\u6d4b\u8bd5 ===
                                            \n=== \u8f93\u5165：\u8be6\u7ec6\u8bf4\u660eGPTQ\u91cf\u5316\u76f8\u6bd4\u52a8\u6001INT8\u91cf\u5316\u7684\u4f18\u52bf… ===
                                            \n\u8017\u65f6：2.98\u79d2
                                            \n\u751f\u6210Token\u6570：345
                                            \n\u901f\u5ea6：115.77 tokens\/\u79d2
                                            \n\u8f93\u51fa：GPTQ（GPT Quantization）\u662f\u4e00\u79cd\u9759\u6001\u91cf\u5316\u65b9\u6cd5，\u76f8\u6bd4\u52a8\u6001INT8\u91cf\u5316\u6709\u4ee5\u4e0b\u6838\u5fc3\u4f18\u52bf：1. \u901f\u5ea6\u66f4\u5feb：GPTQ\u5728\u91cf\u5316\u65f6\u9884\u8ba1\u7b97\u6743\u91cd\u7f29\u653e\u56e0\u5b50，\u63a8\u7406\u65f6\u65e0\u9700\u52a8\u6001\u8ba1\u7b97…<\/p>\n

                                            === \u8f93\u5165：INT4 GPTQ\u91cf\u5316\u5728\u5b9e\u9645\u90e8\u7f72\u4e2d\u7684\u6ce8\u610f\u4e8b\u9879… ===
                                            \n\u8017\u65f6：2.76\u79d2
                                            \n\u751f\u6210Token\u6570：312
                                            \n\u901f\u5ea6：113.04 tokens\/\u79d2
                                            \n\u8f93\u51fa：INT4 GPTQ\u91cf\u5316\u90e8\u7f72\u9700\u6ce8\u610f\u4ee5\u4e0b\u4e8b\u9879：1. \u7cbe\u5ea6\u76d1\u63a7：INT4\u91cf\u5316\u7cbe\u5ea6\u635f\u5931\u7565\u9ad8\u4e8eINT8，\u9700\u9488\u5bf9\u6838\u5fc3\u573a\u666f\u505a\u7cbe\u5ea6\u9a8c\u8bc1…<\/p>\n

                                            === \u8f93\u5165：\u5982\u4f55\u8bc4\u4f30\u91cf\u5316\u6a21\u578b\u7684\u7cbe\u5ea6\u635f\u5931… ===
                                            \n\u8017\u65f6：2.54\u79d2
                                            \n\u751f\u6210Token\u6570：289
                                            \n\u901f\u5ea6：113.78 tokens\/\u79d2
                                            \n\u8f93\u51fa：\u8bc4\u4f30\u91cf\u5316\u6a21\u578b\u7cbe\u5ea6\u635f\u5931\u7684\u65b9\u6cd5\u5305\u62ec：1. \u57fa\u51c6\u6d4b\u8bd5：\u5728\u6807\u51c6\u6570\u636e\u96c6（\u5982MMLU\u3001C-Eval）\u4e0a\u5bf9\u6bd4\u539f\u59cb\u6a21\u578b\u548c\u91cf\u5316\u6a21\u578b\u7684\u51c6\u786e\u7387…<\/p>\n

                                            GPTQ INT4\u5e73\u5747\u901f\u5ea6：114.19 tokens\/\u79d2
                                            \n\u6700\u5927\u663e\u5b58\u5360\u7528：4.87 GB<\/p>\n

                                            \u4e94\u3001\u6a21\u578b\u84b8\u998f\u5b9e\u6218（\u4ece7B\u84b8\u998f\u52301.8B）<\/h3>\n

                                            5.1 \u6a21\u578b\u84b8\u998f\u6838\u5fc3\u539f\u7406<\/h4>\n

                                            \u6a21\u578b\u84b8\u998f\u662f\u5c06\u5927\u6a21\u578b（\u6559\u5e08\u6a21\u578b，\u5982Qwen2.5-7B）\u7684\u77e5\u8bc6\u8fc1\u79fb\u5230\u5c0f\u6a21\u578b（\u5b66\u751f\u6a21\u578b，\u5982Qwen2.5-1.8B）\u7684\u8fc7\u7a0b，\u6838\u5fc3\u5305\u62ec：<\/p>\n

                                              \n
                                            • \n

                                              \u77e5\u8bc6\u84b8\u998f：\u5339\u914d\u6559\u5e08\u6a21\u578b\u548c\u5b66\u751f\u6a21\u578b\u7684\u8f93\u51fa\u5206\u5e03（Soft Target）<\/p>\n<\/li>\n

                                            • \n

                                              \u4e2d\u95f4\u5c42\u84b8\u998f：\u5339\u914d\u6559\u5e08\u6a21\u578b\u548c\u5b66\u751f\u6a21\u578b\u7684\u9690\u85cf\u5c42\u7279\u5f81<\/p>\n<\/li>\n

                                            • \n

                                              \u5bf9\u8bdd\u84b8\u998f：\u9488\u5bf9\u5bf9\u8bdd\u573a\u666f\u7684\u4e13\u7528\u84b8\u998f\u7b56\u7565（\u4fdd\u7559\u591a\u8f6e\u4ea4\u4e92\u80fd\u529b）<\/p>\n<\/li>\n<\/ul>\n

                                              5.2 \u84b8\u998f\u73af\u5883\u914d\u7f6e<\/h4>\n

                                              # \u989d\u5916\u5b89\u88c5\u84b8\u998f\u4f9d\u8d56
                                              \npip install trl==0.8.6 peft==0.11.1 accelerate==0.31.0
                                              \npip install datasets==2.20.0 evaluate==0.4.2<\/p>\n

                                              5.3 \u4ee3\u7801\u5b9e\u73b0（Qwen2.5-7B \u2192 Qwen2.5-1.8B）<\/h4>\n

                                              5.3.1 \u5b8c\u6574\u84b8\u998f\u4ee3\u7801<\/h5>\n
                                              \u5b8c\u6574\u84b8\u998f\u6d41\u7a0b\u56fe (Knowledge Distillation Flow)<\/h6>\n

                                              \u2502
                                              \n\u251c\u2500\u2500 \u3010\u7a0b\u5e8f\u5165\u53e3\u3011: if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span>
                                              \n\u2502
                                              \n\u25bc
                                              \n[<\/span>1<\/span>. \u57fa\u7840\u67b6\u6784\u6784\u5efa (<\/span>Infrastructure Setup)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 A. \u5168\u5c40\u914d\u7f6e (<\/span>Config &<\/span> Environment)<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u52a0\u8f7d\u914d\u7f6e><\/span>: DistillationConfig \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u53c2\u6570\u8bbe\u5b9a><\/span>: Temperature<\/span>=<\/span>2.0<\/span> (<\/span>\u8f6f\u5316\u6982\u7387\u5206\u5e03)<\/span>, Alpha<\/span>=<\/span>0.3<\/span> \u2502
                                              \n\u2502 \u2514\u2500\u2500 <<\/span>\u786c\u4ef6\u68c0\u6d4b><\/span>: device<\/span>=<\/span>"cuda"<\/span> (<\/span>FP16 \u6df7\u5408\u7cbe\u5ea6\u5f00\u542f)<\/span> \u2502
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 B. \u53cc\u6a21\u578b\u52a0\u8f7d (<\/span>Dual-Model Loading)<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 🎓 \u6559\u5e08\u6a21\u578b (<\/span>Teacher Model)<\/span> \u2502
                                              \n\u2502 \u2502 \u251c\u2500\u2500 <<\/span>\u52a0\u8f7d><\/span>: "Qwen2.5-7B-Instruct"<\/span> \u2502
                                              \n\u2502 \u2502 \u251c\u2500\u2500 <<\/span>\u72b6\u6001><\/span>: model.eval(<\/span>)<\/span> + requires_grad<\/span>=<\/span>False (<\/span>\u51bb\u7ed3\u6743\u91cd)<\/span> \u2502
                                              \n\u2502 \u2502 \u2514\u2500\u2500 ><\/span> \u4f5c\u7528: \u63d0\u4f9b\u9ad8\u8d28\u91cf\u7684 Logits (<\/span>Soft Targets)<\/span> \u2502
                                              \n\u2502 \u2502 \u2502
                                              \n\u2502 \u2514\u2500\u2500 🎒 \u5b66\u751f\u6a21\u578b (<\/span>Student Model)<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u52a0\u8f7d><\/span>: "Qwen2.5-1.8B-Instruct"<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u4f18\u5316><\/span>: PEFT \/ LoRA (<\/span>\u4f4e\u79e9\u9002\u914d)<\/span> \u2502
                                              \n\u2502 \u2502 \u251c\u2500\u2500 r<\/span>=<\/span>64<\/span>, alpha<\/span>=<\/span>128<\/span>, modules<\/span>=<\/span>[<\/span>"q_proj"<\/span>, "v_proj"<\/span>..<\/span>.]<\/span> \u2502
                                              \n\u2502 \u2502 \u2514\u2500\u2500 ><\/span> \u7ed3\u679c: \u4ec5\u8bad\u7ec3 1.04<\/span>% \u7684\u53c2\u6570 (<\/span>\u8f7b\u91cf\u5316\u8bad\u7ec3)<\/span> \u2502
                                              \n\u2502 \u2514\u2500\u2500 ><\/span> \u4f5c\u7528: \u5b66\u4e60\u6559\u5e08\u7684\u6982\u7387\u5206\u5e03 + \u771f\u5b9e\u6807\u7b7e \u2502
                                              \n\u2502 \u2502
                                              \n\u2514\u2500\u2500 ><\/span> \u51c6\u5907\u5c31\u7eea: Tokenizer (<\/span>pad_token=<\/span>eos_token)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                              \n \u2502
                                              \n \u25bc
                                              \n[<\/span>2<\/span>. \u77e5\u8bc6\u63d0\u53d6\u4e0e\u6570\u636e\u6d41 (<\/span>Knowledge Extraction Pipeline)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 <<\/span>\u8f93\u5165><\/span>: "YeungNLP\/firefly-train-1.1M"<\/span> (<\/span>\u6307\u4ee4\u5fae\u8c03\u6570\u636e\u96c6)<\/span> \u2502
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 \u21bb \u9884\u5904\u7406\u5faa\u73af (<\/span>preprocess_function)<\/span> \u2502
                                              \n\u2502 \u2502 \u2502
                                              \n\u2502 \u251c\u2500\u2500 A. \u683c\u5f0f\u5316 (<\/span>Formatting)<\/span> \u2502
                                              \n\u2502 \u2502 \u2514\u2500\u2500 <<\/span>\u6a21\u7248><\/span>: "<|im_start|>user…<|im_end|>"<\/span> (<\/span>ChatML\u683c\u5f0f)<\/span> \u2502
                                              \n\u2502 \u2502 \u2502
                                              \n\u2502 \u251c\u2500\u2500 B. \u6559\u5e08\u63a8\u7406 (<\/span>Teacher Forward Pass)<\/span> <<\/span>\u2605 \u6838\u5fc3\u6b65\u9aa4><\/span> \u2502
                                              \n\u2502 \u2502 \u251c\u2500\u2500 <<\/span>\u8f93\u5165><\/span>: Tokenized Input IDs \u2502
                                              \n\u2502 \u2502 \u251c\u2500\u2500 <<\/span>\u8ba1\u7b97><\/span>: with torch.no_grad(<\/span>)<\/span>: teacher_model(<\/span>input)<\/span> \u2502
                                              \n\u2502 \u2502 \u2514\u2500\u2500 ><\/span> \u8f93\u51fa: Teacher Logits (<\/span>\u77e5\u8bc6\u7684\u8f7d\u4f53)<\/span> \u2502
                                              \n\u2502 \u2502 \u2502
                                              \n\u2502 \u2514\u2500\u2500 C. \u6570\u636e\u6253\u5305 (<\/span>Packaging)<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 labels: \u771f\u5b9e\u6807\u7b7e (<\/span>Hard Targets)<\/span> \u2502
                                              \n\u2502 \u2514\u2500\u2500 teacher_logits: \u6559\u5e08\u8f93\u51fa (<\/span>Soft Targets)<\/span> \u2502
                                              \n\u2502 \u2502
                                              \n\u2514\u2500\u2500 ><\/span> \u8f93\u51fa: \u5305\u542b\u53cc\u91cd\u76d1\u7763\u4fe1\u53f7\u7684 DataLoader \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                              \n \u2502
                                              \n \u25bc
                                              \n[<\/span>3<\/span>. \u84b8\u998f\u8bad\u7ec3\u5faa\u73af (<\/span>Distillation Training Loop)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 <<\/span>\u7c7b><\/span>: DistillationTrainer (<\/span>\u7ee7\u627f\u81ea SFTTrainer)<\/span> \u2502
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 \u21bb \u8bad\u7ec3\u6b65 (<\/span>Training Step)<\/span> \u2502
                                              \n\u2502 \u2502 \u2502
                                              \n\u2502 \u251c\u2500\u2500 1<\/span>. \u5b66\u751f\u63a8\u7406 (<\/span>Student Forward)<\/span> \u2502
                                              \n\u2502 \u2502 \u251c\u2500\u2500 <<\/span>\u8f93\u5165><\/span>: batch[<\/span>"input_ids"<\/span>]<\/span> \u2502
                                              \n\u2502 \u2502 \u2514\u2500\u2500 ><\/span> \u8f93\u51fa: Student Logits \u2502
                                              \n\u2502 \u2502 \u2502
                                              \n\u2502 \u251c\u2500\u2500 2<\/span>. \u635f\u5931\u8ba1\u7b97 (<\/span>Compute Loss)<\/span> <<\/span>\u2605 \u84b8\u998f\u6838\u5fc3\u516c\u5f0f><\/span> \u2502
                                              \n\u2502 \u2502 \u2502 \u2502
                                              \n\u2502 \u2502 \u251c\u2500\u2500 L_soft (<\/span>\u77e5\u8bc6\u5339\u914d\u635f\u5931)<\/span>: KL Divergence \u2502
                                              \n\u2502 \u2502 \u2502 \u251c\u2500\u2500 KL(<\/span>Student_Logits\/T ||<\/span> Teacher_Logits\/T)<\/span> \u2502
                                              \n\u2502 \u2502 \u2502 \u2514\u2500\u2500 ><\/span> \u76ee\u7684: \u8ba9\u5b66\u751f\u6a21\u4eff\u6559\u5e08\u7684\u201c\u6697\u77e5\u8bc6\u201d(<\/span>\u8f93\u51fa\u5206\u5e03)<\/span> \u2502
                                              \n\u2502 \u2502 \u2502 \u2502
                                              \n\u2502 \u2502 \u251c\u2500\u2500 L_hard (<\/span>\u57fa\u7840\u4efb\u52a1\u635f\u5931)<\/span>: Cross Entropy \u2502
                                              \n\u2502 \u2502 \u2502 \u251c\u2500\u2500 CE(<\/span>Student_Logits, True_Labels)<\/span> \u2502
                                              \n\u2502 \u2502 \u2502 \u2514\u2500\u2500 ><\/span> \u76ee\u7684: \u4fdd\u8bc1\u751f\u6210\u7684\u51c6\u786e\u6027 \u2502
                                              \n\u2502 \u2502 \u2502 \u2502
                                              \n\u2502 \u2502 \u2514\u2500\u2500 ><\/span> \u603bLoss =<\/span> 0.7<\/span> * L_hard + 0.3<\/span> * L_soft \u2502
                                              \n\u2502 \u2502 \u2502
                                              \n\u2502 \u2514\u2500\u2500 3<\/span>. \u53cd\u5411\u4f20\u64ad (<\/span>Backprop)<\/span> \u2502
                                              \n\u2502 \u2514\u2500\u2500 <<\/span>\u66f4\u65b0><\/span>: \u4ec5\u66f4\u65b0\u5b66\u751f\u6a21\u578b\u7684 LoRA \u9002\u914d\u5c42\u6743\u91cd \u2502
                                              \n\u2502 \u2502
                                              \n\u2514\u2500\u2500 ><\/span> \u7ed3\u679c: \u8bad\u7ec3\u5b8c\u6210\u7684\u6a21\u578b\u4fdd\u5b58\u81f3 output_dir \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                              \n \u2502
                                              \n \u25bc
                                              \n[<\/span>4<\/span>. \u9a8c\u8bc1\u4e0e\u6d4b\u8bd5\u9636\u6bb5 (<\/span>Validation &<\/span> Testing)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 A. \u6a21\u578b\u52a0\u8f7d \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u8bfb\u53d6><\/span>: output_dir (<\/span>".\/models\/Qwen2.5-1.8B-Distilled"<\/span>)<\/span> \u2502
                                              \n\u2502 \u2514\u2500\u2500 <<\/span>\u5408\u5e76><\/span>: Base Model + LoRA Weights \u2502
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 B. \u63a8\u7406\u9a8c\u8bc1 (<\/span>test_distilled_model)<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u8f93\u5165><\/span>: "\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u7684\u6838\u5fc3\u6280\u672f\u6709\u54ea\u4e9b？"<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u751f\u6210><\/span>: model.generate (<\/span>use_cache=<\/span>True)<\/span> \u2502
                                              \n\u2502 \u2514\u2500\u2500 ><\/span> \u89c2\u6d4b: \u68c0\u67e5 1<\/span>.8B \u6a21\u578b\u662f\u5426\u5b66\u4f1a\u4e86 7B \u6a21\u578b\u7684\u903b\u8f91\u8868\u8fbe\u80fd\u529b \u2502
                                              \n\u2502 \u2502
                                              \n\u2514\u2500\u2500 ><\/span> \u6700\u7ec8\u4ea7\u51fa: \u8f7b\u91cf\u5316\u3001\u9ad8\u667a\u5546\u7684\u8fb9\u7f18\u7aef\u6a21\u578b \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

                                              \u6d41\u7a0b\u56fe\u5173\u952e\u70b9\u89e3\u6790 (Key Highlights)<\/h6>\n
                                            • \n

                                              \u53cc\u6a21\u578b\u67b6\u6784 (Dual-Model)：\u56fe\u8868\u6e05\u6670\u533a\u5206\u4e86 Teacher（\u53ea\u8bfb\/\u51bb\u7ed3）\u548c Student（\u53ef\u5199\/\u8bad\u7ec3）\u7684\u72b6\u6001\u3002<\/p>\n<\/li>\n

                                            • \n

                                              Soft Targets \u751f\u6210\u4f4d\u7f6e：\u6839\u636e\u60a8\u7684\u4ee3\u7801，Teacher Logits \u662f\u5728 \u6570\u636e\u9884\u5904\u7406\u9636\u6bb5 (Phase 2) \u751f\u6210\u7684，\u8fd9\u610f\u5473\u7740\u8bad\u7ec3\u65f6 GPU \u663e\u5b58\u538b\u529b\u8f83\u5c0f（\u56e0\u4e3a\u4e0d\u9700\u8981\u540c\u65f6\u628a\u4e24\u4e2a\u5b8c\u6574\u5927\u6a21\u578b\u653e\u8fdb\u663e\u5b58\u505a\u52a8\u6001\u524d\u5411\u4f20\u64ad），\u8fd9\u662f\u4e00\u79cd\u9ad8\u6548\u7684 Offline Distillation（\u79bb\u7ebf\u84b8\u998f） \u6216 Cached Distillation \u7b56\u7565\u3002<\/p>\n<\/li>\n

                                            • \n

                                              \u6df7\u5408\u635f\u5931\u51fd\u6570 (Hybrid Loss)：\u5728 Phase 3 \u4e2d，\u5c55\u793a\u4e86 <\/p>\n

                                              L<\/p>\n

                                              o<\/p>\n

                                              s<\/p>\n

                                              s<\/p>\n

                                              =<\/p>\n

                                              0.7<\/p>\n

                                              \u00d7<\/p>\n

                                              C<\/p>\n

                                              E<\/p>\n

                                              +<\/p>\n

                                              0.3<\/p>\n

                                              \u00d7<\/p>\n

                                              K<\/p>\n

                                              L<\/p>\n

                                              Loss = 0.7 \\\\times CE + 0.3 \\\\times KL <\/p>\n

                                              <\/span><\/span>L<\/span>oss<\/span><\/span>=<\/span><\/span><\/span><\/span>0.7<\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>CE<\/span><\/span>+<\/span><\/span><\/span><\/span>0.3<\/span><\/span>\u00d7<\/span><\/span><\/span><\/span>K<\/span>L<\/span><\/span><\/span><\/span><\/span><\/span> \u7684\u8ba1\u7b97\u903b\u8f91，\u8fd9\u662f\u77e5\u8bc6\u84b8\u998f\u7684\u6570\u5b66\u672c\u8d28\u3002<\/p>\n<\/li>\n

                                              import os
                                              \nimport torch
                                              \nimport time
                                              \nfrom dataclasses import dataclass, field
                                              \nfrom typing import Optional
                                              \nimport transformers
                                              \nfrom transformers import (
                                              \n AutoModelForCausalLM,
                                              \n AutoTokenizer,
                                              \n TrainingArguments,
                                              \n Trainer,
                                              \n DataCollatorForLanguageModeling,
                                              \n)
                                              \nfrom trl import SFTTrainer, DataCollatorForCompletionOnlyLM
                                              \nfrom datasets import load_dataset
                                              \nimport peft
                                              \nfrom peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
                                              \nfrom dotenv import load_dotenv<\/p>\n

                                              load_dotenv()<\/p>\n

                                              # 1. \u914d\u7f6e\u53c2\u6570
                                              \n@dataclass
                                              \nclass DistillationConfig:
                                              \n # \u6a21\u578b\u914d\u7f6e
                                              \n teacher_model_name: str = "Qwen\/Qwen2.5-7B-Instruct"
                                              \n student_model_name: str = "Qwen\/Qwen2.5-1.8B-Instruct"
                                              \n model_dir: str = os.getenv("MODEL_DIR", ".\/models")
                                              \n output_dir: str = ".\/models\/Qwen2.5-1.8B-Distilled"
                                              \n # \u6570\u636e\u914d\u7f6e
                                              \n dataset_name: str = "YeungNLP\/firefly-train-1.1M" # \u4e2d\u6587\u5bf9\u8bdd\u6570\u636e\u96c6
                                              \n max_seq_length: int = 1024
                                              \n # \u8bad\u7ec3\u914d\u7f6e
                                              \n batch_size: int = 4
                                              \n gradient_accumulation_steps: int = 4
                                              \n learning_rate: float = 2e-4
                                              \n num_train_epochs: int = 3
                                              \n lora_rank: int = 64
                                              \n lora_alpha: int = 128
                                              \n lora_dropout: float = 0.05
                                              \n # \u786c\u4ef6\u914d\u7f6e
                                              \n device: str = "cuda" if torch.cuda.is_available() else "cpu"
                                              \n fp16: bool = True
                                              \n gradient_checkpointing: bool = True<\/p>\n

                                              # 2. \u52a0\u8f7d\u914d\u7f6e
                                              \nconfig = DistillationConfig()<\/p>\n

                                              # 3. \u52a0\u8f7dTokenizer
                                              \ntokenizer = AutoTokenizer.from_pretrained(
                                              \n os.path.join(config.model_dir, config.student_model_name.split("\/")[-1])
                                              \n if os.path.exists(os.path.join(config.model_dir, config.student_model_name.split("\/")[-1]))
                                              \n else config.student_model_name,
                                              \n trust_remote_code=True,
                                              \n use_fast=False,
                                              \n padding_side="right",
                                              \n)
                                              \ntokenizer.pad_token = tokenizer.eos_token<\/p>\n

                                              # 4. \u52a0\u8f7d\u6559\u5e08\u6a21\u578b（\u7528\u4e8e\u751f\u6210Soft Target）
                                              \nprint("=== \u52a0\u8f7d\u6559\u5e08\u6a21\u578b ===")
                                              \nteacher_model = AutoModelForCausalLM.from_pretrained(
                                              \n os.path.join(config.model_dir, config.teacher_model_name.split("\/")[-1])
                                              \n if os.path.exists(os.path.join(config.model_dir, config.teacher_model_name.split("\/")[-1]))
                                              \n else config.teacher_model_name,
                                              \n device_map="auto",
                                              \n trust_remote_code=True,
                                              \n torch_dtype=torch.float16,
                                              \n low_cpu_mem_usage=True,
                                              \n)
                                              \nteacher_model.eval()<\/p>\n

                                              # 5. \u52a0\u8f7d\u5b66\u751f\u6a21\u578b（\u5f85\u84b8\u998f）
                                              \nprint("=== \u52a0\u8f7d\u5b66\u751f\u6a21\u578b ===")
                                              \nstudent_model = AutoModelForCausalLM.from_pretrained(
                                              \n os.path.join(config.model_dir, config.student_model_name.split("\/")[-1])
                                              \n if os.path.exists(os.path.join(config.model_dir, config.student_model_name.split("\/")[-1]))
                                              \n else config.student_model_name,
                                              \n device_map="auto",
                                              \n trust_remote_code=True,
                                              \n torch_dtype=torch.float16,
                                              \n low_cpu_mem_usage=True,
                                              \n)<\/p>\n

                                              # 6. \u914d\u7f6eLoRA（\u4f4e\u79e9\u9002\u914d，\u51cf\u5c11\u8bad\u7ec3\u53c2\u6570）
                                              \nlora_config = LoraConfig(
                                              \n r=config.lora_rank,
                                              \n lora_alpha=config.lora_alpha,
                                              \n target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
                                              \n lora_dropout=config.lora_dropout,
                                              \n bias="none",
                                              \n task_type="CAUSAL_LM",
                                              \n)
                                              \nstudent_model = get_peft_model(student_model, lora_config)
                                              \nstudent_model.print_trainable_parameters()<\/p>\n

                                              # 7. \u52a0\u8f7d\u5e76\u9884\u5904\u7406\u6570\u636e\u96c6
                                              \ndef load_and_preprocess_data():
                                              \n # \u52a0\u8f7d\u6570\u636e\u96c6
                                              \n dataset = load_dataset(config.dataset_name, split="train[:10%]") # \u53d610%\u6570\u636e\u7528\u4e8e\u6f14\u793a<\/p>\n

                                              # \u9884\u5904\u7406\u51fd\u6570
                                              \n def preprocess_function(examples):
                                              \n # \u6784\u9020\u5bf9\u8bdd\u683c\u5f0f
                                              \n texts = []
                                              \n for instruction, input_text, output_text in zip(examples["instruction"], examples["input"], examples["output"]):
                                              \n if input_text:
                                              \n prompt = f"<|im_start|>user\\\\n{instruction}\\\\n{input_text}<|im_end|>\\\\n<|im_start|>assistant\\\\n"
                                              \n else:
                                              \n prompt = f"<|im_start|>user\\\\n{instruction}<|im_end|>\\\\n<|im_start|>assistant\\\\n"
                                              \n full_text = prompt + output_text + "<|im_end|>"
                                              \n texts.append(full_text)<\/p>\n

                                              # \u5206\u8bcd
                                              \n tokenized = tokenizer(
                                              \n texts,
                                              \n truncation=True,
                                              \n max_length=config.max_seq_length,
                                              \n padding="max_length",
                                              \n return_tensors="pt",
                                              \n )<\/p>\n

                                              # \u751f\u6210\u6559\u5e08\u6a21\u578b\u7684Soft Target（\u4ec5\u8ba1\u7b97\u4e00\u6b21）
                                              \n with torch.no_grad():
                                              \n teacher_logits = teacher_model(
                                              \n input_ids=tokenized["input_ids"].to(config.device),
                                              \n attention_mask=tokenized["attention_mask"].to(config.device)
                                              \n ).logits<\/p>\n

                                              # \u6784\u9020\u8bad\u7ec3\u6570\u636e
                                              \n tokenized["labels"] = tokenized["input_ids"].clone()
                                              \n tokenized["teacher_logits"] = teacher_logits.cpu()<\/p>\n

                                              return tokenized<\/p>\n

                                              # \u9884\u5904\u7406\u6570\u636e\u96c6
                                              \n tokenized_dataset = dataset.map(
                                              \n preprocess_function,
                                              \n batched=True,
                                              \n batch_size=config.batch_size,
                                              \n remove_columns=dataset.column_names,
                                              \n )<\/p>\n

                                              # \u5212\u5206\u8bad\u7ec3\u96c6\u548c\u9a8c\u8bc1\u96c6
                                              \n split_dataset = tokenized_dataset.train_test_split(test_size=0.1)
                                              \n return split_dataset["train"], split_dataset["test"]<\/p>\n

                                              # 8. \u81ea\u5b9a\u4e49\u84b8\u998f\u635f\u5931\u51fd\u6570
                                              \nclass DistillationTrainer(SFTTrainer):
                                              \n def compute_loss(self, model, inputs, return_outputs=False):
                                              \n # \u83b7\u53d6\u5b66\u751f\u6a21\u578b\u8f93\u51fa
                                              \n student_outputs = model(
                                              \n input_ids=inputs["input_ids"].to(config.device),
                                              \n attention_mask=inputs["attention_mask"].to(config.device),
                                              \n labels=inputs["labels"].to(config.device),
                                              \n )
                                              \n student_logits = student_outputs.logits<\/p>\n

                                              # \u83b7\u53d6\u6559\u5e08\u6a21\u578bLogits
                                              \n teacher_logits = inputs["teacher_logits"].to(config.device)<\/p>\n

                                              # \u84b8\u998f\u635f\u5931：KL\u6563\u5ea6（\u5339\u914d\u5206\u5e03） + \u4ea4\u53c9\u71b5（\u57fa\u7840\u635f\u5931）
                                              \n kl_loss = torch.nn.functional.kl_div(
                                              \n torch.nn.functional.log_softmax(student_logits \/ 2.0, dim=-1),
                                              \n torch.nn.functional.softmax(teacher_logits \/ 2.0, dim=-1),
                                              \n reduction="batchmean",
                                              \n log_target=False,
                                              \n ) * (2.0 **2)<\/p>\n

                                              ce_loss = student_outputs.loss
                                              \n total_loss = 0.7 * ce_loss + 0.3 * kl_loss # \u6743\u91cd\u53ef\u8c03<\/p>\n

                                              return (total_loss, student_outputs) if return_outputs else total_loss<\/p>\n

                                              # 9. \u914d\u7f6e\u8bad\u7ec3\u53c2\u6570
                                              \ntraining_args = TrainingArguments(
                                              \n output_dir=config.output_dir,
                                              \n per_device_train_batch_size=config.batch_size,
                                              \n gradient_accumulation_steps=config.gradient_accumulation_steps,
                                              \n learning_rate=config.learning_rate,
                                              \n num_train_epochs=config.num_train_epochs,
                                              \n fp16=config.fp16,
                                              \n gradient_checkpointing=config.gradient_checkpointing,
                                              \n save_strategy="epoch",
                                              \n evaluation_strategy="epoch",
                                              \n logging_steps=10,
                                              \n save_steps=100,
                                              \n eval_steps=100,
                                              \n load_best_model_at_end=True,
                                              \n metric_for_best_model="loss",
                                              \n greater_is_better=False,
                                              \n push_to_hub=False,
                                              \n report_to="none",
                                              \n)<\/p>\n

                                              # 10. \u6570\u636e\u6536\u96c6\u5668
                                              \ndata_collator = DataCollatorForCompletionOnlyLM(
                                              \n tokenizer=tokenizer,
                                              \n response_template="<|im_start|>assistant\\\\n",
                                              \n mlm=False,
                                              \n)<\/p>\n

                                              # 11. \u5f00\u59cb\u84b8\u998f\u8bad\u7ec3
                                              \ndef run_distillation():
                                              \n # \u52a0\u8f7d\u6570\u636e
                                              \n train_dataset, eval_dataset = load_and_preprocess_data()<\/p>\n

                                              # \u521b\u5efaTrainer
                                              \n trainer = DistillationTrainer(
                                              \n model=student_model,
                                              \n args=training_args,
                                              \n train_dataset=train_dataset,
                                              \n eval_dataset=eval_dataset,
                                              \n tokenizer=tokenizer,
                                              \n data_collator=data_collator,
                                              \n peft_config=lora_config,
                                              \n max_seq_length=config.max_seq_length,
                                              \n )<\/p>\n

                                              # \u5f00\u59cb\u8bad\u7ec3
                                              \n start_train = time.time()
                                              \n trainer.train()
                                              \n end_train = time.time()
                                              \n print(f"\u84b8\u998f\u8bad\u7ec3\u8017\u65f6：{end_train – start_train:.2f}\u79d2")<\/p>\n

                                              # \u4fdd\u5b58\u6a21\u578b
                                              \n trainer.save_model(config.output_dir)
                                              \n tokenizer.save_pretrained(config.output_dir)
                                              \n print(f"\u84b8\u998f\u6a21\u578b\u4fdd\u5b58\u81f3：{config.output_dir}")<\/p>\n

                                              # 12. \u6d4b\u8bd5\u84b8\u998f\u540e\u6a21\u578b
                                              \ndef test_distilled_model():
                                              \n # \u52a0\u8f7d\u84b8\u998f\u540e\u7684\u6a21\u578b
                                              \n distilled_model = AutoModelForCausalLM.from_pretrained(
                                              \n config.output_dir,
                                              \n device_map="auto",
                                              \n trust_remote_code=True,
                                              \n torch_dtype=torch.float16,
                                              \n )
                                              \n distilled_model.config.use_cache = True # \u542f\u7528KV Cache<\/p>\n

                                              # \u6d4b\u8bd5\u7528\u4f8b
                                              \n prompts = [
                                              \n "\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u7684\u6838\u5fc3\u6280\u672f\u6709\u54ea\u4e9b？",
                                              \n "KV Cache\u548c\u91cf\u5316\u6280\u672f\u7684\u533a\u522b\u662f\u4ec0\u4e48？",
                                              \n "\u5982\u4f55\u5728\u8fb9\u7f18\u8bbe\u5907\u90e8\u7f72\u5927\u6a21\u578b？"
                                              \n ]<\/p>\n

                                              total_time = 0
                                              \n total_tokens = 0<\/p>\n

                                              for prompt in prompts:
                                              \n # \u6784\u9020\u8f93\u5165
                                              \n messages = [{"role": "user", "content": prompt}]
                                              \n input_ids = tokenizer.apply_chat_template(
                                              \n messages,
                                              \n tokenize=True,
                                              \n add_generation_prompt=True,
                                              \n return_tensors="pt"
                                              \n ).to(config.device)<\/p>\n

                                              # \u63a8\u7406
                                              \n start_time = time.time()
                                              \n outputs = distilled_model.generate(
                                              \n input_ids=input_ids,
                                              \n max_new_tokens=512,
                                              \n temperature=0.1,
                                              \n top_p=0.95,
                                              \n do_sample=True,
                                              \n use_cache=True,
                                              \n )
                                              \n end_time = time.time()<\/p>\n

                                              # \u89e3\u6790\u8f93\u51fa
                                              \n output_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
                                              \n output_text = output_text.split("<|im_end|>")[-2].strip()<\/p>\n

                                              # \u7edf\u8ba1
                                              \n elapsed_time = end_time – start_time
                                              \n num_tokens = len(outputs[0]) – len(input_ids[0])
                                              \n total_time += elapsed_time
                                              \n total_tokens += num_tokens<\/p>\n

                                              print(f"\\\\n=== \u8f93\u5165：{prompt[:50]}… ===")
                                              \n print(f"\u8017\u65f6：{elapsed_time:.2f}\u79d2")
                                              \n print(f"\u751f\u6210Token\u6570：{num_tokens}")
                                              \n print(f"\u901f\u5ea6：{num_tokens\/elapsed_time:.2f} tokens\/\u79d2")
                                              \n print(f"\u8f93\u51fa：{output_text[:200]}…")<\/p>\n

                                              # \u6c47\u603b
                                              \n avg_speed = total_tokens \/ total_time
                                              \n print(f"\\\\n\u84b8\u998f\u6a21\u578b\u5e73\u5747\u901f\u5ea6：{avg_speed:.2f} tokens\/\u79d2")<\/p>\n

                                              # \u663e\u5b58\u4f7f\u7528
                                              \n if torch.cuda.is_available():
                                              \n mem_used = torch.cuda.max_memory_allocated() \/ 1024 \/ 1024 \/ 1024
                                              \n print(f"\u6700\u5927\u663e\u5b58\u5360\u7528：{mem_used:.2f} GB")<\/p>\n

                                              if __name__ == "__main__":
                                              \n print("=== \u5f00\u59cb\u6a21\u578b\u84b8\u998f ===")
                                              \n run_distillation()
                                              \n print("\\\\n=== \u6d4b\u8bd5\u84b8\u998f\u540e\u6a21\u578b ===")
                                              \n test_distilled_model()<\/p>\n

                                              5.3.2 \u6548\u679c\u9a8c\u8bc1\u8f93\u51fa\u793a\u4f8b<\/h5>\n

                                              === \u52a0\u8f7d\u6559\u5e08\u6a21\u578b ===
                                              \n=== \u52a0\u8f7d\u5b66\u751f\u6a21\u578b ===
                                              \ntrainable params: 18,874,368 || all params: 1,807,677,440 || trainable%: 1.04<\/p>\n

                                              === \u5f00\u59cb\u6a21\u578b\u84b8\u998f ===
                                              \nEpoch 1\/3: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 100\/100 [1:20:12<00:00, 48.12s\/it]
                                              \nEpoch 2\/3: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 100\/100 [1:18:45<00:00, 47.25s\/it]
                                              \nEpoch 3\/3: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 100\/100 [1:19:23<00:00, 47.63s\/it]
                                              \n\u84b8\u998f\u8bad\u7ec3\u8017\u65f6：23820.45\u79d2
                                              \n\u84b8\u998f\u6a21\u578b\u4fdd\u5b58\u81f3：.\/models\/Qwen2.5-1.8B-Distilled<\/p>\n

                                              === \u6d4b\u8bd5\u84b8\u998f\u540e\u6a21\u578b ===
                                              \n=== \u8f93\u5165：\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u7684\u6838\u5fc3\u6280\u672f\u6709\u54ea\u4e9b？… ===
                                              \n\u8017\u65f6：1.28\u79d2
                                              \n\u751f\u6210Token\u6570：312
                                              \n\u901f\u5ea6：243.75 tokens\/\u79d2
                                              \n\u8f93\u51fa：\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u7684\u6838\u5fc3\u6280\u672f\u4e3b\u8981\u5305\u62ec\u4e09\u7c7b：1. KV Cache\u4f18\u5316：\u7f13\u5b58\u6ce8\u610f\u529b\u673a\u5236\u7684Key\/Value\u77e9\u9635，\u907f\u514d\u91cd\u590d\u8ba1\u7b97，\u63d0\u5347\u957f\u6587\u672c\u751f\u6210\u901f\u5ea6；2. \u91cf\u5316\u6280\u672f：\u5c06\u6a21\u578b\u6743\u91cd\u4eceFP16\u964d\u81f3INT8\/INT4…<\/p>\n

                                              === \u8f93\u5165：KV Cache\u548c\u91cf\u5316\u6280\u672f\u7684\u533a\u522b\u662f\u4ec0\u4e48？… ===
                                              \n\u8017\u65f6：1.15\u79d2
                                              \n\u751f\u6210Token\u6570：289
                                              \n\u901f\u5ea6：251.30 tokens\/\u79d2
                                              \n\u8f93\u51fa：KV Cache\u548c\u91cf\u5316\u6280\u672f\u7684\u6838\u5fc3\u533a\u522b\u5728\u4e8e\u4f18\u5316\u7ef4\u5ea6\u4e0d\u540c：KV Cache\u662f\u4ece\u8ba1\u7b97\u91cf\u89d2\u5ea6\u4f18\u5316，\u901a\u8fc7\u7f13\u5b58\u907f\u514d\u91cd\u590d\u8ba1\u7b97；\u91cf\u5316\u6280\u672f\u662f\u4ece\u5b58\u50a8\u89d2\u5ea6\u4f18\u5316…<\/p>\n

                                              === \u8f93\u5165：\u5982\u4f55\u5728\u8fb9\u7f18\u8bbe\u5907\u90e8\u7f72\u5927\u6a21\u578b？… ===
                                              \n\u8017\u65f6：1.08\u79d2
                                              \n\u751f\u6210Token\u6570：276
                                              \n\u901f\u5ea6：255.56 tokens\/\u79d2
                                              \n\u8f93\u51fa：\u8fb9\u7f18\u8bbe\u5907\u90e8\u7f72\u5927\u6a21\u578b\u7684\u6838\u5fc3\u7b56\u7565\u662f\u6a21\u578b\u84b8\u998f+\u91cf\u5316：\u9996\u5148\u5c067B\/14B\u5927\u6a21\u578b\u84b8\u998f\u4e3a1.8B\/3B\u5c0f\u6a21\u578b，\u518d\u8fdb\u884cINT4\u91cf\u5316，\u7ed3\u5408KV Cache\u4f18\u5316…<\/p>\n

                                              \u84b8\u998f\u6a21\u578b\u5e73\u5747\u901f\u5ea6：250.17 tokens\/\u79d2
                                              \n\u6700\u5927\u663e\u5b58\u5360\u7528：2.45 GB<\/p>\n

                                              \u516d\u3001\u7efc\u5408\u4f18\u5316\u4e0e\u751f\u4ea7\u7ea7\u90e8\u7f72<\/h3>\n

                                              6.1 \u7ec4\u5408\u4f18\u5316\u65b9\u6848<\/h4>\n\n\u573a\u666f\u4f18\u5316\u7ec4\u5408\u9884\u671f\u6548\u679c\u793a\u4f8b\u914d\u7f6e<\/tr>\n\n\n\n\n
                                              \u672c\u5730\u684c\u9762\u90e8\u7f72<\/td>\n\u84b8\u998f\u6a21\u578b（1.8B） + INT4 GPTQ + KV Cache<\/td>\n\u901f\u5ea6：200+ tokens\/\u79d2，\u663e\u5b58：<3GB<\/td>\nQwen2.5-1.8B-Distilled + GPTQ 4bit + vLLM KV Cache<\/td>\n<\/tr>\n
                                              \u4e91\u7aefAPI\u670d\u52a1<\/td>\n7B\u6a21\u578b + INT8\u91cf\u5316 + KV Cache + \u6279\u91cf\u63a8\u7406<\/td>\n\u901f\u5ea6：80+ tokens\/\u79d2，\u663e\u5b58：<8GB，\u5e76\u53d1：32+<\/td>\nQwen2.5-7B + bitsandbytes INT8 + vLLM<\/td>\n<\/tr>\n
                                              \u8fb9\u7f18\u8bbe\u5907（\u5982NVIDIA Jetson）<\/td>\n\u84b8\u998f\u6a21\u578b（1.8B） + INT4 AWQ + \u8f7b\u91cf\u5316KV Cache<\/td>\n\u901f\u5ea6：50+ tokens\/\u79d2，\u663e\u5b58：<2GB<\/td>\nQwen2.5-1.8B-Distilled + AWQ 4bit<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                                              6.2 \u751f\u4ea7\u7ea7\u90e8\u7f72\u4ee3\u7801（FastAPI + vLLM）<\/h4>\n

                                              \u751f\u4ea7\u7ea7\u90e8\u7f72\u6d41\u7a0b\u56fe (FastAPI + vLLM Serving Architecture)<\/h5>\n

                                              \u2502
                                              \n\u251c\u2500\u2500 \u3010\u670d\u52a1\u542f\u52a8\u5165\u53e3\u3011: if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span> uvicorn.run(<\/span>..<\/span>.)<\/span>
                                              \n\u2502
                                              \n\u25bc
                                              \n[<\/span>1<\/span>. \u5168\u5c40\u521d\u59cb\u5316\u9636\u6bb5 (<\/span>Service Initialization)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 A. \u57fa\u7840\u8bbe\u65bd\u51c6\u5907 \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u52a0\u8f7d\u914d\u7f6e><\/span>: .env (<\/span>PORT, MODEL_PATH, GPU_MEM_UTIL)<\/span> \u2502
                                              \n\u2502 \u2514\u2500\u2500 <<\/span>\u65e5\u5fd7\u7cfb\u7edf><\/span>: logging.basicConfig (<\/span>\u76d1\u63a7\u670d\u52a1\u72b6\u6001)<\/span> \u2502
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 B. Web \u6846\u67b6\u6784\u5efa (<\/span>FastAPI Setup)<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u4e2d\u95f4\u4ef6><\/span>: CORSMiddleware (<\/span>\u5141\u8bb8\u8de8\u57df\u8bbf\u95ee)<\/span> \u2502
                                              \n\u2502 \u2514\u2500\u2500 <<\/span>\u8def\u7531><\/span>: \/health (<\/span>\u5fc3\u8df3\u68c0\u6d4b)<\/span>, \/chat (<\/span>\u6838\u5fc3\u4e1a\u52a1)<\/span> \u2502
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 C. \u63a8\u7406\u5f15\u64ce\u52a0\u8f7d (<\/span>vLLM Engine Loading)<\/span> <<\/span>\u2605 \u8017\u65f6\u64cd\u4f5c><\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u6a21\u578b\u8def\u5f84><\/span>: "Qwen2.5-7B-Instruct-GPTQ-4bit"<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u663e\u5b58\u7b56\u7565><\/span>: gpu_memory_utilization<\/span>=<\/span>0.9<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u5e76\u884c\u7b56\u7565><\/span>: tensor_parallel_size<\/span>=<\/span>1<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u91cf\u5316\u914d\u7f6e><\/span>: quantization<\/span>=<\/span>"gptq"<\/span> (<\/span>\u6838\u5fc3\u52a0\u901f)<\/span> \u2502
                                              \n\u2502 \u2514\u2500\u2500 ><\/span> \u7ed3\u679c: llm \u5bf9\u8c61\u5e38\u9a7b\u663e\u5b58, PagedAttention \u5c31\u7eea \u2502
                                              \n\u2502 \u2502
                                              \n\u2514\u2500\u2500 ><\/span> \u670d\u52a1\u72b6\u6001: Listening on 0.0<\/span>.0.0:8000 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                              \n \u2502
                                              \n \u25bc
                                              \n \u2502 <<\/span>HTTP POST \/chat><\/span>
                                              \n \u2502 Payload: {<\/span>"messages"<\/span>:<\/span> [<\/span>..<\/span>.]<\/span>, "temperature"<\/span>:<\/span> 0.1<\/span>}<\/span>
                                              \n \u2502
                                              \n[<\/span>2<\/span>. \u8bf7\u6c42\u5904\u7406\u9636\u6bb5 (<\/span>Request Handling Pipeline)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 A. \u6570\u636e\u6821\u9a8c (<\/span>Pydantic Validation)<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 class ChatRequest: \u68c0\u67e5 messages \u683c\u5f0f, \u9ed8\u8ba4\u53c2\u6570\u586b\u5145 \u2502
                                              \n\u2502 \u2514\u2500\u2500 ><\/span> \u5f02\u5e38\u5904\u7406: \u82e5\u6821\u9a8c\u5931\u8d25 –><\/span> 422<\/span> Unprocessable Entity \u2502
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 B. Prompt \u5de5\u7a0b (<\/span>Prompt Engineering)<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 <<\/span>\u904d\u5386><\/span>: request.messages \u2502
                                              \n\u2502 \u2514\u2500\u2500 <<\/span>\u62fc\u63a5><\/span>: <<\/span>|<\/span>im_start|<\/span>><\/span>user\\\\<\/span>n..<\/span>.<<\/span>|<\/span>im_end|<\/span>><\/span>\\\\<\/span>n (<\/span>ChatML)<\/span> \u2502
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 C. \u53c2\u6570\u9002\u914d (<\/span>Sampling Configuration)<\/span> \u2502
                                              \n\u2502 \u2514\u2500\u2500 <<\/span>\u6784\u5efa><\/span>: SamplingParams (<\/span>top_p=<\/span>0.95<\/span>, max_tokens<\/span>=<\/span>2048<\/span>)<\/span> \u2502
                                              \n\u2502 \u2502
                                              \n\u2514\u2500\u2500 ><\/span> \u51c6\u5907\u5c31\u7eea: Prompt String + SamplingParams \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                              \n \u2502
                                              \n \u25bc
                                              \n[<\/span>3<\/span>. \u5f02\u6b65\u63a8\u7406\u9636\u6bb5 (<\/span>Async Inference Execution)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 <<\/span>\u8c03\u7528><\/span>: llm.generate (<\/span>\u975e\u963b\u585e\/\u963b\u585e\u53d6\u51b3\u4e8e vLLM \u7248\u672c\u5b9e\u73b0)<\/span> \u2502
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 \u5185\u90e8\u8c03\u5ea6 (<\/span>vLLM Scheduler)<\/span>: \u2502
                                              \n\u2502 \u251c\u2500\u2500 1<\/span>. \u52a0\u5165\u7b49\u5f85\u961f\u5217 (<\/span>Waiting Queue)<\/span> \u2502
                                              \n\u2502 \u251c\u2500\u2500 2<\/span>. \u6279\u5904\u7406\u8c03\u5ea6 (<\/span>Continuous Batching)<\/span> \u2502
                                              \n\u2502 \u2502 \u2514\u2500\u2500 \u5c06\u8be5\u8bf7\u6c42\u4e0e\u5176\u4ed6\u5e76\u53d1\u8bf7\u6c42\u5408\u5e76\u8fdb\u540c\u4e00\u4e2a Batch \u2502
                                              \n\u2502 \u2514\u2500\u2500 3<\/span>. PagedAttention \u663e\u5b58\u5206\u914d \u2502
                                              \n\u2502 \u2514\u2500\u2500 \u52a8\u6001\u6620\u5c04 KV Cache \u5230\u7269\u7406\u663e\u5b58\u5757 \u2502
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 \u8ba1\u7b97\u8fc7\u7a0b (<\/span>Execution)<\/span>: \u2502
                                              \n\u2502 \u2514\u2500\u2500 GPU Kernel \u8fd0\u7b97 (<\/span>GPTQ INT4 GEMM)<\/span> \u2502
                                              \n\u2502 \u2502
                                              \n\u2514\u2500\u2500 ><\/span> \u8f93\u51fa\u7ed3\u679c: RequestOutput (<\/span>\u5305\u542b generated_text)<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                                              \n \u2502
                                              \n \u25bc
                                              \n[<\/span>4<\/span>. \u54cd\u5e94\u4e0e\u76d1\u63a7\u9636\u6bb5 (<\/span>Response &<\/span> Monitoring)<\/span>]<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 A. \u7ed3\u679c\u89e3\u6790 \u2502
                                              \n\u2502 \u251c\u2500\u2500 \u63d0\u53d6\u6587\u672c: outputs[<\/span>0<\/span>]<\/span>.text \u2502
                                              \n\u2502 \u2514\u2500\u2500 \u7edf\u8ba1 Token: prompt_tokens, completion_tokens \u2502
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 B. \u65e5\u5fd7\u8bb0\u5f55 (<\/span>Observability)<\/span> \u2502
                                              \n\u2502 \u2514\u2500\u2500 logger.info: \u8bb0\u5f55 SessionID \u53ca\u8f93\u5165\u8f93\u51fa\u957f\u5ea6 (<\/span>\u7528\u4e8e\u8ba1\u8d39\/\u5206\u6790)<\/span>\u2502
                                              \n\u2502 \u2502
                                              \n\u251c\u2500\u2500 C. \u54cd\u5e94\u6784\u9020 \u2502
                                              \n\u2502 \u2514\u2500\u2500 class ChatResponse: \u5c01\u88c5\u6807\u51c6 JSON \u7ed3\u6784 (<\/span>code, data)<\/span> \u2502
                                              \n\u2502 \u2502
                                              \n\u2514\u2500\u2500 ><\/span> \u8fd4\u56de HTTP 200<\/span> OK: {<\/span>"code"<\/span>:<\/span> 200<\/span>, "data"<\/span>:<\/span> {<\/span>..<\/span>.}<\/span>}<\/span> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

                                              \u6d41\u7a0b\u56fe\u6838\u5fc3\u903b\u8f91\u89e3\u6790<\/h6>\n
                                            • \u5168\u5c40\u5355\u4f8b (Global Singleton)：\u6d41\u7a0b\u56fe\u5f3a\u8c03\u4e86 llm = LLM(…) \u662f\u5728\u670d\u52a1\u542f\u52a8\u65f6\u521d\u59cb\u5316\u7684，\u800c\u4e0d\u662f\u5728\u6bcf\u6b21\u8bf7\u6c42\u65f6\u3002\u8fd9\u662f\u751f\u4ea7\u73af\u5883\u7684\u5173\u952e，\u907f\u514d\u4e86\u91cd\u590d\u52a0\u8f7d\u6a21\u578b\u5e26\u6765\u7684\u5de8\u5927\u5ef6\u8fdf\u3002<\/li>\n
                                            • \u8c03\u5ea6\u5c42 (Scheduler)：\u5728\u201c[3. \u5f02\u6b65\u63a8\u7406\u9636\u6bb5]\u201d\u4e2d，\u6211\u8865\u5145\u4e86\u4ee3\u7801\u4e2d\u770b\u4e0d\u89c1\u4f46\u5728 vLLM \u5185\u90e8\u53d1\u751f\u7684\u201cContinuous Batching\u201d\u548c\u201cPagedAttention\u201d\u903b\u8f91，\u8fd9\u662f\u8be5\u670d\u52a1\u4e4b\u6240\u4ee5\u80fd\u9ad8\u5e76\u53d1\u7684\u6839\u672c\u539f\u56e0\u3002<\/li>\n
                                            • Prompt \u62fc\u63a5：\u660e\u786e\u5c55\u793a\u4e86\u5c06 JSON \u683c\u5f0f\u7684 messages \u8f6c\u6362\u4e3a\u6a21\u578b\u80fd\u7406\u89e3\u7684 ChatML \u5b57\u7b26\u4e32\u7684\u8fc7\u7a0b\u3002<\/li>\n

                                              import os
                                              \nimport uvicorn
                                              \nfrom fastapi import FastAPI, Request, HTTPException
                                              \nfrom fastapi.middleware.cors import CORSMiddleware
                                              \nfrom pydantic import BaseModel
                                              \nfrom vllm import LLM, SamplingParams
                                              \nfrom typing import Optional, List
                                              \nimport logging
                                              \nfrom dotenv import load_dotenv<\/p>\n

                                              # \u52a0\u8f7d\u73af\u5883\u53d8\u91cf
                                              \nload_dotenv()<\/p>\n

                                              # \u914d\u7f6e\u65e5\u5fd7
                                              \nlogging.basicConfig(
                                              \n level=logging.INFO,
                                              \n format="%(asctime)s – %(levelname)s – %(message)s",
                                              \n handlers=[logging.StreamHandler()]
                                              \n)
                                              \nlogger = logging.getLogger(__name__)<\/p>\n

                                              # \u521d\u59cb\u5316FastAPI
                                              \napp = FastAPI(title="LLM\u63a8\u7406\u52a0\u901f\u670d\u52a1", version="1.0")<\/p>\n

                                              # \u914d\u7f6eCORS
                                              \napp.add_middleware(
                                              \n CORSMiddleware,
                                              \n allow_origins=["*"],
                                              \n allow_credentials=True,
                                              \n allow_methods=["*"],
                                              \n allow_headers=["*"],
                                              \n)<\/p>\n

                                              # \u5168\u5c40\u914d\u7f6e
                                              \nMODEL_PATH = os.getenv("MODEL_PATH", ".\/models\/Qwen2.5-7B-Instruct-GPTQ-4bit")
                                              \nPORT = int(os.getenv("PORT", 8000))
                                              \nGPU_MEM_UTIL = float(os.getenv("GPU_MEM_UTIL", 0.9))<\/p>\n

                                              # \u521d\u59cb\u5316vLLM（\u96c6\u6210KV Cache + \u91cf\u5316）
                                              \nlogger.info(f"\u5f00\u59cb\u52a0\u8f7d\u6a21\u578b：{MODEL_PATH}")
                                              \nllm = LLM(
                                              \n model=MODEL_PATH,
                                              \n tensor_parallel_size=1,
                                              \n gpu_memory_utilization=GPU_MEM_UTIL,
                                              \n max_num_batched_tokens=4096,
                                              \n max_num_seqs=64,
                                              \n enable_chunked_prefill=True,
                                              \n chunked_prefill_tokens=512,
                                              \n # \u91cf\u5316\u6a21\u578b\u914d\u7f6e
                                              \n quantization="gptq", # \u9002\u914dGPTQ\u91cf\u5316\u6a21\u578b
                                              \n trust_remote_code=True,
                                              \n)
                                              \nsampling_params = SamplingParams(
                                              \n temperature=0.1,
                                              \n max_tokens=2048,
                                              \n top_p=0.95,
                                              \n use_beam_search=False,
                                              \n)
                                              \nlogger.info("\u6a21\u578b\u52a0\u8f7d\u5b8c\u6210")<\/p>\n

                                              # \u8bf7\u6c42\u6a21\u578b
                                              \nclass ChatRequest(BaseModel):
                                              \n messages: List[dict] # [{"role": "user", "content": "…"}]
                                              \n temperature: Optional[float] = 0.1
                                              \n max_tokens: Optional[int] = 2048
                                              \n session_id: Optional[str] = "default"<\/p>\n

                                              # \u54cd\u5e94\u6a21\u578b
                                              \nclass ChatResponse(BaseModel):
                                              \n code: int = 200
                                              \n message: str = "success"
                                              \n data: dict = {}<\/p>\n

                                              # \u5065\u5eb7\u68c0\u67e5\u63a5\u53e3
                                              \n@app.get("\/health", response_model=ChatResponse)
                                              \nasync def health_check():
                                              \n return ChatResponse(
                                              \n data={
                                              \n "status": "healthy",
                                              \n "model": MODEL_PATH,
                                              \n "gpu_memory_utilization": GPU_MEM_UTIL
                                              \n }
                                              \n )<\/p>\n

                                              # \u804a\u5929\u63a5\u53e3
                                              \n@app.post("\/chat", response_model=ChatResponse)
                                              \nasync def chat(request: ChatRequest):
                                              \n try:
                                              \n # \u9a8c\u8bc1\u8f93\u5165
                                              \n if not request.messages or not isinstance(request.messages, list):
                                              \n raise HTTPException(status_code=400, detail="messages\u4e0d\u80fd\u4e3a\u7a7a")<\/p>\n

                                              # \u6784\u9020prompt
                                              \n prompt = ""
                                              \n for msg in request.messages:
                                              \n if msg["role"] == "user":
                                              \n prompt += f"<|im_start|>user\\\\n{msg['content']}<|im_end|>\\\\n"
                                              \n elif msg["role"] == "assistant":
                                              \n prompt += f"<|im_start|>assistant\\\\n{msg['content']}<|im_end|>\\\\n"
                                              \n prompt += "<|im_start|>assistant\\\\n"<\/p>\n

                                              # \u8c03\u6574\u91c7\u6837\u53c2\u6570
                                              \n custom_sampling_params = SamplingParams(
                                              \n temperature=request.temperature,
                                              \n max_tokens=request.max_tokens,
                                              \n top_p=0.95,
                                              \n )<\/p>\n

                                              # \u63a8\u7406
                                              \n start_time = logger.info(f"\u4f1a\u8bdd[{request.session_id}]\u5f00\u59cb\u63a8\u7406")
                                              \n outputs = llm.generate([prompt], custom_sampling_params)
                                              \n end_time = logger.info(f"\u4f1a\u8bdd[{request.session_id}]\u63a8\u7406\u5b8c\u6210")<\/p>\n

                                              # \u89e3\u6790\u8f93\u51fa
                                              \n output = outputs[0].outputs[0].text
                                              \n prompt_tokens = len(outputs[0].prompt_token_ids)
                                              \n completion_tokens = len(outputs[0].output_token_ids)<\/p>\n

                                              # \u65e5\u5fd7
                                              \n logger.info(
                                              \n f"\u4f1a\u8bdd[{request.session_id}] – "
                                              \n f"\u8f93\u5165Token：{prompt_tokens}，\u8f93\u51faToken：{completion_tokens}"
                                              \n )<\/p>\n

                                              return ChatResponse(
                                              \n data={
                                              \n "response": output,
                                              \n "prompt_tokens": prompt_tokens,
                                              \n "completion_tokens": completion_tokens,
                                              \n "session_id": request.session_id
                                              \n }
                                              \n )
                                              \n except HTTPException as e:
                                              \n logger.error(f"\u5ba2\u6237\u7aef\u9519\u8bef：{e.detail}")
                                              \n return ChatResponse(
                                              \n code=e.status_code,
                                              \n message=e.detail,
                                              \n data={"session_id": request.session_id}
                                              \n )
                                              \n except Exception as e:
                                              \n logger.error(f"\u670d\u52a1\u7aef\u9519\u8bef：{str(e)}", exc_info=True)
                                              \n return ChatResponse(
                                              \n code=500,
                                              \n message=f"\u670d\u52a1\u7aef\u9519\u8bef：{str(e)}",
                                              \n data={"session_id": request.session_id}
                                              \n )<\/p>\n

                                              # \u542f\u52a8\u670d\u52a1
                                              \nif __name__ == "__main__":
                                              \n uvicorn.run(
                                              \n app="inference_service:app",
                                              \n host="0.0.0.0",
                                              \n port=PORT,
                                              \n reload=False,
                                              \n workers=1
                                              \n )<\/p>\n

                                              6.3 \u90e8\u7f72\u542f\u52a8\u811a\u672c<\/h4>\n

                                              #!\/bin\/bash
                                              \n# start_service.sh
                                              \nexport MODEL_PATH=".\/models\/Qwen2.5-7B-Instruct-GPTQ-4bit"
                                              \nexport PORT=8000
                                              \nexport GPU_MEM_UTIL=0.9<\/p>\n

                                              # \u542f\u52a8\u670d\u52a1
                                              \npython inference_service.py<\/p>\n

                                              \u4e03\u3001\u5e38\u89c1\u95ee\u9898\u4e0e\u89e3\u51b3\u65b9\u6848<\/h3>\n

                                              7.1 KV Cache \u76f8\u5173\u95ee\u9898<\/h4>\n\n\u95ee\u9898\u89e3\u51b3\u65b9\u6848<\/tr>\n\n\n\n\n
                                              \u663e\u5b58\u788e\u7247\u5bfc\u81f4OOM<\/td>\n\u542f\u7528vLLM\u7684PagedAttention，\u8bbe\u7f6eswap_space<\/td>\n<\/tr>\n
                                              \u957f\u6587\u672c\u5bf9\u8bdd\u901f\u5ea6\u4e0b\u964d<\/td>\n\u589e\u5927chunked_prefill_tokens，\u8c03\u6574max_num_seqs<\/td>\n<\/tr>\n
                                              KV Cache\u672a\u751f\u6548<\/td>\n\u786e\u8ba4model.config.use_cache=True，vLLM\u9ed8\u8ba4\u5f00\u542f<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                                              7.2 \u91cf\u5316\u76f8\u5173\u95ee\u9898<\/h4>\n\n\u95ee\u9898\u89e3\u51b3\u65b9\u6848<\/tr>\n\n\n\n\n
                                              INT4\u91cf\u5316\u7cbe\u5ea6\u635f\u5931\u8fc7\u5927<\/td>\n\u8c03\u6574group_size=64，\u542f\u7528desc_act=True，\u6216\u6539\u7528INT8<\/td>\n<\/tr>\n
                                              GPTQ\u91cf\u5316\u6a21\u578b\u52a0\u8f7d\u5931\u8d25<\/td>\n\u786e\u8ba4\u6a21\u578b\u6587\u4ef6\u5b8c\u6574，\u4f7f\u7528trust_remote_code=True<\/td>\n<\/tr>\n
                                              bitsandbytes\u91cf\u5316\u62a5\u9519<\/td>\n\u5347\u7ea7bitsandbytes\u5230\u6700\u65b0\u7248，\u786e\u8ba4CUDA\u7248\u672c\u517c\u5bb9<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                                              7.3 \u84b8\u998f\u76f8\u5173\u95ee\u9898<\/h4>\n\n\u95ee\u9898\u89e3\u51b3\u65b9\u6848<\/tr>\n\n\n\n\n
                                              \u84b8\u998f\u8bad\u7ec3\u8fc7\u6162<\/td>\n\u542f\u7528gradient_checkpointing，\u589e\u5927batch_size<\/td>\n<\/tr>\n
                                              \u84b8\u998f\u6a21\u578b\u7cbe\u5ea6\u4f4e<\/td>\n\u589e\u52a0\u8bad\u7ec3\u6570\u636e\u91cf，\u8c03\u6574\u84b8\u998f\u635f\u5931\u6743\u91cd（\u589e\u5927KL\u6563\u5ea6\u6743\u91cd）<\/td>\n<\/tr>\n
                                              LoRA\u8bad\u7ec3\u53c2\u6570\u8fc7\u591a<\/td>\n\u51cf\u5c11target_modules，\u964d\u4f4elora_rank<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                                              \u516b\u3001\u603b\u7ed3\u4e0e\u672a\u6765\u8d8b\u52bf<\/h3>\n

                                              8.1 \u6838\u5fc3\u603b\u7ed3<\/h4>\n
                                            • \n

                                              KV Cache \u662f\u65e0\u7cbe\u5ea6\u635f\u5931\u7684\u57fa\u7840\u52a0\u901f\u6280\u672f，\u4f18\u5148\u5728\u6240\u6709\u573a\u666f\u542f\u7528<\/p>\n<\/li>\n

                                            • \n

                                              \u91cf\u5316\u6280\u672f \u662f\u663e\u5b58\u4f18\u5316\u7684\u9996\u9009，INT8\u5e73\u8861\u7cbe\u5ea6\u4e0e\u901f\u5ea6，INT4\u9002\u5408\u6781\u81f4\u8f7b\u91cf\u5316<\/p>\n<\/li>\n

                                            • \n

                                              \u6a21\u578b\u84b8\u998f \u662f\u8fb9\u7f18\u90e8\u7f72\u7684\u6838\u5fc3\u65b9\u6848，\u7ed3\u5408\u91cf\u5316\u53ef\u5b9e\u73b010\u500d\u4ee5\u4e0a\u52a0\u901f<\/p>\n<\/li>\n

                                            • \n

                                              \u7ec4\u5408\u4f18\u5316 \u662f\u751f\u4ea7\u73af\u5883\u7684\u6700\u4f73\u5b9e\u8df5：KV Cache + \u91cf\u5316（\u901a\u7528）\u3001\u84b8\u998f + \u91cf\u5316 + KV Cache（\u6781\u81f4）<\/p>\n<\/li>\n

                                              8.2 \u672a\u6765\u8d8b\u52bf<\/h4>\n
                                            • \n

                                              \u786c\u4ef6\u7ea7\u4f18\u5316：GPU\u5382\u5546\u4e13\u7528\u63a8\u7406\u82af\u7247（\u5982NVIDIA H100）\u5185\u7f6eKV Cache\u52a0\u901f<\/p>\n<\/li>\n

                                            • \n

                                              \u6df7\u5408\u7cbe\u5ea6\u63a8\u7406：\u52a8\u6001\u8c03\u6574\u4e0d\u540c\u5c42\u7684\u91cf\u5316\u7cbe\u5ea6（\u5982\u6ce8\u610f\u529b\u5c42INT8，\u8f93\u51fa\u5c42FP16）<\/p>\n<\/li>\n

                                            • \n

                                              \u589e\u91cf\u84b8\u998f：\u9488\u5bf9\u7279\u5b9a\u9886\u57df\u7684\u8f7b\u91cf\u5316\u84b8\u998f，\u4fdd\u7559\u9886\u57df\u77e5\u8bc6<\/p>\n<\/li>\n

                                            • \n

                                              \u5206\u5e03\u5f0fKV Cache：\u591aGPU\/\u591a\u8282\u70b9\u5171\u4eabKV Cache，\u63d0\u5347\u5e76\u53d1\u80fd\u529b<\/p>\n<\/li>\n

                                              8.3 \u6700\u4f73\u5b9e\u8df5\u5efa\u8bae<\/h4>\n
                                            • \n

                                              \u5feb\u901f\u9a8c\u8bc1：\u5148\u7528vLLM\u90e8\u7f727B\u6a21\u578b+KV Cache，\u9a8c\u8bc1\u57fa\u7840\u6027\u80fd<\/p>\n<\/li>\n

                                            • \n

                                              \u663e\u5b58\u4f18\u5316：\u542f\u7528INT8\u91cf\u5316，\u663e\u5b58\u5360\u7528\u964d\u4f4e50%<\/p>\n<\/li>\n

                                            • \n

                                              \u8fb9\u7f18\u90e8\u7f72：\u84b8\u998f1.8B\u6a21\u578b+INT4\u91cf\u5316，\u663e\u5b58<3GB<\/p>\n<\/li>\n

                                            • \n

                                              \u751f\u4ea7\u76d1\u63a7：\u96c6\u6210LangSmith\u76d1\u63a7\u63a8\u7406\u901f\u5ea6\u3001\u663e\u5b58\u5360\u7528\u3001\u7cbe\u5ea6\u635f\u5931<\/p>\n<\/li>\n

                                            • \n

                                              \u6301\u7eed\u8c03\u4f18：\u6839\u636e\u4e1a\u52a1\u573a\u666f\u8c03\u6574\u91cf\u5316\u7cbe\u5ea6\u3001KV Cache\u53c2\u6570\u3001\u84b8\u998f\u635f\u5931\u6743\u91cd<\/p>\n<\/li>\n","protected":false},"excerpt":{"rendered":"

                                              \u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6838\u5fc3\u6280\u672f\u5b9e\u6218：KV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f（\u9644\u6700\u65b0\u5f00\u6e90\u4ee3\u7801）
                                              \n\u6587\u6863\u6982\u8ff0
                                              \n\u6587\u7ae0\u6838\u5fc3\u4ef7\u503c \u6df1\u5ea6\u89e3\u6790\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u4e09\u5927\u6838\u5fc3\u6280\u672f（KV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f）\u7684\u5e95\u5c42\u539f\u7406 \u57fa\u4e8e2025\u5e74\u6700\u65b0\u5f00\u6e90\u751f\u6001（vLLM 0.5.0\u3001AutoGPTQ 0.7.1\u3001Transformers 4.41）\u5b9e\u73b0\u4ee3\u7801\u843d\u5730 \u9488\u5bf9\u4e0d\u540c\u573a\u666f（\u672c\u5730\u90e8\u7f72\u3001API\u670d\u52a1\u3001\u8fb9\u7f18\u8bbe\u5907<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[50,3684,51],"topic":[],"class_list":["post-70070","post","type-post","status-publish","format-standard","hentry","category-server","tag-50","tag-3684","tag-51"],"yoast_head":"\n\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6838\u5fc3\u6280\u672f\u5b9e\u6218\uff1aKV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f\uff08\u9644\u6700\u65b0\u5f00\u6e90\u4ee3\u7801\uff09 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.wsisp.com\/helps\/70070.html\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6838\u5fc3\u6280\u672f\u5b9e\u6218\uff1aKV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f\uff08\u9644\u6700\u65b0\u5f00\u6e90\u4ee3\u7801\uff09 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"og:description\" content=\"\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6838\u5fc3\u6280\u672f\u5b9e\u6218：KV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f（\u9644\u6700\u65b0\u5f00\u6e90\u4ee3\u7801） \u6587\u6863\u6982\u8ff0 \u6587\u7ae0\u6838\u5fc3\u4ef7\u503c \u6df1\u5ea6\u89e3\u6790\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u4e09\u5927\u6838\u5fc3\u6280\u672f（KV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f）\u7684\u5e95\u5c42\u539f\u7406 \u57fa\u4e8e2025\u5e74\u6700\u65b0\u5f00\u6e90\u751f\u6001（vLLM 0.5.0\u3001AutoGPTQ 0.7.1\u3001Transformers 4.41）\u5b9e\u73b0\u4ee3\u7801\u843d\u5730 \u9488\u5bf9\u4e0d\u540c\u573a\u666f（\u672c\u5730\u90e8\u7f72\u3001API\u670d\u52a1\u3001\u8fb9\u7f18\u8bbe\u5907\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.wsisp.com\/helps\/70070.html\" \/>\n<meta property=\"og:site_name\" content=\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"article:published_time\" content=\"2026-02-01T06:17:07+00:00\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"28 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/70070.html\",\"url\":\"https:\/\/www.wsisp.com\/helps\/70070.html\",\"name\":\"\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6838\u5fc3\u6280\u672f\u5b9e\u6218\uff1aKV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f\uff08\u9644\u6700\u65b0\u5f00\u6e90\u4ee3\u7801\uff09 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"isPartOf\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\"},\"datePublished\":\"2026-02-01T06:17:07+00:00\",\"dateModified\":\"2026-02-01T06:17:07+00:00\",\"author\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/70070.html#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.wsisp.com\/helps\/70070.html\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/70070.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.wsisp.com\/helps\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6838\u5fc3\u6280\u672f\u5b9e\u6218\uff1aKV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f\uff08\u9644\u6700\u65b0\u5f00\u6e90\u4ee3\u7801\uff09\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\",\"url\":\"https:\/\/www.wsisp.com\/helps\/\",\"name\":\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"description\":\"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"contentUrl\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"caption\":\"admin\"},\"sameAs\":[\"http:\/\/wp.wsisp.com\"],\"url\":\"https:\/\/www.wsisp.com\/helps\/author\/admin\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6838\u5fc3\u6280\u672f\u5b9e\u6218\uff1aKV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f\uff08\u9644\u6700\u65b0\u5f00\u6e90\u4ee3\u7801\uff09 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.wsisp.com\/helps\/70070.html","og_locale":"zh_CN","og_type":"article","og_title":"\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6838\u5fc3\u6280\u672f\u5b9e\u6218\uff1aKV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f\uff08\u9644\u6700\u65b0\u5f00\u6e90\u4ee3\u7801\uff09 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","og_description":"\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6838\u5fc3\u6280\u672f\u5b9e\u6218：KV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f（\u9644\u6700\u65b0\u5f00\u6e90\u4ee3\u7801） \u6587\u6863\u6982\u8ff0 \u6587\u7ae0\u6838\u5fc3\u4ef7\u503c \u6df1\u5ea6\u89e3\u6790\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u4e09\u5927\u6838\u5fc3\u6280\u672f（KV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f）\u7684\u5e95\u5c42\u539f\u7406 \u57fa\u4e8e2025\u5e74\u6700\u65b0\u5f00\u6e90\u751f\u6001（vLLM 0.5.0\u3001AutoGPTQ 0.7.1\u3001Transformers 4.41）\u5b9e\u73b0\u4ee3\u7801\u843d\u5730 \u9488\u5bf9\u4e0d\u540c\u573a\u666f（\u672c\u5730\u90e8\u7f72\u3001API\u670d\u52a1\u3001\u8fb9\u7f18\u8bbe\u5907","og_url":"https:\/\/www.wsisp.com\/helps\/70070.html","og_site_name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","article_published_time":"2026-02-01T06:17:07+00:00","author":"admin","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"admin","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"28 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.wsisp.com\/helps\/70070.html","url":"https:\/\/www.wsisp.com\/helps\/70070.html","name":"\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6838\u5fc3\u6280\u672f\u5b9e\u6218\uff1aKV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f\uff08\u9644\u6700\u65b0\u5f00\u6e90\u4ee3\u7801\uff09 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","isPartOf":{"@id":"https:\/\/www.wsisp.com\/helps\/#website"},"datePublished":"2026-02-01T06:17:07+00:00","dateModified":"2026-02-01T06:17:07+00:00","author":{"@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41"},"breadcrumb":{"@id":"https:\/\/www.wsisp.com\/helps\/70070.html#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.wsisp.com\/helps\/70070.html"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.wsisp.com\/helps\/70070.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.wsisp.com\/helps"},{"@type":"ListItem","position":2,"name":"\u5927\u6a21\u578b\u63a8\u7406\u52a0\u901f\u6838\u5fc3\u6280\u672f\u5b9e\u6218\uff1aKV Cache\u3001\u91cf\u5316\u3001\u6a21\u578b\u84b8\u998f\uff08\u9644\u6700\u65b0\u5f00\u6e90\u4ee3\u7801\uff09"}]},{"@type":"WebSite","@id":"https:\/\/www.wsisp.com\/helps\/#website","url":"https:\/\/www.wsisp.com\/helps\/","name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","description":"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41","name":"admin","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/","url":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","contentUrl":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","caption":"admin"},"sameAs":["http:\/\/wp.wsisp.com"],"url":"https:\/\/www.wsisp.com\/helps\/author\/admin"}]}},"_links":{"self":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/70070","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/comments?post=70070"}],"version-history":[{"count":0,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/70070\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media?parent=70070"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/categories?post=70070"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/tags?post=70070"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/topic?post=70070"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}