{"id":55101,"date":"2025-08-13T16:29:58","date_gmt":"2025-08-13T08:29:58","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/55101.html"},"modified":"2025-08-13T16:29:58","modified_gmt":"2025-08-13T08:29:58","slug":"%e4%ba%94%e7%ac%94bpe%e5%88%86%e8%af%8d%e5%99%a8%e7%9a%84%e6%8a%80%e6%9c%af%e6%bc%94%e8%bf%9b%e4%b8%8e%e5%85%b3%e9%94%ae%e4%bf%ae%e5%a4%8d","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/55101.html","title":{"rendered":"\u4e94\u7b14BPE\u5206\u8bcd\u5668\u7684\u6280\u672f\u6f14\u8fdb\u4e0e\u5173\u952e\u4fee\u590d"},"content":{"rendered":"

\u5728\u4e2d\u6587\u81ea\u7136\u8bed\u8a00\u5904\u7406\u4efb\u52a1\u4e2d，\u4e94\u7b14\u7f16\u7801\u4e0eByte Pair Encoding (BPE)\u7684\u7ed3\u5408\u63d0\u4f9b\u4e86\u4e00\u79cd\u72ec\u7279\u7684\u65b9\u6cd5\u3002\u672c\u6587\u5206\u4eab\u5728\u6784\u5efa\u4e94\u7b14BPE\u5206\u8bcd\u5668\u8fc7\u7a0b\u4e2d\u9047\u5230\u7684\u6280\u672f\u6311\u6218\u53ca\u5176\u89e3\u51b3\u65b9\u6848\u3002<\/p>\n

\u95ee\u9898\u80cc\u666f<\/h3>\n

\u4e94\u7b14BPE\u5206\u8bcd\u5668\u65e8\u5728：<\/p>\n

  • \u4f7f\u7528\u4e94\u7b14\u7f16\u7801\u8868\u793a\u4e2d\u6587\u5b57\u7b26<\/li>\n
  • \u5e94\u7528BPE\u7b97\u6cd5\u5904\u7406\u6df7\u5408\u5185\u5bb9（\u4e2d\u6587+\u975e\u4e2d\u6587）<\/li>\n
  • \u5b9e\u73b0\u9ad8\u6548\u7f16\u7801\/\u89e3\u7801\u6d41\u7a0b<\/li>\n

    \u4f46\u5728\u5b9e\u73b0\u8fc7\u7a0b\u4e2d，\u53d1\u73b0\u4e86\u4e24\u4e2a\u5173\u952e\u95ee\u9898：<\/p>\n

    \u95ee\u9898\u4e00：\u4e34\u65f6\u6587\u4ef6\u6743\u9650\u51b2\u7a81<\/h4>\n

    \u75c7\u72b6：\u5728Windows\u7cfb\u7edf\u4e0a\u8bad\u7ec3\u540e，\u4e34\u65f6\u6587\u4ef6\u65e0\u6cd5\u6b63\u5e38\u5220\u9664，\u51fa\u73b0\u6743\u9650\u9519\u8bef：<\/p>\n

    PermissionError: [WinError 32] The process cannot access the file because it is being used by another process<\/p>\n

    \u6839\u6e90：<\/p>\n

  • \u8bad\u7ec3\u65f6\u540c\u65f6\u6253\u5f00\u591a\u4e2a\u6587\u4ef6\u53e5\u67c4<\/li>\n
  • Windows\u7cfb\u7edf\u5bf9\u6587\u4ef6\u5220\u9664\u7684\u4e25\u683c\u9501\u5b9a<\/li>\n
  • \u4e34\u65f6\u6587\u4ef6\u547d\u540d\u51b2\u7a81\u5bfc\u81f4\u610f\u5916\u8986\u76d6<\/li>\n

    \u95ee\u9898\u4e8c：\u89e3\u7801\u903b\u8f91\u7f3a\u9677<\/h4>\n

    \u75c7\u72b6：\u5bf9\u6df7\u5408\u5185\u5bb9\u89e3\u7801\u65f6\u51fa\u73b0\u9519\u8bef：<\/p>\n

    "\u4f60\u597dabc" \u2192 \u89e3\u7801\u4e3a "\u4f60bca\u597d" (\u975e\u4e2d\u6587\u5b57\u7b26\u4f4d\u7f6e\u9519\u4e71)<\/p>\n

    \u6839\u6e90：<\/p>\n

  • \u539f\u59cb\u89e3\u7801\u903b\u8f91\u76f4\u63a5\u62fc\u63a5\u6807\u8bb0\u800c\u672a\u6b63\u786e\u5904\u7406\u5206\u9694\u7b26<\/li>\n
  • \u4e94\u7b14\u7f16\u7801("\u4e94"\u524d\u7f00)\u4e0e\u975e\u4e2d\u6587("\u975e"\u524d\u7f00)\u6df7\u5408\u65f6\u8fb9\u754c\u8bc6\u522b\u9519\u8bef<\/li>\n

    \u4fee\u590d\u65b9\u6848<\/h3>\n

    \u4fee\u590d\u4e00：\u5065\u58ee\u7684\u4e34\u65f6\u6587\u4ef6\u7ba1\u7406<\/h4>\n

    def<\/span> train<\/span>(<\/span>self,<\/span> corpus_path:<\/span> str<\/span>,<\/span> vocab_size:<\/span> int<\/span>,<\/span> chunk_size:<\/span> int<\/span> =<\/span> 10000<\/span>)<\/span>:<\/span>
    \n # \u751f\u6210\u552f\u4e00\u6587\u4ef6\u8def\u5f84<\/span>
    \n timestamp =<\/span> str<\/span>(<\/span>int<\/span>(<\/span>time.<\/span>time(<\/span>)<\/span>)<\/span>)<\/span>
    \n temp_path =<\/span> os.<\/span>path.<\/span>join(<\/span>"temp"<\/span>,<\/span> f"wubi_bpe_temp_<\/span>{<\/span>timestamp}<\/span><\/span>.txt"<\/span><\/span>)<\/span><\/p>\n

    try<\/span>:<\/span>
    \n # \u786e\u4fdd\u76ee\u5f55\u5b58\u5728<\/span>
    \n os.<\/span>makedirs(<\/span>"temp"<\/span>,<\/span> exist_ok=<\/span>True<\/span>)<\/span><\/p>\n

    # \u5904\u7406\u6570\u636e(\u7565)<\/span>
    \n with<\/span> open<\/span>(<\/span>temp_path,<\/span> 'w'<\/span>,<\/span> encoding=<\/span>'utf-8'<\/span>)<\/span> as<\/span> temp_file:<\/span>
    \n # \u6570\u636e\u5904\u7406\u903b\u8f91<\/span><\/p>\n

    # \u6267\u884cBPE\u5408\u5e76(\u7565)<\/span><\/p>\n

    finally<\/span>:<\/span>
    \n # \u5b89\u5168\u5220\u9664\u673a\u5236<\/span>
    \n try<\/span>:<\/span>
    \n if<\/span> os.<\/span>path.<\/span>exists(<\/span>temp_path)<\/span>:<\/span>
    \n os.<\/span>remove(<\/span>temp_path)<\/span>
    \n except<\/span> PermissionError:<\/span>
    \n time.<\/span>sleep(<\/span>0.5<\/span>)<\/span> # \u7b49\u5f85\u8d44\u6e90\u91ca\u653e<\/span>
    \n if<\/span> os.<\/span>path.<\/span>exists(<\/span>temp_path)<\/span>:<\/span>
    \n os.<\/span>remove(<\/span>temp_path)<\/span><\/p>\n

    \u5173\u952e\u6539\u8fdb：<\/p>\n

  • \u65f6\u95f4\u6233\u552f\u4e00\u547d\u540d\u907f\u514d\u51b2\u7a81<\/li>\n
  • try\/finally\u786e\u4fdd\u5220\u9664\u6267\u884c<\/li>\n
  • \u91cd\u8bd5\u673a\u5236\u89e3\u51b3Windows\u9501\u5b9a\u95ee\u9898<\/li>\n
  • \u76ee\u5f55\u9884\u5148\u521b\u5efa\u907f\u514d\u8def\u5f84\u9519\u8bef<\/li>\n

    \u4fee\u590d\u4e8c：\u7cbe\u51c6\u7684\u89e3\u7801\u903b\u8f91\u91cd\u6784<\/h4>\n

    def<\/span> decode<\/span>(<\/span>self,<\/span> ids:<\/span> List[<\/span>int<\/span>]<\/span>)<\/span> –<\/span>><\/span> str<\/span>:<\/span>
    \n tokens =<\/span> [<\/span>self.<\/span>id_to_token.<\/span>get(<\/span>id<\/span>,<\/span> "<unk>"<\/span>)<\/span> for<\/span> id<\/span> in<\/span> ids]<\/span>
    \n text =<\/span> ""<\/span>.<\/span>join(<\/span>tokens)<\/span>
    \n result =<\/span> [<\/span>]<\/span>
    \n sep =<\/span> ""<\/span>
    \n current =<\/span> ""<\/span><\/p>\n

    for<\/span> one in<\/span> text:<\/span>
    \n if<\/span> one ==<\/span> "\u975e"<\/span> or<\/span> one ==<\/span> "\u4e94"<\/span>:<\/span>
    \n if<\/span> sep ==<\/span> "\u975e"<\/span>:<\/span>
    \n result.<\/span>append(<\/span>current)<\/span>
    \n else<\/span>:<\/span>
    \n result.<\/span>append(<\/span>self.<\/span>wubi_converter.<\/span>convert_to_chinese(<\/span>[<\/span>current]<\/span>)<\/span>)<\/span>
    \n current =<\/span> ""<\/span>
    \n sep =<\/span> one
    \n else<\/span>:<\/span>
    \n current +=<\/span> one<\/p>\n

    # \u5904\u7406\u672b\u5c3e\u5185\u5bb9<\/span>
    \n if<\/span> sep ==<\/span> "\u4e94"<\/span>:<\/span>
    \n result.<\/span>append(<\/span>self.<\/span>wubi_converter.<\/span>convert_to_chinese(<\/span>[<\/span>current]<\/span>)<\/span>)<\/span>
    \n elif<\/span> sep ==<\/span> "\u975e"<\/span>:<\/span>
    \n result.<\/span>append(<\/span>current)<\/span><\/p>\n

    return<\/span> ''<\/span>.<\/span>join(<\/span>result)<\/span><\/p>\n

    \u5173\u952e\u6539\u8fdb：<\/p>\n

  • \u72b6\u6001\u8ddf\u8e2a(sep\u53d8\u91cf)\u8bb0\u5f55\u5f53\u524d\u524d\u7f00\u7c7b\u578b<\/li>\n
  • \u8fb9\u754c\u68c0\u6d4b\u51c6\u786e\u533a\u5206\u4e2d\u6587\/\u975e\u4e2d\u6587\u5185\u5bb9<\/li>\n
  • \u5355\u72ec\u5904\u7406\u672b\u5c3e\u5185\u5bb9\u907f\u514d\u622a\u65ad<\/li>\n
  • \u4f7f\u7528\u4e94\u7b14\u8f6c\u6362\u5668\u6b63\u786e\u8fd8\u539f\u4e2d\u6587<\/li>\n

    \u8865\u5145\u6539\u8fdb<\/h3>\n
  • \u589e\u5f3a\u7684\u9519\u8bef\u5904\u7406：<\/li>\n

    if<\/span> file_size ==<\/span> 0<\/span>:<\/span>
    \n raise<\/span> RuntimeError(<\/span>"Temporary file is empty…"<\/span>)<\/span><\/p>\n

  • \u8bad\u7ec3\u8fdb\u5ea6\u53cd\u9988：<\/li>\n

    print<\/span>(<\/span>f"Processing chunk <\/span>{<\/span>chunk_count}<\/span><\/span>…"<\/span><\/span>)<\/span>
    \nprint<\/span>(<\/span>f"Highest frequency pair: <\/span>{<\/span>best_pair}<\/span><\/span> (<\/span>{<\/span>best_count}<\/span><\/span>\u6b21)"<\/span><\/span>)<\/span><\/p>\n

  • \u8fb9\u7f18\u60c5\u51b5\u5904\u7406：<\/li>\n

    if<\/span> not<\/span> pair_freqs:<\/span>
    \n print<\/span>(<\/span>"No pairs found. Stopping merge process."<\/span>)<\/span>
    \n break<\/span><\/p>\n

    \u6d4b\u8bd5\u9a8c\u8bc1<\/h3>\n

    \u6d4b\u8bd5\u7528\u4f8b：\u201c\u4f60\u597d，\u7ae0\u8282\u6d4b\u8bd5abc!\u201d \u7f16\u89e3\u7801\u6d41\u7a0b:<\/p>\n

    \u539f\u59cb\u6587\u672c \u2192 \u7f16\u7801 \u2192 [<\/span>201<\/span>,<\/span> 42<\/span>,<\/span> 307<\/span>,<\/span> 15<\/span>,<\/span> 89<\/span>,<\/span> 305<\/span>]<\/span> \u2192 \u89e3\u7801 \u2192 \u8fd8\u539f\u6587\u672c<\/p>\n

    \u6d4b\u8bd5\u7ed3\u679c：<\/p>\n

    \u6d4b\u8bd5\u6587\u672c: '\u4f60\u597d，\u7ae0\u8282\u6d4b\u8bd5abc!'
    \n\u89e3\u7801\u7ed3\u679c: '\u4f60\u597d，\u7ae0\u8282\u6d4b\u8bd5abc!'
    \n\u91cd\u65b0\u7f16\u7801\u7ed3\u679c: [201, 42, 307, 15, 89, 305]
    \n\u662f\u5426\u4e00\u81f4: True<\/p>\n

    \u5e94\u7528\u4ef7\u503c<\/h3>\n

    \u8fd9\u4e9b\u4fee\u590d\u4f7f\u5206\u8bcd\u5668\u80fd\u591f：<\/p>\n

      \n
    • \u5728Windows\/Linux\u7cfb\u7edf\u7a33\u5b9a\u8fd0\u884c<\/li>\n
    • \u6b63\u786e\u5904\u7406\u6df7\u5408\u8bed\u8a00\u6587\u672c<\/li>\n
    • \u9002\u5e94\u751f\u4ea7\u73af\u5883\u9700\u6c42<\/li>\n
    • \u4e3a\u4e2d\u6587NLP\u4efb\u52a1\u63d0\u4f9b\u53ef\u9760\u57fa\u7840<\/li>\n<\/ul>\n

      \u5b8c\u6574\u5b9e\u73b0\u5df2\u5e94\u7528\u4e8e\u6587\u5b57\u5904\u7406\u7cfb\u7edf\u548c\u8f93\u5165\u6cd5\u5f15\u64ce，\u663e\u8457\u63d0\u5347\u4e86\u4e2d\u6587\u6587\u672c\u7684\u5904\u7406\u6548\u7387\u548c\u51c6\u786e\u5ea6\u3002<\/p>\n

      \u63a2\u7d22\u66f4\u591a\u4e2d\u6587\u5904\u7406\u6280\u672f：\u5173\u6ce8\u6211\u7684\u535a\u5ba2\u83b7\u53d6\u6700\u65b0\u66f4\u65b0！<\/p>\n

      import<\/span> json
      \nimport<\/span> re
      \nimport<\/span> tempfile
      \nimport<\/span> os
      \nimport<\/span> time
      \nimport<\/span> shutil
      \nfrom<\/span> collections import<\/span> defaultdict
      \nfrom<\/span> typing import<\/span> Dict,<\/span> List,<\/span> Tuple,<\/span> Generator,<\/span> Set,<\/span> Any<\/p>\n

      class<\/span> WubiConverter<\/span>:<\/span>
      \n def<\/span> __init__<\/span>(<\/span>self,<\/span> wubi_dict_path:<\/span> str<\/span> =<\/span> "wubi86.json"<\/span>)<\/span>:<\/span>
      \n self.<\/span>wubi_dict =<\/span> self.<\/span>_load_wubi_dict(<\/span>wubi_dict_path)<\/span>
      \n self.<\/span>reverse_wubi_dict =<\/span> defaultdict(<\/span>list<\/span>)<\/span>
      \n for<\/span> char,<\/span> code in<\/span> self.<\/span>wubi_dict.<\/span>items(<\/span>)<\/span>:<\/span>
      \n # \u79fb\u9664\u7f16\u7801\u672b\u5c3e\u6570\u5b57<\/span><\/p>\n

      self.<\/span>reverse_wubi_dict[<\/span>code]<\/span>.<\/span>append(<\/span>char)<\/span><\/p>\n

      def<\/span> _load_wubi_dict<\/span>(<\/span>self,<\/span> path:<\/span> str<\/span>)<\/span> –<\/span>><\/span> Dict[<\/span>str<\/span>,<\/span> str<\/span>]<\/span>:<\/span>
      \n try<\/span>:<\/span>
      \n with<\/span> open<\/span>(<\/span>path,<\/span> 'r'<\/span>,<\/span> encoding=<\/span>'utf-8'<\/span>)<\/span> as<\/span> f:<\/span>
      \n return<\/span> json.<\/span>load(<\/span>f)<\/span>
      \n except<\/span> Exception:<\/span>
      \n # \u786e\u4fdd\u57fa\u7840\u5b57\u5178\u5305\u542b\u6d4b\u8bd5\u5b57\u7b26<\/span>
      \n return<\/span> {<\/span>
      \n "\u4e00"<\/span>:<\/span> "g"<\/span>,<\/span> "\u7684"<\/span>:<\/span> "r"<\/span>,<\/span> "\u6211"<\/span>:<\/span> "trnt"<\/span>,<\/span> "\u597d"<\/span>:<\/span> "vb"<\/span>,<\/span> "\u4f60"<\/span>:<\/span> "wq"<\/span>,<\/span>
      \n "\u7ae0"<\/span>:<\/span> "ujj"<\/span>,<\/span> "\u8282"<\/span>:<\/span> "ab"<\/span>,<\/span> "\u6d4b"<\/span>:<\/span> "imj"<\/span>,<\/span> "\u8bd5"<\/span>:<\/span> "ya"<\/span>,<\/span> "\u8fd9"<\/span>:<\/span> "p"<\/span>,<\/span>
      \n "\u662f"<\/span>:<\/span> "j"<\/span>,<\/span> "\u6837"<\/span>:<\/span> "sud"<\/span>,<\/span> "\u6587"<\/span>:<\/span> "yygy"<\/span>,<\/span> "\u672c"<\/span>:<\/span> "sg"<\/span>,<\/span> "\u9a8c"<\/span>:<\/span> "cwgi"<\/span>,<\/span>
      \n "\u4e2d"<\/span>:<\/span> "k"<\/span>,<\/span> "\u6587"<\/span>:<\/span> "yygy"<\/span>,<\/span> "\u5904"<\/span>:<\/span> "thi"<\/span>,<\/span> "\u7406"<\/span>:<\/span> "gj"<\/span>,<\/span> "\u80fd"<\/span>:<\/span> "ce"<\/span>,<\/span>
      \n "\u529b"<\/span>:<\/span> "lt"<\/span>,<\/span> "\u6d4b"<\/span>:<\/span> "imj"<\/span>,<\/span> "\u8bd5"<\/span>:<\/span> "ya"<\/span>,<\/span> "\u4e94"<\/span>:<\/span> "gg"<\/span>,<\/span> "\u7b14"<\/span>:<\/span> "ttfn"<\/span>,<\/span>
      \n "\u8f93"<\/span>:<\/span> "lwg"<\/span>,<\/span> "\u5165"<\/span>:<\/span> "ty"<\/span>,<\/span> "\u6cd5"<\/span>:<\/span> "if"<\/span>,<\/span> "\u5e38"<\/span>:<\/span> "ipkh"<\/span>,<\/span> "\u89c1"<\/span>:<\/span> "mqb"<\/span>
      \n }<\/span><\/p>\n

      def<\/span> convert_to_wubi<\/span>(<\/span>self,<\/span> char:<\/span> str<\/span>)<\/span> –<\/span>><\/span> str<\/span>:<\/span>
      \n """\u8fd4\u56de\u5b57\u7b26\u7684\u4e94\u7b14\u7f16\u7801"""<\/span>
      \n return<\/span> self.<\/span>wubi_dict.<\/span>get(<\/span>char,<\/span> char)<\/span><\/p>\n

      def<\/span> convert_to_chinese<\/span>(<\/span>self,<\/span> wubi_codes:<\/span> List[<\/span>str<\/span>]<\/span>)<\/span> –<\/span>><\/span> str<\/span>:<\/span>
      \n """\u5c06\u4e94\u7b14\u7f16\u7801\u5217\u8868\u8f6c\u6362\u56de\u4e2d\u6587\u5b57\u7b26\u4e32"""<\/span>
      \n chars =<\/span> [<\/span>]<\/span>
      \n for<\/span> code in<\/span> wubi_codes:<\/span><\/p>\n

      candidates =<\/span> self.<\/span>reverse_wubi_dict.<\/span>get(<\/span>code,<\/span> [<\/span>code]<\/span>)<\/span>
      \n chars.<\/span>append(<\/span>candidates[<\/span>0<\/span>]<\/span>)<\/span>
      \n return<\/span> ''<\/span>.<\/span>join(<\/span>chars)<\/span><\/p>\n

      class<\/span> WubiBPETokenizer<\/span>:<\/span>
      \n def<\/span> __init__<\/span>(<\/span>self,<\/span> wubi_converter:<\/span> WubiConverter,<\/span> vocab:<\/span> Dict[<\/span>str<\/span>,<\/span> int<\/span>]<\/span> =<\/span> None<\/span>,<\/span> merges:<\/span> List[<\/span>str<\/span>]<\/span> =<\/span> None<\/span>)<\/span>:<\/span>
      \n self.<\/span>wubi_converter =<\/span> wubi_converter
      \n self.<\/span>special_tokens =<\/span> {<\/span>"<unk>"<\/span>:<\/span> 0<\/span>,<\/span> "<pad>"<\/span>:<\/span> 1<\/span>,<\/span> "<s>"<\/span>:<\/span> 2<\/span>,<\/span> "<\/s>"<\/span>:<\/span> 3<\/span>}<\/span>
      \n self.<\/span>vocab =<\/span> vocab or<\/span> self.<\/span>special_tokens.<\/span>copy(<\/span>)<\/span>
      \n self.<\/span>id_to_token =<\/span> {<\/span>idx:<\/span> token for<\/span> token,<\/span> idx in<\/span> self.<\/span>vocab.<\/span>items(<\/span>)<\/span>}<\/span>
      \n self.<\/span>merges =<\/span> merges or<\/span> [<\/span>]<\/span>
      \n self.<\/span>merge_dict =<\/span> {<\/span>}<\/span>
      \n if<\/span> merges:<\/span>
      \n for<\/span> pair in<\/span> merges:<\/span>
      \n a,<\/span> b =<\/span> pair.<\/span>split(<\/span>)<\/span>
      \n self.<\/span>merge_dict[<\/span>(<\/span>a,<\/span> b)<\/span>]<\/span> =<\/span> a +<\/span> b<\/p>\n

      @staticmethod<\/span>
      \n def<\/span> preprocess_stream<\/span>(<\/span>text_stream:<\/span> Generator[<\/span>str<\/span>,<\/span> None<\/span>,<\/span> None<\/span>]<\/span>)<\/span> –<\/span>><\/span> Generator[<\/span>str<\/span>,<\/span> None<\/span>,<\/span> None<\/span>]<\/span>:<\/span>
      \n """\u6d41\u5f0f\u9884\u5904\u7406：\u8bc6\u522b\u4e2d\u6587\u5b57\u7b26\u5e76\u6dfb\u52a0\u6807\u8bb0"""<\/span>
      \n current_non_cn =<\/span> [<\/span>]<\/span> # \u5b58\u653e\u8fde\u7eed\u975e\u4e2d\u6587\u5b57\u7b26<\/span><\/p>\n

      for<\/span> char in<\/span> text_stream:<\/span>
      \n # \u4e2d\u6587\u5b57\u7b26\u76f4\u63a5\u8f93\u51fa<\/span>
      \n if<\/span> '\\\\u4e00'<\/span> <=<\/span> char <=<\/span> '\\\\u9fff'<\/span>:<\/span>
      \n if<\/span> current_non_cn:<\/span>
      \n yield<\/span> "\u975e"<\/span> +<\/span> ""<\/span>.<\/span>join(<\/span>current_non_cn)<\/span>
      \n current_non_cn =<\/span> [<\/span>]<\/span>
      \n yield<\/span> char
      \n # \u975e\u4e2d\u6587\u5b57\u7b26\u7f13\u5b58<\/span>
      \n else<\/span>:<\/span>
      \n # \u5904\u7406\u7a7a\u683c\u548c\u6362\u884c\u7b26<\/span>
      \n if<\/span> char.<\/span>isspace(<\/span>)<\/span>:<\/span>
      \n if<\/span> current_non_cn:<\/span>
      \n yield<\/span> "\u975e"<\/span> +<\/span> ""<\/span>.<\/span>join(<\/span>current_non_cn)<\/span>
      \n current_non_cn =<\/span> [<\/span>]<\/span>
      \n yield<\/span> char
      \n else<\/span>:<\/span>
      \n current_non_cn.<\/span>append(<\/span>char)<\/span><\/p>\n

      # \u5904\u7406\u672b\u5c3e\u7684\u975e\u4e2d\u6587\u5b57\u7b26<\/span>
      \n if<\/span> current_non_cn:<\/span>
      \n yield<\/span> "\u975e"<\/span> +<\/span> ""<\/span>.<\/span>join(<\/span>current_non_cn)<\/span><\/p>\n

      @staticmethod<\/span>
      \n def<\/span> get_pairs<\/span>(<\/span>word:<\/span> Tuple[<\/span>str<\/span>,<\/span> .<\/span>.<\/span>.<\/span>]<\/span>)<\/span> –<\/span>><\/span> Generator[<\/span>Tuple[<\/span>str<\/span>,<\/span> str<\/span>]<\/span>,<\/span> None<\/span>,<\/span> None<\/span>]<\/span>:<\/span>
      \n """\u751f\u6210\u76f8\u90bbtoken\u5bf9（\u907f\u514d\u4e2d\u95f4\u5217\u8868）"""<\/span>
      \n prev_char =<\/span> word[<\/span>0<\/span>]<\/span>
      \n for<\/span> char in<\/span> word[<\/span>1<\/span>:<\/span>]<\/span>:<\/span>
      \n yield<\/span> (<\/span>prev_char,<\/span> char)<\/span>
      \n prev_char =<\/span> char<\/p>\n

      def<\/span> train<\/span>(<\/span>self,<\/span> corpus_path:<\/span> str<\/span>,<\/span> vocab_size:<\/span> int<\/span>,<\/span> chunk_size:<\/span> int<\/span> =<\/span> 10000<\/span>)<\/span>:<\/span>
      \n """\u4fee\u590d\u7684\u6587\u4ef6\u5199\u5165\u95ee\u9898\u548c\u6743\u9650\u95ee\u9898"""<\/span>
      \n # \u786e\u4fdd\u8f93\u51fa\u76ee\u5f55\u5b58\u5728<\/span>
      \n os.<\/span>makedirs(<\/span>"temp"<\/span>,<\/span> exist_ok=<\/span>True<\/span>)<\/span><\/p>\n

      # 1. \u751f\u6210\u552f\u4e00\u7684\u4e34\u65f6\u6587\u4ef6\u8def\u5f84（\u89e3\u51b3\u6743\u9650\u95ee\u9898）<\/span>
      \n timestamp =<\/span> str<\/span>(<\/span>int<\/span>(<\/span>time.<\/span>time(<\/span>)<\/span>)<\/span>)<\/span>
      \n temp_path =<\/span> os.<\/span>path.<\/span>join(<\/span>"temp"<\/span>,<\/span> f"wubi_bpe_temp_<\/span>{<\/span>timestamp}<\/span><\/span>.txt"<\/span><\/span>)<\/span>
      \n print<\/span>(<\/span>f"Using temporary file: <\/span>{<\/span>temp_path}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

      # 2. \u5b89\u5168\u521b\u5efa\u4e34\u65f6\u6587\u4ef6\u5e76\u5199\u5165\u6570\u636e<\/span>
      \n try<\/span>:<\/span>
      \n with<\/span> open<\/span>(<\/span>temp_path,<\/span> 'w'<\/span>,<\/span> encoding=<\/span>'utf-8'<\/span>)<\/span> as<\/span> temp_file:<\/span>
      \n # \u5904\u7406\u8bed\u6599\u6587\u4ef6<\/span>
      \n with<\/span> open<\/span>(<\/span>corpus_path,<\/span> 'r'<\/span>,<\/span> encoding=<\/span>'utf-8'<\/span>)<\/span> as<\/span> f:<\/span>
      \n chunk_count =<\/span> 0<\/span>
      \n char_count =<\/span> 0<\/span><\/p>\n

      while<\/span> True<\/span>:<\/span>
      \n chunk =<\/span> f.<\/span>read(<\/span>chunk_size)<\/span>
      \n if<\/span> not<\/span> chunk:<\/span>
      \n break<\/span><\/p>\n

      chunk_count +=<\/span> 1<\/span>
      \n # \u6253\u5370\u5f53\u524d\u5904\u7406\u8fdb\u5ea6<\/span>
      \n if<\/span> chunk_count %<\/span> 10<\/span> ==<\/span> 0<\/span>:<\/span>
      \n print<\/span>(<\/span>f"Processing chunk <\/span>{<\/span>chunk_count}<\/span><\/span>…"<\/span><\/span>)<\/span><\/p>\n

      # \u9884\u5904\u7406\u5e76\u8f6c\u6362\u4e2d\u6587<\/span>
      \n processed =<\/span> [<\/span>]<\/span>
      \n for<\/span> token in<\/span> self.<\/span>preprocess_stream(<\/span>(<\/span>c for<\/span> c in<\/span> chunk)<\/span>)<\/span>:<\/span>
      \n if<\/span> token and<\/span> not<\/span> token.<\/span>isspace(<\/span>)<\/span>:<\/span> # \u8df3\u8fc7\u7a7atoken\u548c\u7a7a\u683c<\/span>
      \n # \u4e2d\u6587\u5b57\u7b26\u8f6c\u6362\u4e3a\u4e94\u7b14\u7f16\u7801<\/span>
      \n if<\/span> '\\\\u4e00'<\/span> <=<\/span> token <=<\/span> '\\\\u9fff'<\/span>:<\/span>
      \n wubi_code =<\/span> self.<\/span>wubi_converter.<\/span>convert_to_wubi(<\/span>token)<\/span>
      \n processed.<\/span>append(<\/span>f"\u4e94<\/span>{<\/span>wubi_code}<\/span><\/span>"<\/span><\/span>)<\/span>
      \n else<\/span>:<\/span>
      \n processed.<\/span>append(<\/span>token)<\/span>
      \n char_count +=<\/span> 1<\/span><\/p>\n

      # \u5199\u5165\u5904\u7406\u540e\u7684\u5185\u5bb9<\/span>
      \n if<\/span> processed:<\/span>
      \n temp_file.<\/span>write(<\/span>" "<\/span>.<\/span>join(<\/span>processed)<\/span> +<\/span> "\\\\n"<\/span>)<\/span><\/p>\n

      # \u68c0\u67e5\u6587\u4ef6\u662f\u5426\u6210\u529f\u5199\u5165<\/span>
      \n file_size =<\/span> os.<\/span>path.<\/span>getsize(<\/span>temp_path)<\/span>
      \n print<\/span>(<\/span>f"Temporary file created: <\/span>{<\/span>file_size}<\/span><\/span> bytes, <\/span>{<\/span>char_count}<\/span><\/span> characters processed"<\/span><\/span>)<\/span><\/p>\n

      if<\/span> file_size ==<\/span> 0<\/span>:<\/span>
      \n raise<\/span> RuntimeError(<\/span>"Temporary file is empty. Please check input corpus and preprocessing."<\/span>)<\/span><\/p>\n

      # 3. \u6784\u5efa\u521d\u59cb\u8bcd\u6c47\u8868<\/span>
      \n self.<\/span>_build_initial_vocab(<\/span>temp_path)<\/span><\/p>\n

      # 4. \u6267\u884cBPE\u5408\u5e76<\/span>
      \n self.<\/span>_perform_merges(<\/span>temp_path,<\/span> vocab_size)<\/span><\/p>\n

      finally<\/span>:<\/span>
      \n # 5. \u5b89\u5168\u5220\u9664\u4e34\u65f6\u6587\u4ef6（\u89e3\u51b3\u6743\u9650\u95ee\u9898）<\/span>
      \n try<\/span>:<\/span>
      \n if<\/span> os.<\/span>path.<\/span>exists(<\/span>temp_path)<\/span>:<\/span>
      \n os.<\/span>remove(<\/span>temp_path)<\/span>
      \n print<\/span>(<\/span>f"Temporary file removed: <\/span>{<\/span>temp_path}<\/span><\/span>"<\/span><\/span>)<\/span>
      \n except<\/span> PermissionError as<\/span> pe:<\/span>
      \n print<\/span>(<\/span>f"Warning: Could not remove temporary file – <\/span>{<\/span>pe}<\/span><\/span>"<\/span><\/span>)<\/span>
      \n # \u5728Windows\u4e0a\u7a0d\u540e\u91cd\u8bd5<\/span>
      \n time.<\/span>sleep(<\/span>0.5<\/span>)<\/span>
      \n if<\/span> os.<\/span>path.<\/span>exists(<\/span>temp_path)<\/span>:<\/span>
      \n os.<\/span>remove(<\/span>temp_path)<\/span><\/p>\n

      def<\/span> _build_initial_vocab<\/span>(<\/span>self,<\/span> temp_path:<\/span> str<\/span>)<\/span>:<\/span>
      \n """\u6784\u5efa\u521d\u59cb\u8bcd\u6c47\u8868（\u4fee\u590d\u7a7a\u6587\u4ef6\u95ee\u9898）"""<\/span>
      \n print<\/span>(<\/span>"Building initial vocabulary…"<\/span>)<\/span>
      \n token_counts =<\/span> defaultdict(<\/span>int<\/span>)<\/span><\/p>\n

      with<\/span> open<\/span>(<\/span>temp_path,<\/span> 'r'<\/span>,<\/span> encoding=<\/span>'utf-8'<\/span>)<\/span> as<\/span> f:<\/span>
      \n line_count =<\/span> 0<\/span>
      \n for<\/span> line in<\/span> f:<\/span>
      \n line_count +=<\/span> 1<\/span>
      \n for<\/span> token in<\/span> line.<\/span>split(<\/span>)<\/span>:<\/span>
      \n token_counts[<\/span>token]<\/span> +=<\/span> 1<\/span><\/p>\n

      # \u786e\u4fdd\u5904\u7406\u4e86\u6570\u636e<\/span>
      \n if<\/span> not<\/span> token_counts:<\/span>
      \n raise<\/span> RuntimeError(<\/span>"No tokens found in temporary file. Check preprocessing."<\/span>)<\/span><\/p>\n

      # \u521d\u59cb\u5316\u8bcd\u6c47\u8868<\/span>
      \n self.<\/span>vocab =<\/span> self.<\/span>special_tokens.<\/span>copy(<\/span>)<\/span>
      \n next_id =<\/span> len<\/span>(<\/span>self.<\/span>special_tokens)<\/span><\/p>\n

      # \u6dfb\u52a0\u9ad8\u9891token<\/span>
      \n sorted_tokens =<\/span> sorted<\/span>(<\/span>token_counts.<\/span>items(<\/span>)<\/span>,<\/span> key=<\/span>lambda<\/span> x:<\/span> x[<\/span>1<\/span>]<\/span>,<\/span> reverse=<\/span>True<\/span>)<\/span>
      \n for<\/span> token,<\/span> count in<\/span> sorted_tokens:<\/span>
      \n if<\/span> token not<\/span> in<\/span> self.<\/span>vocab:<\/span>
      \n self.<\/span>vocab[<\/span>token]<\/span> =<\/span> next_id
      \n self.<\/span>id_to_token[<\/span>next_id]<\/span> =<\/span> token
      \n next_id +=<\/span> 1<\/span><\/p>\n

      # \u6dfb\u52a0\u57fa\u7840\u5b57\u7b26<\/span>
      \n all_chars =<\/span> set<\/span>(<\/span>""<\/span>.<\/span>join(<\/span>token_counts.<\/span>keys(<\/span>)<\/span>)<\/span>)<\/span>
      \n for<\/span> char in<\/span> sorted<\/span>(<\/span>all_chars)<\/span>:<\/span>
      \n if<\/span> char not<\/span> in<\/span> self.<\/span>vocab:<\/span>
      \n self.<\/span>vocab[<\/span>char]<\/span> =<\/span> next_id
      \n self.<\/span>id_to_token[<\/span>next_id]<\/span> =<\/span> char
      \n next_id +=<\/span> 1<\/span><\/p>\n

      print<\/span>(<\/span>f"Initial vocabulary built with <\/span>{<\/span>len<\/span>(<\/span>self.<\/span>vocab)<\/span>}<\/span><\/span> tokens"<\/span><\/span>)<\/span><\/p>\n

      def<\/span> _perform_merges<\/span>(<\/span>self,<\/span> temp_path:<\/span> str<\/span>,<\/span> vocab_size:<\/span> int<\/span>)<\/span>:<\/span>
      \n """\u6267\u884cBPE\u5408\u5e76（\u589e\u52a0\u65e5\u5fd7\u8f93\u51fa）"""<\/span>
      \n print<\/span>(<\/span>"Performing BPE merges…"<\/span>)<\/span>
      \n self.<\/span>merges =<\/span> [<\/span>]<\/span>
      \n self.<\/span>merge_dict =<\/span> {<\/span>}<\/span>
      \n next_id =<\/span> len<\/span>(<\/span>self.<\/span>vocab)<\/span><\/p>\n

      # \u8fed\u4ee3\u76f4\u5230\u8fbe\u5230\u76ee\u6807\u8bcd\u6c47\u8868\u5927\u5c0f<\/span>
      \n while<\/span> len<\/span>(<\/span>self.<\/span>vocab)<\/span> <<\/span> vocab_size:<\/span>
      \n pair_freqs =<\/span> defaultdict(<\/span>int<\/span>)<\/span>
      \n total_pairs =<\/span> 0<\/span><\/p>\n

      # \u626b\u63cf\u4e34\u65f6\u6587\u4ef6\u7edf\u8ba1\u76f8\u90bb\u5bf9\u9891\u7387<\/span>
      \n with<\/span> open<\/span>(<\/span>temp_path,<\/span> 'r'<\/span>,<\/span> encoding=<\/span>'utf-8'<\/span>)<\/span> as<\/span> f:<\/span>
      \n for<\/span> line in<\/span> f:<\/span>
      \n tokens =<\/span> line.<\/span>split(<\/span>)<\/span>
      \n if<\/span> not<\/span> tokens:<\/span>
      \n continue<\/span><\/p>\n

      # \u5e94\u7528\u73b0\u6709\u5408\u5e76\u89c4\u5219<\/span>
      \n merged_tokens =<\/span> [<\/span>]<\/span>
      \n for<\/span> token in<\/span> tokens:<\/span>
      \n chars =<\/span> list<\/span>(<\/span>token)<\/span>
      \n changed =<\/span> True<\/span>
      \n while<\/span> changed and<\/span> len<\/span>(<\/span>chars)<\/span> ><\/span> 1<\/span>:<\/span>
      \n changed =<\/span> False<\/span>
      \n i =<\/span> 0<\/span>
      \n new_chars =<\/span> [<\/span>]<\/span>
      \n while<\/span> i <<\/span> len<\/span>(<\/span>chars)<\/span>:<\/span>
      \n if<\/span> i <<\/span> len<\/span>(<\/span>chars)<\/span> –<\/span> 1<\/span> and<\/span> (<\/span>chars[<\/span>i]<\/span>,<\/span> chars[<\/span>i +<\/span> 1<\/span>]<\/span>)<\/span> in<\/span> self.<\/span>merge_dict:<\/span>
      \n new_chars.<\/span>append(<\/span>self.<\/span>merge_dict[<\/span>(<\/span>chars[<\/span>i]<\/span>,<\/span> chars[<\/span>i +<\/span> 1<\/span>]<\/span>)<\/span>]<\/span>)<\/span>
      \n i +=<\/span> 2<\/span>
      \n changed =<\/span> True<\/span>
      \n else<\/span>:<\/span>
      \n new_chars.<\/span>append(<\/span>chars[<\/span>i]<\/span>)<\/span>
      \n i +=<\/span> 1<\/span>
      \n chars =<\/span> new_chars
      \n merged_tokens.<\/span>append(<\/span>chars)<\/span><\/p>\n

      # \u7edf\u8ba1\u6bcf\u4e2a\u6807\u8bb0\u5185\u90e8\u7684\u76f8\u90bb\u5bf9<\/span>
      \n for<\/span> chars in<\/span> merged_tokens:<\/span>
      \n for<\/span> i in<\/span> range<\/span>(<\/span>len<\/span>(<\/span>chars)<\/span> –<\/span> 1<\/span>)<\/span>:<\/span>
      \n pair =<\/span> (<\/span>chars[<\/span>i]<\/span>,<\/span> chars[<\/span>i +<\/span> 1<\/span>]<\/span>)<\/span>
      \n pair_freqs[<\/span>pair]<\/span> +=<\/span> 1<\/span>
      \n total_pairs +=<\/span> 1<\/span><\/p>\n

      # \u5982\u679c\u6ca1\u6709\u627e\u5230\u4efb\u4f55\u5bf9，\u63d0\u524d\u9000\u51fa<\/span>
      \n if<\/span> not<\/span> pair_freqs:<\/span>
      \n print<\/span>(<\/span>"No pairs found. Stopping merge process."<\/span>)<\/span>
      \n break<\/span><\/p>\n

      # \u9009\u62e9\u6700\u9ad8\u9891\u5bf9<\/span>
      \n best_pair,<\/span> best_count =<\/span> max<\/span>(<\/span>pair_freqs.<\/span>items(<\/span>)<\/span>,<\/span> key=<\/span>lambda<\/span> x:<\/span> x[<\/span>1<\/span>]<\/span>)<\/span><\/p>\n

      # \u5982\u679c\u6700\u9ad8\u9891\u5bf9\u53ea\u51fa\u73b0\u4e00\u6b21，\u63d0\u524d\u9000\u51fa<\/span>
      \n if<\/span> best_count <=<\/span> 1<\/span>:<\/span>
      \n print<\/span>(<\/span>f"Highest frequency pair only appears once (<\/span>{<\/span>best_count}<\/span><\/span>). Stopping merge process."<\/span><\/span>)<\/span>
      \n break<\/span><\/p>\n

      new_token =<\/span> best_pair[<\/span>0<\/span>]<\/span> +<\/span> best_pair[<\/span>1<\/span>]<\/span><\/p>\n

      # \u6dfb\u52a0\u5230\u8bcd\u6c47\u8868<\/span>
      \n if<\/span> new_token not<\/span> in<\/span> self.<\/span>vocab:<\/span>
      \n self.<\/span>vocab[<\/span>new_token]<\/span> =<\/span> next_id
      \n self.<\/span>id_to_token[<\/span>next_id]<\/span> =<\/span> new_token
      \n next_id +=<\/span> 1<\/span>
      \n self.<\/span>merges.<\/span>append(<\/span>f"<\/span>{<\/span>best_pair[<\/span>0<\/span>]<\/span>}<\/span><\/span> <\/span>{<\/span>best_pair[<\/span>1<\/span>]<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span>
      \n self.<\/span>merge_dict[<\/span>best_pair]<\/span> =<\/span> new_token
      \n print<\/span>(<\/span>f"Merged: <\/span>{<\/span>best_pair}<\/span><\/span> -> '<\/span>{<\/span>new_token}<\/span><\/span>' (frequency: <\/span>{<\/span>best_count}<\/span><\/span>)"<\/span><\/span>)<\/span>
      \n else<\/span>:<\/span>
      \n print<\/span>(<\/span>f"Pair <\/span>{<\/span>best_pair}<\/span><\/span> already merged, skipping."<\/span><\/span>)<\/span>
      \n break<\/span><\/p>\n

      # \u6253\u5370\u8fdb\u5ea6<\/span>
      \n print<\/span>(<\/span>f"Progress: vocab size <\/span>{<\/span>len<\/span>(<\/span>self.<\/span>vocab)<\/span>}<\/span><\/span>\/<\/span>{<\/span>vocab_size}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

      print<\/span>(<\/span>f"Final vocabulary size: <\/span>{<\/span>len<\/span>(<\/span>self.<\/span>vocab)<\/span>}<\/span><\/span>, merges: <\/span>{<\/span>len<\/span>(<\/span>self.<\/span>merges)<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

      def<\/span> encode<\/span>(<\/span>self,<\/span> text:<\/span> str<\/span>)<\/span> –<\/span>><\/span> List[<\/span>int<\/span>]<\/span>:<\/span>
      \n """\u7f16\u7801\u6587\u672c\u4e3atoken ID\u5217\u8868"""<\/span>
      \n # \u9884\u5904\u7406\u548c\u4e2d\u6587\u8f6c\u6362<\/span>
      \n tokens =<\/span> [<\/span>]<\/span>
      \n for<\/span> token in<\/span> self.<\/span>preprocess_stream(<\/span>(<\/span>c for<\/span> c in<\/span> text)<\/span>)<\/span>:<\/span>
      \n if<\/span> token and<\/span> not<\/span> token.<\/span>isspace(<\/span>)<\/span>:<\/span>
      \n if<\/span> '\\\\u4e00'<\/span> <=<\/span> token <=<\/span> '\\\\u9fff'<\/span>:<\/span>
      \n wubi_code =<\/span> self.<\/span>wubi_converter.<\/span>convert_to_wubi(<\/span>token)<\/span>
      \n tokens.<\/span>append(<\/span>f"\u4e94<\/span>{<\/span>wubi_code}<\/span><\/span>"<\/span><\/span>)<\/span>
      \n else<\/span>:<\/span>
      \n tokens.<\/span>append(<\/span>token)<\/span><\/p>\n

      # \u5e94\u7528BPE\u5408\u5e76<\/span>
      \n merged_tokens =<\/span> [<\/span>]<\/span>
      \n for<\/span> token in<\/span> tokens:<\/span>
      \n chars =<\/span> list<\/span>(<\/span>token)<\/span>
      \n changed =<\/span> True<\/span>
      \n while<\/span> changed and<\/span> len<\/span>(<\/span>chars)<\/span> ><\/span> 1<\/span>:<\/span>
      \n changed =<\/span> False<\/span>
      \n i =<\/span> 0<\/span>
      \n new_chars =<\/span> [<\/span>]<\/span>
      \n while<\/span> i <<\/span> len<\/span>(<\/span>chars)<\/span>:<\/span>
      \n if<\/span> i <<\/span> len<\/span>(<\/span>chars)<\/span> –<\/span> 1<\/span> and<\/span> (<\/span>chars[<\/span>i]<\/span>,<\/span> chars[<\/span>i +<\/span> 1<\/span>]<\/span>)<\/span> in<\/span> self.<\/span>merge_dict:<\/span>
      \n new_chars.<\/span>append(<\/span>self.<\/span>merge_dict[<\/span>(<\/span>chars[<\/span>i]<\/span>,<\/span> chars[<\/span>i +<\/span> 1<\/span>]<\/span>)<\/span>]<\/span>)<\/span>
      \n i +=<\/span> 2<\/span>
      \n changed =<\/span> True<\/span>
      \n else<\/span>:<\/span>
      \n new_chars.<\/span>append(<\/span>chars[<\/span>i]<\/span>)<\/span>
      \n i +=<\/span> 1<\/span>
      \n chars =<\/span> new_chars
      \n merged_tokens.<\/span>extend(<\/span>chars)<\/span><\/p>\n

      # \u8f6c\u6362\u4e3aID<\/span>
      \n return<\/span> [<\/span>self.<\/span>vocab.<\/span>get(<\/span>token,<\/span> self.<\/span>vocab[<\/span>"<unk>"<\/span>]<\/span>)<\/span> for<\/span> token in<\/span> merged_tokens]<\/span><\/p>\n

      def<\/span> decode<\/span>(<\/span>self,<\/span> ids:<\/span> List[<\/span>int<\/span>]<\/span>)<\/span> –<\/span>><\/span> str<\/span>:<\/span>
      \n """\u4fee\u590d：\u6b63\u786e\u5904\u7406\u5206\u9694\u7b26\u548c\u4e2d\u6587\u8f6c\u6362"""<\/span>
      \n tokens =<\/span> [<\/span>self.<\/span>id_to_token.<\/span>get(<\/span>id<\/span>,<\/span> "<unk>"<\/span>)<\/span> for<\/span> id<\/span> in<\/span> ids]<\/span>
      \n text =<\/span> ""<\/span>.<\/span>join(<\/span>tokens)<\/span>
      \n result =<\/span> [<\/span>]<\/span>
      \n sep =<\/span> ""<\/span>
      \n current =<\/span> ""<\/span>
      \n for<\/span> one in<\/span> text:<\/span>
      \n if<\/span> one ==<\/span> "\u975e"<\/span> or<\/span> one ==<\/span> "\u4e94"<\/span>:<\/span>
      \n if<\/span> sep ==<\/span> "\u975e"<\/span>:<\/span>
      \n result.<\/span>append(<\/span>current)<\/span>
      \n current =<\/span> ""<\/span>
      \n else<\/span>:<\/span>
      \n result.<\/span>append(<\/span>self.<\/span>wubi_converter.<\/span>convert_to_chinese(<\/span>[<\/span>current]<\/span>)<\/span>)<\/span>
      \n current =<\/span> ""<\/span><\/p>\n

      sep =<\/span> one
      \n else<\/span>:<\/span>
      \n current +=<\/span> one
      \n if<\/span> sep ==<\/span> "\u4e94"<\/span>:<\/span>
      \n result.<\/span>append(<\/span>self.<\/span>wubi_converter.<\/span>convert_to_chinese(<\/span>[<\/span>current]<\/span>)<\/span>)<\/span>
      \n elif<\/span> sep ==<\/span> "\u975e"<\/span>:<\/span>
      \n result.<\/span>append(<\/span>current)<\/span><\/p>\n

      return<\/span> ''<\/span>.<\/span>join(<\/span>result)<\/span><\/p>\n

      def<\/span> save<\/span>(<\/span>self,<\/span> file_path:<\/span> str<\/span>)<\/span>:<\/span>
      \n """\u4fdd\u5b58\u5206\u8bcd\u5668\u6a21\u578b"""<\/span>
      \n tokenizer_config =<\/span> {<\/span>
      \n "model"<\/span>:<\/span> {<\/span>
      \n "type"<\/span>:<\/span> "BPE"<\/span>,<\/span>
      \n "vocab"<\/span>:<\/span> self.<\/span>vocab,<\/span>
      \n "merges"<\/span>:<\/span> self.<\/span>merges,<\/span>
      \n "unk_token"<\/span>:<\/span> "<unk>"<\/span>
      \n }<\/span>
      \n }<\/span>
      \n with<\/span> open<\/span>(<\/span>file_path,<\/span> 'w'<\/span>,<\/span> encoding=<\/span>'utf-8'<\/span>)<\/span> as<\/span> f:<\/span>
      \n json.<\/span>dump(<\/span>tokenizer_config,<\/span> f,<\/span> indent=<\/span>2<\/span>,<\/span> ensure_ascii=<\/span>False<\/span>)<\/span><\/p>\n

      @classmethod<\/span>
      \n def<\/span> load<\/span>(<\/span>cls,<\/span> file_path:<\/span> str<\/span>,<\/span> wubi_converter:<\/span> WubiConverter)<\/span> –<\/span>><\/span> 'WubiBPETokenizer'<\/span>:<\/span>
      \n """\u52a0\u8f7d\u5206\u8bcd\u5668\u6a21\u578b"""<\/span>
      \n with<\/span> open<\/span>(<\/span>file_path,<\/span> 'r'<\/span>,<\/span> encoding=<\/span>'utf-8'<\/span>)<\/span> as<\/span> f:<\/span>
      \n config =<\/span> json.<\/span>load(<\/span>f)<\/span>
      \n model =<\/span> config[<\/span>"model"<\/span>]<\/span>
      \n return<\/span> cls(<\/span>wubi_converter,<\/span> vocab=<\/span>model[<\/span>"vocab"<\/span>]<\/span>,<\/span> merges=<\/span>model[<\/span>"merges"<\/span>]<\/span>)<\/span><\/p>\n

      # =============== \u66f4\u5065\u58ee\u7684\u6d4b\u8bd5\u7528\u4f8b ================<\/span>
      \nif<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span>
      \n # \u786e\u4fdd\u4e34\u65f6\u76ee\u5f55\u5b58\u5728<\/span>
      \n os.<\/span>makedirs(<\/span>"temp"<\/span>,<\/span> exist_ok=<\/span>True<\/span>)<\/span><\/p>\n

      # \u521b\u5efa\u66f4\u4e30\u5bcc\u7684\u6d4b\u8bd5\u8bed\u6599<\/span>
      \n corpus_path =<\/span> "corpus.txt"<\/span>
      \n # with open(corpus_path, "w", encoding="utf-8") as f:<\/span>
      \n # f.write("\u4f60\u597d，\u8fd9\u662f\u4e00\u4e2a\u6d4b\u8bd5\u6587\u672c\u3002\u7528\u4e8e\u9a8c\u8bc1\u4e94\u7b14BPE\u5206\u8bcd\u5668\u3002\\\\n")<\/span>
      \n # f.write("Hello world! 123 \u6d4b\u8bd5\u6df7\u5408\u5185\u5bb9\u3002\u4e2d\u6587\u5904\u7406\u80fd\u529b\u6d4b\u8bd5\u3002\\\\n")<\/span>
      \n # f.write("\u4e94\u7b14\u8f93\u5165\u6cd5\u662f\u4e00\u79cd\u5e38\u89c1\u7684\u4e2d\u6587\u8f93\u5165\u65b9\u6cd5\u3002\\\\n")<\/span>
      \n # f.write("\u6211\u4eec\u9700\u8981\u786e\u4fdd\u8fd9\u4e2a\u8f6c\u6362\u5668\u80fd\u591f\u6b63\u786e\u5904\u7406\u5404\u79cd\u60c5\u51b5\u3002\\\\n")<\/span>
      \n # f.write("\u66f4\u591a\u6837\u5316\u7684\u5185\u5bb9\u6709\u52a9\u4e8e\u8bad\u7ec3\u66f4\u597d\u7684\u5206\u8bcd\u5668\u6a21\u578b\u3002\\\\n")<\/span>
      \n # f.write("\u6dfb\u52a0\u66f4\u591a\u53e5\u5b50\u4ee5\u589e\u52a0\u8bed\u6599\u5e93\u7684\u591a\u6837\u6027\u3002\\\\n")<\/span>
      \n # f.write("\u4e2d\u6587\u5206\u8bcd\u662f\u81ea\u7136\u8bed\u8a00\u5904\u7406\u4e2d\u7684\u91cd\u8981\u4efb\u52a1\u3002\\\\n")<\/span>
      \n # f.write("\u4e94\u7b14\u7f16\u7801\u53ef\u4ee5\u6709\u6548\u5730\u8868\u793a\u4e2d\u6587\u5b57\u7b26\u3002\\\\n")<\/span>
      \n # f.write("BPE\u7b97\u6cd5\u80fd\u591f\u4ece\u6570\u636e\u4e2d\u5b66\u4e60\u5408\u5e76\u89c4\u5219\u3002\\\\n")<\/span>
      \n # f.write("\u8fd9\u4e2a\u5b9e\u73b0\u7ed3\u5408\u4e86\u4e94\u7b14\u7f16\u7801\u548cBPE\u7b97\u6cd5\u7684\u4f18\u70b9\u3002\\\\n")<\/span><\/p>\n

      # \u521d\u59cb\u5316\u8f6c\u6362\u5668<\/span>
      \n print<\/span>(<\/span>"\u52a0\u8f7d\u4e94\u7b14\u5b57\u5178…"<\/span>)<\/span>
      \n wubi_conv =<\/span> WubiConverter(<\/span>)<\/span><\/p>\n

      # \u8bad\u7ec3\u5206\u8bcd\u5668<\/span>
      \n print<\/span>(<\/span>"\\\\n\u8bad\u7ec3\u5206\u8bcd\u5668…"<\/span>)<\/span>
      \n tokenizer =<\/span> WubiBPETokenizer(<\/span>wubi_conv)<\/span><\/p>\n

      # \u8bbe\u7f6e\u5408\u7406\u7684\u8bcd\u6c47\u8868\u5927\u5c0f<\/span>
      \n tokenizer.<\/span>train(<\/span>corpus_path,<\/span> vocab_size=<\/span>5000<\/span>)<\/span><\/p>\n

      # \u6253\u5370\u8bad\u7ec3\u7ed3\u679c<\/span>
      \n print<\/span>(<\/span>f"\\\\n\u8bcd\u6c47\u8868\u5927\u5c0f: <\/span>{<\/span>len<\/span>(<\/span>tokenizer.<\/span>vocab)<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span>
      \n print<\/span>(<\/span>f"\u7279\u6b8a\u6807\u8bb0: <\/span>{<\/span>tokenizer.<\/span>special_tokens}<\/span><\/span>"<\/span><\/span>)<\/span>
      \n print<\/span>(<\/span>f"\u5408\u5e76\u89c4\u5219 (<\/span>{<\/span>len<\/span>(<\/span>tokenizer.<\/span>merges)<\/span>}<\/span><\/span>\u6761):"<\/span><\/span>)<\/span>
      \n for<\/span> i,<\/span> merge in<\/span> enumerate<\/span>(<\/span>tokenizer.<\/span>merges[<\/span>:<\/span>10<\/span>]<\/span>)<\/span>:<\/span>
      \n print<\/span>(<\/span>f" <\/span>{<\/span>i +<\/span> 1<\/span>}<\/span><\/span>. <\/span>{<\/span>merge}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

      # \u6d4b\u8bd5\u7f16\u7801\u89e3\u7801<\/span>
      \n test_text =<\/span> "\u4f60\u597d，\u7ae0\u8282\u6d4b\u8bd5abc!"<\/span>
      \n print<\/span>(<\/span>f"\\\\n\u6d4b\u8bd5\u6587\u672c: '<\/span>{<\/span>test_text}<\/span><\/span>'"<\/span><\/span>)<\/span><\/p>\n

      # \u7f16\u7801<\/span>
      \n ids =<\/span> tokenizer.<\/span>encode(<\/span>test_text)<\/span>
      \n print<\/span>(<\/span>f"\u7f16\u7801\u7ed3\u679c (<\/span>{<\/span>len<\/span>(<\/span>ids)<\/span>}<\/span><\/span>\u4e2atoken): <\/span>{<\/span>ids}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

      # \u89e3\u7801<\/span>
      \n decoded =<\/span> tokenizer.<\/span>decode(<\/span>ids)<\/span>
      \n print<\/span>(<\/span>f"\u89e3\u7801\u7ed3\u679c: '<\/span>{<\/span>decoded}<\/span><\/span>'"<\/span><\/span>)<\/span><\/p>\n

      # \u4fdd\u5b58\u6a21\u578b<\/span>
      \n tokenizer.<\/span>save(<\/span>"wubi_bpe_tokenizer.json"<\/span>)<\/span>
      \n print<\/span>(<\/span>"\\\\n\u5206\u8bcd\u5668\u6a21\u578b\u5df2\u4fdd\u5b58"<\/span>)<\/span><\/p>\n

      # \u52a0\u8f7d\u6a21\u578b\u6d4b\u8bd5<\/span>
      \n print<\/span>(<\/span>"\\\\n\u52a0\u8f7d\u4fdd\u5b58\u7684\u5206\u8bcd\u5668…"<\/span>)<\/span>
      \n loaded_tokenizer =<\/span> WubiBPETokenizer.<\/span>load(<\/span>"wubi_bpe_tokenizer.json"<\/span>,<\/span> wubi_conv)<\/span>
      \n reloaded_ids =<\/span> loaded_tokenizer.<\/span>encode(<\/span>test_text)<\/span>
      \n print<\/span>(<\/span>f"\u91cd\u65b0\u7f16\u7801\u7ed3\u679c: <\/span>{<\/span>reloaded_ids}<\/span><\/span>"<\/span><\/span>)<\/span>
      \n print<\/span>(<\/span>f"\u662f\u5426\u4e00\u81f4: <\/span>{<\/span>ids ==<\/span> reloaded_ids}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

      \u6587\u7ae0\u6d4f\u89c8\u9605\u8bfb399\u6b21\uff0c\u70b9\u8d5e6\u6b21\uff0c\u6536\u85cf2\u6b21\u3002\u5728\u4e2d\u6587\u81ea\u7136\u8bed\u8a00\u5904\u7406\u4efb\u52a1\u4e2d\uff0c\u4e94\u7b14\u7f16\u7801\u4e0eByte Pair Encoding (BPE)\u7684\u7ed3\u5408\u63d0\u4f9b\u4e86\u4e00\u79cd\u72ec\u7279\u7684\u65b9\u6cd5\u3002\u672c\u6587\u5206\u4eab\u5728\u6784\u5efa\u4e94\u7b14BPE\u5206\u8bcd\u5668\u8fc7\u7a0b\u4e2d\u9047\u5230\u7684\u6280\u672f\u6311\u6218\u53ca\u5176\u89e3\u51b3\u65b9\u6848\u3002<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[50],"topic":[],"class_list":["post-55101","post","type-post","status-publish","format-standard","hentry","category-server","tag-50"],"yoast_head":"\n\u4e94\u7b14BPE\u5206\u8bcd\u5668\u7684\u6280\u672f\u6f14\u8fdb\u4e0e\u5173\u952e\u4fee\u590d - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.wsisp.com\/helps\/55101.html\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u4e94\u7b14BPE\u5206\u8bcd\u5668\u7684\u6280\u672f\u6f14\u8fdb\u4e0e\u5173\u952e\u4fee\u590d - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"og:description\" content=\"\u6587\u7ae0\u6d4f\u89c8\u9605\u8bfb399\u6b21\uff0c\u70b9\u8d5e6\u6b21\uff0c\u6536\u85cf2\u6b21\u3002\u5728\u4e2d\u6587\u81ea\u7136\u8bed\u8a00\u5904\u7406\u4efb\u52a1\u4e2d\uff0c\u4e94\u7b14\u7f16\u7801\u4e0eByte Pair Encoding (BPE)\u7684\u7ed3\u5408\u63d0\u4f9b\u4e86\u4e00\u79cd\u72ec\u7279\u7684\u65b9\u6cd5\u3002\u672c\u6587\u5206\u4eab\u5728\u6784\u5efa\u4e94\u7b14BPE\u5206\u8bcd\u5668\u8fc7\u7a0b\u4e2d\u9047\u5230\u7684\u6280\u672f\u6311\u6218\u53ca\u5176\u89e3\u51b3\u65b9\u6848\u3002\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.wsisp.com\/helps\/55101.html\" \/>\n<meta property=\"og:site_name\" content=\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"article:published_time\" content=\"2025-08-13T08:29:58+00:00\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/55101.html\",\"url\":\"https:\/\/www.wsisp.com\/helps\/55101.html\",\"name\":\"\u4e94\u7b14BPE\u5206\u8bcd\u5668\u7684\u6280\u672f\u6f14\u8fdb\u4e0e\u5173\u952e\u4fee\u590d - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"isPartOf\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\"},\"datePublished\":\"2025-08-13T08:29:58+00:00\",\"dateModified\":\"2025-08-13T08:29:58+00:00\",\"author\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/55101.html#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.wsisp.com\/helps\/55101.html\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/55101.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.wsisp.com\/helps\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u4e94\u7b14BPE\u5206\u8bcd\u5668\u7684\u6280\u672f\u6f14\u8fdb\u4e0e\u5173\u952e\u4fee\u590d\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\",\"url\":\"https:\/\/www.wsisp.com\/helps\/\",\"name\":\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"description\":\"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"contentUrl\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"caption\":\"admin\"},\"sameAs\":[\"http:\/\/wp.wsisp.com\"],\"url\":\"https:\/\/www.wsisp.com\/helps\/author\/admin\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"\u4e94\u7b14BPE\u5206\u8bcd\u5668\u7684\u6280\u672f\u6f14\u8fdb\u4e0e\u5173\u952e\u4fee\u590d - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.wsisp.com\/helps\/55101.html","og_locale":"zh_CN","og_type":"article","og_title":"\u4e94\u7b14BPE\u5206\u8bcd\u5668\u7684\u6280\u672f\u6f14\u8fdb\u4e0e\u5173\u952e\u4fee\u590d - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","og_description":"\u6587\u7ae0\u6d4f\u89c8\u9605\u8bfb399\u6b21\uff0c\u70b9\u8d5e6\u6b21\uff0c\u6536\u85cf2\u6b21\u3002\u5728\u4e2d\u6587\u81ea\u7136\u8bed\u8a00\u5904\u7406\u4efb\u52a1\u4e2d\uff0c\u4e94\u7b14\u7f16\u7801\u4e0eByte Pair Encoding (BPE)\u7684\u7ed3\u5408\u63d0\u4f9b\u4e86\u4e00\u79cd\u72ec\u7279\u7684\u65b9\u6cd5\u3002\u672c\u6587\u5206\u4eab\u5728\u6784\u5efa\u4e94\u7b14BPE\u5206\u8bcd\u5668\u8fc7\u7a0b\u4e2d\u9047\u5230\u7684\u6280\u672f\u6311\u6218\u53ca\u5176\u89e3\u51b3\u65b9\u6848\u3002","og_url":"https:\/\/www.wsisp.com\/helps\/55101.html","og_site_name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","article_published_time":"2025-08-13T08:29:58+00:00","author":"admin","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"admin","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"9 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.wsisp.com\/helps\/55101.html","url":"https:\/\/www.wsisp.com\/helps\/55101.html","name":"\u4e94\u7b14BPE\u5206\u8bcd\u5668\u7684\u6280\u672f\u6f14\u8fdb\u4e0e\u5173\u952e\u4fee\u590d - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","isPartOf":{"@id":"https:\/\/www.wsisp.com\/helps\/#website"},"datePublished":"2025-08-13T08:29:58+00:00","dateModified":"2025-08-13T08:29:58+00:00","author":{"@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41"},"breadcrumb":{"@id":"https:\/\/www.wsisp.com\/helps\/55101.html#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.wsisp.com\/helps\/55101.html"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.wsisp.com\/helps\/55101.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.wsisp.com\/helps"},{"@type":"ListItem","position":2,"name":"\u4e94\u7b14BPE\u5206\u8bcd\u5668\u7684\u6280\u672f\u6f14\u8fdb\u4e0e\u5173\u952e\u4fee\u590d"}]},{"@type":"WebSite","@id":"https:\/\/www.wsisp.com\/helps\/#website","url":"https:\/\/www.wsisp.com\/helps\/","name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","description":"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41","name":"admin","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/","url":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","contentUrl":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","caption":"admin"},"sameAs":["http:\/\/wp.wsisp.com"],"url":"https:\/\/www.wsisp.com\/helps\/author\/admin"}]}},"_links":{"self":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/55101","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/comments?post=55101"}],"version-history":[{"count":0,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/55101\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media?parent=55101"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/categories?post=55101"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/tags?post=55101"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/topic?post=55101"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}