{"id":104173,"date":"2026-09-11T23:25:09","date_gmt":"2026-09-11T15:25:09","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/104173.html"},"modified":"2026-09-11T23:25:09","modified_gmt":"2026-09-11T15:25:09","slug":"nlp%e9%a2%84%e5%a4%84%e7%90%86python%e5%86%85%e7%bd%ae%e5%87%bd%e6%95%b0_02_%e5%88%86%e8%af%8d%e8%be%85%e5%8a%a9%e4%b8%8e%e8%bf%87%e6%bb%a4%e7%ad%9b%e9%80%89","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/104173.html","title":{"rendered":"NLP\u9884\u5904\u7406Python\u5185\u7f6e\u51fd\u6570_02_\u5206\u8bcd\u8f85\u52a9\u4e0e\u8fc7\u6ee4\u7b5b\u9009"},"content":{"rendered":"

NLP \u9884\u5904\u7406\u5fc5\u5907\u7684 Python \u5185\u7f6e\u51fd\u6570（\u4e8c）：\u5206\u8bcd\u8f85\u52a9\u4e0e\u8fc7\u6ee4\u7b5b\u9009<\/h2>\n

\u5199\u5728\u524d\u9762：\u4e0a\u4e00\u7bc7\u6211\u4eec\u628a\u300c\u6587\u672c\u6e05\u6d17\u300d\u548c\u300c\u6587\u672c\u89c4\u8303\u5316\u300d\u8bb2\u900f\u4e86（strip()\u3001replace()\u3001re.sub()\u3001lower()\u3001unicodedata.normalize() \u7b49）\u3002\u672c\u7bc7\u8fdb\u5165\u6d41\u6c34\u7ebf\u7684\u4e2d\u95f4\u73af\u8282\u2014\u2014\u5206\u8bcd\u8f85\u52a9 \u4e0e \u8fc7\u6ee4\u7b5b\u9009\u3002<\/p>\n

\u6587\u672c\u6d17\u767d\u4e4b\u540e，\u5c31\u8981\u628a\u5b83\u5207\u6210\u4e00\u4e2a\u4e2a token（\u8bcd），\u518d\u628a\u566a\u58f0 token（\u505c\u7528\u8bcd\u3001\u7a7a\u4e32\u3001\u7eaf\u6570\u5b57）\u8fc7\u6ee4\u6389\u3002\u8fd9\u4e2d\u95f4\u7528\u5230\u7684\u65b9\u6cd5，\u5168\u90e8\u662f Python \u5185\u7f6e，\u96f6\u7b2c\u4e09\u65b9\u4f9d\u8d56\u3002<\/p>\n

\n

\u76ee\u5f55<\/h3>\n
  • \u5206\u8bcd\u4e0e\u5207\u5206\u8f85\u52a9<\/li>\n
  • \u8fc7\u6ee4\u4e0e\u7b5b\u9009<\/li>\n
  • \u7efc\u5408\u5b9e\u6218：\u5b8c\u6574\u7684\u8fc7\u6ee4\u6e05\u6d17\u6d41\u6c34\u7ebf<\/li>\n
  • \u5c0f\u7ed3<\/li>\n
    \n

    \u4e00\u3001\u5206\u8bcd\u4e0e\u5207\u5206\u8f85\u52a9<\/h3>\n

    1. str.split() \u2014\u2014 \u5206\u8bcd\u7684\u4e3b\u529b\u5de5\u5177<\/h4>\n

    \u8bed\u6cd5<\/p>\n

    str<\/span>.<\/span>split(<\/span>sep=<\/span>None<\/span>,<\/span> maxsplit=<\/span>–<\/span>1<\/span>)<\/span><\/p>\n

    \u53c2\u6570\u8bf4\u660e<\/p>\n\n\u53c2\u6570\u7c7b\u578b\u8bf4\u660e<\/tr>\n\n\n\n
    sep<\/td>\nstr，\u53ef\u9009<\/td>\n\u5206\u9694\u7b26\u3002\u4e0d\u4f20\u65f6\u6309\u4efb\u610f\u7a7a\u767d（\u7a7a\u683c\u3001\\\\t\u3001\\\\n）\u5207\u5206，\u4e14\u81ea\u52a8\u8fc7\u6ee4\u7a7a\u4e32；\u4f20\u5165\u65f6\u4e25\u683c\u6309\u8be5\u5b57\u7b26\u4e32\u5207\u5206，\u7a7a\u4e32\u4f1a\u4fdd\u7559<\/td>\n<\/tr>\n
    maxsplit<\/td>\nint，\u53ef\u9009<\/td>\n\u6700\u5927\u5207\u5206\u6b21\u6570，-1 \u8868\u793a\u4e0d\u9650\u6b21\u6570<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    \u8fd4\u56de\u503c：\u8fd4\u56de\u5b57\u7b26\u4e32\u5217\u8868\u3002<\/p>\n

    \u793a\u4f8b\u4ee3\u7801<\/p>\n

    # \u4e0d\u4f20 sep：\u6309\u4efb\u610f\u7a7a\u767d\u5207\u5206，\u81ea\u52a8\u5904\u7406\u591a\u4e2a\u7a7a\u683c<\/span>
    \ns =<\/span> " hello world python "<\/span>
    \nprint<\/span>(<\/span>s.<\/span>split(<\/span>)<\/span>)<\/span> # ['hello', 'world', 'python']<\/span><\/p>\n

    # \u4f20 sep：\u4e25\u683c\u6309\u5206\u9694\u7b26\u5207\u5206，\u7a7a\u4e32\u4f1a\u4fdd\u7559<\/span>
    \ns2 =<\/span> "a,,b,c"<\/span>
    \nprint<\/span>(<\/span>s2.<\/span>split(<\/span>","<\/span>)<\/span>)<\/span> # ['a', '', 'b', 'c'] \u2190 \u6ce8\u610f\u4e2d\u95f4\u7684\u7a7a\u4e32<\/span><\/p>\n

    # maxsplit \u9650\u5236\u6b21\u6570<\/span>
    \ns3 =<\/span> "a,b,c,d"<\/span>
    \nprint<\/span>(<\/span>s3.<\/span>split(<\/span>","<\/span>,<\/span> 2<\/span>)<\/span>)<\/span> # ['a', 'b', 'c,d'] \u2190 \u53ea\u5207\u524d 2 \u6b21<\/span><\/p>\n

    # \u4e2d\u6587\u6309\u6807\u70b9\u5207\u5206<\/span>
    \ns4 =<\/span> "\u81ea\u7136\u8bed\u8a00\u5904\u7406,\u4eba\u5de5\u667a\u80fd,\u673a\u5668\u5b66\u4e60"<\/span>
    \nprint<\/span>(<\/span>s4.<\/span>split(<\/span>","<\/span>)<\/span>)<\/span> # ['\u81ea\u7136\u8bed\u8a00\u5904\u7406', '\u4eba\u5de5\u667a\u80fd', '\u673a\u5668\u5b66\u4e60']<\/span><\/p>\n

    NLP \u573a\u666f\u5e94\u7528<\/p>\n

    # \u82f1\u6587\u6309\u7a7a\u767d\u5206\u8bcd（\u6700\u5e38\u7528）<\/span>
    \nsentence =<\/span> "I love natural language processing"<\/span>
    \ntokens =<\/span> sentence.<\/span>split(<\/span>)<\/span>
    \nprint<\/span>(<\/span>tokens)<\/span> # ['I', 'love', 'natural', 'language', 'processing']<\/span><\/p>\n

    # \u89e3\u6790 CSV \u884c<\/span>
    \nrow =<\/span> "\u5f20\u4e09,25,\u5317\u4eac,\u5de5\u7a0b\u5e08"<\/span>
    \nfields =<\/span> row.<\/span>split(<\/span>","<\/span>)<\/span>
    \nprint<\/span>(<\/span>fields)<\/span> # ['\u5f20\u4e09', '25', '\u5317\u4eac', '\u5de5\u7a0b\u5e08']<\/span><\/p>\n

    \u6ce8\u610f\u4e8b\u9879（\u9ad8\u9891\u6613\u9519\u70b9）<\/p>\n

  • "a b".split() \u548c "a b".split(" ") \u7ed3\u679c\u4e0d\u4e00\u6837：\u524d\u8005\u81ea\u52a8\u5408\u5e76\u8fde\u7eed\u7a7a\u767d\u5e76\u53bb\u7a7a\u4e32，\u540e\u8005\u9047\u5230\u8fde\u7eed\u7a7a\u683c\u4f1a\u4ea7\u751f\u7a7a\u4e32；<\/li>\n
  • \u4e2d\u6587\u5206\u8bcd\u4e0d\u8981\u76f4\u63a5\u7528 split()（\u4e2d\u6587\u8bcd\u4e4b\u95f4\u6ca1\u6709\u7a7a\u683c），\u4ea4\u7ed9 jieba \u8fd9\u7c7b\u5206\u8bcd\u5e93；split() \u53ea\u8d1f\u8d23\u6309\u6807\u70b9\/\u7a7a\u767d\u7c97\u5207\u3002<\/li>\n
    \n

    2. str.join() \u2014\u2014 \u5206\u8bcd\u7684\u9006\u64cd\u4f5c<\/h4>\n

    \u8bed\u6cd5<\/p>\n

    str<\/span>.<\/span>join(<\/span>iterable)<\/span><\/p>\n

    \u53c2\u6570\u8bf4\u660e<\/p>\n\n\u53c2\u6570\u7c7b\u578b\u8bf4\u660e<\/tr>\n\n\n
    iterable<\/td>\n\u53ef\u8fed\u4ee3\u5bf9\u8c61<\/td>\n\u5217\u8868\u3001\u5143\u7ec4\u3001\u751f\u6210\u5668\u7b49，\u5143\u7d20\u5fc5\u987b\u662f\u5b57\u7b26\u4e32<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    \u8fd4\u56de\u503c：\u7528\u8c03\u7528\u8005\u5b57\u7b26\u4e32\u4f5c\u4e3a\u5206\u9694\u7b26，\u62fc\u63a5\u5404\u5143\u7d20\u540e\u8fd4\u56de\u65b0\u5b57\u7b26\u4e32\u3002<\/p>\n

    \u793a\u4f8b\u4ee3\u7801<\/p>\n

    print<\/span>(<\/span>" "<\/span>.<\/span>join(<\/span>[<\/span>"I"<\/span>,<\/span> "love"<\/span>,<\/span> "Python"<\/span>]<\/span>)<\/span>)<\/span> # "I love Python"<\/span>
    \nprint<\/span>(<\/span>","<\/span>.<\/span>join(<\/span>[<\/span>"a"<\/span>,<\/span> "b"<\/span>,<\/span> "c"<\/span>]<\/span>)<\/span>)<\/span> # "a,b,c"<\/span>
    \nprint<\/span>(<\/span>""<\/span>.<\/span>join(<\/span>[<\/span>"a"<\/span>,<\/span> "b"<\/span>,<\/span> "c"<\/span>]<\/span>)<\/span>)<\/span> # "abc"<\/span>
    \nprint<\/span>(<\/span>"-"<\/span>.<\/span>join(<\/span>"abc"<\/span>)<\/span>)<\/span> # "a-b-c"（\u5b57\u7b26\u4e32\u672c\u8eab\u53ef\u8fed\u4ee3）<\/span><\/p>\n

    # \u751f\u6210\u5668\u4e5f\u53ef\u4ee5<\/span>
    \nprint<\/span>(<\/span>"|"<\/span>.<\/span>join(<\/span>str<\/span>(<\/span>i)<\/span> for<\/span> i in<\/span> range<\/span>(<\/span>3<\/span>)<\/span>)<\/span>)<\/span> # "0|1|2"<\/span><\/p>\n

    NLP \u573a\u666f\u5e94\u7528<\/p>\n

    # \u5206\u8bcd \u2192 \u5904\u7406 \u2192 \u91cd\u7ec4\u53e5\u5b50（\u6bd4\u5982\u53bb\u6389\u505c\u7528\u8bcd\u540e\u518d\u62fc\u56de\u53bb）<\/span>
    \ntokens =<\/span> [<\/span>"\u6211"<\/span>,<\/span> "\u559c\u6b22"<\/span>,<\/span> "\u7684"<\/span>,<\/span> "Python"<\/span>]<\/span>
    \nfiltered =<\/span> [<\/span>t for<\/span> t in<\/span> tokens if<\/span> t !=<\/span> "\u7684"<\/span>]<\/span>
    \nprint<\/span>(<\/span>" "<\/span>.<\/span>join(<\/span>filtered)<\/span>)<\/span> # "\u6211 \u559c\u6b22 Python"<\/span><\/p>\n

    # \u751f\u6210 CSV \u884c<\/span>
    \nrow =<\/span> [<\/span>"\u5f20\u4e09"<\/span>,<\/span> "25"<\/span>,<\/span> "\u5317\u4eac"<\/span>]<\/span>
    \nprint<\/span>(<\/span>","<\/span>.<\/span>join(<\/span>row)<\/span>)<\/span> # "\u5f20\u4e09,25,\u5317\u4eac"<\/span><\/p>\n

    \u6ce8\u610f\u4e8b\u9879（\u9ad8\u9891\u6613\u9519\u70b9）<\/p>\n

  • \u5143\u7d20\u5fc5\u987b\u662f\u5b57\u7b26\u4e32：",".join([1, 2, 3]) \u4f1a\u629b TypeError，\u9700\u8981\u5148 map(str, …) \u8f6c\u6362；<\/li>\n
  • \u5927\u91cf\u5b57\u7b26\u4e32\u62fc\u63a5\u65f6，join() \u6548\u7387\u8fdc\u9ad8\u4e8e +（+ \u662f O(n\u00b2)，join() \u662f O(n)），\u6279\u91cf\u62fc\u63a5\u6c38\u8fdc\u4f18\u5148 join()\u3002<\/li>\n
    \n

    3. str.splitlines() \u2014\u2014 \u6309\u884c\u5207\u5206<\/h4>\n

    \u8bed\u6cd5<\/p>\n

    str<\/span>.<\/span>splitlines(<\/span>[<\/span>keepends]<\/span>)<\/span><\/p>\n

    \u53c2\u6570\u8bf4\u660e<\/p>\n\n\u53c2\u6570\u7c7b\u578b\u8bf4\u660e<\/tr>\n\n\n
    keepends<\/td>\nbool，\u53ef\u9009<\/td>\n\u662f\u5426\u5728\u7ed3\u679c\u4e2d\u4fdd\u7559\u6362\u884c\u7b26，\u9ed8\u8ba4 False \u53bb\u6389<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    \u8fd4\u56de\u503c：\u6309\u6362\u884c\u7b26（\\\\n\u3001\\\\r\\\\n\u3001\\\\r \u7b49）\u5207\u5206\u540e\u7684\u5b57\u7b26\u4e32\u5217\u8868\u3002<\/p>\n

    \u793a\u4f8b\u4ee3\u7801<\/p>\n

    text =<\/span> "\u7b2c\u4e00\u884c\\\\r\\\\n\u7b2c\u4e8c\u884c\\\\n\u7b2c\u4e09\u884c\\\\r\u7b2c\u56db\u884c"<\/span>
    \nprint<\/span>(<\/span>text.<\/span>splitlines(<\/span>)<\/span>)<\/span>
    \n# ['\u7b2c\u4e00\u884c', '\u7b2c\u4e8c\u884c', '\u7b2c\u4e09\u884c', '\u7b2c\u56db\u884c']<\/span><\/p>\n

    # \u4fdd\u7559\u6362\u884c\u7b26<\/span>
    \nprint<\/span>(<\/span>"a\\\\nb"<\/span>.<\/span>splitlines(<\/span>True<\/span>)<\/span>)<\/span> # ['a\\\\n', 'b']<\/span><\/p>\n

    # \u5bf9\u6bd4：split("\\\\n") \u5904\u7406\u4e0d\u4e86 \\\\r\\\\n<\/span>
    \nprint<\/span>(<\/span>"a\\\\r\\\\nb"<\/span>.<\/span>split(<\/span>"\\\\n"<\/span>)<\/span>)<\/span> # ['a\\\\r', 'b'] \u2190 \u6b8b\u7559 \\\\r，\u9700\u518d strip<\/span><\/p>\n

    NLP \u573a\u666f\u5e94\u7528<\/p>\n

    # \u6309\u884c\u8bfb\u53d6\u5e76\u6e05\u6d17\u8bed\u6599（\u6bd4 split("\\\\n") \u66f4\u5065\u58ee）<\/span>
    \nwith<\/span> open<\/span>(<\/span>"corpus.txt"<\/span>,<\/span> encoding=<\/span>"utf-8"<\/span>)<\/span> as<\/span> f:<\/span>
    \n lines =<\/span> [<\/span>line.<\/span>strip(<\/span>)<\/span> for<\/span> line in<\/span> f.<\/span>read(<\/span>)<\/span>.<\/span>splitlines(<\/span>)<\/span>]<\/span><\/p>\n

    \u6ce8\u610f\u4e8b\u9879<\/p>\n

    splitlines() \u80fd\u8bc6\u522b\u5168\u90e8 Unicode \u6362\u884c\u7b26，\u8de8\u5e73\u53f0\u5904\u7406\u6587\u672c（Windows \\\\r\\\\n\u3001Unix \\\\n\u3001Mac \\\\r）\u65f6\u6bd4 split("\\\\n") \u53ef\u9760\u5f97\u591a\u3002<\/p>\n

    \n

    4. str.find() \u4e0e str.index() \u2014\u2014 \u5b9a\u4f4d\u5b50\u4e32<\/h4>\n

    \u8bed\u6cd5<\/p>\n

    str<\/span>.<\/span>find(<\/span>sub[<\/span>,<\/span> start[<\/span>,<\/span> end]<\/span>]<\/span>)<\/span> # \u627e\u4e0d\u5230\u8fd4\u56de -1<\/span>
    \nstr<\/span>.<\/span>index(<\/span>sub[<\/span>,<\/span> start[<\/span>,<\/span> end]<\/span>]<\/span>)<\/span> # \u627e\u4e0d\u5230\u629b ValueError<\/span>
    \nstr<\/span>.<\/span>rfind(<\/span>sub)<\/span> # \u4ece\u53f3\u4fa7\u5f00\u59cb\u627e<\/span>
    \nstr<\/span>.<\/span>rindex(<\/span>sub)<\/span><\/p>\n

    \u8fd4\u56de\u503c：\u5b50\u4e32\u9996\u6b21\u51fa\u73b0\u7684\u7d22\u5f15（int）\u3002<\/p>\n

    \u793a\u4f8b\u4ee3\u7801<\/p>\n

    s =<\/span> "\u81ea\u7136\u8bed\u8a00\u5904\u7406\u662f\u4eba\u5de5\u667a\u80fd\u7684\u4e00\u4e2a\u5206\u652f"<\/span><\/p>\n

    print<\/span>(<\/span>s.<\/span>find(<\/span>"\u4eba\u5de5\u667a\u80fd"<\/span>)<\/span>)<\/span> # 7（0 \u8d77\u59cb\u7d22\u5f15）<\/span>
    \nprint<\/span>(<\/span>s.<\/span>find(<\/span>"\u4e0d\u5b58\u5728"<\/span>)<\/span>)<\/span> # -1<\/span><\/p>\n

    print<\/span>(<\/span>s.<\/span>index(<\/span>"\u4eba\u5de5\u667a\u80fd"<\/span>)<\/span>)<\/span> # 7<\/span>
    \n# print(s.index("\u4e0d\u5b58\u5728")) # \u4f1a\u629b ValueError: substring not found<\/span><\/p>\n

    print<\/span>(<\/span>s.<\/span>rfind(<\/span>"\u7684"<\/span>)<\/span>)<\/span> # \u4ece\u53f3\u5f80\u5de6\u627e<\/span><\/p>\n

    # \u6307\u5b9a\u641c\u7d22\u8303\u56f4<\/span>
    \nprint<\/span>(<\/span>"hello world"<\/span>.<\/span>find(<\/span>"o"<\/span>,<\/span> 5<\/span>)<\/span>)<\/span> # 7，\u4ece\u7d22\u5f15 5 \u5f00\u59cb\u627e<\/span><\/p>\n

    NLP \u573a\u666f\u5e94\u7528<\/p>\n

      \n
    • \u5b9a\u4f4d\u654f\u611f\u8bcd\u4f4d\u7f6e\u505a\u8131\u654f：<\/li>\n<\/ul>\n

      def<\/span> mask_sensitive<\/span>(<\/span>text,<\/span> keyword)<\/span>:<\/span>
      \n idx =<\/span> text.<\/span>find(<\/span>keyword)<\/span>
      \n if<\/span> idx ==<\/span> –<\/span>1<\/span>:<\/span>
      \n return<\/span> text
      \n return<\/span> text[<\/span>:<\/span>idx]<\/span> +<\/span> "*"<\/span> *<\/span> len<\/span>(<\/span>keyword)<\/span> +<\/span> text[<\/span>idx +<\/span> len<\/span>(<\/span>keyword)<\/span>:<\/span>]<\/span><\/p>\n

      print<\/span>(<\/span>mask_sensitive(<\/span>"\u6211\u7684\u624b\u673a\u53f7\u662f13812345678"<\/span>,<\/span> "13812345678"<\/span>)<\/span>)<\/span>
      \n# "\u6211\u7684\u624b\u673a\u53f7\u662f***********"<\/span><\/p>\n

      \u6ce8\u610f\u4e8b\u9879<\/p>\n

      find() \u8fd4\u56de -1 \u8868\u793a\u627e\u4e0d\u5230\u3001\u4e0d\u4f1a\u629b\u5f02\u5e38，\u9002\u5408"\u5224\u65ad\u5728\u4e0d\u5728"；index() \u627e\u4e0d\u5230\u76f4\u63a5\u62a5\u9519，\u9002\u5408"\u786e\u5b9a\u5b58\u5728\u518d\u53d6\u4f4d\u7f6e"\u7684\u573a\u666f\u3002\u4e0d\u786e\u5b9a\u5b58\u5728\u6027\u65f6\u4f18\u5148 find()\u3002<\/p>\n

      \n

      5. str.startswith() \u4e0e str.endswith() \u2014\u2014 \u9996\u5c3e\u5224\u65ad<\/h4>\n

      \u8bed\u6cd5<\/p>\n

      str<\/span>.<\/span>startswith(<\/span>prefix[<\/span>,<\/span> start[<\/span>,<\/span> end]<\/span>]<\/span>)<\/span> # prefix \u53ef\u4ee5\u662f\u5b57\u7b26\u4e32\u6216\u5143\u7ec4<\/span>
      \nstr<\/span>.<\/span>endswith(<\/span>suffix[<\/span>,<\/span> start[<\/span>,<\/span> end]<\/span>]<\/span>)<\/span><\/p>\n

      \u8fd4\u56de\u503c：\u5e03\u5c14\u503c\u3002<\/p>\n

      \u793a\u4f8b\u4ee3\u7801<\/p>\n

      s =<\/span> "\u4eba\u5de5\u667a\u80fd\u5728\u81ea\u7136\u8bed\u8a00\u5904\u7406\u4e2d\u7684\u5e94\u7528"<\/span><\/p>\n

      print<\/span>(<\/span>s.<\/span>startswith(<\/span>"\u4eba\u5de5"<\/span>)<\/span>)<\/span> # True<\/span>
      \nprint<\/span>(<\/span>s.<\/span>startswith(<\/span>"\u81ea\u7136"<\/span>)<\/span>)<\/span> # False<\/span>
      \nprint<\/span>(<\/span>s.<\/span>endswith(<\/span>"\u5e94\u7528"<\/span>)<\/span>)<\/span> # True<\/span><\/p>\n

      # \u5143\u7ec4\u53c2\u6570：\u6ee1\u8db3\u5176\u4e2d\u4e00\u4e2a\u5373\u53ef<\/span>
      \nurl =<\/span> "https:\/\/example.com"<\/span>
      \nprint<\/span>(<\/span>url.<\/span>startswith(<\/span>(<\/span>"http:\/\/"<\/span>,<\/span> "https:\/\/"<\/span>)<\/span>)<\/span>)<\/span> # True<\/span><\/p>\n

      # \u6307\u5b9a\u8303\u56f4\u5224\u65ad<\/span>
      \nprint<\/span>(<\/span>"hello world"<\/span>.<\/span>endswith(<\/span>"world"<\/span>)<\/span>)<\/span> # True<\/span><\/p>\n

      NLP \u573a\u666f\u5e94\u7528<\/p>\n

        \n
      • \u8fc7\u6ee4\u6389\u4ee5\u7279\u6b8a\u5b57\u7b26\u5f00\u5934\/\u7ed3\u5c3e\u7684\u566a\u58f0\u884c：<\/li>\n<\/ul>\n

        lines =<\/span> [<\/span>"# \u6807\u9898"<\/span>,<\/span> "\u6b63\u6587\u5185\u5bb9"<\/span>,<\/span> "\u3010\u5e7f\u544a\u3011xxx"<\/span>,<\/span> "\u2014\u2014\u5206\u5272\u7ebf\u2014\u2014"<\/span>]<\/span>
        \nclean =<\/span> [<\/span>l for<\/span> l in<\/span> lines
        \n if<\/span> not<\/span> l.<\/span>startswith(<\/span>(<\/span>"#"<\/span>,<\/span> "\u3010"<\/span>)<\/span>)<\/span>
        \n and<\/span> not<\/span> l.<\/span>endswith(<\/span>"\u2014\u2014"<\/span>)<\/span>]<\/span>
        \nprint<\/span>(<\/span>clean)<\/span> # ['\u6b63\u6587\u5185\u5bb9']<\/span><\/p>\n

        \n

        \u4e8c\u3001\u8fc7\u6ee4\u4e0e\u7b5b\u9009<\/h3>\n

        6. filter() \u2014\u2014 \u51fd\u6570\u5f0f\u8fc7\u6ee4<\/h4>\n

        \u8bed\u6cd5<\/p>\n

        filter<\/span>(<\/span>function,<\/span> iterable)<\/span><\/p>\n

        \u53c2\u6570\u8bf4\u660e<\/p>\n\n\u53c2\u6570\u7c7b\u578b\u8bf4\u660e<\/tr>\n\n\n\n
        function<\/td>\n\u51fd\u6570 \/ None<\/td>\n\u8fd4\u56de True \u7684\u5143\u7d20\u88ab\u4fdd\u7559；\u4f20 None \u65f6\u8fc7\u6ee4\u6389\u6240\u6709"\u5047\u503c"（\u7a7a\u4e32\u30010\u3001None \u7b49）<\/td>\n<\/tr>\n
        iterable<\/td>\n\u53ef\u8fed\u4ee3\u5bf9\u8c61<\/td>\n\u5f85\u8fc7\u6ee4\u7684\u6570\u636e<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

        \u8fd4\u56de\u503c：Python 3 \u4e2d\u8fd4\u56de\u8fed\u4ee3\u5668（\u60f0\u6027\u6c42\u503c），\u9700\u8981\u7528 list() \u7b49\u8f6c\u6210\u5217\u8868\u3002<\/p>\n

        \u793a\u4f8b\u4ee3\u7801<\/p>\n

        nums =<\/span> [<\/span>1<\/span>,<\/span> 0<\/span>,<\/span> 5<\/span>,<\/span> –<\/span>3<\/span>,<\/span> 8<\/span>]<\/span>
        \nprint<\/span>(<\/span>list<\/span>(<\/span>filter<\/span>(<\/span>lambda<\/span> x:<\/span> x ><\/span> 0<\/span>,<\/span> nums)<\/span>)<\/span>)<\/span> # [1, 5, 8]<\/span><\/p>\n

        # function \u4e3a None：\u8fc7\u6ee4\u5047\u503c<\/span>
        \ndata =<\/span> [<\/span>"hello"<\/span>,<\/span> ""<\/span>,<\/span> "world"<\/span>,<\/span> None<\/span>,<\/span> 0<\/span>]<\/span>
        \nprint<\/span>(<\/span>list<\/span>(<\/span>filter<\/span>(<\/span>None<\/span>,<\/span> data)<\/span>)<\/span>)<\/span> # ['hello', 'world']<\/span><\/p>\n

        NLP \u573a\u666f\u5e94\u7528<\/p>\n

        # \u53bb\u7a7a\u4e32 + \u53bb\u505c\u7528\u8bcd，\u4e00\u6761 filter \u641e\u5b9a<\/span>
        \nstopwords =<\/span> {<\/span>"\u7684"<\/span>,<\/span> "\u4e86"<\/span>,<\/span> "\u548c"<\/span>,<\/span> "\u662f"<\/span>}<\/span>
        \ntokens =<\/span> [<\/span>"\u6211"<\/span>,<\/span> "\u7231"<\/span>,<\/span> "\u7684"<\/span>,<\/span> "Python"<\/span>,<\/span> ""<\/span>,<\/span> "\u662f"<\/span>]<\/span><\/p>\n

        clean =<\/span> list<\/span>(<\/span>filter<\/span>(<\/span>
        \n lambda<\/span> w:<\/span> w and<\/span> w not<\/span> in<\/span> stopwords,<\/span>
        \n tokens
        \n)<\/span>)<\/span>
        \nprint<\/span>(<\/span>clean)<\/span> # ['\u6211', '\u7231', 'Python']<\/span><\/p>\n

        \u6ce8\u610f\u4e8b\u9879<\/p>\n

        filter() \u662f\u60f0\u6027\u7684，\u76f4\u63a5 print(filter(…)) \u770b\u5230\u7684\u662f\u8fed\u4ee3\u5668\u5bf9\u8c61；\u52a1\u5fc5\u5305\u4e00\u5c42 list()\u3002\u60f3\u540c\u65f6\u505a\u8f6c\u6362（\u6bd4\u5982 str()），\u53ef\u4ee5 map() + filter() \u8fde\u7528，\u6216\u76f4\u63a5\u4e0a\u5217\u8868\u63a8\u5bfc\u5f0f\u3002<\/p>\n

        \n

        7. len() \u2014\u2014 \u957f\u5ea6\u8fc7\u6ee4\u4e0e\u7edf\u8ba1<\/h4>\n

        \u8bed\u6cd5<\/p>\n

        len<\/span>(<\/span>obj)<\/span><\/p>\n

        \u8fd4\u56de\u503c：\u5bf9\u8c61\u957f\u5ea6\u3002\u5b57\u7b26\u4e32\u8fd4\u56de\u5b57\u7b26\u6570，\u5217\u8868\/\u5b57\u5178\u8fd4\u56de\u5143\u7d20\u4e2a\u6570\u3002<\/p>\n

        \u793a\u4f8b\u4ee3\u7801<\/p>\n

        print<\/span>(<\/span>len<\/span>(<\/span>"\u81ea\u7136\u8bed\u8a00\u5904\u7406"<\/span>)<\/span>)<\/span> # 6，\u4e2d\u6587\u5b57\u7b26\u6309 1 \u4e2a\u5b57\u7b26\u8ba1<\/span>
        \nprint<\/span>(<\/span>len<\/span>(<\/span>[<\/span>"a"<\/span>,<\/span> "b"<\/span>,<\/span> "c"<\/span>]<\/span>)<\/span>)<\/span> # 3<\/span>
        \nprint<\/span>(<\/span>len<\/span>(<\/span>{<\/span>}<\/span>)<\/span>)<\/span> # 0<\/span><\/p>\n

        NLP \u573a\u666f\u5e94\u7528<\/p>\n

          \n
        • \u8fc7\u6ee4\u8fc7\u77ed\u7684\u566a\u58f0 token（\u5355\u5b57\u3001\u7a7a\u767d\u4e32）：<\/li>\n<\/ul>\n

          tokens =<\/span> [<\/span>"\u6211"<\/span>,<\/span> "\u7231"<\/span>,<\/span> "\u81ea\u7136\u8bed\u8a00\u5904\u7406"<\/span>,<\/span> "Python"<\/span>,<\/span> "\u7684"<\/span>]<\/span>
          \nmeaningful =<\/span> [<\/span>t for<\/span> t in<\/span> tokens if<\/span> len<\/span>(<\/span>t)<\/span> >=<\/span> 2<\/span>]<\/span>
          \nprint<\/span>(<\/span>meaningful)<\/span> # ['\u81ea\u7136\u8bed\u8a00\u5904\u7406', 'Python']（"\u6211""\u7231""\u7684"\u957f\u5ea61\u88ab\u8fc7\u6ee4）<\/span><\/p>\n

            \n
          • \u622a\u65ad\u8d85\u957f\u6587\u672c（\u9650\u5236\u6a21\u578b\u8f93\u5165\u957f\u5ea6）：<\/li>\n<\/ul>\n

            def<\/span> truncate<\/span>(<\/span>text,<\/span> max_len=<\/span>200<\/span>)<\/span>:<\/span>
            \n return<\/span> text[<\/span>:<\/span>max_len]<\/span> if<\/span> len<\/span>(<\/span>text)<\/span> ><\/span> max_len else<\/span> text<\/p>\n

            \u6ce8\u610f\u4e8b\u9879<\/p>\n

            \u5bf9\u4e2d\u6587\u5206\u8bcd\u7ed3\u679c\u7528 len() \u65f6\u6ce8\u610f：\u4e2d\u6587\u6309\u5b57\u7b26\u8ba1\u6570，\u82f1\u6587 token \u6309\u5b57\u6bcd\u8ba1\u6570，\u4e24\u8005\u7684"\u957f\u5ea6"\u8bed\u4e49\u4e0d\u540c，\u8fc7\u6ee4\u9608\u503c\u8981\u5206\u522b\u8bbe\u5b9a\u3002<\/p>\n

            \n

            8. str.isalpha() \/ isdigit() \/ isalnum() \u2014\u2014 \u5b57\u7b26\u7c7b\u578b\u5224\u65ad<\/h4>\n

            \u8bed\u6cd5<\/p>\n

            str<\/span>.<\/span>isalpha(<\/span>)<\/span> # \u662f\u5426\u5168\u662f\u5b57\u6bcd（\u4e2d\u6587\u4e5f\u8fd4\u56de True）<\/span>
            \nstr<\/span>.<\/span>isdigit(<\/span>)<\/span> # \u662f\u5426\u5168\u662f\u6570\u5b57<\/span>
            \nstr<\/span>.<\/span>isalnum(<\/span>)<\/span> # \u662f\u5426\u5168\u662f\u5b57\u6bcd\u6216\u6570\u5b57<\/span>
            \nstr<\/span>.<\/span>isdecimal(<\/span>)<\/span> # \u662f\u5426\u5168\u662f\u5341\u8fdb\u5236\u6570\u5b57（\u6bd4 isdigit \u66f4\u4e25\u683c）<\/span>
            \nstr<\/span>.<\/span>isnumeric(<\/span>)<\/span> # \u662f\u5426\u5168\u662f\u6570\u5b57\u5b57\u7b26（\u6bd4 isdigit \u66f4\u5bbd\u677e）<\/span><\/p>\n

            \u8fd4\u56de\u503c：\u5e03\u5c14\u503c\u3002<\/p>\n

            \u793a\u4f8b\u4ee3\u7801<\/p>\n

            print<\/span>(<\/span>"abc"<\/span>.<\/span>isalpha(<\/span>)<\/span>)<\/span> # True<\/span>
            \nprint<\/span>(<\/span>"\u81ea\u7136\u8bed\u8a00"<\/span>.<\/span>isalpha(<\/span>)<\/span>)<\/span> # True \u2190 \u4e2d\u6587\u4e5f\u7b97\u5b57\u6bcd！<\/span>
            \nprint<\/span>(<\/span>"abc123"<\/span>.<\/span>isalpha(<\/span>)<\/span>)<\/span> # False<\/span><\/p>\n

            print<\/span>(<\/span>"123"<\/span>.<\/span>isdigit(<\/span>)<\/span>)<\/span> # True<\/span>
            \nprint<\/span>(<\/span>"abc"<\/span>.<\/span>isdigit(<\/span>)<\/span>)<\/span> # False<\/span><\/p>\n

            print<\/span>(<\/span>"abc123"<\/span>.<\/span>isalnum(<\/span>)<\/span>)<\/span> # True<\/span>
            \nprint<\/span>(<\/span>"abc 123"<\/span>.<\/span>isalnum(<\/span>)<\/span>)<\/span> # False（\u7a7a\u683c\u4e0d\u662f\u5b57\u6bcd\u6570\u5b57）<\/span><\/p>\n

            \u4e09\u7c7b\u6570\u5b57\u5224\u65ad\u7684\u533a\u522b（\u6613\u6df7\u6dc6，\u91cd\u70b9\u8bb0）<\/p>\n\n\u65b9\u6cd5\u8bc6\u522b\u5185\u5bb9\u793a\u4f8b<\/tr>\n\n\n\n\n
            isdigit()<\/td>\n0-9 \u53ca\u90e8\u5206 Unicode \u6570\u5b57<\/td>\n"\u00b2".isdigit() \u2192 True<\/td>\n<\/tr>\n
            isdecimal()<\/td>\n\u4ec5\u5341\u8fdb\u5236\u6570\u5b57（\u6700\u4e25\u683c）<\/td>\n"\u00b2".isdecimal() \u2192 False<\/td>\n<\/tr>\n
            isnumeric()<\/td>\n\u6570\u5b57\u5b57\u7b26（\u542b\u4e2d\u6587\u6570\u5b57\u3001\u7f57\u9a6c\u6570\u5b57）<\/td>\n"\u4e09".isnumeric() \u2192 True<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

            NLP \u573a\u666f\u5e94\u7528<\/p>\n

            # \u8fc7\u6ee4\u7eaf\u6570\u5b57 token（\u5bf9\u60c5\u611f\u5206\u6790\u7b49\u4efb\u52a1\u901a\u5e38\u662f\u566a\u58f0）<\/span>
            \ntokens =<\/span> [<\/span>"\u623f\u4ef7"<\/span>,<\/span> "30000"<\/span>,<\/span> "\u5143"<\/span>,<\/span> "2024\u5e74"<\/span>]<\/span>
            \nclean =<\/span> [<\/span>t for<\/span> t in<\/span> tokens if<\/span> not<\/span> t.<\/span>isdigit(<\/span>)<\/span>]<\/span>
            \nprint<\/span>(<\/span>clean)<\/span> # ['\u623f\u4ef7', '\u5143', '2024\u5e74']<\/span><\/p>\n

            # \u8bc6\u522b\u4e2d\u6587（\u5229\u7528 isalpha \u5bf9\u4e2d\u6587\u8fd4\u56de True \u7684\u7279\u6027）<\/span>
            \ndef<\/span> is_chinese_text<\/span>(<\/span>s)<\/span>:<\/span>
            \n return<\/span> all<\/span>(<\/span>'\\\\u4e00'<\/span> <=<\/span> ch <=<\/span> '\\\\u9fff'<\/span> for<\/span> ch in<\/span> s)<\/span><\/p>\n

            \u6ce8\u610f\u4e8b\u9879<\/p>\n

          • isalpha() \u5bf9\u4e2d\u6587\u8fd4\u56de True，\u60f3\u53ea\u5224\u65ad\u82f1\u6587\u5b57\u6bcd\u9700\u914d\u5408\u6b63\u5219 [a-zA-Z]；<\/li>\n
          • \u5904\u7406\u6587\u672c\u4e2d\u7684\u5e74\u4efd"2024\u5e74"\u8fd9\u7c7b\u6df7\u5408\u4e32\u65f6，isdigit() \u8fd4\u56de False，\u9700\u8981 re.search(r'\\\\d+', t) \u5224\u65ad\u662f\u5426\u542b\u6570\u5b57\u3002<\/li>\n
            \n

            9. set() \u2014\u2014 \u53bb\u91cd\u4e0e\u5efa\u8bcd\u8868<\/h4>\n

            \u8bed\u6cd5<\/p>\n

            set<\/span>(<\/span>iterable)<\/span><\/p>\n

            \u8fd4\u56de\u503c：\u53bb\u91cd\u540e\u7684\u96c6\u5408（\u65e0\u5e8f）\u3002<\/p>\n

            \u793a\u4f8b\u4ee3\u7801<\/p>\n

            words =<\/span> [<\/span>"\u6211"<\/span>,<\/span> "\u7231"<\/span>,<\/span> "Python"<\/span>,<\/span> "\u6211"<\/span>,<\/span> "\u7231"<\/span>,<\/span> "\u7f16\u7a0b"<\/span>]<\/span>
            \nprint<\/span>(<\/span>set<\/span>(<\/span>words)<\/span>)<\/span> # {'\u7f16\u7a0b', '\u6211', '\u7231', 'Python'}（\u987a\u5e8f\u4e0d\u56fa\u5b9a）<\/span><\/p>\n

            NLP \u573a\u666f\u5e94\u7528<\/p>\n

            # \u5feb\u901f\u7edf\u8ba1\u552f\u4e00\u8bcd\u6570\u91cf（\u8bcd\u8868\u5927\u5c0f）<\/span>
            \nvocab =<\/span> set<\/span>(<\/span>tokens)<\/span>
            \nprint<\/span>(<\/span>f"\u8bcd\u8868\u5927\u5c0f：<\/span>{<\/span>len<\/span>(<\/span>vocab)<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

            # \u8bcd\u8868\u8f6c\u7d22\u5f15\u5b57\u5178（\u9700\u8981\u7a33\u5b9a\u987a\u5e8f\u65f6\u5148\u6392\u5e8f）<\/span>
            \nvocab =<\/span> sorted<\/span>(<\/span>set<\/span>(<\/span>tokens)<\/span>)<\/span>
            \nword2idx =<\/span> {<\/span>w:<\/span> i for<\/span> i,<\/span> w in<\/span> enumerate<\/span>(<\/span>vocab)<\/span>}<\/span><\/p>\n

            \u6ce8\u610f\u4e8b\u9879<\/p>\n

          • set() \u662f\u65e0\u5e8f\u7684，\u6bcf\u6b21\u8fd0\u884c\u987a\u5e8f\u53ef\u80fd\u4e0d\u540c；<\/li>\n
          • \u9700\u8981\u56fa\u5b9a\u987a\u5e8f\u5efa\u8bcd\u8868\u65f6，\u7528 sorted(set(…)) \u6216 dict.fromkeys(tokens)（Python 3.7+ \u5b57\u5178\u4fdd\u5e8f）：# \u6309\u9996\u6b21\u51fa\u73b0\u987a\u5e8f\u53bb\u91cd<\/span>
            \nordered_unique =<\/span> list<\/span>(<\/span>dict<\/span>.<\/span>fromkeys(<\/span>words)<\/span>)<\/span>
            \n# ['\u6211', '\u7231', 'Python', '\u7f16\u7a0b']<\/span>\n <\/li>\n
            \n

            10. map() \u2014\u2014 \u6279\u91cf\u8f6c\u6362<\/h4>\n

            \u8bed\u6cd5<\/p>\n

            map<\/span>(<\/span>function,<\/span> iterable,<\/span> .<\/span>.<\/span>.<\/span>)<\/span><\/p>\n

            \u8fd4\u56de\u503c：\u8fed\u4ee3\u5668（\u60f0\u6027），\u9700 list() \u8f6c\u6362\u3002<\/p>\n

            \u793a\u4f8b\u4ee3\u7801<\/p>\n

            nums =<\/span> [<\/span>"1"<\/span>,<\/span> "2"<\/span>,<\/span> "3"<\/span>]<\/span>
            \nprint<\/span>(<\/span>list<\/span>(<\/span>map<\/span>(<\/span>int<\/span>,<\/span> nums)<\/span>)<\/span>)<\/span> # [1, 2, 3]<\/span><\/p>\n

            # \u591a\u53c2\u6570：zip \u98ce\u683c\u5e76\u884c\u4f20\u5165<\/span>
            \na =<\/span> [<\/span>1<\/span>,<\/span> 2<\/span>,<\/span> 3<\/span>]<\/span>
            \nb =<\/span> [<\/span>10<\/span>,<\/span> 20<\/span>,<\/span> 30<\/span>]<\/span>
            \nprint<\/span>(<\/span>list<\/span>(<\/span>map<\/span>(<\/span>lambda<\/span> x,<\/span> y:<\/span> x +<\/span> y,<\/span> a,<\/span> b)<\/span>)<\/span>)<\/span> # [11, 22, 33]<\/span><\/p>\n

            NLP \u573a\u666f\u5e94\u7528<\/p>\n

            # \u5206\u8bcd\u7ed3\u679c\u7edf\u4e00\u8f6c\u5c0f\u5199 \/ \u53bb\u7a7a\u683c<\/span>
            \ntokens =<\/span> [<\/span>" Hello "<\/span>,<\/span> "WORLD"<\/span>,<\/span> "Python "<\/span>]<\/span>
            \nclean =<\/span> list<\/span>(<\/span>map<\/span>(<\/span>lambda<\/span> t:<\/span> t.<\/span>strip(<\/span>)<\/span>.<\/span>lower(<\/span>)<\/span>,<\/span> tokens)<\/span>)<\/span>
            \nprint<\/span>(<\/span>clean)<\/span> # ['hello', 'world', 'python']<\/span><\/p>\n

            # join \u524d\u6279\u91cf\u8f6c\u5b57\u7b26\u4e32<\/span>
            \nnums =<\/span> [<\/span>1<\/span>,<\/span> 2<\/span>,<\/span> 3<\/span>]<\/span>
            \nprint<\/span>(<\/span>","<\/span>.<\/span>join(<\/span>map<\/span>(<\/span>str<\/span>,<\/span> nums)<\/span>)<\/span>)<\/span> # "1,2,3"<\/span><\/p>\n

            \u6ce8\u610f\u4e8b\u9879<\/p>\n

            map() \u60f0\u6027\u6c42\u503c，\u4e14\u53c2\u6570\u662f\u51fd\u6570\u5728\u524d\u3001\u53ef\u8fed\u4ee3\u5bf9\u8c61\u5728\u540e，\u548c filter() \u7684\u4f20\u53c2\u987a\u5e8f\u4e00\u81f4，\u522b\u5199\u53cd\u3002<\/p>\n

            \n

            \u4e09\u3001\u7efc\u5408\u5b9e\u6218：\u5b8c\u6574\u7684\u8fc7\u6ee4\u6e05\u6d17\u6d41\u6c34\u7ebf<\/h3>\n

            \u628a\u7b2c 2 \u7bc7\u7684\u65b9\u6cd5\u7ec4\u5408\u6210\u4e00\u6bb5\u53ef\u590d\u7528\u7684\u8fc7\u6ee4\u51fd\u6570：<\/p>\n

            import<\/span> re<\/p>\n

            STOPWORDS =<\/span> {<\/span>"\u7684"<\/span>,<\/span> "\u4e86"<\/span>,<\/span> "\u548c"<\/span>,<\/span> "\u662f"<\/span>,<\/span> "\u5728"<\/span>,<\/span> "\u6709"<\/span>,<\/span> "\u6211"<\/span>,<\/span> "\u4f60"<\/span>}<\/span><\/p>\n

            def<\/span> filter_tokens<\/span>(<\/span>tokens,<\/span> min_len=<\/span>1<\/span>,<\/span> stopwords=<\/span>None<\/span>)<\/span>:<\/span>
            \n """\u5206\u8bcd\u7ed3\u679c\u8fc7\u6ee4：\u53bb\u7a7a\u4e32\u3001\u53bb\u505c\u7528\u8bcd\u3001\u53bb\u7eaf\u6570\u5b57\u3001\u53bb\u8fc7\u77ed token"""<\/span>
            \n stopwords =<\/span> stopwords or<\/span> STOPWORDS
            \n return<\/span> [<\/span>
            \n t for<\/span> t in<\/span> tokens
            \n if<\/span> t # \u975e\u7a7a<\/span>
            \n and<\/span> t not<\/span> in<\/span> stopwords # \u975e\u505c\u7528\u8bcd<\/span>
            \n and<\/span> not<\/span> t.<\/span>isdigit(<\/span>)<\/span> # \u975e\u7eaf\u6570\u5b57<\/span>
            \n and<\/span> len<\/span>(<\/span>t)<\/span> >=<\/span> min_len # \u957f\u5ea6\u8fbe\u6807<\/span>
            \n and<\/span> t.<\/span>isalnum(<\/span>)<\/span> # \u53ea\u4fdd\u7559\u5b57\u6bcd\/\u4e2d\u6587\/\u6570\u5b57\u7ec4\u5408（\u53bb\u6389\u6807\u70b9 token）<\/span>
            \n ]<\/span><\/p>\n

            def<\/span> tokenize_and_filter<\/span>(<\/span>text)<\/span>:<\/span>
            \n # \u6e05\u6d17 \u2192 \u5206\u8bcd \u2192 \u8fc7\u6ee4<\/span>
            \n text =<\/span> re.<\/span>sub(<\/span>r'[^\\\\w\\\\s\\\\u4e00-\\\\u9fff]'<\/span>,<\/span> ' '<\/span>,<\/span> text)<\/span>
            \n tokens =<\/span> text.<\/span>split(<\/span>)<\/span>
            \n return<\/span> filter_tokens(<\/span>tokens)<\/span><\/p>\n

            text =<\/span> "\u6211 \u7231 Python！\u7684 \u81ea\u7136\u8bed\u8a00\u5904\u7406 \u662f 2024 \u6700\u70ed\u95e8\u7684 \u65b9\u5411 ！！！"<\/span>
            \nprint<\/span>(<\/span>tokenize_and_filter(<\/span>text)<\/span>)<\/span>
            \n# ['\u7231', 'Python', '\u81ea\u7136\u8bed\u8a00\u5904\u7406', '\u6700\u70ed\u95e8\u7684', '\u65b9\u5411']<\/span><\/p>\n

            \u53ef\u4ee5\u770b\u5230：\u6211\/\u7684\/\u662f（\u505c\u7528\u8bcd）\u30012024（\u7eaf\u6570\u5b57）\u3001！！！（\u6807\u70b9）\u90fd\u88ab\u8fc7\u6ee4\u6389\u4e86，\u5269\u4e0b\u7684\u662f\u6709\u5b9e\u9645\u8bed\u4e49\u7684 token\u3002<\/p>\n

            \n

            \u56db\u3001\u5c0f\u7ed3<\/h3>\n

            \u672c\u7bc7\u8986\u76d6\u4e86\u6d41\u6c34\u7ebf\u7684\u4e2d\u95f4\u4e24\u4e2a\u9636\u6bb5：<\/p>\n

              \n
            • \u5206\u8bcd\u8f85\u52a9：split()（\u6309\u7a7a\u767d\/\u5206\u9694\u7b26\u5207\u5206）\u3001join()（\u91cd\u7ec4）\u3001splitlines()（\u6309\u884c）\u3001find() \/ index()（\u5b9a\u4f4d）\u3001startswith() \/ endswith()（\u9996\u5c3e\u5224\u65ad）；<\/li>\n
            • \u8fc7\u6ee4\u7b5b\u9009：filter()（\u51fd\u6570\u5f0f\u8fc7\u6ee4）\u3001len()（\u957f\u5ea6\u8fc7\u6ee4）\u3001isalpha() \/ isdigit() \u5bb6\u65cf（\u7c7b\u578b\u5224\u65ad）\u3001set()（\u53bb\u91cd\u5efa\u8bcd\u8868）\u3001map()（\u6279\u91cf\u8f6c\u6362）\u3002<\/li>\n<\/ul>\n

              \u4e0b\u4e00\u7bc7（\u7cfb\u5217\u5b8c\u7ed3\u7bc7）\u5c06\u8bb2\u89e3 \u7edf\u8ba1\u3001\u5efa\u8bcd\u8868\u4e0e\u5b8c\u6574\u5b9e\u6218：collections.Counter\u3001sorted()\u3001enumerate()\u3001zip()\u3001ord() \/ chr()\u3001isinstance()，\u5e76\u7ed9\u51fa\u4e00\u6bb5\u4ece\u539f\u59cb\u6587\u672c\u5230\u8bcd\u8868\u7d22\u5f15\u7684\u5168\u6d41\u7a0b\u53ef\u8fd0\u884c\u4ee3\u7801\u3002\u656c\u8bf7\u5173\u6ce8\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"

              NLP \u9884\u5904\u7406\u5fc5\u5907\u7684 Python \u5185\u7f6e\u51fd\u6570（\u4e8c）：\u5206\u8bcd\u8f85\u52a9\u4e0e\u8fc7\u6ee4\u7b5b\u9009 \u5199\u5728\u524d\u9762：\u4e0a\u4e00\u7bc7\u6211\u4eec\u628a\u300c\u6587\u672c\u6e05\u6d17\u300d\u548c\u300c\u6587\u672c\u89c4\u8303\u5316\u300d\u8bb2\u900f\u4e86（strip()\u3001replace()\u3001re.sub()\u3001lower()\u3001unicodedata.normalize() \u7b49）\u3002\u672c\u7bc7\u8fdb\u5165\u6d41\u6c34\u7ebf\u7684\u4e2d\u95f4\u73af\u8282\u2014\u2014\u5206\u8bcd\u8f85\u52a9 \u4e0e \u8fc7\u6ee4\u7b5b\u9009\u3002 \u6587\u672c\u6d17\u767d\u4e4b\u540e，\u5c31\u8981\u628a\u5b83\u5207\u6210\u4e00\u4e2a\u4e2a token（\u8bcd&#xff0<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[7549,11653,81,224],"topic":[],"class_list":["post-104173","post","type-post","status-publish","format-standard","hentry","category-server","tag-easyui","tag-11653","tag-python","tag-224"],"yoast_head":"\nNLP\u9884\u5904\u7406Python\u5185\u7f6e\u51fd\u6570_02_\u5206\u8bcd\u8f85\u52a9\u4e0e\u8fc7\u6ee4\u7b5b\u9009 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.wsisp.com\/helps\/104173.html\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"NLP\u9884\u5904\u7406Python\u5185\u7f6e\u51fd\u6570_02_\u5206\u8bcd\u8f85\u52a9\u4e0e\u8fc7\u6ee4\u7b5b\u9009 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"og:description\" content=\"NLP \u9884\u5904\u7406\u5fc5\u5907\u7684 Python \u5185\u7f6e\u51fd\u6570（\u4e8c）：\u5206\u8bcd\u8f85\u52a9\u4e0e\u8fc7\u6ee4\u7b5b\u9009 \u5199\u5728\u524d\u9762：\u4e0a\u4e00\u7bc7\u6211\u4eec\u628a\u300c\u6587\u672c\u6e05\u6d17\u300d\u548c\u300c\u6587\u672c\u89c4\u8303\u5316\u300d\u8bb2\u900f\u4e86（strip()\u3001replace()\u3001re.sub()\u3001lower()\u3001unicodedata.normalize() \u7b49）\u3002\u672c\u7bc7\u8fdb\u5165\u6d41\u6c34\u7ebf\u7684\u4e2d\u95f4\u73af\u8282\u2014\u2014\u5206\u8bcd\u8f85\u52a9 \u4e0e \u8fc7\u6ee4\u7b5b\u9009\u3002 \u6587\u672c\u6d17\u767d\u4e4b\u540e，\u5c31\u8981\u628a\u5b83\u5207\u6210\u4e00\u4e2a\u4e2a token（\u8bcd&#xff0\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.wsisp.com\/helps\/104173.html\" \/>\n<meta property=\"og:site_name\" content=\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"article:published_time\" content=\"2026-09-11T15:25:09+00:00\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/104173.html\",\"url\":\"https:\/\/www.wsisp.com\/helps\/104173.html\",\"name\":\"NLP\u9884\u5904\u7406Python\u5185\u7f6e\u51fd\u6570_02_\u5206\u8bcd\u8f85\u52a9\u4e0e\u8fc7\u6ee4\u7b5b\u9009 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"isPartOf\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\"},\"datePublished\":\"2026-09-11T15:25:09+00:00\",\"dateModified\":\"2026-09-11T15:25:09+00:00\",\"author\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/104173.html#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.wsisp.com\/helps\/104173.html\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/104173.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.wsisp.com\/helps\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"NLP\u9884\u5904\u7406Python\u5185\u7f6e\u51fd\u6570_02_\u5206\u8bcd\u8f85\u52a9\u4e0e\u8fc7\u6ee4\u7b5b\u9009\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\",\"url\":\"https:\/\/www.wsisp.com\/helps\/\",\"name\":\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"description\":\"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"contentUrl\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"caption\":\"admin\"},\"sameAs\":[\"http:\/\/wp.wsisp.com\"],\"url\":\"https:\/\/www.wsisp.com\/helps\/author\/admin\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"NLP\u9884\u5904\u7406Python\u5185\u7f6e\u51fd\u6570_02_\u5206\u8bcd\u8f85\u52a9\u4e0e\u8fc7\u6ee4\u7b5b\u9009 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.wsisp.com\/helps\/104173.html","og_locale":"zh_CN","og_type":"article","og_title":"NLP\u9884\u5904\u7406Python\u5185\u7f6e\u51fd\u6570_02_\u5206\u8bcd\u8f85\u52a9\u4e0e\u8fc7\u6ee4\u7b5b\u9009 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","og_description":"NLP \u9884\u5904\u7406\u5fc5\u5907\u7684 Python \u5185\u7f6e\u51fd\u6570（\u4e8c）：\u5206\u8bcd\u8f85\u52a9\u4e0e\u8fc7\u6ee4\u7b5b\u9009 \u5199\u5728\u524d\u9762：\u4e0a\u4e00\u7bc7\u6211\u4eec\u628a\u300c\u6587\u672c\u6e05\u6d17\u300d\u548c\u300c\u6587\u672c\u89c4\u8303\u5316\u300d\u8bb2\u900f\u4e86（strip()\u3001replace()\u3001re.sub()\u3001lower()\u3001unicodedata.normalize() \u7b49）\u3002\u672c\u7bc7\u8fdb\u5165\u6d41\u6c34\u7ebf\u7684\u4e2d\u95f4\u73af\u8282\u2014\u2014\u5206\u8bcd\u8f85\u52a9 \u4e0e \u8fc7\u6ee4\u7b5b\u9009\u3002 \u6587\u672c\u6d17\u767d\u4e4b\u540e，\u5c31\u8981\u628a\u5b83\u5207\u6210\u4e00\u4e2a\u4e2a token（\u8bcd&#xff0","og_url":"https:\/\/www.wsisp.com\/helps\/104173.html","og_site_name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","article_published_time":"2026-09-11T15:25:09+00:00","author":"admin","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"admin","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"5 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.wsisp.com\/helps\/104173.html","url":"https:\/\/www.wsisp.com\/helps\/104173.html","name":"NLP\u9884\u5904\u7406Python\u5185\u7f6e\u51fd\u6570_02_\u5206\u8bcd\u8f85\u52a9\u4e0e\u8fc7\u6ee4\u7b5b\u9009 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","isPartOf":{"@id":"https:\/\/www.wsisp.com\/helps\/#website"},"datePublished":"2026-09-11T15:25:09+00:00","dateModified":"2026-09-11T15:25:09+00:00","author":{"@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41"},"breadcrumb":{"@id":"https:\/\/www.wsisp.com\/helps\/104173.html#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.wsisp.com\/helps\/104173.html"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.wsisp.com\/helps\/104173.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.wsisp.com\/helps"},{"@type":"ListItem","position":2,"name":"NLP\u9884\u5904\u7406Python\u5185\u7f6e\u51fd\u6570_02_\u5206\u8bcd\u8f85\u52a9\u4e0e\u8fc7\u6ee4\u7b5b\u9009"}]},{"@type":"WebSite","@id":"https:\/\/www.wsisp.com\/helps\/#website","url":"https:\/\/www.wsisp.com\/helps\/","name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","description":"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41","name":"admin","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/","url":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","contentUrl":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","caption":"admin"},"sameAs":["http:\/\/wp.wsisp.com"],"url":"https:\/\/www.wsisp.com\/helps\/author\/admin"}]}},"_links":{"self":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/104173","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/comments?post=104173"}],"version-history":[{"count":0,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/104173\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media?parent=104173"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/categories?post=104173"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/tags?post=104173"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/topic?post=104173"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}