{"id":97387,"date":"2026-08-29T11:40:45","date_gmt":"2026-08-29T03:40:45","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/97387.html"},"modified":"2026-08-29T11:40:45","modified_gmt":"2026-08-29T03:40:45","slug":"python-%e7%88%ac%e8%99%ab%e6%8a%80%e6%9c%af%e5%85%a8%e6%a0%88%e6%8c%87%e5%8d%97","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/97387.html","title":{"rendered":"Python \u722c\u866b\u6280\u672f\u5168\u6808\u6307\u5357"},"content":{"rendered":"

Python \u722c\u866b\u6280\u672f\u5168\u6808\u6307\u5357<\/h2>\n

\u9762\u5411\u8bfb\u8005：\u6709\u4e00\u5b9a Python \u57fa\u7840\u7684\u6570\u636e\u5f00\u53d1\u5de5\u7a0b\u5e08 \u7248\u672c\u8bf4\u660e：\u878d\u5165 2025-2026 \u5e74\u6700\u65b0\u6280\u672f\u52a8\u6001，\u8986\u76d6 curl_cffi\u3001Playwright \u65b0\u7248\u3001AI \u8f85\u52a9\u9006\u5411\u7b49\u524d\u6cbf\u5185\u5bb9<\/p>\n

\n

\u76ee\u5f55<\/h3>\n
  • \u722c\u866b\u5168\u666f\u6982\u8ff0<\/li>\n
  • HTTP\u534f\u8bae\u4e0e\u7f51\u7edc\u57fa\u7840<\/li>\n
  • Python\u722c\u866b\u57fa\u7840\u5e93<\/li>\n
  • HTML\u89e3\u6790\u4e0e\u6570\u636e\u63d0\u53d6<\/li>\n
  • \u4e3b\u6d41\u722c\u866b\u6846\u67b6\u8be6\u89e3<\/li>\n
  • \u52a8\u6001\u9875\u9762\u722c\u53d6<\/li>\n
  • \u53cd\u722c\u866b\u6280\u672f\u5168\u89e3<\/li>\n
  • \u53cd\u53cd\u722c\u866b\u5b9e\u6218<\/li>\n
  • \u9006\u5411\u5de5\u7a0b\u57fa\u7840<\/li>\n
  • JS\u9006\u5411\u6df1\u5ea6\u5b9e\u6218<\/li>\n
  • APP\u722c\u866b\u4e0e\u6293\u5305<\/li>\n
  • \u5206\u5e03\u5f0f\u722c\u866b\u67b6\u6784<\/li>\n
  • \u6570\u636e\u5b58\u50a8\u4e0e\u6e05\u6d17<\/li>\n
  • \u6cd5\u5f8b\u5408\u89c4\u4e0erobots\u534f\u8bae<\/li>\n
  • \u9a8c\u8bc1\u7801\u4e13\u9879\u7a81\u7834<\/li>\n
  • \u9ad8\u5e76\u53d1\u4e0e\u6027\u80fd\u4f18\u5316<\/li>\n
  • \u5b9e\u6218\u9879\u76ee<\/li>\n
  • 2025-2026\u722c\u866b\u65b0\u6280\u672f\u8d8b\u52bf<\/li>\n
    \n

    1. \u722c\u866b\u5168\u666f\u6982\u8ff0<\/h3>\n

    1.1 \u4ec0\u4e48\u662f\u722c\u866b<\/h4>\n

    \u7f51\u7edc\u722c\u866b（Web Crawler \/ Spider）\u662f\u4e00\u79cd\u6309\u7167\u4e00\u5b9a\u89c4\u5219\u81ea\u52a8\u6293\u53d6\u4e07\u7ef4\u7f51\u4fe1\u606f\u7684\u7a0b\u5e8f\u6216\u811a\u672c\u3002\u5176\u6838\u5fc3\u5de5\u4f5c\u6d41\u7a0b\u53ef\u4ee5\u6982\u62ec\u4e3a：<\/p>\n

    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
    \n\u2502 \u722c\u866b\u5de5\u4f5c\u6d41\u7a0b\u5168\u666f\u56fe \u2502
    \n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
    \n\u2502 \u2502
    \n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\u2502
    \n\u2502 \u2502 \u79cd\u5b50URL \u2502\u2500\u2500\u2500>\u2502 \u53d1\u9001\u8bf7\u6c42 \u2502\u2500\u2500\u2500>\u2502 \u63a5\u6536\u54cd\u5e94 \u2502\u2500\u2500\u2500>\u2502 \u89e3\u6790\u63d0 \u2502\u2502
    \n\u2502 \u2502 \u961f\u5217 \u2502 \u2502 (HTTP) \u2502 \u2502 (HTML) \u2502 \u2502 \u53d6\u6570\u636e \u2502\u2502
    \n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2518\u2502
    \n\u2502 ^ \u2502 \u2502
    \n\u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502
    \n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2502 \u8c03\u5ea6\u63a7\u5236 \u2502<\u2500\u2500\u2500\u2502 \u6570\u636e\u5b58\u50a8 \u2502<\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502
    \n\u2502 \u2502 \u5f15\u64ce \u2502 \u2502 Pipeline \u2502 \u2502
    \n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502
    \n\u2502 \u2502
    \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

    1.2 \u722c\u866b\u53d1\u5c55\u5386\u53f2<\/h4>\n\n\u9636\u6bb5\u65f6\u95f4\u7279\u5f81\u4ee3\u8868\u6280\u672f<\/tr>\n\n\n\n\n\n\n
    \u840c\u82bd\u671f<\/td>\n1993-2000<\/td>\n\u641c\u7d22\u5f15\u64ce\u722c\u866b\u8bde\u751f<\/td>\nWorld Wide Web Wanderer\u3001Googlebot<\/td>\n<\/tr>\n
    \u6210\u957f\u671f<\/td>\n2000-2010<\/td>\n\u5f00\u6e90\u6846\u67b6\u6d8c\u73b0<\/td>\nScrapy(2008)\u3001Nutch\u3001Heritrix<\/td>\n<\/tr>\n
    \u5bf9\u6297\u671f<\/td>\n2010-2020<\/td>\n\u53cd\u722c\u4e0e\u53cd\u53cd\u722c\u535a\u5f08<\/td>\nSelenium\u3001PhantomJS\u3001\u4ee3\u7406\u6c60<\/td>\n<\/tr>\n
    \u667a\u80fd\u5316<\/td>\n2020-2025<\/td>\nAI\u8f85\u52a9\u9006\u5411\u3001\u65e0\u5934\u6d4f\u89c8\u5668\u666e\u53ca<\/td>\nPlaywright\u3001curl_cffi\u3001GPT\u9006\u5411<\/td>\n<\/tr>\n
    \u65b0\u7eaa\u5143<\/td>\n2025-2026<\/td>\nTLS\u6307\u7eb9\u4f2a\u88c5\u3001LLM\u5206\u6790JS<\/td>\ncurl_cffi 0.14+\u3001AI Agent\u722c\u866b<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    1.3 \u722c\u866b\u5206\u7c7b<\/h4>\n\n\u7c7b\u578b\u8bf4\u660e\u9002\u7528\u573a\u666f\u793a\u4f8b<\/tr>\n\n\n\n\n\n\n
    \u901a\u7528\u722c\u866b<\/td>\n\u5927\u89c4\u6a21\u6293\u53d6\u5168\u7f51\u6570\u636e<\/td>\n\u641c\u7d22\u5f15\u64ce\u7d22\u5f15<\/td>\nGooglebot\u3001BaiduSpider<\/td>\n<\/tr>\n
    \u805a\u7126\u722c\u866b<\/td>\n\u9488\u5bf9\u7279\u5b9a\u4e3b\u9898\/\u7f51\u7ad9<\/td>\n\u7535\u5546\u6bd4\u4ef7\u3001\u8206\u60c5\u76d1\u63a7<\/td>\n\u5546\u54c1\u4ef7\u683c\u91c7\u96c6\u5668<\/td>\n<\/tr>\n
    \u589e\u91cf\u722c\u866b<\/td>\n\u53ea\u6293\u53d6\u65b0\u589e\/\u53d8\u66f4\u5185\u5bb9<\/td>\n\u65b0\u95fb\u76d1\u63a7\u3001\u66f4\u65b0\u68c0\u6d4b<\/td>\n\u65b0\u95fb\u805a\u5408\u7cfb\u7edf<\/td>\n<\/tr>\n
    \u5206\u5e03\u5f0f\u722c\u866b<\/td>\n\u591a\u673a\u534f\u540c\u5927\u89c4\u6a21\u91c7\u96c6<\/td>\n\u5168\u7f51\u6570\u636e\u3001PB\u7ea7<\/td>\nScrapy-Redis\u96c6\u7fa4<\/td>\n<\/tr>\n
    \u6df1\u5ea6\u5b66\u4e60\u722c\u866b<\/td>\n\u7ed3\u5408AI\u8fdb\u884c\u667a\u80fd\u63d0\u53d6<\/td>\n\u7ed3\u6784\u5316\u4fe1\u606f\u62bd\u53d6<\/td>\n\u57fa\u4e8eBERT\u7684\u7f51\u9875\u89e3\u6790<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    1.4 \u722c\u866b vs RPA vs API\u8c03\u7528<\/h4>\n\n\u5bf9\u6bd4\u7ef4\u5ea6\u7f51\u7edc\u722c\u866bRPA（\u673a\u5668\u4eba\u6d41\u7a0b\u81ea\u52a8\u5316）API\u8c03\u7528<\/tr>\n\n\n\n\n\n\n\n\n
    \u5b9e\u73b0\u65b9\u5f0f<\/td>\n\u6a21\u62dfHTTP\u8bf7\u6c42<\/td>\n\u6a21\u62df\u4eba\u7c7bUI\u64cd\u4f5c<\/td>\n\u76f4\u63a5\u8c03\u7528\u63a5\u53e3<\/td>\n<\/tr>\n
    \u5f00\u53d1\u96be\u5ea6<\/td>\n\u4e2d\u7b49<\/td>\n\u4f4e（\u53ef\u89c6\u5316\u914d\u7f6e）<\/td>\n\u4f4e（\u6709\u6587\u6863\u65f6）<\/td>\n<\/tr>\n
    \u6267\u884c\u6548\u7387<\/td>\n\u9ad8<\/td>\n\u4f4e（\u9700\u6e32\u67d3UI）<\/td>\n\u6700\u9ad8<\/td>\n<\/tr>\n
    \u7a33\u5b9a\u6027<\/td>\n\u4e2d（\u53d7\u53cd\u722c\u5f71\u54cd）<\/td>\n\u4f4e（UI\u53d8\u5316\u654f\u611f）<\/td>\n\u9ad8<\/td>\n<\/tr>\n
    \u6570\u636e\u5b8c\u6574\u6027<\/td>\n\u53ef\u63a7<\/td>\n\u4e0e\u4eba\u7c7b\u64cd\u4f5c\u4e00\u81f4<\/td>\n\u53d6\u51b3\u4e8eAPI\u8bbe\u8ba1<\/td>\n<\/tr>\n
    \u6cd5\u5f8b\u98ce\u9669<\/td>\n\u4e2d-\u9ad8<\/td>\n\u4e2d<\/td>\n\u4f4e<\/td>\n<\/tr>\n
    \u5178\u578b\u5de5\u5177<\/td>\nScrapy\/Playwright<\/td>\nUiPath\/\u5f71\u5200<\/td>\nrequests\/httpx<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    \u9762\u8bd5\u52a0\u5206\u63d0\u793a：\u5728\u6280\u672f\u9009\u578b\u65f6，\u4f18\u5148\u8bc4\u4f30\u76ee\u6807\u7cfb\u7edf\u662f\u5426\u63d0\u4f9b\u5b98\u65b9API\u3002API\u8c03\u7528\u662f\u6700\u7a33\u5b9a\u3001\u5408\u89c4\u3001\u9ad8\u6548\u7684\u65b9\u5f0f\u3002\u53ea\u6709\u5f53API\u65e0\u6cd5\u6ee1\u8db3\u9700\u6c42（\u5982\u6570\u636e\u4e0d\u5b8c\u6574\u3001\u9891\u7387\u53d7\u9650）\u65f6，\u624d\u8003\u8651\u722c\u866b\u65b9\u6848\u3002<\/p>\n

    1.5 \u5e94\u7528\u573a\u666f\u5168\u666f<\/h4>\n\n\u9886\u57df\u5178\u578b\u5e94\u7528\u6570\u636e\u91cf\u7ea7<\/tr>\n\n\n\n\n\n\n\n
    \u7535\u5546<\/td>\n\u5546\u54c1\u6bd4\u4ef7\u3001\u9500\u91cf\u76d1\u63a7\u3001\u8bc4\u4ef7\u5206\u6790<\/td>\n\u767e\u4e07\u7ea7SKU<\/td>\n<\/tr>\n
    \u91d1\u878d<\/td>\n\u80a1\u7968\u884c\u60c5\u3001\u8d22\u62a5\u6293\u53d6\u3001\u8206\u60c5\u9884\u8b66<\/td>\n\u5b9e\u65f6\/\u65e5\u66f4<\/td>\n<\/tr>\n
    \u5b66\u672f<\/td>\n\u8bba\u6587\u68c0\u7d22\u3001\u671f\u520a\u4e0b\u8f7d\u3001\u4e13\u5229\u5206\u6790<\/td>\n\u5341\u4e07\u7ea7\u6587\u6863<\/td>\n<\/tr>\n
    \u793e\u4ea4\u5a92\u4f53<\/td>\n\u8bdd\u9898\u8ffd\u8e2a\u3001KOL\u5206\u6790\u3001\u8d8b\u52bf\u9884\u6d4b<\/td>\n\u4ebf\u7ea7\u6d88\u606f<\/td>\n<\/tr>\n
    \u62db\u8058<\/td>\n\u804c\u4f4d\u805a\u5408\u3001\u85aa\u8d44\u5206\u6790\u3001\u6280\u80fd\u9700\u6c42<\/td>\n\u767e\u4e07\u7ea7\u5c97\u4f4d<\/td>\n<\/tr>\n
    \u623f\u5730\u4ea7<\/td>\n\u623f\u4ef7\u76d1\u63a7\u3001\u697c\u76d8\u5bf9\u6bd4\u3001\u79df\u91d1\u6307\u6570<\/td>\n\u5341\u4e07\u7ea7\u623f\u6e90<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    \n

    2. HTTP\u534f\u8bae\u4e0e\u7f51\u7edc\u57fa\u7840<\/h3>\n

    2.1 HTTP\/HTTPS \u539f\u7406<\/h4>\n

    HTTP（HyperText Transfer Protocol）\u662f Web \u6570\u636e\u901a\u4fe1\u7684\u57fa\u7840\u534f\u8bae\u3002HTTPS \u5728 HTTP \u57fa\u7840\u4e0a\u589e\u52a0\u4e86 TLS\/SSL \u52a0\u5bc6\u5c42\u3002<\/p>\n

    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
    \n\u2502 HTTP vs HTTPS \u534f\u8bae\u6808\u5bf9\u6bd4 \u2502
    \n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
    \n\u2502 \u2502
    \n\u2502 HTTP \u534f\u8bae\u6808: HTTPS \u534f\u8bae\u6808: \u2502
    \n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502
    \n\u2502 \u2502 HTTP\/1.1 \u2502 \u2502 HTTP\/2 \u2502 \u2502
    \n\u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502
    \n\u2502 \u2502 \u2502 \u2502 TLS 1.3 \u2502 \u2502
    \n\u2502 \u2502 \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502
    \n\u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 \u2502 \u2502
    \n\u2502 \u2502 TCP \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502
    \n\u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502 TCP \u2502 \u2502
    \n\u2502 \u2502 IP \u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502
    \n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 IP \u2502 \u2502
    \n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502
    \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

    HTTPS \u63e1\u624b\u8fc7\u7a0b（TLS 1.3 \u7b80\u5316\u7248）：<\/p>\n

    \u5ba2\u6237\u7aef \u670d\u52a1\u7aef
    \n \u2502 \u2502
    \n \u2502\u2500\u2500\u2500\u2500 ClientHello \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500>\u2502 (\u652f\u6301\u7684\u5bc6\u7801\u5957\u4ef6\u3001\u6269\u5c55\u3001SNI)
    \n \u2502 \u2502
    \n \u2502<\u2500\u2500\u2500\u2500 ServerHello \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2502 (\u9009\u5b9a\u7684\u5bc6\u7801\u5957\u4ef6\u3001\u8bc1\u4e66)
    \n \u2502<\u2500\u2500\u2500\u2500 Certificate \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2502 (\u670d\u52a1\u5668\u8bc1\u4e66\u94fe)
    \n \u2502<\u2500\u2500\u2500\u2500 EncryptedExtensions \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2502
    \n \u2502<\u2500\u2500\u2500\u2500 Finished \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2502
    \n \u2502 \u2502
    \n \u2502\u2500\u2500\u2500\u2500 Finished \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500>\u2502 (\u63e1\u624b\u5b8c\u6210，\u5f00\u59cb\u52a0\u5bc6\u901a\u4fe1)
    \n \u2502 \u2502
    \n \u2502<\u2550\u2550\u2550\u2550 \u52a0\u5bc6\u7684\u5e94\u7528\u6570\u636e \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550>\u2502<\/p>\n

    2.2 \u8bf7\u6c42\u54cd\u5e94\u6a21\u578b<\/h4>\n

    import<\/span> requests<\/p>\n

    # \u5b8c\u6574\u7684\u8bf7\u6c42\u793a\u4f8b<\/span>
    \nresponse =<\/span> requests.<\/span>get(<\/span>
    \n url=<\/span>"https:\/\/httpbin.org\/get"<\/span>,<\/span>
    \n params=<\/span>{<\/span>"page"<\/span>:<\/span> 1<\/span>,<\/span> "size"<\/span>:<\/span> 20<\/span>}<\/span>,<\/span> # URL\u67e5\u8be2\u53c2\u6570<\/span>
    \n headers=<\/span>{<\/span>
    \n "User-Agent"<\/span>:<\/span> "Mozilla\/5.0 (Windows NT 10.0; Win64; x64) "<\/span>
    \n "AppleWebKit\/537.36 (KHTML, like Gecko) "<\/span>
    \n "Chrome\/131.0.0.0 Safari\/537.36"<\/span>,<\/span>
    \n "Accept"<\/span>:<\/span> "application\/json"<\/span>,<\/span>
    \n "Accept-Language"<\/span>:<\/span> "zh-CN,zh;q=0.9,en;q=0.8"<\/span>,<\/span>
    \n "Accept-Encoding"<\/span>:<\/span> "gzip, deflate, br"<\/span>,<\/span>
    \n "Connection"<\/span>:<\/span> "keep-alive"<\/span>,<\/span>
    \n }<\/span>,<\/span>
    \n cookies=<\/span>{<\/span>"session_id"<\/span>:<\/span> "abc123def456"<\/span>}<\/span>,<\/span>
    \n timeout=<\/span>(<\/span>5<\/span>,<\/span> 30<\/span>)<\/span>,<\/span> # (\u8fde\u63a5\u8d85\u65f6, \u8bfb\u53d6\u8d85\u65f6)<\/span>
    \n allow_redirects=<\/span>True<\/span>,<\/span>
    \n verify=<\/span>True<\/span>,<\/span> # SSL\u8bc1\u4e66\u9a8c\u8bc1<\/span>
    \n)<\/span><\/p>\n

    print<\/span>(<\/span>f"\u72b6\u6001\u7801: <\/span>{<\/span>response.<\/span>status_code}<\/span><\/span>"<\/span><\/span>)<\/span>
    \nprint<\/span>(<\/span>f"\u54cd\u5e94\u5934: <\/span>{<\/span>response.<\/span>headers[<\/span>'Content-Type'<\/span>]<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span>
    \nprint<\/span>(<\/span>f"\u7f16\u7801: <\/span>{<\/span>response.<\/span>encoding}<\/span><\/span>"<\/span><\/span>)<\/span>
    \nprint<\/span>(<\/span>f"JSON\u6570\u636e: <\/span>{<\/span>response.<\/span>json(<\/span>)<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span>
    \nprint<\/span>(<\/span>f"\u8bf7\u6c42\u8017\u65f6: <\/span>{<\/span>response.<\/span>elapsed.<\/span>total_seconds(<\/span>)<\/span>:<\/span>.3f<\/span>}<\/span><\/span>s"<\/span><\/span>)<\/span><\/p>\n

    2.3 \u5e38\u89c1\u72b6\u6001\u7801<\/h4>\n\n\u72b6\u6001\u7801\u542b\u4e49\u722c\u866b\u5e94\u5bf9\u7b56\u7565<\/tr>\n\n\n\n\n\n\n\n\n\n\n\n\n
    200<\/td>\n\u6210\u529f<\/td>\n\u6b63\u5e38\u89e3\u6790<\/td>\n<\/tr>\n
    301<\/td>\n\u6c38\u4e45\u91cd\u5b9a\u5411<\/td>\n\u66f4\u65b0URL\u5e93<\/td>\n<\/tr>\n
    302<\/td>\n\u4e34\u65f6\u91cd\u5b9a\u5411<\/td>\n\u8ddf\u8e2aLocation\u5934<\/td>\n<\/tr>\n
    304<\/td>\n\u672a\u4fee\u6539（\u7f13\u5b58）<\/td>\n\u5229\u7528\u7f13\u5b58\u7b56\u7565<\/td>\n<\/tr>\n
    400<\/td>\n\u8bf7\u6c42\u9519\u8bef<\/td>\n\u68c0\u67e5\u8bf7\u6c42\u53c2\u6570<\/td>\n<\/tr>\n
    401<\/td>\n\u672a\u6388\u6743<\/td>\n\u9700\u8981\u8ba4\u8bc1\u4fe1\u606f<\/td>\n<\/tr>\n
    403<\/td>\n\u7981\u6b62\u8bbf\u95ee<\/td>\n\u53ef\u80fd\u89e6\u53d1\u53cd\u722c，\u68c0\u67e5UA\/IP<\/td>\n<\/tr>\n
    404<\/td>\n\u8d44\u6e90\u4e0d\u5b58\u5728<\/td>\n\u8df3\u8fc7\u8be5URL<\/td>\n<\/tr>\n
    429<\/td>\n\u8bf7\u6c42\u8fc7\u591a<\/td>\n\u5fc5\u987b\u964d\u901f，\u589e\u52a0\u5ef6\u8fdf<\/td>\n<\/tr>\n
    500<\/td>\n\u670d\u52a1\u5668\u9519\u8bef<\/td>\n\u91cd\u8bd5\u6216\u8df3\u8fc7<\/td>\n<\/tr>\n
    503<\/td>\n\u670d\u52a1\u4e0d\u53ef\u7528<\/td>\n\u53ef\u80fd\u662f\u53cd\u722c（\u5982Cloudflare），\u7b49\u5f85\u91cd\u8bd5<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    \u751f\u4ea7\u8e29\u5751\u63d0\u9192：429 \u72b6\u6001\u7801\u662f\u6700\u660e\u663e\u7684"\u4f60\u88ab\u5c01\u4e86"\u4fe1\u53f7\u3002\u751f\u4ea7\u73af\u5883\u5fc5\u987b\u5b9e\u73b0\u6307\u6570\u9000\u907f\u91cd\u8bd5\u7b56\u7565，\u9047\u5230 429 \u7acb\u5373\u505c\u6b62\u8bf7\u6c42\u81f3\u5c11 60 \u79d2\u3002<\/p>\n

    2.4 Cookie\/Session \u673a\u5236<\/h4>\n

    import<\/span> requests<\/p>\n

    # \u65b9\u5f0f1：\u4f7f\u7528Session\u81ea\u52a8\u7ba1\u7406Cookie<\/span>
    \nsession =<\/span> requests.<\/span>Session(<\/span>)<\/span><\/p>\n

    # \u767b\u5f55<\/span>
    \nlogin_resp =<\/span> session.<\/span>post(<\/span>"https:\/\/httpbin.org\/post"<\/span>,<\/span> data=<\/span>{<\/span>
    \n "username"<\/span>:<\/span> "test_user"<\/span>,<\/span>
    \n "password"<\/span>:<\/span> "test_pass_123"<\/span>
    \n}<\/span>)<\/span><\/p>\n

    # Session\u4f1a\u81ea\u52a8\u643a\u5e26\u767b\u5f55\u540e\u7684Cookie<\/span>
    \nprofile_resp =<\/span> session.<\/span>get(<\/span>"https:\/\/httpbin.org\/cookies"<\/span>)<\/span>
    \nprint<\/span>(<\/span>f"Cookies: <\/span>{<\/span>session.<\/span>cookies.<\/span>get_dict(<\/span>)<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    # \u65b9\u5f0f2：\u624b\u52a8\u8bbe\u7f6eCookie<\/span>
    \nsession.<\/span>cookies.<\/span>set<\/span>(<\/span>"token"<\/span>,<\/span> "eyJhbGciOiJIUzI1NiJ9.xxxx"<\/span>,<\/span> domain=<\/span>".example.com"<\/span>)<\/span><\/p>\n

    2.5 WebSocket \u57fa\u7840<\/h4>\n

    import<\/span> asyncio
    \nimport<\/span> websockets
    \nimport<\/span> json<\/p>\n

    async<\/span> def<\/span> ws_crawler<\/span>(<\/span>)<\/span>:<\/span>
    \n """WebSocket\u6570\u636e\u6293\u53d6\u793a\u4f8b"""<\/span>
    \n uri =<\/span> "wss:\/\/stream.binance.com:9443\/ws\/btcusdt@trade"<\/span>
    \n async<\/span> with<\/span> websockets.<\/span>connect(<\/span>uri)<\/span> as<\/span> websocket:<\/span>
    \n for<\/span> i in<\/span> range<\/span>(<\/span>100<\/span>)<\/span>:<\/span>
    \n message =<\/span> await<\/span> websocket.<\/span>recv(<\/span>)<\/span>
    \n data =<\/span> json.<\/span>loads(<\/span>message)<\/span>
    \n print<\/span>(<\/span>f"\u4ef7\u683c: <\/span>{<\/span>data[<\/span>'p'<\/span>]<\/span>}<\/span><\/span>, \u6570\u91cf: <\/span>{<\/span>data[<\/span>'q'<\/span>]<\/span>}<\/span><\/span>, \u65f6\u95f4: <\/span>{<\/span>data[<\/span>'T'<\/span>]<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    # asyncio.run(ws_crawler())<\/span><\/p>\n

    2.6 HTTP\/2 \u7279\u6027<\/h4>\n\n\u7279\u6027HTTP\/1.1HTTP\/2\u722c\u866b\u5f71\u54cd<\/tr>\n\n\n\n\n\n\n
    \u591a\u8def\u590d\u7528<\/td>\n\u4e0d\u652f\u6301，\u961f\u5934\u963b\u585e<\/td>\n\u652f\u6301，\u5355\u8fde\u63a5\u591a\u8bf7\u6c42<\/td>\n\u51cf\u5c11\u8fde\u63a5\u6570<\/td>\n<\/tr>\n
    \u5934\u90e8\u538b\u7f29<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u652f\u6301 HPACK<\/td>\n\u51cf\u5c11\u5e26\u5bbd<\/td>\n<\/tr>\n
    \u670d\u52a1\u5668\u63a8\u9001<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n\u53ef\u9884\u53d6\u8d44\u6e90<\/td>\n<\/tr>\n
    \u4e8c\u8fdb\u5236\u5e27<\/td>\n\u6587\u672c<\/td>\n\u4e8c\u8fdb\u5236<\/td>\n\u89e3\u6790\u66f4\u9ad8\u6548<\/td>\n<\/tr>\n
    \u6d41\u4f18\u5148\u7ea7<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n\u8d44\u6e90\u52a0\u8f7d\u4f18\u5316<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    \u9762\u8bd5\u52a0\u5206\u63d0\u793a：2025 \u5e74\u8d77，\u8d8a\u6765\u8d8a\u591a\u7f51\u7ad9\u5f3a\u5236\u8981\u6c42 HTTP\/2 \u652f\u6301\u3002Python \u7684 httpx \u548c curl_cffi \u90fd\u539f\u751f\u652f\u6301 HTTP\/2，\u800c\u4f20\u7edf requests \u5e93\u53ea\u652f\u6301\u5230 HTTP\/1.1\u3002<\/p>\n

    \n

    3. Python\u722c\u866b\u57fa\u7840\u5e93<\/h3>\n

    3.1 \u56db\u5927HTTP\u5ba2\u6237\u7aef\u5bf9\u6bd4<\/h4>\n\n\u7279\u6027requestshttpxaiohttpcurl_cffi<\/tr>\n\n\n\n\n\n\n\n\n\n\n\n
    \u540c\u6b65\/\u5f02\u6b65<\/td>\n\u540c\u6b65<\/td>\n\u540c\u6b65+\u5f02\u6b65<\/td>\n\u5f02\u6b65<\/td>\n\u540c\u6b65+\u5f02\u6b65<\/td>\n<\/tr>\n
    HTTP\/2<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n<\/tr>\n
    TLS\u6307\u7eb9\u4f2a\u88c5<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n<\/tr>\n
    \u6d4f\u89c8\u5668\u6a21\u62df<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u652f\u6301 impersonate<\/td>\n<\/tr>\n
    \u8fde\u63a5\u6c60<\/td>\n\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n<\/tr>\n
    WebSocket<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n\u4e0d\u652f\u6301<\/td>\n<\/tr>\n
    \u81ea\u52a8\u91cd\u8bd5<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n<\/tr>\n
    \u5b66\u4e60\u66f2\u7ebf<\/td>\n\u4f4e<\/td>\n\u4f4e<\/td>\n\u4e2d<\/td>\n\u4f4e<\/td>\n<\/tr>\n
    \u9002\u7528\u573a\u666f<\/td>\n\u7b80\u5355\u811a\u672c<\/td>\n\u73b0\u4ee3\u66ff\u4ee3<\/td>\n\u9ad8\u5e76\u53d1<\/td>\n\u53cd\u722c\u7ed5\u8fc7<\/td>\n<\/tr>\n
    \u7ef4\u62a4\u72b6\u6001<\/td>\n\u7ef4\u62a4\u4e2d<\/td>\n\u6d3b\u8dc3<\/td>\n\u6d3b\u8dc3<\/td>\n\u6d3b\u8dc3(2025\u70ed\u95e8)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    3.2 requests \u8fdb\u9636\u7528\u6cd5<\/h4>\n

    import<\/span> requests
    \nfrom<\/span> requests.<\/span>adapters import<\/span> HTTPAdapter
    \nfrom<\/span> urllib3.<\/span>util.<\/span>retry import<\/span> Retry<\/p>\n

    # \u521b\u5efa\u5e26\u91cd\u8bd5\u673a\u5236\u7684Session<\/span>
    \ndef<\/span> create_robust_session<\/span>(<\/span>)<\/span>:<\/span>
    \n session =<\/span> requests.<\/span>Session(<\/span>)<\/span><\/p>\n

    # \u914d\u7f6e\u91cd\u8bd5\u7b56\u7565<\/span>
    \n retry_strategy =<\/span> Retry(<\/span>
    \n total=<\/span>3<\/span>,<\/span> # \u6700\u5927\u91cd\u8bd5\u6b21\u6570<\/span>
    \n backoff_factor=<\/span>1<\/span>,<\/span> # \u9000\u907f\u56e0\u5b50：1s, 2s, 4s<\/span>
    \n status_forcelist=<\/span>[<\/span>429<\/span>,<\/span> 500<\/span>,<\/span> 502<\/span>,<\/span> 503<\/span>,<\/span> 504<\/span>]<\/span>,<\/span>
    \n allowed_methods=<\/span>[<\/span>"GET"<\/span>,<\/span> "POST"<\/span>]<\/span>,<\/span> # \u5141\u8bb8\u91cd\u8bd5\u7684\u65b9\u6cd5<\/span>
    \n raise_on_status=<\/span>False<\/span>,<\/span> # \u4e0d\u629b\u5f02\u5e38，\u8fd4\u56deResponse<\/span>
    \n )<\/span><\/p>\n

    # \u914d\u7f6e\u8fde\u63a5\u6c60<\/span>
    \n adapter =<\/span> HTTPAdapter(<\/span>
    \n max_retries=<\/span>retry_strategy,<\/span>
    \n pool_connections=<\/span>10<\/span>,<\/span> # \u8fde\u63a5\u6c60\u5927\u5c0f<\/span>
    \n pool_maxsize=<\/span>20<\/span>,<\/span> # \u6700\u5927\u8fde\u63a5\u6570<\/span>
    \n )<\/span><\/p>\n

    session.<\/span>mount(<\/span>"http:\/\/"<\/span>,<\/span> adapter)<\/span>
    \n session.<\/span>mount(<\/span>"https:\/\/"<\/span>,<\/span> adapter)<\/span><\/p>\n

    return<\/span> session<\/p>\n

    # \u4ee3\u7406\u914d\u7f6e<\/span>
    \nproxies =<\/span> {<\/span>
    \n "http"<\/span>:<\/span> "http:\/\/user:pass@proxy.example.com:8080"<\/span>,<\/span>
    \n "https"<\/span>:<\/span> "http:\/\/user:pass@proxy.example.com:8080"<\/span>,<\/span>
    \n}<\/span><\/p>\n

    session =<\/span> create_robust_session(<\/span>)<\/span>
    \nresp =<\/span> session.<\/span>get(<\/span>"https:\/\/httpbin.org\/get"<\/span>,<\/span> proxies=<\/span>proxies,<\/span> timeout=<\/span>10<\/span>)<\/span><\/p>\n

    3.3 httpx \u5f02\u6b65\u5b9e\u6218<\/h4>\n

    import<\/span> httpx
    \nimport<\/span> asyncio<\/p>\n

    async<\/span> def<\/span> async_crawler<\/span>(<\/span>)<\/span>:<\/span>
    \n """httpx\u5f02\u6b65\u722c\u866b\u793a\u4f8b – \u9ad8\u5e76\u53d1\u8bf7\u6c42"""<\/span>
    \n urls =<\/span> [<\/span>
    \n "https:\/\/httpbin.org\/get"<\/span>,<\/span>
    \n "https:\/\/httpbin.org\/headers"<\/span>,<\/span>
    \n "https:\/\/httpbin.org\/cookies"<\/span>,<\/span>
    \n ]<\/span><\/p>\n

    # \u4f7f\u7528\u8fde\u63a5\u6c60\u590d\u7528<\/span>
    \n async<\/span> with<\/span> httpx.<\/span>AsyncClient(<\/span>
    \n http2=<\/span>True<\/span>,<\/span> # \u542f\u7528HTTP\/2<\/span>
    \n timeout=<\/span>httpx.<\/span>Timeout(<\/span>10.0<\/span>)<\/span>,<\/span>
    \n limits=<\/span>httpx.<\/span>Limits(<\/span>
    \n max_connections=<\/span>100<\/span>,<\/span>
    \n max_keepalive_connections=<\/span>20<\/span>,<\/span>
    \n )<\/span>,<\/span>
    \n follow_redirects=<\/span>True<\/span>,<\/span>
    \n )<\/span> as<\/span> client:<\/span>
    \n # \u5e76\u53d1\u8bf7\u6c42<\/span>
    \n tasks =<\/span> [<\/span>client.<\/span>get(<\/span>url)<\/span> for<\/span> url in<\/span> urls]<\/span>
    \n responses =<\/span> await<\/span> asyncio.<\/span>gather(<\/span>*<\/span>tasks,<\/span> return_exceptions=<\/span>True<\/span>)<\/span><\/p>\n

    for<\/span> url,<\/span> resp in<\/span> zip<\/span>(<\/span>urls,<\/span> responses)<\/span>:<\/span>
    \n if<\/span> isinstance<\/span>(<\/span>resp,<\/span> httpx.<\/span>Response)<\/span>:<\/span>
    \n print<\/span>(<\/span>f"<\/span>{<\/span>url}<\/span><\/span> -> <\/span>{<\/span>resp.<\/span>status_code}<\/span><\/span>"<\/span><\/span>)<\/span>
    \n else<\/span>:<\/span>
    \n print<\/span>(<\/span>f"<\/span>{<\/span>url}<\/span><\/span> -> ERROR: <\/span>{<\/span>resp}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    # asyncio.run(async_crawler())<\/span><\/p>\n

    3.4 curl_cffi：TLS\u6307\u7eb9\u4f2a\u88c5\u7684\u5229\u5668<\/h4>\n

    from<\/span> curl_cffi import<\/span> requests as<\/span> cffi_requests<\/p>\n

    # \u6a21\u62dfChrome\u6d4f\u89c8\u5668\u7684TLS\u6307\u7eb9<\/span>
    \nsession =<\/span> cffi_requests.<\/span>Session(<\/span>impersonate=<\/span>"chrome136"<\/span>)<\/span><\/p>\n

    # \u4e00\u6b21impersonate\u53c2\u6570\u81ea\u52a8\u5904\u7406：<\/span>
    \n# – TLS\u6307\u7eb9 (JA3\/JA4)<\/span>
    \n# – HTTP\/2 SETTINGS<\/span>
    \n# – \u4f2a\u5934\u90e8\u987a\u5e8f (masp for Chrome)<\/span>
    \n# – User-Agent\u3001Sec-Ch-Ua \u7b49Headers<\/span>
    \nresponse =<\/span> session.<\/span>get(<\/span>"https:\/\/tls.browserleaks.com\/json"<\/span>)<\/span>
    \ndata =<\/span> response.<\/span>json(<\/span>)<\/span>
    \nprint<\/span>(<\/span>f"JA3 Hash: <\/span>{<\/span>data.<\/span>get(<\/span>'ja3_hash'<\/span>,<\/span> 'N\/A'<\/span>)<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span>
    \nprint<\/span>(<\/span>f"HTTP Version: <\/span>{<\/span>data.<\/span>get(<\/span>'http_version'<\/span>,<\/span> 'N\/A'<\/span>)<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    # \u652f\u6301\u7684\u6d4f\u89c8\u5668\u76ee\u6807（2025\u5e74curl_cffi 0.14\u7248\u672c）<\/span>
    \n# chrome99 ~ chrome142, safari153 ~ safari2601, firefox133\/135\/144<\/span>
    \n# edge99\/101, tor145<\/span>
    \n# \u7b80\u5199: chrome, safari, firefox \u81ea\u52a8\u6307\u5411\u6700\u65b0\u7248\u672c<\/span><\/p>\n

    \u751f\u4ea7\u8e29\u5751\u63d0\u9192：curl_cffi \u7684 JA3 \u54c8\u5e0c\u6bcf\u6b21\u8fd0\u884c\u90fd\u4f1a\u53d8\u5316，\u8fd9\u662f\u6b63\u5e38\u884c\u4e3a\u3002\u56e0\u4e3a GREASE \u673a\u5236\u4f1a\u5728 ClientHello \u4e2d\u63d2\u5165\u968f\u673a\u503c，\u771f\u5b9e Chrome \u4e5f\u662f\u5982\u6b64\u3002\u5173\u952e\u662f\u63e1\u624b\u7684"\u5f62\u72b6"（\u5bc6\u7801\u5957\u4ef6\u3001\u6269\u5c55\u7684\u7ec4\u6210\u548c\u987a\u5e8f）\u8981\u5339\u914d\u76ee\u6807\u6d4f\u89c8\u5668\u3002<\/p>\n

    4. HTML\u89e3\u6790\u4e0e\u6570\u636e\u63d0\u53d6<\/h3>\n

    4.1 \u516d\u5927\u89e3\u6790\u65b9\u6848\u5bf9\u6bd4<\/h4>\n\n\u65b9\u6848\u901f\u5ea6\u6613\u7528\u6027\u529f\u80fd\u4f9d\u8d56\u63a8\u8350\u573a\u666f<\/tr>\n\n\n\n\n\n\n\n
    \u6b63\u5219\u8868\u8fbe\u5f0f<\/td>\n\u6781\u5feb<\/td>\n\u4e2d\u7b49<\/td>\n\u57fa\u7840<\/td>\n\u65e0<\/td>\n\u7b80\u5355\u6a21\u5f0f\u5339\u914d<\/td>\n<\/tr>\n
    BeautifulSoup<\/td>\n\u8f83\u6162<\/td>\n\u6781\u597d<\/td>\n\u5168\u9762<\/td>\nbs4<\/td>\n\u5feb\u901f\u5f00\u53d1<\/td>\n<\/tr>\n
    lxml\/XPath<\/td>\n\u6781\u5feb<\/td>\n\u4e2d\u7b49<\/td>\n\u5f3a\u5927<\/td>\nlxml<\/td>\n\u9ad8\u6027\u80fd\u89e3\u6790<\/td>\n<\/tr>\n
    CSS\u9009\u62e9\u5668<\/td>\n\u5feb<\/td>\n\u597d<\/td>\n\u4e2d\u7b49<\/td>\ncssselect<\/td>\n\u524d\u7aef\u5f00\u53d1\u8005<\/td>\n<\/tr>\n
    pyquery<\/td>\n\u5feb<\/td>\n\u597d<\/td>\njQuery\u98ce\u683c<\/td>\npyquery<\/td>\njQuery\u719f\u6089\u8005<\/td>\n<\/tr>\n
    selectolax<\/td>\n\u6781\u5feb<\/td>\n\u4e2d\u7b49<\/td>\n\u57fa\u7840<\/td>\nselectolax<\/td>\n\u6781\u81f4\u6027\u80fd<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    4.2 \u5b9e\u6218\u4ee3\u7801\u5bf9\u6bd4<\/h4>\n

    html =<\/span> """
    \n<html>
    \n<body>
    \n <div class="product-list">
    \n <div class="item" data-id="1001">
    \n <h3 class="title">Python\u7f16\u7a0b\u4ece\u5165\u95e8\u5230\u5b9e\u8df5<\/h3>
    \n <span class="price">89.00<\/span>
    \n <span class="stock">\u6709\u8d27<\/span>
    \n <a href="\/book\/1001" class="link">\u8be6\u60c5<\/a>
    \n <\/div>
    \n <div class="item" data-id="1002">
    \n <h3 class="title">\u6d41\u7545\u7684Python<\/h3>
    \n <span class="price">128.00<\/span>
    \n <span class="stock">\u7f3a\u8d27<\/span>
    \n <a href="\/book\/1002" class="link">\u8be6\u60c5<\/a>
    \n <\/div>
    \n <\/div>
    \n<\/body>
    \n<\/html>
    \n"""<\/span><\/p>\n

    # ========== \u65b9\u68481: BeautifulSoup ==========<\/span>
    \nfrom<\/span> bs4 import<\/span> BeautifulSoup<\/p>\n

    soup =<\/span> BeautifulSoup(<\/span>html,<\/span> "lxml"<\/span>)<\/span>
    \nfor<\/span> item in<\/span> soup.<\/span>select(<\/span>".item"<\/span>)<\/span>:<\/span>
    \n title =<\/span> item.<\/span>select_one(<\/span>".title"<\/span>)<\/span>.<\/span>get_text(<\/span>)<\/span>
    \n price =<\/span> item.<\/span>select_one(<\/span>".price"<\/span>)<\/span>.<\/span>get_text(<\/span>)<\/span>
    \n book_id =<\/span> item.<\/span>get(<\/span>"data-id"<\/span>)<\/span>
    \n print<\/span>(<\/span>f"[BS4] ID:<\/span>{<\/span>book_id}<\/span><\/span> <\/span>{<\/span>title}<\/span><\/span> <\/span>{<\/span>price}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    # ========== \u65b9\u68482: lxml + XPath ==========<\/span>
    \nfrom<\/span> lxml import<\/span> etree<\/p>\n

    tree =<\/span> etree.<\/span>HTML(<\/span>html)<\/span>
    \nitems =<\/span> tree.<\/span>xpath(<\/span>'\/\/div[@class="item"]'<\/span>)<\/span>
    \nfor<\/span> item in<\/span> items:<\/span>
    \n title =<\/span> item.<\/span>xpath(<\/span>'.\/\/h3[@class="title"]\/text()'<\/span>)<\/span>[<\/span>0<\/span>]<\/span>
    \n price =<\/span> item.<\/span>xpath(<\/span>'.\/\/span[@class="price"]\/text()'<\/span>)<\/span>[<\/span>0<\/span>]<\/span>
    \n book_id =<\/span> item.<\/span>get(<\/span>"data-id"<\/span>)<\/span>
    \n print<\/span>(<\/span>f"[XPath] ID:<\/span>{<\/span>book_id}<\/span><\/span> <\/span>{<\/span>title}<\/span><\/span> <\/span>{<\/span>price}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    # ========== \u65b9\u68483: \u6b63\u5219\u8868\u8fbe\u5f0f ==========<\/span>
    \nimport<\/span> re<\/p>\n

    titles =<\/span> re.<\/span>findall(<\/span>r'class="title">(.*?)<\/h3>'<\/span>,<\/span> html)<\/span>
    \nprices =<\/span> re.<\/span>findall(<\/span>r'class="price">(.*?)<\/span>'<\/span>,<\/span> html)<\/span>
    \nids =<\/span> re.<\/span>findall(<\/span>r'data-id="(\\\\d+)"'<\/span>,<\/span> html)<\/span>
    \nfor<\/span> i,<\/span> t,<\/span> p in<\/span> zip<\/span>(<\/span>ids,<\/span> titles,<\/span> prices)<\/span>:<\/span>
    \n print<\/span>(<\/span>f"[Regex] ID:<\/span>{<\/span>i}<\/span><\/span> <\/span>{<\/span>t}<\/span><\/span> <\/span>{<\/span>p}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    # ========== \u65b9\u68484: pyquery ==========<\/span>
    \nfrom<\/span> pyquery import<\/span> PyQuery as<\/span> pq<\/p>\n

    doc =<\/span> pq(<\/span>html)<\/span>
    \nfor<\/span> item in<\/span> doc(<\/span>".item"<\/span>)<\/span>.<\/span>items(<\/span>)<\/span>:<\/span>
    \n title =<\/span> item(<\/span>".title"<\/span>)<\/span>.<\/span>text(<\/span>)<\/span>
    \n price =<\/span> item(<\/span>".price"<\/span>)<\/span>.<\/span>text(<\/span>)<\/span>
    \n book_id =<\/span> item.<\/span>attr(<\/span>"data-id"<\/span>)<\/span>
    \n print<\/span>(<\/span>f"[PyQuery] ID:<\/span>{<\/span>book_id}<\/span><\/span> <\/span>{<\/span>title}<\/span><\/span> <\/span>{<\/span>price}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    4.3 \u6027\u80fd\u57fa\u51c6\u6d4b\u8bd5\u53c2\u8003<\/h4>\n

    \u6d4b\u8bd5\u6761\u4ef6：10000\u6b21\u91cd\u590d\u89e3\u6790\u540c\u4e00HTML\u9875\u9762<\/p>\n

    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
    \n\u2502 \u65b9\u6848 \u2502 \u8017\u65f6(s) \u2502 \u5185\u5b58(MB) \u2502
    \n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
    \n\u2502 selectolax \u2502 0.3 \u2502 2.1 \u2502
    \n\u2502 lxml (XPath) \u2502 0.8 \u2502 5.3 \u2502
    \n\u2502 parsel \u2502 1.2 \u2502 6.1 \u2502
    \n\u2502 pyquery \u2502 1.5 \u2502 7.8 \u2502
    \n\u2502 BS4 + lxml \u2502 3.2 \u2502 12.4 \u2502
    \n\u2502 BS4+html.parser \u2502 8.5 \u2502 10.2 \u2502
    \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

    \u9762\u8bd5\u52a0\u5206\u63d0\u793a：\u5728\u9ad8\u5e76\u53d1\u722c\u866b\u573a\u666f（\u5982 Scrapy），\u63a8\u8350\u4f7f\u7528 parsel（Scrapy\u5185\u7f6e）\u6216 lxml，\u5b83\u4eec\u57fa\u4e8e C \u6269\u5c55，\u6027\u80fd\u662f BeautifulSoup \u7684 3-10 \u500d\u3002selectolax \u57fa\u4e8e Modest\/lexbor \u5f15\u64ce，\u662f\u76ee\u524d\u6700\u5feb\u7684 Python HTML \u89e3\u6790\u5e93\u3002<\/p>\n

    5. \u4e3b\u6d41\u722c\u866b\u6846\u67b6\u8be6\u89e3<\/h3>\n

    5.1 Scrapy \u67b6\u6784\u5168\u666f<\/h4>\n

    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
    \n\u2502 Scrapy \u6838\u5fc3\u67b6\u6784 \u2502
    \n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
    \n\u2502 \u2502
    \n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502
    \n\u2502 \u2502 Engine \u2502 (\u6838\u5fc3\u5f15\u64ce，\u9a71\u52a8\u6570\u636e\u6d41) \u2502
    \n\u2502 \u2514\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2518 \u2502
    \n\u2502 \u2502 \u2502
    \n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502
    \n\u2502 \u2502 \u2502 \u2502 \u2502
    \n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502
    \n\u2502 \u2502 Scheduler \u2502 \u2502 Downloader \u2502 \u2502 Spider \u2502 \u2502
    \n\u2502 \u2502 (\u8c03\u5ea6\u5668) \u2502 \u2502 (\u4e0b\u8f7d\u5668) \u2502 \u2502 (\u722c\u866b\u903b\u8f91) \u2502 \u2502
    \n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502
    \n\u2502 \u2502 \u2502 \u2502 \u2502
    \n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502
    \n\u2502 \u2502 \u8bf7\u6c42\u961f\u5217 \u2502 \u2502 \u4e0b\u8f7d\u4e2d\u95f4\u4ef6 \u2502 \u2502 \u89e3\u6790\u56de\u8c03 \u2502 \u2502
    \n\u2502 \u2502(Redis\/\u672c\u5730)\u2502 \u2502(\u4ee3\u7406\/\u91cd\u8bd5\/UA)\u2502 \u2502 (parse) \u2502 \u2502
    \n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502
    \n\u2502 \u2502 \u2502
    \n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502
    \n\u2502 \u2502 Pipeline \u2502 \u2502
    \n\u2502 \u2502(\u6e05\u6d17\/\u9a8c\u8bc1\/\u5b58\u50a8)\u2502 \u2502
    \n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502
    \n\u2502 \u2502
    \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

    5.2 Scrapy \u6838\u5fc3\u7ec4\u4ef6<\/h4>\n\n\u7ec4\u4ef6\u804c\u8d23\u5173\u952e\u914d\u7f6e<\/tr>\n\n\n\n\n\n\n\n\n
    Engine<\/td>\n\u63a7\u5236\u6570\u636e\u6d41，\u89e6\u53d1\u4e8b\u4ef6<\/td>\n\u81ea\u52a8\u7ba1\u7406<\/td>\n<\/tr>\n
    Scheduler<\/td>\n\u7ba1\u7406\u8bf7\u6c42\u961f\u5217，\u8c03\u5ea6\u4f18\u5148\u7ea7<\/td>\nSCHEDULER<\/td>\n<\/tr>\n
    Downloader<\/td>\n\u6267\u884cHTTP\u8bf7\u6c42，\u83b7\u53d6\u7f51\u9875<\/td>\nDOWNLOAD_TIMEOUT<\/td>\n<\/tr>\n
    Spider<\/td>\n\u89e3\u6790\u54cd\u5e94，\u63d0\u53d6\u6570\u636e\/\u65b0\u8bf7\u6c42<\/td>\n\u81ea\u5b9a\u4e49 parse()<\/td>\n<\/tr>\n
    Item<\/td>\n\u5b9a\u4e49\u6570\u636e\u7ed3\u6784<\/td>\nscrapy.Item<\/td>\n<\/tr>\n
    Pipeline<\/td>\n\u6e05\u6d17\u3001\u9a8c\u8bc1\u3001\u5b58\u50a8\u6570\u636e<\/td>\nITEM_PIPELINES<\/td>\n<\/tr>\n
    Middleware<\/td>\n\u62e6\u622a\/\u4fee\u6539\u8bf7\u6c42\u548c\u54cd\u5e94<\/td>\nDOWNLOADER_MIDDLEWARES<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    5.3 Scrapy \u5b9e\u6218\u793a\u4f8b<\/h4>\n

    # items.py – \u5b9a\u4e49\u6570\u636e\u7ed3\u6784<\/span>
    \nimport<\/span> scrapy<\/p>\n

    class<\/span> BookItem<\/span>(<\/span>scrapy.<\/span>Item)<\/span>:<\/span>
    \n title =<\/span> scrapy.<\/span>Field(<\/span>)<\/span>
    \n price =<\/span> scrapy.<\/span>Field(<\/span>)<\/span>
    \n rating =<\/span> scrapy.<\/span>Field(<\/span>)<\/span>
    \n url =<\/span> scrapy.<\/span>Field(<\/span>)<\/span><\/p>\n

    # spiders\/books.py – \u722c\u866b\u903b\u8f91<\/span>
    \nimport<\/span> scrapy
    \nfrom<\/span> .<\/span>.<\/span>items import<\/span> BookItem<\/p>\n

    class<\/span> BooksSpider<\/span>(<\/span>scrapy.<\/span>Spider)<\/span>:<\/span>
    \n name =<\/span> "books"<\/span>
    \n allowed_domains =<\/span> [<\/span>"books.toscrape.com"<\/span>]<\/span>
    \n start_urls =<\/span> [<\/span>"https:\/\/books.toscrape.com\/"<\/span>]<\/span><\/p>\n

    # \u81ea\u5b9a\u4e49\u8bbe\u7f6e<\/span>
    \n custom_settings =<\/span> {<\/span>
    \n "DOWNLOAD_DELAY"<\/span>:<\/span> 1<\/span>,<\/span>
    \n "CONCURRENT_REQUESTS_PER_DOMAIN"<\/span>:<\/span> 4<\/span>,<\/span>
    \n }<\/span><\/p>\n

    def<\/span> parse<\/span>(<\/span>self,<\/span> response)<\/span>:<\/span>
    \n for<\/span> book in<\/span> response.<\/span>css(<\/span>"article.product_pod"<\/span>)<\/span>:<\/span>
    \n item =<\/span> BookItem(<\/span>)<\/span>
    \n item[<\/span>"title"<\/span>]<\/span> =<\/span> book.<\/span>css(<\/span>"h3 a::attr(title)"<\/span>)<\/span>.<\/span>get(<\/span>)<\/span>
    \n item[<\/span>"price"<\/span>]<\/span> =<\/span> book.<\/span>css(<\/span>".price_color::text"<\/span>)<\/span>.<\/span>get(<\/span>)<\/span>
    \n item[<\/span>"rating"<\/span>]<\/span> =<\/span> book.<\/span>css(<\/span>"p.star-rating::attr(class)"<\/span>)<\/span>.<\/span>get(<\/span>)<\/span>
    \n item[<\/span>"url"<\/span>]<\/span> =<\/span> response.<\/span>urljoin(<\/span>book.<\/span>css(<\/span>"h3 a::attr(href)"<\/span>)<\/span>.<\/span>get(<\/span>)<\/span>)<\/span>
    \n yield<\/span> item<\/p>\n

    # \u81ea\u52a8\u7ffb\u9875<\/span>
    \n next_page =<\/span> response.<\/span>css(<\/span>"li.next a::attr(href)"<\/span>)<\/span>.<\/span>get(<\/span>)<\/span>
    \n if<\/span> next_page:<\/span>
    \n yield<\/span> response.<\/span>follow(<\/span>next_page,<\/span> self.<\/span>parse)<\/span><\/p>\n

    # pipelines.py – \u6570\u636e\u5b58\u50a8\u7ba1\u9053<\/span>
    \nimport<\/span> pymongo<\/p>\n

    class<\/span> MongoDBPipeline<\/span>:<\/span>
    \n def<\/span> __init__<\/span>(<\/span>self,<\/span> mongo_uri,<\/span> db_name)<\/span>:<\/span>
    \n self.<\/span>mongo_uri =<\/span> mongo_uri
    \n self.<\/span>db_name =<\/span> db_name<\/p>\n

    @classmethod<\/span>
    \n def<\/span> from_crawler<\/span>(<\/span>cls,<\/span> crawler)<\/span>:<\/span>
    \n return<\/span> cls(<\/span>
    \n mongo_uri=<\/span>crawler.<\/span>settings.<\/span>get(<\/span>"MONGO_URI"<\/span>)<\/span>,<\/span>
    \n db_name=<\/span>crawler.<\/span>settings.<\/span>get(<\/span>"MONGO_DB"<\/span>)<\/span>,<\/span>
    \n )<\/span><\/p>\n

    def<\/span> open_spider<\/span>(<\/span>self,<\/span> spider)<\/span>:<\/span>
    \n self.<\/span>client =<\/span> pymongo.<\/span>MongoClient(<\/span>self.<\/span>mongo_uri)<\/span>
    \n self.<\/span>db =<\/span> self.<\/span>client[<\/span>self.<\/span>db_name]<\/span><\/p>\n

    def<\/span> process_item<\/span>(<\/span>self,<\/span> item,<\/span> spider)<\/span>:<\/span>
    \n self.<\/span>db[<\/span>"books"<\/span>]<\/span>.<\/span>update_one(<\/span>
    \n {<\/span>"url"<\/span>:<\/span> item[<\/span>"url"<\/span>]<\/span>}<\/span>,<\/span>
    \n {<\/span>"$set"<\/span>:<\/span> dict<\/span>(<\/span>item)<\/span>}<\/span>,<\/span>
    \n upsert=<\/span>True<\/span>,<\/span>
    \n )<\/span>
    \n return<\/span> item<\/p>\n

    def<\/span> close_spider<\/span>(<\/span>self,<\/span> spider)<\/span>:<\/span>
    \n self.<\/span>client.<\/span>close(<\/span>)<\/span><\/p>\n

    5.4 \u6846\u67b6\u6a2a\u5411\u5bf9\u6bd4<\/h4>\n\n\u6846\u67b6\u8bed\u8a00\u5f02\u6b65\u5206\u5e03\u5f0f\u52a8\u6001\u6e32\u67d3\u5b66\u4e60\u66f2\u7ebf\u9002\u7528\u89c4\u6a21<\/tr>\n\n\n\n\n\n\n\n
    Scrapy<\/td>\nPython<\/td>\n\u652f\u6301(asyncio)<\/td>\n\u652f\u6301(scrapy-redis)<\/td>\n\u9700\u6269\u5c55<\/td>\n\u4e2d<\/td>\n\u5927\u578b<\/td>\n<\/tr>\n
    httpx+bs4<\/td>\nPython<\/td>\n\u652f\u6301(asyncio)<\/td>\n\u9700\u81ea\u5efa<\/td>\n\u9700\u914d\u5408<\/td>\n\u4f4e<\/td>\n\u5c0f\u578b<\/td>\n<\/tr>\n
    Feapder<\/td>\nPython<\/td>\n\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n\u5185\u7f6ePlaywright<\/td>\n\u4f4e<\/td>\n\u4e2d\u578b<\/td>\n<\/tr>\n
    Crawley<\/td>\nPython<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u4f4e<\/td>\n\u5c0f\u578b<\/td>\n<\/tr>\n
    PySpider<\/td>\nPython<\/td>\n\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n\u652f\u6301<\/td>\n\u4e2d<\/td>\n\u4e2d\u578b(\u5df2\u505c\u66f4)<\/td>\n<\/tr>\n
    Colly<\/td>\nGo<\/td>\n\u652f\u6301<\/td>\n\u9700\u81ea\u5efa<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u4e2d<\/td>\n\u5927\u578b<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    \u751f\u4ea7\u8e29\u5751\u63d0\u9192：2025\u5e74 Scrapy \u9ed8\u8ba4 download_delay \u4ece 0 \u79d2\u6539\u4e3a 1 \u79d2，concurrent_requests_per_domain \u4ece 8 \u964d\u4e3a 1\u3002\u8fd9\u662f\u4e3a\u4e86\u964d\u4f4e\u5bf9\u76ee\u6807\u7f51\u7ad9\u7684\u538b\u529b\u3001\u964d\u4f4e\u6cd5\u5f8b\u98ce\u9669\u3002\u5982\u9700\u9ad8\u5e76\u53d1，\u9700\u624b\u52a8\u8c03\u6574\u914d\u7f6e\u3002<\/p>\n

    \n

    6. \u52a8\u6001\u9875\u9762\u722c\u53d6<\/h3>\n

    6.1 \u4e09\u5927\u6d4f\u89c8\u5668\u81ea\u52a8\u5316\u5de5\u5177\u5bf9\u6bd4<\/h4>\n\n\u7279\u6027SeleniumPlaywrightPuppeteer<\/tr>\n\n\n\n\n\n\n\n\n\n\n\n
    \u5f00\u53d1\u5546<\/td>\nSeleniumHQ<\/td>\nMicrosoft<\/td>\nGoogle<\/td>\n<\/tr>\n
    \u6d4f\u89c8\u5668\u652f\u6301<\/td>\nChrome\/Firefox\/Safari\/Edge<\/td>\nChromium\/Firefox\/WebKit<\/td>\n\u4ec5Chromium<\/td>\n<\/tr>\n
    \u8bed\u8a00\u652f\u6301<\/td>\nPython\/Java\/C#\/JS<\/td>\nPython\/JS\/Java\/.NET<\/td>\nJS\/Python(\u975e\u5b98\u65b9)<\/td>\n<\/tr>\n
    \u81ea\u52a8\u7b49\u5f85<\/td>\n\u4e0d\u652f\u6301，\u9700\u624b\u52a8<\/td>\n\u652f\u6301\u667a\u80fd\u7b49\u5f85<\/td>\n\u4e0d\u652f\u6301，\u9700\u624b\u52a8<\/td>\n<\/tr>\n
    \u7f51\u7edc\u62e6\u622a<\/td>\n\u6709\u9650<\/td>\n\u5f3a\u5927<\/td>\n\u5f3a\u5927<\/td>\n<\/tr>\n
    \u6267\u884c\u901f\u5ea6<\/td>\n\u6162<\/td>\n\u5feb（\u5feb30%\u4ee5\u4e0a）<\/td>\n\u5feb<\/td>\n<\/tr>\n
    \u65e0\u5934\u6a21\u5f0f<\/td>\n\u652f\u6301<\/td>\n\u652f\u6301（\u9ed8\u8ba4）<\/td>\n\u652f\u6301<\/td>\n<\/tr>\n
    \u79fb\u52a8\u7aef\u6a21\u62df<\/td>\n\u6709\u9650<\/td>\n\u652f\u6301\u8bbe\u5907\u63cf\u8ff0\u7b26<\/td>\n\u652f\u6301<\/td>\n<\/tr>\n
    \u622a\u56fe\/\u5f55\u5c4f<\/td>\n\u622a\u56fe<\/td>\n\u622a\u56fe+\u5f55\u5c4f<\/td>\n\u622a\u56fe+\u5f55\u5c4f<\/td>\n<\/tr>\n
    \u793e\u533a\u6d3b\u8dc3\u5ea6<\/td>\n\u9ad8（\u8001\u724c）<\/td>\n\u6781\u9ad8（\u65b0\u661f）<\/td>\n\u4e2d<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    6.2 Playwright \u5b9e\u6218<\/h4>\n

    from<\/span> playwright.<\/span>sync_api import<\/span> sync_playwright
    \nimport<\/span> json<\/p>\n

    def<\/span> crawl_spa_page<\/span>(<\/span>)<\/span>:<\/span>
    \n """\u4f7f\u7528Playwright\u722c\u53d6SPA\u52a8\u6001\u9875\u9762"""<\/span>
    \n with<\/span> sync_playwright(<\/span>)<\/span> as<\/span> p:<\/span>
    \n # \u542f\u52a8\u6d4f\u89c8\u5668（Chromium，\u65e0\u5934\u6a21\u5f0f）<\/span>
    \n browser =<\/span> p.<\/span>chromium.<\/span>launch(<\/span>
    \n headless=<\/span>True<\/span>,<\/span>
    \n args=<\/span>[<\/span>"–disable-blink-features=AutomationControlled"<\/span>]<\/span>
    \n )<\/span><\/p>\n

    # \u521b\u5efa\u4e0a\u4e0b\u6587（\u6a21\u62df\u771f\u5b9e\u8bbe\u5907）<\/span>
    \n context =<\/span> browser.<\/span>new_context(<\/span>
    \n viewport=<\/span>{<\/span>"width"<\/span>:<\/span> 1920<\/span>,<\/span> "height"<\/span>:<\/span> 1080<\/span>}<\/span>,<\/span>
    \n user_agent=<\/span>"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) "<\/span>
    \n "AppleWebKit\/537.36 (KHTML, like Gecko) "<\/span>
    \n "Chrome\/131.0.0.0 Safari\/537.36"<\/span>,<\/span>
    \n locale=<\/span>"zh-CN"<\/span>,<\/span>
    \n timezone_id=<\/span>"Asia\/Shanghai"<\/span>,<\/span>
    \n )<\/span><\/p>\n

    page =<\/span> context.<\/span>new_page(<\/span>)<\/span><\/p>\n

    # \u62e6\u622aXHR\/Fetch\u8bf7\u6c42，\u63d0\u53d6API\u6570\u636e<\/span>
    \n api_responses =<\/span> [<\/span>]<\/span>
    \n def<\/span> handle_response<\/span>(<\/span>response)<\/span>:<\/span>
    \n if<\/span> "\/api\/"<\/span> in<\/span> response.<\/span>url:<\/span>
    \n try<\/span>:<\/span>
    \n api_responses.<\/span>append(<\/span>{<\/span>
    \n "url"<\/span>:<\/span> response.<\/span>url,<\/span>
    \n "status"<\/span>:<\/span> response.<\/span>status,<\/span>
    \n "body"<\/span>:<\/span> response.<\/span>json(<\/span>)<\/span>
    \n }<\/span>)<\/span>
    \n except<\/span> Exception:<\/span>
    \n pass<\/span><\/p>\n

    page.<\/span>on(<\/span>"response"<\/span>,<\/span> handle_response)<\/span><\/p>\n

    # \u5bfc\u822a\u5230\u76ee\u6807\u9875\u9762<\/span>
    \n page.<\/span>goto(<\/span>"https:\/\/quotes.toscrape.com\/js\/"<\/span>)<\/span><\/p>\n

    # \u7b49\u5f85\u52a8\u6001\u5185\u5bb9\u52a0\u8f7d<\/span>
    \n page.<\/span>wait_for_selector(<\/span>".quote"<\/span>,<\/span> state=<\/span>"visible"<\/span>,<\/span> timeout=<\/span>10000<\/span>)<\/span><\/p>\n

    # \u63d0\u53d6\u6e32\u67d3\u540e\u7684\u6570\u636e<\/span>
    \n quotes =<\/span> page.<\/span>query_selector_all(<\/span>".quote"<\/span>)<\/span>
    \n results =<\/span> [<\/span>]<\/span>
    \n for<\/span> quote in<\/span> quotes:<\/span>
    \n text =<\/span> quote.<\/span>query_selector(<\/span>".text"<\/span>)<\/span>.<\/span>inner_text(<\/span>)<\/span>
    \n author =<\/span> quote.<\/span>query_selector(<\/span>".author"<\/span>)<\/span>.<\/span>inner_text(<\/span>)<\/span>
    \n results.<\/span>append(<\/span>{<\/span>"text"<\/span>:<\/span> text,<\/span> "author"<\/span>:<\/span> author}<\/span>)<\/span><\/p>\n

    print<\/span>(<\/span>f"\u63d0\u53d6\u5230 <\/span>{<\/span>len<\/span>(<\/span>results)<\/span>}<\/span><\/span> \u6761\u540d\u8a00"<\/span><\/span>)<\/span>
    \n print<\/span>(<\/span>f"\u62e6\u622a\u5230 <\/span>{<\/span>len<\/span>(<\/span>api_responses)<\/span>}<\/span><\/span> \u4e2aAPI\u54cd\u5e94"<\/span><\/span>)<\/span><\/p>\n

    browser.<\/span>close(<\/span>)<\/span>
    \n return<\/span> results<\/p>\n

    # crawl_spa_page()<\/span><\/p>\n

    6.3 JS\u6e32\u67d3\u7b49\u5f85\u7b56\u7565<\/h4>\n\n\u7b56\u7565\u8bf4\u660e\u9002\u7528\u573a\u666f\u4ee3\u7801\u793a\u4f8b<\/tr>\n\n\n\n\n\n\n
    \u56fa\u5b9a\u7b49\u5f85<\/td>\ntime.sleep(n)<\/td>\n\u8c03\u8bd5\u7528，\u4e0d\u63a8\u8350<\/td>\ntime.sleep(3)<\/td>\n<\/tr>\n
    \u5143\u7d20\u7b49\u5f85<\/td>\n\u7b49\u5f85\u7279\u5b9a\u5143\u7d20\u51fa\u73b0<\/td>\n\u6700\u5e38\u7528<\/td>\nwait_for_selector(".data")<\/td>\n<\/tr>\n
    \u7f51\u7edc\u7a7a\u95f2<\/td>\n\u7b49\u5f85\u65e0\u65b0\u8bf7\u6c42<\/td>\nSPA\u52a0\u8f7d<\/td>\nwait_for_load_state("networkidle")<\/td>\n<\/tr>\n
    DOM\u52a0\u8f7d<\/td>\nDOM\u89e3\u6790\u5b8c\u6210<\/td>\n\u9759\u6001\u5185\u5bb9<\/td>\nwait_for_load_state("domcontentloaded")<\/td>\n<\/tr>\n
    \u81ea\u5b9a\u4e49\u6761\u4ef6<\/td>\n\u8f6e\u8be2\u68c0\u67e5<\/td>\n\u590d\u6742\u573a\u666f<\/td>\npage.wait_for_function(…)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    # \u9ad8\u7ea7\u7b49\u5f85\u7b56\u7565\u793a\u4f8b<\/span>
    \n# \u7b49\u5f85\u76f4\u5230\u67d0\u4e2a\u5143\u7d20\u7684\u6587\u672c\u4e0d\u518d\u5305\u542b"\u52a0\u8f7d\u4e2d"<\/span>
    \npage.<\/span>wait_for_function(<\/span>
    \n "() => !document.querySelector('.status').textContent.includes('loading')"<\/span>
    \n)<\/span><\/p>\n

    # \u7b49\u5f85API\u8bf7\u6c42\u5b8c\u6210<\/span>
    \nwith<\/span> page.<\/span>expect_response(<\/span>"**\/api\/data*"<\/span>)<\/span> as<\/span> response_info:<\/span>
    \n page.<\/span>click(<\/span>"#load-button"<\/span>)<\/span>
    \nresponse =<\/span> response_info.<\/span>value
    \ndata =<\/span> response.<\/span>json(<\/span>)<\/span><\/p>\n

    \u9762\u8bd5\u52a0\u5206\u63d0\u793a：Playwright \u7684 expect_response \u662f\u62e6\u622a API \u6570\u636e\u7684\u5229\u5668\u3002\u5f88\u591a SPA \u9875\u9762\u7684\u6570\u636e\u901a\u8fc7 JSON API \u8fd4\u56de，\u76f4\u63a5\u62e6\u622a API \u54cd\u5e94\u6bd4\u89e3\u6790 DOM \u66f4\u9ad8\u6548\u3001\u66f4\u7a33\u5b9a\u3002<\/p>\n

    \n

    7. \u53cd\u722c\u866b\u6280\u672f\u5168\u89e3<\/h3>\n

    7.1 \u53cd\u722c\u6280\u672f\u5206\u7c7b\u5168\u666f<\/h4>\n

    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
    \n\u2502 \u53cd\u722c\u866b\u6280\u672f\u4f53\u7cfb\u5168\u666f \u2502
    \n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
    \n\u2502 \u2502
    \n\u2502 \u8bf7\u6c42\u5c42\u68c0\u6d4b: \u2502
    \n\u2502 +————————————————————+ \u2502
    \n\u2502 | User-Agent\u68c0\u6d4b | Referer\u6821\u9a8c | Accept\u5934\u6821\u9a8c | \u2502
    \n\u2502 | Cookie\u5b8c\u6574\u6027 | \u8bf7\u6c42\u5934\u987a\u5e8f | Accept-Language | \u2502
    \n\u2502 +————————————————————+ \u2502
    \n\u2502 \u2502
    \n\u2502 \u7f51\u7edc\u5c42\u68c0\u6d4b: \u2502
    \n\u2502 +————————————————————+ \u2502
    \n\u2502 | IP\u9891\u7387\u9650\u5236 | IP\u4fe1\u8a89\u8bc4\u5206 | TLS\u6307\u7eb9(JA3\/JA4) | \u2502
    \n\u2502 | HTTP\/2\u6307\u7eb9 | TCP\u6307\u7eb9 | \u5730\u7406\u4f4d\u7f6e\u68c0\u6d4b | \u2502
    \n\u2502 +————————————————————+ \u2502
    \n\u2502 \u2502
    \n\u2502 \u884c\u4e3a\u5c42\u68c0\u6d4b: \u2502
    \n\u2502 +————————————————————+ \u2502
    \n\u2502 | \u8bf7\u6c42\u9891\u7387\u5f02\u5e38 | \u9f20\u6807\u8f68\u8ff9\u5206\u6790 | \u9875\u9762\u505c\u7559\u65f6\u95f4 | \u2502
    \n\u2502 | \u6eda\u52a8\u884c\u4e3a\u6a21\u5f0f | \u70b9\u51fb\u6a21\u5f0f | \u6d4f\u89c8\u5668\u6307\u7eb9(Canvas\/WebGL) | \u2502
    \n\u2502 +————————————————————+ \u2502
    \n\u2502 \u2502
    \n\u2502 \u9a8c\u8bc1\u5c42\u6311\u6218: \u2502
    \n\u2502 +————————————————————+ \u2502
    \n\u2502 | \u56fe\u7247\u9a8c\u8bc1\u7801 | \u6ed1\u5757\u9a8c\u8bc1\u7801 | \u70b9\u9009\u9a8c\u8bc1\u7801 | \u2502
    \n\u2502 | \u884c\u4e3a\u9a8c\u8bc1\u7801 | reCAPTCHA v3 | hCaptcha | \u2502
    \n\u2502 +————————————————————+ \u2502
    \n\u2502 \u2502
    \n\u2502 \u9ad8\u7ea7\u9632\u62a4: \u2502
    \n\u2502 +————————————————————+ \u2502
    \n\u2502 | JS\u6df7\u6dc6\/JSVMP | \u871c\u7f50\u9677\u9631 | \u52a8\u6001Token | \u2502
    \n\u2502 | \u5b57\u4f53\u53cd\u722c | Cookie\u52a0\u5bc6 | \u745e\u6570\/\u6570\u7f8e\u5546\u4e1a\u9632\u62a4 | \u2502
    \n\u2502 +————————————————————+ \u2502
    \n\u2502 \u2502
    \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

    7.2 \u5404\u5c42\u53cd\u722c\u8be6\u89e3<\/h4>\n\n\u53cd\u722c\u7c7b\u578b\u68c0\u6d4b\u539f\u7406\u96be\u5ea6\u5178\u578b\u4ea7\u54c1<\/tr>\n\n\n\n\n\n\n\n\n\n
    User-Agent\u68c0\u6d4b<\/td>\n\u68c0\u67e5UA\u662f\u5426\u4e3a\u5df2\u77e5\u722c\u866b<\/td>\n\u4f4e<\/td>\n\u57fa\u7840\u9632\u62a4<\/td>\n<\/tr>\n
    IP\u9650\u5236<\/td>\n\u5355IP\u8bf7\u6c42\u9891\u7387\u8fc7\u9ad8\u5219\u5c01\u7981<\/td>\n\u4e2d<\/td>\n\u5927\u591a\u6570\u7f51\u7ad9<\/td>\n<\/tr>\n
    Cookie\u9a8c\u8bc1<\/td>\n\u9996\u6b21\u8bbf\u95ee\u8bbe\u7f6e\u52a0\u5bc6Cookie<\/td>\n\u9ad8<\/td>\nCloudflare<\/td>\n<\/tr>\n
    JS\u6311\u6218<\/td>\n\u6267\u884cJS\u8ba1\u7b97\u751f\u6210Token<\/td>\n\u9ad8<\/td>\n\u745e\u6570\u4fe1\u606f<\/td>\n<\/tr>\n
    TLS\u6307\u7eb9<\/td>\n\u68c0\u6d4bJA3\/JA4\u6307\u7eb9<\/td>\n\u5f88\u9ad8<\/td>\nAkamai\/DataDome<\/td>\n<\/tr>\n
    \u6d4f\u89c8\u5668\u6307\u7eb9<\/td>\nCanvas\/WebGL\/Audio\u6307\u7eb9<\/td>\n\u5f88\u9ad8<\/td>\nCloudflare<\/td>\n<\/tr>\n
    \u884c\u4e3a\u5206\u6790<\/td>\n\u9f20\u6807\u8f68\u8ff9\u3001\u70b9\u51fb\u6a21\u5f0f<\/td>\n\u5f88\u9ad8<\/td>\n\u6570\u7f8e\/\u6781\u9a8c<\/td>\n<\/tr>\n
    JSVMP<\/td>\nJS\u865a\u62df\u5316\u4fdd\u62a4<\/td>\n\u6781\u9ad8<\/td>\n\u745e\u6570\/\u6570\u7f8e<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    7.3 \u871c\u7f50\u9677\u9631\u68c0\u6d4b<\/h4>\n

    # \u871c\u7f50\u68c0\u6d4b\u793a\u4f8b — \u7f51\u7ad9\u5728HTML\u4e2d\u653e\u7f6e\u9690\u85cf\u94fe\u63a5<\/span>
    \n# \u6b63\u5e38\u7528\u6237\u4e0d\u53ef\u89c1，\u4f46\u722c\u866b\u4f1a\u8ddf\u968f<\/span><\/p>\n

    from<\/span> lxml import<\/span> etree<\/p>\n

    hidden_html =<\/span> """
    \n<div style="display:none; position:absolute; left:-9999px;">
    \n <a href="\/admin\/secret" class="trap">\u7ba1\u7406\u5165\u53e3<\/a>
    \n <a href="\/api\/internal" class="trap">\u5185\u90e8\u63a5\u53e3<\/a>
    \n<\/div>
    \n<a href="\/real-page" class="real">\u6b63\u5e38\u94fe\u63a5<\/a>
    \n"""<\/span><\/p>\n

    def<\/span> is_visible<\/span>(<\/span>element)<\/span>:<\/span>
    \n """\u68c0\u67e5\u5143\u7d20\u662f\u5426\u53ef\u89c1（\u7b80\u5316\u7248）"""<\/span>
    \n style =<\/span> element.<\/span>get(<\/span>"style"<\/span>,<\/span> ""<\/span>)<\/span>
    \n parent =<\/span> element.<\/span>getparent(<\/span>)<\/span><\/p>\n

    # \u68c0\u67e5display:none<\/span>
    \n if<\/span> "display:none"<\/span> in<\/span> style or<\/span> "display: none"<\/span> in<\/span> style:<\/span>
    \n return<\/span> False<\/span>
    \n # \u68c0\u67e5position\u79fb\u51fa\u89c6\u53e3<\/span>
    \n if<\/span> "left:-9999"<\/span> in<\/span> style or<\/span> "top:-9999"<\/span> in<\/span> style:<\/span>
    \n return<\/span> False<\/span>
    \n # \u9012\u5f52\u68c0\u67e5\u7236\u5143\u7d20<\/span>
    \n if<\/span> parent is<\/span> not<\/span> None<\/span> and<\/span> parent.<\/span>tag !=<\/span> "html"<\/span>:<\/span>
    \n return<\/span> is_visible(<\/span>parent)<\/span>
    \n return<\/span> True<\/span><\/p>\n

    tree =<\/span> etree.<\/span>HTML(<\/span>hidden_html)<\/span>
    \nfor<\/span> link in<\/span> tree.<\/span>xpath(<\/span>"\/\/a"<\/span>)<\/span>:<\/span>
    \n if<\/span> is_visible(<\/span>link)<\/span>:<\/span>
    \n print<\/span>(<\/span>f"\u53ef\u89c1\u94fe\u63a5: <\/span>{<\/span>link.<\/span>get(<\/span>'href'<\/span>)<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span>
    \n else<\/span>:<\/span>
    \n print<\/span>(<\/span>f"\u871c\u7f50\u94fe\u63a5，\u8df3\u8fc7: <\/span>{<\/span>link.<\/span>get(<\/span>'href'<\/span>)<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    \u751f\u4ea7\u8e29\u5751\u63d0\u9192：\u5f88\u591a\u7f51\u7ad9\u4f7f\u7528\u871c\u7f50\u94fe\u63a5\u68c0\u6d4b\u722c\u866b\u3002\u5982\u679c\u4f60\u7684\u722c\u866b\u65e0\u5dee\u522b\u5730\u8ddf\u968f\u6240\u6709\u94fe\u63a5，\u5f88\u53ef\u80fd\u89e6\u53d1\u871c\u7f50\u800c\u88ab\u5c01\u7981\u3002\u5efa\u8bae\u5728\u63d0\u53d6\u94fe\u63a5\u65f6，\u68c0\u67e5\u5143\u7d20\u7684\u53ef\u89c1\u6027\u3002<\/p>\n

    8. \u53cd\u53cd\u722c\u866b\u5b9e\u6218<\/h3>\n

    8.1 \u4ee3\u7406\u6c60\u642d\u5efa（Redis + FastAPI）<\/h4>\n

    # proxy_pool.py – \u8f7b\u91cf\u7ea7\u4ee3\u7406\u6c60\u670d\u52a1<\/span>
    \nimport<\/span> redis
    \nimport<\/span> time
    \nimport<\/span> random
    \nimport<\/span> asyncio
    \nfrom<\/span> fastapi import<\/span> FastAPI
    \nfrom<\/span> typing import<\/span> Optional
    \nfrom<\/span> datetime import<\/span> datetime<\/p>\n

    app =<\/span> FastAPI(<\/span>title=<\/span>"Proxy Pool Service"<\/span>)<\/span><\/p>\n

    # Redis\u8fde\u63a5<\/span>
    \nredis_client =<\/span> redis.<\/span>Redis(<\/span>host=<\/span>"localhost"<\/span>,<\/span> port=<\/span>6379<\/span>,<\/span> db=<\/span>0<\/span>,<\/span> decode_responses=<\/span>True<\/span>)<\/span><\/p>\n

    PROXY_KEY =<\/span> "proxy_pool"<\/span> # \u4ee3\u7406\u6c60Hash<\/span><\/p>\n

    class<\/span> ProxyManager<\/span>:<\/span>
    \n """\u4ee3\u7406\u6c60\u7ba1\u7406\u5668"""<\/span><\/p>\n

    MAX_SCORE =<\/span> 100<\/span> # \u6700\u9ad8\u5206<\/span>
    \n MIN_SCORE =<\/span> 0<\/span> # \u6700\u4f4e\u5206<\/span>
    \n INIT_SCORE =<\/span> 10<\/span> # \u521d\u59cb\u5206<\/span>
    \n CHECK_INTERVAL =<\/span> 300<\/span> # \u68c0\u6d4b\u95f4\u9694(\u79d2)<\/span><\/p>\n

    @classmethod<\/span>
    \n async<\/span> def<\/span> is_valid_proxy<\/span>(<\/span>cls,<\/span> proxy:<\/span> str<\/span>)<\/span> –<\/span>><\/span> bool<\/span>:<\/span>
    \n """\u9a8c\u8bc1\u4ee3\u7406\u662f\u5426\u53ef\u7528"""<\/span>
    \n import<\/span> httpx
    \n try<\/span>:<\/span>
    \n async<\/span> with<\/span> httpx.<\/span>AsyncClient(<\/span>
    \n proxies=<\/span>{<\/span>"all:\/\/"<\/span>:<\/span> f"http:\/\/<\/span>{<\/span>proxy}<\/span><\/span>"<\/span><\/span>}<\/span>,<\/span>
    \n timeout=<\/span>10<\/span>
    \n )<\/span> as<\/span> client:<\/span>
    \n resp =<\/span> await<\/span> client.<\/span>get(<\/span>"https:\/\/httpbin.org\/get"<\/span>)<\/span>
    \n return<\/span> resp.<\/span>status_code ==<\/span> 200<\/span>
    \n except<\/span> Exception:<\/span>
    \n return<\/span> False<\/span><\/p>\n

    @classmethod<\/span>
    \n async<\/span> def<\/span> add_proxy<\/span>(<\/span>cls,<\/span> proxy:<\/span> str<\/span>)<\/span>:<\/span>
    \n """\u6dfb\u52a0\u4ee3\u7406\u5230\u6c60\u4e2d"""<\/span>
    \n if<\/span> not<\/span> redis_client.<\/span>hexists(<\/span>PROXY_KEY,<\/span> proxy)<\/span>:<\/span>
    \n redis_client.<\/span>hset(<\/span>PROXY_KEY,<\/span> proxy,<\/span> cls.<\/span>INIT_SCORE)<\/span><\/p>\n

    @classmethod<\/span>
    \n async<\/span> def<\/span> get_random_proxy<\/span>(<\/span>cls)<\/span> –<\/span>><\/span> Optional[<\/span>str<\/span>]<\/span>:<\/span>
    \n """\u968f\u673a\u83b7\u53d6\u4e00\u4e2a\u9ad8\u5206\u4ee3\u7406"""<\/span>
    \n proxies =<\/span> {<\/span>}<\/span>
    \n all_proxies =<\/span> redis_client.<\/span>hgetall(<\/span>PROXY_KEY)<\/span>
    \n for<\/span> proxy,<\/span> score in<\/span> all_proxies.<\/span>items(<\/span>)<\/span>:<\/span>
    \n if<\/span> int<\/span>(<\/span>score)<\/span> >=<\/span> 50<\/span>:<\/span>
    \n proxies[<\/span>proxy]<\/span> =<\/span> int<\/span>(<\/span>score)<\/span><\/p>\n

    if<\/span> not<\/span> proxies:<\/span>
    \n return<\/span> None<\/span><\/p>\n

    # \u52a0\u6743\u968f\u673a\u9009\u62e9<\/span>
    \n total =<\/span> sum<\/span>(<\/span>proxies.<\/span>values(<\/span>)<\/span>)<\/span>
    \n r =<\/span> random.<\/span>uniform(<\/span>0<\/span>,<\/span> total)<\/span>
    \n cumulative =<\/span> 0<\/span>
    \n for<\/span> proxy,<\/span> score in<\/span> proxies.<\/span>items(<\/span>)<\/span>:<\/span>
    \n cumulative +=<\/span> score
    \n if<\/span> r <=<\/span> cumulative:<\/span>
    \n return<\/span> proxy
    \n return<\/span> list<\/span>(<\/span>proxies.<\/span>keys(<\/span>)<\/span>)<\/span>[<\/span>0<\/span>]<\/span><\/p>\n

    @classmethod<\/span>
    \n async<\/span> def<\/span> increase_score<\/span>(<\/span>cls,<\/span> proxy:<\/span> str<\/span>)<\/span>:<\/span>
    \n """\u4ee3\u7406\u53ef\u7528\u65f6\u52a0\u5206"""<\/span>
    \n current =<\/span> int<\/span>(<\/span>redis_client.<\/span>hget(<\/span>PROXY_KEY,<\/span> proxy)<\/span> or<\/span> 0<\/span>)<\/span>
    \n if<\/span> current <<\/span> cls.<\/span>MAX_SCORE:<\/span>
    \n redis_client.<\/span>hset(<\/span>PROXY_KEY,<\/span> proxy,<\/span> current +<\/span> 1<\/span>)<\/span><\/p>\n

    @classmethod<\/span>
    \n async<\/span> def<\/span> decrease_score<\/span>(<\/span>cls,<\/span> proxy:<\/span> str<\/span>)<\/span>:<\/span>
    \n """\u4ee3\u7406\u4e0d\u53ef\u7528\u65f6\u51cf\u5206"""<\/span>
    \n current =<\/span> int<\/span>(<\/span>redis_client.<\/span>hget(<\/span>PROXY_KEY,<\/span> proxy)<\/span> or<\/span> 0<\/span>)<\/span>
    \n if<\/span> current ><\/span> cls.<\/span>MIN_SCORE:<\/span>
    \n redis_client.<\/span>hset(<\/span>PROXY_KEY,<\/span> proxy,<\/span> current –<\/span> 1<\/span>)<\/span>
    \n else<\/span>:<\/span>
    \n redis_client.<\/span>hdel(<\/span>PROXY_KEY,<\/span> proxy)<\/span><\/p>\n

    proxy_mgr =<\/span> ProxyManager(<\/span>)<\/span><\/p>\n

    @app.<\/span>get<\/span>(<\/span>"\/proxy\/random"<\/span>)<\/span>
    \nasync<\/span> def<\/span> get_proxy<\/span>(<\/span>)<\/span>:<\/span>
    \n """\u83b7\u53d6\u968f\u673a\u4ee3\u7406"""<\/span>
    \n proxy =<\/span> await<\/span> proxy_mgr.<\/span>get_random_proxy(<\/span>)<\/span>
    \n if<\/span> proxy:<\/span>
    \n return<\/span> {<\/span>"proxy"<\/span>:<\/span> proxy,<\/span> "url"<\/span>:<\/span> f"http:\/\/<\/span>{<\/span>proxy}<\/span><\/span>"<\/span><\/span>}<\/span>
    \n return<\/span> {<\/span>"error"<\/span>:<\/span> "no available proxy"<\/span>}<\/span><\/p>\n

    @app.<\/span>post<\/span>(<\/span>"\/proxy\/add"<\/span>)<\/span>
    \nasync<\/span> def<\/span> add_proxy<\/span>(<\/span>proxy:<\/span> str<\/span>)<\/span>:<\/span>
    \n """\u6dfb\u52a0\u4ee3\u7406"""<\/span>
    \n await<\/span> proxy_mgr.<\/span>add_proxy(<\/span>proxy)<\/span>
    \n return<\/span> {<\/span>"status"<\/span>:<\/span> "added"<\/span>,<\/span> "proxy"<\/span>:<\/span> proxy}<\/span><\/p>\n

    8.2 \u9a8c\u8bc1\u7801\u8bc6\u522b（ddddocr）<\/h4>\n

    import<\/span> ddddocr
    \nimport<\/span> requests<\/p>\n

    # \u521d\u59cb\u5316OCR\u8bc6\u522b\u5668<\/span>
    \nocr =<\/span> ddddocr.<\/span>DdddOcr(<\/span>show_ad=<\/span>False<\/span>)<\/span><\/p>\n

    def<\/span> recognize_captcha<\/span>(<\/span>image_url:<\/span> str<\/span>)<\/span> –<\/span>><\/span> str<\/span>:<\/span>
    \n """\u8bc6\u522b\u56fe\u7247\u9a8c\u8bc1\u7801"""<\/span>
    \n response =<\/span> requests.<\/span>get(<\/span>image_url,<\/span> timeout=<\/span>10<\/span>)<\/span>
    \n image_bytes =<\/span> response.<\/span>content
    \n result =<\/span> ocr.<\/span>classification(<\/span>image_bytes)<\/span>
    \n return<\/span> result<\/p>\n

    # \u4f7f\u7528\u793a\u4f8b<\/span>
    \ncaptcha_text =<\/span> recognize_captcha(<\/span>"https:\/\/example.com\/captcha.jpg"<\/span>)<\/span>
    \nprint<\/span>(<\/span>f"\u9a8c\u8bc1\u7801\u8bc6\u522b\u7ed3\u679c: <\/span>{<\/span>captcha_text}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    # \u6ed1\u5757\u9a8c\u8bc1\u7801 – \u7f3a\u53e3\u68c0\u6d4b<\/span>
    \ndet =<\/span> ddddocr.<\/span>DdddOcr(<\/span>det=<\/span>False<\/span>,<\/span> ocr=<\/span>False<\/span>,<\/span> show_ad=<\/span>False<\/span>)<\/span><\/p>\n

    def<\/span> find_slider_gap<\/span>(<\/span>bg_bytes:<\/span> bytes<\/span>,<\/span> slider_bytes:<\/span> bytes<\/span>)<\/span> –<\/span>><\/span> int<\/span>:<\/span>
    \n """\u627e\u5230\u6ed1\u5757\u7f3a\u53e3\u7684x\u5750\u6807"""<\/span>
    \n result =<\/span> det.<\/span>slide_match(<\/span>slider_bytes,<\/span> bg_bytes,<\/span> simple_target=<\/span>True<\/span>)<\/span>
    \n return<\/span> result.<\/span>get(<\/span>"target"<\/span>,<\/span> [<\/span>0<\/span>,<\/span> 0<\/span>]<\/span>)<\/span>[<\/span>0<\/span>]<\/span><\/p>\n

    8.3 curl_cffi \u5b9e\u6218\u7ed5\u8fc7 TLS \u68c0\u6d4b<\/h4>\n

    from<\/span> curl_cffi import<\/span> requests as<\/span> cffi_requests<\/p>\n

    def<\/span> bypass_cloudflare<\/span>(<\/span>url:<\/span> str<\/span>)<\/span>:<\/span>
    \n """\u4f7f\u7528curl_cffi\u7ed5\u8fc7Cloudflare\u7684TLS\u6307\u7eb9\u68c0\u6d4b"""<\/span>
    \n session =<\/span> cffi_requests.<\/span>Session(<\/span>
    \n impersonate=<\/span>"chrome136"<\/span>,<\/span>
    \n verify=<\/span>True<\/span>,<\/span>
    \n )<\/span><\/p>\n

    response =<\/span> session.<\/span>get(<\/span>url,<\/span> timeout=<\/span>30<\/span>)<\/span><\/p>\n

    if<\/span> "cf-chl-bypass"<\/span> in<\/span> response.<\/span>text.<\/span>lower(<\/span>)<\/span>:<\/span>
    \n print<\/span>(<\/span>"Cloudflare\u6311\u6218\u9875\u9762，\u53ef\u80fd\u9700\u8981\u989d\u5916\u5904\u7406"<\/span>)<\/span>
    \n elif<\/span> response.<\/span>status_code ==<\/span> 200<\/span>:<\/span>
    \n print<\/span>(<\/span>f"\u6210\u529f\u7ed5\u8fc7，\u72b6\u6001\u7801: <\/span>{<\/span>response.<\/span>status_code}<\/span><\/span>"<\/span><\/span>)<\/span>
    \n print<\/span>(<\/span>f"\u9875\u9762\u957f\u5ea6: <\/span>{<\/span>len<\/span>(<\/span>response.<\/span>text)<\/span>}<\/span><\/span> \u5b57\u7b26"<\/span><\/span>)<\/span>
    \n else<\/span>:<\/span>
    \n print<\/span>(<\/span>f"\u8bf7\u6c42\u5931\u8d25: <\/span>{<\/span>response.<\/span>status_code}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    return<\/span> response<\/p>\n

    # \u5bf9\u6bd4：requests vs curl_cffi<\/span>
    \n# requests.get(url) -> 403 Forbidden (JA3\u88ab\u6807\u8bb0\u4e3aPython)<\/span>
    \n# curl_cffi.get(url, impersonate="chrome136") -> 200 OK (JA3\u5339\u914dChrome)<\/span><\/p>\n

    8.4 \u8bf7\u6c42\u4f2a\u88c5\u6700\u4f73\u5b9e\u8df5<\/h4>\n

    import<\/span> random
    \nimport<\/span> time<\/p>\n

    # \u771f\u5b9e\u6d4f\u89c8\u5668Header\u6a21\u677f（2025-2026\u7248\u672c）<\/span>
    \nCHROME_HEADERS_TEMPLATES =<\/span> [<\/span>
    \n {<\/span>
    \n "User-Agent"<\/span>:<\/span> "Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 "<\/span>
    \n "(KHTML, like Gecko) Chrome\/131.0.0.0 Safari\/537.36"<\/span>,<\/span>
    \n "Accept"<\/span>:<\/span> "text\/html,application\/xhtml+xml,application\/xml;q=0.9,image\/avif,"<\/span>
    \n "image\/webp,image\/apng,*\/*;q=0.8"<\/span>,<\/span>
    \n "Accept-Language"<\/span>:<\/span> "zh-CN,zh;q=0.9,en;q=0.8"<\/span>,<\/span>
    \n "Accept-Encoding"<\/span>:<\/span> "gzip, deflate, br, zstd"<\/span>,<\/span>
    \n "Sec-Ch-Ua"<\/span>:<\/span> '"Chromium";v="131", "Not_A Brand";v="24"'<\/span>,<\/span>
    \n "Sec-Ch-Ua-Mobile"<\/span>:<\/span> "?0"<\/span>,<\/span>
    \n "Sec-Ch-Ua-Platform"<\/span>:<\/span> '"Windows"'<\/span>,<\/span>
    \n "Sec-Fetch-Dest"<\/span>:<\/span> "document"<\/span>,<\/span>
    \n "Sec-Fetch-Mode"<\/span>:<\/span> "navigate"<\/span>,<\/span>
    \n "Sec-Fetch-Site"<\/span>:<\/span> "none"<\/span>,<\/span>
    \n "Sec-Fetch-User"<\/span>:<\/span> "?1"<\/span>,<\/span>
    \n "Upgrade-Insecure-Requests"<\/span>:<\/span> "1"<\/span>,<\/span>
    \n }<\/span>,<\/span>
    \n]<\/span><\/p>\n

    def<\/span> random_delay<\/span>(<\/span>min_sec:<\/span> float<\/span> =<\/span> 1.0<\/span>,<\/span> max_sec:<\/span> float<\/span> =<\/span> 5.0<\/span>)<\/span>:<\/span>
    \n """\u968f\u673a\u5ef6\u8fdf，\u6a21\u62df\u4eba\u7c7b\u884c\u4e3a"""<\/span>
    \n delay =<\/span> random.<\/span>uniform(<\/span>min_sec,<\/span> max_sec)<\/span>
    \n time.<\/span>sleep(<\/span>delay)<\/span><\/p>\n

    def<\/span> get_stealth_headers<\/span>(<\/span>)<\/span> –<\/span>><\/span> dict<\/span>:<\/span>
    \n """\u83b7\u53d6\u4f2a\u88c5Headers"""<\/span>
    \n return<\/span> random.<\/span>choice(<\/span>CHROME_HEADERS_TEMPLATES)<\/span>.<\/span>copy(<\/span>)<\/span><\/p>\n

    9. \u9006\u5411\u5de5\u7a0b\u57fa\u7840<\/h3>\n

    9.1 Chrome DevTools \u6838\u5fc3\u9762\u677f<\/h4>\n\n\u9762\u677f\u7528\u9014\u722c\u866b\u76f8\u5173\u529f\u80fd<\/tr>\n\n\n\n\n\n\n\n
    Network<\/td>\n\u7f51\u7edc\u8bf7\u6c42\u5206\u6790<\/td>\nXHR\/Fetch\u8fc7\u6ee4\u3001\u8bf7\u6c42\u91cd\u653e\u3001\u53c2\u6570\u67e5\u770b<\/td>\n<\/tr>\n
    Sources<\/td>\n\u6e90\u7801\u8c03\u8bd5<\/td>\n\u65ad\u70b9\u3001\u5355\u6b65\u8c03\u8bd5\u3001Call Stack\u8ffd\u8e2a<\/td>\n<\/tr>\n
    Elements<\/td>\nDOM\u68c0\u67e5<\/td>\n\u5143\u7d20\u5ba1\u67e5\u3001\u4e8b\u4ef6\u76d1\u542c\u5668\u67e5\u770b<\/td>\n<\/tr>\n
    Console<\/td>\n\u63a7\u5236\u53f0<\/td>\nHook\u6ce8\u5165\u3001\u4ee3\u7801\u6267\u884c\u3001\u8c03\u8bd5\u8f93\u51fa<\/td>\n<\/tr>\n
    Application<\/td>\n\u5b58\u50a8\u67e5\u770b<\/td>\nCookie\/LocalStorage\/SessionStorage<\/td>\n<\/tr>\n
    Performance<\/td>\n\u6027\u80fd\u5206\u6790<\/td>\n\u8bc6\u522b\u5173\u952e\u6e32\u67d3\u8def\u5f84<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    9.2 \u9006\u5411\u6838\u5fc3\u6d41\u7a0b<\/h4>\n

    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
    \n\u2502 JS\u9006\u5411\u5de5\u7a0b\u6807\u51c6\u6d41\u7a0b \u2502
    \n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
    \n\u2502 \u2502
    \n\u2502 Step 1: \u6293\u5305\u5206\u6790 Step 2: \u5b9a\u4f4d\u5165\u53e3 \u2502
    \n\u2502 +——————+ +——————+ \u2502
    \n\u2502 | Network | | XHR\u65ad\u70b9 | \u2502
    \n\u2502 | \u9762\u677f\u8fc7\u6ee4XHR |–> | \u5168\u5c40\u641c\u7d22sign | \u2502
    \n\u2502 | \u5206\u6790\u8bf7\u6c42\u53c2\u6570 | | \u641c\u7d22\u52a0\u5bc6\u51fd\u6570 | \u2502
    \n\u2502 +——————+ +——————+ \u2502
    \n\u2502 | \u2502
    \n\u2502 Step 3: \u65ad\u70b9\u8c03\u8bd5 v \u2502
    \n\u2502 +——————+ \u2502
    \n\u2502 | CallStack\u8ffd\u8e2a | \u2502
    \n\u2502 | \u9010\u6b65\u8ddf\u8e2a | \u2502
    \n\u2502 | \u627e\u5230\u52a0\u5bc6\u903b\u8f91 | \u2502
    \n\u2502 +——————+ \u2502
    \n\u2502 | \u2502
    \n\u2502 Step 4: \u7b97\u6cd5\u8bc6\u522b Step 5: \u903b\u8f91\u8fd8\u539f Step 6: Python\u590d\u73b0 \u2502
    \n\u2502 +——————+ +——————+ +—————-+\u2502
    \n\u2502 | MD5\/SHA\/AES | | AST\u8fd8\u539f\u63a7\u5236\u6d41 | | requests\u672c\u5730 \u2502\u2502
    \n\u2502 | RSA\/Base64 |->| \u5b57\u7b26\u4e32\u8868\u8fd8\u539f |->| \u9a8c\u8bc1\u7b7e\u540d \u2502\u2502
    \n\u2502 | HMAC\/\u81ea\u5b9a\u4e49 | | \u6b7b\u4ee3\u7801\u6d88\u9664 | | \u5bf9\u6bd4\u7ed3\u679c \u2502\u2502
    \n\u2502 +——————+ +——————+ +—————-+\u2502
    \n\u2502 \u2502
    \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

    9.3 \u5e38\u89c1\u52a0\u5bc6\u7b97\u6cd5\u8bc6\u522b<\/h4>\n\n\u4ee3\u7801\u7279\u5f81\u7b97\u6cd5\u8f93\u51fa\u7279\u5f81\u5e94\u5bf9\u65b9\u6848<\/tr>\n\n\n\n\n\n\n\n
    CryptoJS.MD5()<\/td>\nMD5<\/td>\n32\u4f4dhex<\/td>\nhashlib.md5<\/td>\n<\/tr>\n
    CryptoJS.SHA256()<\/td>\nSHA-256<\/td>\n64\u4f4dhex<\/td>\nhashlib.sha256<\/td>\n<\/tr>\n
    CryptoJS.AES.encrypt()<\/td>\nAES<\/td>\nBase64\u6216hex<\/td>\npycryptodome<\/td>\n<\/tr>\n
    JSEncrypt \/ encrypt<\/td>\nRSA<\/td>\nBase64<\/td>\nrsa\u5e93<\/td>\n<\/tr>\n
    btoa() \/ atob()<\/td>\nBase64<\/td>\n\u7ed3\u5c3e=<\/td>\nbase64\u6a21\u5757<\/td>\n<\/tr>\n
    encodeURIComponent<\/td>\nURL\u7f16\u7801<\/td>\n%XX\u683c\u5f0f<\/td>\nurllib.parse<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    9.4 Hook \u6280\u672f\u5b9e\u6218<\/h4>\n

    \/\/ \u5728Chrome Console\u4e2d\u6ce8\u5165\u7684Hook\u811a\u672c<\/span><\/p>\n

    \/\/ 1. Hook XMLHttpRequest – \u62e6\u622a\u6240\u6709XHR\u8bf7\u6c42<\/span>
    \n(<\/span>function<\/span>(<\/span>)<\/span> {<\/span>
    \n const<\/span> originalOpen =<\/span> XMLHttpRequest<\/span>.<\/span>prototype.<\/span>open;<\/span>
    \n const<\/span> originalSend =<\/span> XMLHttpRequest<\/span>.<\/span>prototype.<\/span>send;<\/span><\/p>\n

    XMLHttpRequest<\/span>.<\/span>prototype.<\/span>open<\/span> =<\/span> function<\/span>(<\/span>method,<\/span> url<\/span>)<\/span> {<\/span>
    \n this<\/span>.<\/span>_url =<\/span> url;<\/span>
    \n this<\/span>.<\/span>_method =<\/span> method;<\/span>
    \n console.<\/span>log<\/span>(<\/span>'[XHR] '<\/span> +<\/span> method +<\/span> ' '<\/span> +<\/span> url)<\/span>;<\/span>
    \n return<\/span> originalOpen<\/span>.<\/span>apply<\/span>(<\/span>this<\/span>,<\/span> arguments)<\/span>;<\/span>
    \n }<\/span>;<\/span><\/p>\n

    XMLHttpRequest<\/span>.<\/span>prototype.<\/span>send<\/span> =<\/span> function<\/span>(<\/span>body<\/span>)<\/span> {<\/span>
    \n if<\/span> (<\/span>this<\/span>.<\/span>_url &&<\/span> this<\/span>.<\/span>_url.<\/span>includes<\/span>(<\/span>'\/api\/'<\/span>)<\/span>)<\/span> {<\/span>
    \n console.<\/span>log<\/span>(<\/span>'[XHR Body]'<\/span>,<\/span> body)<\/span>;<\/span>
    \n console.<\/span>trace<\/span>(<\/span>)<\/span>;<\/span> \/\/ \u6253\u5370\u8c03\u7528\u6808<\/span>
    \n }<\/span>
    \n return<\/span> originalSend<\/span>.<\/span>apply<\/span>(<\/span>this<\/span>,<\/span> arguments)<\/span>;<\/span>
    \n }<\/span>;<\/span>
    \n}<\/span>)<\/span>(<\/span>)<\/span>;<\/span><\/p>\n

    \/\/ 2. Hook fetch – \u62e6\u622aFetch\u8bf7\u6c42<\/span>
    \n(<\/span>function<\/span>(<\/span>)<\/span> {<\/span>
    \n const<\/span> originalFetch =<\/span> window.<\/span>fetch;<\/span>
    \n window.<\/span>fetch<\/span> =<\/span> function<\/span>(<\/span>…<\/span>args<\/span>)<\/span> {<\/span>
    \n console.<\/span>log<\/span>(<\/span>'[Fetch]'<\/span>,<\/span> args[<\/span>0<\/span>]<\/span>,<\/span> args[<\/span>1<\/span>]<\/span>)<\/span>;<\/span>
    \n console.<\/span>trace<\/span>(<\/span>)<\/span>;<\/span>
    \n return<\/span> originalFetch<\/span>.<\/span>apply<\/span>(<\/span>this<\/span>,<\/span> args)<\/span>;<\/span>
    \n }<\/span>;<\/span>
    \n}<\/span>)<\/span>(<\/span>)<\/span>;<\/span><\/p>\n

    \/\/ 3. Hook setRequestHeader – \u6355\u83b7\u52a0\u5bc6Header<\/span>
    \n(<\/span>function<\/span>(<\/span>)<\/span> {<\/span>
    \n const<\/span> originalSetHeader =<\/span> XMLHttpRequest<\/span>.<\/span>prototype.<\/span>setRequestHeader;<\/span>
    \n XMLHttpRequest<\/span>.<\/span>prototype.<\/span>setRequestHeader<\/span> =<\/span> function<\/span>(<\/span>name,<\/span> value<\/span>)<\/span> {<\/span>
    \n if<\/span> (<\/span>\/<\/span>sign|token|auth|x-<\/span>\/<\/span>i<\/span><\/span>.<\/span>test<\/span>(<\/span>name)<\/span>)<\/span> {<\/span>
    \n console.<\/span>log<\/span>(<\/span>'[Header] '<\/span> +<\/span> name +<\/span> ': '<\/span> +<\/span> value)<\/span>;<\/span>
    \n console.<\/span>trace<\/span>(<\/span>)<\/span>;<\/span>
    \n }<\/span>
    \n return<\/span> originalSetHeader<\/span>.<\/span>apply<\/span>(<\/span>this<\/span>,<\/span> arguments)<\/span>;<\/span>
    \n }<\/span>;<\/span>
    \n}<\/span>)<\/span>(<\/span>)<\/span>;<\/span><\/p>\n

    \/\/ 4. Object.defineProperty Hook – \u8ffd\u8e2a\u5c5e\u6027\u8bfb\u53d6<\/span>
    \n(<\/span>function<\/span>(<\/span>)<\/span> {<\/span>
    \n \/\/ \u76d1\u63a7 navigator.webdriver \u4f55\u65f6\u88ab\u8bfb\u53d6（\u5e38\u89c1\u53cd\u722c\u68c0\u6d4b\u70b9）<\/span>
    \n Object.<\/span>defineProperty<\/span>(<\/span>navigator,<\/span> 'webdriver'<\/span>,<\/span> {<\/span>
    \n get<\/span>:<\/span> function<\/span>(<\/span>)<\/span> {<\/span>
    \n console.<\/span>trace<\/span>(<\/span>'[\u68c0\u6d4b\u70b9] navigator.webdriver \u88ab\u8bfb\u53d6'<\/span>)<\/span>;<\/span>
    \n return<\/span> false<\/span>;<\/span>
    \n }<\/span>
    \n }<\/span>)<\/span>;<\/span>
    \n}<\/span>)<\/span>(<\/span>)<\/span>;<\/span><\/p>\n

    \u9762\u8bd5\u52a0\u5206\u63d0\u793a：XHR\u65ad\u70b9+Call Stack\u8ffd\u8e2a\u662fJS\u9006\u5411\u6700\u57fa\u672c\u4e5f\u6700\u6709\u6548\u7684\u65b9\u6cd5\u3002\u5728 DevTools Sources XHR\/fetch Breakpoints \u4e2d\u6dfb\u52a0\u63a5\u53e3\u5173\u952e\u8bcd，\u89e6\u53d1\u8bf7\u6c42\u540e\u5373\u53ef\u81ea\u52a8\u65ad\u5728\u53c2\u6570\u62fc\u88c5\u4f4d\u7f6e，\u6cbf\u8c03\u7528\u6808\u5411\u4e0a\u8ffd\u6eaf\u5373\u53ef\u627e\u5230\u52a0\u5bc6\u903b\u8f91\u3002<\/p>\n

    10. JS\u9006\u5411\u6df1\u5ea6\u5b9e\u6218<\/h3>\n

    10.1 \u7b7e\u540d\u53c2\u6570（sign）\u7834\u89e3\u6d41\u7a0b<\/h4>\n

    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
    \n\u2502 \u7b7e\u540d\u53c2\u6570\u9006\u5411\u5b8c\u6574\u6d41\u7a0b \u2502
    \n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
    \n\u2502 \u2502
    \n\u2502 Step 1: \u6293\u5305\u53d1\u73b0 \u2502
    \n\u2502 GET \/api\/products?timestamp=1720000000&sign=a1b2c3d4e5 \u2502
    \n\u2502 ^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^ \u2502
    \n\u2502 \u65f6\u95f4\u6233\u53c2\u6570 \u7b7e\u540d\u53c2\u6570(32\u4f4dhex=MD5?) \u2502
    \n\u2502 \u2502
    \n\u2502 Step 2: \u641c\u7d22\u5b9a\u4f4d \u2502
    \n\u2502 DevTools -> Ctrl+Shift+F -> \u641c\u7d22 "sign=" \u6216 "generateSign" \u2502
    \n\u2502 \u2502
    \n\u2502 Step 3: \u65ad\u70b9\u5206\u6790 \u2502
    \n\u2502 \u627e\u5230\u5173\u952e\u51fd\u6570: \u2502
    \n\u2502 function generateSign(params, secret) { \u2502
    \n\u2502 let sorted = Object.keys(params).sort() \u2502
    \n\u2502 .map(k => k + '=' + params[k]).join('&'); \u2502
    \n\u2502 return CryptoJS.MD5(secret + sorted).toString(); \u2502
    \n\u2502 } \u2502
    \n\u2502 \u2502
    \n\u2502 Step 4: Python\u590d\u73b0 \u2502
    \n\u2502 sign = md5(secret + sorted_params).hexdigest() \u2502
    \n\u2502 \u2502
    \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

    import<\/span> hashlib
    \nimport<\/span> time
    \nimport<\/span> requests<\/p>\n

    def<\/span> generate_sign<\/span>(<\/span>params:<\/span> dict<\/span>,<\/span> secret:<\/span> str<\/span> =<\/span> "x$12*_d#@"<\/span>)<\/span> –<\/span>><\/span> str<\/span>:<\/span>
    \n """
    \n \u8fd8\u539f\u76ee\u6807\u7f51\u7ad9\u7684\u7b7e\u540d\u7b97\u6cd5
    \n :param params: \u8bf7\u6c42\u53c2\u6570\u5b57\u5178
    \n :param secret: \u901a\u8fc7JS\u9006\u5411\u83b7\u53d6\u7684\u5bc6\u94a5
    \n """<\/span>
    \n # \u53c2\u6570\u6309key\u6392\u5e8f，\u62fc\u63a5\u4e3a\u5b57\u7b26\u4e32<\/span>
    \n sorted_str =<\/span> "&"<\/span>.<\/span>join(<\/span>
    \n f"<\/span>{<\/span>k}<\/span><\/span>=<\/span>{<\/span>params[<\/span>k]<\/span>}<\/span><\/span>"<\/span><\/span> for<\/span> k in<\/span> sorted<\/span>(<\/span>params.<\/span>keys(<\/span>)<\/span>)<\/span>
    \n )<\/span>
    \n # \u62fc\u63a5\u5bc6\u94a5\u540e\u8ba1\u7b97MD5<\/span>
    \n sign_str =<\/span> secret +<\/span> sorted_str +<\/span> secret
    \n return<\/span> hashlib.<\/span>md5(<\/span>sign_str.<\/span>encode(<\/span>"utf-8"<\/span>)<\/span>)<\/span>.<\/span>hexdigest(<\/span>)<\/span>.<\/span>upper(<\/span>)<\/span><\/p>\n

    def<\/span> crawl_with_sign<\/span>(<\/span>)<\/span>:<\/span>
    \n """\u5e26\u7b7e\u540d\u7684\u8bf7\u6c42"""<\/span>
    \n params =<\/span> {<\/span>
    \n "page"<\/span>:<\/span> 1<\/span>,<\/span>
    \n "size"<\/span>:<\/span> 20<\/span>,<\/span>
    \n "category"<\/span>:<\/span> "electronics"<\/span>,<\/span>
    \n "timestamp"<\/span>:<\/span> int<\/span>(<\/span>time.<\/span>time(<\/span>)<\/span>)<\/span>,<\/span>
    \n }<\/span><\/p>\n

    # \u751f\u6210\u7b7e\u540d<\/span>
    \n params[<\/span>"sign"<\/span>]<\/span> =<\/span> generate_sign(<\/span>params)<\/span><\/p>\n

    response =<\/span> requests.<\/span>get(<\/span>
    \n "https:\/\/api.example.com\/products"<\/span>,<\/span>
    \n params=<\/span>params,<\/span>
    \n headers=<\/span>{<\/span>
    \n "User-Agent"<\/span>:<\/span> "Mozilla\/5.0 (Windows NT 10.0; Win64; x64) "<\/span>
    \n "AppleWebKit\/537.36 Chrome\/131.0.0.0 Safari\/537.36"<\/span>,<\/span>
    \n "Referer"<\/span>:<\/span> "https:\/\/www.example.com\/"<\/span>,<\/span>
    \n }<\/span>,<\/span>
    \n )<\/span><\/p>\n

    return<\/span> response.<\/span>json(<\/span>)<\/span><\/p>\n

    10.2 Cookie \u52a0\u5bc6\u9006\u5411<\/h4>\n

    import<\/span> re
    \nimport<\/span> json
    \nimport<\/span> base64
    \nfrom<\/span> Crypto.<\/span>Cipher import<\/span> AES
    \nfrom<\/span> Crypto.<\/span>Util.<\/span>Padding import<\/span> unpad<\/p>\n

    def<\/span> decrypt_cookie_token<\/span>(<\/span>token:<\/span> str<\/span>,<\/span> key:<\/span> str<\/span>)<\/span> –<\/span>><\/span> dict<\/span>:<\/span>
    \n """
    \n \u89e3\u5bc6Cookie\u4e2d\u7684\u52a0\u5bc6Token
    \n \u5e38\u89c1\u4e8e\u7535\u5546\u7f51\u7ad9\u7684\u52a0\u5bc6Cookie
    \n """<\/span>
    \n try<\/span>:<\/span>
    \n # Base64\u89e3\u7801<\/span>
    \n encrypted =<\/span> base64.<\/span>b64decode(<\/span>token)<\/span><\/p>\n

    # AES-CBC\u89e3\u5bc6<\/span>
    \n cipher =<\/span> AES.<\/span>new(<\/span>
    \n key.<\/span>encode(<\/span>"utf-8"<\/span>)<\/span>[<\/span>:<\/span>16<\/span>]<\/span>,<\/span> # 128\u4f4d\u5bc6\u94a5<\/span>
    \n AES.<\/span>MODE_CBC,<\/span>
    \n iv=<\/span>key.<\/span>encode(<\/span>"utf-8"<\/span>)<\/span>[<\/span>:<\/span>16<\/span>]<\/span>,<\/span>
    \n )<\/span>
    \n decrypted =<\/span> unpad(<\/span>cipher.<\/span>decrypt(<\/span>encrypted)<\/span>,<\/span> AES.<\/span>block_size)<\/span>
    \n return<\/span> json.<\/span>loads(<\/span>decrypted.<\/span>decode(<\/span>"utf-8"<\/span>)<\/span>)<\/span>
    \n except<\/span> Exception as<\/span> e:<\/span>
    \n print<\/span>(<\/span>f"\u89e3\u5bc6\u5931\u8d25: <\/span>{<\/span>e}<\/span><\/span>"<\/span><\/span>)<\/span>
    \n return<\/span> {<\/span>}<\/span><\/p>\n

    10.3 \u5b57\u4f53\u53cd\u722c\u7834\u89e3<\/h4>\n

    import<\/span> io
    \nimport<\/span> fontTools
    \nfrom<\/span> fontTools.<\/span>ttLib import<\/span> TTFont
    \nimport<\/span> requests<\/p>\n

    def<\/span> parse_woff_font<\/span>(<\/span>font_bytes:<\/span> bytes<\/span>)<\/span> –<\/span>><\/span> dict<\/span>:<\/span>
    \n """
    \n \u89e3\u6790woff\u5b57\u4f53\u6587\u4ef6，\u63d0\u53d6\u5b57\u7b26\u6620\u5c04\u5173\u7cfb
    \n \u7528\u4e8e\u7834\u89e3\u5b57\u4f53\u53cd\u722c（\u5e38\u89c1\u4e8e\u7535\u5546\u4ef7\u683c\u3001\u7535\u8bdd\u53f7\u7801）
    \n """<\/span>
    \n font =<\/span> TTFont(<\/span>io.<\/span>BytesIO(<\/span>font_bytes)<\/span>)<\/span><\/p>\n

    # \u83b7\u53d6\u5b57\u7b26\u6620\u5c04\u8868<\/span>
    \n cmap =<\/span> font.<\/span>getBestCmap(<\/span>)<\/span><\/p>\n

    # \u6784\u5efa Unicode -> \u5b57\u5f62\u540d \u7684\u6620\u5c04<\/span>
    \n mapping =<\/span> {<\/span>}<\/span>
    \n for<\/span> code,<\/span> name in<\/span> cmap.<\/span>items(<\/span>)<\/span>:<\/span>
    \n mapping[<\/span>chr<\/span>(<\/span>code)<\/span>]<\/span> =<\/span> name<\/p>\n

    return<\/span> mapping<\/p>\n

    # \u4f7f\u7528\u793a\u4f8b<\/span>
    \n# resp = requests.get("https:\/\/example.com\/font.woff")<\/span>
    \n# mapping = parse_woff_font(resp.content)<\/span>
    \n# print(f"\u5b57\u7b26\u6620\u5c04: {mapping}")<\/span><\/p>\n

    \u751f\u4ea7\u8e29\u5751\u63d0\u9192：\u5b57\u4f53\u53cd\u722c\u7684\u6620\u5c04\u5173\u7cfb\u901a\u5e38\u662f\u52a8\u6001\u751f\u6210\u7684，\u6bcf\u6b21\u8bf7\u6c42\u90fd\u53ef\u80fd\u4e0d\u540c\u3002\u4f60\u9700\u8981\u5728\u6bcf\u6b21\u8bf7\u6c42\u65f6\u540c\u6b65\u4e0b\u8f7d\u5b57\u4f53\u6587\u4ef6\u5e76\u91cd\u65b0\u89e3\u6790\u6620\u5c04\u8868\u3002<\/p>\n

    10.4 JS\u6df7\u6dc6\u5bf9\u6297<\/h4>\n\n\u6df7\u6dc6\u6280\u672f\u7279\u5f81\u7834\u89e3\u65b9\u6cd5<\/tr>\n\n\n\n\n\n\n
    \u53d8\u91cf\u540d\u6df7\u6dc6<\/td>\n_0x3f2a12 \u7b49\u65e0\u610f\u4e49\u540d\u79f0<\/td>\nAST\u53cd\u6df7\u6dc6+\u8bed\u4e49\u5206\u6790<\/td>\n<\/tr>\n
    \u63a7\u5236\u6d41\u5e73\u5766\u5316<\/td>\n\u5de8\u5927\u7684switch-case<\/td>\nAST\u5206\u6790+\u72b6\u6001\u673a\u8fd8\u539f<\/td>\n<\/tr>\n
    \u5b57\u7b26\u4e32\u52a0\u5bc6<\/td>\n\u5b57\u7b26\u4e32\u6570\u7ec4+\u7d22\u5f15\u8bbf\u95ee<\/td>\n\u89e3\u5bc6\u51fd\u6570\u5b9a\u4f4d+\u6279\u91cf\u8fd8\u539f<\/td>\n<\/tr>\n
    \u6b7b\u4ee3\u7801\u6ce8\u5165<\/td>\n\u672a\u4f7f\u7528\u7684\u51fd\u6570\/\u53d8\u91cf<\/td>\n\u4ee3\u7801\u8986\u76d6\u7387\u5206\u6790<\/td>\n<\/tr>\n
    JSVMP<\/td>\n\u5b57\u8282\u7801+\u89e3\u91ca\u5668\u5faa\u73af<\/td>\n\u63d2\u6869\u8ffd\u8e2a+Hook\u51fa\u53e3<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    11. APP\u722c\u866b\u4e0e\u6293\u5305<\/h3>\n

    11.1 \u6293\u5305\u5de5\u5177\u5bf9\u6bd4<\/h4>\n\n\u5de5\u5177\u5e73\u53f0HTTPS\u89e3\u5bc6\u811a\u672c\u80fd\u529b\u514d\u8d39\u63a8\u8350\u573a\u666f<\/tr>\n\n\n\n\n\n\n
    Charles<\/td>\n\u5168\u5e73\u53f0<\/td>\n\u652f\u6301<\/td>\nRewrite\u89c4\u5219<\/td>\n\u8bd5\u752830min<\/td>\nMac\u7528\u6237\u9996\u9009<\/td>\n<\/tr>\n
    Fiddler<\/td>\nWindows<\/td>\n\u652f\u6301<\/td>\nFiddlerScript<\/td>\n\u514d\u8d39<\/td>\nWindows\u7528\u6237<\/td>\n<\/tr>\n
    Burp Suite<\/td>\n\u5168\u5e73\u53f0<\/td>\n\u652f\u6301<\/td>\nExtension\u6269\u5c55<\/td>\nCommunity\u7248<\/td>\n\u5b89\u5168\u6d4b\u8bd5<\/td>\n<\/tr>\n
    mitmproxy<\/td>\n\u5168\u5e73\u53f0<\/td>\n\u652f\u6301<\/td>\nPython\u811a\u672c<\/td>\n\u5f00\u6e90\u514d\u8d39<\/td>\n\u7a0b\u5e8f\u5458\u9996\u9009<\/td>\n<\/tr>\n
    Wireshark<\/td>\n\u5168\u5e73\u53f0<\/td>\n\u9700\u5bc6\u94a5<\/td>\n\u4e0d\u652f\u6301<\/td>\n\u514d\u8d39<\/td>\n\u5e95\u5c42\u534f\u8bae\u5206\u6790<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    11.2 mitmproxy \u811a\u672c\u7f16\u5199<\/h4>\n

    # mitm_proxy.py – mitmproxy\u722c\u866b\u6293\u5305\u811a\u672c<\/span>
    \n# \u542f\u52a8\u547d\u4ee4: mitmweb -s mitm_proxy.py -p 8080<\/span><\/p>\n

    from<\/span> mitmproxy import<\/span> http
    \nimport<\/span> json
    \nimport<\/span> os<\/p>\n

    # \u914d\u7f6e\u8981\u6293\u53d6\u7684API\u8def\u5f84\u5173\u952e\u8bcd<\/span>
    \nAPI_KEYWORDS =<\/span> [<\/span>"\/api\/"<\/span>,<\/span> "\/graphql"<\/span>,<\/span> "\/v1\/"<\/span>,<\/span> "\/v2\/"<\/span>]<\/span><\/p>\n

    class<\/span> CrawlAddon<\/span>:<\/span>
    \n """mitmproxy\u722c\u866b\u63d2\u4ef6"""<\/span><\/p>\n

    def<\/span> __init__<\/span>(<\/span>self)<\/span>:<\/span>
    \n self.<\/span>output_dir =<\/span> ".\/captured_data"<\/span>
    \n os.<\/span>makedirs(<\/span>self.<\/span>output_dir,<\/span> exist_ok=<\/span>True<\/span>)<\/span>
    \n self.<\/span>request_count =<\/span> 0<\/span><\/p>\n

    def<\/span> response<\/span>(<\/span>self,<\/span> flow:<\/span> http.<\/span>HTTPFlow)<\/span>:<\/span>
    \n """\u62e6\u622a\u54cd\u5e94"""<\/span>
    \n url =<\/span> flow.<\/span>request.<\/span>url<\/p>\n

    if<\/span> not<\/span> any<\/span>(<\/span>kw in<\/span> url for<\/span> kw in<\/span> API_KEYWORDS)<\/span>:<\/span>
    \n return<\/span><\/p>\n

    content_type =<\/span> flow.<\/span>response.<\/span>headers.<\/span>get(<\/span>"content-type"<\/span>,<\/span> ""<\/span>)<\/span>
    \n if<\/span> "json"<\/span> not<\/span> in<\/span> content_type:<\/span>
    \n return<\/span><\/p>\n

    self.<\/span>request_count +=<\/span> 1<\/span><\/p>\n

    try<\/span>:<\/span>
    \n data =<\/span> json.<\/span>loads(<\/span>flow.<\/span>response.<\/span>get_text(<\/span>)<\/span>)<\/span><\/p>\n

    filename =<\/span> f"<\/span>{<\/span>self.<\/span>output_dir}<\/span><\/span>\/req_<\/span>{<\/span>self.<\/span>request_count:<\/span>04d<\/span>}<\/span><\/span>.json"<\/span><\/span>
    \n with<\/span> open<\/span>(<\/span>filename,<\/span> "w"<\/span>,<\/span> encoding=<\/span>"utf-8"<\/span>)<\/span> as<\/span> f:<\/span>
    \n json.<\/span>dump(<\/span>{<\/span>
    \n "url"<\/span>:<\/span> url,<\/span>
    \n "method"<\/span>:<\/span> flow.<\/span>request.<\/span>method,<\/span>
    \n "request_headers"<\/span>:<\/span> dict<\/span>(<\/span>flow.<\/span>request.<\/span>headers)<\/span>,<\/span>
    \n "request_body"<\/span>:<\/span> flow.<\/span>request.<\/span>get_text(<\/span>)<\/span>,<\/span>
    \n "status_code"<\/span>:<\/span> flow.<\/span>response.<\/span>status_code,<\/span>
    \n "response_data"<\/span>:<\/span> data,<\/span>
    \n }<\/span>,<\/span> f,<\/span> ensure_ascii=<\/span>False<\/span>,<\/span> indent=<\/span>2<\/span>)<\/span><\/p>\n

    print<\/span>(<\/span>f"[<\/span>{<\/span>self.<\/span>request_count}<\/span><\/span>] <\/span>{<\/span>flow.<\/span>request.<\/span>method}<\/span><\/span> <\/span>{<\/span>url}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    except<\/span> json.<\/span>JSONDecodeError:<\/span>
    \n pass<\/span><\/p>\n

    addons =<\/span> [<\/span>CrawlAddon(<\/span>)<\/span>]<\/span><\/p>\n

    11.3 APP\u6293\u5305\u5173\u952e\u6b65\u9aa4<\/h4>\n

    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
    \n\u2502 APP\u6293\u5305\u5b8c\u6574\u6d41\u7a0b \u2502
    \n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
    \n\u2502 \u2502
    \n\u2502 [1.\u5b89\u88c5\u8bc1\u4e66] [2.\u914d\u7f6e\u4ee3\u7406] [3.\u4fe1\u4efb\u8bc1\u4e66] \u2502
    \n\u2502 \u5bfc\u51faCA\u8bc1\u4e66 –> WiFi\u4ee3\u7406 –> \u7cfb\u7edf\u8bbe\u7f6e \u2502
    \n\u2502 \u5230\u624b\u673a \u6307\u5411\u7535\u8111 \u5b89\u88c5\u8bc1\u4e66 \u2502
    \n\u2502 | \u2502
    \n\u2502 [4.\u9a8c\u8bc1\u6293\u5305] [5.\u5206\u6790\u534f\u8bae] [6.\u6a21\u62df\u8bf7\u6c42] \u2502
    \n\u2502 \u6253\u5f00APP –> \u7b5b\u9009API –> Python\u590d\u73b0 \u2502
    \n\u2502 \u89e6\u53d1\u64cd\u4f5c \u5206\u6790\u53c2\u6570 \u8bf7\u6c42 \u2502
    \n\u2502 \u2502
    \n\u2502 \u5e38\u89c1\u95ee\u9898: \u2502
    \n\u2502 – Android 7+\u4e0d\u4fe1\u4efb\u7528\u6237\u8bc1\u4e66 -> Root\u540e\u5b89\u88c5\u5230\u7cfb\u7edf\u76ee\u5f55 \u2502
    \n\u2502 – APP\u8bc1\u4e66\u7ed1\u5b9a(Certificate Pinning) -> Frida Hook \u2502
    \n\u2502 – \u8bf7\u6c42\u53c2\u6570\u52a0\u5bc6 -> \u9006\u5411APP\u6216Hook\u52a0\u5bc6\u51fd\u6570 \u2502
    \n\u2502 \u2502
    \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

    \u9762\u8bd5\u52a0\u5206\u63d0\u793a：Android 7.0+ \u9ed8\u8ba4\u4e0d\u4fe1\u4efb\u7528\u6237\u5b89\u88c5\u7684 CA \u8bc1\u4e66\u3002\u89e3\u51b3\u65b9\u6848：(1) Root \u540e\u5c06\u8bc1\u4e66\u5b89\u88c5\u5230\u7cfb\u7edf\u76ee\u5f55；(2) \u4f7f\u7528 Frida \u7ed5\u8fc7 Certificate Pinning；(3) \u4f7f\u7528\u5df2 Root \u7684\u6a21\u62df\u5668\u914d\u5408 Xposed \u7684 JustTrustMe \u6a21\u5757\u3002<\/p>\n

    12. \u5206\u5e03\u5f0f\u722c\u866b\u67b6\u6784<\/h3>\n

    12.1 Scrapy-Redis \u5206\u5e03\u5f0f\u539f\u7406<\/h4>\n

    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
    \n\u2502 Scrapy-Redis \u5206\u5e03\u5f0f\u67b6\u6784 \u2502
    \n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
    \n\u2502 \u2502
    \n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502
    \n\u2502 \u2502 Redis \u4e2d\u592e\u5b58\u50a8 \u2502 \u2502
    \n\u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502
    \n\u2502 \u2502 \u2502 \u8bf7\u6c42\u961f\u5217 \u2502 \u2502 \u53bb\u91cd\u96c6\u5408 \u2502 \u2502 \u7ed3\u679c\u961f\u5217 \u2502 \u2502 \u2502
    \n\u2502 \u2502 \u2502 (List\/ZSET)\u2502 \u2502 (Set) \u2502 \u2502 (List) \u2502 \u2502 \u2502
    \n\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502
    \n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502
    \n\u2502 \u2502 \u2502 \u2502 \u2502
    \n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502
    \n\u2502 \u2502 \u2502 \u2502 \u2502 \u2502 \u2502
    \n\u2502 [Worker 1] [Worker 2] [Worker N] \u2502
    \n\u2502 +———+ +———-+ +———+ \u2502
    \n\u2502 |Scrapy | | Scrapy | | Scrapy | \u2502
    \n\u2502 |Engine | | Engine | | Engine | \u2502
    \n\u2502 +—-+—-+ +—-+—–+ +—-+—-+ \u2502
    \n\u2502 | | | \u2502
    \n\u2502 [Pipeline] [Pipeline] [Pipeline] \u2502
    \n\u2502 (\u672c\u5730\u5b58\u50a8) (\u672c\u5730\u5b58\u50a8) (\u672c\u5730\u5b58\u50a8) \u2502
    \n\u2502 \u2502
    \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

    12.2 Scrapy-Redis \u914d\u7f6e<\/h4>\n

    # settings.py – Scrapy-Redis \u5206\u5e03\u5f0f\u914d\u7f6e<\/span><\/p>\n

    # \u4f7f\u7528Redis\u8c03\u5ea6\u5668<\/span>
    \nSCHEDULER =<\/span> "scrapy_redis.scheduler.Scheduler"<\/span><\/p>\n

    # \u4f7f\u7528Redis\u53bb\u91cd<\/span>
    \nDUPEFILTER_CLASS =<\/span> "scrapy_redis.dupefilter.RFPDupeFilter"<\/span><\/p>\n

    # Redis\u8fde\u63a5\u914d\u7f6e<\/span>
    \nREDIS_URL =<\/span> "redis:\/\/:your_password@redis-master:6379\/0"<\/span><\/p>\n

    # \u6301\u4e45\u5316\u961f\u5217（\u65ad\u70b9\u7eed\u722c）<\/span>
    \nSCHEDULER_PERSIST =<\/span> True<\/span><\/p>\n

    # \u961f\u5217\u7c7b\u578b<\/span>
    \n# SCHEDULER_QUEUE_CLASS = "scrapy_redis.queue.FifoQueue" # \u5148\u8fdb\u5148\u51fa<\/span>
    \n# SCHEDULER_QUEUE_CLASS = "scrapy_redis.queue.LifoQueue" # \u540e\u8fdb\u5148\u51fa(\u6df1\u5ea6\u4f18\u5148)<\/span>
    \n# SCHEDULER_QUEUE_CLASS = "scrapy_redis.queue.PriorityQueue" # \u4f18\u5148\u7ea7\u961f\u5217<\/span><\/p>\n

    12.3 \u5e03\u9686\u8fc7\u6ee4\u5668\u53bb\u91cd<\/h4>\n

    import<\/span> math
    \nimport<\/span> mmh3
    \nfrom<\/span> bitarray import<\/span> bitarray<\/p>\n

    class<\/span> BloomFilter<\/span>:<\/span>
    \n """
    \n \u5e03\u9686\u8fc7\u6ee4\u5668 – \u5185\u5b58\u9ad8\u6548\u7684URL\u53bb\u91cd\u65b9\u6848
    \n \u9002\u7528\u4e8e\u6d77\u91cfURL\u53bb\u91cd，\u8bef\u5224\u7387\u53ef\u63a7
    \n """<\/span><\/p>\n

    def<\/span> __init__<\/span>(<\/span>self,<\/span> expected_count:<\/span> int<\/span>,<\/span> error_rate:<\/span> float<\/span> =<\/span> 0.001<\/span>)<\/span>:<\/span>
    \n self.<\/span>size =<\/span> self.<\/span>_optimal_size(<\/span>expected_count,<\/span> error_rate)<\/span>
    \n self.<\/span>hash_count =<\/span> self.<\/span>_optimal_hash_count(<\/span>self.<\/span>size,<\/span> expected_count)<\/span>
    \n self.<\/span>bit_array =<\/span> bitarray(<\/span>self.<\/span>size)<\/span>
    \n self.<\/span>bit_array.<\/span>setall(<\/span>0<\/span>)<\/span>
    \n self.<\/span>count =<\/span> 0<\/span><\/p>\n

    @staticmethod<\/span>
    \n def<\/span> _optimal_size<\/span>(<\/span>n:<\/span> int<\/span>,<\/span> p:<\/span> float<\/span>)<\/span> –<\/span>><\/span> int<\/span>:<\/span>
    \n m =<\/span> –<\/span>(<\/span>n *<\/span> math.<\/span>log(<\/span>p)<\/span>)<\/span> \/<\/span> (<\/span>math.<\/span>log(<\/span>2<\/span>)<\/span> **<\/span> 2<\/span>)<\/span>
    \n return<\/span> int<\/span>(<\/span>m)<\/span><\/p>\n

    @staticmethod<\/span>
    \n def<\/span> _optimal_hash_count<\/span>(<\/span>m:<\/span> int<\/span>,<\/span> n:<\/span> int<\/span>)<\/span> –<\/span>><\/span> int<\/span>:<\/span>
    \n k =<\/span> (<\/span>m \/<\/span> n)<\/span> *<\/span> math.<\/span>log(<\/span>2<\/span>)<\/span>
    \n return<\/span> int<\/span>(<\/span>k)<\/span><\/p>\n

    def<\/span> _hash_values<\/span>(<\/span>self,<\/span> item:<\/span> str<\/span>)<\/span>:<\/span>
    \n for<\/span> i in<\/span> range<\/span>(<\/span>self.<\/span>hash_count)<\/span>:<\/span>
    \n yield<\/span> mmh3.<\/span>hash<\/span>(<\/span>item,<\/span> i)<\/span> %<\/span> self.<\/span>size<\/p>\n

    def<\/span> add<\/span>(<\/span>self,<\/span> item:<\/span> str<\/span>)<\/span>:<\/span>
    \n for<\/span> bit_pos in<\/span> self.<\/span>_hash_values(<\/span>item)<\/span>:<\/span>
    \n self.<\/span>bit_array[<\/span>bit_pos]<\/span> =<\/span> 1<\/span>
    \n self.<\/span>count +=<\/span> 1<\/span><\/p>\n

    def<\/span> __contains__<\/span>(<\/span>self,<\/span> item:<\/span> str<\/span>)<\/span> –<\/span>><\/span> bool<\/span>:<\/span>
    \n return<\/span> all<\/span>(<\/span>
    \n self.<\/span>bit_array[<\/span>bit_pos]<\/span>
    \n for<\/span> bit_pos in<\/span> self.<\/span>_hash_values(<\/span>item)<\/span>
    \n )<\/span><\/p>\n

    # \u4f7f\u7528\u793a\u4f8b<\/span>
    \nbf =<\/span> BloomFilter(<\/span>expected_count=<\/span>10_000_000<\/span>,<\/span> error_rate=<\/span>0.001<\/span>)<\/span>
    \nprint<\/span>(<\/span>f"\u4f4d\u6570\u7ec4\u5927\u5c0f: <\/span>{<\/span>bf.<\/span>size \/<\/span> 1024<\/span> \/<\/span> 1024<\/span>:<\/span>.1f<\/span>}<\/span><\/span> MB"<\/span><\/span>)<\/span>
    \nprint<\/span>(<\/span>f"\u54c8\u5e0c\u51fd\u6570\u6570: <\/span>{<\/span>bf.<\/span>hash_count}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    bf.<\/span>add(<\/span>"https:\/\/example.com\/page\/1"<\/span>)<\/span>
    \nbf.<\/span>add(<\/span>"https:\/\/example.com\/page\/2"<\/span>)<\/span>
    \nprint<\/span>(<\/span>"page\/1\u5b58\u5728:"<\/span>,<\/span> "https:\/\/example.com\/page\/1"<\/span> in<\/span> bf)<\/span> # True<\/span>
    \nprint<\/span>(<\/span>"page\/3\u5b58\u5728:"<\/span>,<\/span> "https:\/\/example.com\/page\/3"<\/span> in<\/span> bf)<\/span> # False<\/span><\/p>\n

    12.4 \u5206\u5e03\u5f0f\u8c03\u5ea6\u65b9\u6848\u5bf9\u6bd4<\/h4>\n\n\u65b9\u6848\u7279\u70b9\u9002\u7528\u89c4\u6a21\u590d\u6742\u5ea6<\/tr>\n\n\n\n\n\n\n
    Scrapy-Redis<\/td>\n\u7b80\u5355，Redis\u961f\u5217<\/td>\n\u4e2d\u5c0f（10\u8282\u70b9\u5185）<\/td>\n\u4f4e<\/td>\n<\/tr>\n
    Celery + Redis<\/td>\n\u4efb\u52a1\u961f\u5217\u6210\u719f<\/td>\n\u4e2d\u7b49<\/td>\n\u4e2d<\/td>\n<\/tr>\n
    Kafka + Scrapy<\/td>\n\u9ad8\u541e\u5410，\u6301\u4e45\u5316<\/td>\n\u5927\u578b（100+\u8282\u70b9）<\/td>\n\u9ad8<\/td>\n<\/tr>\n
    RabbitMQ + Scrapy<\/td>\n\u53ef\u9760\u6d88\u606f\u4f20\u9012<\/td>\n\u4e2d\u5927\u578b<\/td>\n\u4e2d<\/td>\n<\/tr>\n
    \u81ea\u7814\u8c03\u5ea6（Redis+FastAPI）<\/td>\n\u5b8c\u5168\u53ef\u63a7<\/td>\n\u6309\u9700<\/td>\n\u9ad8<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    \n

    13. \u6570\u636e\u5b58\u50a8\u4e0e\u6e05\u6d17<\/h3>\n

    13.1 \u5b58\u50a8\u65b9\u6848\u5bf9\u6bd4<\/h4>\n\n\u6570\u636e\u5e93\u7c7b\u578b\u5199\u5165\u901f\u5ea6\u67e5\u8be2\u80fd\u529b\u9002\u7528\u573a\u666f\u63a8\u8350\u89c4\u6a21<\/tr>\n\n\n\n\n\n\n\n
    MongoDB<\/td>\n\u6587\u6863\u578b<\/td>\n\u5feb<\/td>\n\u7075\u6d3b<\/td>\n\u534a\u7ed3\u6784\u5316\u6570\u636e<\/td>\n\u767e\u4e07-\u5343\u4e07<\/td>\n<\/tr>\n
    MySQL<\/td>\n\u5173\u7cfb\u578b<\/td>\n\u4e2d<\/td>\nSQL\u5f3a\u5927<\/td>\n\u7ed3\u6784\u5316\u6570\u636e<\/td>\n\u5341\u4e07-\u767e\u4e07<\/td>\n<\/tr>\n
    PostgreSQL<\/td>\n\u5173\u7cfb\u578b<\/td>\n\u4e2d<\/td>\nSQL+JSON<\/td>\n\u590d\u6742\u67e5\u8be2<\/td>\n\u767e\u4e07-\u5343\u4e07<\/td>\n<\/tr>\n
    Elasticsearch<\/td>\n\u641c\u7d22\u5f15\u64ce<\/td>\n\u5feb<\/td>\n\u5168\u6587\u68c0\u7d22<\/td>\n\u65e5\u5fd7\/\u641c\u7d22<\/td>\n\u5343\u4e07-\u4ebf<\/td>\n<\/tr>\n
    CSV\/JSON<\/td>\n\u6587\u4ef6<\/td>\n\u6781\u5feb<\/td>\n\u65e0<\/td>\n\u4e34\u65f6\u5b58\u50a8<\/td>\n\u4e07\u7ea7<\/td>\n<\/tr>\n
    Redis<\/td>\n\u7f13\u5b58<\/td>\n\u6781\u5feb<\/td>\nK-V<\/td>\n\u961f\u5217\/\u7f13\u5b58<\/td>\n\u5185\u5b58\u9650\u5236<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    13.2 MongoDB \u5b58\u50a8\u5b9e\u6218<\/h4>\n

    import<\/span> pymongo
    \nfrom<\/span> datetime import<\/span> datetime<\/p>\n

    class<\/span> MongoStorage<\/span>:<\/span>
    \n """MongoDB\u6570\u636e\u5b58\u50a8\u7ba1\u9053"""<\/span><\/p>\n

    def<\/span> __init__<\/span>(<\/span>self,<\/span> uri:<\/span> str<\/span> =<\/span> "mongodb:\/\/localhost:27017"<\/span>,<\/span>
    \n db_name:<\/span> str<\/span> =<\/span> "crawler_data"<\/span>)<\/span>:<\/span>
    \n self.<\/span>client =<\/span> pymongo.<\/span>MongoClient(<\/span>uri)<\/span>
    \n self.<\/span>db =<\/span> self.<\/span>client[<\/span>db_name]<\/span><\/p>\n

    def<\/span> save_items<\/span>(<\/span>self,<\/span> collection_name:<\/span> str<\/span>,<\/span> items:<\/span> list<\/span>)<\/span>:<\/span>
    \n """\u6279\u91cf\u4fdd\u5b58\u6570\u636e"""<\/span>
    \n collection =<\/span> self.<\/span>db[<\/span>collection_name]<\/span><\/p>\n

    for<\/span> item in<\/span> items:<\/span>
    \n item[<\/span>"crawled_at"<\/span>]<\/span> =<\/span> datetime.<\/span>utcnow(<\/span>)<\/span><\/p>\n

    operations =<\/span> [<\/span>]<\/span>
    \n for<\/span> item in<\/span> items:<\/span>
    \n operations.<\/span>append(<\/span>
    \n pymongo.<\/span>UpdateOne(<\/span>
    \n {<\/span>"url"<\/span>:<\/span> item.<\/span>get(<\/span>"url"<\/span>)<\/span>}<\/span>,<\/span>
    \n {<\/span>"$set"<\/span>:<\/span> item}<\/span>,<\/span>
    \n upsert=<\/span>True<\/span>,<\/span>
    \n )<\/span>
    \n )<\/span><\/p>\n

    if<\/span> operations:<\/span>
    \n result =<\/span> collection.<\/span>bulk_write(<\/span>operations)<\/span>
    \n print<\/span>(<\/span>f"\u63d2\u5165: <\/span>{<\/span>result.<\/span>upserted_count}<\/span><\/span>, \u66f4\u65b0: <\/span>{<\/span>result.<\/span>modified_count}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    def<\/span> create_indexes<\/span>(<\/span>self,<\/span> collection_name:<\/span> str<\/span>)<\/span>:<\/span>
    \n """\u521b\u5efa\u7d22\u5f15\u52a0\u901f\u67e5\u8be2"""<\/span>
    \n collection =<\/span> self.<\/span>db[<\/span>collection_name]<\/span>
    \n collection.<\/span>create_index(<\/span>[<\/span>(<\/span>"url"<\/span>,<\/span> pymongo.<\/span>HASHED)<\/span>]<\/span>)<\/span>
    \n collection.<\/span>create_index(<\/span>[<\/span>(<\/span>"crawled_at"<\/span>,<\/span> pymongo.<\/span>DESCENDING)<\/span>]<\/span>)<\/span>
    \n collection.<\/span>create_index(<\/span>[<\/span>(<\/span>"title"<\/span>,<\/span> pymongo.<\/span>TEXT)<\/span>]<\/span>)<\/span><\/p>\n

    13.3 \u6570\u636e\u6e05\u6d17 Pipeline<\/h4>\n

    import<\/span> pandas as<\/span> pd
    \nimport<\/span> re
    \nfrom<\/span> typing import<\/span> List,<\/span> Dict<\/p>\n

    class<\/span> DataCleaner<\/span>:<\/span>
    \n """\u6570\u636e\u6e05\u6d17\u7ba1\u9053"""<\/span><\/p>\n

    @staticmethod<\/span>
    \n def<\/span> clean_price<\/span>(<\/span>price_str:<\/span> str<\/span>)<\/span> –<\/span>><\/span> float<\/span>:<\/span>
    \n """\u6e05\u6d17\u4ef7\u683c\u5b57\u6bb5"""<\/span>
    \n if<\/span> not<\/span> price_str:<\/span>
    \n return<\/span> 0.0<\/span>
    \n cleaned =<\/span> re.<\/span>sub(<\/span>r'[^0-9.]'<\/span>,<\/span> ''<\/span>,<\/span> price_str)<\/span>
    \n try<\/span>:<\/span>
    \n return<\/span> float<\/span>(<\/span>cleaned)<\/span>
    \n except<\/span> ValueError:<\/span>
    \n return<\/span> 0.0<\/span><\/p>\n

    @staticmethod<\/span>
    \n def<\/span> clean_text<\/span>(<\/span>text:<\/span> str<\/span>)<\/span> –<\/span>><\/span> str<\/span>:<\/span>
    \n """\u6e05\u6d17\u6587\u672c\u5b57\u6bb5"""<\/span>
    \n if<\/span> not<\/span> text:<\/span>
    \n return<\/span> ""<\/span>
    \n text =<\/span> re.<\/span>sub(<\/span>r'\\\\s+'<\/span>,<\/span> ' '<\/span>,<\/span> text)<\/span>.<\/span>strip(<\/span>)<\/span>
    \n text =<\/span> re.<\/span>sub(<\/span>r'[\\\\x00-\\\\x08\\\\x0b\\\\x0c\\\\x0e-\\\\x1f\\\\x7f]'<\/span>,<\/span> ''<\/span>,<\/span> text)<\/span>
    \n return<\/span> text<\/p>\n

    @staticmethod<\/span>
    \n def<\/span> deduplicate<\/span>(<\/span>items:<\/span> List[<\/span>Dict]<\/span>,<\/span> key_field:<\/span> str<\/span>)<\/span> –<\/span>><\/span> List[<\/span>Dict]<\/span>:<\/span>
    \n """\u57fa\u4e8e\u5173\u952e\u5b57\u6bb5\u53bb\u91cd"""<\/span>
    \n seen =<\/span> set<\/span>(<\/span>)<\/span>
    \n unique_items =<\/span> [<\/span>]<\/span>
    \n for<\/span> item in<\/span> items:<\/span>
    \n key =<\/span> item.<\/span>get(<\/span>key_field)<\/span>
    \n if<\/span> key and<\/span> key not<\/span> in<\/span> seen:<\/span>
    \n seen.<\/span>add(<\/span>key)<\/span>
    \n unique_items.<\/span>append(<\/span>item)<\/span>
    \n return<\/span> unique_items<\/p>\n

    @staticmethod<\/span>
    \n def<\/span> to_dataframe<\/span>(<\/span>items:<\/span> List[<\/span>Dict]<\/span>)<\/span> –<\/span>><\/span> pd.<\/span>DataFrame:<\/span>
    \n """\u8f6c\u6362\u4e3aDataFrame\u8fdb\u884c\u5206\u6790"""<\/span>
    \n df =<\/span> pd.<\/span>DataFrame(<\/span>items)<\/span>
    \n if<\/span> 'price'<\/span> in<\/span> df.<\/span>columns:<\/span>
    \n df[<\/span>'price'<\/span>]<\/span> =<\/span> pd.<\/span>to_numeric(<\/span>df[<\/span>'price'<\/span>]<\/span>,<\/span> errors=<\/span>'coerce'<\/span>)<\/span>
    \n df.<\/span>dropna(<\/span>how=<\/span>'all'<\/span>,<\/span> inplace=<\/span>True<\/span>)<\/span>
    \n return<\/span> df<\/p>\n

    @staticmethod<\/span>
    \n def<\/span> export_csv<\/span>(<\/span>df:<\/span> pd.<\/span>DataFrame,<\/span> filepath:<\/span> str<\/span>)<\/span>:<\/span>
    \n """\u5bfc\u51fa\u4e3aCSV"""<\/span>
    \n df.<\/span>to_csv(<\/span>filepath,<\/span> index=<\/span>False<\/span>,<\/span> encoding=<\/span>'utf-8-sig'<\/span>)<\/span><\/p>\n

    \u751f\u4ea7\u8e29\u5751\u63d0\u9192：MongoDB \u7684 update_one + upsert \u6a21\u5f0f\u5728\u9ad8\u9891\u5199\u5165\u65f6\u53ef\u80fd\u4ea7\u751f\u7ade\u6001\u6761\u4ef6\u3002\u751f\u4ea7\u73af\u5883\u5efa\u8bae\u4f7f\u7528 bulk_write \u6279\u91cf\u64cd\u4f5c，\u5e76\u5728 url \u5b57\u6bb5\u4e0a\u521b\u5efa\u552f\u4e00\u7d22\u5f15\u3002<\/p>\n

    \n

    14. \u6cd5\u5f8b\u5408\u89c4\u4e0erobots\u534f\u8bae<\/h3>\n

    14.1 \u4e2d\u56fd\u76f8\u5173\u6cd5\u5f8b\u6cd5\u89c4<\/h4>\n\n\u6cd5\u5f8b\u6cd5\u89c4\u751f\u6548\u65f6\u95f4\u4e0e\u722c\u866b\u76f8\u5173\u7684\u6838\u5fc3\u6761\u6b3e<\/tr>\n\n\n\n\n\n\n
    \u300a\u7f51\u7edc\u5b89\u5168\u6cd5\u300b<\/td>\n2017.06<\/td>\n\u7b2c27\u6761：\u7981\u6b62\u4ece\u4e8b\u975e\u6cd5\u4fb5\u5165\u7f51\u7edc\u3001\u5e72\u6270\u7f51\u7edc\u6b63\u5e38\u529f\u80fd<\/td>\n<\/tr>\n
    \u300a\u6570\u636e\u5b89\u5168\u6cd5\u300b<\/td>\n2021.09<\/td>\n\u7b2c32\u6761：\u4efb\u4f55\u7ec4\u7ec7\u548c\u4e2a\u4eba\u6536\u96c6\u6570\u636e\u5e94\u5f53\u5408\u6cd5\u6b63\u5f53<\/td>\n<\/tr>\n
    \u300a\u4e2a\u4eba\u4fe1\u606f\u4fdd\u62a4\u6cd5\u300b<\/td>\n2021.11<\/td>\n\u7b2c13\u6761：\u5904\u7406\u4e2a\u4eba\u4fe1\u606f\u9700\u53d6\u5f97\u540c\u610f\u6216\u6709\u6cd5\u5b9a\u4e8b\u7531<\/td>\n<\/tr>\n
    \u300a\u53cd\u4e0d\u6b63\u5f53\u7ade\u4e89\u6cd5\u300b<\/td>\n2019.04<\/td>\n\u7b2c12\u6761：\u7981\u6b62\u5229\u7528\u6280\u672f\u624b\u6bb5\u59a8\u788d\u7f51\u7edc\u4ea7\u54c1\u6b63\u5e38\u8fd0\u884c<\/td>\n<\/tr>\n
    \u300a\u5211\u6cd5\u300b\u7b2c285\u6761<\/td>\n–<\/td>\n\u975e\u6cd5\u83b7\u53d6\u8ba1\u7b97\u673a\u4fe1\u606f\u7cfb\u7edf\u6570\u636e\u7f6a（\u6700\u9ad87\u5e74）<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    14.2 robots.txt \u89c4\u8303<\/h4>\n

    from<\/span> urllib.<\/span>robotparser import<\/span> RobotFileParser<\/p>\n

    def<\/span> check_robots_txt<\/span>(<\/span>base_url:<\/span> str<\/span>,<\/span> user_agent:<\/span> str<\/span> =<\/span> "*"<\/span>,<\/span> path:<\/span> str<\/span> =<\/span> "\/"<\/span>)<\/span> –<\/span>><\/span> bool<\/span>:<\/span>
    \n """\u68c0\u67e5robots.txt\u662f\u5426\u5141\u8bb8\u722c\u53d6"""<\/span>
    \n rp =<\/span> RobotFileParser(<\/span>)<\/span>
    \n robots_url =<\/span> f"<\/span>{<\/span>base_url}<\/span><\/span>\/robots.txt"<\/span><\/span>
    \n rp.<\/span>set_url(<\/span>robots_url)<\/span><\/p>\n

    try<\/span>:<\/span>
    \n rp.<\/span>read(<\/span>)<\/span>
    \n can_fetch =<\/span> rp.<\/span>can_fetch(<\/span>user_agent,<\/span> path)<\/span>
    \n crawl_delay =<\/span> rp.<\/span>crawl_delay(<\/span>user_agent)<\/span><\/p>\n

    print<\/span>(<\/span>f"URL: <\/span>{<\/span>path}<\/span><\/span>"<\/span><\/span>)<\/span>
    \n print<\/span>(<\/span>f"\u5141\u8bb8\u722c\u53d6: <\/span>{<\/span>can_fetch}<\/span><\/span>"<\/span><\/span>)<\/span>
    \n print<\/span>(<\/span>f"\u8bf7\u6c42\u95f4\u9694: <\/span>{<\/span>crawl_delay}<\/span><\/span>\u79d2"<\/span><\/span>)<\/span><\/p>\n

    return<\/span> can_fetch
    \n except<\/span> Exception as<\/span> e:<\/span>
    \n print<\/span>(<\/span>f"\u8bfb\u53d6robots.txt\u5931\u8d25: <\/span>{<\/span>e}<\/span><\/span>"<\/span><\/span>)<\/span>
    \n return<\/span> True<\/span><\/p>\n

    14.3 \u722c\u866b\u5408\u89c4\u8fb9\u754c<\/h4>\n

    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
    \n\u2502 \u722c\u866b\u884c\u4e3a\u5408\u89c4\u5149\u8c31 \u2502
    \n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
    \n\u2502 \u2502
    \n\u2502 [\u5408\u89c4\u533a\u57df] [\u8fdd\u6cd5\u533a\u57df] \u2502
    \n\u2502 +————————-+ +————————-+ \u2502
    \n\u2502 | \u9075\u5b88robots.txt | | \u7ed5\u8fc7\u767b\u5f55\u8ba4\u8bc1 | \u2502
    \n\u2502 | \u5408\u7406\u8bf7\u6c42\u9891\u7387 | | \u6279\u91cf\u6293\u53d6\u4e2a\u4eba\u4fe1\u606f | \u2502
    \n\u2502 | \u6293\u53d6\u516c\u5f00\u6570\u636e | | \u7834\u89e3\u52a0\u5bc6\u63a5\u53e3 | \u2502
    \n\u2502 | \u6807\u6ce8\u6570\u636e\u6765\u6e90 | | \u5e72\u6270\u7f51\u7ad9\u8fd0\u884c | \u2502
    \n\u2502 | \u7528\u4e8e\u5185\u90e8\u5206\u6790 | | \u5546\u4e1a\u7ade\u4e89\u725f\u5229 | \u2502
    \n\u2502 | \u5b66\u672f\u7814\u7a76\u7528\u9014 | | \u51fa\u552e\u4e2a\u4eba\u4fe1\u606f | \u2502
    \n\u2502 +————————-+ +————————-+ \u2502
    \n\u2502 \u2502
    \n\u2502 [\u7070\u8272\u5730\u5e26 – \u9700\u8c28\u614e\u8bc4\u4f30] \u2502
    \n\u2502 +——————————————————-+ \u2502
    \n\u2502 | \u6293\u53d6\u975e\u516c\u5f00\u4f46\u53ef\u8bbf\u95ee\u7684\u6570\u636e（\u9700\u767b\u5f55\u4f46\u65e0\u6743\u9650\u63a7\u5236） | \u2502
    \n\u2502 | \u5927\u89c4\u6a21\u6293\u53d6\u7528\u4e8e\u5546\u4e1a\u5206\u6790 | \u2502
    \n\u2502 | \u7a81\u7834\u53cd\u722c\u63aa\u65bd（\u6280\u672f\u5bf9\u6297\u672c\u8eab\u53ef\u80fd\u8fdd\u6cd5） | \u2502
    \n\u2502 | \u6293\u53d6\u5185\u5bb9\u6d89\u53ca\u7248\u6743\u4f5c\u54c1 | \u2502
    \n\u2502 +——————————————————-+ \u2502
    \n\u2502 \u2502
    \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

    14.4 \u5178\u578b\u6848\u4f8b\u5206\u6790<\/h4>\n\n\u6848\u4f8b\u5e74\u4efd\u884c\u4e3a\u5224\u51b3\u542f\u793a<\/tr>\n\n\n\n\n
    \u5fae\u8109\u5b9d\u722c\u866b\u6848<\/td>\n2019<\/td>\n\u6279\u91cf\u722c\u53d6\u7528\u6237\u6570\u636e\u5e76\u51fa\u552e<\/td>\n\u6709\u671f\u5f92\u52113\u5e74<\/td>\n\u4e2a\u4eba\u4fe1\u606f\u4e0d\u53ef\u968f\u610f\u6293\u53d6<\/td>\n<\/tr>\n
    \u67d0\u6570\u636e\u516c\u53f8\u6848<\/td>\n2020<\/td>\n\u7834\u89e3\u53cd\u722c\u83b7\u53d6\u7ade\u54c1\u6570\u636e<\/td>\n\u6784\u6210\u4e0d\u6b63\u5f53\u7ade\u4e89<\/td>\n\u5546\u4e1a\u7ade\u4e89\u722c\u866b\u98ce\u9669\u6781\u9ad8<\/td>\n<\/tr>\n
    \u65b0\u6d6a\u8bc9\u8109\u8109\u6848<\/td>\n2019<\/td>\n\u8d85\u6388\u6743\u8303\u56f4\u6293\u53d6\u7528\u6237\u4fe1\u606f<\/td>\n\u8d54\u507f+\u505c\u6b62\u4fb5\u6743<\/td>\n\u5f00\u653e\u5e73\u53f0\u4e0d\u7b49\u4e8e\u65e0\u9650\u6293\u53d6<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    \u751f\u4ea7\u8e29\u5751\u63d0\u9192：2025\u5e74\u6267\u6cd5\u8d8b\u52bf\u660e\u663e\u6536\u7d27\u3002\u5373\u4f7f\u6293\u53d6\u7684\u662f"\u516c\u5f00"\u6570\u636e，\u5982\u679c\u6d89\u53ca\u4e2a\u4eba\u4fe1\u606f（\u624b\u673a\u53f7\u3001\u8eab\u4efd\u8bc1\u7b49），\u5373\u4f7f\u7f51\u7ad9\u672a\u505a\u8bbf\u95ee\u9650\u5236，\u4e5f\u53ef\u80fd\u89e6\u72af\u300a\u4e2a\u4eba\u4fe1\u606f\u4fdd\u62a4\u6cd5\u300b\u3002\u5efa\u8bae\u5728\u722c\u866b\u9879\u76ee\u4e2d\u5f15\u5165\u6cd5\u52a1\u5ba1\u6838\u73af\u8282\u3002<\/p>\n

    \n

    15. \u9a8c\u8bc1\u7801\u4e13\u9879\u7a81\u7834<\/h3>\n

    15.1 \u9a8c\u8bc1\u7801\u7c7b\u578b\u4e0e\u8bc6\u522b\u65b9\u6848<\/h4>\n\n\u9a8c\u8bc1\u7801\u7c7b\u578b\u539f\u7406\u96be\u5ea6\u63a8\u8350\u65b9\u6848\u6210\u529f\u7387<\/tr>\n\n\n\n\n\n\n\n\n
    \u6570\u5b57\u5b57\u6bcd\u9a8c\u8bc1\u7801<\/td>\n\u56fe\u7247+\u566a\u70b9<\/td>\n\u4e2d<\/td>\nddddocr\/Tesseract<\/td>\n85-95%<\/td>\n<\/tr>\n
    \u6ed1\u5757\u9a8c\u8bc1\u7801<\/td>\n\u62fc\u56fe\u7f3a\u53e3\u5339\u914d<\/td>\n\u9ad8<\/td>\nOpenCV\u6a21\u677f\u5339\u914d<\/td>\n90-98%<\/td>\n<\/tr>\n
    \u70b9\u9009\u6587\u5b57\u9a8c\u8bc1\u7801<\/td>\nOCR+\u5750\u6807\u5b9a\u4f4d<\/td>\n\u5f88\u9ad8<\/td>\nddddocr+\u5750\u6807\u8ba1\u7b97<\/td>\n70-85%<\/td>\n<\/tr>\n
    \u70b9\u9009\u56fe\u6807\u9a8c\u8bc1\u7801<\/td>\n\u56fe\u50cf\u8bc6\u522b+\u5b9a\u4f4d<\/td>\n\u5f88\u9ad8<\/td>\nCNN\u6a21\u578b<\/td>\n60-80%<\/td>\n<\/tr>\n
    \u884c\u4e3a\u9a8c\u8bc1\u7801（\u6781\u9a8c）<\/td>\n\u9f20\u6807\u8f68\u8ff9\u5206\u6790<\/td>\n\u6781\u9ad8<\/td>\n\u4e13\u4e1a\u6253\u7801\u5e73\u53f0<\/td>\n90%\u4ee5\u4e0a<\/td>\n<\/tr>\n
    reCAPTCHA v3<\/td>\n\u884c\u4e3a\u8bc4\u5206<\/td>\n\u6781\u9ad8<\/td>\n\u771f\u4eba\u6253\u7801\/\u517bCookie<\/td>\n80%\u4ee5\u4e0a<\/td>\n<\/tr>\n
    hCaptcha<\/td>\n\u56fe\u50cf\u5206\u7c7b<\/td>\n\u5f88\u9ad8<\/td>\n\u6253\u7801\u5e73\u53f0<\/td>\n75-90%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    15.2 \u6ed1\u5757\u9a8c\u8bc1\u7801\u8bc6\u522b<\/h4>\n

    import<\/span> cv2
    \nimport<\/span> numpy as<\/span> np
    \nimport<\/span> requests
    \nimport<\/span> random
    \nimport<\/span> time<\/p>\n

    def<\/span> find_slider_gap<\/span>(<\/span>bg_bytes:<\/span> bytes<\/span>,<\/span> piece_bytes:<\/span> bytes<\/span>)<\/span> –<\/span>><\/span> int<\/span>:<\/span>
    \n """
    \n \u4f7f\u7528OpenCV\u6a21\u677f\u5339\u914d\u627e\u5230\u6ed1\u5757\u7f3a\u53e3\u4f4d\u7f6e
    \n """<\/span>
    \n bg_array =<\/span> np.<\/span>asarray(<\/span>bytearray<\/span>(<\/span>bg_bytes)<\/span>,<\/span> dtype=<\/span>np.<\/span>uint8)<\/span>
    \n piece_array =<\/span> np.<\/span>asarray(<\/span>bytearray<\/span>(<\/span>piece_bytes)<\/span>,<\/span> dtype=<\/span>np.<\/span>uint8)<\/span><\/p>\n

    bg_img =<\/span> cv2.<\/span>imdecode(<\/span>bg_array,<\/span> cv2.<\/span>IMREAD_COLOR)<\/span>
    \n piece_img =<\/span> cv2.<\/span>imdecode(<\/span>piece_array,<\/span> cv2.<\/span>IMREAD_COLOR)<\/span><\/p>\n

    bg_gray =<\/span> cv2.<\/span>cvtColor(<\/span>bg_img,<\/span> cv2.<\/span>COLOR_BGR2GRAY)<\/span>
    \n piece_gray =<\/span> cv2.<\/span>cvtColor(<\/span>piece_img,<\/span> cv2.<\/span>COLOR_BGR2GRAY)<\/span><\/p>\n

    bg_edge =<\/span> cv2.<\/span>Canny(<\/span>bg_gray,<\/span> 100<\/span>,<\/span> 200<\/span>)<\/span>
    \n piece_edge =<\/span> cv2.<\/span>Canny(<\/span>piece_gray,<\/span> 100<\/span>,<\/span> 200<\/span>)<\/span><\/p>\n

    result =<\/span> cv2.<\/span>matchTemplate(<\/span>bg_edge,<\/span> piece_edge,<\/span> cv2.<\/span>TM_CCOEFF_NORMED)<\/span>
    \n _,<\/span> max_val,<\/span> _,<\/span> max_loc =<\/span> cv2.<\/span>minMaxLoc(<\/span>result)<\/span><\/p>\n

    gap_x =<\/span> max_loc[<\/span>0<\/span>]<\/span>
    \n print<\/span>(<\/span>f"\u7f3a\u53e3\u4f4d\u7f6e: x=<\/span>{<\/span>gap_x}<\/span><\/span>, \u5339\u914d\u7f6e\u4fe1\u5ea6: <\/span>{<\/span>max_val:<\/span>.3f<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span>
    \n return<\/span> gap_x<\/p>\n

    def<\/span> generate_track<\/span>(<\/span>distance:<\/span> int<\/span>)<\/span> –<\/span>><\/span> list<\/span>:<\/span>
    \n """
    \n \u751f\u6210\u6a21\u62df\u4eba\u7c7b\u62d6\u62fd\u7684\u8f68\u8ff9（\u52a0\u901f -> \u51cf\u901f -> \u5fae\u8c03）
    \n """<\/span>
    \n track =<\/span> [<\/span>]<\/span>
    \n current =<\/span> 0<\/span>
    \n mid =<\/span> distance *<\/span> 4<\/span> \/<\/span> 5<\/span>
    \n t =<\/span> 0.2<\/span>
    \n v =<\/span> 0<\/span><\/p>\n

    while<\/span> current <<\/span> distance:<\/span>
    \n if<\/span> current <<\/span> mid:<\/span>
    \n a =<\/span> 2<\/span> +<\/span> random.<\/span>uniform(<\/span>–<\/span>0.5<\/span>,<\/span> 0.5<\/span>)<\/span>
    \n else<\/span>:<\/span>
    \n a =<\/span> –<\/span>3<\/span> +<\/span> random.<\/span>uniform(<\/span>–<\/span>0.5<\/span>,<\/span> 0.5<\/span>)<\/span><\/p>\n

    v0 =<\/span> v
    \n v =<\/span> v0 +<\/span> a *<\/span> t
    \n move =<\/span> v0 *<\/span> t +<\/span> 0.5<\/span> *<\/span> a *<\/span> t *<\/span> t<\/p>\n

    current +=<\/span> move
    \n track.<\/span>append(<\/span>round<\/span>(<\/span>move)<\/span>)<\/span><\/p>\n

    # \u56de\u9000\u5fae\u8c03（\u6a21\u62df\u4eba\u7c7b\u884c\u4e3a）<\/span>
    \n track.<\/span>append(<\/span>–<\/span>random.<\/span>randint(<\/span>1<\/span>,<\/span> 3<\/span>)<\/span>)<\/span>
    \n track.<\/span>append(<\/span>random.<\/span>randint(<\/span>1<\/span>,<\/span> 2<\/span>)<\/span>)<\/span><\/p>\n

    return<\/span> track<\/p>\n

    15.3 \u6253\u7801\u5e73\u53f0\u96c6\u6210<\/h4>\n

    import<\/span> requests
    \nimport<\/span> base64
    \nimport<\/span> time<\/p>\n

    class<\/span> CaptchaSolver<\/span>:<\/span>
    \n """\u9a8c\u8bc1\u7801\u8bc6\u522b\u670d\u52a1 – \u652f\u6301\u672c\u5730OCR\u548c\u6253\u7801\u5e73\u53f0"""<\/span><\/p>\n

    def<\/span> __init__<\/span>(<\/span>self,<\/span> api_key:<\/span> str<\/span> =<\/span> None<\/span>,<\/span> api_user:<\/span> str<\/span> =<\/span> None<\/span>)<\/span>:<\/span>
    \n self.<\/span>api_key =<\/span> api_key
    \n self.<\/span>api_user =<\/span> api_user<\/p>\n

    def<\/span> solve_local<\/span>(<\/span>self,<\/span> image_bytes:<\/span> bytes<\/span>)<\/span> –<\/span>><\/span> str<\/span>:<\/span>
    \n """\u672c\u5730ddddocr\u8bc6\u522b"""<\/span>
    \n import<\/span> ddddocr
    \n ocr =<\/span> ddddocr.<\/span>DdddOcr(<\/span>show_ad=<\/span>False<\/span>)<\/span>
    \n return<\/span> ocr.<\/span>classification(<\/span>image_bytes)<\/span><\/p>\n

    def<\/span> solve_remote<\/span>(<\/span>self,<\/span> image_bytes:<\/span> bytes<\/span>,<\/span> captcha_type:<\/span> str<\/span> =<\/span> "1901"<\/span>)<\/span> –<\/span>><\/span> str<\/span>:<\/span>
    \n """\u8fdc\u7a0b\u6253\u7801\u5e73\u53f0\u8bc6\u522b"""<\/span>
    \n if<\/span> not<\/span> self.<\/span>api_key:<\/span>
    \n raise<\/span> ValueError(<\/span>"\u672a\u914d\u7f6e\u6253\u7801\u5e73\u53f0API Key"<\/span>)<\/span><\/p>\n

    img_base64 =<\/span> base64.<\/span>b64encode(<\/span>image_bytes)<\/span>.<\/span>decode(<\/span>)<\/span><\/p>\n

    data =<\/span> {<\/span>
    \n "user"<\/span>:<\/span> self.<\/span>api_user,<\/span>
    \n "pass2"<\/span>:<\/span> self.<\/span>api_key,<\/span>
    \n "softid"<\/span>:<\/span> "your_software_id"<\/span>,<\/span>
    \n "codetype"<\/span>:<\/span> captcha_type,<\/span>
    \n "file_base64"<\/span>:<\/span> img_base64,<\/span>
    \n }<\/span><\/p>\n

    response =<\/span> requests.<\/span>post(<\/span>
    \n "http:\/\/upload.chaojiying.net\/Upload\/Processing.php"<\/span>,<\/span>
    \n data=<\/span>data,<\/span>
    \n )<\/span>
    \n result =<\/span> response.<\/span>json(<\/span>)<\/span><\/p>\n

    if<\/span> result.<\/span>get(<\/span>"err_no"<\/span>)<\/span> ==<\/span> 0<\/span>:<\/span>
    \n return<\/span> result.<\/span>get(<\/span>"pic_str"<\/span>)<\/span>
    \n else<\/span>:<\/span>
    \n raise<\/span> Exception(<\/span>f"\u8bc6\u522b\u5931\u8d25: <\/span>{<\/span>result}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    \n

    16. \u9ad8\u5e76\u53d1\u4e0e\u6027\u80fd\u4f18\u5316<\/h3>\n

    16.1 asyncio \u5f02\u6b65\u722c\u866b<\/h4>\n

    import<\/span> asyncio
    \nimport<\/span> aiohttp
    \nimport<\/span> time
    \nfrom<\/span> typing import<\/span> List,<\/span> Dict<\/p>\n

    class<\/span> AsyncCrawler<\/span>:<\/span>
    \n """\u9ad8\u6027\u80fd\u5f02\u6b65\u722c\u866b"""<\/span><\/p>\n

    def<\/span> __init__<\/span>(<\/span>self,<\/span> concurrency:<\/span> int<\/span> =<\/span> 50<\/span>,<\/span> timeout:<\/span> int<\/span> =<\/span> 30<\/span>)<\/span>:<\/span>
    \n self.<\/span>concurrency =<\/span> concurrency
    \n self.<\/span>timeout =<\/span> timeout
    \n self.<\/span>semaphore =<\/span> asyncio.<\/span>Semaphore(<\/span>concurrency)<\/span>
    \n self.<\/span>results:<\/span> List[<\/span>Dict]<\/span> =<\/span> [<\/span>]<\/span>
    \n self.<\/span>stats =<\/span> {<\/span>"success"<\/span>:<\/span> 0<\/span>,<\/span> "failed"<\/span>:<\/span> 0<\/span>,<\/span> "total_time"<\/span>:<\/span> 0<\/span>}<\/span><\/p>\n

    async<\/span> def<\/span> fetch<\/span>(<\/span>self,<\/span> session:<\/span> aiohttp.<\/span>ClientSession,<\/span> url:<\/span> str<\/span>)<\/span> –<\/span>><\/span> Dict:<\/span>
    \n """\u5355\u4e2a\u8bf7\u6c42"""<\/span>
    \n async<\/span> with<\/span> self.<\/span>semaphore:<\/span>
    \n try<\/span>:<\/span>
    \n async<\/span> with<\/span> session.<\/span>get(<\/span>
    \n url,<\/span>
    \n timeout=<\/span>aiohttp.<\/span>ClientTimeout(<\/span>total=<\/span>self.<\/span>timeout)<\/span>,<\/span>
    \n )<\/span> as<\/span> response:<\/span>
    \n html =<\/span> await<\/span> response.<\/span>text(<\/span>)<\/span>
    \n self.<\/span>stats[<\/span>"success"<\/span>]<\/span> +=<\/span> 1<\/span>
    \n return<\/span> {<\/span>
    \n "url"<\/span>:<\/span> url,<\/span>
    \n "status"<\/span>:<\/span> response.<\/span>status,<\/span>
    \n "length"<\/span>:<\/span> len<\/span>(<\/span>html)<\/span>,<\/span>
    \n "html"<\/span>:<\/span> html,<\/span>
    \n }<\/span>
    \n except<\/span> Exception as<\/span> e:<\/span>
    \n self.<\/span>stats[<\/span>"failed"<\/span>]<\/span> +=<\/span> 1<\/span>
    \n return<\/span> {<\/span>"url"<\/span>:<\/span> url,<\/span> "error"<\/span>:<\/span> str<\/span>(<\/span>e)<\/span>}<\/span><\/p>\n

    async<\/span> def<\/span> crawl_all<\/span>(<\/span>self,<\/span> urls:<\/span> List[<\/span>str<\/span>]<\/span>)<\/span> –<\/span>><\/span> List[<\/span>Dict]<\/span>:<\/span>
    \n """\u6279\u91cf\u722c\u53d6"""<\/span>
    \n start_time =<\/span> time.<\/span>time(<\/span>)<\/span><\/p>\n

    connector =<\/span> aiohttp.<\/span>TCPConnector(<\/span>
    \n limit=<\/span>self.<\/span>concurrency,<\/span>
    \n limit_per_host=<\/span>10<\/span>,<\/span>
    \n ttl_dns_cache=<\/span>300<\/span>,<\/span>
    \n enable_cleanup_closed=<\/span>True<\/span>,<\/span>
    \n )<\/span><\/p>\n

    async<\/span> with<\/span> aiohttp.<\/span>ClientSession(<\/span>
    \n connector=<\/span>connector,<\/span>
    \n headers=<\/span>{<\/span>
    \n "User-Agent"<\/span>:<\/span> "Mozilla\/5.0 (Windows NT 10.0; Win64; x64) "<\/span>
    \n "AppleWebKit\/537.36 Chrome\/131.0.0.0 Safari\/537.36"<\/span>,<\/span>
    \n }<\/span>,<\/span>
    \n )<\/span> as<\/span> session:<\/span>
    \n tasks =<\/span> [<\/span>self.<\/span>fetch(<\/span>session,<\/span> url)<\/span> for<\/span> url in<\/span> urls]<\/span>
    \n self.<\/span>results =<\/span> await<\/span> asyncio.<\/span>gather(<\/span>*<\/span>tasks)<\/span><\/p>\n

    self.<\/span>stats[<\/span>"total_time"<\/span>]<\/span> =<\/span> time.<\/span>time(<\/span>)<\/span> –<\/span> start_time
    \n return<\/span> self.<\/span>results<\/p>\n

    # \u6027\u80fd\u5bf9\u6bd4\u6d4b\u8bd5<\/span>
    \nasync<\/span> def<\/span> benchmark<\/span>(<\/span>)<\/span>:<\/span>
    \n urls =<\/span> [<\/span>f"https:\/\/httpbin.org\/get?id=<\/span>{<\/span>i}<\/span><\/span>"<\/span><\/span> for<\/span> i in<\/span> range<\/span>(<\/span>100<\/span>)<\/span>]<\/span><\/p>\n

    crawler =<\/span> AsyncCrawler(<\/span>concurrency=<\/span>50<\/span>)<\/span>
    \n results =<\/span> await<\/span> crawler.<\/span>crawl_all(<\/span>urls)<\/span><\/p>\n

    print<\/span>(<\/span>f"\u5f02\u6b65\u722c\u53d6\u7edf\u8ba1: <\/span>{<\/span>crawler.<\/span>stats}<\/span><\/span>"<\/span><\/span>)<\/span>
    \n # \u9884\u671f: 100\u4e2a\u8bf7\u6c42\u7ea62-3\u79d2\u5b8c\u6210<\/span><\/p>\n

    16.2 \u751f\u4ea7\u73af\u5883\u90e8\u7f72<\/h4>\n

    # docker-compose.yml – \u722c\u866b\u96c6\u7fa4\u90e8\u7f72<\/span>
    \nversion<\/span>:<\/span> '3.8'<\/span><\/p>\n

    services<\/span>:<\/span>
    \n redis<\/span>:<\/span>
    \n image<\/span>:<\/span> redis:<\/span>7–<\/span>alpine
    \n ports<\/span>:<\/span>
    \n –<\/span> "6379:6379"<\/span>
    \n command<\/span>:<\/span> redis–<\/span>server –<\/span>–<\/span>requirepass crawler_pass
    \n volumes<\/span>:<\/span>
    \n –<\/span> redis_data:<\/span>\/data<\/p>\n

    crawler-worker-1<\/span>:<\/span>
    \n build<\/span>:<\/span> .\/crawler
    \n environment<\/span>:<\/span>
    \n –<\/span> REDIS_URL=redis:<\/span>\/\/:<\/span>crawler_pass@redis:<\/span>6379\/0
    \n –<\/span> SPIDER_NAME=books
    \n –<\/span> CONCURRENT_REQUESTS=32
    \n depends_on<\/span>:<\/span>
    \n –<\/span> redis
    \n deploy<\/span>:<\/span>
    \n resources<\/span>:<\/span>
    \n limits<\/span>:<\/span>
    \n cpus<\/span>:<\/span> '2'<\/span>
    \n memory<\/span>:<\/span> 2G<\/p>\n

    crawler-worker-2<\/span>:<\/span>
    \n build<\/span>:<\/span> .\/crawler
    \n environment<\/span>:<\/span>
    \n –<\/span> REDIS_URL=redis:<\/span>\/\/:<\/span>crawler_pass@redis:<\/span>6379\/0
    \n –<\/span> SPIDER_NAME=books
    \n –<\/span> CONCURRENT_REQUESTS=32
    \n depends_on<\/span>:<\/span>
    \n –<\/span> redis<\/p>\n

    mongo<\/span>:<\/span>
    \n image<\/span>:<\/span> mongo:<\/span>7<\/span>
    \n ports<\/span>:<\/span>
    \n –<\/span> "27017:27017"<\/span>
    \n environment<\/span>:<\/span>
    \n –<\/span> MONGO_INITDB_ROOT_USERNAME=crawler
    \n –<\/span> MONGO_INITDB_ROOT_PASSWORD=mongo_pass_2025
    \n volumes<\/span>:<\/span>
    \n –<\/span> mongo_data:<\/span>\/data\/db<\/p>\n

    proxy-api<\/span>:<\/span>
    \n build<\/span>:<\/span> .\/proxy_pool
    \n ports<\/span>:<\/span>
    \n –<\/span> "8080:8080"<\/span>
    \n depends_on<\/span>:<\/span>
    \n –<\/span> redis<\/p>\n

    volumes<\/span>:<\/span>
    \n redis_data<\/span>:<\/span>
    \n mongo_data<\/span>:<\/span><\/p>\n

    ; supervisord.conf – \u8fdb\u7a0b\u7ba1\u7406\u914d\u7f6e
    \n[program:crawler-worker]
    \ncommand=scrapy crawl %(ENV_SPIDER_NAME)s
    \ndirectory=\/app
    \nuser=crawler
    \nautostart=true
    \nautorestart=true
    \nstartsecs=10
    \nstopwaitsecs=60
    \nredirect_stderr=true
    \nstdout_logfile=\/var\/log\/crawler\/%(program_name)s.log
    \nstdout_logfile_maxbytes=50MB
    \nstdout_logfile_backups=5
    \nenvironment=
    \n PYTHONUNBUFFERED="1",
    \n SCRAPY_SETTINGS_MODULE="crawler.settings"<\/p>\n

    [program:proxy-pool]
    \ncommand=uvicorn proxy_pool:app –host 0.0.0.0 –port 8080
    \ndirectory=\/app
    \nuser=crawler
    \nautostart=true
    \nautorestart=true
    \nredirect_stderr=true
    \nstdout_logfile=\/var\/log\/crawler\/proxy_pool.log<\/p>\n

    \u9762\u8bd5\u52a0\u5206\u63d0\u793a：\u751f\u4ea7\u73af\u5883\u722c\u866b\u90e8\u7f72\u7684\u4e09\u4ef6\u5957\u662f Docker（\u73af\u5883\u9694\u79bb）+ Supervisor（\u8fdb\u7a0b\u5b88\u62a4）+ Prometheus+Grafana（\u76d1\u63a7\u544a\u8b66）\u3002\u5173\u952e\u76d1\u63a7\u6307\u6807\u5305\u62ec：\u8bf7\u6c42\u6210\u529f\u7387\u3001\u54cd\u5e94\u65f6\u95f4P99\u3001\u961f\u5217\u79ef\u538b\u91cf\u3001\u5185\u5b58\u4f7f\u7528\u7387\u3002<\/p>\n

    \n

    17. \u5b9e\u6218\u9879\u76ee<\/h3>\n

    17.1 \u7535\u5546\u5546\u54c1\u6570\u636e\u91c7\u96c6（\u542b\u53cd\u53cd\u722c）<\/h4>\n

    """
    \n\u7535\u5546\u5546\u54c1\u6570\u636e\u91c7\u96c6\u5b8c\u6574\u793a\u4f8b
    \n\u6280\u672f\u6808: curl_cffi + lxml + MongoDB + \u4ee3\u7406\u6c60
    \n"""<\/span>
    \nimport<\/span> json
    \nimport<\/span> time
    \nimport<\/span> random
    \nfrom<\/span> curl_cffi import<\/span> requests as<\/span> cffi_requests
    \nfrom<\/span> lxml import<\/span> etree
    \nfrom<\/span> urllib.<\/span>parse import<\/span> urljoin
    \nimport<\/span> pymongo<\/p>\n

    class<\/span> EcommerceCrawler<\/span>:<\/span>
    \n """\u7535\u5546\u722c\u866b – \u5e26\u5b8c\u6574\u53cd\u53cd\u722c\u7b56\u7565"""<\/span><\/p>\n

    def<\/span> __init__<\/span>(<\/span>self)<\/span>:<\/span>
    \n # 1. \u521d\u59cb\u5316Session（TLS\u6307\u7eb9\u4f2a\u88c5）<\/span>
    \n self.<\/span>session =<\/span> cffi_requests.<\/span>Session(<\/span>
    \n impersonate=<\/span>"chrome136"<\/span>,<\/span>
    \n verify=<\/span>True<\/span>,<\/span>
    \n )<\/span><\/p>\n

    # 2. MongoDB\u5b58\u50a8<\/span>
    \n self.<\/span>mongo =<\/span> pymongo.<\/span>MongoClient(<\/span>"mongodb:\/\/localhost:27017"<\/span>)<\/span>
    \n self.<\/span>db =<\/span> self.<\/span>mongo[<\/span>"ecommerce"<\/span>]<\/span>
    \n self.<\/span>collection =<\/span> self.<\/span>db[<\/span>"products"<\/span>]<\/span><\/p>\n

    # 3. \u4ee3\u7406\u6c60<\/span>
    \n self.<\/span>proxies =<\/span> [<\/span>
    \n "http:\/\/proxy1.example.com:8080"<\/span>,<\/span>
    \n "http:\/\/proxy2.example.com:8080"<\/span>,<\/span>
    \n ]<\/span><\/p>\n

    def<\/span> get_headers<\/span>(<\/span>self)<\/span> –<\/span>><\/span> dict<\/span>:<\/span>
    \n """\u751f\u6210\u968f\u673aHeaders"""<\/span>
    \n return<\/span> {<\/span>
    \n "User-Agent"<\/span>:<\/span> "Mozilla\/5.0 (Windows NT 10.0; Win64; x64) "<\/span>
    \n "AppleWebKit\/537.36 Chrome\/131.0.0.0 Safari\/537.36"<\/span>,<\/span>
    \n "Accept"<\/span>:<\/span> "text\/html,application\/xhtml+xml,*\/*;q=0.8"<\/span>,<\/span>
    \n "Accept-Language"<\/span>:<\/span> "zh-CN,zh;q=0.9,en;q=0.8"<\/span>,<\/span>
    \n "Referer"<\/span>:<\/span> "https:\/\/www.example.com\/"<\/span>,<\/span>
    \n }<\/span><\/p>\n

    def<\/span> fetch_page<\/span>(<\/span>self,<\/span> url:<\/span> str<\/span>,<\/span> max_retries:<\/span> int<\/span> =<\/span> 3<\/span>)<\/span> –<\/span>><\/span> str<\/span>:<\/span>
    \n """\u5e26\u91cd\u8bd5\u7684\u9875\u9762\u83b7\u53d6"""<\/span>
    \n for<\/span> attempt in<\/span> range<\/span>(<\/span>max_retries)<\/span>:<\/span>
    \n try<\/span>:<\/span>
    \n proxy =<\/span> random.<\/span>choice(<\/span>self.<\/span>proxies)<\/span> if<\/span> self.<\/span>proxies else<\/span> None<\/span>
    \n response =<\/span> self.<\/span>session.<\/span>get(<\/span>
    \n url,<\/span>
    \n headers=<\/span>self.<\/span>get_headers(<\/span>)<\/span>,<\/span>
    \n proxies=<\/span>{<\/span>"http"<\/span>:<\/span> proxy,<\/span> "https"<\/span>:<\/span> proxy}<\/span> if<\/span> proxy else<\/span> None<\/span>,<\/span>
    \n timeout=<\/span>30<\/span>,<\/span>
    \n )<\/span><\/p>\n

    if<\/span> response.<\/span>status_code ==<\/span> 200<\/span>:<\/span>
    \n return<\/span> response.<\/span>text
    \n elif<\/span> response.<\/span>status_code ==<\/span> 429<\/span>:<\/span>
    \n wait_time =<\/span> (<\/span>attempt +<\/span> 1<\/span>)<\/span> *<\/span> 30<\/span>
    \n print<\/span>(<\/span>f"\u88ab\u9650\u6d41，\u7b49\u5f85<\/span>{<\/span>wait_time}<\/span><\/span>\u79d2…"<\/span><\/span>)<\/span>
    \n time.<\/span>sleep(<\/span>wait_time)<\/span><\/p>\n

    except<\/span> Exception as<\/span> e:<\/span>
    \n print<\/span>(<\/span>f"\u8bf7\u6c42\u5931\u8d25(\u7b2c<\/span>{<\/span>attempt+<\/span>1<\/span>}<\/span><\/span>\u6b21): <\/span>{<\/span>e}<\/span><\/span>"<\/span><\/span>)<\/span>
    \n time.<\/span>sleep(<\/span>5<\/span>)<\/span><\/p>\n

    return<\/span> None<\/span><\/p>\n

    def<\/span> parse_product<\/span>(<\/span>self,<\/span> html:<\/span> str<\/span>,<\/span> url:<\/span> str<\/span>)<\/span> –<\/span>><\/span> dict<\/span>:<\/span>
    \n """\u89e3\u6790\u5546\u54c1\u9875\u9762"""<\/span>
    \n tree =<\/span> etree.<\/span>HTML(<\/span>html)<\/span><\/p>\n

    product =<\/span> {<\/span>
    \n "url"<\/span>:<\/span> url,<\/span>
    \n "title"<\/span>:<\/span> self.<\/span>_safe_text(<\/span>tree,<\/span> '\/\/h1[@class="title"]\/text()'<\/span>)<\/span>,<\/span>
    \n "price"<\/span>:<\/span> self.<\/span>_safe_text(<\/span>tree,<\/span> '\/\/span[@class="price"]\/text()'<\/span>)<\/span>,<\/span>
    \n "sales"<\/span>:<\/span> self.<\/span>_safe_text(<\/span>tree,<\/span> '\/\/span[@class="sales-count"]\/text()'<\/span>)<\/span>,<\/span>
    \n "shop_name"<\/span>:<\/span> self.<\/span>_safe_text(<\/span>tree,<\/span> '\/\/a[@class="shop-name"]\/text()'<\/span>)<\/span>,<\/span>
    \n "crawled_at"<\/span>:<\/span> time.<\/span>strftime(<\/span>"%Y-%m-%d %H:%M:%S"<\/span>)<\/span>,<\/span>
    \n }<\/span><\/p>\n

    return<\/span> {<\/span>k:<\/span> v for<\/span> k,<\/span> v in<\/span> product.<\/span>items(<\/span>)<\/span> if<\/span> v}<\/span><\/p>\n

    @staticmethod<\/span>
    \n def<\/span> _safe_text<\/span>(<\/span>tree,<\/span> xpath:<\/span> str<\/span>)<\/span> –<\/span>><\/span> str<\/span>:<\/span>
    \n result =<\/span> tree.<\/span>xpath(<\/span>xpath)<\/span>
    \n return<\/span> result[<\/span>0<\/span>]<\/span>.<\/span>strip(<\/span>)<\/span> if<\/span> result else<\/span> ""<\/span><\/p>\n

    def<\/span> save_product<\/span>(<\/span>self,<\/span> product:<\/span> dict<\/span>)<\/span>:<\/span>
    \n """\u4fdd\u5b58\u5546\u54c1\u6570\u636e（\u53bb\u91cd\u66f4\u65b0）"""<\/span>
    \n self.<\/span>collection.<\/span>update_one(<\/span>
    \n {<\/span>"url"<\/span>:<\/span> product[<\/span>"url"<\/span>]<\/span>}<\/span>,<\/span>
    \n {<\/span>"$set"<\/span>:<\/span> product}<\/span>,<\/span>
    \n upsert=<\/span>True<\/span>,<\/span>
    \n )<\/span><\/p>\n

    # \u4f7f\u7528\u793a\u4f8b<\/span>
    \n# crawler = EcommerceCrawler()<\/span><\/p>\n

    17.2 \u65b0\u95fb\u805a\u5408\u722c\u866b<\/h4>\n

    import<\/span> asyncio
    \nimport<\/span> httpx
    \nfrom<\/span> lxml import<\/span> etree
    \nfrom<\/span> datetime import<\/span> datetime
    \nfrom<\/span> typing import<\/span> List,<\/span> Dict<\/p>\n

    class<\/span> NewsAggregator<\/span>:<\/span>
    \n """\u591a\u6e90\u65b0\u95fb\u805a\u5408\u722c\u866b"""<\/span><\/p>\n

    SOURCES =<\/span> {<\/span>
    \n "\u6f8e\u6e43\u65b0\u95fb"<\/span>:<\/span> {<\/span>
    \n "url"<\/span>:<\/span> "https:\/\/www.thepaper.cn\/"<\/span>,<\/span>
    \n "title_xpath"<\/span>:<\/span> '\/\/h2\/a\/text()'<\/span>,<\/span>
    \n "link_xpath"<\/span>:<\/span> '\/\/h2\/a\/@href'<\/span>,<\/span>
    \n }<\/span>,<\/span>
    \n "36\u6c2a"<\/span>:<\/span> {<\/span>
    \n "url"<\/span>:<\/span> "https:\/\/36kr.com\/newsflashes"<\/span>,<\/span>
    \n "title_xpath"<\/span>:<\/span> '\/\/a[@class="item-title"]\/text()'<\/span>,<\/span>
    \n "link_xpath"<\/span>:<\/span> '\/\/a[@class="item-title"]\/@href'<\/span>,<\/span>
    \n }<\/span>,<\/span>
    \n }<\/span><\/p>\n

    def<\/span> __init__<\/span>(<\/span>self)<\/span>:<\/span>
    \n self.<\/span>results:<\/span> List[<\/span>Dict]<\/span> =<\/span> [<\/span>]<\/span><\/p>\n

    async<\/span> def<\/span> crawl_source<\/span>(<\/span>self,<\/span> client:<\/span> httpx.<\/span>AsyncClient,<\/span>
    \n name:<\/span> str<\/span>,<\/span> config:<\/span> dict<\/span>)<\/span> –<\/span>><\/span> List[<\/span>Dict]<\/span>:<\/span>
    \n """\u722c\u53d6\u5355\u4e2a\u65b0\u95fb\u6e90"""<\/span>
    \n try<\/span>:<\/span>
    \n response =<\/span> await<\/span> client.<\/span>get(<\/span>
    \n config[<\/span>"url"<\/span>]<\/span>,<\/span> timeout=<\/span>15<\/span>,<\/span> follow_redirects=<\/span>True<\/span>,<\/span>
    \n )<\/span><\/p>\n

    tree =<\/span> etree.<\/span>HTML(<\/span>response.<\/span>text)<\/span>
    \n items =<\/span> [<\/span>]<\/span><\/p>\n

    titles =<\/span> tree.<\/span>xpath(<\/span>config[<\/span>"title_xpath"<\/span>]<\/span>)<\/span>
    \n links =<\/span> tree.<\/span>xpath(<\/span>config[<\/span>"link_xpath"<\/span>]<\/span>)<\/span><\/p>\n

    for<\/span> title,<\/span> link in<\/span> zip<\/span>(<\/span>titles[<\/span>:<\/span>20<\/span>]<\/span>,<\/span> links[<\/span>:<\/span>20<\/span>]<\/span>)<\/span>:<\/span>
    \n items.<\/span>append(<\/span>{<\/span>
    \n "source"<\/span>:<\/span> name,<\/span>
    \n "title"<\/span>:<\/span> title.<\/span>strip(<\/span>)<\/span>,<\/span>
    \n "url"<\/span>:<\/span> link if<\/span> link.<\/span>startswith(<\/span>"http"<\/span>)<\/span>
    \n else<\/span> f"<\/span>{<\/span>config[<\/span>'url'<\/span>]<\/span>.<\/span>rstrip(<\/span>'\/'<\/span>)<\/span>}<\/span><\/span>{<\/span>link}<\/span><\/span>"<\/span><\/span>,<\/span>
    \n "crawled_at"<\/span>:<\/span> datetime.<\/span>now(<\/span>)<\/span>.<\/span>isoformat(<\/span>)<\/span>,<\/span>
    \n }<\/span>)<\/span><\/p>\n

    print<\/span>(<\/span>f"[<\/span>{<\/span>name}<\/span><\/span>] \u83b7\u53d6\u5230 <\/span>{<\/span>len<\/span>(<\/span>items)<\/span>}<\/span><\/span> \u6761\u65b0\u95fb"<\/span><\/span>)<\/span>
    \n return<\/span> items<\/p>\n

    except<\/span> Exception as<\/span> e:<\/span>
    \n print<\/span>(<\/span>f"[<\/span>{<\/span>name}<\/span><\/span>] \u722c\u53d6\u5931\u8d25: <\/span>{<\/span>e}<\/span><\/span>"<\/span><\/span>)<\/span>
    \n return<\/span> [<\/span>]<\/span><\/p>\n

    async<\/span> def<\/span> crawl_all<\/span>(<\/span>self)<\/span> –<\/span>><\/span> List[<\/span>Dict]<\/span>:<\/span>
    \n """\u5e76\u53d1\u722c\u53d6\u6240\u6709\u6e90"""<\/span>
    \n async<\/span> with<\/span> httpx.<\/span>AsyncClient(<\/span>
    \n headers=<\/span>{<\/span>"User-Agent"<\/span>:<\/span> "Mozilla\/5.0 Chrome\/131.0.0.0"<\/span>}<\/span>,<\/span>
    \n http2=<\/span>True<\/span>,<\/span>
    \n )<\/span> as<\/span> client:<\/span>
    \n tasks =<\/span> [<\/span>
    \n self.<\/span>crawl_source(<\/span>client,<\/span> name,<\/span> config)<\/span>
    \n for<\/span> name,<\/span> config in<\/span> self.<\/span>SOURCES.<\/span>items(<\/span>)<\/span>
    \n ]<\/span>
    \n results =<\/span> await<\/span> asyncio.<\/span>gather(<\/span>*<\/span>tasks)<\/span>
    \n self.<\/span>results =<\/span> [<\/span>item for<\/span> sublist in<\/span> results for<\/span> item in<\/span> sublist]<\/span>
    \n print<\/span>(<\/span>f"\u603b\u8ba1\u83b7\u53d6 <\/span>{<\/span>len<\/span>(<\/span>self.<\/span>results)<\/span>}<\/span><\/span> \u6761\u65b0\u95fb"<\/span><\/span>)<\/span>
    \n return<\/span> self.<\/span>results<\/p>\n

    17.3 \u5b66\u672f\u8bba\u6587\u6279\u91cf\u4e0b\u8f7d<\/h4>\n

    import<\/span> requests
    \nimport<\/span> os
    \nimport<\/span> time
    \nimport<\/span> hashlib
    \nfrom<\/span> lxml import<\/span> etree<\/p>\n

    class<\/span> PaperDownloader<\/span>:<\/span>
    \n """\u5b66\u672f\u8bba\u6587\u4e0b\u8f7d\u5668（\u4ee5ArXiv\u4e3a\u4f8b）"""<\/span><\/p>\n

    BASE_URL =<\/span> "https:\/\/arxiv.org"<\/span><\/p>\n

    def<\/span> __init__<\/span>(<\/span>self,<\/span> download_dir:<\/span> str<\/span> =<\/span> ".\/papers"<\/span>)<\/span>:<\/span>
    \n self.<\/span>download_dir =<\/span> download_dir
    \n os.<\/span>makedirs(<\/span>download_dir,<\/span> exist_ok=<\/span>True<\/span>)<\/span>
    \n self.<\/span>session =<\/span> requests.<\/span>Session(<\/span>)<\/span>
    \n self.<\/span>session.<\/span>headers.<\/span>update(<\/span>{<\/span>
    \n "User-Agent"<\/span>:<\/span> "Mozilla\/5.0 Research Bot\/1.0"<\/span>
    \n }<\/span>)<\/span><\/p>\n

    def<\/span> search_papers<\/span>(<\/span>self,<\/span> query:<\/span> str<\/span>,<\/span> max_results:<\/span> int<\/span> =<\/span> 50<\/span>)<\/span> –<\/span>><\/span> list<\/span>:<\/span>
    \n """\u641c\u7d22\u8bba\u6587"""<\/span>
    \n params =<\/span> {<\/span>"query"<\/span>:<\/span> query,<\/span> "searchtype"<\/span>:<\/span> "all"<\/span>,<\/span> "start"<\/span>:<\/span> 0<\/span>}<\/span>
    \n papers =<\/span> [<\/span>]<\/span><\/p>\n

    while<\/span> len<\/span>(<\/span>papers)<\/span> <<\/span> max_results:<\/span>
    \n response =<\/span> self.<\/span>session.<\/span>get(<\/span>
    \n f"<\/span>{<\/span>self.<\/span>BASE_URL}<\/span><\/span>\/search\/"<\/span><\/span>,<\/span> params=<\/span>params
    \n )<\/span>
    \n tree =<\/span> etree.<\/span>HTML(<\/span>response.<\/span>text)<\/span>
    \n items =<\/span> tree.<\/span>xpath(<\/span>'\/\/li[@class="arxiv-result"]'<\/span>)<\/span><\/p>\n

    for<\/span> item in<\/span> items:<\/span>
    \n title =<\/span> item.<\/span>xpath(<\/span>'.\/\/p[@class="title"]\/text()'<\/span>)<\/span>
    \n pdf_link =<\/span> item.<\/span>xpath(<\/span>'.\/\/a[contains(text(),"PDF")]\/@href'<\/span>)<\/span><\/p>\n

    if<\/span> title and<\/span> pdf_link:<\/span>
    \n papers.<\/span>append(<\/span>{<\/span>
    \n "title"<\/span>:<\/span> title[<\/span>0<\/span>]<\/span>.<\/span>strip(<\/span>)<\/span>,<\/span>
    \n "pdf_url"<\/span>:<\/span> pdf_link[<\/span>0<\/span>]<\/span> if<\/span> pdf_link else<\/span> ""<\/span>,<\/span>
    \n }<\/span>)<\/span><\/p>\n

    next_page =<\/span> tree.<\/span>xpath(<\/span>'\/\/a[@class="next"]\/text()'<\/span>)<\/span>
    \n if<\/span> not<\/span> next_page or<\/span> len<\/span>(<\/span>papers)<\/span> >=<\/span> max_results:<\/span>
    \n break<\/span>
    \n params[<\/span>"start"<\/span>]<\/span> +=<\/span> 25<\/span>
    \n time.<\/span>sleep(<\/span>3<\/span>)<\/span><\/p>\n

    return<\/span> papers[<\/span>:<\/span>max_results]<\/span><\/p>\n

    def<\/span> download_pdf<\/span>(<\/span>self,<\/span> pdf_url:<\/span> str<\/span>)<\/span> –<\/span>><\/span> str<\/span>:<\/span>
    \n """\u4e0b\u8f7dPDF\u6587\u4ef6"""<\/span>
    \n filename =<\/span> hashlib.<\/span>md5(<\/span>pdf_url.<\/span>encode(<\/span>)<\/span>)<\/span>.<\/span>hexdigest(<\/span>)<\/span>[<\/span>:<\/span>12<\/span>]<\/span> +<\/span> ".pdf"<\/span>
    \n filepath =<\/span> os.<\/span>path.<\/span>join(<\/span>self.<\/span>download_dir,<\/span> filename)<\/span><\/p>\n

    if<\/span> os.<\/span>path.<\/span>exists(<\/span>filepath)<\/span>:<\/span>
    \n return<\/span> filepath<\/p>\n

    response =<\/span> self.<\/span>session.<\/span>get(<\/span>pdf_url,<\/span> timeout=<\/span>60<\/span>,<\/span> stream=<\/span>True<\/span>)<\/span>
    \n if<\/span> response.<\/span>status_code ==<\/span> 200<\/span>:<\/span>
    \n with<\/span> open<\/span>(<\/span>filepath,<\/span> "wb"<\/span>)<\/span> as<\/span> f:<\/span>
    \n for<\/span> chunk in<\/span> response.<\/span>iter_content(<\/span>chunk_size=<\/span>8192<\/span>)<\/span>:<\/span>
    \n f.<\/span>write(<\/span>chunk)<\/span>
    \n print<\/span>(<\/span>f"\u5df2\u4e0b\u8f7d: <\/span>{<\/span>filepath}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    return<\/span> filepath<\/p>\n

    def<\/span> batch_download<\/span>(<\/span>self,<\/span> query:<\/span> str<\/span>,<\/span> max_papers:<\/span> int<\/span> =<\/span> 20<\/span>)<\/span>:<\/span>
    \n """\u6279\u91cf\u641c\u7d22\u5e76\u4e0b\u8f7d"""<\/span>
    \n print<\/span>(<\/span>f"\u641c\u7d22\u8bba\u6587: <\/span>{<\/span>query}<\/span><\/span>"<\/span><\/span>)<\/span>
    \n papers =<\/span> self.<\/span>search_papers(<\/span>query,<\/span> max_papers)<\/span><\/p>\n

    print<\/span>(<\/span>f"\u627e\u5230 <\/span>{<\/span>len<\/span>(<\/span>papers)<\/span>}<\/span><\/span> \u7bc7\u8bba\u6587，\u5f00\u59cb\u4e0b\u8f7d…"<\/span><\/span>)<\/span>
    \n for<\/span> i,<\/span> paper in<\/span> enumerate<\/span>(<\/span>papers,<\/span> 1<\/span>)<\/span>:<\/span>
    \n print<\/span>(<\/span>f"[<\/span>{<\/span>i}<\/span><\/span>\/<\/span>{<\/span>len<\/span>(<\/span>papers)<\/span>}<\/span><\/span>] <\/span>{<\/span>paper[<\/span>'title'<\/span>]<\/span>[<\/span>:<\/span>60]<\/span>}<\/span><\/span>…"<\/span><\/span>)<\/span>
    \n if<\/span> paper[<\/span>"pdf_url"<\/span>]<\/span>:<\/span>
    \n self.<\/span>download_pdf(<\/span>paper[<\/span>"pdf_url"<\/span>]<\/span>)<\/span>
    \n time.<\/span>sleep(<\/span>3<\/span>)<\/span><\/p>\n

    18. 2025-2026\u722c\u866b\u65b0\u6280\u672f\u8d8b\u52bf<\/h3>\n

    18.1 AI\u8f85\u52a9\u9006\u5411（LLM\u5206\u6790JS）<\/h4>\n

    2025\u5e74\u51fa\u73b0\u4e86\u4e00\u4e2a\u9769\u547d\u6027\u7684\u8d8b\u52bf：\u4f7f\u7528\u5927\u8bed\u8a00\u6a21\u578b\u8f85\u52a9JS\u9006\u5411\u5206\u6790\u3002<\/p>\n

    """
    \n\u4f7f\u7528LLM\u8f85\u52a9JS\u9006\u5411\u5206\u6790\u7684\u5de5\u4f5c\u6d41
    \n\u5c06\u6df7\u6dc6\u7684JS\u4ee3\u7801\u7247\u6bb5\u53d1\u9001\u7ed9LLM，\u83b7\u53d6\u7b97\u6cd5\u8fd8\u539f\u5efa\u8bae
    \n"""<\/span><\/p>\n

    def<\/span> prepare_js_for_analysis<\/span>(<\/span>js_code:<\/span> str<\/span>,<\/span> max_tokens:<\/span> int<\/span> =<\/span> 4000<\/span>)<\/span> –<\/span>><\/span> str<\/span>:<\/span>
    \n """\u9884\u5904\u7406JS\u4ee3\u7801，\u6784\u5efa\u53d1\u9001\u7ed9LLM\u7684prompt"""<\/span>
    \n prompt =<\/span> f"""
    \n\u4f60\u662f\u4e00\u4e2aJavaScript\u9006\u5411\u5de5\u7a0b\u4e13\u5bb6\u3002\u8bf7\u5206\u6790\u4ee5\u4e0b\u6df7\u6dc6\u7684JS\u4ee3\u7801，
    \n\u8fd8\u539f\u5176\u6838\u5fc3\u7b97\u6cd5\u903b\u8f91，\u5e76\u7ed9\u51faPython\u5b9e\u73b0\u3002<\/p>\n

    \u8981\u6c42：
    \n1. \u8bc6\u522b\u4f7f\u7528\u7684\u52a0\u5bc6\u7b97\u6cd5（MD5\/SHA\/AES\/RSA\u7b49）
    \n2. \u627e\u51fa\u5173\u952e\u53c2\u6570\u548c\u5bc6\u94a5
    \n3. \u7ed9\u51fa\u7b49\u4ef7\u7684Python\u4ee3\u7801<\/p>\n

    \u6df7\u6dc6JS\u4ee3\u7801：
    \n<\/span>{<\/span>js_code[<\/span>:<\/span>max_tokens]<\/span>}<\/span><\/span>
    \n"""<\/span><\/span>
    \n return<\/span> prompt<\/p>\n

    # \u5de5\u4f5c\u6d41\u7a0b\u793a\u4f8b:<\/span>
    \n# 1. \u901a\u8fc7DevTools\u5b9a\u4f4d\u5230\u52a0\u5bc6\u51fd\u6570<\/span>
    \n# 2. \u590d\u5236\u6df7\u6dc6\u540e\u7684\u51fd\u6570\u4ee3\u7801<\/span>
    \n# 3. \u6784\u9020prompt\u53d1\u9001\u7ed9LLM<\/span>
    \n# 4. \u83b7\u53d6LLM\u8fd4\u56de\u7684Python\u5b9e\u73b0<\/span>
    \n# 5. \u672c\u5730\u9a8c\u8bc1\u5e76\u8c03\u8bd5<\/span><\/p>\n

    18.2 \u65e0\u6d4f\u89c8\u5668\u722c\u866b\u65b0\u6280\u672f<\/h4>\n\n\u5de5\u5177\u539f\u7406\u4f18\u52bf\u5c40\u9650<\/tr>\n\n\n\n\n\n\n
    curl_cffi<\/td>\nTLS\u6307\u7eb9\u6a21\u62df<\/td>\n\u65e0\u9700\u6d4f\u89c8\u5668，\u901f\u5ea6\u5feb<\/td>\n\u4e0d\u6267\u884cJS<\/td>\n<\/tr>\n
    tls-client<\/td>\nGo\u5e95\u5c42TLS\u6a21\u62df<\/td>\n\u8de8\u5e73\u53f0，\u9ad8\u6027\u80fd<\/td>\n\u4e0d\u6267\u884cJS<\/td>\n<\/tr>\n
    nodriver<\/td>\nCDP\u534f\u8bae\u7cbe\u7b80\u7248<\/td>\n\u6bd4Selenium\u8f7b\u91cf<\/td>\n\u4ecd\u9700Chromium<\/td>\n<\/tr>\n
    botasaurus<\/td>\n\u9ad8\u7ea7\u722c\u866b\u6846\u67b6<\/td>\n\u81ea\u52a8\u53cd\u53cd\u722c<\/td>\n\u76f8\u5bf9\u8f83\u65b0<\/td>\n<\/tr>\n
    crawl4ai<\/td>\nAI\u9a71\u52a8\u722c\u866b<\/td>\n\u667a\u80fd\u5185\u5bb9\u63d0\u53d6<\/td>\n\u9700\u8981GPU<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    18.3 tls-client：Go\u5e95\u5c42TLS\u6a21\u62df<\/h4>\n

    # pip install tls-client<\/span>
    \nimport<\/span> tls_client<\/p>\n

    # \u6a21\u62df\u6d4f\u89c8\u5668TLS\u6307\u7eb9<\/span>
    \nsession =<\/span> tls_client.<\/span>Session(<\/span>
    \n client_identifier=<\/span>"chrome_131"<\/span>,<\/span>
    \n random_tls_extension_order=<\/span>True<\/span>,<\/span>
    \n)<\/span><\/p>\n

    response =<\/span> session.<\/span>get(<\/span>
    \n "https:\/\/target-website.com\/api\/data"<\/span>,<\/span>
    \n headers=<\/span>{<\/span>"User-Agent"<\/span>:<\/span> "Mozilla\/5.0 Chrome\/131.0.0.0"<\/span>}<\/span>,<\/span>
    \n proxy_url=<\/span>"http:\/\/proxy:8080"<\/span>,<\/span>
    \n)<\/span><\/p>\n

    print<\/span>(<\/span>f"Status: <\/span>{<\/span>response.<\/span>status_code}<\/span><\/span>"<\/span><\/span>)<\/span><\/p>\n

    18.4 crawl4ai：AI\u9a71\u52a8\u7684\u722c\u866b<\/h4>\n

    # pip install crawl4ai<\/span>
    \n# crawl4ai \u662f2025\u5e74\u65b0\u5174\u7684AI\u9a71\u52a8\u722c\u866b\u6846\u67b6<\/span><\/p>\n

    from<\/span> crawl4ai import<\/span> AsyncWebCrawler
    \nfrom<\/span> crawl4ai.<\/span>extraction import<\/span> LLMExtraction
    \nimport<\/span> asyncio<\/p>\n

    async<\/span> def<\/span> ai_crawl<\/span>(<\/span>)<\/span>:<\/span>
    \n """\u4f7f\u7528AI\u8fdb\u884c\u667a\u80fd\u5185\u5bb9\u63d0\u53d6"""<\/span>
    \n async<\/span> with<\/span> AsyncWebCrawler(<\/span>)<\/span> as<\/span> crawler:<\/span>
    \n result =<\/span> await<\/span> crawler.<\/span>arun(<\/span>
    \n url=<\/span>"https:\/\/example.com\/products"<\/span>,<\/span>
    \n wait_until=<\/span>"networkidle"<\/span>,<\/span>
    \n )<\/span><\/p>\n

    if<\/span> result.<\/span>success:<\/span>
    \n print<\/span>(<\/span>f"\u9875\u9762\u957f\u5ea6: <\/span>{<\/span>len<\/span>(<\/span>result.<\/span>html)<\/span>}<\/span><\/span>"<\/span><\/span>)<\/span>
    \n # \u53ef\u8fdb\u4e00\u6b65\u4f7f\u7528LLM\u8fdb\u884c\u7ed3\u6784\u5316\u63d0\u53d6<\/span><\/p>\n

    # asyncio.run(ai_crawl())<\/span><\/p>\n

    18.5 \u6280\u672f\u8d8b\u52bf\u603b\u7ed3<\/h4>\n

    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
    \n\u2502 2025-2026 \u722c\u866b\u6280\u672f\u6f14\u8fdb\u8def\u7ebf \u2502
    \n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524
    \n\u2502 \u2502
    \n\u2502 \u6d4f\u89c8\u5668\u81ea\u52a8\u5316\u6f14\u8fdb: \u2502
    \n\u2502 Selenium –> Playwright\u4e3b\u6d41 –> AI Agent\u722c\u866b \u2502
    \n\u2502 \u2502
    \n\u2502 \u53cd\u68c0\u6d4b\u6280\u672f\u6f14\u8fdb: \u2502
    \n\u2502 \u624b\u52a8\u9006\u5411JS –> curl_cffi\u6d41\u884c –> AI\u8f85\u52a9\u9006\u5411 \u2502
    \n\u2502 \u2502
    \n\u2502 \u67b6\u6784\u6f14\u8fdb: \u2502
    \n\u2502 \u5355\u673a\u90e8\u7f72 –> \u5206\u5e03\u5f0f\u96c6\u7fa4 –> \u8fb9\u7f18\u8ba1\u7b97\u5bf9\u6297 \u2502
    \n\u2502 \u2502
    \n\u2502 \u5173\u952e\u53d8\u5316: \u2502
    \n\u2502 – TLS\u6307\u7eb9\u68c0\u6d4b\u6210\u4e3a\u4e3b\u6d41\u53cd\u722c\u624b\u6bb5 \u2502
    \n\u2502 – LLM\u88ab\u7528\u4e8eJS\u9006\u5411\u5206\u6790\u548c\u667a\u80fd\u5185\u5bb9\u63d0\u53d6 \u2502
    \n\u2502 – \u65e0\u5934\u6d4f\u89c8\u5668\u68c0\u6d4b\u4e0e\u53cd\u68c0\u6d4b\u7684\u535a\u5f08\u6301\u7eed\u5347\u7ea7 \u2502
    \n\u2502 – \u6d4f\u89c8\u5668\u6307\u7eb9\u68c0\u6d4b\u7ef4\u5ea6\u6269\u5c55\u5230WebGL\/Audio\/Fonts \u2502
    \n\u2502 – \u5408\u89c4\u8981\u6c42\u65e5\u76ca\u4e25\u683c，\u722c\u866b\u6cd5\u5f8b\u98ce\u9669\u4e0a\u5347 \u2502
    \n\u2502 \u2502
    \n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

    \u9644\u5f55：\u722c\u866b\u6280\u672f\u9009\u578b\u901f\u67e5\u8868<\/h3>\n\n\u573a\u666f\u63a8\u8350\u6280\u672f\u6808\u7406\u7531<\/tr>\n\n\n\n\n\n\n\n\n\n
    \u7b80\u5355\u9759\u6001\u9875\u9762<\/td>\nrequests + lxml<\/td>\n\u5feb\u901f\u5f00\u53d1，\u6027\u80fd\u591f\u7528<\/td>\n<\/tr>\n
    \u4e2d\u7b49\u89c4\u6a21\u9879\u76ee<\/td>\nScrapy + MongoDB<\/td>\n\u6846\u67b6\u6210\u719f，\u6269\u5c55\u6027\u597d<\/td>\n<\/tr>\n
    \u53cdTLS\u68c0\u6d4b<\/td>\ncurl_cffi<\/td>\n\u4e00\u884c\u4ee3\u7801\u6a21\u62df\u6d4f\u89c8\u5668\u6307\u7eb9<\/td>\n<\/tr>\n
    SPA\u52a8\u6001\u9875\u9762<\/td>\nPlaywright<\/td>\n\u81ea\u52a8\u7b49\u5f85，API\u62e6\u622a<\/td>\n<\/tr>\n
    \u5927\u89c4\u6a21\u5206\u5e03\u5f0f<\/td>\nScrapy-Redis + Kafka<\/td>\n\u9ad8\u541e\u5410，\u53ef\u6269\u5c55<\/td>\n<\/tr>\n
    APP\u6570\u636e<\/td>\nmitmproxy + Frida<\/td>\n\u6293\u5305+Hook<\/td>\n<\/tr>\n
    \u9a8c\u8bc1\u7801\u7a81\u7834<\/td>\nddddocr + \u6253\u7801\u5e73\u53f0<\/td>\n\u672c\u5730\u8bc6\u522b+\u8fdc\u7a0b\u515c\u5e95<\/td>\n<\/tr>\n
    JS\u9006\u5411<\/td>\nDevTools + LLM\u8f85\u52a9<\/td>\n\u4f20\u7edf\u65b9\u6cd5+AI\u52a0\u901f<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    \u514d\u8d23\u58f0\u660e：\u672c\u6307\u5357\u4e2d\u7684\u6240\u6709\u6280\u672f\u548c\u4ee3\u7801\u4ec5\u4f9b\u5b66\u4e60\u548c\u7814\u7a76\u76ee\u7684\u4f7f\u7528\u3002\u4f7f\u7528\u722c\u866b\u6280\u672f\u65f6，\u8bf7\u52a1\u5fc5\u9075\u5b88\u76f8\u5173\u6cd5\u5f8b\u6cd5\u89c4\u3001\u7f51\u7ad9\u670d\u52a1\u6761\u6b3e\u548c robots.txt \u534f\u8bae\u3002\u4efb\u4f55\u56e0\u4e0d\u5f53\u4f7f\u7528\u672c\u6307\u5357\u5185\u5bb9\u800c\u4ea7\u751f\u7684\u6cd5\u5f8b\u95ee\u9898，\u7531\u4f7f\u7528\u8005\u81ea\u884c\u627f\u62c5\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"

    Python \u722c\u866b\u6280\u672f\u5168\u6808\u6307\u5357 \u9762\u5411\u8bfb\u8005：\u6709\u4e00\u5b9a Python \u57fa\u7840\u7684\u6570\u636e\u5f00\u53d1\u5de5\u7a0b\u5e08 \u7248\u672c\u8bf4\u660e：\u878d\u5165 2025-2026 \u5e74\u6700\u65b0\u6280\u672f\u52a8\u6001，\u8986\u76d6 curl_cffi\u3001Playwright \u65b0\u7248\u3001AI \u8f85\u52a9\u9006\u5411\u7b49\u524d\u6cbf\u5185\u5bb9 \u76ee\u5f55
    \n\u722c\u866b\u5168\u666f\u6982\u8ff0HTTP\u534f\u8bae\u4e0e\u7f51\u7edc\u57fa\u7840Python\u722c\u866b\u57fa\u7840\u5e93HTML\u89e3\u6790\u4e0e\u6570\u636e\u63d0\u53d6\u4e3b\u6d41\u722c\u866b\u6846\u67b6\u8be6\u89e3\u52a8\u6001\u9875\u9762\u722c\u53d6\u53cd\u722c\u866b\u6280\u672f\u5168\u89e3\u53cd\u53cd\u722c\u866b\u5b9e\u6218\u9006\u5411\u5de5\u7a0b\u57fa\u7840JS\u9006\u5411\u6df1\u5ea6\u5b9e\u6218APP\u722c\u866b\u4e0e\u6293\u5305\u5206\u5e03\u5f0f\u722c\u866b\u67b6\u6784\u6570\u636e\u5b58\u50a8\u4e0e\u6e05\u6d17\u6cd5\u5f8b\u5408\u89c4<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[10187,81,190,677],"topic":[],"class_list":["post-97387","post","type-post","status-publish","format-standard","hentry","category-server","tag-codingplan","tag-python","tag-190","tag-677"],"yoast_head":"\nPython \u722c\u866b\u6280\u672f\u5168\u6808\u6307\u5357 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.wsisp.com\/helps\/97387.html\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Python \u722c\u866b\u6280\u672f\u5168\u6808\u6307\u5357 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"og:description\" content=\"Python \u722c\u866b\u6280\u672f\u5168\u6808\u6307\u5357 \u9762\u5411\u8bfb\u8005：\u6709\u4e00\u5b9a Python \u57fa\u7840\u7684\u6570\u636e\u5f00\u53d1\u5de5\u7a0b\u5e08 \u7248\u672c\u8bf4\u660e：\u878d\u5165 2025-2026 \u5e74\u6700\u65b0\u6280\u672f\u52a8\u6001，\u8986\u76d6 curl_cffi\u3001Playwright \u65b0\u7248\u3001AI \u8f85\u52a9\u9006\u5411\u7b49\u524d\u6cbf\u5185\u5bb9 \u76ee\u5f55 \u722c\u866b\u5168\u666f\u6982\u8ff0HTTP\u534f\u8bae\u4e0e\u7f51\u7edc\u57fa\u7840Python\u722c\u866b\u57fa\u7840\u5e93HTML\u89e3\u6790\u4e0e\u6570\u636e\u63d0\u53d6\u4e3b\u6d41\u722c\u866b\u6846\u67b6\u8be6\u89e3\u52a8\u6001\u9875\u9762\u722c\u53d6\u53cd\u722c\u866b\u6280\u672f\u5168\u89e3\u53cd\u53cd\u722c\u866b\u5b9e\u6218\u9006\u5411\u5de5\u7a0b\u57fa\u7840JS\u9006\u5411\u6df1\u5ea6\u5b9e\u6218APP\u722c\u866b\u4e0e\u6293\u5305\u5206\u5e03\u5f0f\u722c\u866b\u67b6\u6784\u6570\u636e\u5b58\u50a8\u4e0e\u6e05\u6d17\u6cd5\u5f8b\u5408\u89c4\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.wsisp.com\/helps\/97387.html\" \/>\n<meta property=\"og:site_name\" content=\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"article:published_time\" content=\"2026-08-29T03:40:45+00:00\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"29 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/97387.html\",\"url\":\"https:\/\/www.wsisp.com\/helps\/97387.html\",\"name\":\"Python \u722c\u866b\u6280\u672f\u5168\u6808\u6307\u5357 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"isPartOf\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\"},\"datePublished\":\"2026-08-29T03:40:45+00:00\",\"dateModified\":\"2026-08-29T03:40:45+00:00\",\"author\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/97387.html#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.wsisp.com\/helps\/97387.html\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/97387.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.wsisp.com\/helps\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Python \u722c\u866b\u6280\u672f\u5168\u6808\u6307\u5357\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\",\"url\":\"https:\/\/www.wsisp.com\/helps\/\",\"name\":\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"description\":\"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"contentUrl\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"caption\":\"admin\"},\"sameAs\":[\"http:\/\/wp.wsisp.com\"],\"url\":\"https:\/\/www.wsisp.com\/helps\/author\/admin\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Python \u722c\u866b\u6280\u672f\u5168\u6808\u6307\u5357 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.wsisp.com\/helps\/97387.html","og_locale":"zh_CN","og_type":"article","og_title":"Python \u722c\u866b\u6280\u672f\u5168\u6808\u6307\u5357 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","og_description":"Python \u722c\u866b\u6280\u672f\u5168\u6808\u6307\u5357 \u9762\u5411\u8bfb\u8005：\u6709\u4e00\u5b9a Python \u57fa\u7840\u7684\u6570\u636e\u5f00\u53d1\u5de5\u7a0b\u5e08 \u7248\u672c\u8bf4\u660e：\u878d\u5165 2025-2026 \u5e74\u6700\u65b0\u6280\u672f\u52a8\u6001，\u8986\u76d6 curl_cffi\u3001Playwright \u65b0\u7248\u3001AI \u8f85\u52a9\u9006\u5411\u7b49\u524d\u6cbf\u5185\u5bb9 \u76ee\u5f55 \u722c\u866b\u5168\u666f\u6982\u8ff0HTTP\u534f\u8bae\u4e0e\u7f51\u7edc\u57fa\u7840Python\u722c\u866b\u57fa\u7840\u5e93HTML\u89e3\u6790\u4e0e\u6570\u636e\u63d0\u53d6\u4e3b\u6d41\u722c\u866b\u6846\u67b6\u8be6\u89e3\u52a8\u6001\u9875\u9762\u722c\u53d6\u53cd\u722c\u866b\u6280\u672f\u5168\u89e3\u53cd\u53cd\u722c\u866b\u5b9e\u6218\u9006\u5411\u5de5\u7a0b\u57fa\u7840JS\u9006\u5411\u6df1\u5ea6\u5b9e\u6218APP\u722c\u866b\u4e0e\u6293\u5305\u5206\u5e03\u5f0f\u722c\u866b\u67b6\u6784\u6570\u636e\u5b58\u50a8\u4e0e\u6e05\u6d17\u6cd5\u5f8b\u5408\u89c4","og_url":"https:\/\/www.wsisp.com\/helps\/97387.html","og_site_name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","article_published_time":"2026-08-29T03:40:45+00:00","author":"admin","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"admin","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"29 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.wsisp.com\/helps\/97387.html","url":"https:\/\/www.wsisp.com\/helps\/97387.html","name":"Python \u722c\u866b\u6280\u672f\u5168\u6808\u6307\u5357 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","isPartOf":{"@id":"https:\/\/www.wsisp.com\/helps\/#website"},"datePublished":"2026-08-29T03:40:45+00:00","dateModified":"2026-08-29T03:40:45+00:00","author":{"@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41"},"breadcrumb":{"@id":"https:\/\/www.wsisp.com\/helps\/97387.html#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.wsisp.com\/helps\/97387.html"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.wsisp.com\/helps\/97387.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.wsisp.com\/helps"},{"@type":"ListItem","position":2,"name":"Python \u722c\u866b\u6280\u672f\u5168\u6808\u6307\u5357"}]},{"@type":"WebSite","@id":"https:\/\/www.wsisp.com\/helps\/#website","url":"https:\/\/www.wsisp.com\/helps\/","name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","description":"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41","name":"admin","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/","url":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","contentUrl":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","caption":"admin"},"sameAs":["http:\/\/wp.wsisp.com"],"url":"https:\/\/www.wsisp.com\/helps\/author\/admin"}]}},"_links":{"self":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/97387","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/comments?post=97387"}],"version-history":[{"count":0,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/97387\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media?parent=97387"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/categories?post=97387"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/tags?post=97387"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/topic?post=97387"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}