{"id":101774,"date":"2026-09-07T15:25:17","date_gmt":"2026-09-07T07:25:17","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/101774.html"},"modified":"2026-09-07T15:25:17","modified_gmt":"2026-09-07T07:25:17","slug":"python-%e5%ae%9e%e6%88%98%ef%bc%9a%e5%8a%a8%e6%80%81%e6%b8%94%e4%b8%9a%e8%b5%84%e6%ba%90%e5%88%86%e5%b8%83%e5%9b%be%e9%87%87%e9%9b%86-%e5%9f%ba%e4%ba%8e-obis-%e6%97%b6%e9%97%b4","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/101774.html","title":{"rendered":"Python \u5b9e\u6218\uff1a\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe\u91c7\u96c6\u2014\u2014\u57fa\u4e8e OBIS \u65f6\u95f4\u8f74\u56fe\u5c42\u7684 API \u91c7\u96c6\u3001\u6e05\u6d17\u3001\u805a\u5408\u4e0e\u5bfc\u51fa"},"content":{"rendered":"<p> <img decoding=\"async\" width=\"1000\" src=\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/09\/20260907072512-6a9e66d805f70.png\" \/> <\/p>\n<p>\u3297\ufe0f\u672c\u671f\u5185\u5bb9\u5df2\u6536\u5f55\u81f3\u4e13\u680f\u300aPython\u722c\u866b\u5b9e\u6218\u300b&#xff0c;\u6301\u7eed\u5b8c\u5584\u77e5\u8bc6\u4f53\u7cfb\u4e0e\u9879\u76ee\u5b9e\u6218&#xff0c;\u5efa\u8bae\u5148\u8ba2\u9605\u6536\u85cf&#xff0c;\u540e\u7eed\u67e5\u9605\u66f4\u65b9\u4fbf&#xff5e; \u3299\ufe0f\u672c\u671f\u722c\u866b\u96be\u5ea6\u6307\u6570&#xff1a;\u2b50\u2b50\u2b50\u2b50\u2606&#xff08;\u9ad8\u7ea7&#xff09; &#x1f250;\u798f\u5229&#xff1a; <font color=\"red\">\u4e00\u6b21\u8ba2\u9605\u540e&#xff0c;\u4e13\u680f\u5185\u7684\u6240\u6709\u6587\u7ae0\u53ef\u6c38\u4e45\u514d\u8d39\u770b&#xff0c;\u6301\u7eed\u66f4\u65b0\u4e2d&#xff0c;\u4fdd\u5e951000&#043;(\u7bc7)\u786c\u6838\u5b9e\u6218\u5185\u5bb9\u3002<\/font><\/p>\n<\/p>\n<h4>\u5168\u6587\u76ee\u5f55&#xff1a;<\/h4>\n<ul>\n<li>\n<ul>\n<li>&#x1f31f; \u5f00\u7bc7\u8bed<\/li>\n<li>0\ufe0f\u20e3 \u524d\u8a00&#xff08;Preface&#xff09;<\/li>\n<li>1\ufe0f\u20e3 \u6458\u8981&#xff08;Abstract&#xff09;<\/li>\n<li>2\ufe0f\u20e3 \u80cc\u666f\u4e0e\u9700\u6c42&#xff08;Why&#xff09;<\/li>\n<li>\n<ul>\n<li>2.1 \u4e3a\u4ec0\u4e48\u8981\u91c7\u96c6\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe<\/li>\n<li>2.2 \u672c\u6587\u76ee\u6807\u7ad9\u70b9\u4e0e\u91c7\u96c6\u5bf9\u8c61<\/li>\n<li>2.3 \u76ee\u6807\u5b57\u6bb5\u6e05\u5355<\/li>\n<\/ul>\n<\/li>\n<li>3\ufe0f\u20e3 \u5408\u89c4\u4e0e\u6ce8\u610f\u4e8b\u9879&#xff08;\u5fc5\u5199&#xff09;<\/li>\n<li>\n<ul>\n<li>3.1 robots.txt \u57fa\u672c\u8bf4\u660e<\/li>\n<li>3.2 \u9891\u7387\u63a7\u5236<\/li>\n<li>3.3 \u4e0d\u91c7\u96c6\u654f\u611f\u4fe1\u606f<\/li>\n<li>3.4 \u6570\u636e\u89e3\u91ca\u8981\u8c28\u614e<\/li>\n<\/ul>\n<\/li>\n<li>4\ufe0f\u20e3 \u6280\u672f\u9009\u578b\u4e0e\u6574\u4f53\u6d41\u7a0b&#xff08;What\/How&#xff09;<\/li>\n<li>\n<ul>\n<li>4.1 \u9759\u6001\u3001\u52a8\u6001\u3001API \u4e09\u7c7b\u91c7\u96c6\u65b9\u5f0f<\/li>\n<li>4.2 \u52a8\u6001\u56fe\u5c42\u91c7\u96c6\u7684\u6838\u5fc3\u601d\u8def<\/li>\n<li>4.3 \u6574\u4f53\u6d41\u7a0b<\/li>\n<li>4.4 \u4e3a\u4ec0\u4e48\u9009 requests&#xff0c;\u800c\u4e0d\u662f\u4e00\u4e0a\u6765\u5c31 Playwright<\/li>\n<\/ul>\n<\/li>\n<li>5\ufe0f\u20e3 \u73af\u5883\u51c6\u5907\u4e0e\u4f9d\u8d56\u5b89\u88c5&#xff08;\u53ef\u590d\u73b0&#xff09;<\/li>\n<li>\n<ul>\n<li>5.1 Python \u7248\u672c<\/li>\n<li>5.2 \u521b\u5efa\u865a\u62df\u73af\u5883<\/li>\n<li>5.3 \u5b89\u88c5\u4f9d\u8d56<\/li>\n<li>5.4 \u63a8\u8350\u9879\u76ee\u7ed3\u6784<\/li>\n<li>5.5 \u914d\u7f6e\u6587\u4ef6<\/li>\n<\/ul>\n<\/li>\n<li>6\ufe0f\u20e3 \u6838\u5fc3\u5b9e\u73b0&#xff1a;\u8bf7\u6c42\u5c42&#xff08;Fetcher&#xff09;<\/li>\n<li>\n<ul>\n<li>6.1 headers \u8bbe\u8ba1<\/li>\n<li>6.2 timeout<\/li>\n<li>6.3 session \/ cookie<\/li>\n<li>6.4 \u5931\u8d25\u5904\u7406&#xff1a;\u91cd\u8bd5\u4e0e\u9000\u907f<\/li>\n<li>6.5 \u4ee3\u7801&#xff1a;&#096;config.py&#096;<\/li>\n<li>6.6 \u4ee3\u7801&#xff1a;&#096;fetcher.py&#096;<\/li>\n<\/ul>\n<\/li>\n<li>7\ufe0f\u20e3 \u6838\u5fc3\u5b9e\u73b0&#xff1a;\u89e3\u6790\u5c42&#xff08;Parser&#xff09;<\/li>\n<li>\n<ul>\n<li>7.1 \u89e3\u6790\u65b9\u5f0f<\/li>\n<li>7.2 \u5217\u8868\u9875\u5982\u4f55\u62ff\u8be6\u60c5\u94fe\u63a5<\/li>\n<li>7.3 \u8be6\u60c5\u9875\u5982\u4f55\u62bd\u5b57\u6bb5<\/li>\n<li>7.4 \u7f3a\u5931\u5b57\u6bb5\u600e\u4e48\u529e<\/li>\n<li>7.5 \u4ee3\u7801&#xff1a;&#096;time_utils.py&#096;<\/li>\n<li>7.6 \u4ee3\u7801&#xff1a;&#096;parser.py&#096;<\/li>\n<\/ul>\n<\/li>\n<li>8\ufe0f\u20e3 \u6570\u636e\u5b58\u50a8\u4e0e\u5bfc\u51fa&#xff08;Storage&#xff09;<\/li>\n<li>\n<ul>\n<li>8.1 \u8d77\u6b65\u9009\u62e9 SQLite<\/li>\n<li>8.2 \u5b57\u6bb5\u6620\u5c04\u8868<\/li>\n<li>8.3 \u53bb\u91cd\u7b56\u7565<\/li>\n<li>8.4 \u4ee3\u7801&#xff1a;&#096;storage.py&#096;<\/li>\n<\/ul>\n<\/li>\n<li>9\ufe0f\u20e3 \u8fd0\u884c\u65b9\u5f0f\u4e0e\u7ed3\u679c\u5c55\u793a&#xff08;\u5fc5\u5199&#xff09;<\/li>\n<li>\n<ul>\n<li>9.1 \u5165\u53e3\u6587\u4ef6&#xff1a;&#096;run_spider.py&#096;<\/li>\n<li>9.2 \u5bfc\u51fa GeoJSON&#xff1a;&#096;exporter.py&#096;<\/li>\n<li>9.3 \u751f\u6210\u65f6\u95f4\u8f74\u5730\u56fe&#xff1a;&#096;map_builder.py&#096;<\/li>\n<li>9.4 \u542f\u52a8\u547d\u4ee4<\/li>\n<li>9.5 \u8f93\u51fa\u8def\u5f84<\/li>\n<li>9.6 \u793a\u4f8b\u7ed3\u679c<\/li>\n<li>9.7 \u67e5\u770b SQLite<\/li>\n<\/ul>\n<\/li>\n<li>&#x1f51f; \u5e38\u89c1\u95ee\u9898\u4e0e\u6392\u9519&#xff08;\u5f3a\u70c8\u5efa\u8bae\u5199&#xff09;<\/li>\n<li>\n<ul>\n<li>10.1 403 \u600e\u4e48\u529e<\/li>\n<li>10.2 429 \u600e\u4e48\u529e<\/li>\n<li>10.3 HTML \u6293\u5230\u7a7a\u58f3\u600e\u4e48\u529e<\/li>\n<li>10.4 \u89e3\u6790\u62a5\u9519\u600e\u4e48\u529e<\/li>\n<li>10.5 \u7f16\u7801\/\u4e71\u7801\u5982\u4f55\u5904\u7406<\/li>\n<li>10.6 \u6570\u636e\u4e3a\u7a7a\u600e\u4e48\u529e<\/li>\n<li>10.7 \u65f6\u95f4\u8f74\u4e0d\u52a8\u600e\u4e48\u529e<\/li>\n<\/ul>\n<\/li>\n<li>1\ufe0f\u20e31\ufe0f\u20e3 \u8fdb\u9636\u4f18\u5316&#xff08;\u53ef\u9009\u4f46\u52a0\u5206&#xff09;<\/li>\n<li>\n<ul>\n<li>11.1 \u5e76\u53d1\u4f18\u5316<\/li>\n<li>11.2 \u65ad\u70b9\u7eed\u8dd1<\/li>\n<li>11.3 \u65e5\u5fd7\u4e0e\u76d1\u63a7<\/li>\n<li>11.4 \u5b9a\u65f6\u4efb\u52a1<\/li>\n<li>11.5 \u7a7a\u95f4\u7f51\u683c\u5347\u7ea7<\/li>\n<li>11.6 \u5f3a\u5ea6\u6307\u6807\u5347\u7ea7<\/li>\n<li>11.7 \u591a\u7269\u79cd\u6279\u91cf\u91c7\u96c6<\/li>\n<li>11.8 \u4e0e Scrapy \u96c6\u6210<\/li>\n<\/ul>\n<\/li>\n<li>1\ufe0f\u20e32\ufe0f\u20e3 \u603b\u7ed3\u4e0e\u5ef6\u4f38\u9605\u8bfb<\/li>\n<li>&#x1f31f; \u6587\u672b<\/li>\n<li>\n<ul>\n<li>\u2705 \u4e13\u680f\u6301\u7eed\u66f4\u65b0\u4e2d&#xff5c;\u5efa\u8bae\u6536\u85cf &#043; \u8ba2\u9605<\/li>\n<li>\u2705 \u4e92\u52a8\u5f81\u96c6<\/li>\n<li>\u2705 \u514d\u8d23\u58f0\u660e<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h3>&#x1f31f; \u5f00\u7bc7\u8bed<\/h3>\n<p>\u54c8\u55bd&#xff0c;\u5404\u4f4d\u5c0f\u4f19\u4f34\u4eec\u4f60\u4eec\u597d\u5440&#xff5e;\u6211\u662f\u3010\u55b5\u624b\u3011\u3002 \u8fd0\u8425\u793e\u533a&#xff1a; C\u7ad9 \/ \u6398\u91d1 \/ \u817e\u8baf\u4e91 \/ \u963f\u91cc\u4e91 \/ \u534e\u4e3a\u4e91 \/ 51CTO \u6b22\u8fce\u5927\u5bb6\u5e38\u6765\u901b\u901b&#xff0c;\u4e00\u8d77\u5b66\u4e60&#xff0c;\u4e00\u8d77\u8fdb\u6b65&#xff5e;&#x1f31f;<\/p>\n<p>\u2003\u2003\u6211\u957f\u671f\u4e13\u6ce8 Python \u722c\u866b\u5de5\u7a0b\u5316\u5b9e\u6218&#xff0c;\u4e3b\u7406\u4e13\u680f&#x1f449; \u300aPython\u722c\u866b\u5b9e\u6218\u300b&#xff1a;\u4ece\u91c7\u96c6\u7b56\u7565\u5230\u53cd\u722c\u5bf9\u6297&#xff0c;\u4ece\u6570\u636e\u6e05\u6d17\u5230\u5206\u5e03\u5f0f\u8c03\u5ea6&#xff0c;\u6301\u7eed\u8f93\u51fa\u53ef\u590d\u7528\u7684\u65b9\u6cd5\u8bba\u4e0e\u53ef\u843d\u5730\u6848\u4f8b\u3002\u5185\u5bb9\u4e3b\u6253\u4e00\u4e2a\u201c\u80fd\u8dd1\u3001\u80fd\u7528\u3001\u80fd\u6269\u5c55\u201d&#xff0c;\u8ba9\u6570\u636e\u4ef7\u503c\u771f\u6b63\u505a\u5230\u2014\u2014\u6293\u5f97\u5230\u3001\u6d17\u5f97\u51c0\u3001\u7528\u5f97\u4e0a\u3002<\/p>\n<p>\u2003\u2003&#x1f4cc; \u4e13\u680f\u98df\u7528\u6307\u5357&#xff08;\u5efa\u8bae\u6536\u85cf&#xff09;<\/p>\n<ul>\n<li>\u2705 \u5165\u95e8\u57fa\u7840&#xff1a;\u73af\u5883\u642d\u5efa \/ \u8bf7\u6c42\u4e0e\u89e3\u6790 \/ \u6570\u636e\u843d\u5e93<\/li>\n<li>\u2705 \u8fdb\u9636\u63d0\u5347&#xff1a;\u767b\u5f55\u9274\u6743 \/ \u52a8\u6001\u6e32\u67d3 \/ \u53cd\u722c\u5bf9\u6297<\/li>\n<li>\u2705 \u5de5\u7a0b\u5b9e\u6218&#xff1a;\u5f02\u6b65\u5e76\u53d1 \/ \u5206\u5e03\u5f0f\u8c03\u5ea6 \/ \u76d1\u63a7\u4e0e\u5bb9\u9519<\/li>\n<li>\u2705 \u9879\u76ee\u843d\u5730&#xff1a;\u6570\u636e\u6cbb\u7406 \/ \u53ef\u89c6\u5316\u5206\u6790 \/ \u573a\u666f\u5316\u5e94\u7528<\/li>\n<\/ul>\n<p>&#x1f4e3; \u4e13\u680f\u63a8\u5e7f\u65f6\u95f4&#xff1a;\u5982\u679c\u4f60\u60f3\u7cfb\u7edf\u5b66\u722c\u866b&#xff0c;\u800c\u4e0d\u662f\u788e\u7247\u5316\u4e1c\u62fc\u897f\u51d1&#xff0c;\u6b22\u8fce\u8ba2\u9605\u4e13\u680f&#x1f449;\u300aPython\u722c\u866b\u5b9e\u6218\u300b&#x1f448;&#xff0c;<font color=\"red\">\u4e00\u6b21\u8ba2\u9605\u540e&#xff0c;\u4e13\u680f\u5185\u7684\u6240\u6709\u6587\u7ae0\u53ef\u6c38\u4e45\u514d\u8d39\u9605\u8bfb&#xff0c;\u6301\u7eed\u66f4\u65b0\u4e2d\u3002<\/font> \u2003\u2003 &#x1f495;\u8ba2\u9605\u540e\u66f4\u65b0\u4f1a\u4f18\u5148\u63a8\u9001&#xff0c;\u6309\u76ee\u5f55\u5b66\u4e60\u66f4\u9ad8\u6548&#x1f4af;&#xff5e;<\/p>\n<h3>0\ufe0f\u20e3 \u524d\u8a00&#xff08;Preface&#xff09;<\/h3>\n<p>\u8fd9\u7bc7\u6587\u7ae0\u8981\u505a\u7684\u4e8b\u60c5\u5f88\u660e\u786e&#xff1a;\u7528 Python \u91c7\u96c6\u516c\u5f00\u6d77\u6d0b\u751f\u7269\u5206\u5e03\u6570\u636e&#xff0c;\u56f4\u7ed5\u201c\u7269\u79cd\/\u8d44\u6e90\u540d\u3001\u6d77\u57df\u3001\u65f6\u95f4\u3001\u5f3a\u5ea6\u6307\u6807\u3001\u6765\u6e90\u201d\u8fd9\u51e0\u4e2a\u5b57\u6bb5&#xff0c;\u628a\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe\u80cc\u540e\u7684\u65f6\u95f4\u8f74\u6570\u636e\u6293\u53d6\u51fa\u6765&#xff0c;\u5e76\u6700\u7ec8\u5bfc\u51fa\u4e3a CSV\u3001SQLite\u3001GeoJSON \u548c\u4e00\u4e2a\u53ef\u64ad\u653e\u65f6\u95f4\u8f74\u7684 HTML \u5730\u56fe\u3002<\/p>\n<p>\u8bfb\u5b8c\u8fd9\u7bc7\u6587\u7ae0&#xff0c;\u4f60\u81f3\u5c11\u80fd\u62ff\u5230\u4e09\u6837\u4e1c\u897f\u3002<\/p>\n<p>\u7b2c\u4e00&#xff0c;\u4f60\u4f1a\u77e5\u9053\u52a8\u6001\u5730\u56fe\u7c7b\u7f51\u7ad9\u7684\u6570\u636e\u5230\u5e95\u85cf\u5728\u54ea\u91cc\u3002\u5f88\u591a\u5730\u56fe\u9875\u9762\u770b\u4e0a\u53bb\u662f\u52a8\u753b\u3001\u56fe\u5c42\u3001\u65f6\u95f4\u8f74&#xff0c;\u672c\u8d28\u4e0a\u5f80\u5f80\u662f\u524d\u7aef\u6309\u65f6\u95f4\u3001\u8303\u56f4\u3001\u7269\u79cd\u6761\u4ef6\u8bf7\u6c42\u63a5\u53e3&#xff0c;\u518d\u628a\u8fd4\u56de\u7684\u70b9\u3001\u7f51\u683c\u6216\u74e6\u7247\u6e32\u67d3\u5230\u5730\u56fe\u4e0a\u3002\u771f\u6b63\u8981\u91c7\u96c6\u7684\u4e0d\u662f\u9875\u9762\u4e0a\u90a3\u5f20\u201c\u4f1a\u52a8\u7684\u56fe\u201d&#xff0c;\u800c\u662f\u56fe\u5c42\u80cc\u540e\u7684\u63a5\u53e3\u53c2\u6570\u548c\u54cd\u5e94\u6570\u636e\u3002<\/p>\n<p>\u7b2c\u4e8c&#xff0c;\u4f60\u4f1a\u5f97\u5230\u4e00\u5957\u53ef\u8fd0\u884c\u7684 Python \u9879\u76ee\u9aa8\u67b6\u3002\u5b83\u5305\u542b\u8bf7\u6c42\u5c42\u3001\u89e3\u6790\u5c42\u3001\u6e05\u6d17\u5c42\u3001\u5b58\u50a8\u5c42\u3001\u5bfc\u51fa\u5c42\u548c\u53ef\u89c6\u5316\u5c42&#xff0c;\u4e0d\u662f\u53ea\u8d34\u4e00\u4e2a\u534a\u622a\u811a\u672c\u5c31\u7ed3\u675f\u3002\u4ee3\u7801\u4f1a\u56f4\u7ed5 OBIS \u516c\u5f00 API \u8bbe\u8ba1&#xff0c;\u9ed8\u8ba4\u7528\u6708\u5ea6\u65f6\u95f4\u7a97\u53e3\u91c7\u96c6\u6307\u5b9a\u7269\u79cd\u5728\u6307\u5b9a\u6d77\u57df\u5185\u7684 occurrence \u8bb0\u5f55&#xff0c;\u518d\u805a\u5408\u6210\u7f51\u683c\u5f3a\u5ea6\u3002<\/p>\n<p>\u7b2c\u4e09&#xff0c;\u4f60\u4f1a\u638c\u63e1\u52a8\u6001\u65f6\u95f4\u8f74\u91c7\u96c6\u7684\u57fa\u672c\u5957\u8def&#xff1a;\u5148\u7528\u6d4f\u89c8\u5668\u5f00\u53d1\u8005\u5de5\u5177\u89c2\u5bdf\u63a5\u53e3&#xff0c;\u518d\u628a\u65f6\u95f4\u8f74\u62c6\u6210\u4e00\u4e2a\u4e2a\u65f6\u95f4\u7247&#xff0c;\u63a5\u7740\u6309\u65f6\u95f4\u7247\u5206\u9875\u6293\u53d6&#xff0c;\u6700\u540e\u628a\u539f\u59cb\u70b9\u6570\u636e\u8f6c\u6362\u6210\u201c\u67d0\u7269\u79cd\u5728\u67d0\u6d77\u57df\u3001\u67d0\u6708\u4efd\u3001\u67d0\u7f51\u683c\u7684\u51fa\u73b0\u5f3a\u5ea6\u201d\u3002<\/p>\n<p>\u6211\u4e2a\u4eba\u5f88\u559c\u6b22\u8fd9\u4e00\u7c7b\u6848\u4f8b&#xff0c;\u56e0\u4e3a\u5b83\u4e0d\u50cf\u666e\u901a\u5217\u8868\u9875\u91c7\u96c6\u90a3\u6837\u53ea\u76ef\u7740\u6807\u9898\u3001\u94fe\u63a5\u548c\u6b63\u6587&#xff0c;\u800c\u662f\u8981\u5904\u7406\u5730\u7406\u8303\u56f4\u3001\u65f6\u95f4\u8303\u56f4\u3001\u56fe\u5c42\u805a\u5408\u3001\u5206\u9875\u3001\u5bb9\u9519\u548c\u5bfc\u51fa\u3002\u5b83\u66f4\u63a5\u8fd1\u771f\u5b9e\u6570\u636e\u5de5\u7a0b\u91cc\u7684\u91c7\u96c6\u4efb\u52a1&#xff0c;\u4e5f\u66f4\u80fd\u770b\u51fa\u4e00\u4e2a\u722c\u866b\u9879\u76ee\u662f\u4e0d\u662f\u6709\u57fa\u672c\u7684\u5de5\u7a0b\u610f\u8bc6\u3002<\/p>\n<hr \/>\n<h3>1\ufe0f\u20e3 \u6458\u8981&#xff08;Abstract&#xff09;<\/h3>\n<p>\u672c\u6587\u4f7f\u7528 Python\u3001requests\u3001pandas\u3001SQLite\u3001GeoJSON \u4e0e Folium&#xff0c;\u91c7\u96c6\u516c\u5f00\u6d77\u6d0b\u751f\u7269\u5206\u5e03\u63a5\u53e3\u4e2d\u7684\u7269\u79cd\u51fa\u73b0\u8bb0\u5f55&#xff0c;\u5e76\u6309\u6708\u4efd\u548c\u7a7a\u95f4\u7f51\u683c\u805a\u5408\u4e3a\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe\u6570\u636e&#xff0c;\u6700\u7ec8\u4ea7\u51fa\u53ef\u5206\u6790\u7684\u8868\u683c\u6570\u636e\u548c\u53ef\u64ad\u653e\u65f6\u95f4\u8f74\u7684 HTML \u5730\u56fe\u3002<\/p>\n<p>\u8bfb\u5b8c\u672c\u6587\u53ef\u4ee5\u83b7\u5f97&#xff1a;<\/p>\n<li>\u4e00\u5957\u9762\u5411\u52a8\u6001\u5730\u56fe\u56fe\u5c42\u7684\u91c7\u96c6\u601d\u8def&#xff0c;\u91cd\u70b9\u662f\u65f6\u95f4\u8f74\u3001\u7a7a\u95f4\u8303\u56f4\u3001\u5206\u9875\u548c\u56fe\u5c42\u6570\u636e\u89e3\u6790\u3002<\/li>\n<li>\u4e00\u4e2a\u5b8c\u6574\u7684 Python \u793a\u4f8b\u9879\u76ee&#xff0c;\u5305\u542b Fetcher\u3001Parser\u3001Storage\u3001Exporter \u548c Map Builder\u3002<\/li>\n<li>\u4e00\u4efd\u5b57\u6bb5\u89c4\u8303&#xff0c;\u8986\u76d6\u7269\u79cd\/\u8d44\u6e90\u540d\u3001\u6d77\u57df\u3001\u65f6\u95f4\u3001\u5f3a\u5ea6\u6307\u6807\u3001\u6765\u6e90\u3001\u7ecf\u7eac\u5ea6\u3001\u7f51\u683c\u7f16\u53f7\u3001\u53bb\u91cd\u952e\u7b49\u6838\u5fc3\u5b57\u6bb5\u3002<\/li>\n<p>\u672c\u6587\u9009\u62e9\u516c\u5f00\u3001\u65e0\u9700\u767b\u5f55\u7684\u6570\u636e\u63a5\u53e3\u505a\u6f14\u793a&#xff0c;\u4e0d\u8ba8\u8bba\u7ed5\u8fc7\u767b\u5f55\u3001\u7834\u89e3\u4ed8\u8d39\u6570\u636e\u3001\u89c4\u907f\u8bbf\u95ee\u9650\u5236\u7b49\u5185\u5bb9\u3002\u6280\u672f\u5206\u4eab\u7684\u91cd\u70b9\u662f\u5408\u89c4\u91c7\u96c6\u3001\u7ed3\u6784\u5316\u6574\u7406\u548c\u53ef\u590d\u73b0\u5206\u6790\u3002<\/p>\n<hr \/>\n<h3>2\ufe0f\u20e3 \u80cc\u666f\u4e0e\u9700\u6c42&#xff08;Why&#xff09;<\/h3>\n<h4>2.1 \u4e3a\u4ec0\u4e48\u8981\u91c7\u96c6\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe<\/h4>\n<p>\u6e14\u4e1a\u8d44\u6e90\u3001\u6d77\u6d0b\u751f\u7269\u51fa\u73b0\u8bb0\u5f55\u3001\u6d77\u57df\u751f\u6001\u89c2\u6d4b\u6570\u636e&#xff0c;\u901a\u5e38\u4e0d\u662f\u4e00\u5f20\u9759\u6001\u8868\u5c31\u80fd\u8bb2\u6e05\u695a\u7684\u3002\u5b83\u4eec\u5929\u7136\u5e26\u6709\u4e09\u4e2a\u7ef4\u5ea6&#xff1a;<\/p>\n<ul>\n<li>\u7a7a\u95f4&#xff1a;\u8d44\u6e90\u51fa\u73b0\u5728\u54ea\u91cc&#xff0c;\u9760\u8fd1\u54ea\u4e2a\u6d77\u57df&#xff0c;\u5206\u5e03\u5bc6\u5ea6\u5982\u4f55&#xff1b;<\/li>\n<li>\u65f6\u95f4&#xff1a;\u67d0\u4e2a\u8d44\u6e90\u5728\u54ea\u4e9b\u6708\u4efd\u66f4\u96c6\u4e2d&#xff0c;\u662f\u5426\u6709\u5b63\u8282\u6027\u53d8\u5316&#xff1b;<\/li>\n<li>\u5f3a\u5ea6&#xff1a;\u67d0\u4e2a\u533a\u57df\u5185\u8bb0\u5f55\u6570\u91cf\u3001\u89c2\u6d4b\u9891\u7387\u3001\u76f8\u5bf9\u5bc6\u5ea6\u6216\u5176\u4ed6\u6307\u6807\u9ad8\u4e0d\u9ad8\u3002<\/li>\n<\/ul>\n<p>\u5f88\u591a\u516c\u5f00\u5e73\u53f0\u4f1a\u628a\u8fd9\u4e9b\u6570\u636e\u505a\u6210\u5730\u56fe\u56fe\u5c42&#xff0c;\u8ba9\u7528\u6237\u901a\u8fc7\u7b5b\u9009\u6761\u4ef6\u67e5\u770b\u5206\u5e03\u7ed3\u679c\u3002\u6709\u7684\u9875\u9762\u8fd8\u6709\u65f6\u95f4\u6ed1\u5757&#xff0c;\u62d6\u52a8\u6ed1\u5757\u540e&#xff0c;\u5730\u56fe\u4e0a\u7684\u70b9\u3001\u70ed\u529b\u56fe\u6216\u7f51\u683c\u989c\u8272\u4f1a\u53d1\u751f\u53d8\u5316\u3002\u8fd9\u79cd\u9875\u9762\u5bf9\u666e\u901a\u7528\u6237\u5f88\u53cb\u597d&#xff0c;\u4f46\u5bf9\u6570\u636e\u5206\u6790\u6765\u8bf4\u4e0d\u591f\u65b9\u4fbf&#xff0c;\u56e0\u4e3a\u5206\u6790\u4eba\u5458\u901a\u5e38\u9700\u8981\u628a\u56fe\u5c42\u80cc\u540e\u7684\u6570\u636e\u843d\u6210\u8868\u683c\u3002<\/p>\n<p>\u6bd4\u5982&#xff0c;\u6211\u4eec\u53ef\u80fd\u4f1a\u95ee&#xff1a;<\/p>\n<ul>\n<li>\u67d0\u4e2a\u9c7c\u7c7b\u8d44\u6e90\u5728\u67d0\u7247\u6d77\u57df\u8fd1\u51e0\u5e74\u8bb0\u5f55\u6700\u5bc6\u96c6\u7684\u6708\u4efd\u662f\u54ea\u51e0\u4e2a\u6708&#xff1f;<\/li>\n<li>\u8d44\u6e90\u5206\u5e03\u662f\u5426\u5448\u73b0\u5411\u67d0\u4e2a\u65b9\u5411\u79fb\u52a8\u7684\u8d8b\u52bf&#xff1f;<\/li>\n<li>\u54ea\u4e9b\u7f51\u683c\u533a\u57df\u957f\u671f\u4fdd\u6301\u8f83\u9ad8\u51fa\u73b0\u5f3a\u5ea6&#xff1f;<\/li>\n<li>\u4e0d\u540c\u7269\u79cd\u5728\u540c\u4e00\u6d77\u57df\u7684\u65f6\u7a7a\u5206\u5e03\u662f\u5426\u6709\u660e\u663e\u5dee\u5f02&#xff1f;<\/li>\n<li>\u662f\u5426\u80fd\u628a\u516c\u5f00\u8bb0\u5f55\u6574\u7406\u6210\u540e\u7eed\u5efa\u6a21\u3001\u770b\u677f\u6216\u9884\u8b66\u7cfb\u7edf\u7684\u6570\u636e\u6e90&#xff1f;<\/li>\n<\/ul>\n<p>\u8fd9\u4e9b\u95ee\u9898\u90fd\u9700\u8981\u7ed3\u6784\u5316\u6570\u636e\u3002\u5149\u770b\u5730\u56fe\u622a\u56fe\u4e0d\u591f&#xff0c;\u5fc5\u987b\u628a\u56fe\u5c42\u6570\u636e\u91c7\u96c6\u4e0b\u6765&#xff0c;\u6e05\u6d17\u3001\u805a\u5408\u3001\u5b58\u50a8\u3001\u5bfc\u51fa\u3002<\/p>\n<h4>2.2 \u672c\u6587\u76ee\u6807\u7ad9\u70b9\u4e0e\u91c7\u96c6\u5bf9\u8c61<\/h4>\n<p>\u672c\u6587\u4ee5\u516c\u5f00\u6d77\u6d0b\u751f\u7269\u8bb0\u5f55\u63a5\u53e3\u4f5c\u4e3a\u6848\u4f8b&#xff0c;\u76ee\u6807\u4e0d\u662f\u91c7\u96c6\u67d0\u4e2a\u5546\u4e1a\u7cfb\u7edf&#xff0c;\u4e5f\u4e0d\u662f\u722c\u53d6\u53d7\u9650\u6570\u636e\u3002\u6211\u4eec\u9009\u62e9\u7684\u601d\u8def\u662f&#xff1a;<\/p>\n<ul>\n<li>\u76ee\u6807\u5e73\u53f0&#xff1a;\u516c\u5f00\u6d77\u6d0b\u751f\u7269\u5206\u5e03\u6570\u636e\u5e73\u53f0&#xff1b;<\/li>\n<li>\u76ee\u6807\u63a5\u53e3&#xff1a;occurrence \u67e5\u8be2\u63a5\u53e3&#xff1b;<\/li>\n<li>\u91c7\u96c6\u5bf9\u8c61&#xff1a;\u6307\u5b9a\u7269\u79cd\u5728\u6307\u5b9a\u6d77\u57df\u3001\u6307\u5b9a\u65f6\u95f4\u8303\u56f4\u5185\u7684\u51fa\u73b0\u8bb0\u5f55&#xff1b;<\/li>\n<li>\u65f6\u95f4\u7c92\u5ea6&#xff1a;\u6309\u6708\u62c6\u5206&#xff1b;<\/li>\n<li>\u7a7a\u95f4\u7c92\u5ea6&#xff1a;\u6309\u7ecf\u7eac\u5ea6\u7f51\u683c\u805a\u5408&#xff1b;<\/li>\n<li>\u8f93\u51fa\u5f62\u5f0f&#xff1a;CSV\u3001SQLite\u3001GeoJSON\u3001HTML \u52a8\u6001\u65f6\u95f4\u8f74\u5730\u56fe\u3002<\/li>\n<\/ul>\n<p>\u672c\u6587\u8bbe\u5b9a\u4e00\u4e2a\u793a\u4f8b\u4efb\u52a1&#xff1a;<\/p>\n<p>\u91c7\u96c6 Gadus morhua \u8fd9\u4e2a\u7269\u79cd\u5728\u5317\u5927\u897f\u6d0b\u90e8\u5206\u6d77\u57df\u4e2d&#xff0c;2020 \u5e74 1 \u6708\u5230 2020 \u5e74 12 \u6708\u4e4b\u95f4\u7684\u51fa\u73b0\u8bb0\u5f55&#xff0c;\u5e76\u6309 1 \u5ea6\u7ecf\u7eac\u5ea6\u7f51\u683c\u805a\u5408\u4e3a\u6708\u5ea6\u5f3a\u5ea6\u6307\u6807\u3002<\/p>\n<p>\u8fd9\u91cc\u7684 Gadus morhua \u662f\u5927\u897f\u6d0b\u9cd5\u7684\u5b66\u540d\u3002\u5b9e\u9645\u9879\u76ee\u91cc&#xff0c;\u4f60\u53ef\u4ee5\u628a\u5b83\u66ff\u6362\u6210\u5176\u4ed6\u7269\u79cd\u540d&#xff0c;\u4e5f\u53ef\u4ee5\u628a\u4e00\u4e2a\u7269\u79cd\u6269\u5c55\u6210\u591a\u4e2a\u7269\u79cd\u6279\u91cf\u91c7\u96c6\u3002<\/p>\n<h4>2.3 \u76ee\u6807\u5b57\u6bb5\u6e05\u5355<\/h4>\n<p>\u672c\u6587\u6700\u7ec8\u5e0c\u671b\u62ff\u5230\u5982\u4e0b\u5b57\u6bb5\u3002<\/p>\n<p>\u6838\u5fc3\u4e1a\u52a1\u5b57\u6bb5&#xff1a;<\/p>\n<table>\n<tr>\u5b57\u6bb5\u542b\u4e49\u793a\u4f8b<\/tr>\n<tbody>\n<tr>\n<td>species_name<\/td>\n<td>\u7269\u79cd\/\u8d44\u6e90\u540d<\/td>\n<td>Gadus morhua<\/td>\n<\/tr>\n<tr>\n<td>sea_area<\/td>\n<td>\u6d77\u57df<\/td>\n<td>North Atlantic Demo Area<\/td>\n<\/tr>\n<tr>\n<td>time_window<\/td>\n<td>\u65f6\u95f4<\/td>\n<td>2020-01<\/td>\n<\/tr>\n<tr>\n<td>intensity_index<\/td>\n<td>\u5f3a\u5ea6\u6307\u6807<\/td>\n<td>18<\/td>\n<\/tr>\n<tr>\n<td>source<\/td>\n<td>\u6765\u6e90<\/td>\n<td>OBIS API<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>\u7a7a\u95f4\u5b57\u6bb5&#xff1a;<\/p>\n<table>\n<tr>\u5b57\u6bb5\u542b\u4e49\u793a\u4f8b<\/tr>\n<tbody>\n<tr>\n<td>longitude<\/td>\n<td>\u7ecf\u5ea6<\/td>\n<td>-63.25<\/td>\n<\/tr>\n<tr>\n<td>latitude<\/td>\n<td>\u7eac\u5ea6<\/td>\n<td>44.71<\/td>\n<\/tr>\n<tr>\n<td>grid_id<\/td>\n<td>\u7f51\u683c\u7f16\u53f7<\/td>\n<td>lon_-64_lat_44<\/td>\n<\/tr>\n<tr>\n<td>grid_lon<\/td>\n<td>\u7f51\u683c\u5de6\u4e0b\u89d2\u7ecf\u5ea6<\/td>\n<td>-64<\/td>\n<\/tr>\n<tr>\n<td>grid_lat<\/td>\n<td>\u7f51\u683c\u5de6\u4e0b\u89d2\u7eac\u5ea6<\/td>\n<td>44<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>\u6eaf\u6e90\u5b57\u6bb5&#xff1a;<\/p>\n<table>\n<tr>\u5b57\u6bb5\u542b\u4e49\u793a\u4f8b<\/tr>\n<tbody>\n<tr>\n<td>occurrence_id<\/td>\n<td>\u539f\u59cb\u8bb0\u5f55 ID<\/td>\n<td>\u516c\u5f00\u63a5\u53e3\u8fd4\u56de\u7684\u8bb0\u5f55\u6807\u8bc6<\/td>\n<\/tr>\n<tr>\n<td>event_date<\/td>\n<td>\u539f\u59cb\u89c2\u6d4b\u65f6\u95f4<\/td>\n<td>2020-03-12<\/td>\n<\/tr>\n<tr>\n<td>dataset_id<\/td>\n<td>\u6570\u636e\u96c6 ID<\/td>\n<td>\u6570\u636e\u63d0\u4f9b\u65b9\u76f8\u5173\u6807\u8bc6<\/td>\n<\/tr>\n<tr>\n<td>basis_of_record<\/td>\n<td>\u8bb0\u5f55\u7c7b\u578b<\/td>\n<td>HumanObservation<\/td>\n<\/tr>\n<tr>\n<td>fetched_at<\/td>\n<td>\u91c7\u96c6\u65f6\u95f4<\/td>\n<td>2026-06-12T10:30:00Z<\/td>\n<\/tr>\n<tr>\n<td>request_url<\/td>\n<td>\u8bf7\u6c42 URL<\/td>\n<td>https:\/\/api.example\/\u2026<\/td>\n<\/tr>\n<tr>\n<td>content_hash<\/td>\n<td>\u5185\u5bb9 hash<\/td>\n<td>sha256\u2026<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>\u5f3a\u5ea6\u6307\u6807\u5728\u672c\u6587\u4e2d\u5b9a\u4e49\u4e3a&#xff1a;\u540c\u4e00\u7269\u79cd\u3001\u540c\u4e00\u6d77\u57df\u3001\u540c\u4e00\u6708\u4efd\u3001\u540c\u4e00\u7f51\u683c\u5185\u7684\u6709\u6548 occurrence \u8bb0\u5f55\u6570\u91cf\u3002\u8fd9\u4e2a\u6307\u6807\u4e0d\u662f\u5b98\u65b9\u8d44\u6e90\u91cf\u4f30\u8ba1&#xff0c;\u4e5f\u4e0d\u662f\u6e14\u83b7\u91cf&#xff0c;\u53ea\u662f\u4e00\u4e2a\u57fa\u4e8e\u516c\u5f00\u8bb0\u5f55\u7684\u76f8\u5bf9\u51fa\u73b0\u5f3a\u5ea6\u3002\u8fd9\u4e2a\u5b9a\u4e49\u5fc5\u987b\u5199\u6e05\u695a&#xff0c;\u5426\u5219\u5bb9\u6613\u88ab\u8bef\u89e3\u4e3a\u201c\u771f\u5b9e\u8d44\u6e90\u50a8\u91cf\u201d\u3002<\/p>\n<hr \/>\n<h3>3\ufe0f\u20e3 \u5408\u89c4\u4e0e\u6ce8\u610f\u4e8b\u9879&#xff08;\u5fc5\u5199&#xff09;<\/h3>\n<p>\u722c\u866b\u6280\u672f\u672c\u8eab\u6ca1\u6709\u95ee\u9898&#xff0c;\u4f46\u4f7f\u7528\u65b9\u5f0f\u5fc5\u987b\u514b\u5236\u3002\u5c24\u5176\u662f\u5730\u56fe\u56fe\u5c42\u3001\u65f6\u7a7a\u6570\u636e\u3001\u79d1\u5b66\u6570\u636e\u5e73\u53f0&#xff0c;\u5f88\u591a\u7531\u516c\u5171\u673a\u6784\u3001\u7814\u7a76\u7ec4\u7ec7\u6216\u5fd7\u613f\u8005\u5171\u540c\u7ef4\u62a4&#xff0c;\u8bbf\u95ee\u65f6\u8981\u5c0a\u91cd\u5bf9\u65b9\u670d\u52a1\u80fd\u529b\u548c\u6570\u636e\u653f\u7b56\u3002<\/p>\n<h4>3.1 robots.txt \u57fa\u672c\u8bf4\u660e<\/h4>\n<p>robots.txt \u662f\u7f51\u7ad9\u653e\u5728\u6839\u76ee\u5f55\u4e0b\u7684\u4e00\u4e2a\u6587\u672c\u6587\u4ef6&#xff0c;\u7528\u6765\u58f0\u660e\u54ea\u4e9b\u8def\u5f84\u5141\u8bb8\u6216\u4e0d\u5efa\u8bae\u88ab\u81ea\u52a8\u5316\u7a0b\u5e8f\u8bbf\u95ee\u3002\u5b83\u4e0d\u662f\u6cd5\u5f8b\u5408\u540c&#xff0c;\u4e5f\u4e0d\u662f\u5b89\u5168\u8fb9\u754c&#xff0c;\u4f46\u5b83\u662f\u6700\u57fa\u672c\u7684\u7f51\u7edc\u793c\u4eea\u3002<\/p>\n<p>\u5728\u6b63\u5f0f\u91c7\u96c6\u524d&#xff0c;\u5efa\u8bae\u505a\u4e09\u4ef6\u4e8b&#xff1a;<\/p>\n<li>\u8bbf\u95ee\u76ee\u6807\u7ad9\u70b9\u6839\u76ee\u5f55\u4e0b\u7684 robots.txt&#xff1b;<\/li>\n<li>\u67e5\u770b\u76ee\u6807\u63a5\u53e3\u8def\u5f84\u662f\u5426\u88ab\u7981\u6b62&#xff1b;<\/li>\n<li>\u5982\u679c\u6587\u6863\u660e\u786e\u63d0\u4f9b API&#xff0c;\u4f18\u5148\u4f7f\u7528 API&#xff0c;\u800c\u4e0d\u662f\u66b4\u529b\u6293\u9875\u9762\u3002<\/li>\n<p>\u793a\u4f8b\u68c0\u67e5\u65b9\u5f0f&#xff1a;<\/p>\n<p><span class=\"token keyword\">import<\/span> requests<\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">check_robots<\/span><span class=\"token punctuation\">(<\/span>base_url<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    url <span class=\"token operator\">&#061;<\/span> base_url<span class=\"token punctuation\">.<\/span>rstrip<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;\/&#034;<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#043;<\/span> <span class=\"token string\">&#034;\/robots.txt&#034;<\/span><br \/>\n    resp <span class=\"token operator\">&#061;<\/span> requests<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span>url<span class=\"token punctuation\">,<\/span> timeout<span class=\"token operator\">&#061;<\/span><span class=\"token number\">15<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    <span class=\"token keyword\">print<\/span><span class=\"token punctuation\">(<\/span>resp<span class=\"token punctuation\">.<\/span>status_code<span class=\"token punctuation\">)<\/span><br \/>\n    <span class=\"token keyword\">print<\/span><span class=\"token punctuation\">(<\/span>resp<span class=\"token punctuation\">.<\/span>text<span class=\"token punctuation\">[<\/span><span class=\"token punctuation\">:<\/span><span class=\"token number\">2000<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p><span class=\"token keyword\">if<\/span> __name__ <span class=\"token operator\">&#061;&#061;<\/span> <span class=\"token string\">&#034;__main__&#034;<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    check_robots<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;https:\/\/api.obis.org&#034;<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>\u5982\u679c robots.txt \u6216\u5b98\u65b9\u6587\u6863\u7ed9\u51fa\u66f4\u5177\u4f53\u7684\u8bbf\u95ee\u8bf4\u660e&#xff0c;\u5e94\u4ee5\u5b98\u65b9\u8bf4\u660e\u4e3a\u51c6\u3002\u6ca1\u6709\u770b\u5230\u9650\u5236&#xff0c;\u4e0d\u4ee3\u8868\u53ef\u4ee5\u65e0\u9650\u5236\u5e76\u53d1&#xff1b;\u770b\u5230\u516c\u5f00\u63a5\u53e3&#xff0c;\u4e5f\u4e0d\u4ee3\u8868\u53ef\u4ee5\u628a\u5bf9\u65b9\u670d\u52a1\u5f53\u4f5c\u81ea\u5df1\u7684\u79bb\u7ebf\u6570\u636e\u5e93\u6765\u9ad8\u9891\u626b\u3002<\/p>\n<h4>3.2 \u9891\u7387\u63a7\u5236<\/h4>\n<p>\u52a8\u6001\u5730\u56fe\u63a5\u53e3\u6700\u5bb9\u6613\u88ab\u8bef\u7528\u3002\u5f88\u591a\u4eba\u6253\u5f00\u5f00\u53d1\u8005\u5de5\u5177\u770b\u5230\u4e00\u4e2a\u63a5\u53e3&#xff0c;\u5c31\u9a6c\u4e0a\u5199\u5faa\u73af&#xff0c;\u628a\u65f6\u95f4\u3001\u7ecf\u7eac\u5ea6\u3001\u7269\u79cd\u3001\u5206\u9875\u5168\u90e8\u7ec4\u5408\u8d77\u6765\u9ad8\u5e76\u53d1\u8bf7\u6c42\u3002\u8fd9\u6837\u505a\u5f88\u4e0d\u4e13\u4e1a&#xff0c;\u4e5f\u5bb9\u6613\u7ed9\u516c\u5171\u670d\u52a1\u9020\u6210\u538b\u529b\u3002<\/p>\n<p>\u672c\u6587\u4ee3\u7801\u91c7\u7528\u6e29\u548c\u7b56\u7565&#xff1a;<\/p>\n<ul>\n<li>\u6bcf\u6b21\u8bf7\u6c42\u8bbe\u7f6e timeout&#xff1b;<\/li>\n<li>\u4f7f\u7528 Session \u590d\u7528\u8fde\u63a5&#xff1b;<\/li>\n<li>\u5bf9\u5931\u8d25\u8bf7\u6c42\u505a\u6709\u9650\u91cd\u8bd5&#xff1b;<\/li>\n<li>\u5bf9 429\u3001503 \u7b49\u72b6\u6001\u7801\u91c7\u7528\u9000\u907f\u7b49\u5f85&#xff1b;<\/li>\n<li>\u9ed8\u8ba4\u6bcf\u6b21\u8bf7\u6c42\u540e sleep \u4e00\u5c0f\u6bb5\u65f6\u95f4&#xff1b;<\/li>\n<li>\u4e0d\u4f7f\u7528\u653b\u51fb\u5f0f\u5e76\u53d1&#xff1b;<\/li>\n<li>\u9ed8\u8ba4\u5206\u9875\u4e0a\u9650\u53ef\u914d\u7f6e&#xff0c;\u907f\u514d\u8bef\u91c7\u8d85\u5927\u7ed3\u679c\u96c6\u3002<\/li>\n<\/ul>\n<p>\u5bf9\u4e8e\u516c\u5f00\u79d1\u5b66\u6570\u636e\u5e73\u53f0&#xff0c;\u5efa\u8bae\u5148\u5c0f\u6837\u672c\u9a8c\u8bc1\u903b\u8f91&#xff0c;\u518d\u6269\u5927\u8303\u56f4\u3002\u5982\u679c\u9700\u8981\u5927\u89c4\u6a21\u5206\u6790&#xff0c;\u5e94\u4f7f\u7528\u5b98\u65b9\u63a8\u8350\u7684\u5927\u6570\u636e\u5bfc\u51fa\u65b9\u5f0f&#xff0c;\u800c\u4e0d\u662f\u7528 API \u6162\u6162\u7ffb\u9875\u786c\u6293\u3002<\/p>\n<h4>3.3 \u4e0d\u91c7\u96c6\u654f\u611f\u4fe1\u606f<\/h4>\n<p>\u672c\u6587\u91c7\u96c6\u7684\u662f\u516c\u5f00\u6d77\u6d0b\u751f\u7269 occurrence \u6570\u636e&#xff0c;\u4e0d\u91c7\u96c6\u4e2a\u4eba\u9690\u79c1\u3001\u8d26\u53f7\u4fe1\u606f\u3001\u540e\u53f0\u6570\u636e\u3001\u767b\u5f55\u6001\u4fe1\u606f&#xff0c;\u4e5f\u4e0d\u7ed5\u8fc7\u4ed8\u8d39\u5899\u6216\u6743\u9650\u63a7\u5236\u3002<\/p>\n<p>\u5982\u679c\u76ee\u6807\u7f51\u7ad9\u9700\u8981\u767b\u5f55\u624d\u80fd\u8bbf\u95ee&#xff0c;\u6216\u8005\u63a5\u53e3\u9700\u8981\u6388\u6743 token&#xff0c;\u5e94\u6309\u7167\u5b98\u65b9\u6388\u6743\u6d41\u7a0b\u4f7f\u7528\u3002\u4e0d\u8981\u901a\u8fc7\u6293\u5305\u590d\u5236\u4ed6\u4eba token&#xff0c;\u4e0d\u8981\u6a21\u62df\u975e\u516c\u5f00\u63a5\u53e3&#xff0c;\u4e0d\u8981\u7ed5\u8fc7\u8bbf\u95ee\u9650\u5236\u3002\u6280\u672f\u4e0a\u80fd\u505a\u5230&#xff0c;\u4e0d\u4ee3\u8868\u5e94\u8be5\u505a\u3002<\/p>\n<h4>3.4 \u6570\u636e\u89e3\u91ca\u8981\u8c28\u614e<\/h4>\n<p>\u516c\u5f00 occurrence \u6570\u636e\u901a\u5e38\u662f\u201c\u8bb0\u5f55\u8fc7\u67d0\u7269\u79cd\u5728\u67d0\u5904\u51fa\u73b0\u201d&#xff0c;\u4f46\u5b83\u5e76\u4e0d\u7b49\u4ef7\u4e8e\u771f\u5b9e\u8d44\u6e90\u91cf\u3002\u67d0\u4e2a\u7f51\u683c\u8bb0\u5f55\u591a&#xff0c;\u53ef\u80fd\u662f\u8d44\u6e90\u4e30\u5bcc&#xff0c;\u4e5f\u53ef\u80fd\u53ea\u662f\u91c7\u6837\u66f4\u9891\u7e41\u3002\u67d0\u4e2a\u6708\u8bb0\u5f55\u5c11&#xff0c;\u53ef\u80fd\u662f\u8d44\u6e90\u5c11&#xff0c;\u4e5f\u53ef\u80fd\u662f\u6ca1\u6709\u8c03\u67e5\u6d3b\u52a8\u3002<\/p>\n<p>\u6240\u4ee5\u672c\u6587\u7684 intensity_index \u53ea\u5b9a\u4e49\u4e3a\u8bb0\u5f55\u6570\u91cf\u805a\u5408&#xff0c;\u4e0d\u628a\u5b83\u5305\u88c5\u6210\u771f\u5b9e\u50a8\u91cf\u3001\u6355\u635e\u91cf\u6216\u751f\u6001\u7ed3\u8bba\u3002\u6570\u636e\u5de5\u7a0b\u53ef\u4ee5\u6574\u7406\u6750\u6599&#xff0c;\u79d1\u5b66\u89e3\u91ca\u8981\u4ea4\u7ed9\u66f4\u4e25\u8c28\u7684\u5206\u6790\u6a21\u578b\u548c\u9886\u57df\u77e5\u8bc6\u3002<\/p>\n<hr \/>\n<h3>4\ufe0f\u20e3 \u6280\u672f\u9009\u578b\u4e0e\u6574\u4f53\u6d41\u7a0b&#xff08;What\/How&#xff09;<\/h3>\n<h4>4.1 \u9759\u6001\u3001\u52a8\u6001\u3001API \u4e09\u7c7b\u91c7\u96c6\u65b9\u5f0f<\/h4>\n<p>\u5e38\u89c1\u7f51\u9875\u91c7\u96c6\u53ef\u4ee5\u7c97\u7565\u5206\u6210\u4e09\u7c7b\u3002<\/p>\n<p>\u7b2c\u4e00\u7c7b\u662f\u9759\u6001\u9875\u9762\u3002HTML \u91cc\u5df2\u7ecf\u5305\u542b\u76ee\u6807\u6570\u636e&#xff0c;\u7528 requests \u6293\u9875\u9762&#xff0c;\u518d\u7528 BeautifulSoup\u3001lxml\u3001XPath \u6216 CSS \u9009\u62e9\u5668\u89e3\u6790\u5373\u53ef\u3002\u6bd4\u5982\u65b0\u95fb\u6807\u9898\u5217\u8868\u3001\u666e\u901a\u8868\u683c\u3001\u516c\u544a\u8be6\u60c5\u9875&#xff0c;\u5927\u591a\u5c5e\u4e8e\u8fd9\u4e00\u7c7b\u3002<\/p>\n<p>\u7b2c\u4e8c\u7c7b\u662f\u52a8\u6001\u9875\u9762\u3002\u9875\u9762\u521d\u59cb HTML \u53ea\u662f\u4e00\u4e2a\u7a7a\u58f3&#xff0c;\u771f\u6b63\u7684\u6570\u636e\u7531 JavaScript \u5728\u6d4f\u89c8\u5668\u91cc\u8bf7\u6c42\u63a5\u53e3\u540e\u6e32\u67d3\u3002\u5730\u56fe\u56fe\u5c42\u3001\u65f6\u95f4\u8f74\u52a8\u753b\u3001\u65e0\u9650\u6eda\u52a8\u5217\u8868\u3001\u524d\u7aef\u770b\u677f&#xff0c;\u7ecf\u5e38\u5c5e\u4e8e\u8fd9\u4e00\u7c7b\u3002\u5982\u679c\u76f4\u63a5 requests \u6293 HTML&#xff0c;\u901a\u5e38\u53ea\u80fd\u770b\u5230\u4e00\u5806 script&#xff0c;\u770b\u4e0d\u5230\u6570\u636e\u3002<\/p>\n<p>\u7b2c\u4e09\u7c7b\u662f\u516c\u5f00 API\u3002\u524d\u7aef\u9875\u9762\u4e5f\u662f\u8c03\u7528 API&#xff0c;\u4f46\u5e73\u53f0\u672c\u8eab\u63d0\u4f9b\u6587\u6863\u3001\u53c2\u6570\u8bf4\u660e\u548c\u7ed3\u6784\u5316\u54cd\u5e94\u3002\u8fd9\u79cd\u60c5\u51b5\u4e0b\u5e94\u8be5\u4f18\u5148\u4f7f\u7528 API&#xff0c;\u800c\u4e0d\u662f\u6a21\u62df\u6d4f\u89c8\u5668\u70b9\u6765\u70b9\u53bb\u3002API \u901a\u5e38\u66f4\u7a33\u5b9a&#xff0c;\u5b57\u6bb5\u66f4\u89c4\u8303&#xff0c;\u4e5f\u66f4\u9002\u5408\u5199\u6210\u53ef\u7ef4\u62a4\u7684\u9879\u76ee\u3002<\/p>\n<p>\u672c\u6587\u5c5e\u4e8e\u201c\u52a8\u6001\u5730\u56fe &#043; API \u91c7\u96c6\u201d\u3002\u6211\u4eec\u4e0d\u662f\u53bb\u89e3\u6790\u5730\u56fe\u753b\u5e03\u4e0a\u7684\u50cf\u7d20&#xff0c;\u4e5f\u4e0d\u662f\u4ece HTML \u91cc\u62a0\u6570\u636e&#xff0c;\u800c\u662f\u8bc6\u522b\u5730\u56fe\u80cc\u540e\u7684 occurrence \u67e5\u8be2\u63a5\u53e3&#xff0c;\u7528\u65f6\u95f4\u7a97\u53e3\u548c\u7a7a\u95f4\u8303\u56f4\u6784\u9020\u8bf7\u6c42&#xff0c;\u518d\u5bf9\u54cd\u5e94 JSON \u505a\u89e3\u6790\u548c\u805a\u5408\u3002<\/p>\n<h4>4.2 \u52a8\u6001\u56fe\u5c42\u91c7\u96c6\u7684\u6838\u5fc3\u601d\u8def<\/h4>\n<p>\u52a8\u6001\u5730\u56fe\u91c7\u96c6\u7684\u5173\u952e\u4e0d\u662f\u201c\u5730\u56fe\u201d&#xff0c;\u800c\u662f\u201c\u56fe\u5c42\u8bf7\u6c42\u201d\u3002<\/p>\n<p>\u4e00\u822c\u53ef\u4ee5\u6309\u8fd9\u4e2a\u6d41\u7a0b\u6392\u67e5&#xff1a;<\/p>\n<li>\u6253\u5f00\u76ee\u6807\u5730\u56fe\u9875\u9762&#xff1b;<\/li>\n<li>\u6253\u5f00\u6d4f\u89c8\u5668\u5f00\u53d1\u8005\u5de5\u5177&#xff1b;<\/li>\n<li>\u5207\u6362\u5230 Network \u9762\u677f&#xff1b;<\/li>\n<li>\u5728\u9875\u9762\u91cc\u9009\u62e9\u7269\u79cd\u3001\u6d77\u57df\u3001\u65f6\u95f4\u8303\u56f4&#xff1b;<\/li>\n<li>\u62d6\u52a8\u65f6\u95f4\u8f74\u6216\u6539\u53d8\u56fe\u5c42&#xff1b;<\/li>\n<li>\u89c2\u5bdf\u65b0\u51fa\u73b0\u7684 XHR\u3001Fetch\u3001MVT\u3001GeoJSON\u3001JSON \u8bf7\u6c42&#xff1b;<\/li>\n<li>\u627e\u5230\u643a\u5e26 species\u3001scientificname\u3001startdate\u3001enddate\u3001geometry\u3001bbox\u3001tile\u3001offset \u7b49\u53c2\u6570\u7684\u8bf7\u6c42&#xff1b;<\/li>\n<li>\u590d\u5236\u8bf7\u6c42 URL&#xff0c;\u5728\u6d4f\u89c8\u5668\u6216 Postman \u4e2d\u9a8c\u8bc1&#xff1b;<\/li>\n<li>\u7528 Python requests \u91cd\u653e&#xff1b;<\/li>\n<li>\u5c06\u65f6\u95f4\u8f74\u62c6\u6210\u591a\u6b21\u8bf7\u6c42\u3002<\/li>\n<p>\u5bf9\u4e8e\u672c\u6587\u6848\u4f8b&#xff0c;\u6211\u4eec\u628a\u65f6\u95f4\u8f74\u62c6\u6210\u6708\u5ea6\u7a97\u53e3&#xff1a;<\/p>\n<p>2020-01-01 ~ 2020-01-31<br \/>\n2020-02-01 ~ 2020-02-29<br \/>\n2020-03-01 ~ 2020-03-31<br \/>\n&#8230;<br \/>\n2020-12-01 ~ 2020-12-31<\/p>\n<p>\u6bcf\u4e2a\u7a97\u53e3\u5185\u6309\u7167\u5206\u9875\u53c2\u6570\u6293\u53d6 occurrence \u8bb0\u5f55\u3002\u62ff\u5230\u539f\u59cb\u8bb0\u5f55\u540e&#xff0c;\u6839\u636e\u7ecf\u7eac\u5ea6\u843d\u5230\u7f51\u683c&#xff0c;\u518d\u7edf\u8ba1\u6bcf\u4e2a\u7f51\u683c\u7684\u8bb0\u5f55\u6570\u91cf\u3002<\/p>\n<h4>4.3 \u6574\u4f53\u6d41\u7a0b<\/h4>\n<p>\u6587\u5b57\u7248\u6d41\u7a0b\u5982\u4e0b&#xff1a;<\/p>\n<p>\u914d\u7f6e\u7269\u79cd\u3001\u6d77\u57df WKT\u3001\u591a\u4e2a\u6708\u4efd<br \/>\n        \u2193<br \/>\n\u8bf7\u6c42 occurrence API<br \/>\n        \u2193<br \/>\n\u6309 offset \/ size \u5206\u9875<br \/>\n        \u2193<br \/>\n\u89e3\u6790 JSON records<br \/>\n        \u2193<br \/>\n\u8fc7\u6ee4\u7f3a\u5931\u7ecf\u7eac\u5ea6\u3001\u7f3a\u5931\u65f6\u95f4\u3001\u65e0\u6548\u8bb0\u5f55<br \/>\n        \u2193<br \/>\n\u5199\u5165 SQLite \u539f\u59cb\u8868<br \/>\n        \u2193<br \/>\n\u6309 species &#043; sea_area &#043; month &#043; grid \u805a\u5408<br \/>\n        \u2193<br \/>\n\u5bfc\u51fa CSV \/ GeoJSON<br \/>\n        \u2193<br \/>\n\u751f\u6210\u5e26\u65f6\u95f4\u8f74\u7684 HTML \u5730\u56fe<\/p>\n<p>\u66f4\u50cf\u5de5\u7a0b\u9879\u76ee\u7684\u6d41\u7a0b\u5982\u4e0b&#xff1a;<\/p>\n<p>\u91c7\u96c6 Fetcher<br \/>\n  &#8211; \u8d1f\u8d23\u8bf7\u6c42\u63a5\u53e3<br \/>\n  &#8211; \u8d1f\u8d23 headers\u3001timeout\u3001retry\u3001rate limit<br \/>\n  &#8211; \u8d1f\u8d23\u5206\u9875<\/p>\n<p>\u89e3\u6790 Parser<br \/>\n  &#8211; \u8d1f\u8d23 JSON \u5b57\u6bb5\u6620\u5c04<br \/>\n  &#8211; \u8d1f\u8d23\u65f6\u95f4\u683c\u5f0f\u3001\u7ecf\u7eac\u5ea6\u3001\u7269\u79cd\u540d\u63d0\u53d6<br \/>\n  &#8211; \u8d1f\u8d23\u5bb9\u9519<\/p>\n<p>\u6e05\u6d17 Cleaner<br \/>\n  &#8211; \u53bb\u6389\u7f3a\u5931\u5750\u6807<br \/>\n  &#8211; \u53bb\u6389\u5f02\u5e38\u7ecf\u7eac\u5ea6<br \/>\n  &#8211; \u7edf\u4e00\u6708\u4efd<br \/>\n  &#8211; \u751f\u6210 grid_id<\/p>\n<p>\u5b58\u50a8 Storage<br \/>\n  &#8211; \u4fdd\u5b58 raw_occurrences<br \/>\n  &#8211; \u4fdd\u5b58 resource_intensity<br \/>\n  &#8211; \u4fdd\u5b58 request_log<br \/>\n  &#8211; \u53bb\u91cd<\/p>\n<p>\u5bfc\u51fa Exporter<br \/>\n  &#8211; \u5bfc\u51fa CSV<br \/>\n  &#8211; \u5bfc\u51fa GeoJSON<br \/>\n  &#8211; \u751f\u6210\u65f6\u95f4\u8f74 HTML \u5730\u56fe<\/p>\n<h4>4.4 \u4e3a\u4ec0\u4e48\u9009 requests&#xff0c;\u800c\u4e0d\u662f\u4e00\u4e0a\u6765\u5c31 Playwright<\/h4>\n<p>\u5f88\u591a\u52a8\u6001\u7f51\u9875\u786e\u5b9e\u9700\u8981 Playwright \u6216 Selenium&#xff0c;\u4f46\u672c\u6587\u4e0d\u4f18\u5148\u9009\u5b83\u4eec&#xff0c;\u539f\u56e0\u5f88\u7b80\u5355&#xff1a;\u5982\u679c\u5df2\u7ecf\u5b9a\u4f4d\u5230\u516c\u5f00\u63a5\u53e3&#xff0c;\u76f4\u63a5\u8bf7\u6c42\u63a5\u53e3\u66f4\u7a33\u5b9a\u3001\u66f4\u8f7b\u91cf\u3001\u66f4\u5bb9\u6613\u6d4b\u8bd5\u3002<\/p>\n<p>Playwright \u9002\u5408\u8fd9\u4e9b\u60c5\u51b5&#xff1a;<\/p>\n<ul>\n<li>\u9875\u9762\u52a0\u5bc6\u5f88\u91cd&#xff0c;\u63a5\u53e3\u53c2\u6570\u7531\u524d\u7aef\u8fd0\u884c\u65f6\u751f\u6210&#xff1b;<\/li>\n<li>\u5fc5\u987b\u6267\u884c JavaScript \u624d\u80fd\u62ff\u5230 token&#xff1b;<\/li>\n<li>\u6570\u636e\u53ea\u5b58\u5728\u6d4f\u89c8\u5668\u6e32\u67d3\u540e\u7684 DOM \u4e2d&#xff1b;<\/li>\n<li>\u9700\u8981\u622a\u56fe\u3001\u70b9\u51fb\u3001\u6eda\u52a8\u3001\u6a21\u62df\u590d\u6742\u4ea4\u4e92\u3002<\/li>\n<\/ul>\n<p>\u672c\u6587\u7684\u76ee\u6807\u662f\u63a5\u53e3\u6570\u636e\u91c7\u96c6&#xff0c;\u6240\u4ee5\u9009\u62e9&#xff1a;<\/p>\n<ul>\n<li>requests&#xff1a;\u5904\u7406 HTTP \u8bf7\u6c42&#xff1b;<\/li>\n<li>pandas&#xff1a;\u5904\u7406\u8868\u683c\u6e05\u6d17\u548c\u5bfc\u51fa&#xff1b;<\/li>\n<li>SQLite&#xff1a;\u672c\u5730\u7ed3\u6784\u5316\u5b58\u50a8&#xff1b;<\/li>\n<li>Folium&#xff1a;\u751f\u6210 Leaflet \u5730\u56fe&#xff1b;<\/li>\n<li>tenacity&#xff1a;\u505a\u91cd\u8bd5\u548c\u9000\u907f&#xff1b;<\/li>\n<li>tqdm&#xff1a;\u5c55\u793a\u8fdb\u5ea6&#xff1b;<\/li>\n<li>python-dotenv&#xff1a;\u7ba1\u7406\u914d\u7f6e\u3002<\/li>\n<\/ul>\n<p>\u8fd9\u5957\u7ec4\u5408\u4e0d\u82b1\u54e8&#xff0c;\u4f46\u975e\u5e38\u5b9e\u7528\u3002\u9879\u76ee\u540e\u7eed\u8981\u5347\u7ea7\u5230 Scrapy\u3001asyncio \u6216 Airflow&#xff0c;\u4e5f\u4e0d\u4f1a\u63a8\u5012\u91cd\u6765\u3002<\/p>\n<hr \/>\n<h3>5\ufe0f\u20e3 \u73af\u5883\u51c6\u5907\u4e0e\u4f9d\u8d56\u5b89\u88c5&#xff08;\u53ef\u590d\u73b0&#xff09;<\/h3>\n<h4>5.1 Python \u7248\u672c<\/h4>\n<p>\u5efa\u8bae\u4f7f\u7528 Python 3.10 \u6216\u4ee5\u4e0a\u7248\u672c\u3002\u672c\u6587\u793a\u4f8b\u5728 Python 3.10&#043; \u8bed\u6cd5\u4e0b\u7f16\u5199&#xff0c;\u4e3b\u8981\u7528\u5230 dataclass\u3001\u7c7b\u578b\u6ce8\u89e3\u548c pathlib\u3002<\/p>\n<p>\u67e5\u770b\u7248\u672c&#xff1a;<\/p>\n<p>python <span class=\"token parameter variable\">&#8211;version<\/span><\/p>\n<p>\u5efa\u8bae\u8f93\u51fa\u7c7b\u4f3c&#xff1a;<\/p>\n<p>Python <span class=\"token number\">3.10<\/span>.13<\/p>\n<h4>5.2 \u521b\u5efa\u865a\u62df\u73af\u5883<\/h4>\n<p>macOS \/ Linux&#xff1a;<\/p>\n<p><span class=\"token function\">mkdir<\/span> fisheries_dynamic_obis<br \/>\n<span class=\"token builtin class-name\">cd<\/span> fisheries_dynamic_obis<\/p>\n<p>python <span class=\"token parameter variable\">-m<\/span> venv .venv<br \/>\n<span class=\"token builtin class-name\">source<\/span> .venv\/bin\/activate<\/p>\n<p>Windows PowerShell&#xff1a;<\/p>\n<p>mkdir fisheries_dynamic_obis<br \/>\ncd fisheries_dynamic_obis<\/p>\n<p>python <span class=\"token operator\">&#8211;<\/span>m venv <span class=\"token punctuation\">.<\/span>venv<br \/>\n<span class=\"token punctuation\">.<\/span>venv\\\\Scripts\\\\Activate<span class=\"token punctuation\">.<\/span>ps1<\/p>\n<h4>5.3 \u5b89\u88c5\u4f9d\u8d56<\/h4>\n<p>\u521b\u5efa requirements.txt&#xff1a;<\/p>\n<p>requests&#061;&#061;2.32.3<br \/>\npandas&#061;&#061;2.2.2<br \/>\ntenacity&#061;&#061;8.5.0<br \/>\npython-dotenv&#061;&#061;1.0.1<br \/>\ntqdm&#061;&#061;4.66.5<br \/>\nfolium&#061;&#061;0.17.0<br \/>\nbranca&#061;&#061;0.7.2<\/p>\n<p>\u5b89\u88c5&#xff1a;<\/p>\n<p>pip <span class=\"token function\">install<\/span> <span class=\"token parameter variable\">-r<\/span> requirements.txt<\/p>\n<h4>5.4 \u63a8\u8350\u9879\u76ee\u7ed3\u6784<\/h4>\n<p>fisheries_dynamic_obis\/<br \/>\n\u251c\u2500\u2500 README.md<br \/>\n\u251c\u2500\u2500 requirements.txt<br \/>\n\u251c\u2500\u2500 .env.example<br \/>\n\u251c\u2500\u2500 data\/<br \/>\n\u2502   \u251c\u2500\u2500 raw\/<br \/>\n\u2502   \u251c\u2500\u2500 processed\/<br \/>\n\u2502   \u2514\u2500\u2500 maps\/<br \/>\n\u251c\u2500\u2500 logs\/<br \/>\n\u251c\u2500\u2500 fisheries_spider\/<br \/>\n\u2502   \u251c\u2500\u2500 __init__.py<br \/>\n\u2502   \u251c\u2500\u2500 config.py<br \/>\n\u2502   \u251c\u2500\u2500 time_utils.py<br \/>\n\u2502   \u251c\u2500\u2500 fetcher.py<br \/>\n\u2502   \u251c\u2500\u2500 parser.py<br \/>\n\u2502   \u251c\u2500\u2500 storage.py<br \/>\n\u2502   \u251c\u2500\u2500 exporter.py<br \/>\n\u2502   \u2514\u2500\u2500 map_builder.py<br \/>\n\u2514\u2500\u2500 run_spider.py<\/p>\n<p>\u8fd9\u4e2a\u7ed3\u6784\u4e0d\u590d\u6742&#xff0c;\u4f46\u8db3\u591f\u6e05\u695a\u3002fisheries_spider \u653e\u6838\u5fc3\u4ee3\u7801&#xff0c;data \u653e\u8f93\u51fa\u7ed3\u679c&#xff0c;logs \u653e\u8fd0\u884c\u65e5\u5fd7&#xff0c;\u6839\u76ee\u5f55\u7684 run_spider.py \u4f5c\u4e3a\u5165\u53e3\u6587\u4ef6\u3002<\/p>\n<h4>5.5 \u914d\u7f6e\u6587\u4ef6<\/h4>\n<p>\u521b\u5efa .env.example&#xff1a;<\/p>\n<p>OBIS_BASE_URL&#061;https:\/\/api.obis.org\/v3<br \/>\nDEFAULT_TIMEOUT&#061;30<br \/>\nREQUEST_SLEEP_SECONDS&#061;0.8<br \/>\nPAGE_SIZE&#061;500<br \/>\nMAX_PAGES_PER_MONTH&#061;20<\/p>\n<p>\u590d\u5236\u4e3a .env&#xff1a;<\/p>\n<p><span class=\"token function\">cp<\/span> .env.example .env<\/p>\n<p>Windows \u4e0b\u53ef\u4ee5\u624b\u52a8\u590d\u5236&#xff0c;\u6216\u4f7f\u7528&#xff1a;<\/p>\n<p><span class=\"token function\">copy<\/span> <span class=\"token punctuation\">.<\/span>env<span class=\"token punctuation\">.<\/span>example <span class=\"token punctuation\">.<\/span>env<\/p>\n<hr \/>\n<h3>6\ufe0f\u20e3 \u6838\u5fc3\u5b9e\u73b0&#xff1a;\u8bf7\u6c42\u5c42&#xff08;Fetcher&#xff09;<\/h3>\n<p>\u8bf7\u6c42\u5c42\u7684\u804c\u8d23\u662f\u201c\u7a33\u5b9a\u3001\u514b\u5236\u3001\u53ef\u8ffd\u8e2a\u5730\u628a\u6570\u636e\u62ff\u56de\u6765\u201d\u3002\u5b83\u4e0d\u5e94\u8be5\u5173\u5fc3\u5b57\u6bb5\u600e\u4e48\u89e3\u6790&#xff0c;\u4e5f\u4e0d\u5e94\u8be5\u76f4\u63a5\u5199\u6570\u636e\u5e93\u3002\u8bf7\u6c42\u5c42\u53ea\u5904\u7406 HTTP\u3002<\/p>\n<h4>6.1 headers \u8bbe\u8ba1<\/h4>\n<p>\u5373\u4f7f\u662f\u516c\u5f00 API&#xff0c;\u4e5f\u5efa\u8bae\u8bbe\u7f6e\u6e05\u6670\u7684 User-Agent\u3002\u4e0d\u8981\u4f2a\u88c5\u6210\u5947\u602a\u7684\u6d4f\u89c8\u5668&#xff0c;\u4e5f\u4e0d\u8981\u5199\u7a7a UA\u3002\u6bd4\u8f83\u7a33\u59a5\u7684\u5199\u6cd5\u662f\u8bf4\u660e\u9879\u76ee\u7528\u9014\u548c\u8054\u7cfb\u4fe1\u606f\u3002\u5982\u679c\u662f\u516c\u53f8\u6216\u56e2\u961f\u9879\u76ee&#xff0c;\u53ef\u4ee5\u5199\u56e2\u961f\u90ae\u7bb1&#xff1b;\u4e2a\u4eba\u6d4b\u8bd5\u53ef\u4ee5\u5199\u9879\u76ee\u540d\u79f0\u3002<\/p>\n<p>\u672c\u6587\u793a\u4f8b&#xff1a;<\/p>\n<p>HEADERS <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">{<\/span><br \/>\n    <span class=\"token string\">&#034;User-Agent&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;FisheriesDynamicMapCollector\/1.0 (educational data collection; contact: example&#064;example.com)&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token string\">&#034;Accept&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;application\/json,text\/plain,*\/*&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token string\">&#034;Referer&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;https:\/\/mapper.obis.org\/&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n<span class=\"token punctuation\">}<\/span><\/p>\n<p>Referer \u4e0d\u662f\u7ed5\u8fc7\u9650\u5236\u7684\u5de5\u5177&#xff0c;\u53ea\u662f\u544a\u8bc9\u670d\u52a1\u7aef\u8bf7\u6c42\u6765\u6e90\u573a\u666f\u3002\u5bf9\u4e8e\u5b98\u65b9 API&#xff0c;\u4e0d\u4e00\u5b9a\u5fc5\u987b\u5e26 Referer&#xff1b;\u4fdd\u7559\u5b83\u4e3b\u8981\u662f\u4e3a\u4e86\u8d34\u8fd1\u5730\u56fe\u56fe\u5c42\u63a5\u53e3\u7684\u5b9e\u9645\u8bf7\u6c42\u98ce\u683c\u3002<\/p>\n<h4>6.2 timeout<\/h4>\n<p>\u6240\u6709\u8bf7\u6c42\u90fd\u5fc5\u987b\u8bbe\u7f6e timeout\u3002\u6ca1\u6709 timeout \u7684\u722c\u866b&#xff0c;\u8fdf\u65e9\u4f1a\u5361\u6b7b\u5728\u67d0\u4e2a\u7f51\u7edc\u6296\u52a8\u4e0a\u3002<\/p>\n<p>\u672c\u6587\u9ed8\u8ba4&#xff1a;<\/p>\n<p>timeout<span class=\"token operator\">&#061;<\/span><span class=\"token number\">30<\/span><\/p>\n<p>\u5b9e\u9645\u9879\u76ee\u4e2d\u53ef\u4ee5\u5206\u5f00\u8bbe\u7f6e\u8fde\u63a5\u8d85\u65f6\u548c\u8bfb\u53d6\u8d85\u65f6&#xff1a;<\/p>\n<p>timeout<span class=\"token operator\">&#061;<\/span><span class=\"token punctuation\">(<\/span><span class=\"token number\">10<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token number\">30<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<h4>6.3 session \/ cookie<\/h4>\n<p>\u672c\u6587\u6848\u4f8b\u4e0d\u9700\u8981\u767b\u5f55 cookie\u3002\u6211\u4eec\u4ecd\u7136\u4f7f\u7528 requests.Session()&#xff0c;\u539f\u56e0\u662f Session \u80fd\u590d\u7528\u8fde\u63a5&#xff0c;\u4e5f\u65b9\u4fbf\u7edf\u4e00 headers\u3001\u91cd\u8bd5\u7b56\u7565\u548c\u65e5\u5fd7\u3002<\/p>\n<p>\u5982\u679c\u76ee\u6807 API \u660e\u786e\u9700\u8981\u5b98\u65b9 token&#xff0c;\u5e94\u901a\u8fc7\u914d\u7f6e\u8bfb\u53d6&#xff0c;\u4e0d\u8981\u628a token \u5199\u6b7b\u5728\u4ee3\u7801\u91cc\u3002\u672c\u6587\u4e0d\u6f14\u793a\u7ed5\u8fc7\u767b\u5f55\u6001&#xff0c;\u4e5f\u4e0d\u91c7\u96c6\u9700\u8981\u6743\u9650\u7684\u6570\u636e\u3002<\/p>\n<h4>6.4 \u5931\u8d25\u5904\u7406&#xff1a;\u91cd\u8bd5\u4e0e\u9000\u907f<\/h4>\n<p>\u5931\u8d25\u5904\u7406\u81f3\u5c11\u8981\u8003\u8651\u8fd9\u4e9b\u60c5\u51b5&#xff1a;<\/p>\n<ul>\n<li>\u8fde\u63a5\u8d85\u65f6&#xff1b;<\/li>\n<li>\u8bfb\u53d6\u8d85\u65f6&#xff1b;<\/li>\n<li>502 \/ 503 \/ 504&#xff1b;<\/li>\n<li>429 \u8bbf\u95ee\u8fc7\u5feb&#xff1b;<\/li>\n<li>JSON \u89e3\u6790\u5931\u8d25&#xff1b;<\/li>\n<li>\u670d\u52a1\u5668\u77ed\u6682\u8fd4\u56de\u7a7a\u7ed3\u679c\u3002<\/li>\n<\/ul>\n<p>\u91cd\u8bd5\u4e0d\u662f\u65e0\u9650\u91cd\u8bd5\u3002\u672c\u6587\u4f7f\u7528\u6700\u591a 3 \u6b21\u91cd\u8bd5&#xff0c;\u6bcf\u6b21\u9000\u907f\u7b49\u5f85\u3002\u9047\u5230 429 \u65f6\u5e94\u8be5\u964d\u4f4e\u9891\u7387&#xff0c;\u800c\u4e0d\u662f\u6362\u4ee3\u7406\u786c\u51b2\u3002<\/p>\n<h4>6.5 \u4ee3\u7801&#xff1a;config.py<\/h4>\n<p><span class=\"token comment\"># fisheries_spider\/config.py<\/span><\/p>\n<p><span class=\"token keyword\">from<\/span> __future__ <span class=\"token keyword\">import<\/span> annotations<\/p>\n<p><span class=\"token keyword\">import<\/span> os<br \/>\n<span class=\"token keyword\">from<\/span> dataclasses <span class=\"token keyword\">import<\/span> dataclass<br \/>\n<span class=\"token keyword\">from<\/span> pathlib <span class=\"token keyword\">import<\/span> Path<\/p>\n<p><span class=\"token keyword\">from<\/span> dotenv <span class=\"token keyword\">import<\/span> load_dotenv<\/p>\n<p>load_dotenv<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>BASE_DIR <span class=\"token operator\">&#061;<\/span> Path<span class=\"token punctuation\">(<\/span>__file__<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">.<\/span>resolve<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">.<\/span>parents<span class=\"token punctuation\">[<\/span><span class=\"token number\">1<\/span><span class=\"token punctuation\">]<\/span><br \/>\nDATA_DIR <span class=\"token operator\">&#061;<\/span> BASE_DIR <span class=\"token operator\">\/<\/span> <span class=\"token string\">&#034;data&#034;<\/span><br \/>\nRAW_DIR <span class=\"token operator\">&#061;<\/span> DATA_DIR <span class=\"token operator\">\/<\/span> <span class=\"token string\">&#034;raw&#034;<\/span><br \/>\nPROCESSED_DIR <span class=\"token operator\">&#061;<\/span> DATA_DIR <span class=\"token operator\">\/<\/span> <span class=\"token string\">&#034;processed&#034;<\/span><br \/>\nMAP_DIR <span class=\"token operator\">&#061;<\/span> DATA_DIR <span class=\"token operator\">\/<\/span> <span class=\"token string\">&#034;maps&#034;<\/span><br \/>\nLOG_DIR <span class=\"token operator\">&#061;<\/span> BASE_DIR <span class=\"token operator\">\/<\/span> <span class=\"token string\">&#034;logs&#034;<\/span><\/p>\n<p><span class=\"token keyword\">for<\/span> folder <span class=\"token keyword\">in<\/span> <span class=\"token punctuation\">[<\/span>DATA_DIR<span class=\"token punctuation\">,<\/span> RAW_DIR<span class=\"token punctuation\">,<\/span> PROCESSED_DIR<span class=\"token punctuation\">,<\/span> MAP_DIR<span class=\"token punctuation\">,<\/span> LOG_DIR<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    folder<span class=\"token punctuation\">.<\/span>mkdir<span class=\"token punctuation\">(<\/span>parents<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">,<\/span> exist_ok<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p><span class=\"token decorator annotation punctuation\">&#064;dataclass<\/span><span class=\"token punctuation\">(<\/span>frozen<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">)<\/span><br \/>\n<span class=\"token keyword\">class<\/span> <span class=\"token class-name\">SpiderConfig<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    base_url<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span> <span class=\"token operator\">&#061;<\/span> os<span class=\"token punctuation\">.<\/span>getenv<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;OBIS_BASE_URL&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;https:\/\/api.obis.org\/v3&#034;<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    timeout<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">int<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">(<\/span>os<span class=\"token punctuation\">.<\/span>getenv<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;DEFAULT_TIMEOUT&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;30&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    request_sleep_seconds<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">float<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">float<\/span><span class=\"token punctuation\">(<\/span>os<span class=\"token punctuation\">.<\/span>getenv<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;REQUEST_SLEEP_SECONDS&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;0.8&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    page_size<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">int<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">(<\/span>os<span class=\"token punctuation\">.<\/span>getenv<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;PAGE_SIZE&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;500&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    max_pages_per_month<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">int<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">(<\/span>os<span class=\"token punctuation\">.<\/span>getenv<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;MAX_PAGES_PER_MONTH&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;20&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    sqlite_path<span class=\"token punctuation\">:<\/span> Path <span class=\"token operator\">&#061;<\/span> PROCESSED_DIR <span class=\"token operator\">\/<\/span> <span class=\"token string\">&#034;fisheries_dynamic.sqlite3&#034;<\/span><br \/>\n    raw_csv_path<span class=\"token punctuation\">:<\/span> Path <span class=\"token operator\">&#061;<\/span> PROCESSED_DIR <span class=\"token operator\">\/<\/span> <span class=\"token string\">&#034;raw_occurrences.csv&#034;<\/span><br \/>\n    intensity_csv_path<span class=\"token punctuation\">:<\/span> Path <span class=\"token operator\">&#061;<\/span> PROCESSED_DIR <span class=\"token operator\">\/<\/span> <span class=\"token string\">&#034;resource_intensity.csv&#034;<\/span><br \/>\n    intensity_geojson_path<span class=\"token punctuation\">:<\/span> Path <span class=\"token operator\">&#061;<\/span> PROCESSED_DIR <span class=\"token operator\">\/<\/span> <span class=\"token string\">&#034;resource_intensity.geojson&#034;<\/span><br \/>\n    timeline_map_path<span class=\"token punctuation\">:<\/span> Path <span class=\"token operator\">&#061;<\/span> MAP_DIR <span class=\"token operator\">\/<\/span> <span class=\"token string\">&#034;resource_timeline_map.html&#034;<\/span><\/p>\n<p>DEFAULT_HEADERS <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">{<\/span><br \/>\n    <span class=\"token string\">&#034;User-Agent&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">(<\/span><br \/>\n        <span class=\"token string\">&#034;FisheriesDynamicMapCollector\/1.0 &#034;<\/span><br \/>\n        <span class=\"token string\">&#034;(educational data collection; contact: example&#064;example.com)&#034;<\/span><br \/>\n    <span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token string\">&#034;Accept&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;application\/json,text\/plain,*\/*&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token string\">&#034;Referer&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;https:\/\/mapper.obis.org\/&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n<span class=\"token punctuation\">}<\/span><\/p>\n<h4>6.6 \u4ee3\u7801&#xff1a;fetcher.py<\/h4>\n<p><span class=\"token comment\"># fisheries_spider\/fetcher.py<\/span><\/p>\n<p><span class=\"token keyword\">from<\/span> __future__ <span class=\"token keyword\">import<\/span> annotations<\/p>\n<p><span class=\"token keyword\">import<\/span> logging<br \/>\n<span class=\"token keyword\">import<\/span> time<br \/>\n<span class=\"token keyword\">from<\/span> typing <span class=\"token keyword\">import<\/span> Any<span class=\"token punctuation\">,<\/span> Dict<span class=\"token punctuation\">,<\/span> Iterator<span class=\"token punctuation\">,<\/span> Optional<br \/>\n<span class=\"token keyword\">from<\/span> urllib<span class=\"token punctuation\">.<\/span>parse <span class=\"token keyword\">import<\/span> urlencode<\/p>\n<p><span class=\"token keyword\">import<\/span> requests<br \/>\n<span class=\"token keyword\">from<\/span> requests<span class=\"token punctuation\">.<\/span>adapters <span class=\"token keyword\">import<\/span> HTTPAdapter<br \/>\n<span class=\"token keyword\">from<\/span> tenacity <span class=\"token keyword\">import<\/span> retry<span class=\"token punctuation\">,<\/span> retry_if_exception_type<span class=\"token punctuation\">,<\/span> stop_after_attempt<span class=\"token punctuation\">,<\/span> wait_exponential<br \/>\n<span class=\"token keyword\">from<\/span> urllib3<span class=\"token punctuation\">.<\/span>util<span class=\"token punctuation\">.<\/span>retry <span class=\"token keyword\">import<\/span> Retry<\/p>\n<p><span class=\"token keyword\">from<\/span> fisheries_spider<span class=\"token punctuation\">.<\/span>config <span class=\"token keyword\">import<\/span> DEFAULT_HEADERS<span class=\"token punctuation\">,<\/span> SpiderConfig<\/p>\n<p>logger <span class=\"token operator\">&#061;<\/span> logging<span class=\"token punctuation\">.<\/span>getLogger<span class=\"token punctuation\">(<\/span>__name__<span class=\"token punctuation\">)<\/span><\/p>\n<p><span class=\"token keyword\">class<\/span> <span class=\"token class-name\">FetchError<\/span><span class=\"token punctuation\">(<\/span>RuntimeError<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;Raised when the fetcher fails after retries.&#034;&#034;&#034;<\/span><\/p>\n<p><span class=\"token keyword\">class<\/span> <span class=\"token class-name\">ObisFetcher<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n    A conservative API fetcher for OBIS occurrence records.<\/p>\n<p>    It is intentionally simple:<br \/>\n    &#8211; requests.Session for connection reuse<br \/>\n    &#8211; explicit headers<br \/>\n    &#8211; timeout<br \/>\n    &#8211; limited retries<br \/>\n    &#8211; sleep between requests<br \/>\n    &#8211; offset\/size pagination<br \/>\n    &#034;&#034;&#034;<\/span><\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">__init__<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">,<\/span> config<span class=\"token punctuation\">:<\/span> SpiderConfig<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>config <span class=\"token operator\">&#061;<\/span> config<br \/>\n        self<span class=\"token punctuation\">.<\/span>session <span class=\"token operator\">&#061;<\/span> requests<span class=\"token punctuation\">.<\/span>Session<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>session<span class=\"token punctuation\">.<\/span>headers<span class=\"token punctuation\">.<\/span>update<span class=\"token punctuation\">(<\/span>DEFAULT_HEADERS<span class=\"token punctuation\">)<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>_mount_retries<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">_mount_retries<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        retry_strategy <span class=\"token operator\">&#061;<\/span> Retry<span class=\"token punctuation\">(<\/span><br \/>\n            total<span class=\"token operator\">&#061;<\/span><span class=\"token number\">2<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            connect<span class=\"token operator\">&#061;<\/span><span class=\"token number\">2<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            read<span class=\"token operator\">&#061;<\/span><span class=\"token number\">2<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            status<span class=\"token operator\">&#061;<\/span><span class=\"token number\">2<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            backoff_factor<span class=\"token operator\">&#061;<\/span><span class=\"token number\">1.2<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            status_forcelist<span class=\"token operator\">&#061;<\/span><span class=\"token punctuation\">(<\/span><span class=\"token number\">429<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token number\">500<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token number\">502<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token number\">503<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token number\">504<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            allowed_methods<span class=\"token operator\">&#061;<\/span><span class=\"token builtin\">frozenset<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;GET&#034;<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            raise_on_status<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">False<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">)<\/span><br \/>\n        adapter <span class=\"token operator\">&#061;<\/span> HTTPAdapter<span class=\"token punctuation\">(<\/span>max_retries<span class=\"token operator\">&#061;<\/span>retry_strategy<span class=\"token punctuation\">)<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>session<span class=\"token punctuation\">.<\/span>mount<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;https:\/\/&#034;<\/span><span class=\"token punctuation\">,<\/span> adapter<span class=\"token punctuation\">)<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>session<span class=\"token punctuation\">.<\/span>mount<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;http:\/\/&#034;<\/span><span class=\"token punctuation\">,<\/span> adapter<span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">build_occurrence_url<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token keyword\">return<\/span> self<span class=\"token punctuation\">.<\/span>config<span class=\"token punctuation\">.<\/span>base_url<span class=\"token punctuation\">.<\/span>rstrip<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;\/&#034;<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#043;<\/span> <span class=\"token string\">&#034;\/occurrence&#034;<\/span><\/p>\n<p>    <span class=\"token decorator annotation punctuation\">&#064;retry<\/span><span class=\"token punctuation\">(<\/span><br \/>\n        retry<span class=\"token operator\">&#061;<\/span>retry_if_exception_type<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">(<\/span>requests<span class=\"token punctuation\">.<\/span>RequestException<span class=\"token punctuation\">,<\/span> ValueError<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        stop<span class=\"token operator\">&#061;<\/span>stop_after_attempt<span class=\"token punctuation\">(<\/span><span class=\"token number\">3<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        wait<span class=\"token operator\">&#061;<\/span>wait_exponential<span class=\"token punctuation\">(<\/span>multiplier<span class=\"token operator\">&#061;<\/span><span class=\"token number\">1<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">min<\/span><span class=\"token operator\">&#061;<\/span><span class=\"token number\">1<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">max<\/span><span class=\"token operator\">&#061;<\/span><span class=\"token number\">10<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        reraise<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">)<\/span><br \/>\n    <span class=\"token keyword\">def<\/span> <span class=\"token function\">get_json<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">,<\/span> url<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> params<span class=\"token punctuation\">:<\/span> Dict<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> Any<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> Dict<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> Any<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n        Execute GET request and parse JSON.<\/p>\n<p>        ValueError is included because response.json() raises it<br \/>\n        when the body is not valid JSON.<br \/>\n        &#034;&#034;&#034;<\/span><br \/>\n        full_url <span class=\"token operator\">&#061;<\/span> <span class=\"token string-interpolation\"><span class=\"token string\">f&#034;<\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>url<span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">?<\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>urlencode<span class=\"token punctuation\">(<\/span>params<span class=\"token punctuation\">,<\/span> doseq<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">&#034;<\/span><\/span><br \/>\n        logger<span class=\"token punctuation\">.<\/span>info<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;GET %s&#034;<\/span><span class=\"token punctuation\">,<\/span> full_url<span class=\"token punctuation\">)<\/span><\/p>\n<p>        resp <span class=\"token operator\">&#061;<\/span> self<span class=\"token punctuation\">.<\/span>session<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span>url<span class=\"token punctuation\">,<\/span> params<span class=\"token operator\">&#061;<\/span>params<span class=\"token punctuation\">,<\/span> timeout<span class=\"token operator\">&#061;<\/span>self<span class=\"token punctuation\">.<\/span>config<span class=\"token punctuation\">.<\/span>timeout<span class=\"token punctuation\">)<\/span><\/p>\n<p>        <span class=\"token keyword\">if<\/span> resp<span class=\"token punctuation\">.<\/span>status_code <span class=\"token operator\">&#061;&#061;<\/span> <span class=\"token number\">429<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            logger<span class=\"token punctuation\">.<\/span>warning<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;429 Too Many Requests. Slowing down before retry.&#034;<\/span><span class=\"token punctuation\">)<\/span><br \/>\n            time<span class=\"token punctuation\">.<\/span>sleep<span class=\"token punctuation\">(<\/span><span class=\"token builtin\">max<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">.<\/span>config<span class=\"token punctuation\">.<\/span>request_sleep_seconds <span class=\"token operator\">*<\/span> <span class=\"token number\">5<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token number\">5<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>        <span class=\"token keyword\">if<\/span> resp<span class=\"token punctuation\">.<\/span>status_code <span class=\"token operator\">&gt;&#061;<\/span> <span class=\"token number\">400<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token keyword\">raise<\/span> FetchError<span class=\"token punctuation\">(<\/span><span class=\"token string-interpolation\"><span class=\"token string\">f&#034;HTTP <\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>resp<span class=\"token punctuation\">.<\/span>status_code<span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">: <\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>resp<span class=\"token punctuation\">.<\/span>text<span class=\"token punctuation\">[<\/span><span class=\"token punctuation\">:<\/span><span class=\"token format-spec\">300]<\/span><span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">&#034;<\/span><\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>        data <span class=\"token operator\">&#061;<\/span> resp<span class=\"token punctuation\">.<\/span>json<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><br \/>\n        time<span class=\"token punctuation\">.<\/span>sleep<span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">.<\/span>config<span class=\"token punctuation\">.<\/span>request_sleep_seconds<span class=\"token punctuation\">)<\/span><br \/>\n        <span class=\"token keyword\">return<\/span> data<\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">iter_occurrences<\/span><span class=\"token punctuation\">(<\/span><br \/>\n        self<span class=\"token punctuation\">,<\/span><br \/>\n        scientific_name<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        start_date<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        end_date<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        geometry_wkt<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        fields<span class=\"token punctuation\">:<\/span> Optional<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">]<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> Iterator<span class=\"token punctuation\">[<\/span>Dict<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> Any<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n        Iterate occurrence records by offset pagination.<\/p>\n<p>        Parameters:<br \/>\n            scientific_name: species\/resource scientific name.<br \/>\n            start_date: YYYY-MM-DD.<br \/>\n            end_date: YYYY-MM-DD.<br \/>\n            geometry_wkt: WKT polygon.<br \/>\n            fields: optional comma-separated fields.<br \/>\n        &#034;&#034;&#034;<\/span><br \/>\n        url <span class=\"token operator\">&#061;<\/span> self<span class=\"token punctuation\">.<\/span>build_occurrence_url<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><br \/>\n        offset <span class=\"token operator\">&#061;<\/span> <span class=\"token number\">0<\/span><br \/>\n        page_index <span class=\"token operator\">&#061;<\/span> <span class=\"token number\">0<\/span><\/p>\n<p>        <span class=\"token keyword\">while<\/span> <span class=\"token boolean\">True<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token keyword\">if<\/span> page_index <span class=\"token operator\">&gt;&#061;<\/span> self<span class=\"token punctuation\">.<\/span>config<span class=\"token punctuation\">.<\/span>max_pages_per_month<span class=\"token punctuation\">:<\/span><br \/>\n                logger<span class=\"token punctuation\">.<\/span>warning<span class=\"token punctuation\">(<\/span><br \/>\n                    <span class=\"token string\">&#034;Reached max_pages_per_month&#061;%s for %s %s ~ %s&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                    self<span class=\"token punctuation\">.<\/span>config<span class=\"token punctuation\">.<\/span>max_pages_per_month<span class=\"token punctuation\">,<\/span><br \/>\n                    scientific_name<span class=\"token punctuation\">,<\/span><br \/>\n                    start_date<span class=\"token punctuation\">,<\/span><br \/>\n                    end_date<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token punctuation\">)<\/span><br \/>\n                <span class=\"token keyword\">break<\/span><\/p>\n<p>            params<span class=\"token punctuation\">:<\/span> Dict<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> Any<span class=\"token punctuation\">]<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">{<\/span><br \/>\n                <span class=\"token string\">&#034;scientificname&#034;<\/span><span class=\"token punctuation\">:<\/span> scientific_name<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;startdate&#034;<\/span><span class=\"token punctuation\">:<\/span> start_date<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;enddate&#034;<\/span><span class=\"token punctuation\">:<\/span> end_date<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;geometry&#034;<\/span><span class=\"token punctuation\">:<\/span> geometry_wkt<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;size&#034;<\/span><span class=\"token punctuation\">:<\/span> self<span class=\"token punctuation\">.<\/span>config<span class=\"token punctuation\">.<\/span>page_size<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;offset&#034;<\/span><span class=\"token punctuation\">:<\/span> offset<span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token punctuation\">}<\/span><\/p>\n<p>            <span class=\"token keyword\">if<\/span> fields<span class=\"token punctuation\">:<\/span><br \/>\n                params<span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;fields&#034;<\/span><span class=\"token punctuation\">]<\/span> <span class=\"token operator\">&#061;<\/span> fields<\/p>\n<p>            payload <span class=\"token operator\">&#061;<\/span> self<span class=\"token punctuation\">.<\/span>get_json<span class=\"token punctuation\">(<\/span>url<span class=\"token punctuation\">,<\/span> params<span class=\"token operator\">&#061;<\/span>params<span class=\"token punctuation\">)<\/span><br \/>\n            records <span class=\"token operator\">&#061;<\/span> self<span class=\"token punctuation\">.<\/span>_extract_results<span class=\"token punctuation\">(<\/span>payload<span class=\"token punctuation\">)<\/span><\/p>\n<p>            <span class=\"token keyword\">if<\/span> <span class=\"token keyword\">not<\/span> records<span class=\"token punctuation\">:<\/span><br \/>\n                logger<span class=\"token punctuation\">.<\/span>info<span class=\"token punctuation\">(<\/span><br \/>\n                    <span class=\"token string\">&#034;No more records for %s %s ~ %s at offset&#061;%s&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                    scientific_name<span class=\"token punctuation\">,<\/span><br \/>\n                    start_date<span class=\"token punctuation\">,<\/span><br \/>\n                    end_date<span class=\"token punctuation\">,<\/span><br \/>\n                    offset<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token punctuation\">)<\/span><br \/>\n                <span class=\"token keyword\">break<\/span><\/p>\n<p>            <span class=\"token keyword\">for<\/span> record <span class=\"token keyword\">in<\/span> records<span class=\"token punctuation\">:<\/span><br \/>\n                record<span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;_request_url&#034;<\/span><span class=\"token punctuation\">]<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token string-interpolation\"><span class=\"token string\">f&#034;<\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>url<span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">?<\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>urlencode<span class=\"token punctuation\">(<\/span>params<span class=\"token punctuation\">,<\/span> doseq<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">&#034;<\/span><\/span><br \/>\n                <span class=\"token keyword\">yield<\/span> record<\/p>\n<p>            got <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">len<\/span><span class=\"token punctuation\">(<\/span>records<span class=\"token punctuation\">)<\/span><br \/>\n            logger<span class=\"token punctuation\">.<\/span>info<span class=\"token punctuation\">(<\/span><br \/>\n                <span class=\"token string\">&#034;Fetched page&#061;%s offset&#061;%s size&#061;%s got&#061;%s&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                page_index<span class=\"token punctuation\">,<\/span><br \/>\n                offset<span class=\"token punctuation\">,<\/span><br \/>\n                self<span class=\"token punctuation\">.<\/span>config<span class=\"token punctuation\">.<\/span>page_size<span class=\"token punctuation\">,<\/span><br \/>\n                got<span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token punctuation\">)<\/span><\/p>\n<p>            <span class=\"token keyword\">if<\/span> got <span class=\"token operator\">&lt;<\/span> self<span class=\"token punctuation\">.<\/span>config<span class=\"token punctuation\">.<\/span>page_size<span class=\"token punctuation\">:<\/span><br \/>\n                <span class=\"token keyword\">break<\/span><\/p>\n<p>            offset <span class=\"token operator\">&#043;&#061;<\/span> self<span class=\"token punctuation\">.<\/span>config<span class=\"token punctuation\">.<\/span>page_size<br \/>\n            page_index <span class=\"token operator\">&#043;&#061;<\/span> <span class=\"token number\">1<\/span><\/p>\n<p>    <span class=\"token decorator annotation punctuation\">&#064;staticmethod<\/span><br \/>\n    <span class=\"token keyword\">def<\/span> <span class=\"token function\">_extract_results<\/span><span class=\"token punctuation\">(<\/span>payload<span class=\"token punctuation\">:<\/span> Any<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token builtin\">list<\/span><span class=\"token punctuation\">[<\/span>Dict<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> Any<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n        OBIS occurrence response is commonly a dict with a &#039;results&#039; list.<br \/>\n        This function is defensive because API wrappers or endpoints can<br \/>\n        return slightly different shapes.<br \/>\n        &#034;&#034;&#034;<\/span><br \/>\n        <span class=\"token keyword\">if<\/span> <span class=\"token builtin\">isinstance<\/span><span class=\"token punctuation\">(<\/span>payload<span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">list<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token keyword\">return<\/span> <span class=\"token punctuation\">[<\/span>x <span class=\"token keyword\">for<\/span> x <span class=\"token keyword\">in<\/span> payload <span class=\"token keyword\">if<\/span> <span class=\"token builtin\">isinstance<\/span><span class=\"token punctuation\">(<\/span>x<span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">dict<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">]<\/span><\/p>\n<p>        <span class=\"token keyword\">if<\/span> <span class=\"token builtin\">isinstance<\/span><span class=\"token punctuation\">(<\/span>payload<span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">dict<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token keyword\">for<\/span> key <span class=\"token keyword\">in<\/span> <span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;results&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;features&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;data&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n                value <span class=\"token operator\">&#061;<\/span> payload<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span>key<span class=\"token punctuation\">)<\/span><br \/>\n                <span class=\"token keyword\">if<\/span> <span class=\"token builtin\">isinstance<\/span><span class=\"token punctuation\">(<\/span>value<span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">list<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n                    <span class=\"token keyword\">if<\/span> key <span class=\"token operator\">&#061;&#061;<\/span> <span class=\"token string\">&#034;features&#034;<\/span><span class=\"token punctuation\">:<\/span><br \/>\n                        <span class=\"token keyword\">return<\/span> <span class=\"token punctuation\">[<\/span><br \/>\n                            item<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;properties&#034;<\/span><span class=\"token punctuation\">,<\/span> item<span class=\"token punctuation\">)<\/span><br \/>\n                            <span class=\"token keyword\">for<\/span> item <span class=\"token keyword\">in<\/span> value<br \/>\n                            <span class=\"token keyword\">if<\/span> <span class=\"token builtin\">isinstance<\/span><span class=\"token punctuation\">(<\/span>item<span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">dict<\/span><span class=\"token punctuation\">)<\/span><br \/>\n                        <span class=\"token punctuation\">]<\/span><br \/>\n                    <span class=\"token keyword\">return<\/span> <span class=\"token punctuation\">[<\/span>x <span class=\"token keyword\">for<\/span> x <span class=\"token keyword\">in<\/span> value <span class=\"token keyword\">if<\/span> <span class=\"token builtin\">isinstance<\/span><span class=\"token punctuation\">(<\/span>x<span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">dict<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">]<\/span><\/p>\n<p>        <span class=\"token keyword\">return<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token punctuation\">]<\/span><\/p>\n<p>\u8bf7\u6c42\u5c42\u5199\u5230\u8fd9\u91cc&#xff0c;\u57fa\u672c\u80fd\u6297\u4f4f\u5c0f\u89c4\u6a21\u91c7\u96c6\u3002\u6ce8\u610f&#xff0c;max_pages_per_month \u662f\u4e00\u4e2a\u5f88\u6709\u5fc5\u8981\u7684\u4fdd\u9669\u4e1d\u3002\u521a\u5f00\u59cb\u8c03\u8bd5\u65f6&#xff0c;\u4e0d\u8981\u8ba9\u7a0b\u5e8f\u5728\u4e00\u4e2a\u6708\u91cc\u7ffb\u51e0\u767e\u9875\u3002\u5148\u62ff\u5c0f\u6837\u672c\u9a8c\u8bc1\u89e3\u6790\u903b\u8f91&#xff0c;\u518d\u6162\u6162\u653e\u5927\u3002<\/p>\n<hr \/>\n<h3>7\ufe0f\u20e3 \u6838\u5fc3\u5b9e\u73b0&#xff1a;\u89e3\u6790\u5c42&#xff08;Parser&#xff09;<\/h3>\n<p>\u89e3\u6790\u5c42\u5904\u7406\u4e09\u4ef6\u4e8b&#xff1a;<\/p>\n<li>\u4ece JSON \u4e2d\u62bd\u53d6\u5b57\u6bb5&#xff1b;<\/li>\n<li>\u5bf9\u7f3a\u5931\u5b57\u6bb5\u505a\u5bb9\u9519&#xff1b;<\/li>\n<li>\u5c06\u70b9\u8bb0\u5f55\u8f6c\u6362\u6210\u53ef\u805a\u5408\u7684\u7ed3\u6784\u3002<\/li>\n<h4>7.1 \u89e3\u6790\u65b9\u5f0f<\/h4>\n<p>\u672c\u6587\u63a5\u53e3\u8fd4\u56de JSON&#xff0c;\u6240\u4ee5\u89e3\u6790\u65b9\u5f0f\u662f JSON \u5b57\u6bb5\u6620\u5c04&#xff0c;\u4e0d\u9700\u8981 XPath\u3001CSS \u6216 BeautifulSoup\u3002<\/p>\n<p>\u4e0d\u8fc7\u52a8\u6001\u5730\u56fe\u91c7\u96c6\u4e0d\u4e00\u5b9a\u90fd\u662f JSON\u3002\u6709\u4e9b\u5730\u56fe\u56fe\u5c42\u53ef\u80fd\u8fd4\u56de&#xff1a;<\/p>\n<ul>\n<li>GeoJSON&#xff1b;<\/li>\n<li>MVT \u77e2\u91cf\u74e6\u7247&#xff1b;<\/li>\n<li>KML&#xff1b;<\/li>\n<li>CSV&#xff1b;<\/li>\n<li>\u538b\u7f29\u540e\u7684\u4e8c\u8fdb\u5236\u6570\u636e&#xff1b;<\/li>\n<li>\u524d\u7aef\u81ea\u5b9a\u4e49\u7ed3\u6784\u3002<\/li>\n<\/ul>\n<p>\u672c\u6587\u4e3a\u4e86\u53ef\u8bfb\u6027&#xff0c;\u4f7f\u7528 occurrence JSON\u3002\u540e\u7eed\u5982\u679c\u9047\u5230 MVT&#xff0c;\u53ef\u4ee5\u7528 mapbox-vector-tile \u89e3\u6790&#xff1b;\u9047\u5230 GeoJSON&#xff0c;\u5219\u4ece features[].properties \u4e2d\u62ff\u5c5e\u6027\u5b57\u6bb5&#xff0c;\u4ece geometry.coordinates \u4e2d\u62ff\u5750\u6807\u3002<\/p>\n<h4>7.2 \u5217\u8868\u9875\u5982\u4f55\u62ff\u8be6\u60c5\u94fe\u63a5<\/h4>\n<p>\u672c\u6587\u4e0d\u8d70\u4f20\u7edf\u5217\u8868\u9875\u548c\u8be6\u60c5\u9875\u6a21\u5f0f\u3002\u52a8\u6001\u5730\u56fe\u6570\u636e\u901a\u5e38\u6ca1\u6709\u201c\u5217\u8868\u9875\u8be6\u60c5\u94fe\u63a5\u201d\u8fd9\u4e2a\u7ed3\u6784&#xff0c;\u5b83\u662f\u76f4\u63a5\u8fd4\u56de\u8bb0\u5f55\u96c6\u5408\u3002<\/p>\n<p>\u4f46\u4e3a\u4e86\u5bf9\u5e94\u666e\u901a\u91c7\u96c6\u9879\u76ee\u91cc\u7684\u201c\u5217\u8868\u9875 \u2192 \u8be6\u60c5\u9875\u201d\u601d\u8def&#xff0c;\u53ef\u4ee5\u8fd9\u6837\u7406\u89e3&#xff1a;<\/p>\n<ul>\n<li>\u5217\u8868\u9875&#xff1a;\u4e00\u6b21 occurrence \u67e5\u8be2\u8fd4\u56de\u7684 results&#xff1b;<\/li>\n<li>\u8be6\u60c5\u94fe\u63a5&#xff1a;\u6bcf\u6761\u8bb0\u5f55\u4e2d\u7684 id \u6216 occurrenceID&#xff1b;<\/li>\n<li>\u8be6\u60c5\u9875&#xff1a;\u5982 API \u652f\u6301 \/occurrence\/{id}&#xff0c;\u53ef\u4ee5\u6309 ID \u518d\u67e5\u8be6\u60c5&#xff1b;<\/li>\n<li>\u672c\u6587\u7b56\u7565&#xff1a;\u4e3a\u4e86\u964d\u4f4e\u8bf7\u6c42\u91cf&#xff0c;\u4f18\u5148\u4f7f\u7528\u5217\u8868\u63a5\u53e3\u8fd4\u56de\u5b57\u6bb5&#xff0c;\u4e0d\u5bf9\u6bcf\u6761\u8bb0\u5f55\u518d\u53d1\u8be6\u60c5\u8bf7\u6c42\u3002<\/li>\n<\/ul>\n<p>\u8fd9\u662f\u5f88\u91cd\u8981\u7684\u5de5\u7a0b\u53d6\u820d\u3002\u5f88\u591a\u65f6\u5019&#xff0c;\u5217\u8868\u63a5\u53e3\u5df2\u7ecf\u5305\u542b\u5206\u6790\u6240\u9700\u5b57\u6bb5\u3002\u4e3a\u4e86\u4e00\u4e2a\u53ef\u6709\u53ef\u65e0\u7684\u5b57\u6bb5\u7ed9\u6bcf\u6761\u8bb0\u5f55\u8865\u8be6\u60c5&#xff0c;\u4f1a\u8ba9\u8bf7\u6c42\u91cf\u66b4\u6da8&#xff0c;\u4e0d\u503c\u5f97\u3002<\/p>\n<h4>7.3 \u8be6\u60c5\u9875\u5982\u4f55\u62bd\u5b57\u6bb5<\/h4>\n<p>\u672c\u6587\u9ed8\u8ba4\u4e0d\u6293\u8be6\u60c5\u9875\u3002\u5982\u679c\u5b9e\u9645\u9700\u8981\u8865\u5145\u8be6\u60c5&#xff0c;\u53ef\u4ee5\u5199\u4e00\u4e2a get_occurrence_detail(occurrence_id)&#xff0c;\u4f46\u8981\u6709\u7f13\u5b58\u548c\u53bb\u91cd&#xff0c;\u907f\u514d\u91cd\u590d\u8bf7\u6c42\u3002<\/p>\n<p>\u5e38\u89c1\u5b57\u6bb5\u6620\u5c04\u5982\u4e0b&#xff1a;<\/p>\n<table>\n<tr>\u76ee\u6807\u5b57\u6bb5JSON \u5019\u9009\u5b57\u6bb5<\/tr>\n<tbody>\n<tr>\n<td>occurrence_id<\/td>\n<td>id \/ occurrenceID \/ record_id<\/td>\n<\/tr>\n<tr>\n<td>species_name<\/td>\n<td>scientificName \/ species \/ acceptedNameUsage<\/td>\n<\/tr>\n<tr>\n<td>event_date<\/td>\n<td>eventDate \/ date_mid \/ date_start<\/td>\n<\/tr>\n<tr>\n<td>longitude<\/td>\n<td>decimalLongitude \/ lon \/ longitude<\/td>\n<\/tr>\n<tr>\n<td>latitude<\/td>\n<td>decimalLatitude \/ lat \/ latitude<\/td>\n<\/tr>\n<tr>\n<td>dataset_id<\/td>\n<td>dataset_id \/ datasetID<\/td>\n<\/tr>\n<tr>\n<td>basis_of_record<\/td>\n<td>basisOfRecord<\/td>\n<\/tr>\n<tr>\n<td>source<\/td>\n<td>\u56fa\u5b9a\u4e3a OBIS API<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h4>7.4 \u7f3a\u5931\u5b57\u6bb5\u600e\u4e48\u529e<\/h4>\n<p>\u6570\u636e\u91c7\u96c6\u4e0d\u80fd\u5047\u8bbe\u6bcf\u6761\u8bb0\u5f55\u90fd\u5b8c\u7f8e\u3002\u5b57\u6bb5\u7f3a\u5931\u65f6\u6309\u4e0b\u9762\u89c4\u5219\u5904\u7406&#xff1a;<\/p>\n<ul>\n<li>\u7f3a\u5931 occurrence_id&#xff1a;\u7528\u5185\u5bb9 hash \u4f5c\u4e3a\u53bb\u91cd\u8865\u5145&#xff1b;<\/li>\n<li>\u7f3a\u5931 species_name&#xff1a;\u4f7f\u7528\u8bf7\u6c42\u53c2\u6570\u91cc\u7684 species \u515c\u5e95&#xff1b;<\/li>\n<li>\u7f3a\u5931 event_date&#xff1a;\u5c1d\u8bd5 date_mid \/ date_start&#xff1b;\u4ecd\u65e0\u5219\u4e22\u5f03\u6216\u6807\u8bb0 unknown&#xff1b;<\/li>\n<li>\u7f3a\u5931\u7ecf\u7eac\u5ea6&#xff1a;\u4e0d\u80fd\u8fdb\u5165\u7a7a\u95f4\u805a\u5408&#xff1b;<\/li>\n<li>\u7ecf\u7eac\u5ea6\u8d85\u8303\u56f4&#xff1a;\u4e22\u5f03&#xff1b;<\/li>\n<li>dataset_id \u7f3a\u5931&#xff1a;\u4fdd\u7559\u4e3a\u7a7a&#xff1b;<\/li>\n<li>basis_of_record \u7f3a\u5931&#xff1a;\u4fdd\u7559\u4e3a\u7a7a&#xff1b;<\/li>\n<li>source \u56fa\u5b9a\u5199\u5165\u91c7\u96c6\u6765\u6e90\u3002<\/li>\n<\/ul>\n<h4>7.5 \u4ee3\u7801&#xff1a;time_utils.py<\/h4>\n<p><span class=\"token comment\"># fisheries_spider\/time_utils.py<\/span><\/p>\n<p><span class=\"token keyword\">from<\/span> __future__ <span class=\"token keyword\">import<\/span> annotations<\/p>\n<p><span class=\"token keyword\">import<\/span> calendar<br \/>\n<span class=\"token keyword\">from<\/span> dataclasses <span class=\"token keyword\">import<\/span> dataclass<br \/>\n<span class=\"token keyword\">from<\/span> datetime <span class=\"token keyword\">import<\/span> date<br \/>\n<span class=\"token keyword\">from<\/span> typing <span class=\"token keyword\">import<\/span> Iterator<\/p>\n<p><span class=\"token decorator annotation punctuation\">&#064;dataclass<\/span><span class=\"token punctuation\">(<\/span>frozen<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">)<\/span><br \/>\n<span class=\"token keyword\">class<\/span> <span class=\"token class-name\">MonthWindow<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    month<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><br \/>\n    start_date<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><br \/>\n    end_date<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">iter_month_windows<\/span><span class=\"token punctuation\">(<\/span>start_month<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> end_month<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> Iterator<span class=\"token punctuation\">[<\/span>MonthWindow<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n    Generate inclusive month windows.<\/p>\n<p>    Example:<br \/>\n        start_month&#061;&#039;2020-01&#039;<br \/>\n        end_month&#061;&#039;2020-03&#039;<\/p>\n<p>    Yields:<br \/>\n        2020-01-01 ~ 2020-01-31<br \/>\n        2020-02-01 ~ 2020-02-29<br \/>\n        2020-03-01 ~ 2020-03-31<br \/>\n    &#034;&#034;&#034;<\/span><br \/>\n    sy<span class=\"token punctuation\">,<\/span> sm <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">map<\/span><span class=\"token punctuation\">(<\/span><span class=\"token builtin\">int<\/span><span class=\"token punctuation\">,<\/span> start_month<span class=\"token punctuation\">.<\/span>split<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;-&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    ey<span class=\"token punctuation\">,<\/span> em <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">map<\/span><span class=\"token punctuation\">(<\/span><span class=\"token builtin\">int<\/span><span class=\"token punctuation\">,<\/span> end_month<span class=\"token punctuation\">.<\/span>split<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;-&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    current_year <span class=\"token operator\">&#061;<\/span> sy<br \/>\n    current_month <span class=\"token operator\">&#061;<\/span> sm<\/p>\n<p>    <span class=\"token keyword\">while<\/span> <span class=\"token punctuation\">(<\/span>current_year<span class=\"token punctuation\">,<\/span> current_month<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&lt;&#061;<\/span> <span class=\"token punctuation\">(<\/span>ey<span class=\"token punctuation\">,<\/span> em<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        last_day <span class=\"token operator\">&#061;<\/span> calendar<span class=\"token punctuation\">.<\/span>monthrange<span class=\"token punctuation\">(<\/span>current_year<span class=\"token punctuation\">,<\/span> current_month<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">[<\/span><span class=\"token number\">1<\/span><span class=\"token punctuation\">]<\/span><br \/>\n        start <span class=\"token operator\">&#061;<\/span> date<span class=\"token punctuation\">(<\/span>current_year<span class=\"token punctuation\">,<\/span> current_month<span class=\"token punctuation\">,<\/span> <span class=\"token number\">1<\/span><span class=\"token punctuation\">)<\/span><br \/>\n        end <span class=\"token operator\">&#061;<\/span> date<span class=\"token punctuation\">(<\/span>current_year<span class=\"token punctuation\">,<\/span> current_month<span class=\"token punctuation\">,<\/span> last_day<span class=\"token punctuation\">)<\/span><\/p>\n<p>        month_text <span class=\"token operator\">&#061;<\/span> <span class=\"token string-interpolation\"><span class=\"token string\">f&#034;<\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>current_year<span class=\"token punctuation\">:<\/span><span class=\"token format-spec\">04d<\/span><span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">&#8211;<\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>current_month<span class=\"token punctuation\">:<\/span><span class=\"token format-spec\">02d<\/span><span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">&#034;<\/span><\/span><br \/>\n        <span class=\"token keyword\">yield<\/span> MonthWindow<span class=\"token punctuation\">(<\/span><br \/>\n            month<span class=\"token operator\">&#061;<\/span>month_text<span class=\"token punctuation\">,<\/span><br \/>\n            start_date<span class=\"token operator\">&#061;<\/span>start<span class=\"token punctuation\">.<\/span>isoformat<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            end_date<span class=\"token operator\">&#061;<\/span>end<span class=\"token punctuation\">.<\/span>isoformat<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">)<\/span><\/p>\n<p>        current_month <span class=\"token operator\">&#043;&#061;<\/span> <span class=\"token number\">1<\/span><br \/>\n        <span class=\"token keyword\">if<\/span> current_month <span class=\"token operator\">&gt;<\/span> <span class=\"token number\">12<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            current_month <span class=\"token operator\">&#061;<\/span> <span class=\"token number\">1<\/span><br \/>\n            current_year <span class=\"token operator\">&#043;&#061;<\/span> <span class=\"token number\">1<\/span><\/p>\n<h4>7.6 \u4ee3\u7801&#xff1a;parser.py<\/h4>\n<p><span class=\"token comment\"># fisheries_spider\/parser.py<\/span><\/p>\n<p><span class=\"token keyword\">from<\/span> __future__ <span class=\"token keyword\">import<\/span> annotations<\/p>\n<p><span class=\"token keyword\">import<\/span> hashlib<br \/>\n<span class=\"token keyword\">import<\/span> json<br \/>\n<span class=\"token keyword\">import<\/span> math<br \/>\n<span class=\"token keyword\">from<\/span> dataclasses <span class=\"token keyword\">import<\/span> asdict<span class=\"token punctuation\">,<\/span> dataclass<br \/>\n<span class=\"token keyword\">from<\/span> datetime <span class=\"token keyword\">import<\/span> datetime<span class=\"token punctuation\">,<\/span> timezone<br \/>\n<span class=\"token keyword\">from<\/span> typing <span class=\"token keyword\">import<\/span> Any<span class=\"token punctuation\">,<\/span> Dict<span class=\"token punctuation\">,<\/span> Optional<\/p>\n<p><span class=\"token decorator annotation punctuation\">&#064;dataclass<\/span><br \/>\n<span class=\"token keyword\">class<\/span> <span class=\"token class-name\">RawOccurrence<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    occurrence_id<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><br \/>\n    species_name<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><br \/>\n    sea_area<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><br \/>\n    time_window<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><br \/>\n    event_date<span class=\"token punctuation\">:<\/span> Optional<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">]<\/span><br \/>\n    longitude<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">float<\/span><br \/>\n    latitude<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">float<\/span><br \/>\n    dataset_id<span class=\"token punctuation\">:<\/span> Optional<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">]<\/span><br \/>\n    basis_of_record<span class=\"token punctuation\">:<\/span> Optional<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">]<\/span><br \/>\n    source<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><br \/>\n    request_url<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><br \/>\n    content_hash<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><br \/>\n    fetched_at<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><\/p>\n<p><span class=\"token decorator annotation punctuation\">&#064;dataclass<\/span><br \/>\n<span class=\"token keyword\">class<\/span> <span class=\"token class-name\">IntensityRecord<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    species_name<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><br \/>\n    sea_area<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><br \/>\n    time_window<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><br \/>\n    grid_id<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><br \/>\n    grid_lon<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">int<\/span><br \/>\n    grid_lat<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">int<\/span><br \/>\n    intensity_index<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">int<\/span><br \/>\n    source<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">now_utc_iso<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token keyword\">return<\/span> datetime<span class=\"token punctuation\">.<\/span>now<span class=\"token punctuation\">(<\/span>timezone<span class=\"token punctuation\">.<\/span>utc<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">.<\/span>replace<span class=\"token punctuation\">(<\/span>microsecond<span class=\"token operator\">&#061;<\/span><span class=\"token number\">0<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">.<\/span>isoformat<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">stable_hash<\/span><span class=\"token punctuation\">(<\/span>obj<span class=\"token punctuation\">:<\/span> Dict<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> Any<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    raw <span class=\"token operator\">&#061;<\/span> json<span class=\"token punctuation\">.<\/span>dumps<span class=\"token punctuation\">(<\/span>obj<span class=\"token punctuation\">,<\/span> ensure_ascii<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">False<\/span><span class=\"token punctuation\">,<\/span> sort_keys<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">,<\/span> default<span class=\"token operator\">&#061;<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    <span class=\"token keyword\">return<\/span> hashlib<span class=\"token punctuation\">.<\/span>sha256<span class=\"token punctuation\">(<\/span>raw<span class=\"token punctuation\">.<\/span>encode<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;utf-8&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">.<\/span>hexdigest<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">pick_first<\/span><span class=\"token punctuation\">(<\/span>record<span class=\"token punctuation\">:<\/span> Dict<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> Any<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span> keys<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">list<\/span><span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> Any<span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token keyword\">for<\/span> key <span class=\"token keyword\">in<\/span> keys<span class=\"token punctuation\">:<\/span><br \/>\n        value <span class=\"token operator\">&#061;<\/span> record<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span>key<span class=\"token punctuation\">)<\/span><br \/>\n        <span class=\"token keyword\">if<\/span> value <span class=\"token keyword\">not<\/span> <span class=\"token keyword\">in<\/span> <span class=\"token punctuation\">(<\/span><span class=\"token boolean\">None<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token keyword\">return<\/span> value<br \/>\n    <span class=\"token keyword\">return<\/span> <span class=\"token boolean\">None<\/span><\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">parse_float<\/span><span class=\"token punctuation\">(<\/span>value<span class=\"token punctuation\">:<\/span> Any<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> Optional<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">float<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token keyword\">if<\/span> value <span class=\"token keyword\">in<\/span> <span class=\"token punctuation\">(<\/span><span class=\"token boolean\">None<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token keyword\">return<\/span> <span class=\"token boolean\">None<\/span><br \/>\n    <span class=\"token keyword\">try<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        result <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">float<\/span><span class=\"token punctuation\">(<\/span>value<span class=\"token punctuation\">)<\/span><br \/>\n    <span class=\"token keyword\">except<\/span> <span class=\"token punctuation\">(<\/span>TypeError<span class=\"token punctuation\">,<\/span> ValueError<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token keyword\">return<\/span> <span class=\"token boolean\">None<\/span><\/p>\n<p>    <span class=\"token keyword\">if<\/span> math<span class=\"token punctuation\">.<\/span>isnan<span class=\"token punctuation\">(<\/span>result<span class=\"token punctuation\">)<\/span> <span class=\"token keyword\">or<\/span> math<span class=\"token punctuation\">.<\/span>isinf<span class=\"token punctuation\">(<\/span>result<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token keyword\">return<\/span> <span class=\"token boolean\">None<\/span><\/p>\n<p>    <span class=\"token keyword\">return<\/span> result<\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">valid_lon_lat<\/span><span class=\"token punctuation\">(<\/span>lon<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">float<\/span><span class=\"token punctuation\">,<\/span> lat<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">float<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token builtin\">bool<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token keyword\">return<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token number\">180<\/span> <span class=\"token operator\">&lt;&#061;<\/span> lon <span class=\"token operator\">&lt;&#061;<\/span> <span class=\"token number\">180<\/span> <span class=\"token keyword\">and<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token number\">90<\/span> <span class=\"token operator\">&lt;&#061;<\/span> lat <span class=\"token operator\">&lt;&#061;<\/span> <span class=\"token number\">90<\/span><\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">parse_event_date<\/span><span class=\"token punctuation\">(<\/span>record<span class=\"token punctuation\">:<\/span> Dict<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> Any<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> Optional<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n    Prefer eventDate. Fall back to date_mid\/date_start if present.<\/p>\n<p>    date_mid\/date_start may be unix timestamps in some datasets. This parser<br \/>\n    is intentionally conservative: it keeps readable date strings and ignores<br \/>\n    numeric timestamps unless you extend it.<br \/>\n    &#034;&#034;&#034;<\/span><br \/>\n    value <span class=\"token operator\">&#061;<\/span> pick_first<span class=\"token punctuation\">(<\/span>record<span class=\"token punctuation\">,<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;eventDate&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;event_date&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;date&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;date_mid&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;date_start&#034;<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    <span class=\"token keyword\">if<\/span> value <span class=\"token keyword\">is<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token keyword\">return<\/span> <span class=\"token boolean\">None<\/span><\/p>\n<p>    <span class=\"token keyword\">if<\/span> <span class=\"token builtin\">isinstance<\/span><span class=\"token punctuation\">(<\/span>value<span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token keyword\">return<\/span> value<span class=\"token punctuation\">[<\/span><span class=\"token punctuation\">:<\/span><span class=\"token number\">30<\/span><span class=\"token punctuation\">]<\/span><\/p>\n<p>    <span class=\"token keyword\">return<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">(<\/span>value<span class=\"token punctuation\">)<\/span><\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">grid_id_from_lon_lat<\/span><span class=\"token punctuation\">(<\/span>lon<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">float<\/span><span class=\"token punctuation\">,<\/span> lat<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">float<\/span><span class=\"token punctuation\">,<\/span> grid_size<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">float<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token number\">1.0<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token builtin\">tuple<\/span><span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n    Convert lon\/lat to a simple degree grid.<\/p>\n<p>    For example:<br \/>\n        lon&#061;-63.25, lat&#061;44.71, grid_size&#061;1<br \/>\n        -&gt; grid_lon&#061;-64, grid_lat&#061;44<\/p>\n<p>    This is simple and explainable. For production analysis, consider geohash,<br \/>\n    H3, S2, or official statistical areas.<br \/>\n    &#034;&#034;&#034;<\/span><br \/>\n    grid_lon <span class=\"token operator\">&#061;<\/span> math<span class=\"token punctuation\">.<\/span>floor<span class=\"token punctuation\">(<\/span>lon <span class=\"token operator\">\/<\/span> grid_size<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">*<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">(<\/span>grid_size<span class=\"token punctuation\">)<\/span><br \/>\n    grid_lat <span class=\"token operator\">&#061;<\/span> math<span class=\"token punctuation\">.<\/span>floor<span class=\"token punctuation\">(<\/span>lat <span class=\"token operator\">\/<\/span> grid_size<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">*<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">(<\/span>grid_size<span class=\"token punctuation\">)<\/span><\/p>\n<p>    grid_id <span class=\"token operator\">&#061;<\/span> <span class=\"token string-interpolation\"><span class=\"token string\">f&#034;lon_<\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>grid_lon<span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">_lat_<\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>grid_lat<span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">&#034;<\/span><\/span><br \/>\n    <span class=\"token keyword\">return<\/span> grid_id<span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">(<\/span>grid_lon<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">(<\/span>grid_lat<span class=\"token punctuation\">)<\/span><\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">parse_occurrence<\/span><span class=\"token punctuation\">(<\/span><br \/>\n    record<span class=\"token punctuation\">:<\/span> Dict<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> Any<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    fallback_species<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    sea_area<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    time_window<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    source<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;OBIS API&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> Optional<span class=\"token punctuation\">[<\/span>RawOccurrence<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    lon <span class=\"token operator\">&#061;<\/span> parse_float<span class=\"token punctuation\">(<\/span>pick_first<span class=\"token punctuation\">(<\/span>record<span class=\"token punctuation\">,<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;decimalLongitude&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;longitude&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;lon&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;x&#034;<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    lat <span class=\"token operator\">&#061;<\/span> parse_float<span class=\"token punctuation\">(<\/span>pick_first<span class=\"token punctuation\">(<\/span>record<span class=\"token punctuation\">,<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;decimalLatitude&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;latitude&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;lat&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;y&#034;<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">if<\/span> lon <span class=\"token keyword\">is<\/span> <span class=\"token boolean\">None<\/span> <span class=\"token keyword\">or<\/span> lat <span class=\"token keyword\">is<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token keyword\">return<\/span> <span class=\"token boolean\">None<\/span><\/p>\n<p>    <span class=\"token keyword\">if<\/span> <span class=\"token keyword\">not<\/span> valid_lon_lat<span class=\"token punctuation\">(<\/span>lon<span class=\"token punctuation\">,<\/span> lat<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token keyword\">return<\/span> <span class=\"token boolean\">None<\/span><\/p>\n<p>    content_hash <span class=\"token operator\">&#061;<\/span> stable_hash<span class=\"token punctuation\">(<\/span>record<span class=\"token punctuation\">)<\/span><br \/>\n    occurrence_id <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">(<\/span><br \/>\n        pick_first<span class=\"token punctuation\">(<\/span>record<span class=\"token punctuation\">,<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;id&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;occurrenceID&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;occurrence_id&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;record_id&#034;<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span><br \/>\n        <span class=\"token keyword\">or<\/span> content_hash<br \/>\n    <span class=\"token punctuation\">)<\/span><\/p>\n<p>    species_name <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">(<\/span><br \/>\n        pick_first<span class=\"token punctuation\">(<\/span>record<span class=\"token punctuation\">,<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;scientificName&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;species&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;acceptedNameUsage&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;taxon&#034;<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span><br \/>\n        <span class=\"token keyword\">or<\/span> fallback_species<br \/>\n    <span class=\"token punctuation\">)<\/span><\/p>\n<p>    event_date <span class=\"token operator\">&#061;<\/span> parse_event_date<span class=\"token punctuation\">(<\/span>record<span class=\"token punctuation\">)<\/span><\/p>\n<p>    dataset_id <span class=\"token operator\">&#061;<\/span> pick_first<span class=\"token punctuation\">(<\/span>record<span class=\"token punctuation\">,<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;dataset_id&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;datasetID&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;resource_id&#034;<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    basis_of_record <span class=\"token operator\">&#061;<\/span> pick_first<span class=\"token punctuation\">(<\/span>record<span class=\"token punctuation\">,<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;basisOfRecord&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#034;basis_of_record&#034;<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    request_url <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">(<\/span>record<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;_request_url&#034;<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token keyword\">or<\/span> <span class=\"token string\">&#034;&#034;<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">return<\/span> RawOccurrence<span class=\"token punctuation\">(<\/span><br \/>\n        occurrence_id<span class=\"token operator\">&#061;<\/span>occurrence_id<span class=\"token punctuation\">,<\/span><br \/>\n        species_name<span class=\"token operator\">&#061;<\/span>species_name<span class=\"token punctuation\">,<\/span><br \/>\n        sea_area<span class=\"token operator\">&#061;<\/span>sea_area<span class=\"token punctuation\">,<\/span><br \/>\n        time_window<span class=\"token operator\">&#061;<\/span>time_window<span class=\"token punctuation\">,<\/span><br \/>\n        event_date<span class=\"token operator\">&#061;<\/span>event_date<span class=\"token punctuation\">,<\/span><br \/>\n        longitude<span class=\"token operator\">&#061;<\/span>lon<span class=\"token punctuation\">,<\/span><br \/>\n        latitude<span class=\"token operator\">&#061;<\/span>lat<span class=\"token punctuation\">,<\/span><br \/>\n        dataset_id<span class=\"token operator\">&#061;<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">(<\/span>dataset_id<span class=\"token punctuation\">)<\/span> <span class=\"token keyword\">if<\/span> dataset_id <span class=\"token keyword\">is<\/span> <span class=\"token keyword\">not<\/span> <span class=\"token boolean\">None<\/span> <span class=\"token keyword\">else<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        basis_of_record<span class=\"token operator\">&#061;<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">(<\/span>basis_of_record<span class=\"token punctuation\">)<\/span> <span class=\"token keyword\">if<\/span> basis_of_record <span class=\"token keyword\">is<\/span> <span class=\"token keyword\">not<\/span> <span class=\"token boolean\">None<\/span> <span class=\"token keyword\">else<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        source<span class=\"token operator\">&#061;<\/span>source<span class=\"token punctuation\">,<\/span><br \/>\n        request_url<span class=\"token operator\">&#061;<\/span>request_url<span class=\"token punctuation\">,<\/span><br \/>\n        content_hash<span class=\"token operator\">&#061;<\/span>content_hash<span class=\"token punctuation\">,<\/span><br \/>\n        fetched_at<span class=\"token operator\">&#061;<\/span>now_utc_iso<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">)<\/span><\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">raw_to_dict<\/span><span class=\"token punctuation\">(<\/span>item<span class=\"token punctuation\">:<\/span> RawOccurrence<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> Dict<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> Any<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token keyword\">return<\/span> asdict<span class=\"token punctuation\">(<\/span>item<span class=\"token punctuation\">)<\/span><\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">intensity_to_dict<\/span><span class=\"token punctuation\">(<\/span>item<span class=\"token punctuation\">:<\/span> IntensityRecord<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> Dict<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> Any<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token keyword\">return<\/span> asdict<span class=\"token punctuation\">(<\/span>item<span class=\"token punctuation\">)<\/span><\/p>\n<p>\u8fd9\u4e2a\u89e3\u6790\u5668\u523b\u610f\u5199\u5f97\u4fdd\u5b88\u3002\u5b83\u4e0d\u4f1a\u56e0\u4e3a\u67d0\u4e2a\u5b57\u6bb5\u7f3a\u5931\u5c31\u8ba9\u6574\u4e2a\u7a0b\u5e8f\u5d29\u6389&#xff0c;\u4e5f\u4e0d\u4f1a\u628a\u6ca1\u6709\u7ecf\u7eac\u5ea6\u7684\u6570\u636e\u786c\u585e\u8fdb\u7a7a\u95f4\u805a\u5408\u3002\u52a8\u6001\u5730\u56fe\u91c7\u96c6\u6700\u6015\u7684\u4e0d\u662f\u201c\u6293\u4e0d\u5230\u201d&#xff0c;\u800c\u662f\u201c\u6293\u5230\u4e86\u810f\u6570\u636e\u8fd8\u4e0d\u77e5\u9053\u201d\u3002<\/p>\n<hr \/>\n<h3>8\ufe0f\u20e3 \u6570\u636e\u5b58\u50a8\u4e0e\u5bfc\u51fa&#xff08;Storage&#xff09;<\/h3>\n<h4>8.1 \u8d77\u6b65\u9009\u62e9 SQLite<\/h4>\n<p>\u672c\u6587\u9009\u62e9 SQLite \u4f5c\u4e3a\u672c\u5730\u5b58\u50a8\u3002\u5b83\u7684\u4f18\u70b9\u662f&#xff1a;<\/p>\n<ul>\n<li>\u4e0d\u9700\u8981\u90e8\u7f72\u6570\u636e\u5e93\u670d\u52a1&#xff1b;<\/li>\n<li>\u4e00\u4e2a\u6587\u4ef6\u5c31\u80fd\u4fdd\u5b58\u7ed3\u679c&#xff1b;<\/li>\n<li>\u652f\u6301 SQL \u67e5\u8be2&#xff1b;<\/li>\n<li>\u9002\u5408\u4e2d\u5c0f\u89c4\u6a21\u91c7\u96c6&#xff1b;<\/li>\n<li>\u540e\u7eed\u8fc1\u79fb\u5230 MySQL\/PostgreSQL \u4e5f\u4e0d\u96be\u3002<\/li>\n<\/ul>\n<p>\u5982\u679c\u6570\u636e\u91cf\u5f88\u5927&#xff0c;\u6216\u8005\u9700\u8981\u591a\u4eba\u5171\u4eab&#xff0c;\u5efa\u8bae\u6362 PostgreSQL &#043; PostGIS\u3002\u7a7a\u95f4\u5206\u6790\u591a\u7684\u9879\u76ee&#xff0c;PostGIS \u4f1a\u8212\u670d\u5f88\u591a\u3002<\/p>\n<h4>8.2 \u5b57\u6bb5\u6620\u5c04\u8868<\/h4>\n<p>\u539f\u59cb\u8868 raw_occurrences&#xff1a;<\/p>\n<table>\n<tr>\u5b57\u6bb5\u540d\u7c7b\u578b\u793a\u4f8b\u503c<\/tr>\n<tbody>\n<tr>\n<td>occurrence_id<\/td>\n<td>TEXT PRIMARY KEY<\/td>\n<td>00003cf7-f2fc<\/td>\n<\/tr>\n<tr>\n<td>species_name<\/td>\n<td>TEXT<\/td>\n<td>Gadus morhua<\/td>\n<\/tr>\n<tr>\n<td>sea_area<\/td>\n<td>TEXT<\/td>\n<td>North Atlantic Demo Area<\/td>\n<\/tr>\n<tr>\n<td>time_window<\/td>\n<td>TEXT<\/td>\n<td>2020-03<\/td>\n<\/tr>\n<tr>\n<td>event_date<\/td>\n<td>TEXT<\/td>\n<td>2020-03-12<\/td>\n<\/tr>\n<tr>\n<td>longitude<\/td>\n<td>REAL<\/td>\n<td>-63.25<\/td>\n<\/tr>\n<tr>\n<td>latitude<\/td>\n<td>REAL<\/td>\n<td>44.71<\/td>\n<\/tr>\n<tr>\n<td>dataset_id<\/td>\n<td>TEXT<\/td>\n<td>dataset uuid<\/td>\n<\/tr>\n<tr>\n<td>basis_of_record<\/td>\n<td>TEXT<\/td>\n<td>HumanObservation<\/td>\n<\/tr>\n<tr>\n<td>source<\/td>\n<td>TEXT<\/td>\n<td>OBIS API<\/td>\n<\/tr>\n<tr>\n<td>request_url<\/td>\n<td>TEXT<\/td>\n<td>request url<\/td>\n<\/tr>\n<tr>\n<td>content_hash<\/td>\n<td>TEXT<\/td>\n<td>sha256<\/td>\n<\/tr>\n<tr>\n<td>fetched_at<\/td>\n<td>TEXT<\/td>\n<td>2026-06-12T10:30:00&#043;00:00<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>\u805a\u5408\u8868 resource_intensity&#xff1a;<\/p>\n<table>\n<tr>\u5b57\u6bb5\u540d\u7c7b\u578b\u793a\u4f8b\u503c<\/tr>\n<tbody>\n<tr>\n<td>species_name<\/td>\n<td>TEXT<\/td>\n<td>Gadus morhua<\/td>\n<\/tr>\n<tr>\n<td>sea_area<\/td>\n<td>TEXT<\/td>\n<td>North Atlantic Demo Area<\/td>\n<\/tr>\n<tr>\n<td>time_window<\/td>\n<td>TEXT<\/td>\n<td>2020-03<\/td>\n<\/tr>\n<tr>\n<td>grid_id<\/td>\n<td>TEXT<\/td>\n<td>lon_-64_lat_44<\/td>\n<\/tr>\n<tr>\n<td>grid_lon<\/td>\n<td>INTEGER<\/td>\n<td>-64<\/td>\n<\/tr>\n<tr>\n<td>grid_lat<\/td>\n<td>INTEGER<\/td>\n<td>44<\/td>\n<\/tr>\n<tr>\n<td>intensity_index<\/td>\n<td>INTEGER<\/td>\n<td>18<\/td>\n<\/tr>\n<tr>\n<td>source<\/td>\n<td>TEXT<\/td>\n<td>OBIS API<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>\u8bf7\u6c42\u65e5\u5fd7\u8868 request_log&#xff1a;<\/p>\n<table>\n<tr>\u5b57\u6bb5\u540d\u7c7b\u578b\u793a\u4f8b\u503c<\/tr>\n<tbody>\n<tr>\n<td>id<\/td>\n<td>INTEGER<\/td>\n<td>1<\/td>\n<\/tr>\n<tr>\n<td>species_name<\/td>\n<td>TEXT<\/td>\n<td>Gadus morhua<\/td>\n<\/tr>\n<tr>\n<td>time_window<\/td>\n<td>TEXT<\/td>\n<td>2020-03<\/td>\n<\/tr>\n<tr>\n<td>start_date<\/td>\n<td>TEXT<\/td>\n<td>2020-03-01<\/td>\n<\/tr>\n<tr>\n<td>end_date<\/td>\n<td>TEXT<\/td>\n<td>2020-03-31<\/td>\n<\/tr>\n<tr>\n<td>sea_area<\/td>\n<td>TEXT<\/td>\n<td>North Atlantic Demo Area<\/td>\n<\/tr>\n<tr>\n<td>records_saved<\/td>\n<td>INTEGER<\/td>\n<td>426<\/td>\n<\/tr>\n<tr>\n<td>created_at<\/td>\n<td>TEXT<\/td>\n<td>2026-06-12T10:30:00&#043;00:00<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h4>8.3 \u53bb\u91cd\u7b56\u7565<\/h4>\n<p>\u672c\u6587\u91c7\u7528\u4e24\u5c42\u53bb\u91cd&#xff1a;<\/p>\n<p>\u7b2c\u4e00\u5c42&#xff1a;occurrence_id \u552f\u4e00\u3002\u5982\u679c\u63a5\u53e3\u8fd4\u56de\u7a33\u5b9a ID&#xff0c;\u5c31\u7528\u5b83\u4f5c\u4e3a\u4e3b\u952e\u3002<\/p>\n<p>\u7b2c\u4e8c\u5c42&#xff1a;content_hash\u3002\u5982\u679c\u6ca1\u6709 ID&#xff0c;\u5c31\u5bf9\u539f\u59cb JSON \u505a sha256&#xff0c;\u751f\u6210\u4e00\u4e2a\u7a33\u5b9a hash&#xff0c;\u4f5c\u4e3a\u515c\u5e95 ID \u6216\u8f85\u52a9\u53bb\u91cd\u5b57\u6bb5\u3002<\/p>\n<p>\u4e3a\u4ec0\u4e48\u4e0d\u53ea\u7528 URL \u53bb\u91cd&#xff1f;\u56e0\u4e3a\u52a8\u6001 API \u540c\u4e00\u4e2a URL \u53ef\u80fd\u8fd4\u56de\u591a\u6761\u8bb0\u5f55&#xff0c;\u800c\u4e00\u6761\u8bb0\u5f55\u4e5f\u53ef\u80fd\u5728\u4e0d\u540c\u65f6\u95f4\u7a97\u53e3\u6216\u4e0d\u540c\u67e5\u8be2\u6761\u4ef6\u91cc\u88ab\u547d\u4e2d\u3002URL \u53bb\u91cd\u9002\u5408\u9875\u9762\u7ea7\u91c7\u96c6&#xff0c;\u4e0d\u9002\u5408 occurrence \u8bb0\u5f55\u7ea7\u91c7\u96c6\u3002<\/p>\n<h4>8.4 \u4ee3\u7801&#xff1a;storage.py<\/h4>\n<p><span class=\"token comment\"># fisheries_spider\/storage.py<\/span><\/p>\n<p><span class=\"token keyword\">from<\/span> __future__ <span class=\"token keyword\">import<\/span> annotations<\/p>\n<p><span class=\"token keyword\">import<\/span> sqlite3<br \/>\n<span class=\"token keyword\">from<\/span> collections <span class=\"token keyword\">import<\/span> Counter<br \/>\n<span class=\"token keyword\">from<\/span> pathlib <span class=\"token keyword\">import<\/span> Path<br \/>\n<span class=\"token keyword\">from<\/span> typing <span class=\"token keyword\">import<\/span> Iterable<\/p>\n<p><span class=\"token keyword\">import<\/span> pandas <span class=\"token keyword\">as<\/span> pd<\/p>\n<p><span class=\"token keyword\">from<\/span> fisheries_spider<span class=\"token punctuation\">.<\/span>parser <span class=\"token keyword\">import<\/span> <span class=\"token punctuation\">(<\/span><br \/>\n    IntensityRecord<span class=\"token punctuation\">,<\/span><br \/>\n    RawOccurrence<span class=\"token punctuation\">,<\/span><br \/>\n    grid_id_from_lon_lat<span class=\"token punctuation\">,<\/span><br \/>\n    intensity_to_dict<span class=\"token punctuation\">,<\/span><br \/>\n    raw_to_dict<span class=\"token punctuation\">,<\/span><br \/>\n<span class=\"token punctuation\">)<\/span><\/p>\n<p><span class=\"token keyword\">class<\/span> <span class=\"token class-name\">SQLiteStorage<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token keyword\">def<\/span> <span class=\"token function\">__init__<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">,<\/span> db_path<span class=\"token punctuation\">:<\/span> Path<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>db_path <span class=\"token operator\">&#061;<\/span> db_path<br \/>\n        self<span class=\"token punctuation\">.<\/span>conn <span class=\"token operator\">&#061;<\/span> sqlite3<span class=\"token punctuation\">.<\/span>connect<span class=\"token punctuation\">(<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">(<\/span>db_path<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>row_factory <span class=\"token operator\">&#061;<\/span> sqlite3<span class=\"token punctuation\">.<\/span>Row<br \/>\n        self<span class=\"token punctuation\">.<\/span>init_db<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">init_db<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        cur <span class=\"token operator\">&#061;<\/span> self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>cursor<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>        cur<span class=\"token punctuation\">.<\/span>execute<span class=\"token punctuation\">(<\/span><br \/>\n            <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n            CREATE TABLE IF NOT EXISTS raw_occurrences (<br \/>\n                occurrence_id TEXT PRIMARY KEY,<br \/>\n                species_name TEXT NOT NULL,<br \/>\n                sea_area TEXT NOT NULL,<br \/>\n                time_window TEXT NOT NULL,<br \/>\n                event_date TEXT,<br \/>\n                longitude REAL NOT NULL,<br \/>\n                latitude REAL NOT NULL,<br \/>\n                dataset_id TEXT,<br \/>\n                basis_of_record TEXT,<br \/>\n                source TEXT NOT NULL,<br \/>\n                request_url TEXT,<br \/>\n                content_hash TEXT NOT NULL,<br \/>\n                fetched_at TEXT NOT NULL<br \/>\n            )<br \/>\n            &#034;&#034;&#034;<\/span><br \/>\n        <span class=\"token punctuation\">)<\/span><\/p>\n<p>        cur<span class=\"token punctuation\">.<\/span>execute<span class=\"token punctuation\">(<\/span><br \/>\n            <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n            CREATE INDEX IF NOT EXISTS idx_raw_species_time<br \/>\n            ON raw_occurrences (species_name, time_window)<br \/>\n            &#034;&#034;&#034;<\/span><br \/>\n        <span class=\"token punctuation\">)<\/span><\/p>\n<p>        cur<span class=\"token punctuation\">.<\/span>execute<span class=\"token punctuation\">(<\/span><br \/>\n            <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n            CREATE INDEX IF NOT EXISTS idx_raw_geo<br \/>\n            ON raw_occurrences (longitude, latitude)<br \/>\n            &#034;&#034;&#034;<\/span><br \/>\n        <span class=\"token punctuation\">)<\/span><\/p>\n<p>        cur<span class=\"token punctuation\">.<\/span>execute<span class=\"token punctuation\">(<\/span><br \/>\n            <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n            CREATE TABLE IF NOT EXISTS resource_intensity (<br \/>\n                species_name TEXT NOT NULL,<br \/>\n                sea_area TEXT NOT NULL,<br \/>\n                time_window TEXT NOT NULL,<br \/>\n                grid_id TEXT NOT NULL,<br \/>\n                grid_lon INTEGER NOT NULL,<br \/>\n                grid_lat INTEGER NOT NULL,<br \/>\n                intensity_index INTEGER NOT NULL,<br \/>\n                source TEXT NOT NULL,<br \/>\n                PRIMARY KEY (species_name, sea_area, time_window, grid_id)<br \/>\n            )<br \/>\n            &#034;&#034;&#034;<\/span><br \/>\n        <span class=\"token punctuation\">)<\/span><\/p>\n<p>        cur<span class=\"token punctuation\">.<\/span>execute<span class=\"token punctuation\">(<\/span><br \/>\n            <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n            CREATE TABLE IF NOT EXISTS request_log (<br \/>\n                id INTEGER PRIMARY KEY AUTOINCREMENT,<br \/>\n                species_name TEXT NOT NULL,<br \/>\n                time_window TEXT NOT NULL,<br \/>\n                start_date TEXT NOT NULL,<br \/>\n                end_date TEXT NOT NULL,<br \/>\n                sea_area TEXT NOT NULL,<br \/>\n                records_saved INTEGER NOT NULL,<br \/>\n                created_at TEXT NOT NULL<br \/>\n            )<br \/>\n            &#034;&#034;&#034;<\/span><br \/>\n        <span class=\"token punctuation\">)<\/span><\/p>\n<p>        self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>commit<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">insert_raw_occurrences<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">,<\/span> rows<span class=\"token punctuation\">:<\/span> Iterable<span class=\"token punctuation\">[<\/span>RawOccurrence<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        sql <span class=\"token operator\">&#061;<\/span> <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n        INSERT OR IGNORE INTO raw_occurrences (<br \/>\n            occurrence_id,<br \/>\n            species_name,<br \/>\n            sea_area,<br \/>\n            time_window,<br \/>\n            event_date,<br \/>\n            longitude,<br \/>\n            latitude,<br \/>\n            dataset_id,<br \/>\n            basis_of_record,<br \/>\n            source,<br \/>\n            request_url,<br \/>\n            content_hash,<br \/>\n            fetched_at<br \/>\n        ) VALUES (<br \/>\n            :occurrence_id,<br \/>\n            :species_name,<br \/>\n            :sea_area,<br \/>\n            :time_window,<br \/>\n            :event_date,<br \/>\n            :longitude,<br \/>\n            :latitude,<br \/>\n            :dataset_id,<br \/>\n            :basis_of_record,<br \/>\n            :source,<br \/>\n            :request_url,<br \/>\n            :content_hash,<br \/>\n            :fetched_at<br \/>\n        )<br \/>\n        &#034;&#034;&#034;<\/span><\/p>\n<p>        items <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">[<\/span>raw_to_dict<span class=\"token punctuation\">(<\/span>row<span class=\"token punctuation\">)<\/span> <span class=\"token keyword\">for<\/span> row <span class=\"token keyword\">in<\/span> rows<span class=\"token punctuation\">]<\/span><br \/>\n        <span class=\"token keyword\">if<\/span> <span class=\"token keyword\">not<\/span> items<span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token keyword\">return<\/span> <span class=\"token number\">0<\/span><\/p>\n<p>        before <span class=\"token operator\">&#061;<\/span> self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>total_changes<br \/>\n        self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>executemany<span class=\"token punctuation\">(<\/span>sql<span class=\"token punctuation\">,<\/span> items<span class=\"token punctuation\">)<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>commit<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><br \/>\n        after <span class=\"token operator\">&#061;<\/span> self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>total_changes<br \/>\n        <span class=\"token keyword\">return<\/span> after <span class=\"token operator\">&#8211;<\/span> before<\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">log_request<\/span><span class=\"token punctuation\">(<\/span><br \/>\n        self<span class=\"token punctuation\">,<\/span><br \/>\n        species_name<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        time_window<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        start_date<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        end_date<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        sea_area<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        records_saved<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        created_at<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>execute<span class=\"token punctuation\">(<\/span><br \/>\n            <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n            INSERT INTO request_log (<br \/>\n                species_name,<br \/>\n                time_window,<br \/>\n                start_date,<br \/>\n                end_date,<br \/>\n                sea_area,<br \/>\n                records_saved,<br \/>\n                created_at<br \/>\n            ) VALUES (?, ?, ?, ?, ?, ?, ?)<br \/>\n            &#034;&#034;&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token punctuation\">(<\/span><br \/>\n                species_name<span class=\"token punctuation\">,<\/span><br \/>\n                time_window<span class=\"token punctuation\">,<\/span><br \/>\n                start_date<span class=\"token punctuation\">,<\/span><br \/>\n                end_date<span class=\"token punctuation\">,<\/span><br \/>\n                sea_area<span class=\"token punctuation\">,<\/span><br \/>\n                records_saved<span class=\"token punctuation\">,<\/span><br \/>\n                created_at<span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">)<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>commit<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">rebuild_intensity<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">,<\/span> grid_size<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">float<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token number\">1.0<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n        Rebuild intensity table from raw_occurrences.<br \/>\n        intensity_index &#061; count of raw occurrence records in each grid\/month\/species.<br \/>\n        &#034;&#034;&#034;<\/span><br \/>\n        df <span class=\"token operator\">&#061;<\/span> pd<span class=\"token punctuation\">.<\/span>read_sql_query<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;SELECT * FROM raw_occurrences&#034;<\/span><span class=\"token punctuation\">,<\/span> self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">)<\/span><\/p>\n<p>        self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>execute<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;DELETE FROM resource_intensity&#034;<\/span><span class=\"token punctuation\">)<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>commit<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>        <span class=\"token keyword\">if<\/span> df<span class=\"token punctuation\">.<\/span>empty<span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token keyword\">return<\/span> <span class=\"token number\">0<\/span><\/p>\n<p>        counter<span class=\"token punctuation\">:<\/span> Counter<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">tuple<\/span><span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">]<\/span> <span class=\"token operator\">&#061;<\/span> Counter<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>        <span class=\"token keyword\">for<\/span> row <span class=\"token keyword\">in<\/span> df<span class=\"token punctuation\">.<\/span>itertuples<span class=\"token punctuation\">(<\/span>index<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">False<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            grid_id<span class=\"token punctuation\">,<\/span> grid_lon<span class=\"token punctuation\">,<\/span> grid_lat <span class=\"token operator\">&#061;<\/span> grid_id_from_lon_lat<span class=\"token punctuation\">(<\/span><br \/>\n                <span class=\"token builtin\">float<\/span><span class=\"token punctuation\">(<\/span>row<span class=\"token punctuation\">.<\/span>longitude<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token builtin\">float<\/span><span class=\"token punctuation\">(<\/span>row<span class=\"token punctuation\">.<\/span>latitude<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                grid_size<span class=\"token operator\">&#061;<\/span>grid_size<span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token punctuation\">)<\/span><br \/>\n            key <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">(<\/span><br \/>\n                row<span class=\"token punctuation\">.<\/span>species_name<span class=\"token punctuation\">,<\/span><br \/>\n                row<span class=\"token punctuation\">.<\/span>sea_area<span class=\"token punctuation\">,<\/span><br \/>\n                row<span class=\"token punctuation\">.<\/span>time_window<span class=\"token punctuation\">,<\/span><br \/>\n                grid_id<span class=\"token punctuation\">,<\/span><br \/>\n                grid_lon<span class=\"token punctuation\">,<\/span><br \/>\n                grid_lat<span class=\"token punctuation\">,<\/span><br \/>\n                row<span class=\"token punctuation\">.<\/span>source<span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token punctuation\">)<\/span><br \/>\n            counter<span class=\"token punctuation\">[<\/span>key<span class=\"token punctuation\">]<\/span> <span class=\"token operator\">&#043;&#061;<\/span> <span class=\"token number\">1<\/span><\/p>\n<p>        intensity_rows <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">[<\/span><br \/>\n            IntensityRecord<span class=\"token punctuation\">(<\/span><br \/>\n                species_name<span class=\"token operator\">&#061;<\/span>species_name<span class=\"token punctuation\">,<\/span><br \/>\n                sea_area<span class=\"token operator\">&#061;<\/span>sea_area<span class=\"token punctuation\">,<\/span><br \/>\n                time_window<span class=\"token operator\">&#061;<\/span>time_window<span class=\"token punctuation\">,<\/span><br \/>\n                grid_id<span class=\"token operator\">&#061;<\/span>grid_id<span class=\"token punctuation\">,<\/span><br \/>\n                grid_lon<span class=\"token operator\">&#061;<\/span>grid_lon<span class=\"token punctuation\">,<\/span><br \/>\n                grid_lat<span class=\"token operator\">&#061;<\/span>grid_lat<span class=\"token punctuation\">,<\/span><br \/>\n                intensity_index<span class=\"token operator\">&#061;<\/span>count<span class=\"token punctuation\">,<\/span><br \/>\n                source<span class=\"token operator\">&#061;<\/span>source<span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token punctuation\">)<\/span><br \/>\n            <span class=\"token keyword\">for<\/span> <span class=\"token punctuation\">(<\/span><br \/>\n                species_name<span class=\"token punctuation\">,<\/span><br \/>\n                sea_area<span class=\"token punctuation\">,<\/span><br \/>\n                time_window<span class=\"token punctuation\">,<\/span><br \/>\n                grid_id<span class=\"token punctuation\">,<\/span><br \/>\n                grid_lon<span class=\"token punctuation\">,<\/span><br \/>\n                grid_lat<span class=\"token punctuation\">,<\/span><br \/>\n                source<span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span> count <span class=\"token keyword\">in<\/span> counter<span class=\"token punctuation\">.<\/span>items<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><br \/>\n        <span class=\"token punctuation\">]<\/span><\/p>\n<p>        sql <span class=\"token operator\">&#061;<\/span> <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n        INSERT OR REPLACE INTO resource_intensity (<br \/>\n            species_name,<br \/>\n            sea_area,<br \/>\n            time_window,<br \/>\n            grid_id,<br \/>\n            grid_lon,<br \/>\n            grid_lat,<br \/>\n            intensity_index,<br \/>\n            source<br \/>\n        ) VALUES (<br \/>\n            :species_name,<br \/>\n            :sea_area,<br \/>\n            :time_window,<br \/>\n            :grid_id,<br \/>\n            :grid_lon,<br \/>\n            :grid_lat,<br \/>\n            :intensity_index,<br \/>\n            :source<br \/>\n        )<br \/>\n        &#034;&#034;&#034;<\/span><\/p>\n<p>        items <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">[<\/span>intensity_to_dict<span class=\"token punctuation\">(<\/span>row<span class=\"token punctuation\">)<\/span> <span class=\"token keyword\">for<\/span> row <span class=\"token keyword\">in<\/span> intensity_rows<span class=\"token punctuation\">]<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>executemany<span class=\"token punctuation\">(<\/span>sql<span class=\"token punctuation\">,<\/span> items<span class=\"token punctuation\">)<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>commit<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>        <span class=\"token keyword\">return<\/span> <span class=\"token builtin\">len<\/span><span class=\"token punctuation\">(<\/span>items<span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">export_raw_csv<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">,<\/span> output_path<span class=\"token punctuation\">:<\/span> Path<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        df <span class=\"token operator\">&#061;<\/span> pd<span class=\"token punctuation\">.<\/span>read_sql_query<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;SELECT * FROM raw_occurrences&#034;<\/span><span class=\"token punctuation\">,<\/span> self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">)<\/span><br \/>\n        df<span class=\"token punctuation\">.<\/span>to_csv<span class=\"token punctuation\">(<\/span>output_path<span class=\"token punctuation\">,<\/span> index<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">False<\/span><span class=\"token punctuation\">,<\/span> encoding<span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#034;utf-8-sig&#034;<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">export_intensity_csv<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">,<\/span> output_path<span class=\"token punctuation\">:<\/span> Path<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        df <span class=\"token operator\">&#061;<\/span> pd<span class=\"token punctuation\">.<\/span>read_sql_query<span class=\"token punctuation\">(<\/span><br \/>\n            <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n            SELECT<br \/>\n                species_name,<br \/>\n                sea_area,<br \/>\n                time_window,<br \/>\n                grid_id,<br \/>\n                grid_lon,<br \/>\n                grid_lat,<br \/>\n                intensity_index,<br \/>\n                source<br \/>\n            FROM resource_intensity<br \/>\n            ORDER BY species_name, time_window, intensity_index DESC<br \/>\n            &#034;&#034;&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">)<\/span><br \/>\n        df<span class=\"token punctuation\">.<\/span>to_csv<span class=\"token punctuation\">(<\/span>output_path<span class=\"token punctuation\">,<\/span> index<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">False<\/span><span class=\"token punctuation\">,<\/span> encoding<span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#034;utf-8-sig&#034;<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">read_intensity_df<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> pd<span class=\"token punctuation\">.<\/span>DataFrame<span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token keyword\">return<\/span> pd<span class=\"token punctuation\">.<\/span>read_sql_query<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;SELECT * FROM resource_intensity&#034;<\/span><span class=\"token punctuation\">,<\/span> self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">close<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        self<span class=\"token punctuation\">.<\/span>conn<span class=\"token punctuation\">.<\/span>close<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>\u8fd9\u90e8\u5206\u4ee3\u7801\u6709\u4e00\u4e2a\u7ec6\u8282&#xff1a;\u805a\u5408\u8868\u4e0d\u662f\u8fb9\u6293\u8fb9\u7b97&#xff0c;\u800c\u662f\u5728\u539f\u59cb\u8868\u5199\u5165\u540e\u7edf\u4e00\u91cd\u5efa\u3002\u8fd9\u6837\u8c03\u8bd5\u66f4\u65b9\u4fbf\u3002\u4f60\u4fee\u6539\u7f51\u683c\u5927\u5c0f\u6216\u5f3a\u5ea6\u5b9a\u4e49\u540e&#xff0c;\u53ef\u4ee5\u76f4\u63a5\u91cd\u8dd1\u805a\u5408&#xff0c;\u4e0d\u9700\u8981\u91cd\u65b0\u8bf7\u6c42\u63a5\u53e3\u3002<\/p>\n<hr \/>\n<h3>9\ufe0f\u20e3 \u8fd0\u884c\u65b9\u5f0f\u4e0e\u7ed3\u679c\u5c55\u793a&#xff08;\u5fc5\u5199&#xff09;<\/h3>\n<h4>9.1 \u5165\u53e3\u6587\u4ef6&#xff1a;run_spider.py<\/h4>\n<p><span class=\"token comment\"># run_spider.py<\/span><\/p>\n<p><span class=\"token keyword\">from<\/span> __future__ <span class=\"token keyword\">import<\/span> annotations<\/p>\n<p><span class=\"token keyword\">import<\/span> logging<br \/>\n<span class=\"token keyword\">from<\/span> typing <span class=\"token keyword\">import<\/span> List<\/p>\n<p><span class=\"token keyword\">from<\/span> tqdm <span class=\"token keyword\">import<\/span> tqdm<\/p>\n<p><span class=\"token keyword\">from<\/span> fisheries_spider<span class=\"token punctuation\">.<\/span>config <span class=\"token keyword\">import<\/span> SpiderConfig<br \/>\n<span class=\"token keyword\">from<\/span> fisheries_spider<span class=\"token punctuation\">.<\/span>fetcher <span class=\"token keyword\">import<\/span> ObisFetcher<br \/>\n<span class=\"token keyword\">from<\/span> fisheries_spider<span class=\"token punctuation\">.<\/span>parser <span class=\"token keyword\">import<\/span> now_utc_iso<span class=\"token punctuation\">,<\/span> parse_occurrence<br \/>\n<span class=\"token keyword\">from<\/span> fisheries_spider<span class=\"token punctuation\">.<\/span>storage <span class=\"token keyword\">import<\/span> SQLiteStorage<br \/>\n<span class=\"token keyword\">from<\/span> fisheries_spider<span class=\"token punctuation\">.<\/span>time_utils <span class=\"token keyword\">import<\/span> iter_month_windows<br \/>\n<span class=\"token keyword\">from<\/span> fisheries_spider<span class=\"token punctuation\">.<\/span>exporter <span class=\"token keyword\">import<\/span> export_intensity_geojson<br \/>\n<span class=\"token keyword\">from<\/span> fisheries_spider<span class=\"token punctuation\">.<\/span>map_builder <span class=\"token keyword\">import<\/span> build_timeline_map<\/p>\n<p>logging<span class=\"token punctuation\">.<\/span>basicConfig<span class=\"token punctuation\">(<\/span><br \/>\n    level<span class=\"token operator\">&#061;<\/span>logging<span class=\"token punctuation\">.<\/span>INFO<span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token builtin\">format<\/span><span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#034;%(asctime)s | %(levelname)s | %(name)s | %(message)s&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n<span class=\"token punctuation\">)<\/span><\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">main<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    config <span class=\"token operator\">&#061;<\/span> SpiderConfig<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token comment\"># \u793a\u4f8b\u7269\u79cd\u3002\u5b9e\u9645\u9879\u76ee\u53ef\u6269\u5c55\u6210\u591a\u4e2a\u7269\u79cd\u3002<\/span><br \/>\n    species_list<span class=\"token punctuation\">:<\/span> List<span class=\"token punctuation\">[<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">]<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">[<\/span><br \/>\n        <span class=\"token string\">&#034;Gadus morhua&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">]<\/span><\/p>\n<p>    <span class=\"token comment\"># \u793a\u4f8b\u6d77\u57df&#xff1a;\u5317\u5927\u897f\u6d0b\u90e8\u5206\u533a\u57df\u3002<\/span><br \/>\n    <span class=\"token comment\"># WKT \u5750\u6807\u987a\u5e8f\u662f lon lat&#xff0c;\u4e0d\u662f lat lon\u3002<\/span><br \/>\n    sea_area <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;North Atlantic Demo Area&#034;<\/span><br \/>\n    geometry_wkt <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;POLYGON((-75 35,-75 55,-45 55,-45 35,-75 35))&#034;<\/span><\/p>\n<p>    start_month <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;2020-01&#034;<\/span><br \/>\n    end_month <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;2020-12&#034;<\/span><\/p>\n<p>    fetcher <span class=\"token operator\">&#061;<\/span> ObisFetcher<span class=\"token punctuation\">(<\/span>config<span class=\"token punctuation\">)<\/span><br \/>\n    storage <span class=\"token operator\">&#061;<\/span> SQLiteStorage<span class=\"token punctuation\">(<\/span>config<span class=\"token punctuation\">.<\/span>sqlite_path<span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token comment\"># \u5efa\u8bae\u9650\u5236\u5b57\u6bb5&#xff0c;\u51cf\u5c11\u54cd\u5e94\u4f53\u79ef\u3002<\/span><br \/>\n    <span class=\"token comment\"># \u6ce8\u610f&#xff1a;\u5982\u679c\u4f60\u540e\u7eed\u8981\u66f4\u591a\u5b57\u6bb5&#xff0c;\u53ef\u4ee5\u5728\u8fd9\u91cc\u589e\u52a0\u3002<\/span><br \/>\n    fields <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;,&#034;<\/span><span class=\"token punctuation\">.<\/span>join<span class=\"token punctuation\">(<\/span><br \/>\n        <span class=\"token punctuation\">[<\/span><br \/>\n            <span class=\"token string\">&#034;id&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token string\">&#034;scientificName&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token string\">&#034;eventDate&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token string\">&#034;decimalLongitude&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token string\">&#034;decimalLatitude&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token string\">&#034;dataset_id&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token string\">&#034;basisOfRecord&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">]<\/span><br \/>\n    <span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">try<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token keyword\">for<\/span> species_name <span class=\"token keyword\">in<\/span> species_list<span class=\"token punctuation\">:<\/span><br \/>\n            windows <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">list<\/span><span class=\"token punctuation\">(<\/span>iter_month_windows<span class=\"token punctuation\">(<\/span>start_month<span class=\"token punctuation\">,<\/span> end_month<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>            <span class=\"token keyword\">for<\/span> window <span class=\"token keyword\">in<\/span> tqdm<span class=\"token punctuation\">(<\/span>windows<span class=\"token punctuation\">,<\/span> desc<span class=\"token operator\">&#061;<\/span><span class=\"token string-interpolation\"><span class=\"token string\">f&#034;Collecting <\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>species_name<span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">&#034;<\/span><\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n                parsed_rows <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token punctuation\">]<\/span><\/p>\n<p>                <span class=\"token keyword\">for<\/span> record <span class=\"token keyword\">in<\/span> fetcher<span class=\"token punctuation\">.<\/span>iter_occurrences<span class=\"token punctuation\">(<\/span><br \/>\n                    scientific_name<span class=\"token operator\">&#061;<\/span>species_name<span class=\"token punctuation\">,<\/span><br \/>\n                    start_date<span class=\"token operator\">&#061;<\/span>window<span class=\"token punctuation\">.<\/span>start_date<span class=\"token punctuation\">,<\/span><br \/>\n                    end_date<span class=\"token operator\">&#061;<\/span>window<span class=\"token punctuation\">.<\/span>end_date<span class=\"token punctuation\">,<\/span><br \/>\n                    geometry_wkt<span class=\"token operator\">&#061;<\/span>geometry_wkt<span class=\"token punctuation\">,<\/span><br \/>\n                    fields<span class=\"token operator\">&#061;<\/span>fields<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n                    parsed <span class=\"token operator\">&#061;<\/span> parse_occurrence<span class=\"token punctuation\">(<\/span><br \/>\n                        record<span class=\"token operator\">&#061;<\/span>record<span class=\"token punctuation\">,<\/span><br \/>\n                        fallback_species<span class=\"token operator\">&#061;<\/span>species_name<span class=\"token punctuation\">,<\/span><br \/>\n                        sea_area<span class=\"token operator\">&#061;<\/span>sea_area<span class=\"token punctuation\">,<\/span><br \/>\n                        time_window<span class=\"token operator\">&#061;<\/span>window<span class=\"token punctuation\">.<\/span>month<span class=\"token punctuation\">,<\/span><br \/>\n                        source<span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#034;OBIS API&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                    <span class=\"token punctuation\">)<\/span><br \/>\n                    <span class=\"token keyword\">if<\/span> parsed <span class=\"token keyword\">is<\/span> <span class=\"token keyword\">not<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n                        parsed_rows<span class=\"token punctuation\">.<\/span>append<span class=\"token punctuation\">(<\/span>parsed<span class=\"token punctuation\">)<\/span><\/p>\n<p>                saved <span class=\"token operator\">&#061;<\/span> storage<span class=\"token punctuation\">.<\/span>insert_raw_occurrences<span class=\"token punctuation\">(<\/span>parsed_rows<span class=\"token punctuation\">)<\/span><br \/>\n                storage<span class=\"token punctuation\">.<\/span>log_request<span class=\"token punctuation\">(<\/span><br \/>\n                    species_name<span class=\"token operator\">&#061;<\/span>species_name<span class=\"token punctuation\">,<\/span><br \/>\n                    time_window<span class=\"token operator\">&#061;<\/span>window<span class=\"token punctuation\">.<\/span>month<span class=\"token punctuation\">,<\/span><br \/>\n                    start_date<span class=\"token operator\">&#061;<\/span>window<span class=\"token punctuation\">.<\/span>start_date<span class=\"token punctuation\">,<\/span><br \/>\n                    end_date<span class=\"token operator\">&#061;<\/span>window<span class=\"token punctuation\">.<\/span>end_date<span class=\"token punctuation\">,<\/span><br \/>\n                    sea_area<span class=\"token operator\">&#061;<\/span>sea_area<span class=\"token punctuation\">,<\/span><br \/>\n                    records_saved<span class=\"token operator\">&#061;<\/span>saved<span class=\"token punctuation\">,<\/span><br \/>\n                    created_at<span class=\"token operator\">&#061;<\/span>now_utc_iso<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token punctuation\">)<\/span><\/p>\n<p>        intensity_count <span class=\"token operator\">&#061;<\/span> storage<span class=\"token punctuation\">.<\/span>rebuild_intensity<span class=\"token punctuation\">(<\/span>grid_size<span class=\"token operator\">&#061;<\/span><span class=\"token number\">1.0<\/span><span class=\"token punctuation\">)<\/span><br \/>\n        logging<span class=\"token punctuation\">.<\/span>info<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;Rebuilt intensity table. rows&#061;%s&#034;<\/span><span class=\"token punctuation\">,<\/span> intensity_count<span class=\"token punctuation\">)<\/span><\/p>\n<p>        storage<span class=\"token punctuation\">.<\/span>export_raw_csv<span class=\"token punctuation\">(<\/span>config<span class=\"token punctuation\">.<\/span>raw_csv_path<span class=\"token punctuation\">)<\/span><br \/>\n        storage<span class=\"token punctuation\">.<\/span>export_intensity_csv<span class=\"token punctuation\">(<\/span>config<span class=\"token punctuation\">.<\/span>intensity_csv_path<span class=\"token punctuation\">)<\/span><br \/>\n        export_intensity_geojson<span class=\"token punctuation\">(<\/span>storage<span class=\"token punctuation\">,<\/span> config<span class=\"token punctuation\">.<\/span>intensity_geojson_path<span class=\"token punctuation\">)<\/span><br \/>\n        build_timeline_map<span class=\"token punctuation\">(<\/span><br \/>\n            geojson_path<span class=\"token operator\">&#061;<\/span>config<span class=\"token punctuation\">.<\/span>intensity_geojson_path<span class=\"token punctuation\">,<\/span><br \/>\n            output_html_path<span class=\"token operator\">&#061;<\/span>config<span class=\"token punctuation\">.<\/span>timeline_map_path<span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">)<\/span><\/p>\n<p>        logging<span class=\"token punctuation\">.<\/span>info<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;Raw CSV: %s&#034;<\/span><span class=\"token punctuation\">,<\/span> config<span class=\"token punctuation\">.<\/span>raw_csv_path<span class=\"token punctuation\">)<\/span><br \/>\n        logging<span class=\"token punctuation\">.<\/span>info<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;Intensity CSV: %s&#034;<\/span><span class=\"token punctuation\">,<\/span> config<span class=\"token punctuation\">.<\/span>intensity_csv_path<span class=\"token punctuation\">)<\/span><br \/>\n        logging<span class=\"token punctuation\">.<\/span>info<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;GeoJSON: %s&#034;<\/span><span class=\"token punctuation\">,<\/span> config<span class=\"token punctuation\">.<\/span>intensity_geojson_path<span class=\"token punctuation\">)<\/span><br \/>\n        logging<span class=\"token punctuation\">.<\/span>info<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;Timeline map: %s&#034;<\/span><span class=\"token punctuation\">,<\/span> config<span class=\"token punctuation\">.<\/span>timeline_map_path<span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">finally<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        storage<span class=\"token punctuation\">.<\/span>close<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p><span class=\"token keyword\">if<\/span> __name__ <span class=\"token operator\">&#061;&#061;<\/span> <span class=\"token string\">&#034;__main__&#034;<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    main<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<h4>9.2 \u5bfc\u51fa GeoJSON&#xff1a;exporter.py<\/h4>\n<p><span class=\"token comment\"># fisheries_spider\/exporter.py<\/span><\/p>\n<p><span class=\"token keyword\">from<\/span> __future__ <span class=\"token keyword\">import<\/span> annotations<\/p>\n<p><span class=\"token keyword\">import<\/span> json<br \/>\n<span class=\"token keyword\">from<\/span> pathlib <span class=\"token keyword\">import<\/span> Path<br \/>\n<span class=\"token keyword\">from<\/span> typing <span class=\"token keyword\">import<\/span> Dict<span class=\"token punctuation\">,<\/span> List<\/p>\n<p><span class=\"token keyword\">from<\/span> fisheries_spider<span class=\"token punctuation\">.<\/span>storage <span class=\"token keyword\">import<\/span> SQLiteStorage<\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">grid_polygon<\/span><span class=\"token punctuation\">(<\/span>grid_lon<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">,<\/span> grid_lat<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">,<\/span> size<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">float<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token number\">1.0<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token builtin\">list<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n    Build a simple polygon for a lon\/lat grid cell.<\/p>\n<p>    Coordinates follow GeoJSON order: [lon, lat].<br \/>\n    &#034;&#034;&#034;<\/span><br \/>\n    lon0 <span class=\"token operator\">&#061;<\/span> grid_lon<br \/>\n    lat0 <span class=\"token operator\">&#061;<\/span> grid_lat<br \/>\n    lon1 <span class=\"token operator\">&#061;<\/span> grid_lon <span class=\"token operator\">&#043;<\/span> size<br \/>\n    lat1 <span class=\"token operator\">&#061;<\/span> grid_lat <span class=\"token operator\">&#043;<\/span> size<\/p>\n<p>    <span class=\"token keyword\">return<\/span> <span class=\"token punctuation\">[<\/span><br \/>\n        <span class=\"token punctuation\">[<\/span>lon0<span class=\"token punctuation\">,<\/span> lat0<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">[<\/span>lon1<span class=\"token punctuation\">,<\/span> lat0<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">[<\/span>lon1<span class=\"token punctuation\">,<\/span> lat1<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">[<\/span>lon0<span class=\"token punctuation\">,<\/span> lat1<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">[<\/span>lon0<span class=\"token punctuation\">,<\/span> lat0<span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">]<\/span><\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">export_intensity_geojson<\/span><span class=\"token punctuation\">(<\/span><br \/>\n    storage<span class=\"token punctuation\">:<\/span> SQLiteStorage<span class=\"token punctuation\">,<\/span><br \/>\n    output_path<span class=\"token punctuation\">:<\/span> Path<span class=\"token punctuation\">,<\/span><br \/>\n    grid_size<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">float<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token number\">1.0<\/span><span class=\"token punctuation\">,<\/span><br \/>\n<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    df <span class=\"token operator\">&#061;<\/span> storage<span class=\"token punctuation\">.<\/span>read_intensity_df<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    features<span class=\"token punctuation\">:<\/span> List<span class=\"token punctuation\">[<\/span>Dict<span class=\"token punctuation\">]<\/span> <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token punctuation\">]<\/span><\/p>\n<p>    <span class=\"token keyword\">if<\/span> <span class=\"token keyword\">not<\/span> df<span class=\"token punctuation\">.<\/span>empty<span class=\"token punctuation\">:<\/span><br \/>\n        max_intensity <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">(<\/span>df<span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;intensity_index&#034;<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">.<\/span><span class=\"token builtin\">max<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    <span class=\"token keyword\">else<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        max_intensity <span class=\"token operator\">&#061;<\/span> <span class=\"token number\">0<\/span><\/p>\n<p>    <span class=\"token keyword\">for<\/span> row <span class=\"token keyword\">in<\/span> df<span class=\"token punctuation\">.<\/span>itertuples<span class=\"token punctuation\">(<\/span>index<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">False<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        intensity <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">(<\/span>row<span class=\"token punctuation\">.<\/span>intensity_index<span class=\"token punctuation\">)<\/span><\/p>\n<p>        feature <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">{<\/span><br \/>\n            <span class=\"token string\">&#034;type&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;Feature&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token string\">&#034;geometry&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">{<\/span><br \/>\n                <span class=\"token string\">&#034;type&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;Polygon&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;coordinates&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">[<\/span><br \/>\n                    grid_polygon<span class=\"token punctuation\">(<\/span><br \/>\n                        grid_lon<span class=\"token operator\">&#061;<\/span><span class=\"token builtin\">int<\/span><span class=\"token punctuation\">(<\/span>row<span class=\"token punctuation\">.<\/span>grid_lon<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                        grid_lat<span class=\"token operator\">&#061;<\/span><span class=\"token builtin\">int<\/span><span class=\"token punctuation\">(<\/span>row<span class=\"token punctuation\">.<\/span>grid_lat<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                        size<span class=\"token operator\">&#061;<\/span>grid_size<span class=\"token punctuation\">,<\/span><br \/>\n                    <span class=\"token punctuation\">)<\/span><br \/>\n                <span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token string\">&#034;properties&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">{<\/span><br \/>\n                <span class=\"token string\">&#034;species_name&#034;<\/span><span class=\"token punctuation\">:<\/span> row<span class=\"token punctuation\">.<\/span>species_name<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;sea_area&#034;<\/span><span class=\"token punctuation\">:<\/span> row<span class=\"token punctuation\">.<\/span>sea_area<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;time_window&#034;<\/span><span class=\"token punctuation\">:<\/span> row<span class=\"token punctuation\">.<\/span>time_window<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;time&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string-interpolation\"><span class=\"token string\">f&#034;<\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>row<span class=\"token punctuation\">.<\/span>time_window<span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">-01&#034;<\/span><\/span><span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;grid_id&#034;<\/span><span class=\"token punctuation\">:<\/span> row<span class=\"token punctuation\">.<\/span>grid_id<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;grid_lon&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">(<\/span>row<span class=\"token punctuation\">.<\/span>grid_lon<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;grid_lat&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">int<\/span><span class=\"token punctuation\">(<\/span>row<span class=\"token punctuation\">.<\/span>grid_lat<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;intensity_index&#034;<\/span><span class=\"token punctuation\">:<\/span> intensity<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;max_intensity_in_export&#034;<\/span><span class=\"token punctuation\">:<\/span> max_intensity<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;source&#034;<\/span><span class=\"token punctuation\">:<\/span> row<span class=\"token punctuation\">.<\/span>source<span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;popup&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">(<\/span><br \/>\n                    <span class=\"token string-interpolation\"><span class=\"token string\">f&#034;Species: <\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>row<span class=\"token punctuation\">.<\/span>species_name<span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">&lt;br&gt;&#034;<\/span><\/span><br \/>\n                    <span class=\"token string-interpolation\"><span class=\"token string\">f&#034;Sea area: <\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>row<span class=\"token punctuation\">.<\/span>sea_area<span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">&lt;br&gt;&#034;<\/span><\/span><br \/>\n                    <span class=\"token string-interpolation\"><span class=\"token string\">f&#034;Month: <\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>row<span class=\"token punctuation\">.<\/span>time_window<span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">&lt;br&gt;&#034;<\/span><\/span><br \/>\n                    <span class=\"token string-interpolation\"><span class=\"token string\">f&#034;Grid: <\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>row<span class=\"token punctuation\">.<\/span>grid_id<span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">&lt;br&gt;&#034;<\/span><\/span><br \/>\n                    <span class=\"token string-interpolation\"><span class=\"token string\">f&#034;Intensity: <\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>intensity<span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">&lt;br&gt;&#034;<\/span><\/span><br \/>\n                    <span class=\"token string-interpolation\"><span class=\"token string\">f&#034;Source: <\/span><span class=\"token interpolation\"><span class=\"token punctuation\">{<\/span>row<span class=\"token punctuation\">.<\/span>source<span class=\"token punctuation\">}<\/span><\/span><span class=\"token string\">&#034;<\/span><\/span><br \/>\n                <span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">}<\/span><br \/>\n        features<span class=\"token punctuation\">.<\/span>append<span class=\"token punctuation\">(<\/span>feature<span class=\"token punctuation\">)<\/span><\/p>\n<p>    geojson <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">{<\/span><br \/>\n        <span class=\"token string\">&#034;type&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;FeatureCollection&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token string\">&#034;features&#034;<\/span><span class=\"token punctuation\">:<\/span> features<span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">}<\/span><\/p>\n<p>    output_path<span class=\"token punctuation\">.<\/span>parent<span class=\"token punctuation\">.<\/span>mkdir<span class=\"token punctuation\">(<\/span>parents<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">,<\/span> exist_ok<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    output_path<span class=\"token punctuation\">.<\/span>write_text<span class=\"token punctuation\">(<\/span><br \/>\n        json<span class=\"token punctuation\">.<\/span>dumps<span class=\"token punctuation\">(<\/span>geojson<span class=\"token punctuation\">,<\/span> ensure_ascii<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">False<\/span><span class=\"token punctuation\">,<\/span> indent<span class=\"token operator\">&#061;<\/span><span class=\"token number\">2<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        encoding<span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#034;utf-8&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">)<\/span><\/p>\n<h4>9.3 \u751f\u6210\u65f6\u95f4\u8f74\u5730\u56fe&#xff1a;map_builder.py<\/h4>\n<p><span class=\"token comment\"># fisheries_spider\/map_builder.py<\/span><\/p>\n<p><span class=\"token keyword\">from<\/span> __future__ <span class=\"token keyword\">import<\/span> annotations<\/p>\n<p><span class=\"token keyword\">import<\/span> json<br \/>\n<span class=\"token keyword\">from<\/span> pathlib <span class=\"token keyword\">import<\/span> Path<\/p>\n<p><span class=\"token keyword\">import<\/span> folium<br \/>\n<span class=\"token keyword\">from<\/span> folium<span class=\"token punctuation\">.<\/span>plugins <span class=\"token keyword\">import<\/span> TimestampedGeoJson<\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">style_function<\/span><span class=\"token punctuation\">(<\/span>feature<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">dict<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token builtin\">dict<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    props <span class=\"token operator\">&#061;<\/span> feature<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;properties&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token punctuation\">{<\/span><span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    intensity <span class=\"token operator\">&#061;<\/span> props<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;intensity_index&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token number\">0<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token keyword\">or<\/span> <span class=\"token number\">0<\/span><br \/>\n    max_intensity <span class=\"token operator\">&#061;<\/span> props<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;max_intensity_in_export&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token number\">1<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token keyword\">or<\/span> <span class=\"token number\">1<\/span><\/p>\n<p>    ratio <span class=\"token operator\">&#061;<\/span> <span class=\"token builtin\">min<\/span><span class=\"token punctuation\">(<\/span><span class=\"token builtin\">float<\/span><span class=\"token punctuation\">(<\/span>intensity<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">\/<\/span> <span class=\"token builtin\">float<\/span><span class=\"token punctuation\">(<\/span>max_intensity<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token number\">1.0<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token keyword\">if<\/span> max_intensity <span class=\"token keyword\">else<\/span> <span class=\"token number\">0<\/span><\/p>\n<p>    <span class=\"token comment\"># Folium style_function can use explicit colors.<\/span><br \/>\n    <span class=\"token comment\"># This is map styling, not chart styling.<\/span><br \/>\n    <span class=\"token keyword\">if<\/span> ratio <span class=\"token operator\">&gt;&#061;<\/span> <span class=\"token number\">0.75<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        fill_color <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;#800026&#034;<\/span><br \/>\n    <span class=\"token keyword\">elif<\/span> ratio <span class=\"token operator\">&gt;&#061;<\/span> <span class=\"token number\">0.5<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        fill_color <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;#BD0026&#034;<\/span><br \/>\n    <span class=\"token keyword\">elif<\/span> ratio <span class=\"token operator\">&gt;&#061;<\/span> <span class=\"token number\">0.25<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        fill_color <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;#E31A1C&#034;<\/span><br \/>\n    <span class=\"token keyword\">elif<\/span> ratio <span class=\"token operator\">&gt;<\/span> <span class=\"token number\">0<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        fill_color <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;#FC4E2A&#034;<\/span><br \/>\n    <span class=\"token keyword\">else<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        fill_color <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;#FFEDA0&#034;<\/span><\/p>\n<p>    <span class=\"token keyword\">return<\/span> <span class=\"token punctuation\">{<\/span><br \/>\n        <span class=\"token string\">&#034;fillColor&#034;<\/span><span class=\"token punctuation\">:<\/span> fill_color<span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token string\">&#034;color&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;#333333&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token string\">&#034;weight&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">0.5<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token string\">&#034;fillOpacity&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">0.55<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">}<\/span><\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">build_timeline_map<\/span><span class=\"token punctuation\">(<\/span>geojson_path<span class=\"token punctuation\">:<\/span> Path<span class=\"token punctuation\">,<\/span> output_html_path<span class=\"token punctuation\">:<\/span> Path<span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token boolean\">None<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    data <span class=\"token operator\">&#061;<\/span> json<span class=\"token punctuation\">.<\/span>loads<span class=\"token punctuation\">(<\/span>geojson_path<span class=\"token punctuation\">.<\/span>read_text<span class=\"token punctuation\">(<\/span>encoding<span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#034;utf-8&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>    fmap <span class=\"token operator\">&#061;<\/span> folium<span class=\"token punctuation\">.<\/span>Map<span class=\"token punctuation\">(<\/span><br \/>\n        location<span class=\"token operator\">&#061;<\/span><span class=\"token punctuation\">[<\/span><span class=\"token number\">45<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token number\">60<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        zoom_start<span class=\"token operator\">&#061;<\/span><span class=\"token number\">4<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        tiles<span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#034;CartoDB positron&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">)<\/span><\/p>\n<p>    TimestampedGeoJson<span class=\"token punctuation\">(<\/span><br \/>\n        data<span class=\"token operator\">&#061;<\/span>data<span class=\"token punctuation\">,<\/span><br \/>\n        period<span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#034;P1M&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        duration<span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#034;P1M&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        transition_time<span class=\"token operator\">&#061;<\/span><span class=\"token number\">400<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        auto_play<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">False<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        loop<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">False<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        add_last_point<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">False<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        time_slider_drag_update<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        style<span class=\"token operator\">&#061;<\/span>style_function<span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">.<\/span>add_to<span class=\"token punctuation\">(<\/span>fmap<span class=\"token punctuation\">)<\/span><\/p>\n<p>    folium<span class=\"token punctuation\">.<\/span>LayerControl<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">.<\/span>add_to<span class=\"token punctuation\">(<\/span>fmap<span class=\"token punctuation\">)<\/span><\/p>\n<p>    output_html_path<span class=\"token punctuation\">.<\/span>parent<span class=\"token punctuation\">.<\/span>mkdir<span class=\"token punctuation\">(<\/span>parents<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">,<\/span> exist_ok<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    fmap<span class=\"token punctuation\">.<\/span>save<span class=\"token punctuation\">(<\/span><span class=\"token builtin\">str<\/span><span class=\"token punctuation\">(<\/span>output_html_path<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>\u8fd9\u4e2a HTML \u5730\u56fe\u4e0d\u662f\u622a\u56fe&#xff0c;\u800c\u662f\u4e00\u4e2a\u53ef\u4ea4\u4e92\u9875\u9762\u3002\u6253\u5f00\u540e\u53ef\u4ee5\u62d6\u52a8\u65f6\u95f4\u8f74&#xff0c;\u67e5\u770b\u4e0d\u540c\u6708\u4efd\u7684\u7f51\u683c\u5f3a\u5ea6\u53d8\u5316\u3002\u8fd9\u4e2a\u6548\u679c\u548c\u5f88\u591a\u52a8\u6001\u5206\u5e03\u56fe\u7684\u57fa\u672c\u4ea4\u4e92\u662f\u4e00\u81f4\u7684\u3002<\/p>\n<h4>9.4 \u542f\u52a8\u547d\u4ee4<\/h4>\n<p>\u5728\u9879\u76ee\u6839\u76ee\u5f55\u8fd0\u884c&#xff1a;<\/p>\n<p>python run_spider.py<\/p>\n<p>\u5982\u679c\u6210\u529f&#xff0c;\u4f1a\u770b\u5230\u7c7b\u4f3c\u65e5\u5fd7&#xff1a;<\/p>\n<p>Collecting Gadus morhua: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 12\/12 [00:42&lt;00:00,  3.51s\/it]<br \/>\n2026-06-12 10:30:12 | INFO | root | Rebuilt intensity table. rows&#061;86<br \/>\n2026-06-12 10:30:12 | INFO | root | Raw CSV: data\/processed\/raw_occurrences.csv<br \/>\n2026-06-12 10:30:12 | INFO | root | Intensity CSV: data\/processed\/resource_intensity.csv<br \/>\n2026-06-12 10:30:12 | INFO | root | GeoJSON: data\/processed\/resource_intensity.geojson<br \/>\n2026-06-12 10:30:12 | INFO | root | Timeline map: data\/maps\/resource_timeline_map.html<\/p>\n<h4>9.5 \u8f93\u51fa\u8def\u5f84<\/h4>\n<p>\u8f93\u51fa\u6587\u4ef6\u5982\u4e0b&#xff1a;<\/p>\n<p>data\/processed\/fisheries_dynamic.sqlite3<br \/>\ndata\/processed\/raw_occurrences.csv<br \/>\ndata\/processed\/resource_intensity.csv<br \/>\ndata\/processed\/resource_intensity.geojson<br \/>\ndata\/maps\/resource_timeline_map.html<\/p>\n<p>SQLite \u4e2d\u5305\u542b\u4e09\u5f20\u8868&#xff1a;<\/p>\n<p><span class=\"token keyword\">SELECT<\/span> name <span class=\"token keyword\">FROM<\/span> sqlite_master <span class=\"token keyword\">WHERE<\/span> <span class=\"token keyword\">type<\/span><span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#039;table&#039;<\/span><span class=\"token punctuation\">;<\/span><\/p>\n<p>\u9884\u671f\u7ed3\u679c&#xff1a;<\/p>\n<p>raw_occurrences<br \/>\nresource_intensity<br \/>\nrequest_log<\/p>\n<h4>9.6 \u793a\u4f8b\u7ed3\u679c<\/h4>\n<p>resource_intensity.csv \u793a\u4f8b&#xff1a;<\/p>\n<p>species_name,sea_area,time_window,grid_id,grid_lon,grid_lat,intensity_index,source<br \/>\nGadus morhua,North Atlantic Demo Area,2020-01,lon_-63_lat_44,-63,44,12,OBIS API<br \/>\nGadus morhua,North Atlantic Demo Area,2020-01,lon_-62_lat_45,-62,45,7,OBIS API<br \/>\nGadus morhua,North Atlantic Demo Area,2020-02,lon_-64_lat_46,-64,46,15,OBIS API<br \/>\nGadus morhua,North Atlantic Demo Area,2020-03,lon_-61_lat_43,-61,43,9,OBIS API<br \/>\nGadus morhua,North Atlantic Demo Area,2020-04,lon_-58_lat_47,-58,47,4,OBIS API<\/p>\n<p>\u8fd9\u51e0\u884c\u8868\u8fbe\u7684\u610f\u601d\u662f&#xff1a;\u5728\u6307\u5b9a\u6d77\u57df\u5185&#xff0c;\u67d0\u7269\u79cd\u5728\u67d0\u4e2a\u6708\u4efd\u3001\u67d0\u4e2a\u7ecf\u7eac\u5ea6\u7f51\u683c\u4e2d\u6709\u591a\u5c11\u6761\u6709\u6548 occurrence \u8bb0\u5f55\u3002intensity_index \u8d8a\u9ad8&#xff0c;\u8bf4\u660e\u8be5\u7f51\u683c\u5728\u8be5\u6708\u7684\u8bb0\u5f55\u8d8a\u591a\u3002\u518d\u6b21\u5f3a\u8c03&#xff0c;\u5b83\u662f\u8bb0\u5f55\u5f3a\u5ea6&#xff0c;\u4e0d\u662f\u8d44\u6e90\u50a8\u91cf\u3002<\/p>\n<h4>9.7 \u67e5\u770b SQLite<\/h4>\n<p>\u53ef\u4ee5\u7528 Python \u5feb\u901f\u68c0\u67e5&#xff1a;<\/p>\n<p><span class=\"token keyword\">import<\/span> sqlite3<br \/>\n<span class=\"token keyword\">import<\/span> pandas <span class=\"token keyword\">as<\/span> pd<\/p>\n<p>conn <span class=\"token operator\">&#061;<\/span> sqlite3<span class=\"token punctuation\">.<\/span>connect<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;data\/processed\/fisheries_dynamic.sqlite3&#034;<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>df <span class=\"token operator\">&#061;<\/span> pd<span class=\"token punctuation\">.<\/span>read_sql_query<span class=\"token punctuation\">(<\/span><br \/>\n    <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n    SELECT *<br \/>\n    FROM resource_intensity<br \/>\n    ORDER BY time_window, intensity_index DESC<br \/>\n    LIMIT 10<br \/>\n    &#034;&#034;&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    conn<span class=\"token punctuation\">,<\/span><br \/>\n<span class=\"token punctuation\">)<\/span><\/p>\n<p><span class=\"token keyword\">print<\/span><span class=\"token punctuation\">(<\/span>df<span class=\"token punctuation\">)<\/span><br \/>\nconn<span class=\"token punctuation\">.<\/span>close<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>\u4e5f\u53ef\u4ee5\u7528 sqlite3 \u547d\u4ee4\u884c&#xff1a;<\/p>\n<p>sqlite3 data\/processed\/fisheries_dynamic.sqlite3<\/p>\n<p>\u8fdb\u5165\u540e\u6267\u884c&#xff1a;<\/p>\n<p><span class=\"token punctuation\">.<\/span>headers <span class=\"token keyword\">on<\/span><br \/>\n<span class=\"token punctuation\">.<\/span><span class=\"token keyword\">mode<\/span> <span class=\"token keyword\">column<\/span><\/p>\n<p><span class=\"token keyword\">SELECT<\/span> species_name<span class=\"token punctuation\">,<\/span> time_window<span class=\"token punctuation\">,<\/span> grid_id<span class=\"token punctuation\">,<\/span> intensity_index<br \/>\n<span class=\"token keyword\">FROM<\/span> resource_intensity<br \/>\n<span class=\"token keyword\">ORDER<\/span> <span class=\"token keyword\">BY<\/span> intensity_index <span class=\"token keyword\">DESC<\/span><br \/>\n<span class=\"token keyword\">LIMIT<\/span> <span class=\"token number\">10<\/span><span class=\"token punctuation\">;<\/span><\/p>\n<hr \/>\n<h3>&#x1f51f; \u5e38\u89c1\u95ee\u9898\u4e0e\u6392\u9519&#xff08;\u5f3a\u70c8\u5efa\u8bae\u5199&#xff09;<\/h3>\n<h4>10.1 403 \u600e\u4e48\u529e<\/h4>\n<p>403 \u901a\u5e38\u8868\u793a\u670d\u52a1\u5668\u62d2\u7edd\u8bbf\u95ee\u3002\u53ef\u80fd\u539f\u56e0\u5305\u62ec&#xff1a;<\/p>\n<ul>\n<li>\u8bf7\u6c42\u8def\u5f84\u4e0d\u5141\u8bb8&#xff1b;<\/li>\n<li>\u7f3a\u5c11\u5fc5\u8981 headers&#xff1b;<\/li>\n<li>\u63a5\u53e3\u4e0d\u5bf9\u5916\u5f00\u653e&#xff1b;<\/li>\n<li>\u8bbf\u95ee\u65b9\u5f0f\u4e0d\u7b26\u5408\u5e73\u53f0\u89c4\u5219&#xff1b;<\/li>\n<li>IP \u88ab\u4e34\u65f6\u9650\u5236\u3002<\/li>\n<\/ul>\n<p>\u5904\u7406\u601d\u8def&#xff1a;<\/p>\n<li>\u5148\u786e\u8ba4\u63a5\u53e3\u662f\u5426\u662f\u5b98\u65b9\u516c\u5f00\u63a5\u53e3&#xff1b;<\/li>\n<li>\u67e5\u770b API \u6587\u6863&#xff0c;\u800c\u4e0d\u662f\u76f2\u76ee\u590d\u5236\u524d\u7aef\u8bf7\u6c42&#xff1b;<\/li>\n<li>\u8bbe\u7f6e\u6b63\u5e38 User-Agent&#xff1b;<\/li>\n<li>\u964d\u4f4e\u8bf7\u6c42\u9891\u7387&#xff1b;<\/li>\n<li>\u4e0d\u8981\u5c1d\u8bd5\u7ed5\u8fc7\u767b\u5f55\u6216\u6743\u9650\u9650\u5236&#xff1b;<\/li>\n<li>\u5982\u679c\u63a5\u53e3\u660e\u786e\u4e0d\u5f00\u653e&#xff0c;\u505c\u6b62\u91c7\u96c6\u3002<\/li>\n<p>\u4e0d\u8981\u628a 403 \u7b80\u5355\u7406\u89e3\u4e3a\u201c\u7f3a\u4ee3\u7406\u201d\u3002\u5f88\u591a 403 \u662f\u89c4\u5219\u95ee\u9898&#xff0c;\u4e0d\u662f\u7f51\u7edc\u95ee\u9898\u3002<\/p>\n<h4>10.2 429 \u600e\u4e48\u529e<\/h4>\n<p>429 \u8868\u793a\u8bf7\u6c42\u8fc7\u5feb\u3002\u6b63\u786e\u5904\u7406\u65b9\u5f0f\u662f\u964d\u901f\u3002<\/p>\n<p>\u672c\u6587\u4ee3\u7801\u91cc\u5df2\u7ecf\u5bf9 429 \u505a\u4e86\u9000\u907f\u7b49\u5f85\u3002\u8fd8\u53ef\u4ee5\u7ee7\u7eed\u4f18\u5316&#xff1a;<\/p>\n<ul>\n<li>\u589e\u5927 REQUEST_SLEEP_SECONDS&#xff1b;<\/li>\n<li>\u964d\u4f4e PAGE_SIZE&#xff1b;<\/li>\n<li>\u51cf\u5c11\u7269\u79cd\u6570\u91cf&#xff1b;<\/li>\n<li>\u51cf\u5c11\u65f6\u95f4\u8303\u56f4&#xff1b;<\/li>\n<li>\u907f\u5f00\u9ad8\u5cf0\u65f6\u6bb5&#xff1b;<\/li>\n<li>\u4f7f\u7528\u5b98\u65b9\u63a8\u8350\u7684\u5927\u6570\u636e\u5bfc\u51fa\u65b9\u5f0f\u3002<\/li>\n<\/ul>\n<p>\u4e0d\u5efa\u8bae\u9047\u5230 429 \u5c31\u6362\u4ee3\u7406\u6c60\u3002\u516c\u5171\u6570\u636e\u5e73\u53f0\u4e0d\u662f\u9776\u573a&#xff0c;\u91c7\u96c6\u8981\u6709\u8fb9\u754c\u3002<\/p>\n<h4>10.3 HTML \u6293\u5230\u7a7a\u58f3\u600e\u4e48\u529e<\/h4>\n<p>\u52a8\u6001\u9875\u9762\u76f4\u63a5 requests \u6293 HTML&#xff0c;\u7ecf\u5e38\u53ea\u80fd\u770b\u5230&#xff1a;<\/p>\n<p><span class=\"token tag\"><span class=\"token tag\"><span class=\"token punctuation\">&lt;<\/span>div<\/span> <span class=\"token attr-name\">id<\/span><span class=\"token attr-value\"><span class=\"token punctuation attr-equals\">&#061;<\/span><span class=\"token punctuation\">&#034;<\/span>app<span class=\"token punctuation\">&#034;<\/span><\/span><span class=\"token punctuation\">&gt;<\/span><\/span><span class=\"token tag\"><span class=\"token tag\"><span class=\"token punctuation\">&lt;\/<\/span>div<\/span><span class=\"token punctuation\">&gt;<\/span><\/span><br \/>\n<span class=\"token tag\"><span class=\"token tag\"><span class=\"token punctuation\">&lt;<\/span>script<\/span> <span class=\"token attr-name\">src<\/span><span class=\"token attr-value\"><span class=\"token punctuation attr-equals\">&#061;<\/span><span class=\"token punctuation\">&#034;<\/span>\/assets\/index.js<span class=\"token punctuation\">&#034;<\/span><\/span><span class=\"token punctuation\">&gt;<\/span><\/span><span class=\"token script\"><\/span><span class=\"token tag\"><span class=\"token tag\"><span class=\"token punctuation\">&lt;\/<\/span>script<\/span><span class=\"token punctuation\">&gt;<\/span><\/span><\/p>\n<p>\u8fd9\u8bf4\u660e\u6570\u636e\u4e0d\u662f\u5728 HTML \u91cc&#xff0c;\u800c\u662f\u7531\u524d\u7aef JavaScript \u8bf7\u6c42\u63a5\u53e3\u540e\u6e32\u67d3\u3002<\/p>\n<p>\u89e3\u51b3\u65b9\u6cd5&#xff1a;<\/p>\n<li>\u6253\u5f00\u6d4f\u89c8\u5668\u5f00\u53d1\u8005\u5de5\u5177&#xff1b;<\/li>\n<li>\u8fdb\u5165 Network&#xff1b;<\/li>\n<li>\u7b5b\u9009 XHR \/ Fetch&#xff1b;<\/li>\n<li>\u64cd\u4f5c\u9875\u9762\u7b5b\u9009\u6761\u4ef6&#xff1b;<\/li>\n<li>\u627e\u5230 JSON\u3001GeoJSON\u3001MVT \u6216 tile \u8bf7\u6c42&#xff1b;<\/li>\n<li>\u590d\u5236\u8bf7\u6c42\u53c2\u6570&#xff1b;<\/li>\n<li>\u7528 Python \u91cd\u653e\u63a5\u53e3\u3002<\/li>\n<p>\u5982\u679c\u63a5\u53e3\u53c2\u6570\u7531\u524d\u7aef\u590d\u6742\u751f\u6210&#xff0c;\u518d\u8003\u8651 Playwright\u3002\u4f46\u53ea\u8981\u80fd\u5b9a\u4f4d\u5230\u6e05\u6670 API&#xff0c;\u5c31\u4e0d\u8981\u7528\u6d4f\u89c8\u5668\u81ea\u52a8\u5316\u786c\u8dd1\u3002<\/p>\n<h4>10.4 \u89e3\u6790\u62a5\u9519\u600e\u4e48\u529e<\/h4>\n<p>\u5e38\u89c1\u89e3\u6790\u9519\u8bef\u5305\u62ec&#xff1a;<\/p>\n<ul>\n<li>\u5b57\u6bb5\u540d\u53d8\u5316&#xff1b;<\/li>\n<li>\u8fd4\u56de\u7ed3\u6784\u53d8\u5316&#xff1b;<\/li>\n<li>\u67d0\u4e9b\u8bb0\u5f55\u7f3a\u5c11\u7ecf\u7eac\u5ea6&#xff1b;<\/li>\n<li>\u67d0\u4e9b\u8bb0\u5f55 eventDate \u4e3a\u7a7a&#xff1b;<\/li>\n<li>\u54cd\u5e94\u4e0d\u662f JSON&#xff1b;<\/li>\n<li>\u63a5\u53e3\u8fd4\u56de\u9519\u8bef\u4fe1\u606f\u4f46\u72b6\u6001\u7801\u4ecd\u662f 200\u3002<\/li>\n<\/ul>\n<p>\u5904\u7406\u65b9\u5f0f&#xff1a;<\/p>\n<ul>\n<li>\u89e3\u6790\u51fd\u6570\u8981\u7528 dict.get()&#xff1b;<\/li>\n<li>\u7528 pick_first() \u652f\u6301\u591a\u4e2a\u5019\u9009\u5b57\u6bb5&#xff1b;<\/li>\n<li>\u7f3a\u5c11\u5173\u952e\u5b57\u6bb5\u5c31\u8df3\u8fc7&#xff1b;<\/li>\n<li>\u4fdd\u5b58\u90e8\u5206\u539f\u59cb\u54cd\u5e94\u7528\u4e8e\u6392\u67e5&#xff1b;<\/li>\n<li>\u8bb0\u5f55\u5931\u8d25\u6837\u672c&#xff1b;<\/li>\n<li>\u5355\u72ec\u5199\u5c0f\u6d4b\u8bd5\u9a8c\u8bc1\u89e3\u6790\u5668\u3002<\/li>\n<\/ul>\n<p>\u4e0d\u8981\u628a\u89e3\u6790\u903b\u8f91\u5199\u6210\u4e00\u4e32\u786c\u7d22\u5f15&#xff0c;\u6bd4\u5982&#xff1a;<\/p>\n<p>record<span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;data&#034;<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;items&#034;<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">[<\/span><span class=\"token number\">0<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;location&#034;<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;lat&#034;<\/span><span class=\"token punctuation\">]<\/span><\/p>\n<p>\u8fd9\u79cd\u5199\u6cd5\u5728\u771f\u5b9e\u6570\u636e\u91cc\u5f88\u8106\u3002\u66f4\u7a33\u59a5\u7684\u65b9\u5f0f\u662f\u9010\u5c42\u5224\u65ad\u3002<\/p>\n<h4>10.5 \u7f16\u7801\/\u4e71\u7801\u5982\u4f55\u5904\u7406<\/h4>\n<p>JSON \u63a5\u53e3\u4e00\u822c\u95ee\u9898\u4e0d\u5927\u3002\u5982\u679c\u5bfc\u51fa CSV \u7ed9 Excel \u6253\u5f00&#xff0c;\u4e2d\u6587\u5b57\u6bb5\u53ef\u80fd\u51fa\u73b0\u4e71\u7801\u3002\u672c\u6587\u4f7f\u7528&#xff1a;<\/p>\n<p>encoding<span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#034;utf-8-sig&#034;<\/span><\/p>\n<p>\u8fd9\u6837 Excel \u517c\u5bb9\u6027\u66f4\u597d\u3002<\/p>\n<p>\u5982\u679c\u7f51\u9875\u91c7\u96c6\u9047\u5230\u4e71\u7801&#xff0c;\u53ef\u4ee5\u68c0\u67e5&#xff1a;<\/p>\n<p>resp<span class=\"token punctuation\">.<\/span>encoding<br \/>\nresp<span class=\"token punctuation\">.<\/span>apparent_encoding<\/p>\n<p>\u5fc5\u8981\u65f6\u624b\u52a8\u8bbe\u7f6e&#xff1a;<\/p>\n<p>resp<span class=\"token punctuation\">.<\/span>encoding <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;utf-8&#034;<\/span><\/p>\n<p>\u4f46\u4e0d\u8981\u76f2\u76ee\u8bbe\u7f6e\u3002\u5148\u770b\u54cd\u5e94\u5934\u548c\u9875\u9762\u5b9e\u9645\u7f16\u7801\u3002<\/p>\n<h4>10.6 \u6570\u636e\u4e3a\u7a7a\u600e\u4e48\u529e<\/h4>\n<p>\u6570\u636e\u4e3a\u7a7a\u4e0d\u4e00\u5b9a\u662f\u7a0b\u5e8f\u9519\u3002\u53ef\u80fd\u539f\u56e0\u5f88\u591a&#xff1a;<\/p>\n<ul>\n<li>\u7269\u79cd\u540d\u4e0d\u5b58\u5728\u6216\u62fc\u5199\u4e0d\u4e00\u81f4&#xff1b;<\/li>\n<li>\u65f6\u95f4\u8303\u56f4\u5185\u6ca1\u6709\u8bb0\u5f55&#xff1b;<\/li>\n<li>\u6d77\u57df WKT \u8303\u56f4\u592a\u5c0f&#xff1b;<\/li>\n<li>\u7ecf\u7eac\u5ea6\u987a\u5e8f\u5199\u53cd&#xff1b;<\/li>\n<li>\u53c2\u6570\u540d\u5199\u9519&#xff1b;<\/li>\n<li>\u63a5\u53e3\u5206\u9875\u903b\u8f91\u4e0d\u5bf9&#xff1b;<\/li>\n<li>\u5b57\u6bb5\u8fc7\u6ee4\u8fc7\u7a84\u3002<\/li>\n<\/ul>\n<p>\u6392\u67e5\u987a\u5e8f\u5efa\u8bae&#xff1a;<\/p>\n<li>\u653e\u5927\u65f6\u95f4\u8303\u56f4&#xff1b;<\/li>\n<li>\u653e\u5927\u7a7a\u95f4\u8303\u56f4&#xff1b;<\/li>\n<li>\u5148\u4e0d\u4f20 fields&#xff1b;<\/li>\n<li>page size \u8bbe\u7f6e\u5c0f\u4e00\u70b9&#xff1b;<\/li>\n<li>\u5728\u6d4f\u89c8\u5668\u4e2d\u76f4\u63a5\u6253\u5f00\u8bf7\u6c42 URL&#xff1b;<\/li>\n<li>\u6362\u4e00\u4e2a\u66f4\u5e38\u89c1\u7684\u7269\u79cd\u6d4b\u8bd5&#xff1b;<\/li>\n<li>\u6253\u5370\u5b8c\u6574\u54cd\u5e94\u7ed3\u6784\u3002<\/li>\n<p>WKT \u6700\u5bb9\u6613\u8e29\u5751\u7684\u662f\u5750\u6807\u987a\u5e8f\u3002WKT \u901a\u5e38\u5199 lon lat&#xff0c;\u4e0d\u662f lat lon\u3002\u4f8b\u5982&#xff1a;<\/p>\n<p>POLYGON((-75 35,-75 55,-45 55,-45 35,-75 35))<\/p>\n<p>\u8fd9\u91cc -75 \u662f\u7ecf\u5ea6&#xff0c;35 \u662f\u7eac\u5ea6\u3002<\/p>\n<h4>10.7 \u65f6\u95f4\u8f74\u4e0d\u52a8\u600e\u4e48\u529e<\/h4>\n<p>Folium \u751f\u6210\u65f6\u95f4\u8f74\u5730\u56fe\u65f6&#xff0c;\u9700\u8981 GeoJSON feature \u91cc\u6709\u53ef\u8bc6\u522b\u7684\u65f6\u95f4\u5c5e\u6027\u3002\u672c\u6587\u5199\u5165\u7684\u662f&#xff1a;<\/p>\n<p><span class=\"token string-property property\">&#034;properties&#034;<\/span><span class=\"token operator\">:<\/span> <span class=\"token punctuation\">{<\/span><br \/>\n  <span class=\"token string-property property\">&#034;time&#034;<\/span><span class=\"token operator\">:<\/span> <span class=\"token string\">&#034;2020-03-01&#034;<\/span><br \/>\n<span class=\"token punctuation\">}<\/span><\/p>\n<p>\u5982\u679c\u65f6\u95f4\u8f74\u4e0d\u52a8&#xff0c;\u68c0\u67e5&#xff1a;<\/p>\n<ul>\n<li>\u662f\u5426\u6709 time \u5b57\u6bb5&#xff1b;<\/li>\n<li>\u65f6\u95f4\u683c\u5f0f\u662f\u5426\u7c7b\u4f3c YYYY-MM-DD&#xff1b;<\/li>\n<li>GeoJSON \u662f\u5426\u4e3a\u7a7a&#xff1b;<\/li>\n<li>feature geometry \u662f\u5426\u6709\u6548&#xff1b;<\/li>\n<li>\u6d4f\u89c8\u5668\u63a7\u5236\u53f0\u662f\u5426\u6709 JS \u62a5\u9519\u3002<\/li>\n<\/ul>\n<p>\u5982\u679c\u6570\u636e\u91cf\u5f88\u5927&#xff0c;HTML \u4f1a\u53d8\u5f97\u5f88\u91cd\u3002\u6b64\u65f6\u5e94\u8be5\u8003\u8651\u6309\u65f6\u95f4\u5207\u7247\u52a0\u8f7d&#xff0c;\u6216\u8005\u7528\u4e13\u95e8\u7684 WebGIS \u670d\u52a1\u53d1\u5e03\u56fe\u5c42\u3002<\/p>\n<hr \/>\n<h3>1\ufe0f\u20e31\ufe0f\u20e3 \u8fdb\u9636\u4f18\u5316&#xff08;\u53ef\u9009\u4f46\u52a0\u5206&#xff09;<\/h3>\n<h4>11.1 \u5e76\u53d1\u4f18\u5316<\/h4>\n<p>\u672c\u6587\u6ca1\u6709\u4f7f\u7528\u5e76\u53d1&#xff0c;\u662f\u6545\u610f\u7684\u3002\u5bf9\u516c\u5f00 API&#xff0c;\u5148\u4fdd\u8bc1\u6e29\u548c\u548c\u7a33\u5b9a&#xff0c;\u518d\u8c08\u901f\u5ea6\u3002<\/p>\n<p>\u5982\u679c\u540e\u7eed\u786e\u5b9e\u9700\u8981\u5e76\u53d1&#xff0c;\u53ef\u4ee5\u8003\u8651\u4e09\u79cd\u65b9\u6848&#xff1a;<\/p>\n<p>\u7b2c\u4e00\u79cd&#xff0c;\u7ebf\u7a0b\u6c60&#xff1a;<\/p>\n<p><span class=\"token keyword\">from<\/span> concurrent<span class=\"token punctuation\">.<\/span>futures <span class=\"token keyword\">import<\/span> ThreadPoolExecutor<span class=\"token punctuation\">,<\/span> as_completed<\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">collect_one_window<\/span><span class=\"token punctuation\">(<\/span>args<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    species_name<span class=\"token punctuation\">,<\/span> window <span class=\"token operator\">&#061;<\/span> args<br \/>\n    <span class=\"token comment\"># call fetcher and parser here<\/span><br \/>\n    <span class=\"token keyword\">return<\/span> species_name<span class=\"token punctuation\">,<\/span> window<span class=\"token punctuation\">.<\/span>month<span class=\"token punctuation\">,<\/span> <span class=\"token number\">0<\/span><\/p>\n<p><span class=\"token keyword\">with<\/span> ThreadPoolExecutor<span class=\"token punctuation\">(<\/span>max_workers<span class=\"token operator\">&#061;<\/span><span class=\"token number\">2<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token keyword\">as<\/span> executor<span class=\"token punctuation\">:<\/span><br \/>\n    futures <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">[<\/span><br \/>\n        executor<span class=\"token punctuation\">.<\/span>submit<span class=\"token punctuation\">(<\/span>collect_one_window<span class=\"token punctuation\">,<\/span> item<span class=\"token punctuation\">)<\/span><br \/>\n        <span class=\"token keyword\">for<\/span> item <span class=\"token keyword\">in<\/span> tasks<br \/>\n    <span class=\"token punctuation\">]<\/span><\/p>\n<p>    <span class=\"token keyword\">for<\/span> future <span class=\"token keyword\">in<\/span> as_completed<span class=\"token punctuation\">(<\/span>futures<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token keyword\">print<\/span><span class=\"token punctuation\">(<\/span>future<span class=\"token punctuation\">.<\/span>result<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>\u6ce8\u610f&#xff0c;max_workers \u4e0d\u8981\u592a\u5927\u3002\u5bf9\u4e8e\u516c\u5171 API&#xff0c;2 \u5230 4 \u5df2\u7ecf\u4e0d\u5c11\u4e86\u3002<\/p>\n<p>\u7b2c\u4e8c\u79cd&#xff0c;asyncio &#043; aiohttp\u3002\u9002\u5408\u5927\u91cf\u5c0f\u8bf7\u6c42&#xff0c;\u4f46\u590d\u6742\u5ea6\u66f4\u9ad8\u3002<\/p>\n<p>\u7b2c\u4e09\u79cd&#xff0c;Scrapy\u3002\u9002\u5408\u4efb\u52a1\u961f\u5217\u3001\u53bb\u91cd\u3001\u91cd\u8bd5\u3001\u65e5\u5fd7\u3001\u9650\u901f\u3001\u6269\u5c55\u4e2d\u95f4\u4ef6\u90fd\u6bd4\u8f83\u590d\u6742\u7684\u9879\u76ee\u3002<\/p>\n<h4>11.2 \u65ad\u70b9\u7eed\u8dd1<\/h4>\n<p>\u5f53\u524d\u4ee3\u7801\u6bcf\u6b21\u8fd0\u884c\u4f1a\u7ee7\u7eed\u63d2\u5165\u539f\u59cb\u8868&#xff0c;\u56e0\u4e3a occurrence_id \u662f\u4e3b\u952e&#xff0c;\u91cd\u590d\u8bb0\u5f55\u4f1a\u88ab\u5ffd\u7565\u3002\u4f46\u8bf7\u6c42\u672c\u8eab\u8fd8\u662f\u4f1a\u91cd\u53d1\u3002<\/p>\n<p>\u66f4\u597d\u7684\u65ad\u70b9\u7eed\u8dd1\u65b9\u5f0f\u662f&#xff1a;<\/p>\n<ul>\n<li>request_log \u8bb0\u5f55\u6bcf\u4e2a species &#043; month &#043; sea_area \u662f\u5426\u5b8c\u6210&#xff1b;<\/li>\n<li>\u542f\u52a8\u65f6\u5148\u67e5 request_log&#xff1b;<\/li>\n<li>\u5df2\u5b8c\u6210\u7684\u7a97\u53e3\u8df3\u8fc7&#xff1b;<\/li>\n<li>\u672a\u5b8c\u6210\u7684\u7a97\u53e3\u7ee7\u7eed\u6293\u3002<\/li>\n<\/ul>\n<p>\u793a\u4f8b\u51fd\u6570&#xff1a;<\/p>\n<p><span class=\"token keyword\">def<\/span> <span class=\"token function\">is_window_done<\/span><span class=\"token punctuation\">(<\/span>conn<span class=\"token punctuation\">,<\/span> species_name<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> sea_area<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">,<\/span> time_window<span class=\"token punctuation\">:<\/span> <span class=\"token builtin\">str<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">&#8211;<\/span><span class=\"token operator\">&gt;<\/span> <span class=\"token builtin\">bool<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    cur <span class=\"token operator\">&#061;<\/span> conn<span class=\"token punctuation\">.<\/span>execute<span class=\"token punctuation\">(<\/span><br \/>\n        <span class=\"token triple-quoted-string string\">&#034;&#034;&#034;<br \/>\n        SELECT COUNT(*) AS cnt<br \/>\n        FROM request_log<br \/>\n        WHERE species_name &#061; ?<br \/>\n          AND sea_area &#061; ?<br \/>\n          AND time_window &#061; ?<br \/>\n        &#034;&#034;&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">(<\/span>species_name<span class=\"token punctuation\">,<\/span> sea_area<span class=\"token punctuation\">,<\/span> time_window<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">)<\/span><br \/>\n    row <span class=\"token operator\">&#061;<\/span> cur<span class=\"token punctuation\">.<\/span>fetchone<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><br \/>\n    <span class=\"token keyword\">return<\/span> row<span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;cnt&#034;<\/span><span class=\"token punctuation\">]<\/span> <span class=\"token operator\">&gt;<\/span> <span class=\"token number\">0<\/span><\/p>\n<p>\u4f46\u8981\u6ce8\u610f&#xff1a;\u5982\u679c\u4e0a\u6b21\u91c7\u96c6\u5931\u8d25\u5374\u5199\u4e86\u65e5\u5fd7&#xff0c;\u5c31\u4f1a\u8bef\u5224\u5b8c\u6210\u3002\u66f4\u4e25\u8c28\u7684\u505a\u6cd5\u662f\u7ed9 request_log \u52a0 status \u5b57\u6bb5&#xff1a;<\/p>\n<p>running \/ success \/ failed<\/p>\n<p>\u5f00\u59cb\u91c7\u96c6\u524d\u5199 running&#xff0c;\u6210\u529f\u540e\u6539 success&#xff0c;\u5931\u8d25\u540e\u6539 failed\u3002<\/p>\n<h4>11.3 \u65e5\u5fd7\u4e0e\u76d1\u63a7<\/h4>\n<p>\u6700\u4f4e\u9650\u5ea6\u7684\u65e5\u5fd7\u5e94\u8be5\u5305\u62ec&#xff1a;<\/p>\n<ul>\n<li>\u8bf7\u6c42 URL&#xff1b;<\/li>\n<li>\u7269\u79cd&#xff1b;<\/li>\n<li>\u65f6\u95f4\u7a97\u53e3&#xff1b;<\/li>\n<li>offset&#xff1b;<\/li>\n<li>\u8fd4\u56de\u6570\u91cf&#xff1b;<\/li>\n<li>\u4fdd\u5b58\u6570\u91cf&#xff1b;<\/li>\n<li>\u5931\u8d25\u539f\u56e0&#xff1b;<\/li>\n<li>\u603b\u8017\u65f6\u3002<\/li>\n<\/ul>\n<p>\u53ef\u4ee5\u628a\u65e5\u5fd7\u5199\u5165\u6587\u4ef6&#xff1a;<\/p>\n<p><span class=\"token keyword\">from<\/span> pathlib <span class=\"token keyword\">import<\/span> Path<br \/>\n<span class=\"token keyword\">import<\/span> logging<\/p>\n<p>log_path <span class=\"token operator\">&#061;<\/span> Path<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;logs\/spider.log&#034;<\/span><span class=\"token punctuation\">)<\/span><br \/>\nlog_path<span class=\"token punctuation\">.<\/span>parent<span class=\"token punctuation\">.<\/span>mkdir<span class=\"token punctuation\">(<\/span>exist_ok<span class=\"token operator\">&#061;<\/span><span class=\"token boolean\">True<\/span><span class=\"token punctuation\">)<\/span><\/p>\n<p>logging<span class=\"token punctuation\">.<\/span>basicConfig<span class=\"token punctuation\">(<\/span><br \/>\n    level<span class=\"token operator\">&#061;<\/span>logging<span class=\"token punctuation\">.<\/span>INFO<span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token builtin\">format<\/span><span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#034;%(asctime)s | %(levelname)s | %(name)s | %(message)s&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    handlers<span class=\"token operator\">&#061;<\/span><span class=\"token punctuation\">[<\/span><br \/>\n        logging<span class=\"token punctuation\">.<\/span>FileHandler<span class=\"token punctuation\">(<\/span>log_path<span class=\"token punctuation\">,<\/span> encoding<span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#034;utf-8&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        logging<span class=\"token punctuation\">.<\/span>StreamHandler<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span><br \/>\n<span class=\"token punctuation\">)<\/span><\/p>\n<p>\u5bf9\u4e8e\u957f\u671f\u4efb\u52a1&#xff0c;\u53ef\u4ee5\u7edf\u8ba1&#xff1a;<\/p>\n<ul>\n<li>\u6210\u529f\u7a97\u53e3\u6570&#xff1b;<\/li>\n<li>\u5931\u8d25\u7a97\u53e3\u6570&#xff1b;<\/li>\n<li>\u7a7a\u7ed3\u679c\u7a97\u53e3\u6570&#xff1b;<\/li>\n<li>\u5e73\u5747\u6bcf\u9875\u8017\u65f6&#xff1b;<\/li>\n<li>\u5e73\u5747\u6bcf\u6708\u8bb0\u5f55\u6570&#xff1b;<\/li>\n<li>429 \u6b21\u6570&#xff1b;<\/li>\n<li>5xx \u6b21\u6570\u3002<\/li>\n<\/ul>\n<p>\u8fd9\u4e9b\u6307\u6807\u6bd4\u201c\u7a0b\u5e8f\u80fd\u8dd1\u201d\u66f4\u6709\u4ef7\u503c\u3002\u91c7\u96c6\u4efb\u52a1\u4e00\u65e6\u8dd1\u4e45\u4e86&#xff0c;\u65e5\u5fd7\u5c31\u662f\u4f60\u7684\u773c\u775b\u3002<\/p>\n<h4>11.4 \u5b9a\u65f6\u4efb\u52a1<\/h4>\n<p>\u5982\u679c\u8981\u6bcf\u6708\u81ea\u52a8\u66f4\u65b0&#xff0c;\u53ef\u4ee5\u4f7f\u7528 cron\u3002<\/p>\n<p>Linux crontab \u793a\u4f8b&#xff1a;<\/p>\n<p><span class=\"token number\">0<\/span> <span class=\"token number\">3<\/span> <span class=\"token number\">1<\/span> * * <span class=\"token builtin class-name\">cd<\/span> \/opt\/fisheries_dynamic_obis <span class=\"token operator\">&amp;&amp;<\/span> \/opt\/fisheries_dynamic_obis\/.venv\/bin\/python run_spider.py <span class=\"token operator\">&gt;&gt;<\/span> logs\/cron.log <span class=\"token operator\"><span class=\"token file-descriptor important\">2<\/span>&gt;<\/span><span class=\"token file-descriptor important\">&amp;1<\/span><\/p>\n<p>\u542b\u4e49\u662f\u6bcf\u6708 1 \u65e5\u51cc\u6668 3 \u70b9\u6267\u884c\u4e00\u6b21\u3002<\/p>\n<p>\u5982\u679c\u4efb\u52a1\u590d\u6742&#xff0c;\u53ef\u4ee5\u4f7f\u7528 Airflow \u6216 Prefect\u3002\u5b83\u4eec\u53ef\u4ee5\u7ba1\u7406\u4f9d\u8d56\u3001\u91cd\u8bd5\u3001\u544a\u8b66\u548c\u4efb\u52a1\u72b6\u6001\u3002\u4f46\u5c0f\u9879\u76ee\u6ca1\u5fc5\u8981\u4e00\u5f00\u59cb\u5c31\u4e0a Airflow\u3002\u5148\u7528\u811a\u672c\u628a\u6d41\u7a0b\u8dd1\u901a&#xff0c;\u518d\u51b3\u5b9a\u662f\u5426\u5e73\u53f0\u5316\u3002<\/p>\n<h4>11.5 \u7a7a\u95f4\u7f51\u683c\u5347\u7ea7<\/h4>\n<p>\u672c\u6587\u7528 1 \u5ea6\u7ecf\u7eac\u5ea6\u7f51\u683c\u662f\u4e3a\u4e86\u7b80\u5355\u53ef\u89e3\u91ca\u3002\u771f\u5b9e\u9879\u76ee\u53ef\u4ee5\u5347\u7ea7\u4e3a&#xff1a;<\/p>\n<ul>\n<li>Geohash&#xff1b;<\/li>\n<li>H3&#xff1b;<\/li>\n<li>S2&#xff1b;<\/li>\n<li>EEZ \u533a\u5212&#xff1b;<\/li>\n<li>FAO fishing area&#xff1b;<\/li>\n<li>\u81ea\u5b9a\u4e49\u6e14\u573a\u8fb9\u754c&#xff1b;<\/li>\n<li>PostGIS \u7a7a\u95f4 join\u3002<\/li>\n<\/ul>\n<p>\u6bd4\u5982\u4f7f\u7528 H3&#xff0c;\u53ef\u4ee5\u628a\u7ecf\u7eac\u5ea6\u7f16\u7801\u6210\u516d\u8fb9\u5f62\u7f51\u683c&#xff0c;\u66f4\u9002\u5408\u505a\u7a7a\u95f4\u805a\u5408\u3002\u4f46 H3 \u9700\u8981\u989d\u5916\u4f9d\u8d56&#xff0c;\u672c\u6587\u4e3a\u4e86\u4fdd\u6301\u4ee3\u7801\u8f7b\u91cf&#xff0c;\u6ca1\u6709\u52a0\u5165\u3002<\/p>\n<h4>11.6 \u5f3a\u5ea6\u6307\u6807\u5347\u7ea7<\/h4>\n<p>\u5f53\u524d\u5f3a\u5ea6\u6307\u6807\u662f\u8bb0\u5f55\u6570\u91cf\u3002\u540e\u7eed\u53ef\u4ee5\u6269\u5c55\u4e3a&#xff1a;<\/p>\n<ul>\n<li>\u8bb0\u5f55\u6570\u91cf&#xff1b;<\/li>\n<li>\u53bb\u91cd\u540e\u7684\u91c7\u6837\u4e8b\u4ef6\u6570\u91cf&#xff1b;<\/li>\n<li>\u5355\u4f4d\u9762\u79ef\u8bb0\u5f55\u6570\u91cf&#xff1b;<\/li>\n<li>\u6309\u6570\u636e\u96c6\u52a0\u6743\u540e\u7684\u8bb0\u5f55\u6570\u91cf&#xff1b;<\/li>\n<li>\u6309\u91c7\u6837\u52aa\u529b\u6821\u6b63\u540e\u7684\u76f8\u5bf9\u6307\u6570&#xff1b;<\/li>\n<li>\u4e0d\u540c\u6765\u6e90\u4e00\u81f4\u6027\u6307\u6807\u3002<\/li>\n<\/ul>\n<p>\u8fd9\u91cc\u8981\u7279\u522b\u8c28\u614e\u3002\u8d8a\u63a5\u8fd1\u751f\u6001\u89e3\u91ca&#xff0c;\u5c31\u8d8a\u9700\u8981\u9886\u57df\u77e5\u8bc6\u3002\u722c\u866b\u53ea\u80fd\u8d1f\u8d23\u628a\u6570\u636e\u53ef\u9760\u5730\u642c\u5230\u5206\u6790\u684c\u9762\u4e0a&#xff0c;\u4e0d\u80fd\u628a\u7b80\u5355 count \u5305\u88c5\u6210\u79d1\u5b66\u7ed3\u8bba\u3002<\/p>\n<h4>11.7 \u591a\u7269\u79cd\u6279\u91cf\u91c7\u96c6<\/h4>\n<p>\u628a species_list \u6269\u5c55\u5373\u53ef&#xff1a;<\/p>\n<p>species_list <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">[<\/span><br \/>\n    <span class=\"token string\">&#034;Gadus morhua&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token string\">&#034;Clupea harengus&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token string\">&#034;Thunnus thynnus&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n<span class=\"token punctuation\">]<\/span><\/p>\n<p>\u6279\u91cf\u91c7\u96c6\u65f6\u5efa\u8bae\u52a0\u4e0a&#xff1a;<\/p>\n<ul>\n<li>\u6bcf\u4e2a\u7269\u79cd\u5355\u72ec\u65e5\u5fd7&#xff1b;<\/li>\n<li>\u6bcf\u4e2a\u7269\u79cd\u5355\u72ec\u8f93\u51fa\u6587\u4ef6&#xff1b;<\/li>\n<li>\u5931\u8d25\u7269\u79cd\u6e05\u5355&#xff1b;<\/li>\n<li>\u53ef\u914d\u7f6e\u7684\u91c7\u96c6\u8303\u56f4&#xff1b;<\/li>\n<li>\u9650\u5236\u6700\u5927\u8bb0\u5f55\u6570\u3002<\/li>\n<\/ul>\n<p>\u4e0d\u8981\u4e00\u6b21\u585e\u51e0\u5341\u4e2a\u7269\u79cd\u3001\u5341\u51e0\u5e74\u65f6\u95f4\u3001\u5168\u7403\u8303\u56f4&#xff0c;\u7136\u540e\u76f4\u63a5\u5f00\u8dd1\u3002\u5148\u505a\u4e00\u4e2a\u7269\u79cd\u4e00\u4e2a\u6708\u7684\u5c0f\u6837\u672c&#xff0c;\u786e\u8ba4\u5b57\u6bb5\u548c\u6570\u636e\u8d28\u91cf&#xff0c;\u518d\u6269\u5927\u3002<\/p>\n<h4>11.8 \u4e0e Scrapy \u96c6\u6210<\/h4>\n<p>\u5982\u679c\u4f60\u66f4\u559c\u6b22 Scrapy&#xff0c;\u53ef\u4ee5\u628a\u8bf7\u6c42\u5c42\u6539\u6210 Scrapy Spider&#xff1a;<\/p>\n<p><span class=\"token keyword\">import<\/span> scrapy<\/p>\n<p><span class=\"token keyword\">class<\/span> <span class=\"token class-name\">ObisOccurrenceSpider<\/span><span class=\"token punctuation\">(<\/span>scrapy<span class=\"token punctuation\">.<\/span>Spider<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    name <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;obis_occurrence&#034;<\/span><\/p>\n<p>    custom_settings <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">{<\/span><br \/>\n        <span class=\"token string\">&#034;DOWNLOAD_DELAY&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">1.0<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token string\">&#034;CONCURRENT_REQUESTS&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">2<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token string\">&#034;RETRY_TIMES&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">2<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token string\">&#034;FEED_EXPORT_ENCODING&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;utf-8&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n    <span class=\"token punctuation\">}<\/span><\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">start_requests<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        url <span class=\"token operator\">&#061;<\/span> <span class=\"token string\">&#034;https:\/\/api.obis.org\/v3\/occurrence&#034;<\/span><br \/>\n        params <span class=\"token operator\">&#061;<\/span> <span class=\"token punctuation\">{<\/span><br \/>\n            <span class=\"token string\">&#034;scientificname&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;Gadus morhua&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token string\">&#034;startdate&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;2020-01-01&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token string\">&#034;enddate&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;2020-01-31&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token string\">&#034;geometry&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;POLYGON((-75 35,-75 55,-45 55,-45 35,-75 35))&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token string\">&#034;size&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">500<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token string\">&#034;offset&#034;<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">0<\/span><span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">}<\/span><br \/>\n        <span class=\"token keyword\">yield<\/span> scrapy<span class=\"token punctuation\">.<\/span>FormRequest<span class=\"token punctuation\">(<\/span><br \/>\n            url<span class=\"token operator\">&#061;<\/span>url<span class=\"token punctuation\">,<\/span><br \/>\n            method<span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#034;GET&#034;<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            formdata<span class=\"token operator\">&#061;<\/span>params<span class=\"token punctuation\">,<\/span><br \/>\n            callback<span class=\"token operator\">&#061;<\/span>self<span class=\"token punctuation\">.<\/span>parse<span class=\"token punctuation\">,<\/span><br \/>\n        <span class=\"token punctuation\">)<\/span><\/p>\n<p>    <span class=\"token keyword\">def<\/span> <span class=\"token function\">parse<\/span><span class=\"token punctuation\">(<\/span>self<span class=\"token punctuation\">,<\/span> response<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        payload <span class=\"token operator\">&#061;<\/span> response<span class=\"token punctuation\">.<\/span>json<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><br \/>\n        <span class=\"token keyword\">for<\/span> row <span class=\"token keyword\">in<\/span> payload<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;results&#034;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token keyword\">yield<\/span> <span class=\"token punctuation\">{<\/span><br \/>\n                <span class=\"token string\">&#034;id&#034;<\/span><span class=\"token punctuation\">:<\/span> row<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;id&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;scientificName&#034;<\/span><span class=\"token punctuation\">:<\/span> row<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;scientificName&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;eventDate&#034;<\/span><span class=\"token punctuation\">:<\/span> row<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;eventDate&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;decimalLongitude&#034;<\/span><span class=\"token punctuation\">:<\/span> row<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;decimalLongitude&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n                <span class=\"token string\">&#034;decimalLatitude&#034;<\/span><span class=\"token punctuation\">:<\/span> row<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span><span class=\"token string\">&#034;decimalLatitude&#034;<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span><br \/>\n            <span class=\"token punctuation\">}<\/span><\/p>\n<p>\u4e0d\u8fc7\u8fd9\u6bb5\u53ea\u662f\u793a\u610f\u3002\u5bf9\u4e8e GET \u53c2\u6570&#xff0c;Scrapy \u4e2d\u66f4\u5e38\u89c1\u7684\u5199\u6cd5\u662f\u81ea\u5df1\u62fc query string \u6216\u4f7f\u7528 urllib.parse.urlencode\u3002\u5982\u679c\u9879\u76ee\u5df2\u7ecf\u7528 requests \u8dd1\u5f97\u5f88\u597d&#xff0c;\u6ca1\u5fc5\u8981\u4e3a\u4e86\u201c\u66f4\u50cf\u722c\u866b\u201d\u800c\u6362 Scrapy\u3002<\/p>\n<hr \/>\n<h3>1\ufe0f\u20e32\ufe0f\u20e3 \u603b\u7ed3\u4e0e\u5ef6\u4f38\u9605\u8bfb<\/h3>\n<p>\u672c\u6587\u5b8c\u6210\u4e86\u4e00\u4ef6\u6bd4\u8f83\u5b8c\u6574\u7684\u4e8b\u60c5&#xff1a;\u628a\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe\u91c7\u96c6\u62c6\u6210\u4e86\u53ef\u843d\u5730\u7684\u5de5\u7a0b\u6d41\u7a0b\u3002<\/p>\n<p>\u6211\u4eec\u4ece\u9700\u6c42\u51fa\u53d1&#xff0c;\u660e\u786e\u4e86\u76ee\u6807\u5b57\u6bb5&#xff1a;<\/p>\n<ul>\n<li>\u7269\u79cd\/\u8d44\u6e90\u540d&#xff1b;<\/li>\n<li>\u6d77\u57df&#xff1b;<\/li>\n<li>\u65f6\u95f4&#xff1b;<\/li>\n<li>\u5f3a\u5ea6\u6307\u6807&#xff1b;<\/li>\n<li>\u6765\u6e90\u3002<\/li>\n<\/ul>\n<p>\u7136\u540e\u9009\u62e9\u516c\u5f00 API \u4f5c\u4e3a\u91c7\u96c6\u5165\u53e3&#xff0c;\u6ca1\u6709\u53bb\u6293\u622a\u56fe&#xff0c;\u4e5f\u6ca1\u6709\u53bb\u6a21\u62df\u590d\u6742\u9875\u9762\u64cd\u4f5c\u3002\u63a5\u7740&#xff0c;\u6211\u4eec\u628a\u65f6\u95f4\u8f74\u62c6\u6210\u6708\u5ea6\u7a97\u53e3&#xff0c;\u628a\u7a7a\u95f4\u8303\u56f4\u5199\u6210 WKT polygon&#xff0c;\u901a\u8fc7 requests Session \u7a33\u5b9a\u8bf7\u6c42 occurrence \u6570\u636e\u3002\u89e3\u6790\u5c42\u8d1f\u8d23\u5b57\u6bb5\u6620\u5c04\u548c\u5bb9\u9519&#xff0c;\u5b58\u50a8\u5c42\u7528 SQLite \u4fdd\u5b58\u539f\u59cb\u8bb0\u5f55\u548c\u805a\u5408\u8bb0\u5f55&#xff0c;\u5bfc\u51fa\u5c42\u751f\u6210 CSV\u3001GeoJSON&#xff0c;\u6700\u540e\u7528 Folium \u505a\u4e86\u4e00\u4e2a\u53ef\u62d6\u52a8\u65f6\u95f4\u8f74\u7684 HTML \u5730\u56fe\u3002<\/p>\n<p>\u8fd9\u5957\u6d41\u7a0b\u7684\u91cd\u70b9\u4e0d\u662f\u67d0\u4e00\u4e2a\u63a5\u53e3&#xff0c;\u800c\u662f\u65b9\u6cd5&#xff1a;<\/p>\n<p>\u52a8\u6001\u5730\u56fe\u9875\u9762<br \/>\n  \u2192 \u627e\u56fe\u5c42\u63a5\u53e3<br \/>\n  \u2192 \u62c6\u65f6\u95f4\u8f74<br \/>\n  \u2192 \u5206\u9875\u91c7\u96c6<br \/>\n  \u2192 \u89e3\u6790 JSON \/ GeoJSON \/ MVT<br \/>\n  \u2192 \u6e05\u6d17\u5750\u6807\u548c\u65f6\u95f4<br \/>\n  \u2192 \u7a7a\u95f4\u805a\u5408<br \/>\n  \u2192 \u6307\u6807\u5b9a\u4e49<br \/>\n  \u2192 \u6570\u636e\u5e93\u5b58\u50a8<br \/>\n  \u2192 \u53ef\u89c6\u5316\u5bfc\u51fa<\/p>\n<p>\u5728\u6211\u770b\u6765&#xff0c;\u722c\u866b\u771f\u6b63\u6709\u610f\u601d\u7684\u5730\u65b9\u4e0d\u5728\u201c\u628a\u6570\u636e\u6293\u4e0b\u6765\u201d\u8fd9\u4e00\u6b65&#xff0c;\u800c\u5728\u4e8e\u6293\u4e0b\u6765\u4e4b\u540e\u80fd\u4e0d\u80fd\u89e3\u91ca\u6e05\u695a&#xff1a;\u5b57\u6bb5\u662f\u4ec0\u4e48&#xff0c;\u6307\u6807\u600e\u4e48\u7b97&#xff0c;\u6570\u636e\u6709\u4ec0\u4e48\u5c40\u9650&#xff0c;\u7ed3\u679c\u80fd\u4e0d\u80fd\u590d\u73b0&#xff0c;\u51fa\u4e86\u95ee\u9898\u80fd\u4e0d\u80fd\u6392\u67e5\u3002\u4e00\u4e2a\u53ea\u4f1a\u8dd1\u7684\u811a\u672c&#xff0c;\u5f88\u5feb\u4f1a\u53d8\u6210\u9ed1\u76d2&#xff1b;\u4e00\u4e2a\u7ed3\u6784\u6e05\u695a\u7684\u91c7\u96c6\u9879\u76ee&#xff0c;\u624d\u6709\u7ee7\u7eed\u7ef4\u62a4\u548c\u6269\u5c55\u7684\u4ef7\u503c\u3002<\/p>\n<p>\u4e0b\u4e00\u6b65\u53ef\u4ee5\u7ee7\u7eed\u505a\u8fd9\u4e9b\u6269\u5c55&#xff1a;<\/p>\n<li>\u4f7f\u7528 Scrapy \u7ba1\u7406\u66f4\u590d\u6742\u7684\u4efb\u52a1\u961f\u5217&#xff1b;<\/li>\n<li>\u4f7f\u7528 Playwright \u5206\u6790\u9700\u8981\u524d\u7aef\u8fd0\u884c\u65f6\u751f\u6210\u53c2\u6570\u7684\u5730\u56fe\u9875\u9762&#xff1b;<\/li>\n<li>\u4f7f\u7528 PostGIS \u505a\u7a7a\u95f4 join \u548c\u7a7a\u95f4\u7d22\u5f15&#xff1b;<\/li>\n<li>\u4f7f\u7528 H3 \u6216 Geohash \u66ff\u4ee3\u7b80\u5355\u7ecf\u7eac\u5ea6\u7f51\u683c&#xff1b;<\/li>\n<li>\u4f7f\u7528 Airflow \u505a\u5b9a\u65f6\u66f4\u65b0&#xff1b;<\/li>\n<li>\u4f7f\u7528 DuckDB \/ GeoParquet \u5904\u7406\u66f4\u5927\u89c4\u6a21\u7684\u5f00\u653e\u6570\u636e&#xff1b;<\/li>\n<li>\u5c06\u7ed3\u679c\u63a5\u5165 WebGIS \u524d\u7aef&#xff0c;\u505a\u6210\u771f\u6b63\u7684\u8d44\u6e90\u5206\u5e03\u770b\u677f\u3002<\/li>\n<p>\u6700\u540e\u518d\u63d0\u9192\u4e00\u53e5&#xff1a;\u52a8\u6001\u5730\u56fe\u91c7\u96c6\u4e0d\u662f\u8d8a\u5feb\u8d8a\u597d&#xff0c;\u5c24\u5176\u662f\u516c\u5171\u79d1\u5b66\u6570\u636e\u5e73\u53f0\u3002\u6162\u4e00\u70b9\u3001\u7a33\u4e00\u70b9\u3001\u53ef\u590d\u73b0\u4e00\u70b9&#xff0c;\u5f80\u5f80\u6bd4\u77ed\u65f6\u95f4\u6293\u5f88\u591a\u6570\u636e\u66f4\u4e13\u4e1a\u3002\u722c\u866b\u5de5\u7a0b\u7684\u6210\u719f&#xff0c;\u4e0d\u662f\u4f53\u73b0\u5728\u5e76\u53d1\u6570\u5b57\u4e0a&#xff0c;\u800c\u662f\u4f53\u73b0\u5728\u8fb9\u754c\u611f\u3001\u6570\u636e\u8d28\u91cf\u548c\u957f\u671f\u7ef4\u62a4\u80fd\u529b\u4e0a\u3002<\/p>\n<h3>&#x1f31f; \u6587\u672b<\/h3>\n<p>\u597d\u5566&#xff5e;\u4ee5\u4e0a\u5c31\u662f\u672c\u671f\u7684\u5168\u90e8\u5185\u5bb9\u5566&#xff01;\u5982\u679c\u4f60\u5728\u5b9e\u8df5\u8fc7\u7a0b\u4e2d\u9047\u5230\u4efb\u4f55\u7591\u95ee&#xff0c;\u6b22\u8fce\u5728\u8bc4\u8bba\u533a\u7559\u8a00\u4ea4\u6d41&#xff0c;\u6211\u770b\u5230\u90fd\u4f1a\u5c3d\u91cf\u56de\u590d&#xff5e;\u54b1\u4eec\u4e0b\u671f\u89c1&#xff01;<\/p>\n<p>\u5c0f\u4f19\u4f34\u4eec\u5728\u6279\u9605\u7684\u8fc7\u7a0b\u4e2d&#xff0c;\u5982\u679c\u89c9\u5f97\u6587\u7ae0\u4e0d\u9519&#xff0c;\u6b22\u8fce\u70b9\u8d5e\u3001\u6536\u85cf\u3001\u5173\u6ce8\u54e6&#xff5e; \u4e09\u8fde\u5c31\u662f\u5bf9\u6211\u5199\u4f5c\u9053\u8def\u4e0a\u6700\u597d\u7684\u9f13\u52b1\u4e0e\u652f\u6301&#xff01; \u2764\ufe0f&#x1f525;<\/p>\n<h4>\u2705 \u4e13\u680f\u6301\u7eed\u66f4\u65b0\u4e2d&#xff5c;\u5efa\u8bae\u6536\u85cf &#043; \u8ba2\u9605<\/h4>\n<p>\u5899\u88c2\u63a8\u8350\u8ba2\u9605\u4e13\u680f &#x1f449; \u300aPython\u722c\u866b\u5b9e\u6218\u300b&#xff0c;\u672c\u4e13\u680f\u79c9\u627f\u7740\u4ee5\u201c\u5165\u95e8 \u2192 \u8fdb\u9636 \u2192 \u5de5\u7a0b\u5316 \u2192 \u9879\u76ee\u843d\u5730\u201d\u7684\u8def\u7ebf\u6301\u7eed\u66f4\u65b0&#xff0c;\u4e89\u53d6\u8ba9\u6bcf\u4e00\u671f\u5185\u5bb9\u90fd\u505a\u5230&#xff1a;<\/p>\n<p>\u2705 \u8bb2\u5f97\u6e05\u695a&#xff08;\u539f\u7406&#xff09;&#xff5c;\u2705 \u8dd1\u5f97\u8d77\u6765&#xff08;\u4ee3\u7801&#xff09;&#xff5c;\u2705 \u7528\u5f97\u4e0a&#xff08;\u573a\u666f&#xff09;&#xff5c;\u2705 \u625b\u5f97\u4f4f&#xff08;\u5de5\u7a0b\u5316&#xff09;<\/p>\n<p>&#x1f4e3; \u60f3\u7cfb\u7edf\u63d0\u5347\u7684\u5c0f\u4f19\u4f34&#xff1a;\u5f3a\u70c8\u5efa\u8bae\u5148\u8ba2\u9605\u4e13\u680f \u300aPython\u722c\u866b\u5b9e\u6218\u300b&#xff0c;\u518d\u6309\u76ee\u5f55\u5927\u7eb2\u987a\u5e8f\u5b66\u4e60&#xff0c;\u6548\u7387\u5341\u500d\u4e0a\u5347&#xff5e;<\/p>\n<p> <img decoding=\"async\" width=\"1000\" src=\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/09\/20260907072512-6a9e66d868cff.png\" \/> <\/p>\n<h4>\u2705 \u4e92\u52a8\u5f81\u96c6<\/h4>\n<p>\u60f3\u8ba9\u6211\u628a\u3010\u67d0\u7ad9\u70b9\/\u67d0\u53cd\u722c\/\u67d0\u9a8c\u8bc1\u7801\/\u67d0\u5206\u5e03\u5f0f\u65b9\u6848\u3011\u7b49\u5199\u6210\u67d0\u671f\u5b9e\u6218&#xff1f;<\/p>\n<p>\u8bc4\u8bba\u533a\u7559\u8a00\u544a\u8bc9\u6211\u4f60\u7684\u9700\u6c42&#xff0c;\u6211\u4f1a\u4f18\u5148\u5b89\u6392\u5b9e\u73b0(\u66f4\u65b0)\u54d2~<\/p>\n<hr \/>\n<p>\u2b50\ufe0f \u82e5\u559c\u6b22\u6211&#xff0c;\u5c31\u8bf7\u5173\u6ce8\u6211\u53ed&#xff5e;&#xff08;\u66f4\u65b0\u4e0d\u8ff7\u8def&#xff09; \u2b50\ufe0f \u82e5\u5bf9\u4f60\u6709\u7528&#xff0c;\u5c31\u8bf7\u70b9\u8d5e\u652f\u6301\u4e00\u4e0b\u53ed&#xff5e;&#xff08;\u7ed9\u6211\u4e00\u70b9\u70b9\u52a8\u529b&#xff09; \u2b50\ufe0f \u82e5\u6709\u7591\u95ee&#xff0c;\u5c31\u8bf7\u8bc4\u8bba\u7559\u8a00\u544a\u8bc9\u6211\u53ed&#xff5e;&#xff08;\u6211\u4f1a\u8865\u5751 &amp; \u66f4\u65b0\u8fed\u4ee3&#xff09;<\/p>\n<hr \/>\n<h4>\u2705 \u514d\u8d23\u58f0\u660e<\/h4>\n<p><font color=\"red\"> \u672c\u6587\u722c\u866b\u601d\u8def\u3001\u76f8\u5173\u6280\u672f\u548c\u4ee3\u7801\u4ec5\u7528\u4e8e\u5b66\u4e60\u53c2\u8003&#xff0c;\u5bf9\u9605\u8bfb\u672c\u6587\u540e\u7684\u8fdb\u884c\u722c\u866b\u884c\u4e3a\u7684\u7528\u6237\u672c\u4f5c\u8005\u4e0d\u627f\u62c5\u4efb\u4f55\u6cd5\u5f8b\u8d23\u4efb\u3002<\/font><\/p>\n<p>\u4f7f\u7528\u6216\u8005\u53c2\u8003\u672c\u9879\u76ee\u5373\u8868\u793a\u60a8\u5df2\u9605\u8bfb\u5e76\u540c\u610f\u4ee5\u4e0b\u6761\u6b3e&#xff1a;<\/p>\n<ul>\n<li>\u5408\u6cd5\u4f7f\u7528&#xff1a; \u4e0d\u5f97\u5c06\u672c\u9879\u76ee\u7528\u4e8e\u4efb\u4f55\u8fdd\u6cd5\u3001\u8fdd\u89c4\u6216\u4fb5\u72af\u4ed6\u4eba\u6743\u76ca\u7684\u884c\u4e3a&#xff0c;\u5305\u62ec\u4f46\u4e0d\u9650\u4e8e\u7f51\u7edc\u653b\u51fb\u3001\u8bc8\u9a97\u3001\u7ed5\u8fc7\u8eab\u4efd\u9a8c\u8bc1\u3001\u672a\u7ecf\u6388\u6743\u7684\u6570\u636e\u6293\u53d6\u7b49\u3002<\/li>\n<li>\u98ce\u9669\u81ea\u8d1f&#xff1a; \u4efb\u4f55\u56e0\u4f7f\u7528\u672c\u9879\u76ee\u800c\u4ea7\u751f\u7684\u6cd5\u5f8b\u8d23\u4efb\u3001\u6280\u672f\u98ce\u9669\u6216\u7ecf\u6d4e\u635f\u5931&#xff0c;\u7531\u4f7f\u7528\u8005\u81ea\u884c\u627f\u62c5&#xff0c;\u9879\u76ee\u4f5c\u8005\u4e0d\u627f\u62c5\u4efb\u4f55\u5f62\u5f0f\u7684\u8d23\u4efb\u3002<\/li>\n<li>\u7981\u6b62\u6ee5\u7528&#xff1a; \u4e0d\u5f97\u5c06\u672c\u9879\u76ee\u7528\u4e8e\u8fdd\u6cd5\u725f\u5229\u3001\u9ed1\u4ea7\u6d3b\u52a8\u6216\u5176\u4ed6\u4e0d\u5f53\u5546\u4e1a\u7528\u9014\u3002<\/li>\n<li>\u4f7f\u7528\u6216\u8005\u53c2\u8003\u672c\u9879\u76ee\u5373\u89c6\u4e3a\u540c\u610f\u4e0a\u8ff0\u6761\u6b3e,\u5373 \u201c\u8c01\u4f7f\u7528&#xff0c;\u8c01\u8d1f\u8d23\u201d \u3002\u5982\u4e0d\u540c\u610f&#xff0c;\u8bf7\u7acb\u5373\u505c\u6b62\u4f7f\u7528\u5e76\u5220\u9664\u672c\u9879\u76ee\u3002&#xff01;&#xff01;&#xff01;<\/li>\n<\/ul>\n<p> <img decoding=\"async\" width=\"1000\" src=\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/09\/20260907072512-6a9e66d8792c2.jpg\" \/> <\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u3297\ufe0f\u672c\u671f\u5185\u5bb9\u5df2\u6536\u5f55\u81f3\u4e13\u680f\u300aPython\u722c\u866b\u5b9e\u6218\u300b&#xff0c;\u6301\u7eed\u5b8c\u5584\u77e5\u8bc6\u4f53\u7cfb\u4e0e\u9879\u76ee\u5b9e\u6218&#xff0c;\u5efa\u8bae\u5148\u8ba2\u9605\u6536\u85cf&#xff0c;\u540e\u7eed\u67e5\u9605\u66f4\u65b9\u4fbf&#xff5e; \u3299\ufe0f\u672c\u671f\u722c\u866b\u96be\u5ea6\u6307\u6570&#xff1a;\u2b50\u2b50\u2b50\u2b50\u2606&#xff08;\u9ad8\u7ea7&#xff09; &#x1f250;\u798f\u5229&#xff1a; \u4e00\u6b21\u8ba2\u9605\u540e&#xff0c;\u4e13\u680f\u5185\u7684\u6240\u6709\u6587\u7ae0\u53ef\u6c38\u4e45\u514d\u8d39\u770b&#xff0c;\u6301\u7eed\u66f4\u65b0\u4e2d&#xff0c;\u4fdd\u5e951000(\u7bc7)\u786c\u6838\u5b9e\u6218\u5185\u5bb9\u3002 \u5168\u6587\u76ee\u5f55&#xff1a;&#x1f31f; \u5f00\u7bc7\u8bed0\ufe0f\u20e3 <\/p>\n","protected":false},"author":2,"featured_media":101771,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[11630,11229,11228,81,677],"topic":[],"class_list":["post-101774","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-server","tag-11630","tag-11229","tag-python","tag-677"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v20.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Python \u5b9e\u6218\uff1a\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe\u91c7\u96c6\u2014\u2014\u57fa\u4e8e OBIS \u65f6\u95f4\u8f74\u56fe\u5c42\u7684 API \u91c7\u96c6\u3001\u6e05\u6d17\u3001\u805a\u5408\u4e0e\u5bfc\u51fa - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.wsisp.com\/helps\/101774.html\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Python \u5b9e\u6218\uff1a\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe\u91c7\u96c6\u2014\u2014\u57fa\u4e8e OBIS \u65f6\u95f4\u8f74\u56fe\u5c42\u7684 API \u91c7\u96c6\u3001\u6e05\u6d17\u3001\u805a\u5408\u4e0e\u5bfc\u51fa - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"og:description\" content=\"\u3297\ufe0f\u672c\u671f\u5185\u5bb9\u5df2\u6536\u5f55\u81f3\u4e13\u680f\u300aPython\u722c\u866b\u5b9e\u6218\u300b&#xff0c;\u6301\u7eed\u5b8c\u5584\u77e5\u8bc6\u4f53\u7cfb\u4e0e\u9879\u76ee\u5b9e\u6218&#xff0c;\u5efa\u8bae\u5148\u8ba2\u9605\u6536\u85cf&#xff0c;\u540e\u7eed\u67e5\u9605\u66f4\u65b9\u4fbf&#xff5e; \u3299\ufe0f\u672c\u671f\u722c\u866b\u96be\u5ea6\u6307\u6570&#xff1a;\u2b50\u2b50\u2b50\u2b50\u2606&#xff08;\u9ad8\u7ea7&#xff09; &#x1f250;\u798f\u5229&#xff1a; \u4e00\u6b21\u8ba2\u9605\u540e&#xff0c;\u4e13\u680f\u5185\u7684\u6240\u6709\u6587\u7ae0\u53ef\u6c38\u4e45\u514d\u8d39\u770b&#xff0c;\u6301\u7eed\u66f4\u65b0\u4e2d&#xff0c;\u4fdd\u5e951000(\u7bc7)\u786c\u6838\u5b9e\u6218\u5185\u5bb9\u3002 \u5168\u6587\u76ee\u5f55&#xff1a;&#x1f31f; \u5f00\u7bc7\u8bed0\ufe0f\u20e3\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.wsisp.com\/helps\/101774.html\" \/>\n<meta property=\"og:site_name\" content=\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"article:published_time\" content=\"2026-09-07T07:25:17+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/09\/20260907072512-6a9e66d805f70.png\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"30 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/101774.html\",\"url\":\"https:\/\/www.wsisp.com\/helps\/101774.html\",\"name\":\"Python \u5b9e\u6218\uff1a\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe\u91c7\u96c6\u2014\u2014\u57fa\u4e8e OBIS \u65f6\u95f4\u8f74\u56fe\u5c42\u7684 API \u91c7\u96c6\u3001\u6e05\u6d17\u3001\u805a\u5408\u4e0e\u5bfc\u51fa - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"isPartOf\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\"},\"datePublished\":\"2026-09-07T07:25:17+00:00\",\"dateModified\":\"2026-09-07T07:25:17+00:00\",\"author\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/101774.html#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.wsisp.com\/helps\/101774.html\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/101774.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.wsisp.com\/helps\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Python \u5b9e\u6218\uff1a\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe\u91c7\u96c6\u2014\u2014\u57fa\u4e8e OBIS \u65f6\u95f4\u8f74\u56fe\u5c42\u7684 API \u91c7\u96c6\u3001\u6e05\u6d17\u3001\u805a\u5408\u4e0e\u5bfc\u51fa\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\",\"url\":\"https:\/\/www.wsisp.com\/helps\/\",\"name\":\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"description\":\"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"contentUrl\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"caption\":\"admin\"},\"sameAs\":[\"http:\/\/wp.wsisp.com\"],\"url\":\"https:\/\/www.wsisp.com\/helps\/author\/admin\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Python \u5b9e\u6218\uff1a\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe\u91c7\u96c6\u2014\u2014\u57fa\u4e8e OBIS \u65f6\u95f4\u8f74\u56fe\u5c42\u7684 API \u91c7\u96c6\u3001\u6e05\u6d17\u3001\u805a\u5408\u4e0e\u5bfc\u51fa - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.wsisp.com\/helps\/101774.html","og_locale":"zh_CN","og_type":"article","og_title":"Python \u5b9e\u6218\uff1a\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe\u91c7\u96c6\u2014\u2014\u57fa\u4e8e OBIS \u65f6\u95f4\u8f74\u56fe\u5c42\u7684 API \u91c7\u96c6\u3001\u6e05\u6d17\u3001\u805a\u5408\u4e0e\u5bfc\u51fa - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","og_description":"\u3297\ufe0f\u672c\u671f\u5185\u5bb9\u5df2\u6536\u5f55\u81f3\u4e13\u680f\u300aPython\u722c\u866b\u5b9e\u6218\u300b&#xff0c;\u6301\u7eed\u5b8c\u5584\u77e5\u8bc6\u4f53\u7cfb\u4e0e\u9879\u76ee\u5b9e\u6218&#xff0c;\u5efa\u8bae\u5148\u8ba2\u9605\u6536\u85cf&#xff0c;\u540e\u7eed\u67e5\u9605\u66f4\u65b9\u4fbf&#xff5e; \u3299\ufe0f\u672c\u671f\u722c\u866b\u96be\u5ea6\u6307\u6570&#xff1a;\u2b50\u2b50\u2b50\u2b50\u2606&#xff08;\u9ad8\u7ea7&#xff09; &#x1f250;\u798f\u5229&#xff1a; \u4e00\u6b21\u8ba2\u9605\u540e&#xff0c;\u4e13\u680f\u5185\u7684\u6240\u6709\u6587\u7ae0\u53ef\u6c38\u4e45\u514d\u8d39\u770b&#xff0c;\u6301\u7eed\u66f4\u65b0\u4e2d&#xff0c;\u4fdd\u5e951000(\u7bc7)\u786c\u6838\u5b9e\u6218\u5185\u5bb9\u3002 \u5168\u6587\u76ee\u5f55&#xff1a;&#x1f31f; \u5f00\u7bc7\u8bed0\ufe0f\u20e3","og_url":"https:\/\/www.wsisp.com\/helps\/101774.html","og_site_name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","article_published_time":"2026-09-07T07:25:17+00:00","og_image":[{"url":"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/09\/20260907072512-6a9e66d805f70.png"}],"author":"admin","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"admin","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"30 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.wsisp.com\/helps\/101774.html","url":"https:\/\/www.wsisp.com\/helps\/101774.html","name":"Python \u5b9e\u6218\uff1a\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe\u91c7\u96c6\u2014\u2014\u57fa\u4e8e OBIS \u65f6\u95f4\u8f74\u56fe\u5c42\u7684 API \u91c7\u96c6\u3001\u6e05\u6d17\u3001\u805a\u5408\u4e0e\u5bfc\u51fa - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","isPartOf":{"@id":"https:\/\/www.wsisp.com\/helps\/#website"},"datePublished":"2026-09-07T07:25:17+00:00","dateModified":"2026-09-07T07:25:17+00:00","author":{"@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41"},"breadcrumb":{"@id":"https:\/\/www.wsisp.com\/helps\/101774.html#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.wsisp.com\/helps\/101774.html"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.wsisp.com\/helps\/101774.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.wsisp.com\/helps"},{"@type":"ListItem","position":2,"name":"Python \u5b9e\u6218\uff1a\u52a8\u6001\u6e14\u4e1a\u8d44\u6e90\u5206\u5e03\u56fe\u91c7\u96c6\u2014\u2014\u57fa\u4e8e OBIS \u65f6\u95f4\u8f74\u56fe\u5c42\u7684 API \u91c7\u96c6\u3001\u6e05\u6d17\u3001\u805a\u5408\u4e0e\u5bfc\u51fa"}]},{"@type":"WebSite","@id":"https:\/\/www.wsisp.com\/helps\/#website","url":"https:\/\/www.wsisp.com\/helps\/","name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","description":"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41","name":"admin","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/","url":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","contentUrl":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","caption":"admin"},"sameAs":["http:\/\/wp.wsisp.com"],"url":"https:\/\/www.wsisp.com\/helps\/author\/admin"}]}},"_links":{"self":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/101774","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/comments?post=101774"}],"version-history":[{"count":0,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/101774\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media\/101771"}],"wp:attachment":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media?parent=101774"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/categories?post=101774"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/tags?post=101774"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/topic?post=101774"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}