{"id":93806,"date":"2026-08-13T09:40:52","date_gmt":"2026-08-13T01:40:52","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/93806.html"},"modified":"2026-08-13T09:40:52","modified_gmt":"2026-08-13T01:40:52","slug":"python-%e5%a4%9a%e8%bf%9b%e7%a8%8b%e5%af%b9%e8%b1%a1%e4%bc%a0%e9%80%92%e6%b7%b1%e5%ba%a6%e8%a7%a3%e6%9e%90%ef%bc%9a%e4%b8%ba%e4%bb%80%e4%b9%88%e5%bf%85%e9%a1%bb%e5%ba%8f%e5%88%97%e5%8c%96%ef%bc%8c","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/93806.html","title":{"rendered":"Python \u591a\u8fdb\u7a0b\u5bf9\u8c61\u4f20\u9012\u6df1\u5ea6\u89e3\u6790\uff1a\u4e3a\u4ec0\u4e48\u5fc5\u987b\u5e8f\u5217\u5316\uff0c\u4ee5\u53ca\u5982\u4f55\u964d\u4f4e\u901a\u4fe1\u6210\u672c"},"content":{"rendered":"

Python \u591a\u8fdb\u7a0b\u5bf9\u8c61\u4f20\u9012\u6df1\u5ea6\u89e3\u6790：\u4e3a\u4ec0\u4e48\u5fc5\u987b\u5e8f\u5217\u5316，\u4ee5\u53ca\u5982\u4f55\u964d\u4f4e\u901a\u4fe1\u6210\u672c<\/h2>\n

\u5728 Python \u5e76\u53d1\u7f16\u7a0b\u4e2d，multiprocessing \u7ecf\u5e38\u88ab\u7528\u6765\u5904\u7406 CPU \u5bc6\u96c6\u578b\u4efb\u52a1，\u4f8b\u5982\u56fe\u7247\u5904\u7406\u3001\u89c6\u9891\u8f6c\u7801\u3001\u6570\u636e\u6e05\u6d17\u3001\u79d1\u5b66\u8ba1\u7b97\u3001\u6279\u91cf\u7279\u5f81\u63d0\u53d6\u4ee5\u53ca\u673a\u5668\u5b66\u4e60\u9884\u5904\u7406\u3002<\/p>\n

\u5f88\u591a\u521d\u5b66\u8005\u7b2c\u4e00\u6b21\u63a5\u89e6\u591a\u8fdb\u7a0b\u65f6，\u4f1a\u4ea7\u751f\u4e00\u4e2a\u5f88\u81ea\u7136\u7684\u7591\u95ee：<\/p>\n

\u65e2\u7136\u6211\u5df2\u7ecf\u5728\u7236\u8fdb\u7a0b\u91cc\u521b\u5efa\u4e86\u4e00\u4e2a Python \u5bf9\u8c61，\u4e3a\u4ec0\u4e48\u4e0d\u80fd\u76f4\u63a5\u628a\u5b83\u4ea4\u7ed9\u5b50\u8fdb\u7a0b\u4f7f\u7528？\u4e3a\u4ec0\u4e48\u8fd8\u9700\u8981 pickle \u5e8f\u5217\u5316？<\/p>\n

\u4f8b\u5982\u4e0b\u9762\u8fd9\u6bb5\u4ee3\u7801，\u770b\u8d77\u6765\u53ea\u662f\u628a\u4e00\u4e2a\u5b57\u5178\u4f20\u7ed9\u5b50\u8fdb\u7a0b：<\/p>\n

from<\/span> multiprocessing import<\/span> Process<\/p>\n

def<\/span> worker<\/span>(<\/span>data)<\/span>:<\/span>
\n print<\/span>(<\/span>data)<\/span><\/p>\n

if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span>
\n user =<\/span> {<\/span>
\n "name"<\/span>:<\/span> "Alice"<\/span>,<\/span>
\n "age"<\/span>:<\/span> 28<\/span>,<\/span>
\n "skills"<\/span>:<\/span> [<\/span>"Python"<\/span>,<\/span> "SQL"<\/span>,<\/span> "Docker"<\/span>]<\/span>,<\/span>
\n }<\/span><\/p>\n

p =<\/span> Process(<\/span>target=<\/span>worker,<\/span> args=<\/span>(<\/span>user,<\/span>)<\/span>)<\/span>
\n p.<\/span>start(<\/span>)<\/span>
\n p.<\/span>join(<\/span>)<\/span><\/p>\n

\u4ece\u4ee3\u7801\u8868\u9762\u770b，\u6211\u4eec\u597d\u50cf\u53ea\u662f\u8fdb\u884c\u4e86：<\/p>\n

\u7236\u8fdb\u7a0b\u4e2d\u7684 user
\n \u2193
\n\u4f20\u7ed9 worker
\n \u2193
\n\u5b50\u8fdb\u7a0b\u4f7f\u7528<\/p>\n

\u4f46\u5e95\u5c42\u771f\u6b63\u53d1\u751f\u7684\u4e8b\u60c5\u8fdc\u6bd4\u8fd9\u590d\u6742\u3002<\/p>\n

\u5728\u5f88\u591a\u573a\u666f\u4e0b，\u5b9e\u9645\u8fc7\u7a0b\u66f4\u63a5\u8fd1：<\/p>\n

Python \u5bf9\u8c61
\n \u2193
\n\u5e8f\u5217\u5316
\n \u2193
\n\u5b57\u8282\u6570\u636e
\n \u2193
\n\u8fdb\u7a0b\u95f4\u901a\u4fe1
\n \u2193
\n\u53cd\u5e8f\u5217\u5316
\n \u2193
\n\u65b0\u7684 Python \u5bf9\u8c61<\/p>\n

\u4e5f\u5c31\u662f\u8bf4，\u591a\u8fdb\u7a0b\u7a0b\u5e8f\u4e2d\u7684\u201c\u4f20\u5bf9\u8c61\u201d，\u5f88\u591a\u65f6\u5019\u5b9e\u9645\u4e0a\u5e76\u4e0d\u662f\u628a\u90a3\u4e2a\u5bf9\u8c61\u672c\u8eab\u642c\u8fc7\u53bb，\u800c\u662f\u5728\u53e6\u4e00\u4e2a\u8fdb\u7a0b\u4e2d\u91cd\u5efa\u4e00\u4e2a\u7b49\u4ef7\u5bf9\u8c61\u3002<\/p>\n

\u7406\u89e3\u8fd9\u4e00\u70b9，\u662f\u638c\u63e1 Python multiprocessing\u3001Queue\u3001Pipe\u3001Pool\u3001ProcessPoolExecutor \u4e43\u81f3\u5206\u5e03\u5f0f\u7cfb\u7edf\u901a\u4fe1\u673a\u5236\u7684\u91cd\u8981\u57fa\u7840\u3002<\/p>\n

\u672c\u6587\u5c31\u56f4\u7ed5\u8fd9\u4e2a\u95ee\u9898\u5c55\u5f00：<\/p>\n

\u4e3a\u4ec0\u4e48\u591a\u8fdb\u7a0b\u5bf9\u8c61\u4f20\u9012\u901a\u5e38\u9700\u8981\u5e8f\u5217\u5316？\u5e8f\u5217\u5316\u5230\u5e95\u89e3\u51b3\u4e86\u4ec0\u4e48\u95ee\u9898？\u5b83\u4e3a\u4ec0\u4e48\u4f1a\u5f71\u54cd\u6027\u80fd？\u54ea\u4e9b\u5bf9\u8c61\u4e0d\u80fd\u4f20？\u53c8\u5e94\u8be5\u5982\u4f55\u8bbe\u8ba1\u9ad8\u6027\u80fd\u7684\u591a\u8fdb\u7a0b\u7a0b\u5e8f？<\/p>\n

\n

\u4e00\u3001\u5148\u4ece\u4e00\u4e2a\u6839\u672c\u95ee\u9898\u5f00\u59cb：\u4ec0\u4e48\u662f Python \u5bf9\u8c61？<\/h3>\n

\u6211\u4eec\u5e73\u65f6\u5199：<\/p>\n

user =<\/span> {<\/span>
\n "name"<\/span>:<\/span> "Alice"<\/span>,<\/span>
\n "age"<\/span>:<\/span> 28<\/span>
\n}<\/span><\/p>\n

\u4ece Python \u5c42\u9762\u770b：<\/p>\n

user<\/p>\n

\u662f\u4e00\u4e2a\u5b57\u5178\u3002<\/p>\n

\u4f46\u4ece\u8fdb\u7a0b\u548c\u64cd\u4f5c\u7cfb\u7edf\u89d2\u5ea6\u770b，\u5b83\u5b9e\u9645\u4e0a\u5b58\u5728\u4e8e：<\/p>\n

\u5f53\u524d Python \u8fdb\u7a0b\u7684\u865a\u62df\u5185\u5b58\u7a7a\u95f4<\/p>\n

\u53ef\u4ee5\u7c97\u7565\u7406\u89e3\u4e3a：<\/p>\n

Process A<\/p>\n

Virtual Memory
\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
\n\u2502 Python Interpreter \u2502
\n\u2502 \u2502
\n\u2502 user \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502
\n\u2502 \u2193 \u2502
\n\u2502 dict object \u2502
\n\u2502 0x7f123456 \u2502
\n\u2502 \u2502
\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

\u8fd9\u91cc\u6240\u8c13：<\/p>\n

0x7f123456<\/p>\n

\u53ea\u662f\u4e3a\u4e86\u5e2e\u52a9\u7406\u89e3\u7684\u201c\u5185\u5b58\u5730\u5740\u201d\u3002<\/p>\n

\u5173\u952e\u95ee\u9898\u5728\u4e8e：<\/p>\n

\u8fd9\u4e2a\u5730\u5740\u53ea\u5728\u5f53\u524d\u8fdb\u7a0b\u81ea\u5df1\u7684\u865a\u62df\u5730\u5740\u7a7a\u95f4\u4e2d\u6709\u610f\u4e49\u3002<\/p>\n

\u5982\u679c\u542f\u52a8\u53e6\u4e00\u4e2a\u8fdb\u7a0b：<\/p>\n

Process B<\/p>\n

Virtual Memory
\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
\n\u2502 Python Interpreter \u2502
\n\u2502 \u2502
\n\u2502 0x7f123456 \u2502
\n\u2502 \u2502
\n\u2502 \u53ef\u80fd\u5bf9\u5e94\u5b8c\u5168\u4e0d\u540c\u7684\u6570\u636e \u2502
\n\u2502 \u6216\u8005\u6839\u672c\u4e0d\u53ef\u8bbf\u95ee \u2502
\n\u2502 \u2502
\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

\u56e0\u6b64\u4f60\u4e0d\u80fd\u7b80\u5355\u5730\u544a\u8bc9\u5b50\u8fdb\u7a0b：<\/p>\n

\u5bf9\u8c61\u5728 0x7f123456，
\n\u4f60\u53bb\u90a3\u91cc\u62ff\u5427\u3002<\/p>\n

\u56e0\u4e3a：<\/p>\n

\u4e0d\u540c\u8fdb\u7a0b\u62e5\u6709\u5f7c\u6b64\u9694\u79bb\u7684\u5730\u5740\u7a7a\u95f4\u3002<\/p>\n

\u8fd9\u5c31\u662f\u6574\u4e2a\u95ee\u9898\u7684\u8d77\u70b9\u3002<\/p>\n

\n

\u4e8c\u3001\u7ebf\u7a0b\u53ef\u4ee5\u76f4\u63a5\u4f20\u5bf9\u8c61，\u4e3a\u4ec0\u4e48\u8fdb\u7a0b\u4e0d\u884c？<\/h2>\n

\u8fd9\u662f\u7406\u89e3\u5e8f\u5217\u5316\u6700\u597d\u7684\u5207\u5165\u70b9\u3002<\/p>\n

\u5148\u770b\u591a\u7ebf\u7a0b：<\/p>\n

import<\/span> threading<\/p>\n

data =<\/span> [<\/span>1<\/span>,<\/span> 2<\/span>,<\/span> 3<\/span>]<\/span><\/p>\n

def<\/span> worker<\/span>(<\/span>)<\/span>:<\/span>
\n data.<\/span>append(<\/span>4<\/span>)<\/span><\/p>\n

t =<\/span> threading.<\/span>Thread(<\/span>target=<\/span>worker)<\/span>
\nt.<\/span>start(<\/span>)<\/span>
\nt.<\/span>join(<\/span>)<\/span><\/p>\n

print<\/span>(<\/span>data)<\/span><\/p>\n

\u8f93\u51fa：<\/p>\n

[1, 2, 3, 4]<\/p>\n

\u539f\u56e0\u5728\u4e8e，\u540c\u4e00\u4e2a\u8fdb\u7a0b\u4e2d\u7684\u591a\u4e2a\u7ebf\u7a0b\u5171\u4eab：<\/p>\n

\u540c\u4e00\u4e2a\u865a\u62df\u5730\u5740\u7a7a\u95f4<\/p>\n

\u7ed3\u6784\u5927\u81f4\u662f：<\/p>\n

Process
\n\u2502
\n\u251c\u2500\u2500 Thread A
\n\u2502
\n\u251c\u2500\u2500 Thread B
\n\u2502
\n\u2514\u2500\u2500 Thread C<\/p>\n

\u5171\u540c\u8bbf\u95ee：<\/p>\n

Heap
\nGlobal Variables
\nPython Objects<\/p>\n

\u56e0\u6b64：<\/p>\n

Thread A
\n \\\\
\n \u2192 same list object
\n \/
\nThread B<\/p>\n

\u7ebf\u7a0b\u4e4b\u95f4\u53ef\u4ee5\u76f4\u63a5\u901a\u8fc7\u5bf9\u8c61\u5f15\u7528\u8bbf\u95ee\u540c\u4e00\u5bf9\u8c61\u3002<\/p>\n

\u4f46\u591a\u8fdb\u7a0b\u5b8c\u5168\u4e0d\u540c：<\/p>\n

Process A Process B<\/p>\n

Memory A Memory B
\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
\n\u2502 object A \u2502 \u2502 \u2502
\n\u2502 list \u2502 \u2502 \u2502
\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

\u4e24\u4e2a\u8fdb\u7a0b：<\/p>\n

\u5185\u5b58\u7a7a\u95f4\u5f7c\u6b64\u9694\u79bb<\/p>\n

\u56e0\u6b64\u9700\u8981\u67d0\u79cd\u673a\u5236\u628a\u4fe1\u606f\u4ece：<\/p>\n

Process A<\/p>\n

\u642c\u5230：<\/p>\n

Process B<\/p>\n

\u8fd9\u5c31\u662f：<\/p>\n

IPC
\nInter-Process Communication
\n\u8fdb\u7a0b\u95f4\u901a\u4fe1<\/p>\n

\u5e38\u89c1 IPC \u673a\u5236\u5305\u62ec：<\/p>\n

Pipe
\nQueue
\nSocket
\nShared Memory
\nMessage Queue
\nFile
\nMemory Map<\/p>\n

\u800c\u5982\u679c\u6211\u4eec\u8981\u4f20\u9012\u7684\u662f\u590d\u6742 Python \u5bf9\u8c61，\u5c31\u4ea7\u751f\u4e86\u4e00\u4e2a\u65b0\u95ee\u9898：<\/p>\n

Pipe \u548c Socket \u672c\u8d28\u4e0a\u80fd\u4f20\u4ec0\u4e48？<\/p>\n

\u7b54\u6848\u901a\u5e38\u4e0d\u662f：<\/p>\n

Python dict
\nPython class
\nPython list<\/p>\n

\u800c\u662f：<\/p>\n

bytes<\/p>\n

\u4e8e\u662f\u5e8f\u5217\u5316\u5c31\u51fa\u73b0\u4e86\u3002<\/p>\n

\n

\u4e09\u3001\u4ec0\u4e48\u53eb\u5e8f\u5217\u5316？<\/h2>\n

\u5e8f\u5217\u5316，\u672c\u8d28\u5c31\u662f：<\/p>\n

\u628a\u4e00\u4e2a\u8fd0\u884c\u65f6\u5bf9\u8c61\u8f6c\u6362\u6210\u4e00\u79cd\u53ef\u4ee5\u4fdd\u5b58\u6216\u4f20\u8f93\u7684\u6570\u636e\u8868\u793a\u3002<\/p>\n

\u4f8b\u5982：<\/p>\n

user =<\/span> {<\/span>
\n "name"<\/span>:<\/span> "Alice"<\/span>,<\/span>
\n "age"<\/span>:<\/span> 28<\/span>
\n}<\/span><\/p>\n

\u7ecf\u8fc7\u5e8f\u5217\u5316\u4e4b\u540e，\u53ef\u4ee5\u53d8\u6210\u4e00\u6bb5\u5b57\u8282\u6570\u636e：<\/p>\n

Python dict
\n \u2193
\npickle
\n \u2193
\nb'\\\\x80\\\\x04\\\\x95…'<\/p>\n

\u53d1\u9001\u5230\u53e6\u4e00\u4e2a\u8fdb\u7a0b：<\/p>\n

Process A<\/p>\n

dict
\n \u2193
\npickle.dumps()
\n \u2193
\nbytes
\n \u2193
\n================= IPC =================>
\n \u2193
\n Process B
\n \u2193
\n pickle.loads()
\n \u2193
\n dict<\/p>\n

\u4ee3\u7801\u793a\u4f8b：<\/p>\n

import<\/span> pickle<\/p>\n

user =<\/span> {<\/span>
\n "name"<\/span>:<\/span> "Alice"<\/span>,<\/span>
\n "age"<\/span>:<\/span> 28<\/span>,<\/span>
\n}<\/span><\/p>\n

raw =<\/span> pickle.<\/span>dumps(<\/span>user)<\/span><\/p>\n

print<\/span>(<\/span>type<\/span>(<\/span>raw)<\/span>)<\/span>
\nprint<\/span>(<\/span>raw)<\/span><\/p>\n

new_user =<\/span> pickle.<\/span>loads(<\/span>raw)<\/span><\/p>\n

print<\/span>(<\/span>type<\/span>(<\/span>new_user)<\/span>)<\/span>
\nprint<\/span>(<\/span>new_user)<\/span><\/p>\n

\u8f93\u51fa\u7c7b\u4f3c：<\/p>\n

<class 'bytes'>
\nb'\\\\x80\\\\x04…'<\/p>\n

<class 'dict'>
\n{'name': 'Alice', 'age': 28}<\/p>\n

\u6ce8\u610f：<\/p>\n

user is<\/span> new_user<\/p>\n

\u7ed3\u679c\u662f：<\/p>\n

False<\/p>\n

\u56e0\u4e3a\u5b83\u4eec\u5e76\u4e0d\u662f\u540c\u4e00\u4e2a\u5bf9\u8c61\u3002<\/p>\n

\u53ea\u662f：<\/p>\n

user ==<\/span> new_user<\/p>\n

\u901a\u5e38\u4e3a：<\/p>\n

True<\/p>\n

\u4e5f\u5c31\u662f\u8bf4：<\/p>\n

\u5e8f\u5217\u5316\u901a\u5e38\u4f20\u9012\u7684\u662f\u201c\u5bf9\u8c61\u7684\u72b6\u6001\u201d，\u800c\u4e0d\u662f\u201c\u5bf9\u8c61\u672c\u8eab\u201d\u3002<\/p>\n

\n

\u56db\u3001\u4e3a\u4ec0\u4e48 multiprocessing \u4e2d\u7279\u522b\u9700\u8981 pickle？<\/h2>\n

Python \u7684 multiprocessing \u5e0c\u671b\u7ed9\u5f00\u53d1\u8005\u4e00\u79cd\u975e\u5e38\u81ea\u7136\u7684\u63a5\u53e3：<\/p>\n

p =<\/span> Process(<\/span>
\n target=<\/span>worker,<\/span>
\n args=<\/span>(<\/span>data,<\/span>)<\/span>
\n)<\/span><\/p>\n

\u6211\u4eec\u5199\u7684\u662f：<\/p>\n

args=<\/span>(<\/span>data,<\/span>)<\/span><\/p>\n

\u770b\u8d77\u6765\u50cf\u666e\u901a\u51fd\u6570\u8c03\u7528：<\/p>\n

worker(<\/span>data)<\/span><\/p>\n

\u4f46\u5b9e\u9645\u4e0a worker \u53ef\u80fd\u8fd0\u884c\u5728\u5b8c\u5168\u4e0d\u540c\u7684 Python \u8fdb\u7a0b\u4e2d\u3002<\/p>\n

\u4e8e\u662f multiprocessing \u5fc5\u987b\u89e3\u51b3：<\/p>\n

\u51fd\u6570\u662f\u8c01？
\n\u53c2\u6570\u662f\u4ec0\u4e48？
\n\u5bf9\u8c61\u662f\u4ec0\u4e48\u7c7b\u578b？
\n\u5bf9\u8c61\u5f53\u524d\u72b6\u6001\u662f\u4ec0\u4e48？<\/p>\n

\u8fd9\u4e9b\u4fe1\u606f\u600e\u4e48\u4f20\u7ed9\u53e6\u4e00\u4e2a\u89e3\u91ca\u5668？<\/p>\n

Python \u6807\u51c6\u5e93\u901a\u5e38\u5229\u7528：<\/p>\n

pickle<\/p>\n

\u5c06\u53ef\u5e8f\u5217\u5316\u5bf9\u8c61\u8f6c\u6362\u6210\u5b57\u8282\u6d41\u3002<\/p>\n

\u56e0\u6b64\u5728 spawn\u3001\u8fdb\u7a0b\u6c60\u3001Queue \u7b49\u5f88\u591a\u573a\u666f\u4e2d：<\/p>\n

object
\n \u2193
\npickle
\n \u2193
\nIPC
\n \u2193
\nunpickle<\/p>\n

\u662f\u5e95\u5c42\u91cd\u8981\u673a\u5236\u3002<\/p>\n

\n

\u4e94\u3001\u4e00\u4e2a\u5b9e\u9a8c：Queue \u5e76\u4e0d\u662f\u7b80\u5355\u5730\u5171\u4eab\u5bf9\u8c61<\/h2>\n

\u6765\u770b：<\/p>\n

from<\/span> multiprocessing import<\/span> Process,<\/span> Queue<\/p>\n

def<\/span> worker<\/span>(<\/span>q)<\/span>:<\/span>
\n data =<\/span> q.<\/span>get(<\/span>)<\/span>
\n print<\/span>(<\/span>"child:"<\/span>,<\/span> data)<\/span><\/p>\n

if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span>
\n q =<\/span> Queue(<\/span>)<\/span><\/p>\n

data =<\/span> [<\/span>1<\/span>,<\/span> 2<\/span>,<\/span> 3<\/span>]<\/span><\/p>\n

p =<\/span> Process(<\/span>
\n target=<\/span>worker,<\/span>
\n args=<\/span>(<\/span>q,<\/span>)<\/span>
\n )<\/span><\/p>\n

p.<\/span>start(<\/span>)<\/span><\/p>\n

q.<\/span>put(<\/span>data)<\/span><\/p>\n

p.<\/span>join(<\/span>)<\/span><\/p>\n

\u770b\u8d77\u6765：<\/p>\n

q.<\/span>put(<\/span>data)<\/span><\/p>\n

\u50cf\u662f\u628a：<\/p>\n

data<\/p>\n

\u76f4\u63a5\u653e\u8fdb\u4e00\u4e2a\u516c\u5171\u5bb9\u5668\u3002<\/p>\n

\u4f46\u6982\u5ff5\u4e0a\u5b83\u66f4\u63a5\u8fd1：<\/p>\n

Parent Process<\/p>\n

data
\n \u2193
\npickle
\n \u2193
\nQueue internal pipe
\n \u2193
\n=========================
\n \u2193
\nChild Process
\n \u2193
\nunpickle
\n \u2193
\nnew data object<\/p>\n

\u56e0\u6b64\u5982\u679c\u5b50\u8fdb\u7a0b\u6267\u884c：<\/p>\n

data.<\/span>append(<\/span>999<\/span>)<\/span><\/p>\n

\u7236\u8fdb\u7a0b\u539f\u6765\u7684\u5217\u8868\u4e0d\u4f1a\u81ea\u52a8\u6539\u53d8\u3002<\/p>\n

\u4f8b\u5982：<\/p>\n

from<\/span> multiprocessing import<\/span> Process,<\/span> Queue<\/p>\n

def<\/span> worker<\/span>(<\/span>q)<\/span>:<\/span>
\n data =<\/span> q.<\/span>get(<\/span>)<\/span>
\n data.<\/span>append(<\/span>999<\/span>)<\/span><\/p>\n

print<\/span>(<\/span>"child:"<\/span>,<\/span> data)<\/span><\/p>\n

if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span>
\n q =<\/span> Queue(<\/span>)<\/span><\/p>\n

data =<\/span> [<\/span>1<\/span>,<\/span> 2<\/span>,<\/span> 3<\/span>]<\/span><\/p>\n

p =<\/span> Process(<\/span>
\n target=<\/span>worker,<\/span>
\n args=<\/span>(<\/span>q,<\/span>)<\/span>
\n )<\/span><\/p>\n

p.<\/span>start(<\/span>)<\/span>
\n q.<\/span>put(<\/span>data)<\/span><\/p>\n

p.<\/span>join(<\/span>)<\/span><\/p>\n

print<\/span>(<\/span>"parent:"<\/span>,<\/span> data)<\/span><\/p>\n

\u53ef\u80fd\u8f93\u51fa：<\/p>\n

child: [1, 2, 3, 999]
\nparent: [1, 2, 3]<\/p>\n

\u8fd9\u6b63\u8bf4\u660e：<\/p>\n

\u7236\u8fdb\u7a0b\u5bf9\u8c61
\n\u2260
\n\u5b50\u8fdb\u7a0b\u5bf9\u8c61<\/p>\n

\u4e24\u8fb9\u53ea\u662f\u5185\u5bb9\u76f8\u540c\u3002<\/p>\n

\n

\u516d\u3001\u5e8f\u5217\u5316\u89e3\u51b3\u7684\u5176\u5b9e\u662f\u201c\u8de8\u5730\u5740\u7a7a\u95f4\u91cd\u5efa\u201d<\/h2>\n

\u5982\u679c\u628a\u95ee\u9898\u8bf4\u5f97\u66f4\u51c6\u786e\u4e00\u70b9：<\/p>\n

\u5e8f\u5217\u5316\u5e76\u4e0d\u662f\u56e0\u4e3a Python\u201c\u6545\u610f\u9ebb\u70e6\u201d\u3002<\/p>\n

\u800c\u662f\u56e0\u4e3a：<\/p>\n

\u4e00\u4e2a Python \u5bf9\u8c61<\/p>\n

\u4e0d\u662f\u7b80\u5355\u7684\u4e00\u6bb5\u5b57\u7b26\u3002<\/p>\n

\u4f8b\u5982：<\/p>\n

data =<\/span> {<\/span>
\n "users"<\/span>:<\/span> [<\/span>
\n {<\/span>"name"<\/span>:<\/span> "Alice"<\/span>,<\/span> "score"<\/span>:<\/span> 95<\/span>}<\/span>,<\/span>
\n {<\/span>"name"<\/span>:<\/span> "Bob"<\/span>,<\/span> "score"<\/span>:<\/span> 88<\/span>}<\/span>,<\/span>
\n ]<\/span>,<\/span>
\n "active"<\/span>:<\/span> True<\/span>,<\/span>
\n}<\/span><\/p>\n

\u8fd9\u4e2a\u5bf9\u8c61\u5728\u5185\u5b58\u4e2d\u53ef\u80fd\u7531：<\/p>\n

dict
\n \u251c\u2500\u2500 str
\n \u251c\u2500\u2500 list
\n \u2502 \u251c\u2500\u2500 dict
\n \u2502 \u2502 \u251c\u2500\u2500 str
\n \u2502 \u2502 \u2514\u2500\u2500 int
\n \u2502 \u2514\u2500\u2500 dict
\n \u2502 \u251c\u2500\u2500 str
\n \u2502 \u2514\u2500\u2500 int
\n \u2514\u2500\u2500 bool<\/p>\n

\u7ec4\u6210\u3002<\/p>\n

\u5176\u4e2d\u6bcf\u4e2a\u5143\u7d20\u90fd\u53ef\u80fd\u662f\u72ec\u7acb\u5bf9\u8c61\u3002<\/p>\n

\u4e5f\u5c31\u662f\u8bf4，Python \u5bf9\u8c61\u5b9e\u9645\u5f62\u6210\u7684\u662f：<\/p>\n

object graph
\n\u5bf9\u8c61\u56fe<\/p>\n

\u800c\u4e0d\u662f：<\/p>\n

\u4e00\u5757\u8fde\u7eed\u7684\u6570\u636e<\/p>\n

\u5e8f\u5217\u5316\u8981\u505a\u7684\u4e8b\u60c5\u5c31\u662f：<\/p>\n

\u590d\u6742\u5bf9\u8c61\u56fe
\n \u2193
\n\u8f6c\u6210\u7a33\u5b9a\u7684\u5b57\u8282\u8868\u793a
\n \u2193
\n\u4f20\u8f93
\n \u2193
\n\u91cd\u65b0\u6784\u9020\u5bf9\u8c61\u56fe<\/p>\n

\u56e0\u6b64 pickle \u4e0d\u53ea\u662f\u4fdd\u5b58：<\/p>\n

value<\/p>\n

\u8fd8\u9700\u8981\u4fdd\u5b58\u8db3\u591f\u7684\u4fe1\u606f\u6765\u6062\u590d：<\/p>\n

type
\nstructure
\nreferences
\nstate<\/p>\n

\u8fd9\u5c31\u662f\u4e3a\u4ec0\u4e48\u5e8f\u5217\u5316\u662f\u4e00\u4e2a\u771f\u6b63\u7684\u8ba1\u7b97\u8fc7\u7a0b，\u800c\u4e0d\u662f\u7b80\u5355\u7684\u5185\u5b58\u590d\u5236\u3002<\/p>\n

\n

\u4e03\u3001\u4e3a\u4ec0\u4e48\u666e\u901a\u53d8\u91cf\u4e0d\u80fd\u76f4\u63a5\u5171\u4eab？<\/h2>\n

\u6765\u770b\u7ecf\u5178\u4ee3\u7801：<\/p>\n

from<\/span> multiprocessing import<\/span> Process<\/p>\n

count =<\/span> 0<\/span><\/p>\n

def<\/span> worker<\/span>(<\/span>)<\/span>:<\/span>
\n global<\/span> count
\n count +=<\/span> 1<\/span><\/p>\n

if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span>
\n p =<\/span> Process(<\/span>target=<\/span>worker)<\/span><\/p>\n

p.<\/span>start(<\/span>)<\/span>
\n p.<\/span>join(<\/span>)<\/span><\/p>\n

print<\/span>(<\/span>count)<\/span><\/p>\n

\u5f88\u591a\u521d\u5b66\u8005\u5e0c\u671b\u8f93\u51fa：<\/p>\n

1<\/p>\n

\u4f46\u901a\u5e38\u4ecd\u7136\u662f：<\/p>\n

0<\/p>\n

\u539f\u56e0\u5c31\u662f：<\/p>\n

Parent Process<\/p>\n

count = 0<\/p>\n

\u548c：<\/p>\n

Child Process<\/p>\n

count = 0
\n \u2193
\ncount = 1<\/p>\n

\u5b9e\u9645\u4e0a\u5b58\u5728\u4e8e\u4e0d\u540c\u5185\u5b58\u7a7a\u95f4\u3002<\/p>\n

\u7ed3\u6784\u4e0a\u662f：<\/p>\n

Parent Child<\/p>\n

count count
\n 0 0
\n \u2193
\n 1<\/p>\n

\u5b50\u8fdb\u7a0b\u4fee\u6539\u7684\u662f\u81ea\u5df1\u7684：<\/p>\n

count<\/p>\n

\u4e0d\u662f\u7236\u8fdb\u7a0b\u7684\u3002<\/p>\n

\u56e0\u6b64：<\/p>\n

\u591a\u8fdb\u7a0b\u7f16\u7a0b\u65f6，\u4e0d\u5e94\u8be5\u628a\u666e\u901a\u5168\u5c40\u53d8\u91cf\u60f3\u8c61\u6210\u7ebf\u7a0b\u4e2d\u7684\u5171\u4eab\u53d8\u91cf\u3002<\/p>\n

\n

\u516b\u3001\u4e3a\u4ec0\u4e48 spawn \u5bf9\u5e8f\u5217\u5316\u8981\u6c42\u7279\u522b\u660e\u663e？<\/h2>\n

Python \u591a\u8fdb\u7a0b\u6709\u4e0d\u540c\u542f\u52a8\u65b9\u5f0f，\u4f8b\u5982：<\/p>\n

fork
\nspawn
\nforkserver<\/p>\n

\u5176\u4e2d spawn \u7684\u884c\u4e3a\u5c24\u5176\u80fd\u5e2e\u52a9\u6211\u4eec\u7406\u89e3\u5e8f\u5217\u5316\u3002<\/p>\n

\u4f7f\u7528：<\/p>\n

ctx =<\/span> multiprocessing.<\/span>get_context(<\/span>"spawn"<\/span>)<\/span><\/p>\n

\u521b\u5efa\u8fdb\u7a0b\u65f6，Python \u4f1a\u542f\u52a8\u4e00\u4e2a：<\/p>\n

\u65b0\u7684 Python \u89e3\u91ca\u5668<\/p>\n

\u53ef\u4ee5\u7c97\u7565\u7406\u89e3\u4e3a：<\/p>\n

Parent<\/p>\n

Python Interpreter
\n |
\n | spawn
\n \u2193
\nNew Python Interpreter
\n |
\n \u2193
\nimport module
\n \u2193
\ndeserialize data
\n \u2193
\nexecute worker<\/p>\n

\u8fd9\u65f6\u5019\u5b50\u8fdb\u7a0b\u4e0d\u50cf\u7ebf\u7a0b\u90a3\u6837\u53ef\u4ee5\u770b\u5230\u7236\u8fdb\u7a0b\u73b0\u6709\u5bf9\u8c61\u3002<\/p>\n

\u5b83\u9700\u8981\u77e5\u9053：<\/p>\n

\u76ee\u6807\u51fd\u6570\u662f\u8c01？
\n\u53c2\u6570\u662f\u4ec0\u4e48？<\/p>\n

\u56e0\u6b64\u76ee\u6807\u51fd\u6570\u548c\u76f8\u5173\u53c2\u6570\u5fc5\u987b\u80fd\u591f\u4ee5\u67d0\u79cd\u65b9\u5f0f\u91cd\u65b0\u5b9a\u4f4d\u6216\u91cd\u5efa\u3002<\/p>\n

\u8fd9\u4e5f\u662f\u4e3a\u4ec0\u4e48\u4e0b\u9762\u7684\u4ee3\u7801\u5bb9\u6613\u51fa\u95ee\u9898：<\/p>\n

import<\/span> multiprocessing as<\/span> mp<\/p>\n

if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span>
\n ctx =<\/span> mp.<\/span>get_context(<\/span>"spawn"<\/span>)<\/span><\/p>\n

func =<\/span> lambda<\/span> x:<\/span> x *<\/span> 2<\/span><\/p>\n

p =<\/span> ctx.<\/span>Process(<\/span>
\n target=<\/span>func,<\/span>
\n args=<\/span>(<\/span>10<\/span>,<\/span>)<\/span>
\n )<\/span><\/p>\n

p.<\/span>start(<\/span>)<\/span>
\n p.<\/span>join(<\/span>)<\/span><\/p>\n

lambda \u901a\u5e38\u65e0\u6cd5\u50cf\u666e\u901a\u6a21\u5757\u9876\u5c42\u51fd\u6570\u90a3\u6837\u88ab\u6807\u51c6 pickle \u6b63\u5e38\u5b9a\u4f4d\u3002<\/p>\n

\n

\u4e5d\u3001\u4e3a\u4ec0\u4e48\u5c40\u90e8\u51fd\u6570\u4e5f\u7ecf\u5e38\u65e0\u6cd5\u4f20\u7ed9\u5b50\u8fdb\u7a0b？<\/h2>\n

\u4f8b\u5982：<\/p>\n

def<\/span> main<\/span>(<\/span>)<\/span>:<\/span><\/p>\n

def<\/span> worker<\/span>(<\/span>x)<\/span>:<\/span>
\n print<\/span>(<\/span>x *<\/span> 2<\/span>)<\/span><\/p>\n

.<\/span>.<\/span>.<\/span><\/p>\n

\u8fd9\u91cc：<\/p>\n

worker<\/p>\n

\u4e0d\u662f\u6a21\u5757\u7ea7\u51fd\u6570\u3002<\/p>\n

\u5b83\u5c5e\u4e8e：<\/p>\n

main.<locals>.worker<\/p>\n

\u65b0\u8fdb\u7a0b\u65e0\u6cd5\u7b80\u5355\u6267\u884c：<\/p>\n

import<\/span> module
\nmodule.<\/span>worker<\/p>\n

\u627e\u5230\u5b83\u3002<\/p>\n

\u56e0\u6b64\u66f4\u63a8\u8350：<\/p>\n

def<\/span> worker<\/span>(<\/span>x)<\/span>:<\/span>
\n print<\/span>(<\/span>x *<\/span> 2<\/span>)<\/span><\/p>\n

def<\/span> main<\/span>(<\/span>)<\/span>:<\/span>
\n .<\/span>.<\/span>.<\/span><\/p>\n

\u4e5f\u5c31\u662f：<\/p>\n

\u591a\u8fdb\u7a0b worker \u5c3d\u91cf\u5b9a\u4e49\u5728\u6a21\u5757\u9876\u5c42\u3002<\/p>\n

\u8fd9\u662f Python \u591a\u8fdb\u7a0b\u4ee3\u7801\u4e2d\u975e\u5e38\u91cd\u8981\u7684\u6700\u4f73\u5b9e\u8df5\u3002<\/p>\n

\n

\u5341\u3001\u7c7b\u5b9e\u4f8b\u80fd\u4e0d\u80fd\u5e8f\u5217\u5316？<\/h2>\n

\u901a\u5e38\u53ef\u4ee5，\u4f46\u9700\u8981\u770b\u7c7b\u7684\u5185\u90e8\u72b6\u6001\u3002<\/p>\n

\u4f8b\u5982：<\/p>\n

import<\/span> pickle<\/p>\n

class<\/span> User<\/span>:<\/span><\/p>\n

def<\/span> __init__<\/span>(<\/span>self,<\/span> name,<\/span> age)<\/span>:<\/span>
\n self.<\/span>name =<\/span> name
\n self.<\/span>age =<\/span> age<\/p>\n

user =<\/span> User(<\/span>"Alice"<\/span>,<\/span> 28<\/span>)<\/span><\/p>\n

raw =<\/span> pickle.<\/span>dumps(<\/span>user)<\/span><\/p>\n

new_user =<\/span> pickle.<\/span>loads(<\/span>raw)<\/span><\/p>\n

print<\/span>(<\/span>new_user.<\/span>name)<\/span>
\nprint<\/span>(<\/span>new_user.<\/span>age)<\/span><\/p>\n

\u901a\u5e38\u53ef\u4ee5\u6b63\u5e38\u6267\u884c\u3002<\/p>\n

\u4f46\u5982\u679c\u7c7b\u5305\u542b：<\/p>\n

thread lock
\nsocket
\ndatabase connection
\nfile handle
\ngenerator
\n\u67d0\u4e9b C \u6269\u5c55\u5bf9\u8c61<\/p>\n

\u5c31\u53ef\u80fd\u5931\u8d25\u3002<\/p>\n

\u4f8b\u5982：<\/p>\n

import<\/span> pickle
\nimport<\/span> threading<\/p>\n

class<\/span> Task<\/span>:<\/span><\/p>\n

def<\/span> __init__<\/span>(<\/span>self)<\/span>:<\/span>
\n self.<\/span>lock =<\/span> threading.<\/span>Lock(<\/span>)<\/span><\/p>\n

task =<\/span> Task(<\/span>)<\/span><\/p>\n

pickle.<\/span>dumps(<\/span>task)<\/span><\/p>\n

\u5f88\u53ef\u80fd\u62a5：<\/p>\n

TypeError:
\ncannot pickle '_thread.lock' object<\/p>\n

\u4e3a\u4ec0\u4e48？<\/p>\n

\u56e0\u4e3a\u4e00\u4e2a\u9501\u4e0d\u4ec5\u4ec5\u662f：<\/p>\n

\u666e\u901a\u6570\u636e<\/p>\n

\u5b83\u4ee3\u8868\u7684\u662f\u5f53\u524d\u8fdb\u7a0b\u8fd0\u884c\u65f6\u4e2d\u7684：<\/p>\n

\u540c\u6b65\u72b6\u6001
\n\u64cd\u4f5c\u7cfb\u7edf\u8d44\u6e90
\n\u7ebf\u7a0b\u72b6\u6001<\/p>\n

\u8fd9\u4e9b\u4e1c\u897f\u4e0d\u80fd\u7b80\u5355\u5730\u8f6c\u6362\u6210\u51e0\u4e2a\u5b57\u6bb5，\u7136\u540e\u5728\u53e6\u4e00\u4e2a\u8fdb\u7a0b\u91cc\u6062\u590d\u6210\u201c\u540c\u4e00\u628a\u9501\u201d\u3002<\/p>\n

\n

\u5341\u4e00\u3001\u54ea\u4e9b\u5bf9\u8c61\u901a\u5e38\u5bb9\u6613 pickle？<\/h2>\n

\u5e38\u89c1\u53ef\u5e8f\u5217\u5316\u5bf9\u8c61\u5305\u62ec：<\/p>\n

int<\/span>
\nfloat<\/span>
\nstr<\/span>
\nbytes<\/span>
\nbool<\/span>
\nNone<\/span><\/p>\n

list<\/span>
\ntuple<\/span>
\ndict<\/span>
\nset<\/span><\/p>\n

\u6a21\u5757\u9876\u5c42\u5b9a\u4e49\u7684\u51fd\u6570
\n\u6a21\u5757\u9876\u5c42\u5b9a\u4e49\u7684\u7c7b
\n\u7531\u53ef pickle \u6570\u636e\u7ec4\u6210\u7684\u7c7b\u5b9e\u4f8b<\/p>\n

\u4f8b\u5982：<\/p>\n

data =<\/span> {<\/span>
\n "name"<\/span>:<\/span> "Alice"<\/span>,<\/span>
\n "scores"<\/span>:<\/span> [<\/span>90<\/span>,<\/span> 88<\/span>,<\/span> 96<\/span>]<\/span>,<\/span>
\n "active"<\/span>:<\/span> True<\/span>,<\/span>
\n}<\/span><\/p>\n

\u975e\u5e38\u9002\u5408\u8fdb\u7a0b\u95f4\u4f20\u9012\u3002<\/p>\n

\n

\u5341\u4e8c\u3001\u54ea\u4e9b\u5bf9\u8c61\u7ecf\u5e38\u4e0d\u80fd\u76f4\u63a5 pickle？<\/h2>\n

\u5e38\u89c1\u9ad8\u98ce\u9669\u5bf9\u8c61\u5305\u62ec：<\/p>\n

lambda
\n\u5c40\u90e8\u51fd\u6570
\n\u751f\u6210\u5668
\n\u7ebf\u7a0b\u9501
\n\u6253\u5f00\u7684\u6587\u4ef6\u5bf9\u8c61
\nsocket
\n\u6570\u636e\u5e93\u8fde\u63a5
\n\u7ebf\u7a0b\u6c60
\n\u67d0\u4e9b\u7b2c\u4e09\u65b9 C \u6269\u5c55\u5bf9\u8c61
\n\u590d\u6742\u8fd0\u884c\u65f6\u4e0a\u4e0b\u6587<\/p>\n

\u56e0\u6b64\u5982\u679c\u4f60\u9047\u5230：<\/p>\n

PicklingError<\/p>\n

\u6216\u8005：<\/p>\n

Can't pickle local object<\/p>\n

\u7b2c\u4e00\u53cd\u5e94\u5e94\u8be5\u68c0\u67e5：<\/p>\n

\u6211\u662f\u4e0d\u662f\u5728\u628a\u201c\u8fd0\u884c\u72b6\u6001\u201d\u5f53\u4f5c\u201c\u666e\u901a\u6570\u636e\u201d\u4f20？<\/p>\n

\n

\u5341\u4e09\u3001\u5b9e\u6218：\u9519\u8bef\u5730\u4f20\u9012\u6570\u636e\u5e93\u8fde\u63a5<\/h2>\n

\u4f8b\u5982：<\/p>\n

from<\/span> multiprocessing import<\/span> Process<\/p>\n

db =<\/span> create_database_connection(<\/span>)<\/span><\/p>\n

def<\/span> worker<\/span>(<\/span>)<\/span>:<\/span>
\n db.<\/span>execute(<\/span>"SELECT …"<\/span>)<\/span><\/p>\n

if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span>
\n p =<\/span> Process(<\/span>target=<\/span>worker)<\/span>
\n p.<\/span>start(<\/span>)<\/span>
\n p.<\/span>join(<\/span>)<\/span><\/p>\n

\u8fd9\u6837\u7684\u4ee3\u7801\u975e\u5e38\u5371\u9669\u3002<\/p>\n

\u6570\u636e\u5e93\u8fde\u63a5\u80cc\u540e\u901a\u5e38\u5305\u542b：<\/p>\n

TCP socket
\nsession
\ntransaction
\nbuffer
\nauthentication state
\nprotocol state<\/p>\n

\u8fd9\u4e9b\u4e1c\u897f\u4e0d\u9002\u5408\u901a\u8fc7\u666e\u901a\u5e8f\u5217\u5316\u5728\u8fdb\u7a0b\u4e4b\u95f4\u4f20\u9012\u3002<\/p>\n

\u66f4\u52a0\u5408\u7406\u7684\u65b9\u6cd5\u662f：<\/p>\n

def<\/span> worker<\/span>(<\/span>config)<\/span>:<\/span><\/p>\n

db =<\/span> create_database_connection(<\/span>
\n host=<\/span>config[<\/span>"host"<\/span>]<\/span>,<\/span>
\n port=<\/span>config[<\/span>"port"<\/span>]<\/span>,<\/span>
\n user=<\/span>config[<\/span>"user"<\/span>]<\/span>,<\/span>
\n )<\/span><\/p>\n

try<\/span>:<\/span>
\n db.<\/span>execute(<\/span>.<\/span>.<\/span>.<\/span>)<\/span>
\n finally<\/span>:<\/span>
\n db.<\/span>close(<\/span>)<\/span><\/p>\n

\u4f20\u9012：<\/p>\n

\u8fde\u63a5\u914d\u7f6e<\/p>\n

\u800c\u4e0d\u662f：<\/p>\n

\u8fde\u63a5\u672c\u8eab<\/p>\n

\u8fd9\u662f\u4e00\u6761\u975e\u5e38\u5b9e\u7528\u7684\u8bbe\u8ba1\u539f\u5219：<\/p>\n

\u8de8\u8fdb\u7a0b\u4f20\u9012\u6570\u636e，\u800c\u4e0d\u662f\u4f20\u9012\u8d44\u6e90\u3002<\/p>\n

\n

\u5341\u56db\u3001\u8fdb\u4e00\u6b65\u7406\u89e3：\u5e8f\u5217\u5316\u5176\u5b9e\u662f\u4e00\u79cd\u201c\u534f\u8bae\u201d<\/h2>\n

\u5047\u8bbe\u7236\u8fdb\u7a0b\u62e5\u6709：<\/p>\n

user =<\/span> User(<\/span>
\n name=<\/span>"Alice"<\/span>,<\/span>
\n age=<\/span>28<\/span>
\n)<\/span><\/p>\n

\u8981\u5728\u5b50\u8fdb\u7a0b\u4e2d\u91cd\u65b0\u6784\u9020\u5b83，\u672c\u8d28\u4e0a\u9700\u8981\u63cf\u8ff0：<\/p>\n

\u8fd9\u662f\u54ea\u4e2a\u7c7b？
\n\u5c5e\u6027\u6709\u54ea\u4e9b？
\n\u5c5e\u6027\u5206\u522b\u662f\u4ec0\u4e48？
\n\u5f15\u7528\u5173\u7cfb\u662f\u4ec0\u4e48？<\/p>\n

\u53ef\u4ee5\u62bd\u8c61\u6210：<\/p>\n

{
\n type: "User",
\n state: {
\n name: "Alice",
\n age: 28
\n }
\n}<\/p>\n

\u4e8e\u662f：<\/p>\n

Serialize<\/p>\n

\u76f8\u5f53\u4e8e\u628a：<\/p>\n

\u8fd0\u884c\u65f6\u5bf9\u8c61<\/p>\n

\u6620\u5c04\u4e3a：<\/p>\n

\u4f20\u8f93\u534f\u8bae<\/p>\n

\u800c：<\/p>\n

Deserialize<\/p>\n

\u5219\u662f：<\/p>\n

\u534f\u8bae\u6570\u636e
\n \u2193
\n\u91cd\u65b0\u6784\u5efa\u8fd0\u884c\u65f6\u5bf9\u8c61<\/p>\n

\u8fd9\u548c\u5206\u5e03\u5f0f\u7cfb\u7edf\u4e2d\u7684：<\/p>\n

JSON
\nMessagePack
\nProtocol Buffers
\nAvro<\/p>\n

\u672c\u8d28\u4e0a\u662f\u540c\u4e00\u79cd\u8bbe\u8ba1\u601d\u60f3\u3002<\/p>\n

\u4e0d\u540c\u4e4b\u5904\u53ea\u662f：<\/p>\n

pickle \u66f4\u504f\u5411：<\/p>\n

Python \u2194 Python<\/p>\n

\u800c JSON\u3001Protobuf \u7b49\u66f4\u9002\u5408：<\/p>\n

Python \u2194 Java
\nPython \u2194 Go
\nPython \u2194 JavaScript<\/p>\n

\n

\u5341\u4e94\u3001\u771f\u6b63\u503c\u5f97\u5173\u6ce8\u7684\u95ee\u9898：\u5e8f\u5217\u5316\u6709\u6210\u672c<\/h2>\n

\u591a\u8fdb\u7a0b\u5e76\u4e0d\u662f：<\/p>\n

\u4efb\u52a1\u62c6\u6210 8 \u4efd
\n=
\n\u901f\u5ea6\u81ea\u52a8\u63d0\u5347 8 \u500d<\/p>\n

\u5b9e\u9645\u6267\u884c\u65f6\u95f4\u53ef\u80fd\u662f：<\/p>\n

T_total
\n=
\nT_pickle
\n+
\nT_transfer
\n+
\nT_unpickle
\n+
\nT_compute
\n+
\nT_sync<\/p>\n

\u5176\u4e2d：<\/p>\n

T_pickle<\/p>\n

\u662f\u5e8f\u5217\u5316\u65f6\u95f4\u3002<\/p>\n

T_transfer<\/p>\n

\u662f\u4f20\u8f93\u65f6\u95f4\u3002<\/p>\n

T_unpickle<\/p>\n

\u662f\u53cd\u5e8f\u5217\u5316\u65f6\u95f4\u3002<\/p>\n

\u5982\u679c\u4efb\u52a1\u672c\u8eab\u8ba1\u7b97\u975e\u5e38\u8f7b：<\/p>\n

T_compute = 1 ms<\/p>\n

\u4f46\u5e8f\u5217\u5316\u52a0\u901a\u4fe1：<\/p>\n

10 ms<\/p>\n

\u90a3\u4e48\u4f7f\u7528\u591a\u8fdb\u7a0b\u53cd\u800c\u4f1a\u66f4\u6162\u3002<\/p>\n

\n

\u5341\u516d\u3001\u4e00\u4e2a\u6027\u80fd\u9677\u9631：\u4f20\u9012\u5de8\u5927\u5217\u8868<\/h2>\n

\u4f8b\u5982：<\/p>\n

from<\/span> concurrent.<\/span>futures import<\/span> ProcessPoolExecutor<\/p>\n

def<\/span> compute<\/span>(<\/span>data)<\/span>:<\/span>
\n return<\/span> sum<\/span>(<\/span>data)<\/span><\/p>\n

if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span><\/p>\n

huge_data =<\/span> list<\/span>(<\/span>
\n range<\/span>(<\/span>10_000_000<\/span>)<\/span>
\n )<\/span><\/p>\n

with<\/span> ProcessPoolExecutor(<\/span>)<\/span> as<\/span> pool:<\/span><\/p>\n

result =<\/span> pool.<\/span>submit(<\/span>
\n compute,<\/span>
\n huge_data
\n )<\/span>.<\/span>result(<\/span>)<\/span><\/p>\n

print<\/span>(<\/span>result)<\/span><\/p>\n

\u95ee\u9898\u5728\u4e8e：<\/p>\n

huge_data<\/p>\n

\u975e\u5e38\u5927\u3002<\/p>\n

\u7a0b\u5e8f\u53ef\u80fd\u9700\u8981\u7ecf\u5386：<\/p>\n

\u5de8\u5927 Python list
\n \u2193
\npickle
\n \u2193
\n\u5de8\u5927 bytes
\n \u2193
\nIPC
\n \u2193
\nunpickle
\n \u2193
\n\u5de8\u5927 Python list<\/p>\n

\u4e8e\u662f\u4f60\u7684\u5b9e\u9645\u74f6\u9888\u53ef\u80fd\u6839\u672c\u4e0d\u662f：<\/p>\n

sum<\/span>(<\/span>data)<\/span><\/p>\n

\u800c\u662f：<\/p>\n

\u5bf9\u8c61\u590d\u5236\u4e0e\u4f20\u8f93<\/p>\n

\u8fd9\u5c31\u662f\u4e3a\u4ec0\u4e48\u5f88\u591a\u4eba\u4f1a\u53d1\u73b0：<\/p>\n

\u201c\u660e\u660e\u7528\u4e86 8 \u6838 CPU，\u4e3a\u4ec0\u4e48\u53cd\u800c\u6bd4\u5355\u8fdb\u7a0b\u8fd8\u6162？\u201d<\/p>\n

\u7b54\u6848\u7ecf\u5e38\u4e0d\u662f GIL，\u800c\u662f：<\/p>\n

serialization overhead<\/p>\n

\n

\u5341\u4e03\u3001\u7528\u4e00\u4e2a\u7b80\u5355\u6d4b\u8bd5\u89c2\u5bdf\u5e8f\u5217\u5316\u6210\u672c<\/h2>\n

\u53ef\u4ee5\u81ea\u5df1\u6d4b\u8bd5：<\/p>\n

import<\/span> pickle
\nimport<\/span> time<\/p>\n

data =<\/span> list<\/span>(<\/span>
\n range<\/span>(<\/span>5_000_000<\/span>)<\/span>
\n)<\/span><\/p>\n

start =<\/span> time.<\/span>perf_counter(<\/span>)<\/span><\/p>\n

raw =<\/span> pickle.<\/span>dumps(<\/span>data)<\/span><\/p>\n

t1 =<\/span> time.<\/span>perf_counter(<\/span>)<\/span><\/p>\n

new_data =<\/span> pickle.<\/span>loads(<\/span>raw)<\/span><\/p>\n

t2 =<\/span> time.<\/span>perf_counter(<\/span>)<\/span><\/p>\n

print<\/span>(<\/span>
\n "serialize:"<\/span>,<\/span>
\n t1 –<\/span> start
\n)<\/span><\/p>\n

print<\/span>(<\/span>
\n "deserialize:"<\/span>,<\/span>
\n t2 –<\/span> t1
\n)<\/span><\/p>\n

print<\/span>(<\/span>
\n "size:"<\/span>,<\/span>
\n len<\/span>(<\/span>raw)<\/span> \/<\/span> 1024<\/span> \/<\/span> 1024<\/span>,<\/span>
\n "MB"<\/span>
\n)<\/span><\/p>\n

\u4f60\u4f1a\u53d1\u73b0：<\/p>\n

\u5bf9\u8c61\u8d8a\u5927
\n \u2193
\n\u5e8f\u5217\u5316\u65f6\u95f4\u8d8a\u957f
\n \u2193
\n\u6570\u636e\u4f20\u8f93\u8d8a\u91cd
\n \u2193
\n\u8fdb\u7a0b\u5e76\u884c\u6536\u76ca\u8d8a\u5bb9\u6613\u88ab\u62b5\u6d88<\/p>\n

\u6240\u4ee5\u4f18\u5316 multiprocessing \u65f6，\u4e0d\u80fd\u53ea\u76ef\u7740：<\/p>\n

CPU utilization<\/p>\n

\u8fd8\u8981\u89c2\u5bdf：<\/p>\n

\u5e8f\u5217\u5316
\n\u590d\u5236
\n\u4f20\u8f93
\n\u540c\u6b65
\n\u4efb\u52a1\u7c92\u5ea6<\/p>\n

\n

\u5341\u516b\u3001Pool \u4e3a\u4ec0\u4e48\u7279\u522b\u5bb9\u6613\u4ea7\u751f\u5e8f\u5217\u5316\u5f00\u9500？<\/h2>\n

\u6765\u770b：<\/p>\n

from<\/span> multiprocessing import<\/span> Pool<\/p>\n

def<\/span> compute<\/span>(<\/span>data)<\/span>:<\/span>
\n return<\/span> sum<\/span>(<\/span>data)<\/span><\/p>\n

if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span><\/p>\n

with<\/span> Pool(<\/span>4<\/span>)<\/span> as<\/span> pool:<\/span><\/p>\n

result =<\/span> pool.<\/span>map<\/span>(<\/span>
\n compute,<\/span>
\n datasets
\n )<\/span><\/p>\n

\u6982\u5ff5\u4e0a：<\/p>\n

Main Process
\n |
\n \u251c\u2500\u2500 pickle(dataset1)
\n | \u2193
\n | Worker 1
\n |
\n \u251c\u2500\u2500 pickle(dataset2)
\n | \u2193
\n | Worker 2
\n |
\n \u251c\u2500\u2500 pickle(dataset3)
\n | \u2193
\n | Worker 3
\n |
\n \u2514\u2500\u2500 pickle(dataset4)
\n \u2193
\n Worker 4<\/p>\n

\u8ba1\u7b97\u5b8c\u6210\u540e\u7ed3\u679c\u53ef\u80fd\u8fd8\u9700\u8981：<\/p>\n

pickle(result)
\n \u2193
\nMain Process<\/p>\n

\u4e5f\u5c31\u662f\u8bf4：<\/p>\n

\u8f93\u5165\u5e8f\u5217\u5316\u4e00\u6b21
\n+
\n\u8f93\u51fa\u5e8f\u5217\u5316\u4e00\u6b21<\/p>\n

\u4efb\u52a1\u8d8a\u788e，\u5f00\u9500\u8d8a\u660e\u663e\u3002<\/p>\n

\n

\u5341\u4e5d\u3001\u6700\u4f73\u5b9e\u8df5\u4e00：\u63d0\u9ad8\u4efb\u52a1\u7c92\u5ea6<\/h2>\n

\u9519\u8bef\u601d\u8def：<\/p>\n

pool.<\/span>map<\/span>(<\/span>
\n process,<\/span>
\n millions_of_tiny_items
\n)<\/span><\/p>\n

\u5982\u679c\u6bcf\u4e2a\u4efb\u52a1\u53ea\u6267\u884c：<\/p>\n

\u51e0\u5341\u5fae\u79d2<\/p>\n

\u901a\u4fe1\u6210\u672c\u53ef\u80fd\u8fdc\u5927\u4e8e\u8ba1\u7b97\u6210\u672c\u3002<\/p>\n

\u53ef\u4ee5\u8003\u8651\u6279\u5904\u7406：<\/p>\n

def<\/span> process_batch<\/span>(<\/span>items)<\/span>:<\/span><\/p>\n

result =<\/span> [<\/span>]<\/span><\/p>\n

for<\/span> item in<\/span> items:<\/span>
\n result.<\/span>append(<\/span>
\n process(<\/span>item)<\/span>
\n )<\/span><\/p>\n

return<\/span> result<\/p>\n

\u7136\u540e：<\/p>\n

100 \u4e07\u4e2a\u4efb\u52a1<\/p>\n

\u53d8\u6210：<\/p>\n

1000 \u4e2a batch<\/p>\n

\u7ed3\u6784\u4ece：<\/p>\n

serialize
\ncompute
\nserialize<\/p>\n

serialize
\ncompute
\nserialize<\/p>\n

serialize
\ncompute
\nserialize<\/p>\n

\u53d8\u6210：<\/p>\n

serialize batch
\n \u2193
\n\u5927\u91cf\u8ba1\u7b97
\n \u2193
\nserialize result<\/p>\n

\u591a\u8fdb\u7a0b\u6548\u7387\u901a\u5e38\u4f1a\u660e\u663e\u63d0\u5347\u3002<\/p>\n

\n

\u4e8c\u5341\u3001map \u4e2d\u7684 chunksize \u4e3a\u4ec0\u4e48\u5f88\u91cd\u8981？<\/h2>\n

Pool.map() \u652f\u6301：<\/p>\n

chunksize<\/p>\n

\u4f8b\u5982：<\/p>\n

with<\/span> Pool(<\/span>4<\/span>)<\/span> as<\/span> pool:<\/span><\/p>\n

result =<\/span> pool.<\/span>map<\/span>(<\/span>
\n calculate,<\/span>
\n range<\/span>(<\/span>1_000_000<\/span>)<\/span>,<\/span>
\n chunksize=<\/span>1000<\/span>,<\/span>
\n )<\/span><\/p>\n

\u5982\u679c\u6bcf\u4e2a\u4efb\u52a1\u975e\u5e38\u5c0f：<\/p>\n

1 item
\n1 IPC message<\/p>\n

\u4f1a\u4ea7\u751f\u5927\u91cf\u901a\u4fe1\u3002<\/p>\n

\u4f7f\u7528 chunks：<\/p>\n

1000 items
\n1 IPC message<\/p>\n

\u53ef\u4ee5\u51cf\u5c11：<\/p>\n

\u4efb\u52a1\u8c03\u5ea6
\n\u5e8f\u5217\u5316
\n\u6d88\u606f\u6570\u91cf
\n\u8fdb\u7a0b\u540c\u6b65<\/p>\n

\u5f00\u9500\u3002<\/p>\n

\u56e0\u6b64\u5904\u7406\u5927\u91cf\u8f7b\u91cf\u4efb\u52a1\u65f6：<\/p>\n

\u8c03\u6574 chunksize \u5f80\u5f80\u6bd4\u5355\u7eaf\u589e\u52a0 worker \u6570\u66f4\u6709\u6548\u3002<\/p>\n

\n

\u4e8c\u5341\u4e00\u3001\u6700\u4f73\u5b9e\u8df5\u4e8c：\u8ba9 worker \u81ea\u5df1\u52a0\u8f7d\u5927\u8d44\u6e90<\/h2>\n

\u5047\u8bbe\u6709\u4e00\u4e2a：<\/p>\n

2 GB \u6a21\u578b<\/p>\n

\u6700\u5dee\u7684\u8bbe\u8ba1：<\/p>\n

def<\/span> infer<\/span>(<\/span>model,<\/span> image)<\/span>:<\/span>
\n .<\/span>.<\/span>.<\/span><\/p>\n

\u7136\u540e\u6bcf\u6b21\u90fd：<\/p>\n

pool.<\/span>submit(<\/span>
\n infer,<\/span>
\n model,<\/span>
\n image
\n)<\/span><\/p>\n

\u6982\u5ff5\u4e0a\u53ef\u80fd\u4e0d\u65ad\u5c1d\u8bd5\u4f20：<\/p>\n

2 GB model<\/p>\n

\u6b63\u786e\u601d\u8def\u901a\u5e38\u662f：<\/p>\n

Worker \u542f\u52a8
\n \u2193
\n\u6bcf\u4e2a Worker \u521d\u59cb\u5316\u6a21\u578b
\n \u2193
\n\u4ee5\u540e\u53ea\u4f20\u8f7b\u91cf\u4efb\u52a1\u53c2\u6570<\/p>\n

\u4f8b\u5982：<\/p>\n

model =<\/span> None<\/span><\/p>\n

def<\/span> init_worker<\/span>(<\/span>model_path)<\/span>:<\/span><\/p>\n

global<\/span> model<\/p>\n

model =<\/span> load_model(<\/span>
\n model_path
\n )<\/span><\/p>\n

def<\/span> infer<\/span>(<\/span>image_path)<\/span>:<\/span><\/p>\n

global<\/span> model<\/p>\n

image =<\/span> load_image(<\/span>
\n image_path
\n )<\/span><\/p>\n

return<\/span> model.<\/span>predict(<\/span>image)<\/span><\/p>\n

\u521b\u5efa Pool：<\/p>\n

from<\/span> multiprocessing import<\/span> Pool<\/p>\n

if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span><\/p>\n

with<\/span> Pool(<\/span>
\n processes=<\/span>4<\/span>,<\/span>
\n initializer=<\/span>init_worker,<\/span>
\n initargs=<\/span>(<\/span>"model.bin"<\/span>,<\/span>)<\/span>,<\/span>
\n )<\/span> as<\/span> pool:<\/span><\/p>\n

results =<\/span> pool.<\/span>map<\/span>(<\/span>
\n infer,<\/span>
\n image_paths,<\/span>
\n )<\/span><\/p>\n

\u8fd9\u6837\u901a\u4fe1\u5185\u5bb9\u53d8\u6210：<\/p>\n

"001.jpg"
\n"002.jpg"
\n"003.jpg"<\/p>\n

\u800c\u4e0d\u662f：<\/p>\n

2 GB model
\n+
\nimage<\/p>\n

\u8fd9\u5c5e\u4e8e\u975e\u5e38\u5178\u578b\u7684\u591a\u8fdb\u7a0b\u4f18\u5316\u6a21\u5f0f\u3002<\/p>\n

\n

\u4e8c\u5341\u4e8c\u3001\u6700\u4f73\u5b9e\u8df5\u4e09：\u4f20 ID，\u4e0d\u8981\u4f20\u5927\u5bf9\u8c61<\/h2>\n

\u5047\u8bbe\u4f60\u9700\u8981\u5904\u7406：<\/p>\n

user =<\/span> {<\/span>
\n "id"<\/span>:<\/span> 10001<\/span>,<\/span>
\n "name"<\/span>:<\/span> "…"<\/span>,<\/span>
\n "history"<\/span>:<\/span> [<\/span>.<\/span>.<\/span>.<\/span>\u5927\u91cf\u6570\u636e.<\/span>.<\/span>.<\/span>]<\/span>,<\/span>
\n}<\/span><\/p>\n

\u4e0d\u8981\u603b\u662f：<\/p>\n

pool.<\/span>submit(<\/span>
\n process_user,<\/span>
\n user
\n)<\/span><\/p>\n

\u53ef\u4ee5\u8003\u8651：<\/p>\n

pool.<\/span>submit(<\/span>
\n process_user,<\/span>
\n user[<\/span>"id"<\/span>]<\/span>
\n)<\/span><\/p>\n

\u7136\u540e worker \u6839\u636e：<\/p>\n

user_id<\/p>\n

\u81ea\u5df1\u8bbf\u95ee：<\/p>\n

\u6570\u636e\u5e93
\n\u672c\u5730\u6587\u4ef6
\n\u5185\u5b58\u6620\u5c04
\n\u5171\u4eab\u5b58\u50a8<\/p>\n

\u8fd9\u79cd\u8bbe\u8ba1\u5728\u5206\u5e03\u5f0f\u7cfb\u7edf\u91cc\u5c24\u5176\u5e38\u89c1：<\/p>\n

\u4f20 reference
\n\u800c\u4e0d\u662f
\n\u4f20\u6574\u4e2a object<\/p>\n

\u672c\u8d28\u4e0a\u662f\u5728\u964d\u4f4e：<\/p>\n

data movement<\/p>\n

\n

\u4e8c\u5341\u4e09\u3001\u6700\u4f73\u5b9e\u8df5\u56db：\u4f7f\u7528\u5171\u4eab\u5185\u5b58<\/h2>\n

\u5982\u679c\u4e0d\u540c\u8fdb\u7a0b\u786e\u5b9e\u9700\u8981\u8bbf\u95ee\u5927\u5757\u6570\u636e：<\/p>\n

\u5927\u578b NumPy \u6570\u7ec4
\n\u56fe\u50cf\u77e9\u9635
\n\u6a21\u578b\u8f93\u5165
\n\u6570\u503c\u6570\u636e<\/p>\n

\u53cd\u590d pickle \u901a\u5e38\u4e0d\u662f\u6700\u4f73\u65b9\u6848\u3002<\/p>\n

\u53ef\u4ee5\u8003\u8651：<\/p>\n

multiprocessing.<\/span>shared_memory<\/p>\n

\u57fa\u672c\u601d\u60f3\u662f：<\/p>\n

Shared Memory
\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
\n \u2502 Large Array \u2502
\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
\n \u2191 \u2191
\n \u2502 \u2502
\n Process A Process B<\/p>\n

\u8fd9\u65f6\u5019\u5e76\u4e0d\u662f：<\/p>\n

\u590d\u5236\u6574\u4e2a\u5927\u6570\u7ec4<\/p>\n

\u800c\u662f：<\/p>\n

\u591a\u4e2a\u8fdb\u7a0b\u8bbf\u95ee\u540c\u4e00\u5757\u5171\u4eab\u5185\u5b58<\/p>\n

\u5982\u679c\u9700\u8981\u4f20\u8f93\u7684\u4fe1\u606f\u53ea\u6709：<\/p>\n

\u5171\u4eab\u5185\u5b58\u540d\u79f0
\nshape
\ndtype
\noffset<\/p>\n

\u901a\u4fe1\u6210\u672c\u4f1a\u5c0f\u5f97\u591a\u3002<\/p>\n

\n

\u4e8c\u5341\u56db\u3001NumPy \u573a\u666f\u5c24\u5176\u9002\u5408\u5171\u4eab\u5185\u5b58<\/h2>\n

\u4f8b\u5982：<\/p>\n

import<\/span> numpy as<\/span> np<\/p>\n

data =<\/span> np.<\/span>random.<\/span>rand(<\/span>
\n 10000<\/span>,<\/span>
\n 10000<\/span>,<\/span>
\n)<\/span><\/p>\n

\u5982\u679c\u76f4\u63a5：<\/p>\n

pool.<\/span>submit(<\/span>
\n calculate,<\/span>
\n data
\n)<\/span><\/p>\n

\u53ef\u80fd\u9700\u8981\u5e8f\u5217\u5316\u5de8\u5927\u6570\u7ec4\u3002<\/p>\n

\u800c\u5171\u4eab\u5185\u5b58\u65b9\u6848\u5927\u81f4\u53ef\u4ee5\u8bbe\u8ba1\u6210：<\/p>\n

Main Process<\/p>\n

NumPy Array
\n \u2193
\nShared Memory
\n \u2193
\nshm_name = "psm_xxxxx"
\n \u2193
\nworker \u53ea\u63a5\u6536：<\/p>\n

{
\n name,
\n shape,
\n dtype
\n}<\/p>\n

worker \u518d\u901a\u8fc7：<\/p>\n

shared memory<\/p>\n

\u91cd\u5efa NumPy view\u3002<\/p>\n

\u8fd9\u91cc\u4f20\u9012\u7684\u4e0d\u662f：<\/p>\n

800 MB \u6570\u7ec4<\/p>\n

\u800c\u53ef\u80fd\u53ea\u662f：<\/p>\n

\u51e0\u5341\u5b57\u8282 metadata<\/p>\n

\u8fd9\u5c31\u662f\u6570\u636e\u5bc6\u96c6\u578b\u5e76\u884c\u7a0b\u5e8f\u4e2d\u975e\u5e38\u91cd\u8981\u7684\u4f18\u5316\u601d\u60f3\u3002<\/p>\n

\n

\u4e8c\u5341\u4e94\u3001\u90a3 Manager \u4e3a\u4ec0\u4e48\u770b\u8d77\u6765\u53ef\u4ee5\u5171\u4eab\u5bf9\u8c61？<\/h2>\n

Python \u63d0\u4f9b：<\/p>\n

multiprocessing.<\/span>Manager<\/p>\n

\u4f8b\u5982：<\/p>\n

from<\/span> multiprocessing import<\/span> Manager<\/p>\n

with<\/span> Manager(<\/span>)<\/span> as<\/span> manager:<\/span><\/p>\n

shared_list =<\/span> manager.<\/span>list<\/span>(<\/span>
\n [<\/span>1<\/span>,<\/span> 2<\/span>,<\/span> 3<\/span>]<\/span>
\n )<\/span><\/p>\n

\u5f88\u591a\u521d\u5b66\u8005\u4ee5\u4e3a：<\/p>\n

Manager
\n=
\n\u771f\u6b63\u5171\u4eab\u666e\u901a Python list<\/p>\n

\u5176\u5b9e\u4e0d\u662f\u8fd9\u4e48\u7b80\u5355\u3002<\/p>\n

Manager \u901a\u5e38\u4f1a\u542f\u52a8\u4e00\u4e2a\u4e13\u95e8\u7684\u7ba1\u7406\u8fdb\u7a0b：<\/p>\n

Worker A
\n |
\n | RPC
\n \u2193<\/p>\n

Manager Process
\n |
\n | owns list
\n \u2193
\n[1, 2, 3]<\/p>\n

\u2191
\n | RPC
\n |
\nWorker B<\/p>\n

Worker \u62ff\u5230\u7684\u662f：<\/p>\n

proxy
\n\u4ee3\u7406\u5bf9\u8c61<\/p>\n

\u4f8b\u5982：<\/p>\n

shared_list.<\/span>append(<\/span>4<\/span>)<\/span><\/p>\n

\u5185\u90e8\u53ef\u80fd\u53d8\u6210：<\/p>\n

\u53d1\u9001\u8bf7\u6c42：
\n"\u8bf7\u6267\u884c append(4)"<\/p>\n

Manager process \u6267\u884c\u540e，\u518d\u8fd4\u56de\u7ed3\u679c\u3002<\/p>\n

\u6240\u4ee5 Manager \u5f88\u65b9\u4fbf，\u4f46\u64cd\u4f5c\u9891\u7e41\u65f6：<\/p>\n

\u5e8f\u5217\u5316
\nIPC
\n\u540c\u6b65
\n\u4ee3\u7406\u8c03\u7528<\/p>\n

\u90fd\u4f1a\u4ea7\u751f\u5f00\u9500\u3002<\/p>\n

\u56e0\u6b64：<\/p>\n

Manager \u9002\u5408\u65b9\u4fbf\u5730\u5171\u4eab\u5c11\u91cf\u590d\u6742\u72b6\u6001，\u4e0d\u9002\u5408\u9ad8\u9891\u3001\u5927\u89c4\u6a21\u6570\u503c\u8ba1\u7b97\u3002<\/p>\n

\n

\u4e8c\u5341\u516d\u3001Value \u548c Array \u4e3a\u4ec0\u4e48\u66f4\u5feb？<\/h2>\n

multiprocessing \u8fd8\u63d0\u4f9b：<\/p>\n

Value
\nArray<\/p>\n

\u4f8b\u5982：<\/p>\n

from<\/span> multiprocessing import<\/span> Value<\/p>\n

counter =<\/span> Value(<\/span>
\n "i"<\/span>,<\/span>
\n 0<\/span>
\n)<\/span><\/p>\n

\u8fd9\u79cd\u7ed3\u6784\u66f4\u63a5\u8fd1：<\/p>\n

\u5171\u4eab\u5185\u5b58\u4e2d\u7684\u57fa\u7840 C \u6570\u636e\u7c7b\u578b<\/p>\n

\u800c\u4e0d\u662f：<\/p>\n

\u6574\u4e2a Python object<\/p>\n

\u56e0\u6b64\u5bf9\u4e8e：<\/p>\n

\u6574\u6570
\n\u6d6e\u70b9\u6570
\n\u56fa\u5b9a\u6570\u7ec4<\/p>\n

\u8fd9\u7c7b\u7ed3\u6784，\u901a\u5e38\u6bd4 Manager \u4ee3\u7406\u5bf9\u8c61\u66f4\u52a0\u76f4\u63a5\u3002<\/p>\n

\u4f46\u4ee3\u4ef7\u662f：<\/p>\n

\u652f\u6301\u7684\u6570\u636e\u7ed3\u6784\u6ca1\u90a3\u4e48\u7075\u6d3b<\/p>\n

\u8fd9\u5c31\u662f\u7ecf\u5178\u5de5\u7a0b\u53d6\u820d：<\/p>\n

\u4fbf\u5229\u6027
\nvs
\n\u6027\u80fd<\/p>\n

\n

\u4e8c\u5341\u4e03\u3001\u5e8f\u5217\u5316\u548c\u5171\u4eab\u5185\u5b58\u5230\u5e95\u600e\u4e48\u9009？<\/h2>\n

\u53ef\u4ee5\u7528\u4e0b\u9762\u7684\u5224\u65ad\u65b9\u5f0f\u3002<\/p>\n

\u5c0f\u6570\u636e\u3001\u4f4e\u9891\u901a\u4fe1<\/h4>\n

\u4f8b\u5982：<\/p>\n

\u4efb\u52a1 ID
\n\u6587\u4ef6\u540d
\n\u7b80\u5355\u5b57\u5178
\n\u914d\u7f6e\u9879
\n\u5c11\u91cf\u6570\u503c<\/p>\n

\u76f4\u63a5：<\/p>\n

pickle + Queue \/ Pool<\/p>\n

\u901a\u5e38\u8db3\u591f\u3002<\/p>\n

\n

\u4e2d\u7b49\u6570\u636e\u3001\u8ba1\u7b97\u4efb\u52a1\u8f83\u91cd<\/h4>\n

\u4f8b\u5982：<\/p>\n

\u51e0 MB \u8f93\u5165
\n\u6bcf\u4e2a\u4efb\u52a1\u8ba1\u7b97\u51e0\u79d2<\/p>\n

\u5373\u4f7f\u6709\u5e8f\u5217\u5316\u6210\u672c：<\/p>\n

T_pickle
\n<<
\nT_compute<\/p>\n

\u4ecd\u7136\u503c\u5f97\u4f7f\u7528\u591a\u8fdb\u7a0b\u3002<\/p>\n

\n

\u8d85\u5927\u6570\u636e\u3001\u9ad8\u9891\u901a\u4fe1<\/h4>\n

\u4f8b\u5982：<\/p>\n

\u6570\u767e MB NumPy \u6570\u7ec4
\n\u5927\u578b\u6a21\u578b\u53c2\u6570
\n\u89c6\u9891\u5e27
\n\u77e9\u9635<\/p>\n

\u5e94\u8be5\u8ba4\u771f\u8003\u8651：<\/p>\n

shared_memory
\nmmap
\nmemory-mapped file
\nworker local cache<\/p>\n

\u800c\u4e0d\u662f\u53cd\u590d pickle\u3002<\/p>\n

\n

\u4e8c\u5341\u516b\u3001\u4e00\u4e2a\u5b9e\u6218\u8bbe\u8ba1：\u6279\u91cf\u56fe\u7247\u5206\u6790\u7cfb\u7edf<\/h2>\n

\u5047\u8bbe\u9700\u8981\u5904\u7406：<\/p>\n

100000 \u5f20\u56fe\u7247<\/p>\n

\u9700\u6c42：<\/p>\n

\u8bfb\u53d6\u56fe\u7247
\n\u2193
\n\u7f29\u653e
\n\u2193
\n\u63d0\u53d6\u7279\u5f81
\n\u2193
\n\u8ba1\u7b97\u7edf\u8ba1\u4fe1\u606f
\n\u2193
\n\u4fdd\u5b58\u7ed3\u679c<\/p>\n

\u9519\u8bef\u67b6\u6784：<\/p>\n

Main Process<\/p>\n

\u8bfb\u53d6\u6240\u6709\u56fe\u7247\u5230\u5185\u5b58
\n \u2193
\n\u6bcf\u5f20\u56fe\u7247\u4f5c\u4e3a NumPy Array
\n \u2193
\npickle
\n \u2193
\nWorker<\/p>\n

\u95ee\u9898：<\/p>\n

\u5185\u5b58\u9ad8
\n\u5e8f\u5217\u5316\u91cd
\nIPC \u6570\u636e\u91cf\u5de8\u5927
\n\u4e3b\u8fdb\u7a0b\u5bb9\u6613\u6210\u4e3a\u74f6\u9888<\/p>\n

\u66f4\u52a0\u5408\u7406：<\/p>\n

Main Process<\/p>\n

\u626b\u63cf\u6587\u4ef6
\n \u2193
\n\u53ea\u83b7\u53d6\u8def\u5f84
\n \u2193<\/p>\n

image_001.jpg
\nimage_002.jpg
\nimage_003.jpg<\/p>\n

\u2193
\nProcessPool
\n \u2193<\/p>\n

Worker
\n \u2193
\n\u8bfb\u53d6\u56fe\u7247
\n \u2193
\n\u5904\u7406
\n \u2193
\n\u8fd4\u56de\u5c0f\u578b\u7edf\u8ba1\u7ed3\u679c<\/p>\n

\u4ee3\u7801：<\/p>\n

from<\/span> concurrent.<\/span>futures import<\/span> ProcessPoolExecutor<\/p>\n

def<\/span> analyze_image<\/span>(<\/span>path)<\/span>:<\/span><\/p>\n

image =<\/span> load_image(<\/span>path)<\/span><\/p>\n

features =<\/span> extract_features(<\/span>
\n image
\n )<\/span><\/p>\n

return<\/span> {<\/span>
\n "path"<\/span>:<\/span> path,<\/span>
\n "score"<\/span>:<\/span> features.<\/span>score,<\/span>
\n }<\/span><\/p>\n

if<\/span> __name__ ==<\/span> "__main__"<\/span>:<\/span><\/p>\n

with<\/span> ProcessPoolExecutor(<\/span>)<\/span> as<\/span> executor:<\/span><\/p>\n

results =<\/span> list<\/span>(<\/span>
\n executor.<\/span>map<\/span>(<\/span>
\n analyze_image,<\/span>
\n image_paths,<\/span>
\n )<\/span>
\n )<\/span><\/p>\n

\u901a\u4fe1\u4ece：<\/p>\n

\u5de8\u5927 image array<\/p>\n

\u53d8\u6210：<\/p>\n

\u51e0\u5341\u5b57\u8282 path<\/p>\n

\u8fd9\u5f80\u5f80\u80fd\u5e26\u6765\u975e\u5e38\u660e\u663e\u7684\u6027\u80fd\u63d0\u5347\u3002<\/p>\n

\n

\u4e8c\u5341\u4e5d\u3001\u518d\u770b\u4e00\u4e2a\u751f\u4ea7\u6848\u4f8b：\u65e5\u5fd7\u5206\u6790<\/h2>\n

\u5047\u8bbe\u6709：<\/p>\n

100 GB \u65e5\u5fd7\u6587\u4ef6<\/p>\n

\u4e0d\u8981\u8fd9\u6837：<\/p>\n

content =<\/span> open<\/span>(<\/span>
\n "server.log"<\/span>
\n)<\/span>.<\/span>read(<\/span>)<\/span><\/p>\n

pool.<\/span>map<\/span>(<\/span>
\n analyze,<\/span>
\n split(<\/span>content)<\/span>
\n)<\/span><\/p>\n

\u56e0\u4e3a：<\/p>\n

\u5927\u91cf\u5b57\u7b26\u4e32
\n\u2193
\n\u5e8f\u5217\u5316
\n\u2193
\n\u590d\u5236
\n\u2193
\n\u5927\u91cf\u5185\u5b58<\/p>\n

\u66f4\u597d\u7684\u65b9\u6848\u662f：<\/p>\n

Main Process<\/p>\n

\u751f\u6210：<\/p>\n

(file_path, start, end)<\/p>\n

\u2193<\/p>\n

Worker<\/p>\n

\u6253\u5f00\u6587\u4ef6
\n\u2193
\nseek(offset)
\n\u2193
\n\u8bfb\u53d6\u6307\u5b9a\u533a\u57df
\n\u2193
\n\u5206\u6790<\/p>\n

\u4e5f\u5c31\u662f\u8bf4：<\/p>\n

\u4f20\u5143\u6570\u636e
\n\u4e0d\u8981\u4f20\u6570\u636e\u672c\u4f53<\/p>\n

\u8fd9\u662f\u9ad8\u6027\u80fd\u5e76\u884c\u7a0b\u5e8f\u4e2d\u7684\u6838\u5fc3\u7406\u5ff5\u4e4b\u4e00\u3002<\/p>\n

\n

\u4e09\u5341\u3001\u4e3a\u4ec0\u4e48\u201c\u5c3d\u91cf\u5c11\u4f20\u5bf9\u8c61\u201d\u662f\u9ad8\u7ea7 multiprocessing \u6280\u5de7？<\/h2>\n

\u521d\u5b66 multiprocessing \u65f6，\u6211\u4eec\u5173\u6ce8：<\/p>\n

\u600e\u4e48\u521b\u5efa Process？<\/p>\n

\u8fdb\u4e00\u6b65\u5b66\u4e60：<\/p>\n

\u600e\u4e48\u7528 Queue？
\n\u600e\u4e48\u7528 Pool？<\/p>\n

\u518d\u5f80\u540e：<\/p>\n

\u600e\u4e48\u5171\u4eab\u6570\u636e？<\/p>\n

\u771f\u6b63\u8fdb\u5165\u6027\u80fd\u4f18\u5316\u9636\u6bb5\u540e，\u95ee\u9898\u5f80\u5f80\u4f1a\u53cd\u8fc7\u6765：<\/p>\n

\u6211\u80fd\u4e0d\u80fd\u6839\u672c\u4e0d\u4f20\u8fd9\u4e9b\u6570\u636e？<\/p>\n

\u8fd9\u662f\u4e00\u79cd\u975e\u5e38\u91cd\u8981\u7684\u601d\u7ef4\u5347\u7ea7\u3002<\/p>\n

\u56e0\u4e3a\u8ba1\u7b97\u673a\u7cfb\u7edf\u4e2d\u6700\u6602\u8d35\u7684\u4e8b\u60c5\u4e4b\u4e00\u901a\u5e38\u4e0d\u662f：<\/p>\n

\u8ba1\u7b97<\/p>\n

\u800c\u662f：<\/p>\n

\u6570\u636e\u79fb\u52a8<\/p>\n

\u4f8b\u5982：<\/p>\n

\u78c1\u76d8 \u2192 \u5185\u5b58
\n\u5185\u5b58 \u2192 CPU Cache
\n\u8fdb\u7a0b A \u2192 \u8fdb\u7a0b B
\n\u673a\u5668 A \u2192 \u673a\u5668 B
\nCPU \u2192 GPU<\/p>\n

\u6240\u4ee5\u9ad8\u7ea7\u6027\u80fd\u4f18\u5316\u7ecf\u5e38\u9075\u5faa：<\/p>\n

Move computation to data<\/p>\n

\u800c\u4e0d\u662f：<\/p>\n

Move huge data to computation<\/p>\n

\n

\u4e09\u5341\u4e00\u3001\u4e00\u4e2a\u5b9e\u7528\u7684 multiprocessing \u6027\u80fd\u516c\u5f0f<\/h2>\n

\u8bc4\u4f30\u4e00\u4e2a\u591a\u8fdb\u7a0b\u4efb\u52a1\u662f\u5426\u503c\u5f97\u5e76\u884c，\u53ef\u4ee5\u7c97\u7565\u8003\u8651：<\/p>\n

T_parallel
\n=
\nT_startup
\n+
\nT_pickle_input
\n+
\nT_transfer_input
\n+
\nT_compute \/ N
\n+
\nT_pickle_output
\n+
\nT_transfer_output
\n+
\nT_sync<\/p>\n

\u5176\u4e2d：<\/p>\n

N<\/p>\n

\u662f worker \u6570\u91cf\u3002<\/p>\n

\u7406\u60f3\u60c5\u51b5\u4e0b\u5927\u5bb6\u53ea\u5173\u6ce8：<\/p>\n

T_compute \/ N<\/p>\n

\u4e8e\u662f\u4ee5\u4e3a：<\/p>\n

8 \u6838
\n\u2248
\n8 \u500d\u6027\u80fd<\/p>\n

\u5b9e\u9645\u4e0a：<\/p>\n

startup
\nserialization
\ncommunication
\nsynchronization<\/p>\n

\u90fd\u4f1a\u6d88\u8017\u65f6\u95f4\u3002<\/p>\n

\u56e0\u6b64\u591a\u8fdb\u7a0b\u6536\u76ca\u901a\u5e38\u6ee1\u8db3：<\/p>\n

\u8ba1\u7b97\u8d8a\u91cd
\n\u4efb\u52a1\u8d8a\u72ec\u7acb
\n\u901a\u4fe1\u8d8a\u5c11
\n\u5e76\u884c\u6536\u76ca\u8d8a\u660e\u663e<\/p>\n

\u53cd\u8fc7\u6765：<\/p>\n

\u4efb\u52a1\u8d8a\u5c0f
\n\u5bf9\u8c61\u8d8a\u5927
\n\u901a\u4fe1\u8d8a\u9891\u7e41
\n\u591a\u8fdb\u7a0b\u8d8a\u5bb9\u6613\u53d8\u6162<\/p>\n

\n

\u4e09\u5341\u4e8c\u3001ProcessPoolExecutor \u7684\u4e00\u4e2a\u7ecf\u5178\u8bef\u533a<\/h2>\n

\u5047\u8bbe：<\/p>\n

def<\/span> add_one<\/span>(<\/span>x)<\/span>:<\/span>
\n return<\/span> x +<\/span> 1<\/span><\/p>\n

\u7136\u540e：<\/p>\n

with<\/span> ProcessPoolExecutor(<\/span>)<\/span> as<\/span> executor:<\/span><\/p>\n

results =<\/span> list<\/span>(<\/span>
\n executor.<\/span>map<\/span>(<\/span>
\n add_one,<\/span>
\n range<\/span>(<\/span>10_000_000<\/span>)<\/span>
\n )<\/span>
\n )<\/span><\/p>\n

\u5355\u6b21：<\/p>\n

x +<\/span> 1<\/span><\/p>\n

\u51e0\u4e4e\u6ca1\u6709\u8ba1\u7b97\u91cf\u3002<\/p>\n

\u4f46\u6bcf\u4e2a\u4efb\u52a1\u53ef\u80fd\u6d89\u53ca：<\/p>\n

\u8c03\u5ea6
\n\u5e8f\u5217\u5316
\n\u4f20\u8f93
\n\u53cd\u5e8f\u5217\u5316
\n\u8ba1\u7b97
\n\u8fd4\u56de
\n\u5e8f\u5217\u5316
\n\u518d\u4f20\u8f93<\/p>\n

\u8fd9\u7c7b\u4efb\u52a1\u53ef\u80fd\u6bd4\u666e\u901a：<\/p>\n

[<\/span>x +<\/span> 1<\/span> for<\/span> x in<\/span> data]<\/span><\/p>\n

\u6162\u5f88\u591a\u3002<\/p>\n

\u56e0\u6b64：<\/p>\n

\u591a\u8fdb\u7a0b\u9002\u5408\u201c\u91cd\u4efb\u52a1\u201d，\u4e0d\u662f\u201c\u4efb\u52a1\u6570\u91cf\u5f88\u591a\u201d\u5c31\u4e00\u5b9a\u9002\u5408\u591a\u8fdb\u7a0b\u3002<\/p>\n

\n

\u4e09\u5341\u4e09\u3001\u5982\u4f55\u5224\u65ad\u4efb\u52a1\u7c92\u5ea6\u662f\u5426\u5408\u7406？<\/h2>\n

\u53ef\u4ee5\u505a\u4e00\u4e2a\u7b80\u5355\u5224\u65ad\u3002<\/p>\n

\u5047\u8bbe\u5355\u4e2a\u4efb\u52a1：<\/p>\n

\u8ba1\u7b97 = 500 ms
\n\u901a\u4fe1 = 2 ms<\/p>\n

\u90a3\u4e48：<\/p>\n

\u901a\u4fe1\u6210\u672c\u5360\u6bd4\u5f88\u4f4e<\/p>\n

\u975e\u5e38\u9002\u5408\u591a\u8fdb\u7a0b\u3002<\/p>\n

\u5982\u679c：<\/p>\n

\u8ba1\u7b97 = 0.05 ms
\n\u901a\u4fe1 = 1 ms<\/p>\n

\u90a3\u4e48：<\/p>\n

\u901a\u4fe1\u6bd4\u8ba1\u7b97\u91cd 20 \u500d<\/p>\n

\u8fd9\u79cd\u65f6\u5019\u5e76\u884c\u5f80\u5f80\u5f97\u4e0d\u507f\u5931\u3002<\/p>\n

\u6240\u4ee5\u5b9e\u8df5\u4e2d\u5e94\u8be5\u6d4b：<\/p>\n

import<\/span> time<\/p>\n

start =<\/span> time.<\/span>perf_counter(<\/span>)<\/span><\/p>\n

result =<\/span> compute(<\/span>task)<\/span><\/p>\n

elapsed =<\/span> time.<\/span>perf_counter(<\/span>)<\/span> –<\/span> start<\/p>\n

print<\/span>(<\/span>elapsed)<\/span><\/p>\n

\u4e0d\u8981\u53ea\u51ed\u611f\u89c9\u51b3\u5b9a：<\/p>\n

\u6211\u8981\u4e0d\u8981 multiprocessing<\/p>\n

\n

\u4e09\u5341\u56db\u3001pickle \u4e0e JSON \u6709\u4ec0\u4e48\u533a\u522b？<\/h2>\n

\u5f88\u591a\u8bfb\u8005\u4f1a\u95ee：<\/p>\n

\u65e2\u7136\u90fd\u662f\u5e8f\u5217\u5316，\u4e3a\u4ec0\u4e48 multiprocessing \u4e0d\u76f4\u63a5\u7528 JSON？<\/p>\n

\u56e0\u4e3a JSON \u80fd\u8868\u8fbe\u7684\u6570\u636e\u7c7b\u578b\u6709\u9650：<\/p>\n

string
\nnumber
\nboolean
\nnull
\narray
\nobject<\/p>\n

\u4f46\u662f Python \u591a\u8fdb\u7a0b\u53ef\u80fd\u9700\u8981\u4f20\u9012：<\/p>\n

tuple
\nset
\nbytes
\n\u81ea\u5b9a\u4e49 class
\n\u51fd\u6570\u5f15\u7528
\n\u590d\u6742 Python \u72b6\u6001<\/p>\n

\u4f8b\u5982：<\/p>\n

data =<\/span> {<\/span>
\n "point"<\/span>:<\/span> (<\/span>10<\/span>,<\/span> 20<\/span>)<\/span>,<\/span>
\n "tags"<\/span>:<\/span> {<\/span>"python"<\/span>,<\/span> "linux"<\/span>}<\/span>,<\/span>
\n}<\/span><\/p>\n

pickle \u66f4\u4e86\u89e3：<\/p>\n

Python object model<\/p>\n

\u6240\u4ee5\u975e\u5e38\u9002\u5408：<\/p>\n

Python \u2192 Python<\/p>\n

\u4f46\u6709\u4e00\u4e2a\u5fc5\u987b\u77e5\u9053\u7684\u91cd\u8981\u539f\u5219：<\/p>\n

\u4e0d\u8981\u5bf9\u4e0d\u53ef\u4fe1\u6765\u6e90\u7684\u6570\u636e\u6267\u884c pickle.loads()\u3002<\/p>\n

\u56e0\u4e3a pickle \u7684\u53cd\u5e8f\u5217\u5316\u8fc7\u7a0b\u53ef\u80fd\u6267\u884c\u5371\u9669\u4ee3\u7801\u3002<\/p>\n

\u56e0\u6b64：<\/p>\n

pickle<\/p>\n

\u9002\u5408：<\/p>\n

\u53d7\u4fe1\u4efb\u7684 Python \u5185\u90e8\u901a\u4fe1<\/p>\n

\u5374\u4e0d\u5e94\u8be5\u76f4\u63a5\u7528\u4e8e：<\/p>\n

\u5904\u7406\u4e92\u8054\u7f51\u4e0a\u964c\u751f\u7528\u6237\u63d0\u4ea4\u7684\u6570\u636e<\/p>\n

\n

\u4e09\u5341\u4e94\u3001\u5982\u679c\u5bf9\u8c61\u4e0d\u80fd pickle，\u5e94\u8be5\u600e\u4e48\u529e？<\/h2>\n

\u9047\u5230：<\/p>\n

Can't pickle …<\/p>\n

\u4e0d\u8981\u7b2c\u4e00\u65f6\u95f4\u60f3\u529e\u6cd5\u201c\u5f3a\u884c\u5e8f\u5217\u5316\u201d\u3002<\/p>\n

\u5148\u5224\u65ad\u5bf9\u8c61\u662f\u4ec0\u4e48\u3002<\/p>\n

\u5982\u679c\u662f：<\/p>\n

\u6570\u636e\u5e93\u8fde\u63a5<\/p>\n

\u4f20\u8fde\u63a5\u53c2\u6570\u3002<\/p>\n

\u5982\u679c\u662f：<\/p>\n

\u6587\u4ef6\u5bf9\u8c61<\/p>\n

\u4f20\u6587\u4ef6\u8def\u5f84\u3002<\/p>\n

\u5982\u679c\u662f：<\/p>\n

\u7f51\u7edc\u8fde\u63a5<\/p>\n

\u8ba9 worker \u81ea\u5df1\u5efa\u7acb\u8fde\u63a5\u3002<\/p>\n

\u5982\u679c\u662f：<\/p>\n

\u7ebf\u7a0b\u9501<\/p>\n

\u91cd\u65b0\u8bbe\u8ba1\u540c\u6b65\u65b9\u5f0f\u3002<\/p>\n

\u5982\u679c\u662f：<\/p>\n

\u5927\u578b\u6570\u636e<\/p>\n

\u8003\u8651\u5171\u4eab\u5185\u5b58\u3002<\/p>\n

\u5982\u679c\u662f：<\/p>\n

\u5c40\u90e8\u51fd\u6570<\/p>\n

\u79fb\u5230\u6a21\u5757\u9876\u5c42\u3002<\/p>\n

\u8fd9\u5957\u601d\u8def\u975e\u5e38\u503c\u5f97\u8bb0\u4f4f：<\/p>\n

\u4e0d\u53ef pickle
\n \u2193
\n\u5148\u95ee：
\n\u201c\u5b83\u5e94\u8be5\u88ab\u4f20\u5417？\u201d<\/p>\n

\u5f88\u591a\u65f6\u5019\u7b54\u6848\u5176\u5b9e\u662f：<\/p>\n

\u4e0d\u5e94\u8be5\u3002<\/p>\n

\n

\u4e09\u5341\u516d\u3001\u5982\u4f55\u8bbe\u8ba1\u4e00\u4e2a\u201c\u9002\u5408 multiprocessing\u201d\u7684\u5bf9\u8c61？<\/h2>\n

\u5047\u8bbe：<\/p>\n

class<\/span> ImageTask<\/span>:<\/span><\/p>\n

def<\/span> __init__<\/span>(<\/span>
\n self,<\/span>
\n image_path,<\/span>
\n model_name,<\/span>
\n )<\/span>:<\/span><\/p>\n

self.<\/span>image_path =<\/span> image_path
\n self.<\/span>model_name =<\/span> model_name<\/p>\n

\u8fd9\u6837\u7684\u5bf9\u8c61\u975e\u5e38\u9002\u5408：<\/p>\n

\u5e8f\u5217\u5316<\/p>\n

\u56e0\u4e3a\u91cc\u9762\u90fd\u662f：<\/p>\n

\u666e\u901a\u5b57\u7b26\u4e32<\/p>\n

\u4f46\u5982\u679c\u8bbe\u8ba1\u6210：<\/p>\n

class<\/span> ImageTask<\/span>:<\/span><\/p>\n

def<\/span> __init__<\/span>(<\/span>
\n self,<\/span>
\n image_path,<\/span>
\n )<\/span>:<\/span><\/p>\n

self.<\/span>image_path =<\/span> image_path<\/p>\n

self.<\/span>file<\/span> =<\/span> open<\/span>(<\/span>
\n image_path,<\/span>
\n "rb"<\/span>
\n )<\/span><\/p>\n

self.<\/span>db =<\/span> create_connection(<\/span>)<\/span><\/p>\n

self.<\/span>lock =<\/span> threading.<\/span>Lock(<\/span>)<\/span><\/p>\n

\u5c31\u975e\u5e38\u4e0d\u9002\u5408\u8de8\u8fdb\u7a0b\u4f20\u9012\u3002<\/p>\n

\u56e0\u6b64\u6570\u636e\u7c7b\u6700\u597d\u627f\u62c5：<\/p>\n

\u63cf\u8ff0\u4efb\u52a1<\/p>\n

\u800c\u4e0d\u662f：<\/p>\n

\u643a\u5e26\u5168\u90e8\u8fd0\u884c\u65f6\u8d44\u6e90<\/p>\n

\u4f8b\u5982：<\/p>\n

from<\/span> dataclasses import<\/span> dataclass<\/p>\n

@dataclass<\/span>
\nclass<\/span> ImageTask<\/span>:<\/span><\/p>\n

image_path:<\/span> str<\/span><\/p>\n

resize_width:<\/span> int<\/span><\/p>\n

resize_height:<\/span> int<\/span><\/p>\n

\u975e\u5e38\u6e05\u6670\u3002<\/p>\n

Worker：<\/p>\n

def<\/span> process<\/span>(<\/span>task)<\/span>:<\/span><\/p>\n

image =<\/span> load_image(<\/span>
\n task.<\/span>image_path
\n )<\/span><\/p>\n

.<\/span>.<\/span>.<\/span><\/p>\n

\u8fd9\u5c31\u662f\u826f\u597d\u7684：<\/p>\n

data-oriented design<\/p>\n

\n

\u4e09\u5341\u4e03\u3001\u591a\u8fdb\u7a0b\u7a0b\u5e8f\u7684\u63a8\u8350\u5206\u5c42\u7ed3\u6784<\/h2>\n

\u771f\u5b9e\u9879\u76ee\u4e2d，\u53ef\u4ee5\u8bbe\u8ba1\u6210：<\/p>\n

\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
\n\u2502 Main Process \u2502
\n\u2502 \u2502
\n\u2502 \u4efb\u52a1\u53d1\u73b0 \/ \u8c03\u5ea6 \u2502
\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
\n \u2502
\n \u2502 small task
\n \u2193
\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
\n\u2502 Worker Process \u2502
\n\u2502 \u2502
\n\u2502 \u521d\u59cb\u5316\u672c\u5730\u8d44\u6e90 \u2502
\n\u2502 \u6267\u884c CPU \u4efb\u52a1 \u2502
\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
\n \u2502
\n \u2502 small result
\n \u2193
\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
\n\u2502 Main Process \u2502
\n\u2502 \u2502
\n\u2502 \u805a\u5408 \/ \u4fdd\u5b58\u7ed3\u679c \u2502
\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n

\u7406\u60f3\u901a\u4fe1\u5bf9\u8c61\u5e94\u8be5：<\/p>\n

\u5c0f
\n\u7b80\u5355
\n\u4e0d\u53ef\u53d8\u4f18\u5148
\n\u6613\u5e8f\u5217\u5316
\n\u4e0e\u8fd0\u884c\u73af\u5883\u89e3\u8026<\/p>\n

\u8fd9\u5957\u8bbe\u8ba1\u4e0d\u4ec5\u9002\u5408 multiprocessing，\u4e5f\u9002\u5408：<\/p>\n

Celery
\nRay
\nSpark
\n\u6d88\u606f\u961f\u5217
\n\u5fae\u670d\u52a1
\n\u5206\u5e03\u5f0f\u8ba1\u7b97<\/p>\n

\n

\u4e09\u5341\u516b\u3001\u8fdb\u7a0b\u95f4\u901a\u4fe1\u7684\u6700\u4f73\u5b9e\u8df5\u603b\u7ed3<\/h2>\n

\u771f\u5b9e Python \u9879\u76ee\u91cc，\u53ef\u4ee5\u9075\u5faa\u4ee5\u4e0b\u539f\u5219：<\/p>\n

1. \u9ed8\u8ba4\u8ba4\u4e3a\u4e0d\u540c\u8fdb\u7a0b\u5185\u5b58\u5f7c\u6b64\u9694\u79bb<\/p>\n

2. \u4e0d\u4f9d\u8d56\u666e\u901a\u5168\u5c40\u53d8\u91cf\u5171\u4eab\u72b6\u6001<\/p>\n

3. Queue \/ Pipe \/ Pool \u4e2d\u4f20\u9012\u5bf9\u8c61\u65f6，
\n \u8981\u610f\u8bc6\u5230\u5e8f\u5217\u5316\u6210\u672c<\/p>\n

4. worker \u5b9a\u4e49\u5728\u6a21\u5757\u9876\u5c42<\/p>\n

5. \u53c2\u6570\u5c3d\u91cf\u7531\u7b80\u5355\u6570\u636e\u7ec4\u6210<\/p>\n

6. \u4e0d\u4f20\u6570\u636e\u5e93\u8fde\u63a5\u3001socket\u3001\u7ebf\u7a0b\u9501\u7b49\u8d44\u6e90<\/p>\n

7. \u5927\u8d44\u6e90\u4f18\u5148\u8ba9 worker \u81ea\u5df1\u521d\u59cb\u5316<\/p>\n

8. \u5de8\u5927\u5bf9\u8c61\u8003\u8651\u5171\u4eab\u5185\u5b58<\/p>\n

9. \u9ad8\u9891\u5c0f\u4efb\u52a1\u8003\u8651\u6279\u5904\u7406<\/p>\n

10. \u4f7f\u7528 Pool.map \u65f6\u5173\u6ce8 chunksize<\/p>\n

11. \u4e0d\u8981\u628a\u201c\u591a\u8fdb\u7a0b\u201d\u7406\u89e3\u6210\u514d\u8d39\u6027\u80fd<\/p>\n

12. \u4f18\u5316\u65f6\u91cd\u70b9\u5173\u6ce8\u6570\u636e\u79fb\u52a8\u6210\u672c<\/p>\n

\u5982\u679c\u53ea\u8bb0\u4f4f\u4e00\u53e5：<\/p>\n

\u8fdb\u7a0b\u4e4b\u95f4\u4f20\u9012\u7684\u6700\u597d\u662f\u201c\u4efb\u52a1\u63cf\u8ff0\u201d，\u800c\u4e0d\u662f\u5b8c\u6574\u8fd0\u884c\u72b6\u6001\u3002<\/p>\n

\n

\u4e09\u5341\u4e5d\u3001\u9762\u8bd5\u4e2d\u5982\u4f55\u56de\u7b54\u201c\u4e3a\u4ec0\u4e48\u591a\u8fdb\u7a0b\u5bf9\u8c61\u4f20\u9012\u9700\u8981\u5e8f\u5217\u5316？\u201d<\/h2>\n

\u5982\u679c\u9762\u8bd5\u5b98\u95ee：<\/p>\n

Python multiprocessing \u4e3a\u4ec0\u4e48\u5bf9\u8c61\u4f20\u9012\u9700\u8981\u5e8f\u5217\u5316？<\/p>\n

\u53ef\u4ee5\u8fd9\u6837\u56de\u7b54：<\/p>\n

\u56e0\u4e3a\u4e0d\u540c\u8fdb\u7a0b\u62e5\u6709\u5f7c\u6b64\u72ec\u7acb\u7684\u865a\u62df\u5730\u5740\u7a7a\u95f4\u3002\u7236\u8fdb\u7a0b\u4e2d\u7684 Python \u5bf9\u8c61\u5f15\u7528\u6216\u5185\u5b58\u5730\u5740，\u5bf9\u53e6\u4e00\u4e2a\u8fdb\u7a0b\u901a\u5e38\u6ca1\u6709\u76f4\u63a5\u610f\u4e49\u3002\u4e3a\u4e86\u901a\u8fc7 Pipe\u3001Queue \u6216\u5176\u4ed6 IPC \u673a\u5236\u628a\u5bf9\u8c61\u4fe1\u606f\u53d1\u9001\u7ed9\u5176\u4ed6\u8fdb\u7a0b，\u9700\u8981\u5148\u628a\u5bf9\u8c61\u8f6c\u6362\u6210\u53ef\u4ee5\u4f20\u8f93\u7684\u5b57\u8282\u8868\u793a，\u518d\u7531\u76ee\u6807\u8fdb\u7a0b\u6839\u636e\u8fd9\u4e9b\u6570\u636e\u91cd\u65b0\u6784\u5efa\u5bf9\u8c61，\u8fd9\u4e2a\u8fc7\u7a0b\u5c31\u662f\u5e8f\u5217\u5316\u548c\u53cd\u5e8f\u5217\u5316\u3002Python multiprocessing \u4e2d\u5f88\u591a\u573a\u666f\u4f7f\u7528 pickle \u5b8c\u6210\u8fd9\u4e00\u8fc7\u7a0b，\u56e0\u6b64\u5bf9\u8c61\u901a\u5e38\u9700\u8981\u5177\u5907\u53ef pickle \u6027\u3002<\/p>\n

\u5982\u679c\u9762\u8bd5\u5b98\u7ee7\u7eed\u8ffd\u95ee\u6027\u80fd\u95ee\u9898，\u53ef\u4ee5\u8865\u5145：<\/p>\n

\u5e8f\u5217\u5316\u4e0d\u662f\u514d\u8d39\u7684\u3002\u5927\u578b Python \u5bf9\u8c61\u4f1a\u5e26\u6765 pickle\u3001\u5185\u5b58\u590d\u5236\u548c IPC \u6210\u672c，\u56e0\u6b64\u591a\u8fdb\u7a0b\u6027\u80fd\u4e0d\u4ec5\u53d6\u51b3\u4e8e\u8ba1\u7b97\u901f\u5ea6，\u8fd8\u53d6\u51b3\u4e8e\u4efb\u52a1\u7c92\u5ea6\u548c\u6570\u636e\u4f20\u8f93\u91cf\u3002\u5927\u91cf\u6570\u636e\u573a\u666f\u901a\u5e38\u5e94\u8be5\u8003\u8651\u5171\u4eab\u5185\u5b58\u3001\u6279\u5904\u7406\u6216\u8005\u8ba9 worker \u81ea\u5df1\u52a0\u8f7d\u8d44\u6e90\u3002<\/p>\n

\u5982\u679c\u518d\u8ffd\u95ee：<\/p>\n

\u4e3a\u4ec0\u4e48\u7ebf\u7a0b\u4e0d\u9700\u8981？<\/p>\n

\u56de\u7b54：<\/p>\n

\u56e0\u4e3a\u540c\u4e00\u8fdb\u7a0b\u5185\u7684\u7ebf\u7a0b\u5171\u4eab\u865a\u62df\u5730\u5740\u7a7a\u95f4，\u53ef\u4ee5\u76f4\u63a5\u8bbf\u95ee\u540c\u4e00\u4e2a Python \u5bf9\u8c61，\u800c\u72ec\u7acb\u8fdb\u7a0b\u4e4b\u95f4\u9ed8\u8ba4\u6ca1\u6709\u8fd9\u79cd\u5171\u4eab\u5173\u7cfb\u3002<\/p>\n

\u8fd9\u4e09\u5c42\u56de\u7b54\u57fa\u672c\u5df2\u7ecf\u8986\u76d6：<\/p>\n

\u64cd\u4f5c\u7cfb\u7edf
\nPython \u5bf9\u8c61\u6a21\u578b
\nIPC
\npickle
\n\u6027\u80fd\u4f18\u5316<\/p>\n

\u51e0\u4e2a\u5173\u952e\u77e5\u8bc6\u70b9\u3002<\/p>\n

\n

\u56db\u5341\u3001\u4ece multiprocessing \u8fdb\u4e00\u6b65\u7406\u89e3\u5206\u5e03\u5f0f\u7cfb\u7edf<\/h2>\n

\u7406\u89e3\u5b8c\u591a\u8fdb\u7a0b\u5e8f\u5217\u5316，\u4f60\u5176\u5b9e\u5df2\u7ecf\u8fc8\u8fdb\u4e86\u5206\u5e03\u5f0f\u8ba1\u7b97\u7684\u5927\u95e8\u3002<\/p>\n

\u5355\u673a\u591a\u8fdb\u7a0b：<\/p>\n

Process A
\n \u2193
\nserialize
\n \u2193
\nIPC
\n \u2193
\nProcess B<\/p>\n

\u8de8\u673a\u5668\u670d\u52a1：<\/p>\n

Machine A
\n \u2193
\nserialize
\n \u2193
\nnetwork
\n \u2193
\nMachine B<\/p>\n

\u672c\u8d28\u7ed3\u6784\u975e\u5e38\u63a5\u8fd1\u3002<\/p>\n

\u53ea\u662f\u534f\u8bae\u53ef\u80fd\u4ece：<\/p>\n

pickle<\/p>\n

\u53d8\u6210：<\/p>\n

JSON
\nProtobuf
\nMessagePack
\nAvro<\/p>\n

\u901a\u4fe1\u65b9\u5f0f\u4ece：<\/p>\n

Pipe<\/p>\n

\u53d8\u6210：<\/p>\n

TCP
\nHTTP
\ngRPC
\nKafka
\nRabbitMQ<\/p>\n

\u6240\u4ee5\u5f53\u4f60\u771f\u6b63\u7406\u89e3：<\/p>\n

\u4e3a\u4ec0\u4e48\u4e0d\u80fd\u628a Python \u5bf9\u8c61\u76f4\u63a5\u4f20\u7ed9\u53e6\u4e00\u4e2a\u8fdb\u7a0b<\/p>\n

\u4e5f\u5c31\u5f88\u5bb9\u6613\u7406\u89e3：<\/p>\n

\u4e3a\u4ec0\u4e48\u5fae\u670d\u52a1\u63a5\u53e3\u9700\u8981\u5b9a\u4e49\u6570\u636e\u7ed3\u6784？<\/p>\n

\u4e3a\u4ec0\u4e48 RPC \u9700\u8981\u5e8f\u5217\u5316？<\/p>\n

\u4e3a\u4ec0\u4e48 Kafka \u6d88\u606f\u4e0d\u80fd\u76f4\u63a5\u4fdd\u5b58 Python \u5185\u5b58\u5bf9\u8c61？<\/p>\n

\u4e3a\u4ec0\u4e48\u5206\u5e03\u5f0f\u7cfb\u7edf\u7279\u522b\u5f3a\u8c03\u6570\u636e\u534f\u8bae？<\/p>\n

\u7b54\u6848\u90fd\u662f\u7c7b\u4f3c\u7684：<\/p>\n

\u8fd0\u884c\u65f6\u5bf9\u8c61\u53ea\u5c5e\u4e8e\u81ea\u5df1\u7684\u8fd0\u884c\u73af\u5883，
\n\u8de8\u8d8a\u8fb9\u754c\u9700\u8981\u660e\u786e\u7684\u6570\u636e\u8868\u793a\u3002<\/p>\n

\n

\u603b\u7ed3：\u5e8f\u5217\u5316\u4e0d\u662f\u989d\u5916\u8d1f\u62c5，\u800c\u662f\u8fdb\u7a0b\u9694\u79bb\u4e4b\u540e\u5fc5\u7136\u51fa\u73b0\u7684\u6865\u6881<\/h2>\n

Python \u591a\u8fdb\u7a0b\u4e2d\u7684\u5bf9\u8c61\u4f20\u9012，\u8868\u9762\u4e0a\u770b\u4f3c：<\/p>\n

worker(<\/span>data)<\/span><\/p>\n

\u5b9e\u9645\u4e0a\u80cc\u540e\u53ef\u80fd\u7ecf\u5386：<\/p>\n

Python Object
\n \u2193
\nSerialization
\n \u2193
\nBytes
\n \u2193
\nIPC
\n \u2193
\nBytes
\n \u2193
\nDeserialization
\n \u2193
\nNew Python Object<\/p>\n

\u4e3a\u4ec0\u4e48\u5fc5\u987b\u8fd9\u4e48\u505a？<\/p>\n

\u6839\u672c\u539f\u56e0\u53ea\u6709\u4e00\u4e2a：<\/p>\n

\u4e0d\u540c\u8fdb\u7a0b\u62e5\u6709\u72ec\u7acb\u7684\u5730\u5740\u7a7a\u95f4，\u4e00\u4e2a\u8fdb\u7a0b\u4e2d\u7684 Python \u5bf9\u8c61\u65e0\u6cd5\u5929\u7136\u6210\u4e3a\u53e6\u4e00\u4e2a\u8fdb\u7a0b\u4e2d\u7684\u540c\u4e00\u4e2a Python \u5bf9\u8c61\u3002<\/p>\n

\u5e8f\u5217\u5316\u627f\u62c5\u7684\u89d2\u8272，\u5c31\u662f\u628a：<\/p>\n

\u8fd0\u884c\u65f6\u5bf9\u8c61<\/p>\n

\u8f6c\u6362\u6210：<\/p>\n

\u53ef\u4f20\u8f93\u6570\u636e<\/p>\n

\u7136\u540e\u518d：<\/p>\n

\u91cd\u5efa\u5bf9\u8c61<\/p>\n

\u7406\u89e3\u8fd9\u4e2a\u673a\u5236\u4e4b\u540e，\u5f88\u591a Python \u591a\u8fdb\u7a0b\u4e2d\u7684\u73b0\u8c61\u90fd\u4f1a\u53d8\u5f97\u975e\u5e38\u81ea\u7136：<\/p>\n

\u4e3a\u4ec0\u4e48\u666e\u901a\u5168\u5c40\u53d8\u91cf\u4e0d\u80fd\u76f4\u63a5\u5171\u4eab？<\/p>\n

\u4e3a\u4ec0\u4e48 Queue \u91cc\u7684\u5bf9\u8c61\u88ab\u4fee\u6539\u540e\u4e0d\u4f1a\u5f71\u54cd\u7236\u8fdb\u7a0b？<\/p>\n

\u4e3a\u4ec0\u4e48 lambda \u548c\u5c40\u90e8\u51fd\u6570\u7ecf\u5e38\u62a5 pickle \u9519\u8bef？<\/p>\n

\u4e3a\u4ec0\u4e48\u6570\u636e\u5e93\u8fde\u63a5\u4e0d\u5e94\u8be5\u76f4\u63a5\u4f20\u7ed9 worker？<\/p>\n

\u4e3a\u4ec0\u4e48\u5927\u578b NumPy \u6570\u7ec4\u4f1a\u62d6\u6162 ProcessPool？<\/p>\n

\u4e3a\u4ec0\u4e48\u5171\u4eab\u5185\u5b58\u80fd\u63d0\u5347\u67d0\u4e9b\u6570\u636e\u5bc6\u96c6\u578b\u4efb\u52a1\u7684\u6027\u80fd？<\/p>\n

\u5b83\u4eec\u5176\u5b9e\u90fd\u662f\u540c\u4e00\u4e2a\u6838\u5fc3\u95ee\u9898\u7684\u4e0d\u540c\u8868\u73b0：<\/p>\n

\u8fdb\u7a0b\u8fb9\u754c
\n+
\n\u5bf9\u8c61\u8868\u793a
\n+
\n\u6570\u636e\u79fb\u52a8\u6210\u672c<\/p>\n

\u800c\u771f\u6b63\u6210\u719f\u7684 Python \u591a\u8fdb\u7a0b\u8bbe\u8ba1，\u5e76\u4e0d\u662f\u60f3\u529e\u6cd5\u201c\u628a\u6240\u6709\u4e1c\u897f\u90fd\u4f20\u7ed9 worker\u201d\u3002<\/p>\n

\u6070\u6070\u76f8\u53cd，\u4f18\u79c0\u7684\u7cfb\u7edf\u4f1a\u4e0d\u65ad\u601d\u8003：<\/p>\n

\u54ea\u4e9b\u6570\u636e\u771f\u7684\u5fc5\u987b\u4f20？<\/p>\n

\u80fd\u4e0d\u80fd\u53ea\u4f20 ID？<\/p>\n

\u80fd\u4e0d\u80fd\u53ea\u4f20\u8def\u5f84？<\/p>\n

\u80fd\u4e0d\u80fd\u6279\u91cf\u4f20？<\/p>\n

\u80fd\u4e0d\u80fd\u8ba9 worker \u81ea\u5df1\u52a0\u8f7d？<\/p>\n

\u80fd\u4e0d\u80fd\u4f7f\u7528\u5171\u4eab\u5185\u5b58？<\/p>\n

\u80fd\u4e0d\u80fd\u628a\u8ba1\u7b97\u79fb\u52a8\u5230\u6570\u636e\u9644\u8fd1？<\/p>\n

\u5f53\u4f60\u7684\u5173\u6ce8\u70b9\u4ece：<\/p>\n

\u201c\u5982\u4f55\u521b\u5efa\u66f4\u591a\u8fdb\u7a0b？\u201d<\/p>\n

\u8f6c\u53d8\u4e3a：<\/p>\n

\u201c\u5982\u4f55\u51cf\u5c11\u8fdb\u7a0b\u4e4b\u95f4\u7684\u6570\u636e\u79fb\u52a8？\u201d<\/p>\n

\u4f60\u5c31\u5df2\u7ecf\u4ece\u4f1a\u4f7f\u7528 multiprocessing，\u8d70\u5411\u771f\u6b63\u7406\u89e3\u9ad8\u6027\u80fd Python \u7f16\u7a0b\u4e86\u3002<\/p>\n

\n

\u5199\u5728\u6700\u540e<\/h3>\n

Python \u7684\u9b45\u529b\u5e38\u5e38\u5c31\u5728\u8fd9\u4e9b\u770b\u4f3c\u7b80\u5355\u7684 API \u80cc\u540e\u3002<\/p>\n

\u4e00\u53e5：<\/p>\n

pool.<\/span>submit(<\/span>worker,<\/span> data)<\/span><\/p>\n

\u80cc\u540e\u8fde\u63a5\u7740：<\/p>\n

Python \u5bf9\u8c61\u6a21\u578b
\n\u64cd\u4f5c\u7cfb\u7edf\u865a\u62df\u5185\u5b58
\n\u8fdb\u7a0b\u9694\u79bb
\nIPC
\npickle
\n\u5171\u4eab\u5185\u5b58
\n\u5e76\u884c\u8ba1\u7b97
\n\u5206\u5e03\u5f0f\u7cfb\u7edf<\/p>\n

\u5982\u679c\u4f60\u613f\u610f\u987a\u7740\u4e00\u4e2a\u5c0f\u95ee\u9898\u4e0d\u65ad\u5f80\u5e95\u5c42\u8ffd\u95ee，\u4f1a\u53d1\u73b0\u5f88\u591a\u9ad8\u7ea7\u77e5\u8bc6\u5176\u5b9e\u5e76\u6ca1\u6709\u60f3\u8c61\u4e2d\u90a3\u4e48\u5272\u88c2\u3002<\/p>\n

\u5b83\u4eec\u5f80\u5f80\u53ea\u662f\u540c\u4e00\u4e2a\u57fa\u672c\u539f\u7406，\u5728\u4e0d\u540c\u5c42\u6b21\u4e0a\u7684\u91cd\u590d\u51fa\u73b0\u3002<\/p>\n

\u6700\u540e\u4e5f\u7559\u4e24\u4e2a\u95ee\u9898：<\/p>\n

\u4f60\u5728\u5b9e\u9645\u9879\u76ee\u4e2d\u9047\u5230\u8fc7\u54ea\u4e9b\u56e0\u4e3a pickle\u3001Queue \u6216 ProcessPool \u5bfc\u81f4\u7684\u6027\u80fd\u95ee\u9898？<\/p>\n

\u5982\u679c\u8981\u5728\u591a\u8fdb\u7a0b\u4e4b\u95f4\u5904\u7406\u4e00\u4e2a\u6570 GB \u7684 NumPy \u6570\u7ec4，\u4f60\u4f1a\u9009\u62e9\u76f4\u63a5\u5e8f\u5217\u5316\u3001\u5171\u4eab\u5185\u5b58，\u8fd8\u662f\u91cd\u65b0\u8bbe\u8ba1\u4efb\u52a1\u8fb9\u754c？<\/p>\n

\u6b22\u8fce\u628a\u4f60\u7684\u5b9e\u8df5\u65b9\u6848\u548c\u8e29\u5751\u7ecf\u5386\u5206\u4eab\u51fa\u6765\u3002\u5f88\u591a\u771f\u6b63\u6709\u4ef7\u503c\u7684 Python \u6700\u4f73\u5b9e\u8df5，\u5f80\u5f80\u5e76\u4e0d\u662f\u6765\u81ea API \u6587\u6863，\u800c\u662f\u6765\u81ea\u4e00\u6b21\u53c8\u4e00\u6b21\u5bf9\u6027\u80fd\u74f6\u9888\u548c\u5f02\u5e38\u884c\u4e3a\u7684\u8ffd\u8e2a\u3002<\/p>\n

\n

\u53c2\u8003\u5b66\u4e60\u65b9\u5411<\/h3>\n

\u5982\u679c\u5e0c\u671b\u7ee7\u7eed\u6df1\u5165 Python \u591a\u8fdb\u7a0b\u4e0e\u9ad8\u6027\u80fd\u8ba1\u7b97，\u53ef\u4ee5\u91cd\u70b9\u5b66\u4e60：<\/p>\n