{"id":33808,"date":"2025-04-28T15:49:45","date_gmt":"2025-04-28T07:49:45","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/33808.html"},"modified":"2025-04-28T15:49:45","modified_gmt":"2025-04-28T07:49:45","slug":"%e6%ba%90%e7%a0%81%e8%a7%a3%e6%9e%90%ef%bc%9a%e4%bb%8e%e9%9b%b6%e8%a7%a3%e8%af%bbsamsegment-anything-model%e5%a4%a7%e6%a8%a1%e5%9e%8b%ef%bc%81","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/33808.html","title":{"rendered":"\u6e90\u7801\u89e3\u6790\uff1a\u4ece\u96f6\u89e3\u8bfbSAM(Segment Anything Model)\u5927\u6a21\u578b\uff01"},"content":{"rendered":"

\u8282\u524d，\u6211\u4eec\u661f\u7403\u7ec4\u7ec7\u4e86\u4e00\u573a\u7b97\u6cd5\u5c97\u6280\u672f&\u9762\u8bd5\u8ba8\u8bba\u4f1a，\u9080\u8bf7\u4e86\u4e00\u4e9b\u4e92\u8054\u7f51\u5927\u5382\u670b\u53cb\u3001\u53c2\u52a0\u793e\u62db\u548c\u6821\u62db\u9762\u8bd5\u7684\u540c\u5b66\u3002<\/p>\n

\u9488\u5bf9\u7b97\u6cd5\u5c97\u6280\u672f\u8d8b\u52bf\u3001\u5927\u6a21\u578b\u843d\u5730\u9879\u76ee\u7ecf\u9a8c\u5206\u4eab\u3001\u65b0\u624b\u5982\u4f55\u5165\u95e8\u7b97\u6cd5\u5c97\u3001\u8be5\u5982\u4f55\u51c6\u5907\u3001\u9762\u8bd5\u5e38\u8003\u70b9\u5206\u4eab\u7b49\u70ed\u95e8\u8bdd\u9898\u8fdb\u884c\u4e86\u6df1\u5165\u7684\u8ba8\u8bba\u3002<\/p>\n

\u5408\u96c6：<\/p>\n

\u300a\u5927\u6a21\u578b\u9762\u8bd5\u5b9d\u5178\u300b(2024\u7248) \u6b63\u5f0f\u53d1\u5e03！<\/p>\n

\u6301\u7eed\u706b\u7206！！！\u300aAIGC \u9762\u8bd5\u5b9d\u5178\u300b\u5df2\u5708\u7c89\u65e0\u6570！<\/p>\n

\n

SAM(Segment Anything Model)，\u987e\u540d\u601d\u4e49，\u5373\u4e3a\u5206\u5272\u4e00\u5207！\u8be5\u6a21\u578b\u7531Facebook\u7684Meta AI\u5b9e\u9a8c\u5ba4，\u80fd\u591f\u6839\u636e\u6587\u672c\u6307\u4ee4\u6216\u56fe\u50cf\u8bc6\u522b，\u5b9e\u73b0\u5bf9\u4efb\u610f\u7269\u4f53\u7684\u8bc6\u522b\u4e0e\u5206\u5272\u3002\u5b83\u7684\u8bde\u751f，\u65e0\u7591\u662fCV\u9886\u57df\u7684\u4e00\u6b21\u91cd\u8981\u91cc\u7a0b\u7891\u3002<\/p>\n

\"\u56fe\u7247\"<\/p>\n

\u8bba\u6587\u5730\u5740：https:\/\/arxiv.org\/abs\/2304.02643 \u9879\u76ee\u5730\u5740：https:\/\/github.com\/facebookresearch\/segment-anything<\/p>\n

SAM Task<\/h3>\n

SAM\u501f\u9274\u4e86NLP\u9886\u57df\u7684Prompt\u7b56\u7565，\u901a\u8fc7\u7ed9\u56fe\u50cf\u5206\u5272\u4efb\u52a1\u63d0\u4f9bPrompt\u63d0\u793a\u6765\u5b8c\u6210\u4efb\u610f\u76ee\u6807\u7684\u5feb\u901f\u5206\u5272\u3002Prompt\u7c7b\u578b\u53ef\u4ee5\u662f**\u300c\u524d\u666f\/\u80cc\u666f\u70b9\u96c6\u3001\u7c97\u7565\u7684\u6846\u6216\u906e\u7f69\u3001\u4efb\u610f\u5f62\u5f0f\u7684\u6587\u672c\u6216\u8005\u4efb\u4f55\u6307\u793a\u56fe\u50cf\u4e2d\u9700\u8981\u8fdb\u884c\u5206\u5272\u300d**\u7684\u4fe1\u606f\u3002\u5982\u4e0b\u56fe(a)\u6240\u793a，\u6a21\u578b\u7684\u8f93\u5165\u662f\u539f\u59cb\u7684\u56fe\u50cf\u548c\u4e00\u4e9bprompt，\u76ee\u6807\u662f\u8f93\u51fa"valid"\u7684\u5206\u5272，\u6240\u8c13valid，\u5c31\u662f\u5f53prompt\u7684\u6307\u5411\u662f\u6a21\u7cca\u65f6，\u6a21\u578b\u80fd\u591f\u8f93\u51fa\u81f3\u5c11\u5176\u4e2d\u4e00\u4e2amask\u3002<\/p>\n

\u8fd9\u6837，\u53ef\u4ee5\u662f\u7684SAM\u80fd\u591f\u9002\u914d\u5404\u79cd\u4e0b\u6e38\u4efb\u52a1\u3002\u4f8b\u5982，\u7ed9\u5b9a\u4e00\u4e2a\u732b\u7684\u8fb9\u754c\u6846，SAM\u80fd\u591f\u8f93\u51fa\u5176mask，\u4ece\u800c\u548c\u5b9e\u4f8b\u5206\u5272\u4efb\u52a1\u642d\u914d\u8d77\u6765\u3002<\/p>\n

\"\u56fe\u7247\"<\/p>\n

SAM Model<\/h3>\n

\u5982\u4e0b\u56fe\u6240\u793a，SAM\u6a21\u578b\u5305\u542b\u4e09\u4e2a\u6838\u5fc3\u7ec4\u4ef6，Image Encoder\u3001Prompt Encoder\u548cMask Decoder\u3002\u56fe\u50cf\u7ecf\u8fc7Image Encoder\u7f16\u7801，Prompt\u63d0\u793a\u7ecf\u8fc7Prompt Encoder\u7f16\u7801，\u4e24\u90e8\u5206Embedding\u518d\u7ecf\u8fc7\u4e00\u4e2a\u8f7b\u91cf\u5316\u7684Mask Decoder\u5f97\u5230\u878d\u5408\u540e\u7684\u7279\u5f81\u3002\u5176\u4e2d，Encoder\u90e8\u5206\u4f7f\u7528\u7684\u662f\u5df2\u6709\u6a21\u578b，Decoder\u90e8\u5206\u4f7f\u7528Transformer\u3002<\/p>\n

\"\u56fe\u7247\"<\/p>\n

Image Encoder<\/h4>\n

Image Encoder\u7684\u4f5c\u7528\u662f\u628a\u56fe\u50cf\u6620\u5c04\u5230\u7279\u5f81\u7a7a\u95f4，\u6574\u4f53\u8fc7\u7a0b\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

\"\u56fe\u7247\"<\/p>\n

\u6b63\u5982\u8bba\u6587\u4e2d\u6240\u8bb2，\u672c\u8d28\u4e0a\u8fd9\u4e2aEncoder\u53ef\u4ee5\u662f\u4efb\u4f55\u7f51\u7edc\u7ed3\u6784，\u5728\u8fd9\u91cc\u4f7f\u7528\u7684\u662f\u5fae\u8c03\u7684Detectron\u7684ViT，\u5f53\u7136\u5b83\u4e5f\u53ef\u4ee5\u88ab\u6539\u6210\u4f20\u7edf\u7684\u5377\u79ef\u7ed3\u6784，\u975e\u5e38\u5408\u7406\u3002<\/p>\n

\u8f93\u5165\u56fe\u50cf\u7ecf\u8fc7ViT\u7ed3\u6784\u7684\u8fc7\u7a0b\u5982\u4e0b：<\/p>\n

1. Patch Embedding<\/h5>\n

\u8f93\u5165\u56fe\u50cf\u901a\u8fc7\u4e00\u4e2a\u5377\u79efbase，\u5c06\u56fe\u50cf\u5212\u5206\u4e3a16×16\u7684patches，\u6b65\u957f\u4e5f\u4e3a16，\u8fd9\u6837feature map\u7684\u5c3a\u5bf8\u5c31\u7f29\u5c0f\u4e8616\u500d，\u540c\u65f6channel\u4ece3\u6620\u5c04\u5230768\u3002Patch Embedding\u793a\u610f\u56fe\u5982\u4e0b\u6240\u793a\u3002<\/p>\n

\"\u56fe\u7247\"<\/p>\n

\u4ee3\u7801\u5b9e\u73b0：<\/p>\n

'''
\n\u5c06\u8f93\u5165\u7684\u56fe\u50cf\u8f6c\u6362\u4e3a\u5e8f\u5217\u5316\u7684\u7279\u5f81\u5411\u91cf
\n'''<\/span>
\nclass<\/span> PatchEmbed<\/span>(<\/span>nn.<\/span>Module)<\/span>:<\/span>
\n def<\/span> __init__<\/span>(<\/span>
\n self,<\/span>
\n # \u5377\u79ef\u6838\u5927\u5c0f<\/span>
\n # \u8fd9\u91cc\u662f (16, 16)，\u610f\u5473\u7740\u56fe\u50cf\u5c06\u88ab\u5212\u5206\u4e3a16×16\u7684patches<\/span>
\n kernel_size:<\/span> Tuple[<\/span>int<\/span>,<\/span> int<\/span>]<\/span> =<\/span> (<\/span>16<\/span>,<\/span> 16<\/span>)<\/span>,<\/span>
\n # \u5377\u79ef\u7684\u6b65\u957f，\u4e0ekernel_size\u76f8\u540c，\u5373(16, 16)，<\/span>
\n # \u610f\u5473\u7740\u6bcf\u4e00\u6b65\u79fb\u52a816\u4e2a\u50cf\u7d20，\u8fd9\u6837\u56fe\u50cf\u7684\u5c3a\u5bf8\u5c31\u4f1a\u51cf\u5c11\u5230\u539f\u6765\u76841\/16<\/span>
\n stride:<\/span> Tuple[<\/span>int<\/span>,<\/span> int<\/span>]<\/span> =<\/span> (<\/span>16<\/span>,<\/span> 16<\/span>)<\/span>,<\/span>
\n # \u63a7\u5236\u8fb9\u7f18\u586b\u5145，\u8fd9\u91cc\u8bbe\u7f6e\u4e3a (0, 0)，\u610f\u5473\u7740\u6ca1\u6709\u989d\u5916\u7684\u586b\u5145<\/span>
\n padding:<\/span> Tuple[<\/span>int<\/span>,<\/span> int<\/span>]<\/span> =<\/span> (<\/span>0<\/span>,<\/span> 0<\/span>)<\/span>,<\/span>
\n # \u8f93\u5165\u56fe\u50cf\u7684\u901a\u9053\u6570，\u901a\u5e38\u4e3a3（RGB\u56fe\u50cf）<\/span>
\n in_chans:<\/span> int<\/span> =<\/span> 3<\/span>,<\/span>
\n # \u8f93\u51fa\u7684\u7279\u5f81\u7ef4\u5ea6，\u4e5f\u5c31\u662f\u6bcf\u4e2apatch\u88ab\u7f16\u7801\u4e3a\u7684\u5411\u91cf\u7684\u957f\u5ea6，\u8fd9\u91cc\u8bbe\u7f6e\u4e3a768<\/span>
\n embed_dim:<\/span> int<\/span> =<\/span> 768<\/span>,<\/span>
\n )<\/span> –<\/span>><\/span> None<\/span>:<\/span>
\n '''
\n \u521d\u59cb\u5316\u8fd9\u4e2a\u5b50\u7c7b\u5b9e\u4f8b\u7684\u5c5e\u6027
\n '''<\/span>
\n # PatchEmbed\u7684\u5b50\u7c7b，\u7ee7\u627f\u81eann.Module，\u7528\u4e8e\u6784\u5efa\u795e\u7ecf\u7f51\u7edc\u6a21\u5757<\/span>
\n super<\/span>(<\/span>)<\/span>.<\/span>__init__(<\/span>)<\/span>
\n self.<\/span>proj =<\/span> nn.<\/span>Conv2d(<\/span>
\n in_chans,<\/span> embed_dim,<\/span> kernel_size=<\/span>kernel_size,<\/span> stride=<\/span>stride,<\/span> padding=<\/span>padding
\n )<\/span>
\n '''\u524d\u5411\u4f20\u64ad：
\n \u63a5\u6536\u8f93\u5165\u5f20\u91cf x，\u5f62\u72b6 (B, C, H, W)，\u5176\u4e2d，
\n – B\u8868\u793a\u6279\u6b21\u5927\u5c0f
\n – C \u662f\u8f93\u5165\u901a\u9053\u6570
\n – H \u548c W \u662f\u56fe\u50cf\u7684\u9ad8\u5ea6\u548c\u5bbd\u5ea6
\n '''<\/span>
\n def<\/span> forward<\/span>(<\/span>self,<\/span> x:<\/span> torch.<\/span>Tensor)<\/span> –<\/span>><\/span> torch.<\/span>Tensor:<\/span>
\n # \u5377\u79ef，\u5c06\u8f93\u5165\u7684\u901a\u9053\u6570\u4ece in_chans \u8f6c\u6362\u4e3a embed_dim<\/span>
\n x =<\/span> self.<\/span>proj(<\/span>x)<\/span>
\n # \u5c06\u5f20\u91cf\u7684\u7ef4\u5ea6\u987a\u5e8f\u4ece (B, C, H, W) \u8c03\u6574\u4e3a (B, H, W, C)<\/span>
\n x =<\/span> x.<\/span>permute(<\/span>0<\/span>,<\/span> 2<\/span>,<\/span> 3<\/span>,<\/span> 1<\/span>)<\/span>
\n return<\/span> x<\/p>\n

Patch Embedding\u8fc7\u7a0b\u5728Vision Transformer\u7ed3\u6784\u56fe\u4e2d\u5bf9\u5e94\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

\"\u56fe\u7247\"<\/p>\n

2. Positiona Embedding<\/h5>\n

\u7ecf\u8fc7Patch Embedding\u540e\u8f93\u51fatokens\u9700\u8981\u52a0\u5165\u4f4d\u7f6e\u7f16\u7801，\u4ee5\u4fdd\u7559\u56fe\u50cf\u7684\u7a7a\u95f4\u4fe1\u606f\u3002\u4f4d\u7f6e\u7f16\u7801\u53ef\u4ee5\u7406\u89e3\u4e3a\u4e00\u5f20map，map\u7684\u884c\u6570\u4e0e\u8f93\u5165\u5e8f\u5217\u4e2a\u6570\u76f8\u540c，\u6bcf\u4e00\u884c\u4ee3\u8868\u4e00\u4e2a\u5411\u91cf，\u5411\u91cf\u7684\u7ef4\u5ea6\u548c\u8f93\u5165\u5e8f\u5217tokens\u7684\u7ef4\u5ea6\u76f8\u540c，\u4f4d\u7f6e\u7f16\u7801\u7684\u64cd\u4f5c\u662fsum，\u6240\u4ee5\u7ef4\u5ea6\u4f9d\u65e7\u4fdd\u6301\u4e0d\u53d8\u3002<\/p>\n

\"\u56fe\u7247\"<\/p>\n

\u56fe\u50cf\u5c3a\u5bf8\u662f1024，\u56e0\u6b64patch\u7684\u6570\u91cf\u662f1024\/16=64\u3002<\/p>\n

\u4ee3\u7801\u5b9e\u73b0：<\/p>\n

# \u5728ImageEncoderViT\u7684__init__\u5b9a\u4e49<\/span>
\nif<\/span> use_abs_pos:<\/span>
\n # \u4f7f\u7528\u9884\u8bad\u7ec3\u56fe\u50cf\u5927\u5c0f\u521d\u59cb\u5316\u7edd\u5bf9\u4f4d\u7f6e\u5d4c\u5165<\/span>
\n self.<\/span>pos_embed =<\/span> nn.<\/span>Parameter(<\/span>
\n torch.<\/span>zeros(<\/span>1<\/span>,<\/span> img_size \/\/<\/span> patch_size,<\/span> img_size \/\/<\/span> patch_size,<\/span> embed_dim)<\/span>
\n )<\/span>
\n# \u5728ImageEncoderViT\u7684forward\u6dfb\u52a0\u4f4d\u7f6e\u7f16\u7801<\/span>
\nif<\/span> self.<\/span>pos_embed is<\/span> not<\/span> None<\/span>:<\/span>
\n x =<\/span> x +<\/span> self.<\/span>pos_embed<\/p>\n

Positiona Embedding\u8fc7\u7a0b\u5728\u7ed3\u6784\u56fe\u4e2d\u5bf9\u5e94\u7684\u90e8\u5206：<\/p>\n

\"\u56fe\u7247\"<\/p>\n

3. Transformer Encoder<\/h5>\n

feature map\u901a\u8fc716\u4e2aTransformer Block，\u5176\u4e2d12\u4e2aBlock\u4f7f\u7528\u4e86\u57fa\u4e8eWindow Partition（\u5c31\u662f\u628a\u7279\u5f81\u56fe\u5206\u621014*14\u7684windows\u505a\u5c40\u90e8\u7684Attention）\u7684\u6ce8\u610f\u529b\u673a\u5236，\u4ee5\u5904\u7406\u5c40\u90e8\u4fe1\u606f\u3002\u53e6\u59164\u4e2aBlock\u662f\u5168\u5c40\u6ce8\u610f\u529b\u6a21\u5757，\u5b83\u4eec\u7a7f\u63d2\u5728Window Partition\u6a21\u5757\u4e4b\u95f4，\u4ee5\u6355\u6349\u56fe\u50cf\u7684\u5168\u5c40\u4e0a\u4e0b\u6587\u3002<\/p>\n

# \u5728ImageEncoderViT\u7684__init__\u5b9a\u4e49<\/span>
\n# —–Transformer Encoder—–<\/span>
\n# \u521d\u59cb\u5316\u4e00\u4e2aModuleList，\u7528\u4e8e\u5b58\u50a8Block\u5b9e\u4f8b<\/span>
\nself.<\/span>blocks =<\/span> nn.<\/span>ModuleList(<\/span>)<\/span>
\n# \u5faa\u73af\u521b\u5efaBlock，depth\u662fTransformer Encoder\u5c42\u6570<\/span>
\nfor<\/span> i in<\/span> range<\/span>(<\/span>depth)<\/span>:<\/span>
\n # \u521b\u5efa\u5355\u4e2aBlock<\/span>
\n block =<\/span> Block(<\/span>
\n # \u8f93\u5165\u7684\u901a\u9053\u6570，\u5373\u6bcf\u4e2apatch\u7f16\u7801\u540e\u7684\u5411\u91cf\u7ef4\u5ea6<\/span>
\n dim=<\/span>embed_dim,<\/span>
\n # \u81ea\u6ce8\u610f\u529b\u673a\u5236\u4e2d\u7684\u6ce8\u610f\u529b\u5934\u6570<\/span>
\n num_heads=<\/span>num_heads,<\/span>
\n # MLP\u5c42\u7684\u901a\u9053\u6570\u76f8\u5bf9\u4e8e\u8f93\u5165\u901a\u9053\u6570\u7684\u6bd4\u4f8b<\/span>
\n mlp_ratio=<\/span>mlp_ratio,<\/span>
\n # \u662f\u5426\u5728QKV\u5168\u8fde\u63a5\u5c42\u4e2d\u4f7f\u7528\u504f\u7f6e<\/span>
\n qkv_bias=<\/span>qkv_bias,<\/span>
\n # \u5f52\u4e00\u5316\u5c42<\/span>
\n norm_layer=<\/span>norm_layer,<\/span>
\n # \u6fc0\u6d3b\u51fd\u6570<\/span>
\n act_layer=<\/span>act_layer,<\/span>
\n # \u662f\u5426\u4f7f\u7528\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801<\/span>
\n use_rel_pos=<\/span>use_rel_pos,<\/span>
\n # \u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801\u7684\u521d\u59cb\u5316\u8bbe\u7f6e<\/span>
\n rel_pos_zero_init=<\/span>rel_pos_zero_init,<\/span>
\n # \u5982\u679c\u5f53\u524dBlock\u4e0d\u662f\u5168\u5c40\u6ce8\u610f\u529b\u5c42，\u5219\u4f7f\u7528\u7a97\u53e3\u5927\u5c0f，\u5426\u5219\u4f7f\u75280<\/span>
\n window_size=<\/span>window_size if<\/span> i not<\/span> in<\/span> global_attn_indexes else<\/span> 0<\/span>,<\/span>
\n # \u8f93\u5165\u7279\u5f81\u7684\u5c3a\u5bf8，\u57fa\u4e8e\u539f\u59cb\u56fe\u50cf\u5927\u5c0f\u548cpatch\u5927\u5c0f\u8ba1\u7b97\u5f97\u51fa<\/span>
\n input_size=<\/span>(<\/span>img_size \/\/<\/span> patch_size,<\/span> img_size \/\/<\/span> patch_size)<\/span>,<\/span>
\n )<\/span>
\n # \u5c06\u521b\u5efa\u7684Block\u5bf9\u8c61\u6dfb\u52a0\u5230self.blocks\u5217\u8868\u4e2d<\/span>
\n self.<\/span>blocks.<\/span>append(<\/span>block)<\/span>
\n# —–Transformer Encoder—–<\/span><\/p>\n

Transformer Encoder\u8fc7\u7a0b\u5728\u7ed3\u6784\u56fe\u4e2d\u5bf9\u5e94\u7684\u90e8\u5206：<\/p>\n

\"\u56fe\u7247\"<\/p>\n

Encoder Block<\/p>\n

\u5982\u4e0a\u56fe\u53f3\u6240\u793a，Encoder Block\u4ece\u4f4e\u5230\u9ad8\u4e3b\u8981\u7531LayerNorm \u3001Multi-Head Attention\u548cMLP\u6784\u6210\u3002<\/p>\n

class<\/span> Block<\/span>(<\/span>nn.<\/span>Module)<\/span>:<\/span>
\n def<\/span> __init__<\/span>(<\/span>
\n self,<\/span>
\n dim:<\/span> int<\/span>,<\/span> # \u8f93\u5165\u901a\u9053\u6570<\/span>
\n num_heads:<\/span> int<\/span>,<\/span> # attention\u4e2dhead\u7684\u4e2a\u6570<\/span>
\n mlp_ratio:<\/span> float<\/span> =<\/span> 4.0<\/span>,<\/span> # MLP\u5c42\u7684\u901a\u9053\u6570\u76f8\u5bf9\u4e8e\u8f93\u5165\u901a\u9053\u6570\u7684\u6bd4\u4f8b\u3002<\/span>
\n qkv_bias:<\/span> bool<\/span> =<\/span> True<\/span>,<\/span> # \u5982\u679c\u4e3aTrue，QKV\u5168\u8fde\u63a5\u5c42\u5305\u542b\u504f\u7f6e\u3002<\/span>
\n norm_layer:<\/span> Type[<\/span>nn.<\/span>Module]<\/span> =<\/span> nn.<\/span>LayerNorm,<\/span> # \u5f52\u4e00\u5316\u5c42<\/span>
\n act_layer:<\/span> Type[<\/span>nn.<\/span>Module]<\/span> =<\/span> nn.<\/span>GELU,<\/span> # \u6fc0\u6d3b\u5c42<\/span>
\n use_rel_pos:<\/span> bool<\/span> =<\/span> False<\/span>,<\/span> # \u662f\u5426\u4f7f\u7528\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801<\/span>
\n rel_pos_zero_init:<\/span> bool<\/span> =<\/span> True<\/span>,<\/span> # \u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801\u7684\u521d\u59cb\u5316\u8bbe\u7f6e<\/span>
\n window_size:<\/span> int<\/span> =<\/span> 0<\/span>,<\/span> # \u6ce8\u610f\u529b\u5c42\u7684\u7a97\u53e3\u5927\u5c0f<\/span>
\n input_size:<\/span> Optional[<\/span>Tuple[<\/span>int<\/span>,<\/span> int<\/span>]<\/span>]<\/span> =<\/span> None<\/span>,<\/span> # \u8f93\u5165\u7279\u5f81\u7684\u5c3a\u5bf8<\/span>
\n )<\/span> –<\/span>><\/span> None<\/span>:<\/span>
\n super<\/span>(<\/span>)<\/span>.<\/span>__init__(<\/span>)<\/span>
\n self.<\/span>norm1 =<\/span> norm_layer(<\/span>dim)<\/span> # \u7b2c\u4e00\u4e2a\u5f52\u4e00\u5316\u5c42，\u7528\u4e8e\u6ce8\u610f\u529b\u5c42<\/span>
\n self.<\/span>attn =<\/span> Attention(<\/span> # Multi-Head Attention<\/span>
\n dim,<\/span>
\n num_heads=<\/span>num_heads,<\/span>
\n qkv_bias=<\/span>qkv_bias,<\/span>
\n use_rel_pos=<\/span>use_rel_pos,<\/span>
\n rel_pos_zero_init=<\/span>rel_pos_zero_init,<\/span>
\n input_size=<\/span>input_size if<\/span> window_size ==<\/span> 0<\/span> else<\/span> (<\/span>window_size,<\/span> window_size)<\/span>,<\/span>
\n )<\/span>
\n self.<\/span>norm2 =<\/span> norm_layer(<\/span>dim)<\/span> #\u7b2c\u4e8c\u4e2a\u5f52\u4e00\u5316\u5c42，\u7528\u4e8eMLP\u4e4b\u524d<\/span>
\n # MLP<\/span>
\n self.<\/span>mlp =<\/span> MLPBlock(<\/span>embedding_dim=<\/span>dim,<\/span> mlp_dim=<\/span>int<\/span>(<\/span>dim *<\/span> mlp_ratio)<\/span>,<\/span> act=<\/span>act_layer)<\/span>
\n self.<\/span>window_size =<\/span> window_size
\n # \u524d\u5411\u4f20\u64ad<\/span>
\n def<\/span> forward<\/span>(<\/span>self,<\/span> x:<\/span> torch.<\/span>Tensor)<\/span> –<\/span>><\/span> torch.<\/span>Tensor:<\/span>
\n # \u4fdd\u5b58\u8f93\u5165\u5f20\u91cf\u7684\u526f\u672c<\/span>
\n shortcut =<\/span> x
\n # \u5bf9\u8f93\u5165\u5f20\u91cf\u5e94\u7528\u7b2c\u4e00\u4e2a\u5f52\u4e00\u5316\u5c42<\/span>
\n x =<\/span> self.<\/span>norm1(<\/span>x)<\/span>
\n # Window partition \u5bf9X\u8fdb\u884cpadding<\/span>
\n if<\/span> self.<\/span>window_size ><\/span> 0<\/span>:<\/span>
\n H,<\/span> W =<\/span> x.<\/span>shape[<\/span>1<\/span>]<\/span>,<\/span> x.<\/span>shape[<\/span>2<\/span>]<\/span>
\n x,<\/span> pad_hw =<\/span> window_partition(<\/span>x,<\/span> self.<\/span>window_size)<\/span>
\n # Multi-Head Attention<\/span>
\n x =<\/span> self.<\/span>attn(<\/span>x)<\/span>
\n # \u5982\u679c window_size > 0，\u4f7f\u7528window_unpartition\u53bb\u9664\u7a97\u53e3\u5206\u533a\u7684padding，\u6062\u590d\u539f\u59cb\u5c3a\u5bf8<\/span>
\n if<\/span> self.<\/span>window_size ><\/span> 0<\/span>:<\/span>
\n x =<\/span> window_unpartition(<\/span>x,<\/span> self.<\/span>window_size,<\/span> pad_hw,<\/span> (<\/span>H,<\/span> W)<\/span>)<\/span>
\n # \u5c06\u6ce8\u610f\u529b\u5c42\u7684\u8f93\u51fa\u4e0e\u8f93\u5165\u5f20\u91cf\u76f8\u52a0，\u5b9e\u73b0\u6b8b\u5dee\u8fde\u63a5<\/span>
\n x =<\/span> shortcut +<\/span> x
\n # \u5bf9\u7ecf\u8fc7\u7b2c\u4e8c\u4e2a\u5f52\u4e00\u5316\u5c42\u7684\u5f20\u91cf\u5e94\u7528MLP\u5c42，\u518d\u6b21\u4f7f\u7528\u6b8b\u5dee\u8fde\u63a5<\/span>
\n x =<\/span> x +<\/span> self.<\/span>mlp(<\/span>self.<\/span>norm2(<\/span>x)<\/span>)<\/span>
\n # \u8fd4\u56de\u6700\u7ec8\u7684\u5f20\u91cf x<\/span>
\n return<\/span> x<\/p>\n

Partition\u64cd\u4f5c<\/p>\n

\u5728\u975e\u5168\u5c40\u6ce8\u610f\u529b\u7684Block\u4e2d，\u4e3a\u4e86\u9002\u5e9414×14\u7684\u7a97\u53e3\u5927\u5c0f，\u8f93\u5165\u7279\u5f81\u56fe\u9700\u8981\u8fdb\u884c\u8865\u8fb9（padding）\u548c\u62c6\u5206\u64cd\u4f5c\u3002\u5177\u4f53\u6d41\u7a0b\u5982\u4e0b：<\/p>\n

  • \n

    \u8f93\u5165\u7279\u5f81\u56fe：\u8f93\u5165\u7279\u5f81\u56fe\u7684\u521d\u59cb\u5c3a\u5bf8\u4e3a 1x64x64x768\u3002<\/p>\n<\/li>\n

  • \n

    \u786e\u5b9a\u6700\u5c0f\u53ef\u6574\u9664\u5c3a\u5bf8：\u7a97\u53e3\u5927\u5c0f\u4e3a14*14，\u8981\u627e\u5230\u80fd\u591f\u88ab14\u6574\u9664\u7684\u6700\u5c0f\u7279\u5f81\u56fe\u5c3a\u5bf8\u3002\u5bf9\u4e8e\u5bbd\u5ea6\u548c\u9ad8\u5ea6，\u6211\u4eec\u9700\u8981\u627e\u5230\u5927\u4e8e\u7b49\u4e8e64\u4e14\u80fd\u88ab14\u6574\u9664\u7684\u6700\u5c0f\u6570\u3002\u8fd9\u4e24\u4e2a\u6570\u5206\u522b\u662f70（64+6）\u548c70（64+6），\u6240\u4ee5\u6700\u5c0f\u53ef\u6574\u9664\u7279\u5f81\u56fe\u7684\u5c3a\u5bf8\u662f 1x70x70x768\u3002<\/p>\n<\/li>\n

  • \n

    padding：\u4e3a\u4e86\u5c06\u7279\u5f81\u56fe\u5c3a\u5bf8\u4ece 64×64 \u6269\u5c55\u5230 70×70，\u6211\u4eec\u9700\u8981\u5728\u53f3\u4e0b\u89d2\u586b\u5145 6×6 \u7684\u533a\u57df，\u56e0\u4e3a70-64=6\u3002\u8fd9\u79cdpadding\u65b9\u5f0f\u786e\u4fdd\u4e86\u7a97\u53e3\u53ef\u4ee5\u5728\u7279\u5f81\u56fe\u7684\u8fb9\u7f18\u6b63\u786e\u5730\u5212\u5206\u3002<\/p>\n<\/li>\n

  • \n

    \u62c6\u5206\u7279\u5f81\u56fe：\u5c06padding\u540e\u7684\u7279\u5f81\u56fe1x70x70x768\u6309\u7167\u7a97\u53e3\u5927\u5c0f14×14\u8fdb\u884c\u62c6\u5206\u3002\u56e0\u4e3a70\/14=5，\u6240\u4ee5\u7279\u5f81\u56fe\u53ef\u4ee5\u88ab\u62c6\u5206\u4e3a 5×5\u4e2a14×14\u7684\u7a97\u53e3，\u603b\u51715×5=25\u4e2a\u7a97\u53e3\u3002\u6bcf\u4e2a\u7a97\u53e3\u7684\u5c3a\u5bf8\u4e3a14x14x768\u3002<\/p>\n<\/li>\n

    \u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

    \"\u56fe\u7247\"<\/p>\n

    # \u5c06\u8f93\u5165\u5f20\u91cfx\u5206\u5272\u6210\u6307\u5b9a\u5927\u5c0f\u7684\u7a97\u53e3<\/span>
    \ndef<\/span> window_partition<\/span>(<\/span>x:<\/span> torch.<\/span>Tensor,<\/span> window_size:<\/span> int<\/span>)<\/span> –<\/span>><\/span> Tuple[<\/span>torch.<\/span>Tensor,<\/span> Tuple[<\/span>int<\/span>,<\/span> int<\/span>]<\/span>]<\/span>:<\/span>
    \n # \u83b7\u53d6\u8f93\u5165\u5f20\u91cf\u5f62\u72b6<\/span>
    \n # B\u8868\u793a\u6279\u6b21\u5927\u5c0f，H\u548cW\u8868\u793a\u9ad8\u548c\u5bbd，C\u8868\u793a\u901a\u9053\u6570<\/span>
    \n B,<\/span> H,<\/span> W,<\/span> C =<\/span> x.<\/span>shape
    \n # \u8ba1\u7b97\u586b\u5145\u9ad8\u5ea6\u548c\u5bbd\u5ea6 pad_h \u548c pad_w，\u4ee5\u4f7f\u5f97\u8f93\u5165\u5c3a\u5bf8\u80fd\u88abwindow_size\u6574\u9664<\/span>
    \n # \u907f\u514d\u5728\u5206\u5272\u65f6\u4ea7\u751f\u975e\u5b8c\u6574\u7684\u7a97\u53e3<\/span>
    \n pad_h =<\/span> (<\/span>window_size –<\/span> H %<\/span> window_size)<\/span> %<\/span> window_size
    \n pad_w =<\/span> (<\/span>window_size –<\/span> W %<\/span> window_size)<\/span> %<\/span> window_size
    \n # \u5982\u679c\u9700\u8981\u586b\u5145，\u4f7f\u7528F.pad\u51fd\u6570\u5728\u5bbd\u5ea6\u548c\u9ad8\u5ea6\u65b9\u5411\u4e0a\u8fdb\u884c\u586b\u5145<\/span>
    \n if<\/span> pad_h ><\/span> 0<\/span> or<\/span> pad_w ><\/span> 0<\/span>:<\/span>
    \n x =<\/span> F.<\/span>pad(<\/span>x,<\/span> (<\/span>0<\/span>,<\/span> 0<\/span>,<\/span> 0<\/span>,<\/span> pad_w,<\/span> 0<\/span>,<\/span> pad_h)<\/span>)<\/span>
    \n # \u66f4\u65b0\u586b\u5145\u540e\u5f20\u91cf\u7684\u9ad8\u5ea6\u548c\u5bbd\u5ea6 Hp \u548c Wp<\/span>
    \n Hp,<\/span> Wp =<\/span> H +<\/span> pad_h,<\/span> W +<\/span> pad_w
    \n # \u5f20\u91cf\u91cd\u5851\u4e3a：B,Hp\/S,S,Wp\/S,S,C，\u8fd9\u6837\u53ef\u4ee5\u5c06\u8f93\u5165\u5f20\u91cf\u5206\u5272\u6210\u591a\u4e2a\u7a97\u53e3<\/span>
    \n x =<\/span> x.<\/span>view(<\/span>B,<\/span> Hp \/\/<\/span> window_size,<\/span> window_size,<\/span> Wp \/\/<\/span> window_size,<\/span> window_size,<\/span> C)<\/span>
    \n # \u8c03\u6574\u5f20\u91cf\u7684\u5f62\u72b6，\u4f7f\u5176\u7531B,Hp\/S,Wp\/S,S,S,C–>B*Hp*Wp\/(S*S),S,S,C<\/span>
    \n # \u8fd9\u6837\u6bcf\u4e2a\u7a97\u53e3\u90fd\u5728\u5f20\u91cf\u7684\u8fde\u7eed\u90e8\u5206<\/span>
    \n windows =<\/span> x.<\/span>permute(<\/span>0<\/span>,<\/span> 1<\/span>,<\/span> 3<\/span>,<\/span> 2<\/span>,<\/span> 4<\/span>,<\/span> 5<\/span>)<\/span>.<\/span>contiguous(<\/span>)<\/span>.<\/span>view(<\/span>–<\/span>1<\/span>,<\/span> window_size,<\/span> window_size,<\/span> C)<\/span>
    \n # \u8fd4\u56de\u4e00\u4e2a\u5305\u542b\u6240\u6709\u7a97\u53e3\u7684\u5f20\u91cf\u548c\u539f\u59cb\u5f20\u91cf\u7684\u586b\u5145\u540e\u5c3a\u5bf8 (Hp, Wp)<\/span>
    \n return<\/span> windows,<\/span> (<\/span>Hp,<\/span> Wp)<\/span><\/p>\n

    \u300cUnpartition\u64cd\u4f5c\u300d<\/p>\n

    \u5728\u975e\u5168\u5c40\u6ce8\u610f\u529b\u7684Block\u4e2d，\u5c06attention\u5c42\u8f93\u51fa\u7684\u7279\u5f81\u56fe1x70x70x768\u8f6c\u5316\u4e3a1x64x64x768\u7684\u7279\u5f81\u56fe，\u5b9e\u9645\u4e0a\u662f\u901a\u8fc7\u5207\u7247\u64cd\u4f5cx = x[:1, :64, :64, :]，\u4ece1x70x70x768\u7684\u7279\u5f81\u56fe\u4e2d\u53d6\u51fa\u5de6\u4e0a\u89d2\u76841x64x64x768\u90e8\u5206\u3002<\/p>\n

    \"\u56fe\u7247\"<\/p>\n

    # \u7528\u4e8e\u5c06window_partition\u51fd\u6570\u5206\u5272\u7684\u7a97\u53e3\u91cd\u65b0\u7ec4\u5408\u56de\u539f\u59cb\u5c3a\u5bf8\u7684\u5f20\u91cf<\/span>
    \ndef<\/span> window_unpartition<\/span>(<\/span>
    \n # \u83b7\u53d6\u8f93\u5165\u5f20\u91cf windows \u7684\u5f62\u72b6，\u4ee5\u53ca\u7a97\u53e3\u5927\u5c0f window_size<\/span>
    \n windows:<\/span> torch.<\/span>Tensor,<\/span> window_size:<\/span> int<\/span>,<\/span> pad_hw:<\/span> Tuple[<\/span>int<\/span>,<\/span> int<\/span>]<\/span>,<\/span> hw:<\/span> Tuple[<\/span>int<\/span>,<\/span> int<\/span>]<\/span>
    \n)<\/span> –<\/span>><\/span> torch.<\/span>Tensor:<\/span>
    \n # \u539f\u59cb\u5c3a\u5bf8\u7684\u586b\u5145\u9ad8\u5ea6\u548c\u5bbd\u5ea6<\/span>
    \n Hp,<\/span> Wp =<\/span> pad_hw
    \n # \u539f\u59cb\u5c3a\u5bf8\u7684\u65e0\u586b\u5145\u9ad8\u5ea6\u548c\u5bbd\u5ea6<\/span>
    \n H,<\/span> W =<\/span> hw
    \n # \u4ece\u7a97\u53e3\u5f20\u91cf\u7684\u603b\u5927\u5c0f\u4e2d\u8ba1\u7b97\u51fa\u539f\u59cb\u6279\u91cf\u5927\u5c0f B<\/span>
    \n B =<\/span> windows.<\/span>shape[<\/span>0<\/span>]<\/span> \/\/<\/span> (<\/span>Hp *<\/span> Wp \/\/<\/span> window_size \/\/<\/span> window_size)<\/span>
    \n # \u91cd\u5851\u7a97\u53e3\u5f20\u91cf：B*Hp*Wp\/(S*S),S,S,C–>B,Hp\/S,Wp\/S,S,S,C<\/span>
    \n x =<\/span> windows.<\/span>view(<\/span>B,<\/span> Hp \/\/<\/span> window_size,<\/span> Wp \/\/<\/span> window_size,<\/span> window_size,<\/span> window_size,<\/span> –<\/span>1<\/span>)<\/span>
    \n # \u518d\u6b21\u91cd\u5851\u5f20\u91cf：B,Hp\/S,Wp\/S,S,S,C–>B,Hp,Wp,C<\/span>
    \n x =<\/span> x.<\/span>permute(<\/span>0<\/span>,<\/span> 1<\/span>,<\/span> 3<\/span>,<\/span> 2<\/span>,<\/span> 4<\/span>,<\/span> 5<\/span>)<\/span>.<\/span>contiguous(<\/span>)<\/span>.<\/span>view(<\/span>B,<\/span> Hp,<\/span> Wp,<\/span> –<\/span>1<\/span>)<\/span>
    \n # \u5982\u679c\u539f\u59cb\u5c3a\u5bf8\u5c0f\u4e8e\u586b\u5145\u540e\u7684\u5c3a\u5bf8<\/span>
    \n if<\/span> Hp ><\/span> H or<\/span> Wp ><\/span> W:<\/span>
    \n # \u901a\u8fc7\u5207\u7247 x[:, :H, :W, :] \u53bb\u9664\u586b\u5145\u90e8\u5206，\u53ea\u4fdd\u7559\u539f\u59cb\u5927\u5c0f\u7684\u533a\u57df<\/span>
    \n x =<\/span> x[<\/span>:<\/span>,<\/span> :<\/span>H,<\/span> :<\/span>W,<\/span> :<\/span>]<\/span>.<\/span>contiguous(<\/span>)<\/span>
    \n # B,H,W,C<\/span>
    \n # \u8fd4\u56de\u5408\u5e76\u540e\u7684\u5f20\u91cf，\u5176\u5f62\u72b6\u4e3a (B,H,W,C)，\u5373\u539f\u59cb\u7684\u6279\u91cf\u5927\u5c0f\u3001\u9ad8\u5ea6\u3001\u5bbd\u5ea6\u548c\u901a\u9053\u6570<\/span>
    \n return<\/span> x<\/p>\n

    Encoder Block\u8fc7\u7a0b\u5982\u4e0b\u56fe\u6240\u793a：<\/p>\n

    \"\u56fe\u7247\"<\/p>\n

    window_partition\u5c06\u8f93\u5165\u7279\u5f81\u7684\u5c3a\u5bf8\u4ece(H, W)\u8c03\u6574\u4e3a(S, S)\u7684\u7a97\u53e3，\u5176\u4e2dS\u662f\u7a97\u53e3\u5927\u5c0f\u3002\u8fd9\u79cd\u8c03\u6574\u662f\u4e3a\u4e86\u5728\u591a\u5934\u6ce8\u610f\u529b（Multi-Head Attention）\u4e2d\u5c06\u76f8\u5bf9\u4f4d\u7f6e\u5d4c\u5165\u6dfb\u52a0\u5230\u6ce8\u610f\u529b\u56fe(attn)\u3002\u7136\u800c，\u5e76\u975e\u6240\u6709Transformer Block\u90fd\u9700\u8981\u5728\u6ce8\u610f\u529b\u56fe\u4e2d\u5d4c\u5165\u76f8\u5bf9\u4f4d\u7f6e\u4fe1\u606f\u3002 window_unpartition \u51fd\u6570\u7684\u4f5c\u7528\u662f\u5c06\u7ecf\u8fc7\u6ce8\u610f\u529b\u8ba1\u7b97\u7684\u7a97\u53e3\u7279\u5f81\u91cd\u65b0\u7ec4\u5408\u56de\u539f\u59cb\u5c3a\u5bf8(S\u00d7S\u2013>H\u00d7W)\u3002 Hp\u548cWp\u662fS\u7684\u6574\u6570\u500d<\/p>\n

    Multi-Head Attention<\/p>\n

    \u5148\u6765\u770bAttention，\u7ed3\u6784\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

    \"\u56fe\u7247\"<\/p>\n

    Attention\u4e2dq\u3001k\u548cv\u7684\u4f5c\u7528：<\/p>\n

    \"\u56fe\u7247\"<\/p>\n

    \u4ee3\u7801\u5b9e\u73b0\u5982\u4e0b：<\/p>\n

    class<\/span> Attention<\/span>(<\/span>nn.<\/span>Module)<\/span>:<\/span>
    \n """Multi-head Attention block with relative position embeddings."""<\/span>
    \n def<\/span> __init__<\/span>(<\/span>
    \n self,<\/span>
    \n dim:<\/span> int<\/span>,<\/span> # \u8f93\u5165\u901a\u9053\u6570<\/span>
    \n num_heads:<\/span> int<\/span> =<\/span> 8<\/span>,<\/span> # head\u6570\u76ee<\/span>
    \n qkv_bias:<\/span> bool<\/span> =<\/span> True<\/span>,<\/span> # \u662f\u5426\u5728QKV\u7ebf\u6027\u53d8\u6362\u4e2d\u4f7f\u7528\u504f\u7f6e\u9879，\u9ed8\u8ba4\u4e3aTrue<\/span>
    \n use_rel_pos:<\/span> bool<\/span> =<\/span> False<\/span>,<\/span> #\u662f\u5426\u4f7f\u7528\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801，\u9ed8\u8ba4\u4e3aFalse<\/span>
    \n rel_pos_zero_init:<\/span> bool<\/span> =<\/span> True<\/span>,<\/span> #\u5982\u679c\u4f7f\u7528\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801，\u662f\u5426\u4ee5\u96f6\u521d\u59cb\u5316，\u9ed8\u8ba4\u4e3aTrue<\/span>
    \n input_size:<\/span> Optional[<\/span>Tuple[<\/span>int<\/span>,<\/span> int<\/span>]<\/span>]<\/span> =<\/span> None<\/span>,<\/span> # \u53ef\u9009\u53c2\u6570，\u7528\u4e8e\u6307\u5b9a\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801\u7684\u5c3a\u5bf8，\u53ea\u6709\u5728\u4f7f\u7528\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801\u65f6\u624d\u9700\u8981<\/span>
    \n )<\/span> –<\/span>><\/span> None<\/span>:<\/span>
    \n super<\/span>(<\/span>)<\/span>.<\/span>__init__(<\/span>)<\/span>
    \n self.<\/span>num_heads =<\/span> num_heads #\u8f93\u5165head\u6570\u76ee<\/span>
    \n head_dim =<\/span> dim \/\/<\/span> num_heads #\u6bcf\u4e2ahead\u7ef4\u5ea6<\/span>
    \n self.<\/span>scale =<\/span> head_dim**<\/span>–<\/span>0.5<\/span> #\u7528\u4e8e\u7f29\u653e\u6ce8\u610f\u529b\u5f97\u5206\u7684\u56e0\u5b50，\u4ee5\u907f\u514d\u6570\u503c\u6ea2\u51fa，\u53d6\u503c\u4e3ahead_dim\u7684\u5e73\u65b9\u6839\u7684\u5012\u6570<\/span>
    \n #\u4e00\u4e2a\u5168\u8fde\u63a5\u5c42（nn.Linear），\u5c06\u8f93\u5165\u6620\u5c04\u5230Q\u3001K\u3001V\u7684\u7ec4\u5408<\/span>
    \n self.<\/span>qkv =<\/span> nn.<\/span>Linear(<\/span>dim,<\/span> dim *<\/span> 3<\/span>,<\/span> bias=<\/span>qkv_bias)<\/span>
    \n # \u4e00\u4e2a\u5168\u8fde\u63a5\u5c42，\u7528\u4e8e\u5c06\u6ce8\u610f\u529b\u673a\u5236\u7684\u8f93\u51fa\u6295\u5f71\u56de\u539f\u59cb\u7ef4\u5ea6<\/span>
    \n self.<\/span>proj =<\/span> nn.<\/span>Linear(<\/span>dim,<\/span> dim)<\/span>
    \n self.<\/span>use_rel_pos =<\/span> use_rel_pos
    \n if<\/span> self.<\/span>use_rel_pos:<\/span> # \u4f7f\u7528\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801<\/span>
    \n assert<\/span> (<\/span>
    \n input_size is<\/span> not<\/span> None<\/span>
    \n )<\/span>,<\/span> "Input size must be provided if using relative positional encoding."<\/span>
    \n # \u521d\u59cb\u5316\u6c34\u5e73\u65b9\u5411(rel_pos_h)\u548c\u5782\u76f4\u65b9\u5411(rel_pos_w)\u7684\u76f8\u5bf9\u4f4d\u7f6e\u5d4c\u5165<\/span>
    \n # 2S-1,Epos<\/span>
    \n # \u8f93\u5165\u5c3a\u5bf8\u4e3a(H, W)，\u5219\u6c34\u5e73\u65b9\u5411\u7684\u4f4d\u7f6e\u5d4c\u5165\u957f\u5ea6\u4e3a2*H-1，\u5782\u76f4\u65b9\u5411\u7684\u4f4d\u7f6e\u5d4c\u5165\u957f\u5ea6\u4e3a2*W-1<\/span>
    \n # \u6bcf\u4e2a\u4f4d\u7f6e\u5d4c\u5165\u7684\u7ef4\u5ea6\u4e3ahead_dim<\/span>
    \n # \u8fd9\u4e9b\u4f4d\u7f6e\u5d4c\u5165\u4ee5\u6a21\u578b\u53c2\u6570\u7684\u5f62\u5f0f\u5b9a\u4e49(nn.Parameter)，\u610f\u5473\u7740\u5b83\u4eec\u4f1a\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u88ab\u5b66\u4e60\u548c\u66f4\u65b0<\/span>
    \n self.<\/span>rel_pos_h =<\/span> nn.<\/span>Parameter(<\/span>torch.<\/span>zeros(<\/span>2<\/span> *<\/span> input_size[<\/span>0<\/span>]<\/span> –<\/span> 1<\/span>,<\/span> head_dim)<\/span>)<\/span>
    \n self.<\/span>rel_pos_w =<\/span> nn.<\/span>Parameter(<\/span>torch.<\/span>zeros(<\/span>2<\/span> *<\/span> input_size[<\/span>1<\/span>]<\/span> –<\/span> 1<\/span>,<\/span> head_dim)<\/span>)<\/span><\/p>\n

    def<\/span> forward<\/span>(<\/span>self,<\/span> x:<\/span> torch.<\/span>Tensor)<\/span> –<\/span>><\/span> torch.<\/span>Tensor:<\/span>
    \n # \u8f93\u5165\u5f20\u91cfx\u7684\u5f62\u72b6\u4e3a(B, H, W, C)，\u5176\u4e2dB\u662f\u6279\u6b21\u5927\u5c0f，H\u548cW\u662f\u9ad8\u5ea6\u548c\u5bbd\u5ea6，C\u662f\u901a\u9053\u6570（\u5373dim）<\/span>
    \n B,<\/span> H,<\/span> W,<\/span> _ =<\/span> x.<\/span>shape
    \n # \u4f7f\u7528qkv\u5c42\u5c06x\u8f6c\u6362\u4e3aQ\u3001K\u3001V\u7684\u7ec4\u5408，\u7136\u540e\u901a\u8fc7\u91cd\u5851\u548c\u91cd\u65b0\u6392\u5217\u6765\u51c6\u5907\u591a\u5934\u6ce8\u610f\u529b\u8ba1\u7b97<\/span>
    \n # qkv with shape (3, B, nHead, H * W, C)<\/span>
    \n qkv =<\/span> self.<\/span>qkv(<\/span>x)<\/span>.<\/span>reshape(<\/span>B,<\/span> H *<\/span> W,<\/span> 3<\/span>,<\/span> self.<\/span>num_heads,<\/span> –<\/span>1<\/span>)<\/span>.<\/span>permute(<\/span>2<\/span>,<\/span> 0<\/span>,<\/span> 3<\/span>,<\/span> 1<\/span>,<\/span> 4<\/span>)<\/span>
    \n # q, k, v with shape (B * nHead, H * W, C)<\/span>
    \n q,<\/span> k,<\/span> v =<\/span> qkv.<\/span>reshape(<\/span>3<\/span>,<\/span> B *<\/span> self.<\/span>num_heads,<\/span> H *<\/span> W,<\/span> –<\/span>1<\/span>)<\/span>.<\/span>unbind(<\/span>0<\/span>)<\/span>
    \n # attn with shape (B * nHead, H * W, H * W)<\/span>
    \n # \u8ba1\u7b97\u6ce8\u610f\u529b\u5206\u6570<\/span>
    \n # q * self.scale: q\u662f\u67e5\u8be2\u5411\u91cf（query vectors），\u5f62\u72b6\u4e3a(B * nHead, H * W, C)，\u5176\u4e2dB\u662f\u6279\u6b21\u5927\u5c0f，nHead\u662f\u6ce8\u610f\u529b\u5934\u7684\u6570\u91cf，H * W\u662f\u5e8f\u5217\u7684\u957f\u5ea6，C\u662f\u6bcf\u4e2a\u4f4d\u7f6e\u7684\u7279\u5f81\u7ef4\u5ea6<\/span>
    \n # self.scale\u662f\u7528\u4e8e\u7f29\u653e\u6ce8\u610f\u529b\u5206\u6570\u7684\u56e0\u5b50，\u901a\u5e38\u53d6head_dim\u7684\u5e73\u65b9\u6839\u7684\u5012\u6570，\u4ee5\u9632\u6b62\u6570\u503c\u8fc7\u5927<\/span>
    \n # \u4e58\u4ee5self.scale\u662f\u4e3a\u4e86\u7a33\u5b9a\u8ba1\u7b97\u5e76\u9632\u6b62\u68af\u5ea6\u6d88\u5931<\/span>
    \n # k.transpose(-2, -1): k\u662f\u952e\u5411\u91cf（key vectors），\u5f62\u72b6\u4e0eq\u76f8\u540c\u3002transpose(-2, -1)\u662f\u5bf9k\u8fdb\u884c\u8f6c\u7f6e\u64cd\u4f5c，\u5373\u5c06\u6700\u540e\u4e00\u4e2a\u548c\u5012\u6570\u7b2c\u4e8c\u4e2a\u7ef4\u5ea6\u4e92\u6362，\u76ee\u7684\u662f\u8ba9q\u548ck\u5728\u8ba1\u7b97\u70b9\u79ef\u65f6\u7684\u7ef4\u5ea6\u5339\u914d\u3002\u8f6c\u7f6e\u540e\u7684k\u5f62\u72b6\u53d8\u4e3a(B * nHead, C, H * W)<\/span>
    \n # \u5c06q\u548c\u8f6c\u7f6e\u540e\u7684k\u8fdb\u884c\u77e9\u9635\u4e58\u6cd5\u3002\u8ba1\u7b97\u6bcf\u4e2a\u67e5\u8be2\u4f4d\u7f6eq\u4e0e\u6240\u6709\u952e\u4f4d\u7f6ek\u7684\u70b9\u79ef，\u751f\u6210\u4e00\u4e2a\u5f62\u72b6\u4e3a(B * nHead, H * W, H * W)\u7684\u6ce8\u610f\u529b\u5206\u6570\u77e9\u9635attn\u3002\u6bcf\u4e2a\u4f4d\u7f6ei\u548cj\u7684\u6ce8\u610f\u529b\u5206\u6570\u8868\u793aq_i\u4e0ek_j\u7684\u76f8\u4f3c\u5ea6<\/span>
    \n attn =<\/span> (<\/span>q *<\/span> self.<\/span>scale)<\/span> @ k.<\/span>transpose(<\/span>–<\/span>2<\/span>,<\/span> –<\/span>1<\/span>)<\/span>
    \n # \u5982\u679c\u542f\u7528\u4e86\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801<\/span>
    \n if<\/span> self.<\/span>use_rel_pos:<\/span>
    \n # (H, W)\u4ee3\u8868\u8f93\u5165\u5e8f\u5217\u7684\u5c3a\u5bf8，\u8fd9\u91cc\u5047\u8bbeH\u548cW\u662f\u76f8\u7b49\u7684(S\u00d7S)，\u5373\u8f93\u5165\u662f\u4e00\u4e2a\u6b63\u65b9\u5f62\u7f51\u683c（\u4f8b\u5982，\u56fe\u50cf\u7684\u50cf\u7d20\u7f51\u683c）<\/span>
    \n # attn: \u4e0a\u8ff0\u8ba1\u7b97\u5f97\u5230\u7684\u6ce8\u610f\u529b\u5206\u6570\u77e9\u9635，\u5f62\u72b6\u4e3a(B * nHead, H * W, H * W)<\/span>
    \n # q: \u67e5\u8be2\u5411\u91cf，\u5f62\u72b6\u4e3a(B * nHead, H * W, C)<\/span>
    \n # self.rel_pos_h\u548cself.rel_pos_w: \u5206\u522b\u8868\u793a\u6c34\u5e73\u548c\u5782\u76f4\u65b9\u5411\u4e0a\u7684\u76f8\u5bf9\u4f4d\u7f6e\u5d4c\u5165，\u5f62\u72b6\u5206\u522b\u4e3a(2 * S – 1, head_dim)<\/span>
    \n # (H, W): \u8f93\u5165\u5e8f\u5217\u7684\u5c3a\u5bf8，\u7528\u4e8e\u6307\u5bfc\u76f8\u5bf9\u4f4d\u7f6e\u5d4c\u5165\u7684\u8ba1\u7b97<\/span>
    \n attn =<\/span> add_decomposed_rel_pos(<\/span>attn,<\/span> q,<\/span> self.<\/span>rel_pos_h,<\/span> self.<\/span>rel_pos_w,<\/span> (<\/span>H,<\/span> W)<\/span>,<\/span> (<\/span>H,<\/span> W)<\/span>)<\/span>
    \n # \u751f\u6210\u7684\u6ce8\u610f\u529b\u5206\u6570\u77e9\u9635attn\u968f\u540e\u4f1a\u7ecf\u8fc7Softmax\u51fd\u6570，\u5c06\u6bcf\u4e2a\u4f4d\u7f6e\u7684\u5206\u6570\u5f52\u4e00\u5316\u5230[0, 1]\u533a\u95f4，\u5f62\u6210\u4e00\u4e2a\u6982\u7387\u5206\u5e03<\/span>
    \n attn =<\/span> attn.<\/span>softmax(<\/span>dim=<\/span>–<\/span>1<\/span>)<\/span>
    \n # \u52a0\u6743\u6c42\u548c: <\/span>
    \n # \u4f7f\u7528attn @ v\u8ba1\u7b97\u52a0\u6743\u548c，\u5176\u4e2d@\u8868\u793a\u77e9\u9635\u4e58\u6cd5,v\u662f\u503c\u5411\u91cf（value vectors），\u5f62\u72b6\u4e3a(B * nHead, H * W, C)<\/span>
    \n # \u6ce8\u610f\u529b\u6743\u91cd\u77e9\u9635attn（\u5f62\u72b6\u4e3a(B * nHead, H * W, H * W)）\u4e0ev\u6309\u5143\u7d20\u76f8\u4e58\u540e，\u518d\u8fdb\u884c\u77e9\u9635\u4e58\u6cd5，\u5f97\u5230\u52a0\u6743\u540e\u7684\u503c\u5411\u91cf，\u5f62\u72b6\u4e3a(B * nHead, H * W, C)<\/span>
    \n # \u4f7f\u7528.view()\u5c06\u52a0\u6743\u540e\u7684\u503c\u5411\u91cf\u91cd\u5851\u4e3a(B, self.num_heads, H, W, -1)，\u7136\u540e\u4f7f\u7528.permute(0, 2, 3, 1, 4)\u8fdb\u884c\u91cd\u6392，\u5c06self.num_heads\u79fb\u52a8\u5230\u7b2c\u56db\u4e2a\u7ef4\u5ea6\u3002\u6700\u540e，\u4f7f\u7528.reshape(B, H, W, -1)\u5c06\u7ed3\u679c\u8fdb\u4e00\u6b65\u91cd\u5851\u4e3a(B, H, W, -1)，\u4e0e\u8f93\u5165\u5f20\u91cf\u7684\u5f62\u72b6\u4e00\u81f4，\u4f46\u4fdd\u7559\u4e86\u591a\u5934\u6ce8\u610f\u529b\u7684\u8f93\u51fa<\/span>
    \n x =<\/span> (<\/span>attn @ v)<\/span>.<\/span>view(<\/span>B,<\/span> self.<\/span>num_heads,<\/span> H,<\/span> W,<\/span> –<\/span>1<\/span>)<\/span>.<\/span>permute(<\/span>0<\/span>,<\/span> 2<\/span>,<\/span> 3<\/span>,<\/span> 1<\/span>,<\/span> 4<\/span>)<\/span>.<\/span>reshape(<\/span>B,<\/span> H,<\/span> W,<\/span> –<\/span>1<\/span>)<\/span>
    \n # \u4f7f\u7528self.proj（\u4e00\u4e2a\u5168\u8fde\u63a5\u5c42，\u5f62\u72b6\u4e3a(dim, dim)）\u5bf9\u4e0a\u8ff0\u5904\u7406\u540e\u7684\u5f20\u91cf\u8fdb\u884c\u7ebf\u6027\u6295\u5f71，\u4ee5\u5c06\u5176\u6295\u5f71\u56de\u539f\u59cb\u7684\u7279\u5f81\u7ef4\u5ea6<\/span>
    \n x =<\/span> self.<\/span>proj(<\/span>x)<\/span>
    \n # \u6700\u7ec8，\u8fd4\u56de\u7ecf\u8fc7\u7ebf\u6027\u6295\u5f71\u7684\u5f20\u91cfx\u4f5c\u4e3a\u6ce8\u610f\u529b\u6a21\u5757\u7684\u8f93\u51fa<\/span>
    \n return<\/span> x<\/p>\n

    \u5728\u591a\u5934\u6ce8\u610f\u529b（Multi-Head Attention）\u6a21\u5757\u4e2d，\u8f93\u5165\u7279\u5f81F(N\u00d7E)\u8868\u793a\u4e00\u4e2a\u5e8f\u5217，\u5176\u4e2dN\u662f\u5e8f\u5217\u4e2d\u7684\u5143\u7d20\u6570\u91cf，E\u662f\u6bcf\u4e2a\u5143\u7d20\u7684\u7279\u5f81\u7ef4\u5ea6\u3002\u5177\u4f53\u6d41\u7a0b\u5982\u4e0b\u3002<\/p>\n

  • \u9996\u5148\u5c06\u6bcf\u4e2atoken\u7684qkv\u7279\u5f81\u7ef4\u5ea6embed_dim\u5747\u62c6\u5206\u5230\u6bcf\u4e2ahead\u4e0a\u3002<\/li>\n

    \"\u56fe\u7247\"<\/p>\n

  • \u6bcf\u4e2ahead\u5206\u522b\u901a\u8fc7q\u548ck\u8ba1\u7b97\u5f97\u5230\u6743\u91cdw，\u6743\u91cdw\u548cv\u5f97\u5230\u8f93\u51faoutput，\u5408\u5e76\u6240\u6709head\u7684output\u5f97\u5230\u6700\u7ec8\u7684output\u3002<\/li>\n

    \"\u56fe\u7247\"<\/p>\n

    get_rel_pos\u7528\u4e8e\u8ba1\u7b97\u67e5\u8be2（query）\u548c\u952e（key）\u4e4b\u95f4\u5728\u4e8c\u7ef4\u7a7a\u95f4\u4e2d\u7684\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801，\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

    \"\u56fe\u7247\"<\/p>\n

    \u5b9e\u73b0\u4ee3\u7801：<\/p>\n

    def<\/span> get_rel_pos<\/span>(<\/span>q_size:<\/span> int<\/span>,<\/span> k_size:<\/span> int<\/span>,<\/span> rel_pos:<\/span> torch.<\/span>Tensor)<\/span> –<\/span>><\/span> torch.<\/span>Tensor:<\/span>
    \n # \u8868\u793a\u67e5\u8be2（query）\u548c\u952e（key）\u5728\u4e8c\u7ef4\u7a7a\u95f4\u4e2d\u7684\u6700\u5927\u76f8\u5bf9\u8ddd\u79bb<\/span>
    \n # max(q_size, k_size)：\u53d6\u67e5\u8be2\u7684\u5bbd\u5ea6q_size\u548c\u952e\u7684\u5bbd\u5ea6k_size\u4e2d\u7684\u8f83\u5927\u503c<\/span>
    \n # \u5982\u679cq_size\u548ck_size\u90fd\u4e3aS，\u5219\u6700\u5927\u7684\u6b63\u5411\u8ddd\u79bb\u662fS-1，\u6700\u5927\u7684\u8d1f\u5411\u8ddd\u79bb\u4e5f\u662fS-1，\u6240\u4ee5\u603b\u7684\u6700\u5927\u8ddd\u79bb\u662f2 * S<\/span>
    \n # – 1：\u51cf\u53bb1\u662f\u56e0\u4e3a\u5728\u8ba1\u7b97\u76f8\u5bf9\u4f4d\u7f6e\u65f6，0\u88ab\u5305\u542b\u5728\u5185，\u6240\u4ee5\u6700\u5927\u8ddd\u79bb\u662f2 * S – 1<\/span>
    \n max_rel_dist =<\/span> int<\/span>(<\/span>2<\/span> *<\/span> max<\/span>(<\/span>q_size,<\/span> k_size)<\/span> –<\/span> 1<\/span>)<\/span>
    \n # \u5982\u679crel_pos\u7684\u5f62\u72b6\u7684\u7b2c0\u4e2a\u7ef4\u5ea6（\u5373\u957f\u5ea6）\u4e0d\u7b49\u4e8emax_rel_dist，\u8bf4\u660e\u9700\u8981\u8fdb\u884c\u63d2\u503c<\/span>
    \n if<\/span> rel_pos.<\/span>shape[<\/span>0<\/span>]<\/span> !=<\/span> max_rel_dist:<\/span>
    \n # \u4f7f\u7528F.interpolate\u8fdb\u884c\u7ebf\u6027\u63d2\u503c<\/span>
    \n rel_pos_resized =<\/span> F.<\/span>interpolate(<\/span>
    \n # 1,N,Ep –> 1,Ep,N –> 1,Ep,2S-1<\/span>
    \n # \u5c06rel_pos\u91cd\u5851\u4e3a(1, N, Ep)，\u5176\u4e2dN\u662f\u539f\u59cb\u7684\u957f\u5ea6，Ep\u662f\u6bcf\u4e2a\u4f4d\u7f6e\u7f16\u7801\u7684\u7279\u5f81\u7ef4\u5ea6<\/span>
    \n # \u901a\u8fc7permute(0, 2, 1)\u8fdb\u884c\u8f6c\u7f6e，\u4f7f\u5176\u5f62\u72b6\u53d8\u4e3a(1, Ep, N)<\/span>
    \n rel_pos.<\/span>reshape(<\/span>1<\/span>,<\/span> rel_pos.<\/span>shape[<\/span>0<\/span>]<\/span>,<\/span> –<\/span>1<\/span>)<\/span>.<\/span>permute(<\/span>0<\/span>,<\/span> 2<\/span>,<\/span> 1<\/span>)<\/span>,<\/span>
    \n # \u8bbe\u7f6e\u63d2\u503c\u7684\u76ee\u6807\u957f\u5ea6\u4e3amax_rel_dist<\/span>
    \n size=<\/span>max_rel_dist,<\/span>
    \n # \u6307\u5b9a\u63d2\u503c\u65b9\u6cd5\u4e3a\u7ebf\u6027\u63d2\u503c<\/span>
    \n mode=<\/span>"linear"<\/span>,<\/span>
    \n )<\/span>
    \n # Ep,2S-1 –> 2S-1,Ep<\/span>
    \n # \u63d2\u503c\u540e\u7684rel_pos\u5f62\u72b6\u4e3a(1, Ep, max_rel_dist)，\u901a\u8fc7reshape(-1, max_rel_dist)\u5c06\u5176\u91cd\u5851\u4e3a(Ep, max_rel_dist)<\/span>
    \n # \u518d\u901a\u8fc7permute(1, 0)\u8f6c\u7f6e\u4e3a(max_rel_dist, Ep)<\/span>
    \n rel_pos_resized =<\/span> rel_pos_resized.<\/span>reshape(<\/span>–<\/span>1<\/span>,<\/span> max_rel_dist)<\/span>.<\/span>permute(<\/span>1<\/span>,<\/span> 0<\/span>)<\/span>
    \n else<\/span>:<\/span>
    \n # \u5982\u679crel_pos\u7684\u957f\u5ea6\u4e0emax_rel_dist\u76f8\u7b49，\u8bf4\u660e\u5df2\u7ecf\u8db3\u591f\u8986\u76d6\u6240\u6709\u53ef\u80fd\u7684\u76f8\u5bf9\u4f4d\u7f6e，\u56e0\u6b64\u76f4\u63a5\u4f7f\u7528rel_pos，\u4e0d\u8fdb\u884c\u4efb\u4f55\u5904\u7406<\/span>
    \n rel_pos_resized =<\/span> rel_pos<\/p>\n

    # \u5982\u679cq\u548ck\u957f\u5ea6\u503c\u4e0d\u540c，\u5219\u7528\u77ed\u8fb9\u957f\u5ea6\u7f29\u653e\u5750\u6807<\/span>
    \n # \u521b\u5efa\u67e5\u8be2\u5750\u6807q_coords<\/span>
    \n # torch.arange(q_size)\u751f\u6210\u4e00\u4e2a\u4ece0\u5230q_size – 1\u7684\u6574\u6570\u5e8f\u5217，\u8868\u793aq_size\u4e2a\u4f4d\u7f6e<\/span>
    \n # [:, None]\u5728\u5e8f\u5217\u672b\u5c3e\u6dfb\u52a0\u4e00\u4e2a\u7ef4\u5ea6，\u4f7f\u5176\u5f62\u72b6\u4e3a(q_size, 1)，\u8fd9\u6837\u53ef\u4ee5\u65b9\u4fbf\u4e0e\u4e00\u4e2a\u6807\u91cf\u8fdb\u884c\u9010\u5143\u7d20\u4e58\u6cd5<\/span>
    \n # max(k_size \/ q_size, 1.0)\u8ba1\u7b97\u6bd4\u4f8b\u56e0\u5b50，\u5982\u679ck_size\u5927\u4e8eq_size，\u5219\u4f7f\u7528k_size \/ q_size，\u5426\u5219\u4f7f\u75281.0<\/span>
    \n # \u8fd9\u786e\u4fdd\u4e86\u5728q_size\u5c0f\u4e8ek_size\u7684\u60c5\u51b5\u4e0b，q_coords\u7684\u5750\u6807\u4f1a\u88ab\u9002\u5f53\u653e\u5927，\u4ee5\u5339\u914dk_coords\u7684\u5c3a\u5ea6<\/span>
    \n q_coords =<\/span> torch.<\/span>arange(<\/span>q_size)<\/span>[<\/span>:<\/span>,<\/span> None<\/span>]<\/span> *<\/span> max<\/span>(<\/span>k_size \/<\/span> q_size,<\/span> 1.0<\/span>)<\/span>
    \n # \u521b\u5efa\u952e\u5750\u6807k_coords<\/span>
    \n k_coords =<\/span> torch.<\/span>arange(<\/span>k_size)<\/span>[<\/span>None<\/span>,<\/span> :<\/span>]<\/span> *<\/span> max<\/span>(<\/span>q_size \/<\/span> k_size,<\/span> 1.0<\/span>)<\/span>
    \n # S,S<\/span>
    \n # \u8ba1\u7b97\u4e86\u67e5\u8be2（query）\u548c\u952e（key）\u5728\u4e8c\u7ef4\u7a7a\u95f4\u4e2d\u7684\u76f8\u5bf9\u5750\u6807relative_coords<\/span>
    \n # (q_coords – k_coords):\u6bcf\u4e2a\u67e5\u8be2\u4f4d\u7f6e\u76f8\u5bf9\u4e8e\u6bcf\u4e2a\u952e\u4f4d\u7f6e\u7684\u6c34\u5e73\u8ddd\u79bb<\/span>
    \n # (k_size – 1) * max(q_size \/ k_size, 1.0)：\u8ba1\u7b97\u4e86\u4e00\u4e2a\u504f\u79fb\u91cf，\u7528\u4e8e\u786e\u4fdd\u76f8\u5bf9\u5750\u6807\u5728\u6b63\u786e\u7684\u8303\u56f4\u5185<\/span>
    \n # (q_coords – k_coords) + (k_size – 1) * max(q_size \/ k_size, 1.0)：\u5c06\u8ba1\u7b97\u51fa\u7684\u5dee\u503c\u548c\u504f\u79fb\u91cf\u76f8\u52a0，\u5f97\u5230\u6700\u7ec8\u7684\u76f8\u5bf9\u5750\u6807relative_coords<\/span>
    \n relative_coords =<\/span> (<\/span>q_coords –<\/span> k_coords)<\/span> +<\/span> (<\/span>k_size –<\/span> 1<\/span>)<\/span> *<\/span> max<\/span>(<\/span>q_size \/<\/span> k_size,<\/span> 1.0<\/span>)<\/span><\/p>\n

    # tensor\u7d22\u5f15\u662ftensor\u65f6,\u5373tensor1[tensor2]<\/span>
    \n # \u5047\u8bbetensor2\u67d0\u4e2a\u5177\u4f53\u4f4d\u7f6e\u503c\u662f2,\u5219tensor1[2]\u4f4d\u7f6e\u7684tensor1\u5207\u7247\u66ff\u6362tensor2\u4e2d\u76842<\/span>
    \n # tensor1->shape 5,5,3 tensor2->shape 2,2,3 tensor1\u5207\u7247->shape 5,3 tensor1[tensor2]->shape 2,2,3,5,3<\/span>
    \n # tensor1->shape 5,5 tensor2->shape 3,2,3 tensor1\u5207\u7247->shape 5 tensor1[tensor2]->shape 3,2,3,5<\/span><\/p>\n

    # 2S-1,Ep–>S,S,Ep<\/span>
    \n return<\/span> rel_pos_resized[<\/span>relative_coords.<\/span>long<\/span>(<\/span>)<\/span>]<\/span><\/p>\n

    add_decomposed_rel_pos\u4e3aatten\u6ce8\u610f\u529b\u7279\u5f81\u6dfb\u52a0\u76f8\u5bf9\u4f4d\u7f6e\u7684\u5d4c\u5165\u7279\u5f81，\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

    \"\u56fe\u7247\"<\/p>\n

    def<\/span> add_decomposed_rel_pos<\/span>(<\/span>
    \n # \u6ce8\u610f\u529b\u5206\u6570\u77e9\u9635<\/span>
    \n attn:<\/span> torch.<\/span>Tensor,<\/span>
    \n q:<\/span> torch.<\/span>Tensor,<\/span>
    \n rel_pos_h:<\/span> torch.<\/span>Tensor,<\/span>
    \n rel_pos_w:<\/span> torch.<\/span>Tensor,<\/span>
    \n q_size:<\/span> Tuple[<\/span>int<\/span>,<\/span> int<\/span>]<\/span>,<\/span>
    \n k_size:<\/span> Tuple[<\/span>int<\/span>,<\/span> int<\/span>]<\/span>,<\/span>
    \n)<\/span> –<\/span>><\/span> torch.<\/span>Tensor:<\/span>
    \n # S,S<\/span>
    \n q_h,<\/span> q_w =<\/span> q_size
    \n k_h,<\/span> k_w =<\/span> k_size
    \n # rel_pos_h -> 2S-1\u00d7Epos<\/span>
    \n # \u67e5\u8be2（query）\u548c\u952e（key）\u5728\u9ad8\u5ea6\u65b9\u5411\u4e0a\u7684\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801<\/span>
    \n Rh =<\/span> get_rel_pos(<\/span>q_h,<\/span> k_h,<\/span> rel_pos_h)<\/span>
    \n # \u67e5\u8be2（query）\u548c\u952e（key）\u5728\u5bbd\u5ea6\u65b9\u5411\u4e0a\u7684\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801<\/span>
    \n Rw =<\/span> get_rel_pos(<\/span>q_w,<\/span> k_w,<\/span> rel_pos_w)<\/span>
    \n # \u91cd\u5851q\u4e3a(B, q_h, q_w, dim)<\/span>
    \n B,<\/span> _,<\/span> dim =<\/span> q.<\/span>shape
    \n r_q =<\/span> q.<\/span>reshape(<\/span>B,<\/span> q_h,<\/span> q_w,<\/span> dim)<\/span>
    \n # \u8ba1\u7b97\u76f8\u5bf9\u4f4d\u7f6e\u52a0\u6743<\/span>
    \n # \u8ba1\u7b97rel_h\u548crel_w，\u8fd9\u4e24\u4e2a\u5f20\u91cf\u8868\u793a\u5728\u6bcf\u4e2a\u4f4d\u7f6e\u4e0a，\u67e5\u8be2\u4e0e\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801\u7684\u52a0\u6743\u548c<\/span>
    \n # B,q_h,q_w,k_h<\/span>
    \n rel_h =<\/span> torch.<\/span>einsum(<\/span>"bhwc,hkc->bhwk"<\/span>,<\/span> r_q,<\/span> Rh)<\/span>
    \n # B,q_h, q_w, k_w<\/span>
    \n rel_w =<\/span> torch.<\/span>einsum(<\/span>"bhwc,wkc->bhwk"<\/span>,<\/span> r_q,<\/span> Rw)<\/span>
    \n # \u5408\u5e76\u6ce8\u610f\u529b\u5206\u6570\u548c\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801<\/span>
    \n # \u5c06attn\u91cd\u5851\u4e3a(B, q_h, q_w, k_h, k_w)，\u7136\u540e\u4e0erel_h\u548crel_w\u6309\u5143\u7d20\u76f8\u52a0<\/span>
    \n # \u5c06attn\u91cd\u5851\u4e3a(B, q_h, q_w, k_h, k_w)，\u7136\u540e\u4e0erel_h\u548crel_w\u6309\u5143\u7d20\u76f8\u52a0<\/span>
    \n attn =<\/span> (<\/span>
    \n # B,q_h, q_w, k_h, k_w<\/span>
    \n attn.<\/span>view(<\/span>B,<\/span> q_h,<\/span> q_w,<\/span> k_h,<\/span> k_w)<\/span> +<\/span> rel_h[<\/span>:<\/span>,<\/span> :<\/span>,<\/span> :<\/span>,<\/span> :<\/span>,<\/span> None<\/span>]<\/span> +<\/span> rel_w[<\/span>:<\/span>,<\/span> :<\/span>,<\/span> :<\/span>,<\/span> None<\/span>,<\/span> :<\/span>]<\/span>
    \n )<\/span>.<\/span>view(<\/span>B,<\/span> q_h *<\/span> q_w,<\/span> k_h *<\/span> k_w)<\/span>
    \n return<\/span> attn<\/p>\n

    Multi-Head Attention\u6a21\u5757\u4e3a\u6ce8\u610f\u529b\u7279\u5f81\u5d4c\u5165\u4e86\u76f8\u5bf9\u4f4d\u7f6e\u7279\u5f81(add_decomposed_rel_pos)：<\/p>\n

    \"\u56fe\u7247\"<\/p>\n

    Neck Convolution<\/h5>\n

    \u6700\u540e，\u901a\u8fc7\u4e24\u5c42\u5377\u79ef（Neck）\u5c06\u901a\u9053\u6570\u964d\u4f4e\u81f3256，\u751f\u6210\u6700\u7ec8\u7684Image Embedding\u3002\u5176\u7ed3\u6784\u56fe\u5982\u4e0b\u6240\u793a\u3002<\/p>\n

    \"\u56fe\u7247\"<\/p>\n

    \u4ee3\u7801\u5b9e\u73b0\u5982\u4e0b：<\/p>\n

    # neck: nn.Sequential，\u5b83\u5305\u542b\u4e24\u4e2a\u5377\u79ef\u5c42\u548c\u4e24\u4e2aLayerNorm2d）<\/span>
    \nself.<\/span>neck =<\/span> nn.<\/span>Sequential(<\/span>
    \n # 1×1\u7684\u5377\u79ef\u5c42，\u7528\u4e8e\u5c06\u8f93\u5165\u901a\u9053\u6570\u4eceembed_dim\u51cf\u5c0f\u5230out_chans<\/span>
    \n # 1×1\u5377\u79ef\u4e3b\u8981\u7528\u4e8e\u901a\u9053\u95f4\u7684\u4fe1\u606f\u878d\u5408，\u800c\u4e0d\u6539\u53d8\u7279\u5f81\u56fe\u7684\u7a7a\u95f4\u5c3a\u5bf8<\/span>
    \n nn.<\/span>Conv2d(<\/span>
    \n embed_dim,<\/span>
    \n out_chans,<\/span>
    \n kernel_size=<\/span>1<\/span>,<\/span>
    \n # \u4e0d\u4f7f\u7528\u504f\u7f6e\u9879<\/span>
    \n bias=<\/span>False<\/span>,<\/span>
    \n )<\/span>,<\/span>
    \n # \u5f52\u4e00\u5316\u5c42，\u7528\u4e8e\u89c4\u8303\u5316\u8f93\u51fa\u901a\u9053\u7684\u5747\u503c\u548c\u65b9\u5dee，\u63d0\u9ad8\u6a21\u578b\u7684\u7a33\u5b9a\u6027\u548c\u6536\u655b\u901f\u5ea6<\/span>
    \n # out_chans：\u5f52\u4e00\u5316\u5c42\u7684\u901a\u9053\u6570<\/span>
    \n LayerNorm2d(<\/span>out_chans)<\/span>,<\/span>
    \n # 3×3\u7684\u5377\u79ef\u5c42<\/span>
    \n nn.<\/span>Conv2d(<\/span>
    \n # \u4f7f\u7528out_chans\u4f5c\u4e3a\u8f93\u5165\u548c\u8f93\u51fa\u901a\u9053\u6570<\/span>
    \n out_chans,<\/span>
    \n out_chans,<\/span>
    \n kernel_size=<\/span>3<\/span>,<\/span>
    \n # \u8f93\u5165\u548c\u8f93\u51fa\u7684\u7279\u5f81\u56fe\u5c3a\u5bf8\u4fdd\u6301\u4e0d\u53d8，\u907f\u514d\u5c3a\u5bf8\u6536\u7f29<\/span>
    \n padding=<\/span>1<\/span>,<\/span>
    \n # \u4e0d\u4f7f\u7528\u504f\u7f6e<\/span>
    \n bias=<\/span>False<\/span>,<\/span>
    \n )<\/span>,<\/span>
    \n # \u7b2c\u4e8c\u4e2a\u5f52\u4e00\u5316\u5c42，\u518d\u6b21\u5bf9\u8f93\u51fa\u8fdb\u884c\u89c4\u8303\u5316<\/span>
    \n LayerNorm2d(<\/span>out_chans)<\/span>,<\/span>
    \n)<\/span><\/p>\n

    # \u5f52\u4e00\u5316<\/span>
    \nclass<\/span> LayerNorm2d<\/span>(<\/span>nn.<\/span>Module)<\/span>:<\/span>
    \n def<\/span> __init__<\/span>(<\/span>self,<\/span> num_channels:<\/span> int<\/span>,<\/span> eps:<\/span> float<\/span> =<\/span> 1e-6<\/span>)<\/span> –<\/span>><\/span> None<\/span>:<\/span>
    \n super<\/span>(<\/span>)<\/span>.<\/span>__init__(<\/span>)<\/span>
    \n # \u521b\u5efa\u4e86\u4e24\u4e2a\u53ef\u5b66\u4e60\u7684\u53c2\u6570：weight\u548cbias<\/span>
    \n # weight\u521d\u59cb\u5316\u4e3a\u51681，bias\u521d\u59cb\u5316\u4e3a\u51680<\/span>
    \n self.<\/span>weight =<\/span> nn.<\/span>Parameter(<\/span>torch.<\/span>ones(<\/span>num_channels)<\/span>)<\/span>
    \n self.<\/span>bias =<\/span> nn.<\/span>Parameter(<\/span>torch.<\/span>zeros(<\/span>num_channels)<\/span>)<\/span>
    \n self.<\/span>eps =<\/span> eps<\/p>\n

    def<\/span> forward<\/span>(<\/span>self,<\/span> x:<\/span> torch.<\/span>Tensor)<\/span> –<\/span>><\/span> torch.<\/span>Tensor:<\/span>
    \n # \u6cbf\u7740\u901a\u9053\u7ef4\u5ea6\u6c42\u5747\u503c，keepdim=True\u4fdd\u7559\u7ef4\u5ea6，\u4f7f\u5f97u\u7684\u5f62\u72b6\u4e0ex\u76f8\u540c，\u9664\u4e86\u901a\u9053\u7ef4\u5ea6\u7684\u5927\u5c0f\u4e3a1<\/span>
    \n u =<\/span> x.<\/span>mean(<\/span>1<\/span>,<\/span> keepdim=<\/span>True<\/span>)<\/span> # dim=1\u7ef4\u5ea6\u6c42\u5747\u503c\u5e76\u4fdd\u7559\u901a\u9053<\/span>
    \n # \u8ba1\u7b97\u6807\u51c6\u5316\u56e0\u5b50 s，\u5373\u51cf\u53bb\u5747\u503c\u540e\u7684\u5e73\u65b9\u5dee\u7684\u5e73\u5747\u503c，\u4e5f\u4fdd\u7559\u901a\u9053\u7ef4\u5ea6<\/span>
    \n s =<\/span> (<\/span>x –<\/span> u)<\/span>.<\/span>pow<\/span>(<\/span>2<\/span>)<\/span>.<\/span>mean(<\/span>1<\/span>,<\/span> keepdim=<\/span>True<\/span>)<\/span>
    \n # \u5f52\u4e00\u5316，\u5c06\u6bcf\u4e2a\u50cf\u7d20\u7684\u503c\u51cf\u53bb\u5747\u503c u，\u7136\u540e\u9664\u4ee5\u6807\u51c6\u5dee\u7684\u5e73\u65b9\u6839\u52a0\u4e0a\u4e00\u4e2a\u5c0f\u7684\u5e38\u6570 eps \u4ee5\u4fdd\u8bc1\u6570\u503c\u7a33\u5b9a\u6027<\/span>
    \n x =<\/span> (<\/span>x –<\/span> u)<\/span> \/<\/span> torch.<\/span>sqrt(<\/span>s +<\/span> self.<\/span>eps)<\/span>
    \n # \u5e94\u7528\u53ef\u5b66\u4e60\u7684\u6743\u91cd\u548c\u504f\u7f6e<\/span>
    \n x =<\/span> self.<\/span>weight[<\/span>:<\/span>,<\/span> None<\/span>,<\/span> None<\/span>]<\/span> *<\/span> x +<\/span> self.<\/span>bias[<\/span>:<\/span>,<\/span> None<\/span>,<\/span> None<\/span>]<\/span>
    \n return<\/span> x<\/p>\n

    Prompt Encoder<\/h4>\n

    SAM\u6a21\u578b\u4e2dPrompt Encoder\u7f51\u7edc\u7ed3\u6784\u5982\u4e0b\u56fe\u6240\u793a\u3002\u4e3b\u8981\u5305\u62ec\u4e09\u6b65\u9aa4：<\/p>\n

      \n
    • \n

      Embed_Points：\u6807\u8bb0\u70b9\u7f16\u7801(\u6807\u8bb0\u70b9\u7531\u70b9\u8f6c\u53d8\u4e3a\u5411\u91cf)<\/p>\n<\/li>\n

    • \n

      Embed_Boxes：\u6807\u8bb0\u6846\u7f16\u7801(\u6807\u8bb0\u6846\u7531\u70b9\u8f6c\u53d8\u4e3a\u5411\u91cf)<\/p>\n<\/li>\n

    • \n

      Embed_Masks：mask\u7f16\u7801(mask\u4e0b\u91c7\u6837\u4fdd\u8bc1\u4e0eImage Encoder\u8f93\u51fa\u4e00\u81f4)<\/p>\n<\/li>\n<\/ul>\n

      \"\u56fe\u7247\"<\/p>\n

      Embed_Points<\/h5>\n

      Embed_Points\u7ed3\u6784\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

      \"\u56fe\u7247\"<\/p>\n

      \u6807\u8bb0\u70b9\u9884\u5904\u7406，\u5c06channel\u75312\u53d8\u4e3aembed_dim(MatMul:forward_with_coords)，\u7136\u540e\u518d\u52a0\u4e0a\u4f4d\u7f6e\u7f16\u7801\u6743\u91cd\u3002\u5176\u4e2d，<\/p>\n

        \n
      • \n

        2：\u5750\u6807(h,w)<\/p>\n<\/li>\n

      • \n

        embed_dim：\u63d0\u793a\u7f16\u7801\u7684channel<\/p>\n<\/li>\n<\/ul>\n

        \u300c\u4ee3\u7801\u5b9e\u73b0：\u300d<\/p>\n

        # \u5c06\u8f93\u5165\u7684\u70b9\u5750\u6807\u548c\u5bf9\u5e94\u7684\u6807\u7b7e\u8f6c\u5316\u4e3a\u9ad8\u7ef4\u7684\u5d4c\u5165\u8868\u793a，\u4ee5\u4fbf\u4e8e\u540e\u7eed\u7684\u6a21\u578b\u5904\u7406<\/span>
        \ndef<\/span> _embed_points<\/span>(<\/span>
        \n self,<\/span>
        \n points:<\/span> torch.<\/span>Tensor,<\/span>
        \n labels:<\/span> torch.<\/span>Tensor,<\/span>
        \n pad:<\/span> bool<\/span>,<\/span>
        \n)<\/span> –<\/span>><\/span> torch.<\/span>Tensor:<\/span>
        \n # \u5c06\u8f93\u5165\u7684\u70b9\u5750\u6807points\u7684\u6bcf\u4e2a\u5750\u6807\u503c\u589e\u52a00.5，\u4ee5\u5c06\u5750\u6807\u4ece\u50cf\u7d20\u7684\u5de6\u4e0a\u89d2\u79fb\u52a8\u5230\u50cf\u7d20\u4e2d\u5fc3<\/span>
        \n points =<\/span> points +<\/span> 0.5<\/span>
        \n # points\u548cboxes\u8054\u5408\u5219\u4e0d\u9700\u8981pad<\/span>
        \n if<\/span> pad:<\/span>
        \n # \u5728\u70b9\u5750\u6807 points \u548c\u6807\u7b7e labels \u4e2d\u6dfb\u52a0\u4e00\u4e2a\u586b\u5145\u9879<\/span>
        \n # \u4ee5\u4fdd\u6301\u6279\u6b21\u5904\u7406\u7684\u4e00\u81f4\u6027，\u5373\u4f7f\u67d0\u4e9b\u6837\u672c\u7684\u70b9\u6570\u91cf\u5c11\u4e8e\u6700\u5927\u6570\u91cf\u3002<\/span>
        \n # \u586b\u5145\u7684\u70b9\u5750\u6807\u4e3a(0,0)，\u6807\u7b7e\u4e3a-1<\/span>
        \n padding_point =<\/span> torch.<\/span>zeros(<\/span>(<\/span>points.<\/span>shape[<\/span>0<\/span>]<\/span>,<\/span> 1<\/span>,<\/span> 2<\/span>)<\/span>,<\/span> device=<\/span>points.<\/span>device)<\/span> # B,1,2<\/span>
        \n padding_label =<\/span> –<\/span>torch.<\/span>ones(<\/span>(<\/span>labels.<\/span>shape[<\/span>0<\/span>]<\/span>,<\/span> 1<\/span>)<\/span>,<\/span> device=<\/span>labels.<\/span>device)<\/span> # B,1<\/span>
        \n points =<\/span> torch.<\/span>cat(<\/span>[<\/span>points,<\/span> padding_point]<\/span>,<\/span> dim=<\/span>1<\/span>)<\/span> # B,N+1,2<\/span>
        \n labels =<\/span> torch.<\/span>cat(<\/span>[<\/span>labels,<\/span> padding_label]<\/span>,<\/span> dim=<\/span>1<\/span>)<\/span> # B,N+1<\/span>
        \n # \u6839\u636e\u8c03\u6574\u540e\u7684\u70b9\u5750\u6807\u548c\u8f93\u5165\u56fe\u50cf\u7684\u5c3a\u5bf8\u751f\u6210\u4f4d\u7f6e\u7f16\u7801<\/span>
        \n # \u751f\u6210\u7684\u5d4c\u5165\u7ef4\u5ea6：B,N+1,2f<\/span>
        \n # 2f \u8868\u793a\u6bcf\u4e2a\u70b9\u4f4d\u7f6e\u7f16\u7801\u7684\u7ef4\u5ea6，\u662f\u901a\u8fc7\u67d0\u79cd\u51fd\u6570（\u5982\u6b63\u5f26\u6216\u4f59\u5f26\u51fd\u6570）\u4ece\u539f\u59cb\u76842D\u5750\u6807\u6269\u5c55\u800c\u6765<\/span>
        \n point_embedding =<\/span> self.<\/span>pe_layer.<\/span>forward_with_coords(<\/span>points,<\/span> self.<\/span>input_image_size)<\/span>
        \n # \u6839\u636e\u6807\u7b7e labels \u7684\u503c，\u5bf9\u6bcf\u4e2a\u70b9\u7684\u5d4c\u5165\u8fdb\u884c\u8c03\u6574\u3002<\/span><\/p>\n

        # labels\u4e3a-1\u662f\u975e\u6807\u8bb0\u70b9，\u8bbe\u4e3a\u975e\u6807\u8bb0\u70b9\u6743\u91cd<\/span>
        \n point_embedding[<\/span>labels ==<\/span> –<\/span>1<\/span>]<\/span> =<\/span> 0.0<\/span>
        \n point_embedding[<\/span>labels ==<\/span> –<\/span>1<\/span>]<\/span> +=<\/span> self.<\/span>not_a_point_embed.<\/span>weight
        \n # labels\u4e3a0\u662f\u80cc\u666f\u70b9，\u52a0\u4e0a\u80cc\u666f\u70b9\u6743\u91cd<\/span>
        \n point_embedding[<\/span>labels ==<\/span> 0<\/span>]<\/span> +=<\/span> self.<\/span>point_embeddings[<\/span>0<\/span>]<\/span>.<\/span>weight
        \n # labels\u4e3a1\u662f\u76ee\u6807\u70b9，\u52a0\u4e0a\u76ee\u6807\u70b9\u6743\u91cd<\/span>
        \n point_embedding[<\/span>labels ==<\/span> 1<\/span>]<\/span> +=<\/span> self.<\/span>point_embeddings[<\/span>1<\/span>]<\/span>.<\/span>weight
        \n return<\/span> point_embedding<\/p>\n

        Embed_Boxes<\/h5>\n

        Embed_Boxes\u7ed3\u6784\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

        \"\u5728\u8fd9\u91cc\u63d2\u5165\u56fe\u7247\u63cf\u8ff0\"<\/p>\n

        \u6807\u8bb0\u6846（Bounding Box）\u4e00\u822c\u6709\u4e24\u4e2a\u70b9，\u7f16\u7801\u6b65\u9aa4\u5982\u4e0b：<\/p>\n

      • \n

        \u5c06\u8f93\u5165\u7684\u8fb9\u754c\u6846\u5750\u6807\u5f20\u91cfboxes\u4eceBxNx4\u8f6c\u6362\u4e3aBxNx2x2；<\/p>\n<\/li>\n

      • \n

        \u518d\u4f7f\u7528point embedding\u7f16\u7801\u7684\u65b9\u5f0f，\u5f97\u5230corner_embedding；<\/p>\n<\/li>\n

      • \n

        \u52a0\u4e0a\u4e4b\u524d\u751f\u6210\u7684\u53ef\u5b66\u4e60\u7684embeding\u5411\u91cf\u3002<\/p>\n<\/li>\n

        \u6700\u540e\u8f93\u51fa\u7684corner_embedding\u5927\u5c0f\u4e3aNx2x256\u3002<\/p>\n

        \u300c\u4ee3\u7801\u5b9e\u73b0：\u300d<\/p>\n

        # \u5c06\u8f93\u5165\u7684\u8fb9\u754c\u6846（boxes）\u8f6c\u6362\u4e3a\u9ad8\u7ef4\u7684\u5d4c\u5165\u8868\u793a<\/span>
        \ndef<\/span> _embed_boxes<\/span>(<\/span>self,<\/span> boxes:<\/span> torch.<\/span>Tensor)<\/span> –<\/span>><\/span> torch.<\/span>Tensor:<\/span>
        \n # \u5c06\u5750\u6807\u4ece\u50cf\u7d20\u7684\u5de6\u4e0a\u89d2\u79fb\u52a8\u5230\u50cf\u7d20\u4e2d\u5fc3<\/span>
        \n boxes =<\/span> boxes +<\/span> 0.5<\/span>
        \n # \u5c06\u8f93\u5165\u7684\u8fb9\u754c\u6846\u5750\u6807\u5f20\u91cfboxes\u4eceBxN*4\u8f6c\u6362\u4e3aB*Nx2x2<\/span>
        \n # \u5176\u4e2dB\u662f\u6279\u6b21\u5927\u5c0f，N\u662f\u6bcf\u4e2a\u6837\u672c\u4e2d\u7684\u8fb9\u754c\u6846\u6570\u91cf<\/span>
        \n coords =<\/span> boxes.<\/span>reshape(<\/span>–<\/span>1<\/span>,<\/span> 2<\/span>,<\/span> 2<\/span>)<\/span>
        \n # \u5bf9\u6bcf\u4e2a\u8fb9\u754c\u6846\u7684\u89d2\u70b9\u5750\u6807\u8fdb\u884c\u4f4d\u7f6e\u7f16\u7801<\/span>
        \n corner_embedding =<\/span> self.<\/span>pe_layer.<\/span>forward_with_coords(<\/span>coords,<\/span> self.<\/span>input_image_size)<\/span> #<\/span>
        \n # \u5206\u522b\u5bf9\u6bcf\u4e2a\u8fb9\u754c\u6846\u7684\u8d77\u59cb\u70b9\u548c\u672b\u5c3e\u70b9\u7684\u5d4c\u5165\u5411\u91cf\u52a0\u4e0a\u7279\u5b9a\u7684\u6743\u91cd<\/span>
        \n corner_embedding[<\/span>:<\/span>,<\/span> 0<\/span>,<\/span> :<\/span>]<\/span> +=<\/span> self.<\/span>point_embeddings[<\/span>2<\/span>]<\/span>.<\/span>weight
        \n corner_embedding[<\/span>:<\/span>,<\/span> 1<\/span>,<\/span> :<\/span>]<\/span> +=<\/span> self.<\/span>point_embeddings[<\/span>3<\/span>]<\/span>.<\/span>weight
        \n # \u8fd4\u56de\u52a0\u6743\u540e\u5d4c\u5165\u5411\u91cf，\u5f62\u72b6\u4e3a B*Nx2xembed_dim，\u5176\u4e2d embed_dim \u662f\u4f4d\u7f6e\u7f16\u7801\u7684\u7ef4\u5ea6<\/span>
        \n return<\/span> corner_embedding<\/p>\n

        Embed_Mask<\/h5>\n

        mask\u63d0\u793a\u5141\u8bb8\u6211\u4eec\u76f4\u63a5\u5728\u539f\u56fe\u4e0a\u6307\u793a\u611f\u5174\u8da3\u533a\u57df\u6765\u5f15\u5bfc\u6a21\u578b\u3002\u8fd9\u4e9bmask\u901a\u8fc7\u5377\u79ef\u64cd\u4f5c\u88ab\u8f6c\u6362\u4e3a\u4e0e\u56fe\u50cf\u5d4c\u5165\u7a7a\u95f4\u76f8\u5339\u914d\u7684\u7279\u5f81，\u7136\u540e\u4e0e\u56fe\u50cf\u5d4c\u5165\u76f8\u52a0\u7ed3\u5408，\u4e3a\u6a21\u578b\u63d0\u4f9b\u5206\u5272\u7684\u7cbe\u786e\u4f4d\u7f6e\u4fe1\u606f\u3002<\/p>\n

        \u5982\u679c\u6ca1\u6709\u4f7f\u7528mask\u63d0\u793a，\u5219\u5c06\u4e00\u7ec4\u53ef\u5b66\u4e60\u5411\u91cf(no_mask_embed,1*256)expand\u4e3a1×256\u00d764\u00d764\u540e\u66ff\u4ee3，\u4f7f\u5f97\u5728\u5904\u7406\u5e8f\u5217\u6570\u636e\u65f6，\u5373\u4f7f\u6ca1\u6709\u5177\u4f53\u7684mask\u4fe1\u606f，\u4e5f\u80fd\u6709\u4e00\u4e2a\u7edf\u4e00\u7684\u5904\u7406\u65b9\u5f0f\u3002<\/p>\n

        \"\u56fe\u7247\"<\/p>\n

        # \u5728PromptEncoder\u7684forward\u5b9a\u4e49<\/span>
        \n'''
        \n\u9996\u5148\u83b7\u53d6no_mask_embed\u6743\u91cd\u77e9\u9635，\u5e76\u5c06\u5176\u91cd\u5851\u6210\u4e00\u4e2a\u5f62\u72b6\u4e3a(1, num_embeddings, 1, 1)\u7684\u56db\u7ef4\u5f20\u91cf\u3002<\/p>\n

        \u518d\u5229\u7528.expand\u65b9\u6cd5\u5c06\u8fd9\u4e2a\u5f20\u91cf\u6269\u5c55\u5230\u4e0e\u56fe\u50cf\u7f16\u7801\u76f8\u540c\u7684\u5c3a\u5bf8\u3002bs\u662fbatch\u5927\u5c0f，-1\u662f\u4e00\u4e2a\u5360\u4f4d\u7b26，\u5b83\u4f1a\u81ea\u52a8\u8ba1\u7b97\u51fa
        \nnum_embeddings\u7684\u503c\u4ee5\u4fdd\u6301\u5f20\u91cf\u7684\u5143\u7d20\u603b\u6570\u4e0d\u53d8\u3002self.image_embedding_size[0]\u548cself.image_embedding_size[1]\u5206\u522b\u8868\u793a\u56fe\u50cf\u7f16\u7801\u7684\u5bbd\u5ea6\u548c\u9ad8\u5ea6\u3002
        \n'''<\/span>
        \nself.<\/span>no_mask_embed =<\/span> nn.<\/span>Embedding(<\/span>1<\/span>,<\/span> embed_dim)<\/span> # embed_dim=256<\/span>
        \ndense_embeddings =<\/span> self.<\/span>no_mask_embed.<\/span>weight.<\/span>reshape(<\/span>1<\/span>,<\/span> –<\/span>1<\/span>,<\/span> 1<\/span>,<\/span> 1<\/span>)<\/span>.<\/span>expand(<\/span>
        \n bs,<\/span> –<\/span>1<\/span>,<\/span> self.<\/span>image_embedding_size[<\/span>0<\/span>]<\/span>,<\/span> self.<\/span>image_embedding_size[<\/span>1<\/span>]<\/span>)<\/span><\/p>\n

        )<\/span><\/p>\n

        \u5982\u679c\u6709\u914d\u7f6emask，Embed_Masks\u7ed3\u6784\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

        \"\u5728\u8fd9\u91cc\u63d2\u5165\u56fe\u7247\u63cf\u8ff0\"<\/p>\n

        \u5df2\u77e5\u8f93\u5165mask\u662fNx1x256x256，\u7ecf\u8fc73\u5c42\u5377\u79ef，\u6700\u540e\u5f97\u5230\u4e0eImage Embedding\u4e00\u6837\u7684size：<\/p>\n

        \u9996\u5148，mask\u8fdb\u5165\u4e00\u4e2a1x2x2x4\u7684\u5377\u79ef，stride=2；LN；\u518d\u8fdb\u5165\u4e00\u4e2a4x2x2x16\u7684\u5377\u79ef，stride=2；LN；\u6700\u540e\u518d\u8fdb\u5165\u4e00\u4e2a16x1x1x256\u7684\u5377\u79ef；\u5f97\u5230\u6700\u540e\u7684mask_embedding\u7684size\u4e3aNx256x64x64，\u6700\u7ec8mask_embedding\u4f5c\u4e3adense_embedding\u8f93\u51fa，\u5927\u5c0f\u4e3aNx256x64x64\u3002<\/p>\n

        mask\u7684\u8f93\u51fa\u5c3a\u5bf8\u662fImage Encoder\u6a21\u5757\u8f93\u51fa\u7684\u56fe\u50cf\u7f16\u7801\u5c3a\u5bf8\u76844\u500d，\u56e0\u6b64\u4e3a\u4e86\u4fdd\u6301\u4e00\u81f4，\u9700\u89814\u500d\u4e0b\u91c7\u6837\u3002<\/p>\n

        \u300c\u4ee3\u7801\u5b9e\u73b0\u300d<\/p>\n

        # \u5c06\u8f93\u5165\u7684\u63a9\u6a21（mask）\u5f20\u91cf\u8f6c\u6362\u4e3a\u4e00\u4e2a\u4f4e\u5206\u8fa8\u7387\u7684\u5d4c\u5165\u8868\u793a<\/span>
        \n# \u63a9\u6a21 masks \u662f\u4e00\u4e2a\u5f62\u72b6\u4e3a BxCxHxW \u7684\u5f20\u91cf<\/span>
        \n# \u5176\u4e2d B \u662f\u6279\u6b21\u5927\u5c0f，C \u662f\u901a\u9053\u6570（\u901a\u5e38\u4e3a1，\u56e0\u4e3a\u63a9\u6a21\u901a\u5e38\u53ea\u6709\u4e00\u901a\u9053），H \u548c W \u5206\u522b\u662f\u9ad8\u5ea6\u548c\u5bbd\u5ea6\u3002<\/span>
        \ndef<\/span> _embed_masks<\/span>(<\/span>self,<\/span> masks:<\/span> torch.<\/span>Tensor)<\/span> –<\/span>><\/span> torch.<\/span>Tensor:<\/span>
        \n # mask\u4e0b\u91c7\u68374\u500d<\/span>
        \n mask_embedding =<\/span> self.<\/span>mask_downscaling(<\/span>masks)<\/span>
        \n # \u8fd4\u56de\u4e0b\u91c7\u6837\u5e76\u8f6c\u6362\u540e\u7684\u63a9\u6a21\u5d4c\u5165，\u5176\u5f62\u72b6\u4e3a B*embed_dim*H'*W',\u5176\u4e2d H' \u548c W' \u662f\u4e0b\u91c7\u6837\u540e\u7684\u9ad8\u5ea6\u548c\u5bbd\u5ea6<\/span>
        \n return<\/span> mask_embedding<\/p>\n

        # mask_downscaling\u5305\u62ec\u591a\u4e2a\u5377\u79ef\u5c42\u3001\u5c42\u5f52\u4e00\u5316（LayerNorm2d）\u548c\u6fc0\u6d3b\u51fd\u6570，\u76ee\u7684\u662f\u51cf\u5c11\u63a9\u6a21\u7684\u7a7a\u95f4\u7ef4\u5ea6，\u540c\u65f6\u589e\u52a0\u901a\u9053\u7ef4\u5ea6<\/span>
        \nself.<\/span>mask_downscaling =<\/span> nn.<\/span>Sequential(<\/span>
        \n # \u5c06\u901a\u9053\u6570\u4ece1\u51cf\u5c11\u5230mask_in_chans\/\/4，\u540c\u65f6\u4f7f\u75282×2\u7684\u5377\u79ef\u6838\u548c\u6b65\u957f2\u8fdb\u884c\u4e0b\u91c7\u6837，\u964d\u4f4e\u4e86\u7a7a\u95f4\u5206\u8fa8\u7387<\/span>
        \n nn.<\/span>Conv2d(<\/span>1<\/span>,<\/span> mask_in_chans \/\/<\/span> 4<\/span>,<\/span> kernel_size=<\/span>2<\/span>,<\/span> stride=<\/span>2<\/span>)<\/span>,<\/span>
        \n # \u89c4\u8303\u5316\u901a\u9053\u7ef4\u5ea6\u4e0a\u7684\u7279\u5f81<\/span>
        \n LayerNorm2d(<\/span>mask_in_chans \/\/<\/span> 4<\/span>)<\/span>,<\/span>
        \n # \u6fc0\u6d3b\u51fd\u6570，\u5f15\u5165\u975e\u7ebf\u6027<\/span>
        \n activation(<\/span>)<\/span>,<\/span>
        \n # \u5c06\u901a\u9053\u6570\u6062\u590d\u5230 mask_in_chans，\u518d\u6b21\u4f7f\u75282×2\u7684\u5377\u79ef\u6838\u548c\u6b65\u957f2\u8fdb\u884c\u4e0b\u91c7\u6837，\u8fdb\u4e00\u6b65\u964d\u4f4e\u7a7a\u95f4\u5206\u8fa8\u7387<\/span>
        \n nn.<\/span>Conv2d(<\/span>mask_in_chans \/\/<\/span> 4<\/span>,<\/span> mask_in_chans,<\/span> kernel_size=<\/span>2<\/span>,<\/span> stride=<\/span>2<\/span>)<\/span>,<\/span>
        \n # LayerNorm2d \u5c42\u548c\u6fc0\u6d3b\u51fd\u6570<\/span>
        \n LayerNorm2d(<\/span>mask_in_chans)<\/span>,<\/span>
        \n activation(<\/span>)<\/span>,<\/span>
        \n # \u5c06\u901a\u9053\u6570\u589e\u52a0\u5230 embed_dim，\u901a\u5e38\u662f\u4e3a\u4e86\u4e0e\u6a21\u578b\u7684\u5176\u4ed6\u90e8\u5206\u4fdd\u6301\u4e00\u81f4<\/span>
        \n nn.<\/span>Conv2d(<\/span>mask_in_chans,<\/span> embed_dim,<\/span> kernel_size=<\/span>1<\/span>)<\/span>,<\/span>
        \n )<\/span><\/p>\n

        \u300cPositionEmbeddingRandom\u300d<\/p>\n

        \u7528\u4e8e\u5c06\u6807\u8bb0\u70b9\u548c\u6807\u8bb0\u6846\u7684\u5750\u6807\u8fdb\u884c\u63d0\u793a\u7f16\u7801\u9884\u5904\u7406\u3002\u5c31\u662f\u5c0664×64\u4e2a\u5750\u6807\u70b9\u5f52\u4e00\u5316\u540e，\u4e0e\u968f\u673a\u9ad8\u65af\u77e9\u9635\u76f8\u4e58(2×128)，\u518d\u5c06\u7ed3\u679c\u5206\u522b\u8fdb\u884csin\u548ccos，\u6700\u540e\u518d\u62fc\u5230\u4e00\u8d77，\u8f93\u51fa\u7684\u5927\u5c0f\u4e3a256x64x64，\u4e0eimage_embedding\u5927\u5c0f\u57fa\u672c\u4e00\u81f4\u4e86\u3002<\/p>\n

        class<\/span> PositionEmbeddingRandom<\/span>(<\/span>nn.<\/span>Module)<\/span>:<\/span>
        \n """
        \n Positional encoding using random spatial frequencies.
        \n """<\/span>
        \n def<\/span> init<\/span>(<\/span>self,<\/span> num_pos_feats:<\/span> int<\/span> =<\/span> 64<\/span>,<\/span> scale:<\/span> Optional[<\/span>float<\/span>]<\/span> =<\/span> None<\/span>)<\/span> –<\/span>><\/span> None<\/span>:<\/span>
        \n super<\/span>(<\/span>)<\/span>.<\/span>init(<\/span>)<\/span>
        \n if<\/span> scale is<\/span> None<\/span> or<\/span> scale <=<\/span> 0.0<\/span>:<\/span>
        \n scale =<\/span> 1.0<\/span>
        \n # \u6784\u5efa\u4e00\u4e2a2×128\u7684\u968f\u673a\u77e9\u9635\u4f5c\u4e3a\u4f4d\u7f6e\u7f16\u7801\u9ad8\u65af\u77e9\u9635<\/span>
        \n self.<\/span>register_buffer(<\/span>
        \n "positional_encoding_gaussian_matrix"<\/span>,<\/span>
        \n scale *<\/span> torch.<\/span>randn(<\/span>(<\/span>2<\/span>,<\/span> num_pos_feats)<\/span>)<\/span>,<\/span>
        \n )<\/span><\/p>\n

        def<\/span> _pe_encoding<\/span>(<\/span>self,<\/span> coords:<\/span> torch.<\/span>Tensor)<\/span> –<\/span>><\/span> torch.<\/span>Tensor:<\/span>
        \n """Positionally encode points that are normalized to [0,1]."""<\/span>
        \n # assuming coords are in [0, 1]^2 square and have d_1 x … x d_n x 2 shape<\/span>
        \n coords =<\/span> 2<\/span> *<\/span> coords –<\/span> 1<\/span><\/p>\n

        # \u77e9\u9635\u4e58\u6cd5：64x64xx2 @ 2×128 —> 64x64x128<\/span>
        \n coords =<\/span> coords @ self.<\/span>positional_encoding_gaussian_matrix
        \n coords =<\/span> 2<\/span> *<\/span> np.<\/span>pi *<\/span> coords<\/p>\n

        # outputs d_1 x … x d_n x C shape<\/span>
        \n # cat, \u6700\u540e\u4e00\u4e2a\u7ef4\u5ea6\u4e0a\u62fc\u63a5：64x64x256<\/span>
        \n return<\/span> torch.<\/span>cat(<\/span>[<\/span>torch.<\/span>sin(<\/span>coords)<\/span>,<\/span> torch.<\/span>cos(<\/span>coords)<\/span>]<\/span>,<\/span> dim=<\/span>–<\/span>1<\/span>)<\/span><\/p>\n

        def<\/span> forward<\/span>(<\/span>self,<\/span> size:<\/span> Tuple[<\/span>int<\/span>,<\/span> int<\/span>]<\/span>)<\/span> –<\/span>><\/span> torch.<\/span>Tensor:<\/span>
        \n """Generate positional encoding for a grid of the specified size."""<\/span>
        \n h,<\/span> w =<\/span> size
        \n device:<\/span> Any =<\/span> self.<\/span>positional_encoding_gaussian_matrix.<\/span>device<\/p>\n

        # \u6784\u9020\u4e00\u4e2a64×64\u7684\u51681\u77e9\u9635<\/span>
        \n grid =<\/span> torch.<\/span>ones(<\/span>(<\/span>h,<\/span> w)<\/span>,<\/span> device=<\/span>device,<\/span> dtype=<\/span>torch.<\/span>float32)<\/span><\/p>\n

        # \u884c\u3001\u5217\u7d2f\u52a0<\/span>
        \n y_embed =<\/span> grid.<\/span>cumsum(<\/span>dim=<\/span>0<\/span>)<\/span> –<\/span> 0.5<\/span>
        \n x_embed =<\/span> grid.<\/span>cumsum(<\/span>dim=<\/span>1<\/span>)<\/span> –<\/span> 0.5<\/span><\/p>\n

        # \u884c\u5217\u7d2f\u52a0\u7ed3\u679c\u5f52\u4e00\u5316<\/span>
        \n y_embed =<\/span> y_embed \/<\/span> h
        \n x_embed =<\/span> x_embed \/<\/span> w<\/p>\n

        # \u884c\u5217\u62fc\u63a5：64x64x2，\u7f16\u7801\u540e\u7684\u7ed3\u679c\u662f64x64x256<\/span>
        \n pe =<\/span> self.<\/span>_pe_encoding(<\/span>torch.<\/span>stack(<\/span>[<\/span>x_embed,<\/span> y_embed]<\/span>,<\/span> dim=<\/span>–<\/span>1<\/span>)<\/span>)<\/span><\/p>\n

        # \u6700\u540e\u8f93\u51fa256x64x64<\/span>
        \n return<\/span> pe.<\/span>permute(<\/span>2<\/span>,<\/span> 0<\/span>,<\/span> 1<\/span>)<\/span> # C x H x W<\/span><\/p>\n

        Mask Decoder<\/h4>\n

        Mask Decoder\u7f51\u7edc\u7ed3\u6784\u53c2\u6570\u914d\u7f6e\u5982\u4e0b\u3002<\/p>\n

        def<\/span> __init__<\/span>(<\/span>
        \n self,<\/span>
        \n *<\/span>,<\/span>
        \n # transformer\u901a\u9053\u6570<\/span>
        \n transformer_dim:<\/span> int<\/span>,<\/span>
        \n # \u7528\u4e8e\u9884\u6d4bmask\u7684Transformer\u7f51\u7edc\u6a21\u5757<\/span>
        \n transformer:<\/span> nn.<\/span>Module,<\/span>
        \n # \u6d88\u9664\u63a9\u7801\u6b67\u4e49\u9884\u6d4b\u7684\u63a9\u7801\u6570\u91cf，\u9ed8\u8ba4\u4e3a3<\/span>
        \n num_multimask_outputs:<\/span> int<\/span> =<\/span> 3<\/span>,<\/span>
        \n # \u6fc0\u6d3b\u51fd\u6570，\u9ed8\u8ba4\u4e3aGELU<\/span>
        \n activation:<\/span> Type[<\/span>nn.<\/span>Module]<\/span> =<\/span> nn.<\/span>GELU,<\/span>
        \n # MLP\u7528\u4e8e\u9884\u6d4b\u63a9\u6a21\u8d28\u91cf\u7684\u6df1\u5ea6<\/span>
        \n iou_head_depth:<\/span> int<\/span> =<\/span> 3<\/span>,<\/span>
        \n # MLP\u7684\u9690\u85cf\u5c42\u901a\u9053\u6570<\/span>
        \n iou_head_hidden_dim:<\/span> int<\/span> =<\/span> 256<\/span>,<\/span>
        \n)<\/span> –<\/span>><\/span> None<\/span>:<\/span>
        \n super<\/span>(<\/span>)<\/span>.<\/span>__init__(<\/span>)<\/span>
        \n self.<\/span>transformer_dim =<\/span> transformer_dim #\u5b58\u50a8\u4f20\u5165\u7684transformer_dim<\/span>
        \n # \u5b58\u50a8\u4f20\u5165\u7684transformer\u6a21\u5757<\/span>
        \n self.<\/span>transformer =<\/span> transformer
        \n # \u5b58\u50a8\u63a9\u7801\u9884\u6d4b\u7684\u8f93\u51fa\u6570\u91cf<\/span>
        \n self.<\/span>num_multimask_outputs =<\/span> num_multimask_outputs
        \n # \u7528\u4e8e\u8868\u793aIoU（Intersection over Union）\u7684\u5d4c\u5165\u5c42，\u5927\u5c0f\u4e3a1\u00d7transformer_dim<\/span>
        \n # \u53ef\u5b66\u4e60\u7684iou tokens：1×256<\/span>
        \n self.<\/span>iou_token =<\/span> nn.<\/span>Embedding(<\/span>1<\/span>,<\/span> transformer_dim)<\/span>
        \n # \u5305\u542bIoU token\u5728\u5185\u7684\u603bmask token\u6570\u91cf<\/span>
        \n # # num_mask_tokens = 3 + 1 = 4, transformer_dim = 256<\/span>
        \n # \u8f93\u51fa\u4e00\u4e2a4×256\u7684\u77e9\u9635<\/span>
        \n self.<\/span>num_mask_tokens =<\/span> num_multimask_outputs +<\/span> 1<\/span>
        \n # \u5b58\u50a8\u6240\u6709mask token\u7684\u5d4c\u5165\u5c42，\u5927\u5c0f\u4e3anum_mask_tokens\u00d7transformer_dim<\/span>
        \n self.<\/span>mask_tokens =<\/span> nn.<\/span>Embedding(<\/span>self.<\/span>num_mask_tokens,<\/span> transformer_dim)<\/span><\/p>\n

        #—– upscaled —–<\/span>
        \n # \u7528\u4e8e4\u500d\u4e0a\u91c7\u6837\u7684\u5e8f\u5217，\u5305\u542b\u4e24\u4e2a\u8f6c\u7f6e\u5377\u79ef\u5c42，\u6bcf\u4e2a\u4e0a\u91c7\u68372\u500d，\u4e2d\u95f4\u5939\u7740LayerNorm\u548c\u6fc0\u6d3b\u51fd\u6570<\/span>
        \n self.<\/span>output_upscaling =<\/span> nn.<\/span>Sequential(<\/span>
        \n nn.<\/span>ConvTranspose2d(<\/span>transformer_dim,<\/span> transformer_dim \/\/<\/span> 4<\/span>,<\/span> kernel_size=<\/span>2<\/span>,<\/span> stride=<\/span>2<\/span>)<\/span>,<\/span> #\u8f6c\u7f6e\u5377\u79ef \u4e0a\u91c7\u68372\u500d<\/span>
        \n LayerNorm2d(<\/span>transformer_dim \/\/<\/span> 4<\/span>)<\/span>,<\/span>
        \n activation(<\/span>)<\/span>,<\/span>
        \n nn.<\/span>ConvTranspose2d(<\/span>transformer_dim \/\/<\/span> 4<\/span>,<\/span> transformer_dim \/\/<\/span> 8<\/span>,<\/span> kernel_size=<\/span>2<\/span>,<\/span> stride=<\/span>2<\/span>)<\/span>,<\/span>
        \n activation(<\/span>)<\/span>,<\/span>
        \n )<\/span>
        \n # —– upscaled —–<\/span><\/p>\n

        # \u591a\u5c42\u611f\u77e5\u673a（MLP）\u6a21\u5757<\/span>
        \n # \u4e00\u4e2a\u6a21\u5757\u5217\u8868，\u5305\u542b\u4e86num_mask_tokens\u4e2aMLP，\u6bcf\u4e2aMLP\u7528\u4e8e\u5904\u7406\u4e0d\u540cmask\u7684\u8f93\u51fa<\/span>
        \n self.<\/span>output_hypernetworks_mlps =<\/span> nn.<\/span>ModuleList(<\/span>
        \n [<\/span>
        \n MLP(<\/span>transformer_dim,<\/span> transformer_dim,<\/span> transformer_dim \/\/<\/span> 8<\/span>,<\/span> 3<\/span>)<\/span>
        \n for<\/span> i in<\/span> range<\/span>(<\/span>self.<\/span>num_mask_tokens)<\/span>
        \n ]<\/span>
        \n )<\/span>
        \n # —– MLP —–<\/span><\/p>\n

        # —– MLP —–<\/span>
        \n # \u4e00\u4e2aMLP，\u7528\u4e8e\u9884\u6d4bIoU，\u8f93\u5165\u662ftransformer_dim，\u7ecf\u8fc7iou_head_hidden_dim\u7684\u9690\u85cf\u5c42，\u8f93\u51fa\u662fnum_mask_tokens<\/span>
        \n self.<\/span>iou_prediction_head =<\/span> MLP(<\/span>
        \n transformer_dim,<\/span> iou_head_hidden_dim,<\/span> self.<\/span>num_mask_tokens,<\/span> iou_head_depth
        \n )<\/span>
        \n # —– MLP —–<\/span><\/p>\n

        SAM\u6a21\u578bMask Decoder\u7f51\u7edc\u7ed3\u6784\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

        \"\u5728\u8fd9\u91cc\u63d2\u5165\u56fe\u7247\u63cf\u8ff0\"<\/p>\n

          \n
        • \n

          spa_pro_emb(sparse embedding)\u3001iou_token\u3001mask_token\u5408\u5e76\u6210\u4e00\u4e2atokens，\u4f5c\u4e3apoint_embeddings\u3002<\/p>\n<\/li>\n

        • \n

          spa_pro_emb： point\u3001bbox prompt\u5408\u5e76\u540e\u7684\u4ea7\u7269，\u4e00\u822c\u4e3aNxXx256\u3002<\/p>\n<\/li>\n

        • \n

          iou_token：\u53ef\u5b66\u4e60\u53c2\u6570，\u5927\u5c0f\u4e3a1×256\u3002<\/p>\n<\/li>\n

        • \n

          mask_token：\u53ef\u5b66\u4e60\u53c2\u6570，\u5927\u5c0f\u4e3a4×256\u3002<\/p>\n<\/li>\n<\/ul>\n

          \u539f\u8bba\u6587\u4e2dMask Decoder\u6a21\u5757\u5404\u90e8\u5206\u7ed3\u6784\u793a\u610f\u56fe\u5982\u4e0b\u3002<\/p>\n

          \"\u5728\u8fd9\u91cc\u63d2\u5165\u56fe\u7247\u63cf\u8ff0\"<\/p>\n

          Mask Decoder\u7f51\u7edc\u5728\u7279\u5f81\u63d0\u53d6\u4e2d\u7684\u57fa\u672c\u6b65\u9aa4\u5982\u4e0b：<\/p>\n

        • \n

          transformer：\u5c06\u6765\u81ea\u7f16\u7801\u5668\u7684\u56fe\u50cf\u7279\u5f81\u4e0e\u989d\u5916\u7684\u63d0\u793a\u4fe1\u606f（\u5982\u63a9\u7801\u63d0\u793a\u6216\u67e5\u8be2\u5411\u91cf）\u878d\u5408，\u4ee5\u6355\u6349\u76ee\u6807\u533a\u57df\u7684\u4e0a\u4e0b\u6587\u4fe1\u606f\u3002<\/p>\n<\/li>\n

        • \n

          upscaled：\u5bf9\u7c97\u7565mask src\u8fdb\u884c\u4e0a\u91c7\u6837，\u4f7f\u5176\u4e0e\u539f\u59cb\u56fe\u50cf\u5c3a\u5bf8\u76f8\u5339\u914d，\u4ee5\u4fbf\u8fdb\u884c\u66f4\u7cbe\u7ec6\u7684mask\u9884\u6d4b\u3002<\/p>\n<\/li>\n

        • \n

          mask_MLP：\u901a\u8fc7\u4e00\u7cfb\u5217\u5168\u8fde\u63a5\u5c42，\u5bf9\u4e0a\u91c7\u6837\u540e\u7684\u7279\u5f81\u8fdb\u884c\u53d8\u6362，\u8ba1\u7b97\u51fa\u9488\u5bf9\u6bcf\u4e2a\u50cf\u7d20\u7684mask\u6982\u7387\u3002\u8fd9\u4e9b\u5c42\u53ef\u4ee5\u8bbe\u8ba1\u4e3a\u5b66\u4e60\u5982\u4f55\u4e3a\u6bcf\u4e2amask\u901a\u9053\u5206\u914d\u6743\u91cd，\u4ece\u800c\u751f\u6210\u6700\u7ec8\u7684mask\u8f93\u51fa\u3002<\/p>\n<\/li>\n

        • \n

          iou_MLP：\u8bc4\u4f30\u751f\u6210\u7684mask\u4e0e\u771f\u5b9emask\u4e4b\u95f4\u7684\u91cd\u53e0\u7a0b\u5ea6，\u5373\u9884\u6d4bmask\u7684\u8d28\u91cf\u3002<\/p>\n<\/li>\n

          def<\/span> forward<\/span>(<\/span>
          \n self,<\/span>
          \n # image encoder \u56fe\u50cf\u7279\u5f81<\/span>
          \n image_embeddings:<\/span> torch.<\/span>Tensor,<\/span>
          \n # \u4f4d\u7f6e\u7f16\u7801<\/span>
          \n # 256x64x64<\/span>
          \n image_pe:<\/span> torch.<\/span>Tensor,<\/span>
          \n # \u6807\u8bb0\u70b9\u548c\u6807\u8bb0\u6846\u7684\u5d4c\u5165\u7f16\u7801<\/span>
          \n sparse_prompt_embeddings:<\/span> torch.<\/span>Tensor,<\/span>
          \n # \u8f93\u5165mask\u7684\u5d4c\u5165\u7f16\u7801<\/span>
          \n dense_prompt_embeddings:<\/span> torch.<\/span>Tensor,<\/span>
          \n # \u662f\u5426\u8f93\u51fa\u591a\u4e2amask<\/span>
          \n multimask_output:<\/span> bool<\/span>,<\/span>
          \n)<\/span> –<\/span>><\/span> Tuple[<\/span>torch.<\/span>Tensor,<\/span> torch.<\/span>Tensor]<\/span>:<\/span>
          \n # \u5c06\u8fd9\u4e9b\u7279\u5f81\u878d\u5408，\u901a\u8fc7Transformer\u548c\u540e\u7eed\u7684\u4e0a\u91c7\u6837\u53caMLP\u5c42，\u751f\u6210\u63a9\u819c\u9884\u6d4b\u548cIoU\u5206\u6570<\/span>
          \n masks,<\/span> iou_pred =<\/span> self.<\/span>predict_masks(<\/span>
          \n image_embeddings=<\/span>image_embeddings,<\/span>
          \n image_pe=<\/span>image_pe,<\/span>
          \n sparse_prompt_embeddings=<\/span>sparse_prompt_embeddings,<\/span>
          \n dense_prompt_embeddings=<\/span>dense_prompt_embeddings,<\/span>
          \n )<\/span>
          \n # \u5982\u679cmultimask_output\u4e3aTrue，\u8868\u793a\u9700\u8981\u8f93\u51fa\u591a\u4e2a\u63a9\u6a21，\u9009\u53d6\u7d22\u5f15\u4e3a1\u5230num_multimask_outputs\u7684\u6240\u6709\u63a9\u6a21<\/span>
          \n if<\/span> multimask_output:<\/span>
          \n mask_slice =<\/span> slice<\/span>(<\/span>1<\/span>,<\/span> None<\/span>)<\/span>
          \n # \u5426\u5219，\u5982\u679cmultimask_output\u4e3aFalse，\u4ec5\u8f93\u51fa\u7b2c\u4e00\u4e2a\u63a9\u6a21（\u901a\u5e38\u662f\u6700\u9ad8\u5f97\u5206\u7684\u63a9\u6a21）<\/span>
          \n else<\/span>:<\/span>
          \n mask_slice =<\/span> slice<\/span>(<\/span>0<\/span>,<\/span> 1<\/span>)<\/span>
          \n # \u6839\u636emultimask_output\u9009\u62e9\u540e\u7684\u63a9\u6a21，\u7ef4\u5ea6\u8c03\u6574\u4e3a(batch_size, num_selected_masks, height, width)<\/span>
          \n masks =<\/span> masks[<\/span>:<\/span>,<\/span> mask_slice,<\/span> :<\/span>,<\/span> :<\/span>]<\/span>
          \n # \u6839\u636emultimask_output\u9009\u62e9\u540e\u7684IoU\u9884\u6d4b，\u7ef4\u5ea6\u8c03\u6574\u4e3a(batch_size, num_selected_masks)<\/span>
          \n iou_pred =<\/span> iou_pred[<\/span>:<\/span>,<\/span> mask_slice]<\/span>
          \n return<\/span> masks,<\/span> iou_pred<\/p>\n

          def<\/span> predict_masks<\/span>(<\/span>
          \n self,<\/span>
          \n # image embedding： \u662fimage encoder\u7684\u8f93\u51fa，\u5927\u5c0f\u4e3a\u4e3a1x256x64x64<\/span>
          \n image_embeddings:<\/span> torch.<\/span>Tensor,<\/span>
          \n # image_pe\u4f4d\u7f6e\u7f16\u7801\u4e5f\u62d3\u5c55\u6210Nx256x64x64\u7684\u77e9\u9635<\/span>
          \n image_pe:<\/span> torch.<\/span>Tensor,<\/span>
          \n sparse_prompt_embeddings:<\/span> torch.<\/span>Tensor,<\/span>
          \n dense_prompt_embeddings:<\/span> torch.<\/span>Tensor,<\/span>
          \n)<\/span> –<\/span>><\/span> Tuple[<\/span>torch.<\/span>Tensor,<\/span> torch.<\/span>Tensor]<\/span>:<\/span>
          \n # \u9996\u5148\u5c06iou token\u548cmask token \u62fc\u63a5\u5f97\u5230\u4e00\u4e2a5×256\u7684\u77e9\u9635，\u518d\u5c06\u5176\u62d3\u5c55\u5230\u4e0esparse embedding\u4e00\u4e2a\u7ef4\u5ea6Nx5x256<\/span>
          \n # 1,E and 4,E –> 5,E<\/span>
          \n output_tokens =<\/span> torch.<\/span>cat(<\/span>[<\/span>self.<\/span>iou_token.<\/span>weight,<\/span> self.<\/span>mask_tokens.<\/span>weight]<\/span>,<\/span> dim=<\/span>0<\/span>)<\/span>
          \n # \u518d\u5c06\u62d3\u5c55\u540e\u7684\u77e9\u9635\u4e0esparse embedding\u62fc\u63a5\u5f97\u5230tokens，\u5176\u5927\u5c0fNx(5+X)x256<\/span>
          \n # 5,E –> B,5,E<\/span>
          \n output_tokens =<\/span> output_tokens.<\/span>unsqueeze(<\/span>0<\/span>)<\/span>.<\/span>expand(<\/span>sparse_prompt_embeddings.<\/span>size(<\/span>0<\/span>)<\/span>,<\/span> –<\/span>1<\/span>,<\/span> –<\/span>1<\/span>)<\/span>
          \n # \u518d\u4e0e\u7a00\u758f\u77e9\u9635\u62fc\u63a5，\u5047\u8bbe\u7a00\u758f\u77e9\u9635\u53ea\u6709point\u4e3aNx2x256，\u62fc\u63a5\u4e4b\u540e\u5219\u4e3aNx(5+2)x256<\/span>
          \n # B,5,E and B,N,E –>B,5+N,E N\u662f\u70b9\u7684\u4e2a\u6570(\u6807\u8bb0\u70b9\u548c\u6807\u8bb0\u6846\u7684\u70b9)<\/span>
          \n tokens =<\/span> torch.<\/span>cat(<\/span>(<\/span>output_tokens,<\/span> sparse_prompt_embeddings)<\/span>,<\/span> dim=<\/span>1<\/span>)<\/span><\/p>\n

          # \u5c06image embedding(1x256x64x64)\u62d3\u5c55\u6210\u7a20\u5bc6prompt\u7684\u7ef4\u5ea6：Nx256x64x64<\/span>
          \n # B,C,H,W<\/span>
          \n src =<\/span> torch.<\/span>repeat_interleave(<\/span>image_embeddings,<\/span> tokens.<\/span>shape[<\/span>0<\/span>]<\/span>,<\/span> dim=<\/span>0<\/span>)<\/span>
          \n #\u5c06\u62d3\u5c55\u540e\u7684image embedding\u76f4\u63a5\u4e0e\u7a20\u5bc6prompt\u76f8\u52a0：Nx256x64x64<\/span>
          \n # B,C,H,W + 1,C,H,W —> B,C,H,W<\/span>
          \n src =<\/span> src +<\/span> dense_prompt_embeddings
          \n # # \u5c06256x64x64\u7684\u4f4d\u7f6e\u7f16\u7801,\u62d3\u5c55\u6210Nx256x64x64<\/span>
          \n # 1,C,H,W—> B,C,H,W<\/span>
          \n pos_src =<\/span> torch.<\/span>repeat_interleave(<\/span>image_pe,<\/span> tokens.<\/span>shape[<\/span>0<\/span>]<\/span>,<\/span> dim=<\/span>0<\/span>)<\/span>
          \n b,<\/span> c,<\/span> h,<\/span> w =<\/span> src.<\/span>shape<\/p>\n

          # —– transformer —–<\/span>
          \n # Run the transformer：\u8fd9\u91cc\u4f7f\u7528\u7684TwoWayTransformer，\u6709\u5fc5\u8981\u5bf9\u8f93\u5165\u518d\u8bf4\u660e\u4e00\u4e0b<\/span>
          \n # src：image_bedding + dense_prompt（mask）,Nx256x64x64<\/span>
          \n # pos_src: \u4f4d\u7f6e\u7f16\u7801,Nx256x64x64<\/span>
          \n # tokens: iou_tokens + mask_tokens + sparse_prompt（point\/bbox）,Nx(5+x)x256<\/span>
          \n # B,N,C<\/span>
          \n hs,<\/span> src =<\/span> self.<\/span>transformer(<\/span>src,<\/span> pos_src,<\/span> tokens)<\/span>
          \n # —– transformer —–<\/span>
          \n # # \u540e\u5904\u7406<\/span>
          \n iou_token_out =<\/span> hs[<\/span>:<\/span>,<\/span> 0<\/span>,<\/span> :<\/span>]<\/span>
          \n mask_tokens_out =<\/span> hs[<\/span>:<\/span>,<\/span> 1<\/span>:<\/span> (<\/span>1<\/span> +<\/span> self.<\/span>num_mask_tokens)<\/span>,<\/span> :<\/span>]<\/span><\/p>\n

          # \u901a\u8fc7\u4e0a\u91c7\u6837\u5c42\u5c06Transformer\u8f93\u51fa\u7684\u63a9\u6a21\u90e8\u5206\u6062\u590d\u5230(batch_size, channels, height, width)\u7684\u5f62\u72b6<\/span>
          \n # B,N,C–>B,C,H,W<\/span>
          \n src =<\/span> src.<\/span>transpose(<\/span>1<\/span>,<\/span> 2<\/span>)<\/span>.<\/span>view(<\/span>b,<\/span> c,<\/span> h,<\/span> w)<\/span>
          \n # —– upscaled —–<\/span>
          \n # 4\u500d\u4e0a\u91c7\u6837<\/span>
          \n upscaled_embedding =<\/span> self.<\/span>output_upscaling(<\/span>src)<\/span>
          \n # —– upscaled —–<\/span><\/p>\n

          # \u5bf9\u6bcf\u4e2amask token，\u901a\u8fc7\u5176\u5bf9\u5e94\u7684MLP\u5f97\u5230\u4e00\u4e2a\u6743\u91cd\u5f20\u91cf，\u4f7f\u7528\u8fd9\u4e9b\u6743\u91cd\u4e0e\u4e0a\u91c7\u6837\u540e\u7684\u7279\u5f81\u5f20\u91cf\u8fdb\u884c\u70b9\u4e58，\u5f97\u5230\u63a9\u6a21\u9884\u6d4b(batch_size, num_mask_tokens, height, width)<\/span>
          \n hyper_in_list:<\/span> List[<\/span>torch.<\/span>Tensor]<\/span> =<\/span> [<\/span>]<\/span><\/p>\n

          # —– mlp —–<\/span>
          \n for<\/span> i in<\/span> range<\/span>(<\/span>self.<\/span>num_mask_tokens)<\/span>:<\/span>
          \n # mask_tokens_out[:, i, :]: B,1,C<\/span>
          \n # output_hypernetworks_mlps: B,1,c<\/span>
          \n hyper_in_list.<\/span>append(<\/span>self.<\/span>output_hypernetworks_mlps[<\/span>i]<\/span>(<\/span>mask_tokens_out[<\/span>:<\/span>,<\/span> i,<\/span> :<\/span>]<\/span>)<\/span>)<\/span>
          \n # B,n,c<\/span>
          \n hyper_in =<\/span> torch.<\/span>stack(<\/span>hyper_in_list,<\/span> dim=<\/span>1<\/span>)<\/span>
          \n # —– mlp —–<\/span><\/p>\n

          b,<\/span> c,<\/span> h,<\/span> w =<\/span> upscaled_embedding.<\/span>shape
          \n # B,n,c \u00d7 B,c,N–>B,n,h,w<\/span>
          \n masks =<\/span> (<\/span>hyper_in @ upscaled_embedding.<\/span>view(<\/span>b,<\/span> c,<\/span> h *<\/span> w)<\/span>)<\/span>.<\/span>view(<\/span>b,<\/span> –<\/span>1<\/span>,<\/span> h,<\/span> w)<\/span><\/p>\n

          # —– mlp —–<\/span>
          \n # \u901a\u8fc7IoU\u9884\u6d4b\u5934（MLP）\u5bf9IoU token\u7684\u8f93\u51fa\u8fdb\u884c\u5904\u7406，\u5f97\u5230(batch_size, num_mask_tokens)\u7684IoU\u5206\u6570<\/span>
          \n # iou_token_out: B,1,n<\/span>
          \n iou_pred =<\/span> self.<\/span>iou_prediction_head(<\/span>iou_token_out)<\/span>
          \n # —– mlp —–<\/span>
          \n # \u8fd4\u56de\u9884\u6d4b\u7684\u63a9\u6a21\u548cIoU\u5206\u6570<\/span>
          \n # masks: B,n,h,w<\/span>
          \n # iou_pred: B,1,n<\/span>
          \n return<\/span> masks,<\/span> iou_pred<\/p>\n

          1. transformer<\/h5>\n

          Mask Decoder\u7531\u591a\u4e2a\u91cd\u590d\u5806\u53e0TwoWayAttention Block\u548c1\u4e2aMulti-Head Attention\u7ec4\u6210\u3002<\/p>\n

          \"\u56fe\u7247\"<\/p>\n

          \u300cTwoWayAttention Block\u300d<\/p>\n

          TwoWayAttention Block\u7531LayerNorm \u3001Multi-Head Attention\u548cMLP\u6784\u6210\u3002\u6240\u8c13\u7684TwoWay：\u5373\u662f\u4e24\u8f6e\u6b21\u5faa\u73af，\u7b2c\u4e00\u6b21point_embedding\u81ea\u6ce8\u610f，\u7b2c\u4e8c\u6b21\u5219\u52a0\u4e0a\u4e0a\u4e00\u8f6e\u8f93\u51fa\u7684queries\u8fdb\u884cattention\u3002<\/p>\n

          \"\u56fe\u7247\"<\/p>\n

          \u539f\u8bba\u6587\u4e2dTwoWayAttention\u90e8\u5206\u793a\u610f\u56fe\u3002<\/p>\n

          \"\u56fe\u7247\"<\/p>\n

          class<\/span> TwoWayAttentionBlock<\/span>(<\/span>nn.<\/span>Module)<\/span>:<\/span>
          \n def<\/span> __init__<\/span>(<\/span>
          \n self,<\/span>
          \n embedding_dim:<\/span> int<\/span>,<\/span> # \u8f93\u5165\u7279\u5f81\u7ef4\u5ea6<\/span>
          \n num_heads:<\/span> int<\/span>,<\/span> # \u6ce8\u610f\u529b\u5934\u7684\u6570\u91cf，\u51b3\u5b9a\u4e86\u6ce8\u610f\u529b\u673a\u5236\u7684\u5e76\u884c\u5ea6<\/span>
          \n mlp_dim:<\/span> int<\/span> =<\/span> 2048<\/span>,<\/span> # MLP（\u591a\u5c42\u611f\u77e5\u673a）\u4e2d\u95f4\u5c42\u7684\u7ef4\u5ea6，\u7528\u4e8e\u7279\u5f81\u53d8\u6362\u548c\u975e\u7ebf\u6027\u589e\u5f3a<\/span>
          \n activation:<\/span> Type[<\/span>nn.<\/span>Module]<\/span> =<\/span> nn.<\/span>ReLU,<\/span> # \u6fc0\u6d3b\u51fd\u6570\u7c7b\u578b，\u9ed8\u8ba4\u4e3aReLU<\/span>
          \n attention_downsample_rate:<\/span> int<\/span> =<\/span> 2<\/span>,<\/span> # \u4e0b\u91c7\u6837\u6bd4\u7387<\/span>
          \n # \u662f\u5426\u5728\u7b2c\u4e00\u5c42\u81ea\u6ce8\u610f\u529b\u4e2d\u8df3\u8fc7\u4f4d\u7f6e\u7f16\u7801\u7684\u6b8b\u5dee\u8fde\u63a5<\/span>
          \n skip_first_layer_pe:<\/span> bool<\/span> =<\/span> False<\/span>,<\/span>
          \n )<\/span> –<\/span>><\/span> None<\/span>:<\/span>
          \n super<\/span>(<\/span>)<\/span>.<\/span>__init__(<\/span>)<\/span>
          \n # \u81ea\u6ce8\u610f\u529b\u6a21\u5757，\u7528\u4e8e\u589e\u5f3aqueries\u5185\u90e8\u7684\u4fe1\u606f\u4ea4\u4e92<\/span>
          \n self.<\/span>self_attn =<\/span> Attention(<\/span>embedding_dim,<\/span> num_heads)<\/span>
          \n # norm1\/2\/3\/4: LayerNorm\u5c42，\u7528\u4e8e\u7a33\u5b9a\u8bad\u7ec3\u548c\u52a0\u901f\u6536\u655b<\/span>
          \n self.<\/span>norm1 =<\/span> nn.<\/span>LayerNorm(<\/span>embedding_dim)<\/span>
          \n # cross_attn_token_to_image\u548ccross_attn_image_to_token: \u4ea4\u53c9\u6ce8\u610f\u529b\u6a21\u5757，\u5206\u522b\u8ba9\u6807\u8bb0\u70b9\u7279\u5f81\u5173\u6ce8\u56fe\u50cf\u7279\u5f81，\u4ee5\u53ca\u56fe\u50cf\u7279\u5f81\u53cd\u8fc7\u6765\u5173\u6ce8\u6807\u8bb0\u70b9\u7279\u5f81<\/span>
          \n self.<\/span>cross_attn_token_to_image =<\/span> Attention(<\/span>
          \n embedding_dim,<\/span> num_heads,<\/span> downsample_rate=<\/span>attention_downsample_rate
          \n )<\/span>
          \n self.<\/span>norm2 =<\/span> nn.<\/span>LayerNorm(<\/span>embedding_dim)<\/span>
          \n # mlp: \u591a\u5c42\u611f\u77e5\u673a\u6a21\u5757，\u589e\u52a0\u6a21\u578b\u7684\u8868\u8fbe\u80fd\u529b<\/span>
          \n self.<\/span>mlp =<\/span> MLPBlock(<\/span>embedding_dim,<\/span> mlp_dim,<\/span> activation)<\/span>
          \n self.<\/span>norm3 =<\/span> nn.<\/span>LayerNorm(<\/span>embedding_dim)<\/span><\/p>\n

          self.<\/span>norm4 =<\/span> nn.<\/span>LayerNorm(<\/span>embedding_dim)<\/span>
          \n self.<\/span>cross_attn_image_to_token =<\/span> Attention(<\/span>
          \n embedding_dim,<\/span> num_heads,<\/span> downsample_rate=<\/span>attention_downsample_rate
          \n )<\/span>
          \n self.<\/span>skip_first_layer_pe =<\/span> skip_first_layer_pe
          \n # \u524d\u5411\u4f20\u64ad<\/span>
          \n def<\/span> forward<\/span>(<\/span>
          \n self,<\/span> queries:<\/span> Tensor,<\/span> keys:<\/span> Tensor,<\/span> query_pe:<\/span> Tensor,<\/span> key_pe:<\/span> Tensor
          \n )<\/span> –<\/span>><\/span> Tuple[<\/span>Tensor,<\/span> Tensor]<\/span>:<\/span><\/p>\n

          # queries：\u6807\u8bb0\u70b9\u7f16\u7801\u76f8\u5173(\u539f\u59cb\u6807\u8bb0\u70b9\u7f16\u7801\u7ecf\u8fc7\u4e00\u7cfb\u5217\u7279\u5f81\u63d0\u53d6)<\/span>
          \n # keys：\u539f\u59cb\u56fe\u50cf\u7f16\u7801\u76f8\u5173(\u539f\u59cb\u56fe\u50cf\u7f16\u7801\u7ecf\u8fc7\u4e00\u7cfb\u5217\u7279\u5f81\u63d0\u53d6)<\/span>
          \n # query_pe：\u539f\u59cb\u6807\u8bb0\u70b9\u7f16\u7801<\/span>
          \n # key_pe：\u539f\u59cb\u56fe\u50cf\u4f4d\u7f6e\u7f16\u7801<\/span>
          \n # \u7b2c\u4e00\u8f6e\u672c\u8eabqueries==query_pe\u6ca1\u6bd4\u8f83\u518d"\u6b8b\u5dee"<\/span><\/p>\n

          # \u9996\u5148\u5bf9queries\u5e94\u7528\u81ea\u6ce8\u610f\u529b，\u82e5skip_first_layer_pe=True，\u76f4\u63a5\u4f7f\u7528queries\u8fdb\u884c\u81ea\u6ce8\u610f\u529b\u8ba1\u7b97；\u5426\u5219，\u5c06queries\u4e0equery_pe\u76f8\u52a0\u540e\u8fdb\u884c\u81ea\u6ce8\u610f\u529b\u8ba1\u7b97，\u5e76\u6b8b\u5dee\u8fde\u63a5\u56dequeries，\u4e4b\u540e\u8fdb\u884cLayerNorm<\/span>
          \n if<\/span> self.<\/span>skip_first_layer_pe:<\/span>
          \n queries =<\/span> self.<\/span>self_attn(<\/span>q=<\/span>queries,<\/span> k=<\/span>queries,<\/span> v=<\/span>queries)<\/span>
          \n else<\/span>:<\/span>
          \n q =<\/span> queries +<\/span> query_pe
          \n attn_out =<\/span> self.<\/span>self_attn(<\/span>q=<\/span>q,<\/span> k=<\/span>q,<\/span> v=<\/span>queries)<\/span>
          \n queries =<\/span> queries +<\/span> attn_out
          \n queries =<\/span> self.<\/span>norm1(<\/span>queries)<\/span><\/p>\n

          # \u8c03\u6574queries\u548ckeys（\u56fe\u50cf\u7279\u5f81）\u52a0\u4e0a\u5404\u81ea\u7684\u4f4d\u7f6e\u7f16\u7801，\u7136\u540e\u901a\u8fc7cross_attn_token_to_image\u4ea4\u53c9\u6ce8\u610f\u529b\u5c42，\u4f7f\u6807\u8bb0\u70b9\u7279\u5f81\u5173\u6ce8\u56fe\u50cf\u7279\u5f81，\u7ed3\u679c\u4e0e\u539f\u59cbqueries\u6b8b\u5dee\u8fde\u63a5\u5e76\u8fdb\u884cLayerNorm<\/span>
          \n q =<\/span> queries +<\/span> query_pe
          \n k =<\/span> keys +<\/span> key_pe
          \n attn_out =<\/span> self.<\/span>cross_attn_token_to_image(<\/span>q=<\/span>q,<\/span> k=<\/span>k,<\/span> v=<\/span>keys)<\/span>
          \n queries =<\/span> queries +<\/span> attn_out
          \n queries =<\/span> self.<\/span>norm2(<\/span>queries)<\/span><\/p>\n

          # MLP block：\u5c06\u66f4\u65b0\u540e\u7684queries\u901a\u8fc7MLP\u6a21\u5757\u8fdb\u884c\u975e\u7ebf\u6027\u53d8\u6362，\u7ed3\u679c\u4e0e\u539fqueries\u6b8b\u5dee\u8fde\u63a5\u5e76\u8fdb\u884cLayerNorm<\/span>
          \n mlp_out =<\/span> self.<\/span>mlp(<\/span>queries)<\/span>
          \n queries =<\/span> queries +<\/span> mlp_out
          \n queries =<\/span> self.<\/span>norm3(<\/span>queries)<\/span><\/p>\n

          # \u4ea4\u53c9\u6ce8\u610f\u529b（\u56fe\u50cf\u5230\u6807\u8bb0\u70b9）：\u518d\u6b21\u8c03\u6574queries\u548ckeys\u52a0\u4e0a\u4f4d\u7f6e\u7f16\u7801，\u4f46\u8fd9\u6b21\u901a\u8fc7cross_attn_image_to_token\u8ba9\u56fe\u50cf\u7279\u5f81\u5173\u6ce8\u6807\u8bb0\u70b9\u7279\u5f81，\u66f4\u65b0\u540e\u7684keys\u4e0e\u539f\u59cbkeys\u6b8b\u5dee\u8fde\u63a5\u5e76\u8fdb\u884cLayerNorm<\/span>
          \n q =<\/span> queries +<\/span> query_pe
          \n k =<\/span> keys +<\/span> key_pe
          \n attn_out =<\/span> self.<\/span>cross_attn_image_to_token(<\/span>q=<\/span>k,<\/span> k=<\/span>q,<\/span> v=<\/span>queries)<\/span>
          \n keys =<\/span> keys +<\/span> attn_out
          \n keys =<\/span> self.<\/span>norm4(<\/span>keys)<\/span>
          \n return<\/span> queries,<\/span> keys<\/p>\n

          \u300cAttention\u300d<\/p>\n

          Mask Decoder\u7684Attention\u4e0eViT\u7684Attention\u6709\u4e9b\u7ec6\u5fae\u7684\u4e0d\u540c：<\/p>\n

            \n
          • \n

            Mask Decoder\u7684Attention\u662f3\u4e2aFC\u5c42\u5206\u522b\u63a5\u53d73\u4e2a\u8f93\u5165\u83b7\u5f97q\u3001k\u548cv\u3002<\/p>\n<\/li>\n

          • \n

            ViT\u7684Attention\u662f1\u4e2aFC\u5c42\u63a5\u53d71\u4e2a\u8f93\u5165\u540e\u5c06\u7ed3\u679c\u5747\u62c6\u5206\u83b7\u5f97q\u3001k\u548cv\u3002<\/p>\n<\/li>\n<\/ul>\n

            \u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

            \"\u56fe\u7247\"<\/p>\n

            \u539f\u8bba\u6587\u4e2dAttention\u90e8\u5206\u793a\u610f\u56fe\u3002<\/p>\n

            \"\u56fe\u7247\"<\/p>\n

            class<\/span> Attention<\/span>(<\/span>nn.<\/span>Module)<\/span>:<\/span><\/p>\n

            def<\/span> __init__<\/span>(<\/span>
            \n self,<\/span>
            \n embedding_dim:<\/span> int<\/span>,<\/span> # \u8f93\u5165\u7279\u5f81\u7684\u7ef4\u5ea6<\/span>
            \n num_heads:<\/span> int<\/span>,<\/span> # attention\u7684head\u6570<\/span>
            \n downsample_rate:<\/span> int<\/span> =<\/span> 1<\/span>,<\/span> # \u4e0b\u91c7\u6837<\/span>
            \n )<\/span> –<\/span>><\/span> None<\/span>:<\/span>
            \n super<\/span>(<\/span>)<\/span>.<\/span>__init__(<\/span>)<\/span>
            \n self.<\/span>embedding_dim =<\/span> embedding_dim
            \n # \u5185\u90e8\u7ef4\u5ea6<\/span>
            \n self.<\/span>internal_dim =<\/span> embedding_dim \/\/<\/span> downsample_rate
            \n self.<\/span>num_heads =<\/span> num_heads
            \n assert<\/span> self.<\/span>internal_dim %<\/span> num_heads ==<\/span> 0<\/span>,<\/span> "num_heads must divide embedding_dim."<\/span>
            \n # \u56db\u4e2a\u7ebf\u6027\u5c42（\u5168\u8fde\u63a5\u5c42）：\u7528\u4e8e\u751f\u6210query\u5411\u91cf\u3001key\u5411\u91cf\u3001value\u5411\u91cf<\/span>
            \n self.<\/span>q_proj =<\/span> nn.<\/span>Linear(<\/span>embedding_dim,<\/span> self.<\/span>internal_dim)<\/span>
            \n self.<\/span>k_proj =<\/span> nn.<\/span>Linear(<\/span>embedding_dim,<\/span> self.<\/span>internal_dim)<\/span>
            \n self.<\/span>v_proj =<\/span> nn.<\/span>Linear(<\/span>embedding_dim,<\/span> self.<\/span>internal_dim)<\/span>
            \n # \u7528\u4e8e\u5c06\u6ce8\u610f\u529b\u673a\u5236\u540e\u7684\u8f93\u51fa\u6295\u5f71\u56de\u539f\u59cb\u7684\u7279\u5f81\u7ef4\u5ea6<\/span>
            \n self.<\/span>out_proj =<\/span> nn.<\/span>Linear(<\/span>self.<\/span>internal_dim,<\/span> embedding_dim)<\/span>
            \n # \u5c06\u8f93\u5165\u5f20\u91cf\u5206\u89e3\u4e3a\u591a\u5934\u6ce8\u610f\u529b\u6240\u9700\u7684\u5f62\u72b6<\/span>
            \n def<\/span> _separate_heads<\/span>(<\/span>self,<\/span> x:<\/span> Tensor,<\/span> num_heads:<\/span> int<\/span>)<\/span> –<\/span>><\/span> Tensor:<\/span>
            \n b,<\/span> n,<\/span> c =<\/span> x.<\/span>shape
            \n x =<\/span> x.<\/span>reshape(<\/span>b,<\/span> n,<\/span> num_heads,<\/span> c \/\/<\/span> num_heads)<\/span>
            \n return<\/span> x.<\/span>transpose(<\/span>1<\/span>,<\/span> 2<\/span>)<\/span> # B x N_heads x N_tokens x C_per_head<\/span>
            \n # \u5728\u6ce8\u610f\u529b\u8ba1\u7b97\u540e\u91cd\u65b0\u7ec4\u5408\u8fd9\u4e9b\u5934\u90e8<\/span>
            \n def<\/span> _recombine_heads<\/span>(<\/span>self,<\/span> x:<\/span> Tensor)<\/span> –<\/span>><\/span> Tensor:<\/span>
            \n b,<\/span> n_heads,<\/span> n_tokens,<\/span> c_per_head =<\/span> x.<\/span>shape
            \n x =<\/span> x.<\/span>transpose(<\/span>1<\/span>,<\/span> 2<\/span>)<\/span>
            \n return<\/span> x.<\/span>reshape(<\/span>b,<\/span> n_tokens,<\/span> n_heads *<\/span> c_per_head)<\/span> # B x N_tokens x C<\/span><\/p>\n

            def<\/span> forward<\/span>(<\/span>self,<\/span> q:<\/span> Tensor,<\/span> k:<\/span> Tensor,<\/span> v:<\/span> Tensor)<\/span> –<\/span>><\/span> Tensor:<\/span>
            \n # \u8f93\u5165\u6295\u5f71：\u5206\u522b\u4f7f\u7528q_proj\u3001k_proj\u548cv_proj\u5bf9query\u3001key\u548cvalue\u8fdb\u884c\u7ebf\u6027\u53d8\u6362<\/span>
            \n q =<\/span> self.<\/span>q_proj(<\/span>q)<\/span>
            \n k =<\/span> self.<\/span>k_proj(<\/span>k)<\/span>
            \n v =<\/span> self.<\/span>v_proj(<\/span>v)<\/span><\/p>\n

            # \u5206\u79bb\u5934\u90e8：\u5c06\u53d8\u6362\u540e\u7684query\u3001key\u548cvalue\u5f20\u91cf\u6309\u7167num_heads\u8fdb\u884c\u91cd\u5851，\u4ee5\u4fbf\u8fdb\u884c\u591a\u5934\u6ce8\u610f\u529b\u8ba1\u7b97<\/span>
            \n # B,N_heads,N_tokens,C_per_head<\/span>
            \n q =<\/span> self.<\/span>_separate_heads(<\/span>q,<\/span> self.<\/span>num_heads)<\/span>
            \n k =<\/span> self.<\/span>_separate_heads(<\/span>k,<\/span> self.<\/span>num_heads)<\/span>
            \n v =<\/span> self.<\/span>_separate_heads(<\/span>v,<\/span> self.<\/span>num_heads)<\/span><\/p>\n

            # \u6ce8\u610f\u529b\u8ba1\u7b97：<\/span>
            \n # \u8ba1\u7b97query\u548ckey\u7684\u70b9\u79ef，\u7136\u540e\u9664\u4ee5c_per_head\u7684\u5e73\u65b9\u6839\u8fdb\u884c\u5f52\u4e00\u5316，\u4ee5\u9632\u6b62\u6570\u503c\u8fc7\u5927<\/span>
            \n _,<\/span> _,<\/span> _,<\/span> c_per_head =<\/span> q.<\/span>shape
            \n attn =<\/span> q @ k.<\/span>permute(<\/span>0<\/span>,<\/span> 1<\/span>,<\/span> 3<\/span>,<\/span> 2<\/span>)<\/span> # B,N_heads,N_tokens,C_per_head<\/span>
            \n # \u5f52\u4e00\u5316Scale<\/span>
            \n attn =<\/span> attn \/<\/span> math.<\/span>sqrt(<\/span>c_per_head)<\/span>
            \n # \u5e94\u7528softmax\u51fd\u6570\u5f97\u5230\u6ce8\u610f\u529b\u6743\u91cd<\/span>
            \n attn =<\/span> torch.<\/span>softmax(<\/span>attn,<\/span> dim=<\/span>–<\/span>1<\/span>)<\/span>
            \n # \u4f7f\u7528\u6ce8\u610f\u529b\u6743\u91cd\u5bf9value\u8fdb\u884c\u52a0\u6743\u6c42\u548c，\u5f97\u5230\u6ce8\u610f\u529b\u8f93\u51fa<\/span>
            \n out =<\/span> attn @ v
            \n # # B,N_tokens,C<\/span>
            \n # \u91cd\u65b0\u7ec4\u5408\u5934\u90e8：\u5c06\u591a\u5934\u6ce8\u610f\u529b\u8f93\u51fa\u5408\u5e76\u56de\u539f\u59cb\u7684\u7279\u5f81\u7ef4\u5ea6\u3002<\/span>
            \n out =<\/span> self.<\/span>_recombine_heads(<\/span>out)<\/span>
            \n # \u8f93\u51fa\u6295\u5f71：\u6700\u540e，\u901a\u8fc7out_proj\u5c06\u8f93\u51fa\u6295\u5f71\u56de\u539f\u59cb\u7684embedding_dim<\/span>
            \n out =<\/span> self.<\/span>out_proj(<\/span>out)<\/span>
            \n return<\/span> out<\/p>\n

            \u300ctransformer_MLP\u300d<\/p>\n

            transformer\u4e2dMLP\u7684\u7ed3\u6784\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

            \"\u56fe\u7247\"<\/p>\n

            # MLPBlock\u7c7b\u662f\u4e00\u4e2a\u7b80\u5355\u7684\u591a\u5c42\u611f\u77e5\u673a（MLP）\u6a21\u5757，\u7531\u4e24\u4e2a\u5168\u8fde\u63a5\u5c42（Linear）\u548c\u4e00\u4e2a\u6fc0\u6d3b\u51fd\u6570\u7ec4\u6210<\/span>
            \nclass<\/span> MLPBlock<\/span>(<\/span>nn.<\/span>Module)<\/span>:<\/span>
            \n def<\/span> __init__<\/span>(<\/span>
            \n self,<\/span>
            \n # \u8f93\u5165\u7684\u7ef4\u5ea6，\u901a\u5e38\u662f\u7279\u5f81\u5411\u91cf\u7684\u957f\u5ea6<\/span>
            \n embedding_dim:<\/span> int<\/span>,<\/span>
            \n # MLP\u4e2d\u95f4\u5c42\u7684\u5bbd\u5ea6，\u53ef\u4ee5\u8bbe\u7f6e\u4e3a\u6bd4\u8f93\u5165\u7ef4\u5ea6\u66f4\u5927\u7684\u503c\u4ee5\u589e\u52a0\u6a21\u578b\u7684\u8868\u8fbe\u80fd\u529b<\/span>
            \n mlp_dim:<\/span> int<\/span>,<\/span>
            \n # \u6fc0\u6d3b\u51fd\u6570，\u8fd9\u91cc\u9ed8\u8ba4\u4f7f\u7528GELU<\/span>
            \n act:<\/span> Type[<\/span>nn.<\/span>Module]<\/span> =<\/span> nn.<\/span>GELU,<\/span>
            \n )<\/span> –<\/span>><\/span> None<\/span>:<\/span>
            \n super<\/span>(<\/span>)<\/span>.<\/span>__init__(<\/span>)<\/span>
            \n # \u7b2c\u4e00\u4e2a\u5168\u8fde\u63a5\u5c42，\u5c06\u8f93\u5165\u4eceembedding_dim\u7ef4\u5ea6\u53d8\u6362\u5230mlp_dim\u7ef4\u5ea6<\/span>
            \n self.<\/span>lin1 =<\/span> nn.<\/span>Linear(<\/span>embedding_dim,<\/span> mlp_dim)<\/span>
            \n # \u7b2c\u4e8c\u4e2a\u5168\u8fde\u63a5\u5c42，\u5c06mlp_dim\u7ef4\u5ea6\u7684\u7ed3\u679c\u53d8\u6362\u56deembedding_dim\u7ef4\u5ea6，\u4ee5\u4fdd\u6301\u4e0e\u8f93\u5165\u76f8\u540c\u7684\u7ef4\u5ea6<\/span>
            \n self.<\/span>lin2 =<\/span> nn.<\/span>Linear(<\/span>mlp_dim,<\/span> embedding_dim)<\/span>
            \n # \u6fc0\u6d3b\u51fd\u6570\u5b9e\u4f8b，\u7528\u4e8e\u5728\u5168\u8fde\u63a5\u5c42\u4e4b\u95f4\u5f15\u5165\u975e\u7ebf\u6027<\/span>
            \n self.<\/span>act =<\/span> act(<\/span>)<\/span>
            \n # \u63a5\u6536\u8f93\u5165\u5f20\u91cfx，\u5c06\u5176\u4f20\u9012\u7ed9lin1，\u7136\u540e\u5e94\u7528\u6fc0\u6d3b\u51fd\u6570act\u3002<\/span>
            \n # \u5c06\u6fc0\u6d3b\u51fd\u6570\u7684\u8f93\u51fa\u4f20\u9012\u7ed9lin2，\u5f97\u5230\u6700\u7ec8\u7684\u8f93\u51fa\u5f20\u91cf<\/span>
            \n def<\/span> forward<\/span>(<\/span>self,<\/span> x:<\/span> torch.<\/span>Tensor)<\/span> –<\/span>><\/span> torch.<\/span>Tensor:<\/span>
            \n return<\/span> self.<\/span>lin2(<\/span>self.<\/span>act(<\/span>self.<\/span>lin1(<\/span>x)<\/span>)<\/span>)<\/span><\/p>\n

            \u300cupscaled\u300d<\/p>\n

            \u8fd9\u4e2a\u4e0a\u91c7\u6837\u8fc7\u7a0b\u5c06Transformer\u7684\u8f93\u51fa\u7279\u5f81\u56fe\u6062\u590d\u5230\u66f4\u63a5\u8fd1\u8f93\u5165\u56fe\u50cf\u7684\u5206\u8fa8\u7387，\u4ee5\u4fbf\u4e8e\u751f\u6210\u63a9\u6a21\u9884\u6d4b\u3002upscaled\u7684\u7ed3\u6784\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

            \"\u56fe\u7247\"<\/p>\n

            # \u5728MaskDecoder\u7684__init__\u5b9a\u4e49<\/span>
            \n# output_upscaling\u662f\u4e00\u4e2a\u5e8f\u5217\u6a21\u5757，\u7528\u4e8e\u4e0a\u91c7\u6837Transformer\u8f93\u51fa\u7684\u7279\u5f81\u56fe<\/span>
            \nself.<\/span>output_upscaling =<\/span> nn.<\/span>Sequential(<\/span>
            \n # \u4f7f\u7528nn.ConvTranspose2d，\u8f93\u5165\u901a\u9053\u6570\u4e3atransformer_dim，\u8f93\u51fa\u901a\u9053\u6570\u4e3atransformer_dim \/\/ 4，\u5185\u6838\u5927\u5c0f\u4e3a2，\u6b65\u957f\u4e3a2<\/span>
            \n # \u5c06\u7279\u5f81\u56fe\u7684\u5c3a\u5bf8\u653e\u5927\u4e24\u500d，\u540c\u65f6\u5c06\u901a\u9053\u6570\u51cf\u534a<\/span>
            \n # \u5185\u6838\u5927\u5c0f\u4e3a2\u7684\u8f6c\u7f6e\u5377\u79ef\u76f8\u5f53\u4e8e\u4e0a\u91c7\u68372\u500d，\u6b65\u957f\u4e3a2\u786e\u4fdd\u8f93\u51fa\u5c3a\u5bf8\u7ffb\u500d<\/span>
            \n nn.<\/span>ConvTranspose2d(<\/span>transformer_dim,<\/span> transformer_dim \/\/<\/span> 4<\/span>,<\/span> kernel_size=<\/span>2<\/span>,<\/span> stride=<\/span>2<\/span>)<\/span>,<\/span> #\u8f6c\u7f6e\u5377\u79ef \u4e0a\u91c7\u68372\u500d<\/span>
            \n # \u5c42\u5f52\u4e00\u5316（LayerNorm2d）<\/span>
            \n LayerNorm2d(<\/span>transformer_dim \/\/<\/span> 4<\/span>)<\/span>,<\/span>
            \n # \u6fc0\u6d3b\u51fd\u6570<\/span>
            \n activation(<\/span>)<\/span>,<\/span>
            \n # \u518d\u6b21\u4f7f\u7528nn.ConvTranspose2d，\u8f93\u5165\u901a\u9053\u6570\u4e3atransformer_dim \/\/ 4，\u8f93\u51fa\u901a\u9053\u6570\u4e3atransformer_dim \/\/ 8，\u5185\u6838\u5927\u5c0f\u4e3a2，\u6b65\u957f\u4e3a2\u3002\u8fd9\u4e00\u6b65\u7ee7\u7eed\u5c06\u7279\u5f81\u56fe\u7684\u5c3a\u5bf8\u653e\u5927\u4e24\u500d，\u540c\u65f6\u901a\u9053\u6570\u518d\u6b21\u51cf\u534a<\/span>
            \n nn.<\/span>ConvTranspose2d(<\/span>transformer_dim \/\/<\/span> 4<\/span>,<\/span> transformer_dim \/\/<\/span> 8<\/span>,<\/span> kernel_size=<\/span>2<\/span>,<\/span> stride=<\/span>2<\/span>)<\/span>,<\/span>
            \n # \u91cd\u590d\u6fc0\u6d3b\u51fd\u6570\u7684\u8fc7\u7a0b，\u4ee5\u8fdb\u4e00\u6b65\u589e\u5f3a\u975e\u7ebf\u6027\u8868\u8fbe<\/span>
            \n activation(<\/span>)<\/span>,<\/span>
            \n)<\/span>
            \n# \u5728MaskDecoder\u7684predict_masks\u6dfb\u52a0\u4f4d\u7f6e\u7f16\u7801<\/span>
            \nupscaled_embedding =<\/span> self.<\/span>output_upscaling(<\/span>src)<\/span><\/p>\n

            \u300cmask_MLP\u300d<\/p>\n

            \u6b64\u5904\u7684MLP\u57fa\u7840\u6a21\u5757\u4e0d\u540c\u4e8eViT\u7684MLP(transformer_MLP)\u57fa\u7840\u6a21\u5757\u3002<\/p>\n

            # \u5728MaskDecoder\u7684__init__\u5b9a\u4e49<\/span>
            \n# output_hypernetworks_mlps\u662f\u4e00\u4e2ann.ModuleList，\u5305\u542b\u4e86\u591a\u4e2a\u591a\u5c42\u611f\u77e5\u673a（MLP）\u3002\u6bcf\u4e2aMLP\u7684\u76ee\u7684\u662f\u6839\u636e\u8f93\u5165\u7684mask_tokens_out\u751f\u6210\u7279\u5b9a\u63a9\u6a21\u7684\u8d85\u7f51\u7edc\u6743\u91cd<\/span>
            \nself.<\/span>output_hypernetworks_mlps =<\/span> nn.<\/span>ModuleList(<\/span>
            \n [<\/span>
            \n # transformer_dim: Transformer\u7684\u8f93\u51fa\u7ef4\u5ea6，\u4e5f\u662f\u8f93\u5165\u5230MLP\u7684\u901a\u9053\u6570<\/span>
            \n # transformer_dim \/\/ 8: MLP\u7684\u8f93\u51fa\u901a\u9053\u6570，\u7528\u4e8e\u751f\u6210\u8d85\u7f51\u7edc\u7684\u6743\u91cd<\/span>
            \n # 3: MLP\u7684\u4e2d\u95f4\u5c42\u7ef4\u5ea6，\u7528\u4e8e\u589e\u52a0\u6a21\u578b\u7684\u8868\u8fbe\u80fd\u529b<\/span>
            \n MLP(<\/span>transformer_dim,<\/span> transformer_dim,<\/span> transformer_dim \/\/<\/span> 8<\/span>,<\/span> 3<\/span>)<\/span>
            \n for<\/span> i in<\/span> range<\/span>(<\/span>self.<\/span>num_mask_tokens)<\/span>
            \n ]<\/span>
            \n)<\/span>
            \n# \u5728MaskDecoder\u7684predict_masks\u6dfb\u52a0\u4f4d\u7f6e\u7f16\u7801<\/span>
            \n# \u5bf9\u4e8eself.num_mask_tokens\u4e2a\u63a9\u6a21token，\u904d\u5386output_hypernetworks_mlps\u5217\u8868<\/span>
            \nfor<\/span> i in<\/span> range<\/span>(<\/span>self.<\/span>num_mask_tokens)<\/span>:<\/span>
            \n # mask_tokens_out[:, i, :]: B,1,C<\/span>
            \n # output_hypernetworks_mlps: B,1,c<\/span>
            \n # \u5bf9\u6bcf\u4e2a\u63a9\u6a21token，\u5e94\u7528\u5bf9\u5e94\u7684MLP，\u8f93\u5165\u662fmask_tokens_out\u4e2d\u5bf9\u5e94\u4f4d\u7f6e\u7684\u7279\u5f81，\u8f93\u51fa\u4e3aB, 1, c\u5f62\u72b6\u7684\u5f20\u91cf，\u5176\u4e2dc\u662f\u8d85\u7f51\u7edc\u7684\u8f93\u51fa\u901a\u9053\u6570<\/span>
            \n # \u5c06\u6bcf\u4e2aMLP\u7684\u8f93\u51fa\u6536\u96c6\u5230hyper_in_list\u5217\u8868\u4e2d<\/span>
            \n hyper_in_list.<\/span>append(<\/span>self.<\/span>output_hypernetworks_mlps[<\/span>i]<\/span>(<\/span>mask_tokens_out[<\/span>:<\/span>,<\/span> i,<\/span> :<\/span>]<\/span>)<\/span>)<\/span>
            \n# B,n,c<\/span>
            \n# \u5c06hyper_in_list\u5806\u53e0\u6210\u4e00\u4e2aB, n, c\u5f62\u72b6\u7684\u5f20\u91cfhyper_in，\u5176\u4e2dn\u662f\u63a9\u6a21token\u7684\u6570\u91cf<\/span>
            \nhyper_in =<\/span> torch.<\/span>stack(<\/span>hyper_in_list,<\/span> dim=<\/span>1<\/span>)<\/span>
            \n# \u83b7\u53d6upscaled_embedding\u7684\u5f62\u72b6b, c, h, w，\u5176\u4e2db\u662f\u6279\u6b21\u5927\u5c0f，c\u662f\u901a\u9053\u6570，h\u548cw\u662f\u9ad8\u5ea6\u548c\u5bbd\u5ea6<\/span>
            \nb,<\/span> c,<\/span> h,<\/span> w =<\/span> upscaled_embedding.<\/span>shape
            \n# B,n,c \u00d7 B,c,N–>B,n,h,w<\/span>
            \n# \u6267\u884c\u77e9\u9635\u4e58\u6cd5（@\u8fd0\u7b97\u7b26）\u5c06hyper_in（B, n, c）\u4e0eupscaled_embedding（\u5728\u901a\u9053\u7ef4\u5ea6\u4e0a\u5c55\u5e73\u4e3aB, c, h * w）\u76f8\u7ed3\u5408<\/span>
            \n# \u8ba1\u7b97\u6bcf\u4e2a\u63a9\u6a21token\u7684\u8d85\u7f51\u7edc\u6743\u91cd\u4e0e\u4e0a\u91c7\u6837\u7279\u5f81\u56fe\u7684\u70b9\u79ef，\u5f97\u5230B, n, h * w\u5f62\u72b6\u7684\u5f20\u91cf<\/span>
            \n# \u901a\u8fc7view\u64cd\u4f5c\u5c06\u7ed3\u679c\u8f6c\u6362\u56deB, n, h, w\u5f62\u72b6，\u751f\u6210\u4e86masks\u5f20\u91cf，\u8868\u793a\u6bcf\u4e2a\u63a9\u6a21token\u5bf9\u5e94\u7684\u9884\u6d4b\u63a9\u6a21<\/span>
            \nmasks =<\/span> (<\/span>hyper_in @ upscaled_embedding.<\/span>view(<\/span>b,<\/span> c,<\/span> h *<\/span> w)<\/span>)<\/span>.<\/span>view(<\/span>b,<\/span> –<\/span>1<\/span>,<\/span> h,<\/span> w)<\/span><\/p>\n

            \u300ciou_MLP\u300d<\/p>\n

            \u6b64\u5904\u7684MLP\u57fa\u7840\u6a21\u5757\u4e0d\u540c\u4e8eViT\u7684MLP(transformer_MLP)\u57fa\u7840\u6a21\u5757\u3002<\/p>\n

            # \u5728MaskDecoder\u7684__init__\u5b9a\u4e49<\/span>
            \n# \u4e00\u4e2a\u591a\u5c42\u611f\u77e5\u673a（MLP）\u6a21\u5757，\u5176\u76ee\u7684\u662f\u9884\u6d4b\u6bcf\u4e2a\u63a9\u6a21token\u5bf9\u5e94\u7684IoU（Intersection over Union，\u4ea4\u5e76\u6bd4）\u503c，\u4ee5\u8bc4\u4f30\u9884\u6d4b\u63a9\u6a21\u4e0e\u771f\u5b9e\u63a9\u6a21\u7684\u91cd\u5408\u7a0b\u5ea6<\/span>
            \nself.<\/span>iou_prediction_head =<\/span> MLP(<\/span>
            \n # transformer_dim: \u8f93\u5165\u5230MLP\u7684\u7279\u5f81\u7ef4\u5ea6，\u901a\u5e38\u4e0eTransformer\u7684\u8f93\u51fa\u7ef4\u5ea6\u76f8\u540c<\/span>
            \n # iou_head_hidden_dim: MLP\u4e2d\u95f4\u5c42\u7684\u7ef4\u5ea6，\u7528\u4e8e\u589e\u5f3a\u6a21\u578b\u7684\u8868\u8fbe\u80fd\u529b<\/span>
            \n # self.num_mask_tokens: \u8f93\u51fa\u7ef4\u5ea6，\u5373\u9884\u6d4b\u7684\u63a9\u6a21\u4ee4\u724c\u6570\u91cf，\u6bcf\u4e2a\u4ee4\u724c\u5bf9\u5e94\u4e00\u4e2aIoU\u9884\u6d4b\u503c<\/span>
            \n transformer_dim,<\/span> iou_head_hidden_dim,<\/span> self.<\/span>num_mask_tokens,<\/span> iou_head_depth
            \n)<\/span>
            \n# \u5728MaskDecoder\u7684predict_masks\u6dfb\u52a0\u4f4d\u7f6e\u7f16\u7801<\/span>
            \niou_pred =<\/span> self.<\/span>iou_prediction_head(<\/span>iou_token_out)<\/span><\/p>\n

            \u300cMaskDeco_MLP\u300d<\/p>\n

            Mask Decoder\u4e2dMLP\u7684\u7ed3\u6784\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n

            \"\u56fe\u7247\"<\/p>\n

            '''
            \n\u5b9a\u4e49\u4e86\u4e00\u4e2a\u591a\u5c42\u611f\u77e5\u673a，\u5b83\u5305\u542b\u4e00\u4e2a\u53ef\u914d\u7f6e\u7684\u9690\u85cf\u5c42\u6570\u76ee\u3001\u8f93\u5165\u548c\u8f93\u51fa\u7ef4\u5ea6，\u5e76\u53ef\u4ee5\u9009\u62e9\u662f\u5426\u5728\u8f93\u51fa\u5c42\u5e94\u7528Sigmoid\u6fc0\u6d3b\u51fd\u6570
            \n'''<\/span>
            \nclass<\/span> MLP<\/span>(<\/span>nn.<\/span>Module)<\/span>:<\/span>
            \n def<\/span> __init__<\/span>(<\/span>
            \n self,<\/span>
            \n input_dim:<\/span> int<\/span>,<\/span> # \u8f93\u5165\u7279\u5f81\u7684\u7ef4\u5ea6，\u5373\u8f93\u5165\u5f20\u91cf\u7684\u901a\u9053\u6570<\/span>
            \n hidden_dim:<\/span> int<\/span>,<\/span> # \u9690\u85cf\u5c42\u7684\u901a\u9053\u6570，\u4e2d\u95f4\u5c42\u7684\u5bbd\u5ea6<\/span>
            \n output_dim:<\/span> int<\/span>,<\/span> # \u8f93\u51fa\u7279\u5f81\u7684\u7ef4\u5ea6，\u5373\u8f93\u51fa\u5f20\u91cf\u7684\u901a\u9053\u6570<\/span>
            \n num_layers:<\/span> int<\/span>,<\/span> # \u591a\u5c42\u611f\u77e5\u673a\u7684\u5c42\u6570，\u5305\u62ec\u8f93\u5165\u5c42\u548c\u8f93\u51fa\u5c42<\/span>
            \n sigmoid_output:<\/span> bool<\/span> =<\/span> False<\/span>,<\/span> # \u4e00\u4e2a\u5e03\u5c14\u503c，\u8868\u793a\u662f\u5426\u5728\u8f93\u51fa\u5c42\u5e94\u7528Sigmoid\u6fc0\u6d3b\u51fd\u6570，\u9ed8\u8ba4\u4e3aFalse<\/span>
            \n )<\/span> –<\/span>><\/span> None<\/span>:<\/span>
            \n '''
            \n \u5185\u90e8\u7ec4\u4ef6
            \n '''<\/span>
            \n super<\/span>(<\/span>)<\/span>.<\/span>__init__(<\/span>)<\/span>
            \n # \u5b58\u50a8\u8f93\u5165\u7684\u5c42\u6570<\/span>
            \n self.<\/span>num_layers =<\/span> num_layers
            \n # \u4e00\u4e2a\u5217\u8868，\u5305\u542bnum_layers – 1\u4e2ahidden_dim，\u7528\u4e8e\u6784\u5efa\u4e2d\u95f4\u5c42\u7684\u7ebf\u6027\u53d8\u6362<\/span>
            \n h =<\/span> [<\/span>hidden_dim]<\/span> *<\/span> (<\/span>num_layers –<\/span> 1<\/span>)<\/span>
            \n # \u4e00\u4e2ann.ModuleList，\u5305\u542bnum_layers\u4e2a\u7ebf\u6027\u5c42（\u5168\u8fde\u63a5\u5c42），\u6bcf\u4e2a\u5c42\u7684\u8f93\u5165\u548c\u8f93\u51fa\u901a\u9053\u6570\u7531h\u548cinput_dim\u3001output_dim\u51b3\u5b9a<\/span>
            \n self.<\/span>layers =<\/span> nn.<\/span>ModuleList(<\/span>
            \n nn.<\/span>Linear(<\/span>n,<\/span> k)<\/span> for<\/span> n,<\/span> k in<\/span> zip<\/span>(<\/span>[<\/span>input_dim]<\/span> +<\/span> h,<\/span> h +<\/span> [<\/span>output_dim]<\/span>)<\/span>
            \n )<\/span>
            \n self.<\/span>sigmoid_output =<\/span> sigmoid_output<\/p>\n

            def<\/span> forward<\/span>(<\/span>self,<\/span> x)<\/span>:<\/span>
            \n # \u5bf9\u8f93\u5165\u5f20\u91cfx，\u904d\u5386layers\u5217\u8868\u4e2d\u7684\u6bcf\u4e2a\u7ebf\u6027\u5c42<\/span>
            \n for<\/span> i,<\/span> layer in<\/span> enumerate<\/span>(<\/span>self.<\/span>layers)<\/span>:<\/span>
            \n # \u5982\u679c\u5f53\u524d\u5c42\u4e0d\u662f\u6700\u540e\u4e00\u5c42，\u5e94\u7528ReLU\u6fc0\u6d3b\u51fd\u6570（F.relu）<\/span>
            \n x =<\/span> F.<\/span>relu(<\/span>layer(<\/span>x)<\/span>)<\/span> if<\/span> i <<\/span> self.<\/span>num_layers –<\/span> 1<\/span> else<\/span> layer(<\/span>x)<\/span>
            \n # \u5982\u679csigmoid_output\u4e3aTrue，\u6700\u540e\u5bf9\u8f93\u51fa\u5e94\u7528Sigmoid\u6fc0\u6d3b\u51fd\u6570<\/span>
            \n if<\/span> self.<\/span>sigmoid_output:<\/span>
            \n x =<\/span> F.<\/span>sigmoid(<\/span>x)<\/span>
            \n return<\/span> x<\/p>\n","protected":false},"excerpt":{"rendered":"

            \u6587\u7ae0\u6d4f\u89c8\u9605\u8bfb9.3k\u6b21\uff0c\u70b9\u8d5e71\u6b21\uff0c\u6536\u85cf299\u6b21\u3002\u8282\u524d\uff0c\u6211\u4eec\u661f\u7403\u7ec4\u7ec7\u4e86\u4e00\u573a\u7b97\u6cd5\u5c97\u6280\u672f&\u9762\u8bd5\u8ba8\u8bba\u4f1a\uff0c\u9080\u8bf7\u4e86\u4e00\u4e9b\u4e92\u8054\u7f51\u5927\u5382\u670b\u53cb\u3001\u53c2\u52a0\u793e\u62db\u548c\u6821\u62db\u9762\u8bd5\u7684\u540c\u5b66\u3002\u9488\u5bf9\u7b97\u6cd5\u5c97\u6280\u672f\u8d8b\u52bf\u3001\u5927\u6a21\u578b\u843d\u5730\u9879\u76ee\u7ecf\u9a8c\u5206\u4eab\u3001\u65b0\u624b\u5982\u4f55\u5165\u95e8\u7b97\u6cd5\u5c97\u3001\u8be5\u5982\u4f55\u51c6\u5907\u3001\u9762\u8bd5\u5e38\u8003\u70b9\u5206\u4eab\u7b49\u70ed\u95e8\u8bdd\u9898\u8fdb\u884c\u4e86\u6df1\u5165\u7684\u8ba8\u8bba\u3002SAM(Segment Anything Model)\uff0c\u987e\u540d\u601d\u4e49\uff0c\u5373\u4e3a\u5206\u5272\u4e00\u5207\uff01\u8be5\u6a21\u578b\u7531Facebook\u7684Meta AI\u5b9e\u9a8c\u5ba4\uff0c\u80fd\u591f\u6839\u636e\u6587\u672c\u6307\u4ee4\u6216\u56fe\u50cf\u8bc6\u522b\uff0c\u5b9e\u73b0\u5bf9\u4efb\u610f\u7269\u4f53\u7684\u8bc6\u522b\u4e0e\u5206\u5272\u3002\u5b83\u7684\u8bde\u751f\uff0c\u65e0\u7591\u662fCV\u9886\u57df\u7684\u4e00\u6b21\u91cd\u8981\u91cc\u7a0b\u7891\u3002_sam\u6a21\u578b<\/p>\n","protected":false},"author":2,"featured_media":33788,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[50,2831,132,86,427,2830],"topic":[],"class_list":["post-33808","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-server","tag-50","tag-2831","tag-132","tag-86","tag-427","tag-2830"],"yoast_head":"\n\u6e90\u7801\u89e3\u6790\uff1a\u4ece\u96f6\u89e3\u8bfbSAM(Segment Anything Model)\u5927\u6a21\u578b\uff01 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.wsisp.com\/helps\/33808.html\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u6e90\u7801\u89e3\u6790\uff1a\u4ece\u96f6\u89e3\u8bfbSAM(Segment Anything Model)\u5927\u6a21\u578b\uff01 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"og:description\" content=\"\u6587\u7ae0\u6d4f\u89c8\u9605\u8bfb9.3k\u6b21\uff0c\u70b9\u8d5e71\u6b21\uff0c\u6536\u85cf299\u6b21\u3002\u8282\u524d\uff0c\u6211\u4eec\u661f\u7403\u7ec4\u7ec7\u4e86\u4e00\u573a\u7b97\u6cd5\u5c97\u6280\u672f&\u9762\u8bd5\u8ba8\u8bba\u4f1a\uff0c\u9080\u8bf7\u4e86\u4e00\u4e9b\u4e92\u8054\u7f51\u5927\u5382\u670b\u53cb\u3001\u53c2\u52a0\u793e\u62db\u548c\u6821\u62db\u9762\u8bd5\u7684\u540c\u5b66\u3002\u9488\u5bf9\u7b97\u6cd5\u5c97\u6280\u672f\u8d8b\u52bf\u3001\u5927\u6a21\u578b\u843d\u5730\u9879\u76ee\u7ecf\u9a8c\u5206\u4eab\u3001\u65b0\u624b\u5982\u4f55\u5165\u95e8\u7b97\u6cd5\u5c97\u3001\u8be5\u5982\u4f55\u51c6\u5907\u3001\u9762\u8bd5\u5e38\u8003\u70b9\u5206\u4eab\u7b49\u70ed\u95e8\u8bdd\u9898\u8fdb\u884c\u4e86\u6df1\u5165\u7684\u8ba8\u8bba\u3002SAM(Segment Anything Model)\uff0c\u987e\u540d\u601d\u4e49\uff0c\u5373\u4e3a\u5206\u5272\u4e00\u5207\uff01\u8be5\u6a21\u578b\u7531Facebook\u7684Meta AI\u5b9e\u9a8c\u5ba4\uff0c\u80fd\u591f\u6839\u636e\u6587\u672c\u6307\u4ee4\u6216\u56fe\u50cf\u8bc6\u522b\uff0c\u5b9e\u73b0\u5bf9\u4efb\u610f\u7269\u4f53\u7684\u8bc6\u522b\u4e0e\u5206\u5272\u3002\u5b83\u7684\u8bde\u751f\uff0c\u65e0\u7591\u662fCV\u9886\u57df\u7684\u4e00\u6b21\u91cd\u8981\u91cc\u7a0b\u7891\u3002_sam\u6a21\u578b\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.wsisp.com\/helps\/33808.html\" \/>\n<meta property=\"og:site_name\" content=\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"article:published_time\" content=\"2025-04-28T07:49:45+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2025\/04\/20250428074938-680f3312d5fa7.png\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"29 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/33808.html\",\"url\":\"https:\/\/www.wsisp.com\/helps\/33808.html\",\"name\":\"\u6e90\u7801\u89e3\u6790\uff1a\u4ece\u96f6\u89e3\u8bfbSAM(Segment Anything Model)\u5927\u6a21\u578b\uff01 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"isPartOf\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\"},\"datePublished\":\"2025-04-28T07:49:45+00:00\",\"dateModified\":\"2025-04-28T07:49:45+00:00\",\"author\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/33808.html#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.wsisp.com\/helps\/33808.html\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/33808.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.wsisp.com\/helps\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u6e90\u7801\u89e3\u6790\uff1a\u4ece\u96f6\u89e3\u8bfbSAM(Segment Anything Model)\u5927\u6a21\u578b\uff01\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\",\"url\":\"https:\/\/www.wsisp.com\/helps\/\",\"name\":\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"description\":\"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"contentUrl\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"caption\":\"admin\"},\"sameAs\":[\"http:\/\/wp.wsisp.com\"],\"url\":\"https:\/\/www.wsisp.com\/helps\/author\/admin\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"\u6e90\u7801\u89e3\u6790\uff1a\u4ece\u96f6\u89e3\u8bfbSAM(Segment Anything Model)\u5927\u6a21\u578b\uff01 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.wsisp.com\/helps\/33808.html","og_locale":"zh_CN","og_type":"article","og_title":"\u6e90\u7801\u89e3\u6790\uff1a\u4ece\u96f6\u89e3\u8bfbSAM(Segment Anything Model)\u5927\u6a21\u578b\uff01 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","og_description":"\u6587\u7ae0\u6d4f\u89c8\u9605\u8bfb9.3k\u6b21\uff0c\u70b9\u8d5e71\u6b21\uff0c\u6536\u85cf299\u6b21\u3002\u8282\u524d\uff0c\u6211\u4eec\u661f\u7403\u7ec4\u7ec7\u4e86\u4e00\u573a\u7b97\u6cd5\u5c97\u6280\u672f&\u9762\u8bd5\u8ba8\u8bba\u4f1a\uff0c\u9080\u8bf7\u4e86\u4e00\u4e9b\u4e92\u8054\u7f51\u5927\u5382\u670b\u53cb\u3001\u53c2\u52a0\u793e\u62db\u548c\u6821\u62db\u9762\u8bd5\u7684\u540c\u5b66\u3002\u9488\u5bf9\u7b97\u6cd5\u5c97\u6280\u672f\u8d8b\u52bf\u3001\u5927\u6a21\u578b\u843d\u5730\u9879\u76ee\u7ecf\u9a8c\u5206\u4eab\u3001\u65b0\u624b\u5982\u4f55\u5165\u95e8\u7b97\u6cd5\u5c97\u3001\u8be5\u5982\u4f55\u51c6\u5907\u3001\u9762\u8bd5\u5e38\u8003\u70b9\u5206\u4eab\u7b49\u70ed\u95e8\u8bdd\u9898\u8fdb\u884c\u4e86\u6df1\u5165\u7684\u8ba8\u8bba\u3002SAM(Segment Anything Model)\uff0c\u987e\u540d\u601d\u4e49\uff0c\u5373\u4e3a\u5206\u5272\u4e00\u5207\uff01\u8be5\u6a21\u578b\u7531Facebook\u7684Meta AI\u5b9e\u9a8c\u5ba4\uff0c\u80fd\u591f\u6839\u636e\u6587\u672c\u6307\u4ee4\u6216\u56fe\u50cf\u8bc6\u522b\uff0c\u5b9e\u73b0\u5bf9\u4efb\u610f\u7269\u4f53\u7684\u8bc6\u522b\u4e0e\u5206\u5272\u3002\u5b83\u7684\u8bde\u751f\uff0c\u65e0\u7591\u662fCV\u9886\u57df\u7684\u4e00\u6b21\u91cd\u8981\u91cc\u7a0b\u7891\u3002_sam\u6a21\u578b","og_url":"https:\/\/www.wsisp.com\/helps\/33808.html","og_site_name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","article_published_time":"2025-04-28T07:49:45+00:00","og_image":[{"url":"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2025\/04\/20250428074938-680f3312d5fa7.png"}],"author":"admin","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"admin","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"29 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.wsisp.com\/helps\/33808.html","url":"https:\/\/www.wsisp.com\/helps\/33808.html","name":"\u6e90\u7801\u89e3\u6790\uff1a\u4ece\u96f6\u89e3\u8bfbSAM(Segment Anything Model)\u5927\u6a21\u578b\uff01 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","isPartOf":{"@id":"https:\/\/www.wsisp.com\/helps\/#website"},"datePublished":"2025-04-28T07:49:45+00:00","dateModified":"2025-04-28T07:49:45+00:00","author":{"@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41"},"breadcrumb":{"@id":"https:\/\/www.wsisp.com\/helps\/33808.html#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.wsisp.com\/helps\/33808.html"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.wsisp.com\/helps\/33808.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.wsisp.com\/helps"},{"@type":"ListItem","position":2,"name":"\u6e90\u7801\u89e3\u6790\uff1a\u4ece\u96f6\u89e3\u8bfbSAM(Segment Anything Model)\u5927\u6a21\u578b\uff01"}]},{"@type":"WebSite","@id":"https:\/\/www.wsisp.com\/helps\/#website","url":"https:\/\/www.wsisp.com\/helps\/","name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","description":"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41","name":"admin","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/","url":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","contentUrl":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","caption":"admin"},"sameAs":["http:\/\/wp.wsisp.com"],"url":"https:\/\/www.wsisp.com\/helps\/author\/admin"}]}},"_links":{"self":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/33808","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/comments?post=33808"}],"version-history":[{"count":0,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/33808\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media\/33788"}],"wp:attachment":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media?parent=33808"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/categories?post=33808"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/tags?post=33808"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/topic?post=33808"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}