{"id":97293,"date":"2026-08-29T08:37:08","date_gmt":"2026-08-29T00:37:08","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/97293.html"},"modified":"2026-08-29T08:37:08","modified_gmt":"2026-08-29T00:37:08","slug":"%e6%b7%b1%e5%85%a5%e7%90%86%e8%a7%a3-transformer%ef%bc%9atransformer-%e7%a9%b6%e7%ab%9f%e6%98%af%e4%bb%80%e4%b9%88%ef%bc%9f","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/97293.html","title":{"rendered":"\u6df1\u5165\u7406\u89e3 Transformer\uff1aTransformer \u7a76\u7adf\u662f\u4ec0\u4e48\uff1f"},"content":{"rendered":"<h3>\u76ee\u5f55<\/h3>\n<ul>\n<li>\u4e00\u3001\u5148\u4ece\u4e00\u5f20\u7ecf\u5178\u7ed3\u6784\u56fe\u5f00\u59cb<\/li>\n<li>\u4e8c\u3001Transformer \u4e0d\u662f\u4e00\u4e2a\u6a21\u5757&#xff0c;\u800c\u662f\u4e00\u7ec4\u6a21\u5757\u7684\u7ec4\u5408<\/li>\n<li>\u4e09\u3001\u4ec0\u4e48\u662f Encoder&#xff1f;<\/li>\n<li>\u56db\u3001Encoder Block&#xff1a;Transformer \u7684\u57fa\u672c\u79ef\u6728<\/li>\n<li>\u4e94\u3001\u7b2c\u4e00\u90e8\u5206&#xff1a;Multi-Head Self-Attention<\/li>\n<li>\u516d\u3001\u7b2c\u4e8c\u90e8\u5206&#xff1a;\u4e3a\u4ec0\u4e48 Attention \u540e\u9762\u8fd8\u9700\u8981 Add&#xff1f;<\/li>\n<li>\u4e03\u3001\u7b2c\u4e09\u90e8\u5206&#xff1a;Norm \u53c8\u662f\u4ec0\u4e48&#xff1f;<\/li>\n<li>\u516b\u3001\u7b2c\u56db\u90e8\u5206&#xff1a;Attention \u4e4b\u540e&#xff0c;\u4e3a\u4ec0\u4e48\u8fd8\u9700\u8981 FFN&#xff1f;<\/li>\n<li>\u4e5d\u3001\u73b0\u5728&#xff0c;\u6211\u4eec\u7ec8\u4e8e\u53ef\u4ee5\u770b\u61c2 Encoder Block \u4e86<\/li>\n<li>\u5341\u3001Encoder \u5230\u5e95\u8f93\u51fa\u4e86\u4ec0\u4e48&#xff1f;<\/li>\n<li>\u5341\u4e00\u3001\u63a5\u4e0b\u6765\u8f6e\u5230 Decoder<\/li>\n<li>\u5341\u4e8c\u3001Decoder Block \u4e3a\u4ec0\u4e48\u6709\u4e24\u4e2a Attention&#xff1f;<\/li>\n<li>\u5341\u4e09\u3001Encoder \u548c Decoder \u5230\u5e95\u6709\u4ec0\u4e48\u533a\u522b&#xff1f;<\/li>\n<li>\u5341\u56db\u3001Transformer \u5230\u5e95\u662f\u5982\u4f55\u5b8c\u6210\u4e00\u6b21\u673a\u5668\u7ffb\u8bd1\u7684&#xff1f;<\/li>\n<li>\u5341\u4e94\u3001\u7b49\u7b49&#xff0c;\u8fd8\u6709\u4e00\u4e2a\u91cd\u8981\u95ee\u9898&#xff1a;Transformer \u600e\u4e48\u77e5\u9053\u987a\u5e8f&#xff1f;<\/li>\n<li>\u5341\u516d\u3001Transformer \u7684\u5b8c\u6574\u6570\u636e\u6d41<\/li>\n<li>\u5341\u4e03\u3001\u4e00\u4e2a\u5bb9\u6613\u4ea7\u751f\u7684\u8bef\u89e3&#xff1a;\u73b0\u5728\u7684 GPT \u8fd8\u662f\u5b8c\u6574 Transformer \u5417&#xff1f;<\/li>\n<li>\u5341\u516b\u3001\u73b0\u5728&#xff0c;\u6211\u4eec\u5bf9 Transformer \u5e94\u8be5\u6709\u4ec0\u4e48\u6574\u4f53\u8ba4\u8bc6&#xff1f;<\/li>\n<li>\u5341\u4e5d\u3001\u5c0f\u7ed3<\/li>\n<li>\u4e0b\u4e00\u7bc7&#xff1a;Self-Attention \u5230\u5e95\u662f\u4ec0\u4e48&#xff1f;<\/li>\n<\/ul>\n<h2>\u300a\u6df1\u5165\u7406\u89e3 Transformer&#xff1a;Transformer \u7a76\u7adf\u662f\u4ec0\u4e48&#xff1f;\u300b<\/h2>\n<h3>\u524d\u8a00<\/h3>\n<p>\u5728\u4e0a\u4e00\u7bc7\u6587\u7ae0\u4e2d&#xff0c;\u6211\u4eec\u6682\u65f6\u6ca1\u6709\u6025\u7740\u8ba8\u8bba Q\u3001K\u3001V&#xff0c;\u4e5f\u6ca1\u6709\u76f4\u63a5\u63a8\u5bfc Attention \u7684\u8ba1\u7b97\u516c\u5f0f\u3002<\/p>\n<p>\u6211\u4eec\u5148\u56de\u7b54\u4e86\u4e00\u4e2a\u66f4\u57fa\u7840\u7684\u95ee\u9898&#xff1a;<\/p>\n<p>\u4e3a\u4ec0\u4e48\u9700\u8981 Transformer&#xff1f;<\/p>\n<p>\u5728 Transformer \u51fa\u73b0\u4e4b\u524d&#xff0c;\u5904\u7406\u6587\u672c\u7b49\u5e8f\u5217\u6570\u636e\u7684\u4e3b\u6d41\u65b9\u6848\u662f RNN\u3001LSTM \u548c GRU\u3002<\/p>\n<p>\u5b83\u4eec\u90fd\u6709\u4e00\u4e2a\u975e\u5e38\u660e\u663e\u7684\u5171\u540c\u7279\u70b9&#xff1a;<\/p>\n<p>x\u2081 \u2192 x\u2082 \u2192 x\u2083 \u2192 x\u2084 \u2192 \u00b7\u00b7\u00b7<\/p>\n<p>\u4e5f\u5c31\u662f\u8bf4&#xff0c;\u6a21\u578b\u901a\u5e38\u9700\u8981\u6309\u7167\u5e8f\u5217\u7684\u987a\u5e8f&#xff0c;\u4e00\u6b65\u4e00\u6b65\u5730\u5904\u7406\u6570\u636e\u3002<\/p>\n<p>\u8fd9\u79cd\u65b9\u5f0f\u867d\u7136\u7b26\u5408\u4eba\u7c7b\u9605\u8bfb\u6587\u672c\u7684\u76f4\u89c9&#xff0c;\u4f46\u4e5f\u5e26\u6765\u4e86\u4e24\u4e2a\u975e\u5e38\u5173\u952e\u7684\u95ee\u9898&#xff1a;<\/p>\n<li>\u8ba1\u7b97\u8fc7\u7a0b\u5b58\u5728\u987a\u5e8f\u4f9d\u8d56&#xff0c;\u96be\u4ee5\u5145\u5206\u5229\u7528\u5e76\u884c\u8ba1\u7b97\u80fd\u529b&#xff1b;<\/li>\n<li>\u8ddd\u79bb\u8f83\u8fdc\u7684\u4fe1\u606f\u9700\u8981\u7ecf\u8fc7\u591a\u6b21\u4f20\u9012&#xff0c;\u957f\u8ddd\u79bb\u4f9d\u8d56\u96be\u4ee5\u5904\u7406\u3002<\/li>\n<p>\u968f\u540e&#xff0c;Attention \u7684\u51fa\u73b0\u6539\u53d8\u4e86\u8fd9\u79cd\u601d\u8def\u3002<\/p>\n<p>\u6211\u4eec\u4e0d\u518d\u8981\u6c42\u4fe1\u606f\u5fc5\u987b&#xff1a;<\/p>\n<p>Token 1<br \/>\n   \u2193<br \/>\nToken 2<br \/>\n   \u2193<br \/>\nToken 3<br \/>\n   \u2193<br \/>\nToken 4<\/p>\n<p>\u4e00\u7ea7\u4e00\u7ea7\u5730\u4f20\u9012\u3002<\/p>\n<p>\u800c\u662f\u53ef\u4ee5\u8ba9\u5f53\u524d\u4fe1\u606f\u76f4\u63a5\u53bb\u5173\u6ce8\u5e8f\u5217\u4e2d\u7684\u5176\u4ed6\u4fe1\u606f&#xff1a;<\/p>\n<p>Token 1 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\nToken 2 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524<br \/>\nToken 3 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2192 \u5f53\u524d Token<br \/>\nToken 4 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524<br \/>\nToken 5 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n<p>\u4e8e\u662f&#xff0c;\u4e00\u4e2a\u65b0\u7684\u95ee\u9898\u51fa\u73b0\u4e86&#xff1a;<\/p>\n<p>\u65e2\u7136 Attention \u5df2\u7ecf\u8fd9\u4e48\u5f3a\u5927&#xff0c;\u90a3\u4e48 Transformer \u5230\u5e95\u662f\u4ec0\u4e48&#xff1f;<\/p>\n<p>\u5f88\u591a\u521a\u5f00\u59cb\u5b66\u4e60 Transformer \u7684\u4eba\u90fd\u4f1a\u6709\u4e00\u79cd\u9519\u89c9&#xff1a;<\/p>\n<p>Transformer \u4e0d\u5c31\u662f Attention \u5417&#xff1f;<\/p>\n<p>\u8fd9\u4e2a\u7406\u89e3\u4e0d\u80fd\u8bf4\u5b8c\u5168\u9519\u8bef&#xff0c;\u4f46\u663e\u7136\u8fd8\u4e0d\u591f\u5b8c\u6574\u3002<\/p>\n<p>Attention \u662f Transformer \u6700\u6838\u5fc3\u7684\u601d\u60f3\u4e4b\u4e00&#xff0c;\u4f46\u4e00\u4e2a\u5b8c\u6574\u7684 Transformer \u5e76\u4e0d\u662f\u53ea\u6709 Attention\u3002<\/p>\n<p>\u5728\u539f\u59cb\u7684 Transformer \u4e2d&#xff0c;\u6211\u4eec\u8fd8\u4f1a\u770b\u5230&#xff1a;<\/p>\n<p>Encoder<br \/>\nDecoder<br \/>\nMulti-Head Attention<br \/>\nFeed Forward Network<br \/>\nResidual Connection<br \/>\nLayer Normalization<br \/>\nPositional Encoding<\/p>\n<p>\u8fd9\u4e9b\u6a21\u5757\u5171\u540c\u7ec4\u6210\u4e86 Transformer\u3002<\/p>\n<p>\u56e0\u6b64&#xff0c;\u8fd9\u4e00\u7bc7\u6587\u7ae0\u6211\u4eec\u7684\u76ee\u6807\u5e76\u4e0d\u662f\u7acb\u523b\u94bb\u8fdb\u67d0\u4e00\u4e2a\u516c\u5f0f&#xff0c;\u800c\u662f\u5148\u5b8c\u6210\u4e00\u4ef6\u66f4\u52a0\u91cd\u8981\u7684\u4e8b\u60c5&#xff1a;<\/p>\n<p>\u5148\u628a Transformer \u7684\u6574\u4f53\u7ed3\u6784\u770b\u660e\u767d\u3002<\/p>\n<p>\u53ea\u6709\u5148\u77e5\u9053&#xff1a;<\/p>\n<p>Transformer \u7531\u54ea\u4e9b\u90e8\u5206\u7ec4\u6210&#xff1f;<br \/>\n\u6bcf\u4e00\u90e8\u5206\u8d1f\u8d23\u4ec0\u4e48&#xff1f;<br \/>\n\u6570\u636e\u5728\u6a21\u578b\u4e2d\u5982\u4f55\u6d41\u52a8&#xff1f;<\/p>\n<p>\u540e\u9762\u5b66\u4e60 Self-Attention\u3001Multi-Head Attention \u7b49\u5185\u5bb9\u65f6&#xff0c;\u624d\u4e0d\u4f1a\u53ea\u770b\u5230\u4e00\u5806\u96f6\u6563\u7684\u516c\u5f0f\u3002<\/p>\n<hr \/>\n<h2>\u4e00\u3001\u5148\u4ece\u4e00\u5f20\u7ecf\u5178\u7ed3\u6784\u56fe\u5f00\u59cb<\/h2>\n<p>Transformer \u6700\u65e9\u7531\u8bba\u6587\u300aAttention Is All You Need\u300b\u63d0\u51fa\u3002<\/p>\n<p>\u539f\u59cb Transformer \u662f\u4e00\u4e2a\u5178\u578b\u7684&#xff1a;<\/p>\n<p>Encoder-Decoder \u67b6\u6784\u3002<\/p>\n<p>\u6211\u4eec\u53ef\u4ee5\u5148\u628a\u5b83\u6700\u5916\u5c42\u7684\u7ed3\u6784\u7406\u89e3\u6210&#xff1a;<\/p>\n<p>\u8f93\u5165\u5e8f\u5217<br \/>\n    \u2193<br \/>\n Encoder<br \/>\n    \u2193<br \/>\n\u4e2d\u95f4\u8868\u793a<br \/>\n    \u2193<br \/>\n Decoder<br \/>\n    \u2193<br \/>\n\u8f93\u51fa\u5e8f\u5217<\/p>\n<p>\u4f8b\u5982\u673a\u5668\u7ffb\u8bd1&#xff1a;<\/p>\n<p>I love artificial intelligence.<br \/>\n            \u2193<br \/>\n         Encoder<br \/>\n            \u2193<br \/>\n      \u7406\u89e3\u8f93\u5165\u4fe1\u606f<br \/>\n            \u2193<br \/>\n         Decoder<br \/>\n            \u2193<br \/>\n\u6211\u559c\u6b22\u4eba\u5de5\u667a\u80fd\u3002<\/p>\n<p>\u8fd9\u91cc\u9700\u8981\u6ce8\u610f&#xff1a;<\/p>\n<p>Encoder \u548c Decoder \u5e76\u4e0d\u662f Transformer \u968f\u4fbf\u5212\u5206\u51fa\u6765\u7684\u4e24\u4e2a\u90e8\u5206&#xff0c;\u5b83\u4eec\u627f\u62c5\u7740\u4e0d\u540c\u7684\u804c\u8d23\u3002<\/p>\n<p>\u7b80\u5355\u6765\u8bf4&#xff1a;<\/p>\n<ul>\n<li>Encoder&#xff1a;\u8d1f\u8d23\u7406\u89e3\u8f93\u5165&#xff1b;<\/li>\n<li>Decoder&#xff1a;\u8d1f\u8d23\u6839\u636e\u5df2\u6709\u4fe1\u606f\u751f\u6210\u8f93\u51fa\u3002<\/li>\n<\/ul>\n<p>\u6211\u4eec\u53ef\u4ee5\u5148\u628a\u5b83\u4eec\u7406\u89e3\u6210&#xff1a;<\/p>\n<p>                 Transformer<\/p>\n<p>        \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n        \u2502                         \u2502<br \/>\n        \u2193                         \u2193<br \/>\n      Encoder                   Decoder<br \/>\n        \u2502                         \u2502<br \/>\n     \u7406\u89e3\u8f93\u5165                   \u751f\u6210\u8f93\u51fa<\/p>\n<p>\u4f46\u662f&#xff0c;\u771f\u5b9e\u7684 Transformer \u5e76\u4e0d\u662f\u53ea\u6709\u4e00\u4e2a Encoder \u548c\u4e00\u4e2a Decoder\u3002<\/p>\n<p>\u5b83\u4eec\u5185\u90e8\u5b9e\u9645\u4e0a\u90fd\u7531\u591a\u4e2a\u76f8\u540c\u7684 Block \u5806\u53e0\u800c\u6210\u3002<\/p>\n<hr \/>\n<p>\u4e0b\u9762\u662f Transformer \u6700\u5916\u5c42\u7684\u6574\u4f53\u67b6\u6784\u56fe&#xff1a;<\/p>\n<p>  #mermaid-svg-yfssAKUcSuKFkME0{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-yfssAKUcSuKFkME0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-yfssAKUcSuKFkME0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-yfssAKUcSuKFkME0 .error-icon{fill:#552222;}#mermaid-svg-yfssAKUcSuKFkME0 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-yfssAKUcSuKFkME0 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-yfssAKUcSuKFkME0 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-yfssAKUcSuKFkME0 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-yfssAKUcSuKFkME0 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-yfssAKUcSuKFkME0 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-yfssAKUcSuKFkME0 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-yfssAKUcSuKFkME0 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-yfssAKUcSuKFkME0 .marker.cross{stroke:#333333;}#mermaid-svg-yfssAKUcSuKFkME0 svg{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-yfssAKUcSuKFkME0 p{margin:0;}#mermaid-svg-yfssAKUcSuKFkME0 .label{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;color:#333;}#mermaid-svg-yfssAKUcSuKFkME0 .cluster-label text{fill:#333;}#mermaid-svg-yfssAKUcSuKFkME0 .cluster-label span{color:#333;}#mermaid-svg-yfssAKUcSuKFkME0 .cluster-label span p{background-color:transparent;}#mermaid-svg-yfssAKUcSuKFkME0 .label text,#mermaid-svg-yfssAKUcSuKFkME0 span{fill:#333;color:#333;}#mermaid-svg-yfssAKUcSuKFkME0 .node rect,#mermaid-svg-yfssAKUcSuKFkME0 .node circle,#mermaid-svg-yfssAKUcSuKFkME0 .node ellipse,#mermaid-svg-yfssAKUcSuKFkME0 .node polygon,#mermaid-svg-yfssAKUcSuKFkME0 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-yfssAKUcSuKFkME0 .rough-node .label text,#mermaid-svg-yfssAKUcSuKFkME0 .node .label text,#mermaid-svg-yfssAKUcSuKFkME0 .image-shape .label,#mermaid-svg-yfssAKUcSuKFkME0 .icon-shape .label{text-anchor:middle;}#mermaid-svg-yfssAKUcSuKFkME0 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-yfssAKUcSuKFkME0 .rough-node .label,#mermaid-svg-yfssAKUcSuKFkME0 .node .label,#mermaid-svg-yfssAKUcSuKFkME0 .image-shape .label,#mermaid-svg-yfssAKUcSuKFkME0 .icon-shape .label{text-align:center;}#mermaid-svg-yfssAKUcSuKFkME0 .node.clickable{cursor:pointer;}#mermaid-svg-yfssAKUcSuKFkME0 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-yfssAKUcSuKFkME0 .arrowheadPath{fill:#333333;}#mermaid-svg-yfssAKUcSuKFkME0 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-yfssAKUcSuKFkME0 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-yfssAKUcSuKFkME0 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-yfssAKUcSuKFkME0 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-yfssAKUcSuKFkME0 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-yfssAKUcSuKFkME0 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-yfssAKUcSuKFkME0 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-yfssAKUcSuKFkME0 .cluster text{fill:#333;}#mermaid-svg-yfssAKUcSuKFkME0 .cluster span{color:#333;}#mermaid-svg-yfssAKUcSuKFkME0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-yfssAKUcSuKFkME0 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-yfssAKUcSuKFkME0 rect.text{fill:none;stroke-width:0;}#mermaid-svg-yfssAKUcSuKFkME0 .icon-shape,#mermaid-svg-yfssAKUcSuKFkME0 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-yfssAKUcSuKFkME0 .icon-shape p,#mermaid-svg-yfssAKUcSuKFkME0 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-yfssAKUcSuKFkME0 .icon-shape .label rect,#mermaid-svg-yfssAKUcSuKFkME0 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-yfssAKUcSuKFkME0 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-yfssAKUcSuKFkME0 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-yfssAKUcSuKFkME0 :root{&#8211;mermaid-font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;}<\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>\u8f93\u5165\u5e8f\u5217<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>Encoder<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>\u4e2d\u95f4\u8868\u793a<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>Decoder<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>\u8f93\u51fa\u5e8f\u5217<\/p>\n<p><\/span><\/p>\n<h2>\u4e8c\u3001Transformer \u4e0d\u662f\u4e00\u4e2a\u6a21\u5757&#xff0c;\u800c\u662f\u4e00\u7ec4\u6a21\u5757\u7684\u7ec4\u5408<\/h2>\n<p>\u5047\u8bbe\u6211\u4eec\u628a Transformer \u770b\u6210\u4e00\u680b\u5efa\u7b51\u3002<\/p>\n<p>\u90a3\u4e48&#xff1a;<\/p>\n<p>Encoder<\/p>\n<p>\u548c&#xff1a;<\/p>\n<p>Decoder<\/p>\n<p>\u66f4\u50cf\u662f\u4e24\u680b\u4e3b\u8981\u5efa\u7b51\u3002<\/p>\n<p>\u800c\u771f\u6b63\u7ec4\u6210\u8fd9\u4e24\u680b\u5efa\u7b51\u7684&#xff0c;\u662f\u4e00\u4e2a\u4e2a\u91cd\u590d\u51fa\u73b0\u7684\u6a21\u5757\u3002<\/p>\n<p>\u539f\u59cb Transformer \u7684\u7ed3\u6784\u53ef\u4ee5\u7b80\u5355\u8868\u793a\u4e3a&#xff1a;<\/p>\n<p>Input<br \/>\n  \u2193<br \/>\nEmbedding<br \/>\n  \u2193<br \/>\nPositional Encoding<br \/>\n  \u2193<br \/>\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n\u2502  Encoder Block  \u2502<br \/>\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<br \/>\n  \u2193<br \/>\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n\u2502  Encoder Block  \u2502<br \/>\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<br \/>\n  \u2193<br \/>\n        \u00b7\u00b7\u00b7<br \/>\n  \u2193<br \/>\nEncoder Output<\/p>\n<p>Decoder \u4e5f\u662f\u7c7b\u4f3c\u7684\u7ed3\u6784&#xff1a;<\/p>\n<p>Output<br \/>\n  \u2193<br \/>\nEmbedding<br \/>\n  \u2193<br \/>\nPositional Encoding<br \/>\n  \u2193<br \/>\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n\u2502  Decoder Block  \u2502<br \/>\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<br \/>\n  \u2193<br \/>\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n\u2502  Decoder Block  \u2502<br \/>\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<br \/>\n  \u2193<br \/>\n        \u00b7\u00b7\u00b7<br \/>\n  \u2193<br \/>\nLinear<br \/>\n  \u2193<br \/>\nSoftmax<br \/>\n  \u2193<br \/>\n\u9884\u6d4b\u4e0b\u4e00\u4e2a Token<\/p>\n<p>\u6240\u4ee5\u7406\u89e3 Transformer \u65f6&#xff0c;\u5b9e\u9645\u4e0a\u53ef\u4ee5\u5206\u6210\u4e09\u4e2a\u5c42\u6b21&#xff1a;<\/p>\n<p>\u7b2c\u4e00\u5c42&#xff1a;\u6574\u4f53\u67b6\u6784<br \/>\n        \u2193<br \/>\nEncoder &#043; Decoder<\/p>\n<p>\u7b2c\u4e8c\u5c42&#xff1a;Block<br \/>\n        \u2193<br \/>\n\u591a\u4e2a\u76f8\u540c\u6a21\u5757\u91cd\u590d\u5806\u53e0<\/p>\n<p>\u7b2c\u4e09\u5c42&#xff1a;\u5177\u4f53\u7ec4\u4ef6<br \/>\n        \u2193<br \/>\nAttention<br \/>\nFFN<br \/>\nResidual<br \/>\nLayerNorm<br \/>\n&#8230;<\/p>\n<p>\u6211\u4eec\u8fd9\u4e00\u7bc7\u4e3b\u8981\u89e3\u51b3\u524d\u4e24\u4e2a\u5c42\u6b21\u7684\u95ee\u9898\u3002<\/p>\n<p>\u81f3\u4e8e\u6700\u5e95\u5c42\u7684 Attention \u662f\u600e\u4e48\u8ba1\u7b97\u7684&#xff0c;\u4f1a\u5728\u540e\u9762\u7684\u6587\u7ae0\u4e2d\u9010\u6b65\u5c55\u5f00\u3002<\/p>\n<hr \/>\n<p>\u4e0b\u9762\u662f Transformer \u4e09\u4e2a\u7406\u89e3\u5c42\u6b21\u7684 Mermaid \u56fe&#xff1a;<\/p>\n<p>  #mermaid-svg-2DAOTt9PxWdA6wru{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-2DAOTt9PxWdA6wru .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-2DAOTt9PxWdA6wru .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-2DAOTt9PxWdA6wru .error-icon{fill:#552222;}#mermaid-svg-2DAOTt9PxWdA6wru .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-2DAOTt9PxWdA6wru .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-2DAOTt9PxWdA6wru .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-2DAOTt9PxWdA6wru .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-2DAOTt9PxWdA6wru .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-2DAOTt9PxWdA6wru .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-2DAOTt9PxWdA6wru .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-2DAOTt9PxWdA6wru .marker{fill:#333333;stroke:#333333;}#mermaid-svg-2DAOTt9PxWdA6wru .marker.cross{stroke:#333333;}#mermaid-svg-2DAOTt9PxWdA6wru svg{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-2DAOTt9PxWdA6wru p{margin:0;}#mermaid-svg-2DAOTt9PxWdA6wru .label{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;color:#333;}#mermaid-svg-2DAOTt9PxWdA6wru .cluster-label text{fill:#333;}#mermaid-svg-2DAOTt9PxWdA6wru .cluster-label span{color:#333;}#mermaid-svg-2DAOTt9PxWdA6wru .cluster-label span p{background-color:transparent;}#mermaid-svg-2DAOTt9PxWdA6wru .label text,#mermaid-svg-2DAOTt9PxWdA6wru span{fill:#333;color:#333;}#mermaid-svg-2DAOTt9PxWdA6wru .node rect,#mermaid-svg-2DAOTt9PxWdA6wru .node circle,#mermaid-svg-2DAOTt9PxWdA6wru .node ellipse,#mermaid-svg-2DAOTt9PxWdA6wru .node polygon,#mermaid-svg-2DAOTt9PxWdA6wru .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-2DAOTt9PxWdA6wru .rough-node .label text,#mermaid-svg-2DAOTt9PxWdA6wru .node .label text,#mermaid-svg-2DAOTt9PxWdA6wru .image-shape .label,#mermaid-svg-2DAOTt9PxWdA6wru .icon-shape .label{text-anchor:middle;}#mermaid-svg-2DAOTt9PxWdA6wru .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-2DAOTt9PxWdA6wru .rough-node .label,#mermaid-svg-2DAOTt9PxWdA6wru .node .label,#mermaid-svg-2DAOTt9PxWdA6wru .image-shape .label,#mermaid-svg-2DAOTt9PxWdA6wru .icon-shape .label{text-align:center;}#mermaid-svg-2DAOTt9PxWdA6wru .node.clickable{cursor:pointer;}#mermaid-svg-2DAOTt9PxWdA6wru .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-2DAOTt9PxWdA6wru .arrowheadPath{fill:#333333;}#mermaid-svg-2DAOTt9PxWdA6wru .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-2DAOTt9PxWdA6wru .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-2DAOTt9PxWdA6wru .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2DAOTt9PxWdA6wru .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-2DAOTt9PxWdA6wru .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2DAOTt9PxWdA6wru .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-2DAOTt9PxWdA6wru .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-2DAOTt9PxWdA6wru .cluster text{fill:#333;}#mermaid-svg-2DAOTt9PxWdA6wru .cluster span{color:#333;}#mermaid-svg-2DAOTt9PxWdA6wru div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-2DAOTt9PxWdA6wru .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-2DAOTt9PxWdA6wru rect.text{fill:none;stroke-width:0;}#mermaid-svg-2DAOTt9PxWdA6wru .icon-shape,#mermaid-svg-2DAOTt9PxWdA6wru .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2DAOTt9PxWdA6wru .icon-shape p,#mermaid-svg-2DAOTt9PxWdA6wru .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-2DAOTt9PxWdA6wru .icon-shape .label rect,#mermaid-svg-2DAOTt9PxWdA6wru .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2DAOTt9PxWdA6wru .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-2DAOTt9PxWdA6wru .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-2DAOTt9PxWdA6wru :root{&#8211;mermaid-font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;}<\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>\u7b2c\u4e09\u5c42&#xff1a;\u5177\u4f53\u7ec4\u4ef6<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>Attention<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>FFN<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>Residual<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>LayerNorm<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>\u7b2c\u4e8c\u5c42&#xff1a;Block<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>\u591a\u4e2a\u76f8\u540c\u6a21\u5757\u91cd\u590d\u5806\u53e0<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>\u7b2c\u4e00\u5c42&#xff1a;\u6574\u4f53\u67b6\u6784<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>Encoder &#043; Decoder<\/p>\n<p><\/span><\/p>\n<h2>\u4e09\u3001\u4ec0\u4e48\u662f Encoder&#xff1f;<\/h2>\n<p>\u5148\u6765\u770b Transformer \u5de6\u8fb9\u7684 Encoder\u3002<\/p>\n<p>\u5047\u8bbe\u8f93\u5165\u4e00\u53e5\u82f1\u6587&#xff1a;<\/p>\n<p>I love artificial intelligence.<\/p>\n<p>\u8ba1\u7b97\u673a\u9996\u5148\u5e76\u4e0d\u80fd\u76f4\u63a5\u7406\u89e3&#xff1a;<\/p>\n<p>I<br \/>\nlove<br \/>\nartificial<br \/>\nintelligence<\/p>\n<p>\u8fd9\u4e9b\u5355\u8bcd\u3002<\/p>\n<p>\u56e0\u4e3a\u5bf9\u4e8e\u795e\u7ecf\u7f51\u7edc\u6765\u8bf4&#xff0c;\u771f\u6b63\u80fd\u591f\u53c2\u4e0e\u8ba1\u7b97\u7684\u662f\u6570\u5b57\u3002<\/p>\n<p>\u6240\u4ee5&#xff0c;\u8f93\u5165\u9996\u5148\u9700\u8981\u7ecf\u8fc7\u4e00\u4e2a\u8fc7\u7a0b&#xff1a;<\/p>\n<p>\u6587\u672c<br \/>\n \u2193<br \/>\nTokenization<br \/>\n \u2193<br \/>\nToken<br \/>\n \u2193<br \/>\nToken ID<br \/>\n \u2193<br \/>\nEmbedding<br \/>\n \u2193<br \/>\n\u5411\u91cf<\/p>\n<p>\u4f8b\u5982&#xff1a;<\/p>\n<p>&#034;I&#034;              \u2192 [0.12, -0.35, 0.71, &#8230;]<br \/>\n&#034;love&#034;           \u2192 [0.43,  0.18, -0.22, &#8230;]<br \/>\n&#034;artificial&#034;     \u2192 [-0.61, 0.29, 0.44, &#8230;]<br \/>\n&#034;intelligence&#034;   \u2192 [0.07, -0.82, 0.15, &#8230;]<\/p>\n<p>\u8fd9\u91cc\u5f97\u5230\u7684\u5411\u91cf&#xff0c;\u5c31\u662f\u540e\u7eed Transformer \u771f\u6b63\u5904\u7406\u7684\u6570\u636e\u3002<\/p>\n<p>Encoder \u7684\u4efb\u52a1&#xff0c;\u5c31\u662f\u4e0d\u65ad\u5904\u7406\u8fd9\u4e9b Token \u5bf9\u5e94\u7684\u8868\u793a&#xff0c;\u5e76\u8ba9\u6a21\u578b\u9010\u6e10\u7406\u89e3&#xff1a;<\/p>\n<p>\u8fd9\u4e9b Token \u5206\u522b\u662f\u4ec0\u4e48&#xff0c;\u4ee5\u53ca\u5b83\u4eec\u4e4b\u95f4\u5b58\u5728\u4ec0\u4e48\u5173\u7cfb\u3002<\/p>\n<p>\u4f8b\u5982&#xff1a;<\/p>\n<p>I love artificial intelligence.<\/p>\n<p>\u7ecf\u8fc7 Encoder \u4e4b\u540e&#xff0c;\u6211\u4eec\u5e0c\u671b\u6a21\u578b\u4e0d\u518d\u53ea\u662f\u5b64\u7acb\u5730\u770b&#xff1a;<\/p>\n<p>I<br \/>\nlove<br \/>\nartificial<br \/>\nintelligence<\/p>\n<p>\u800c\u662f\u80fd\u591f\u5efa\u7acb\u8fd9\u6837\u7684\u8054\u7cfb&#xff1a;<\/p>\n<p>I \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2192 love<br \/>\n              \u2502<br \/>\n              \u2193<br \/>\n      artificial intelligence<\/p>\n<p>\u4e5f\u5c31\u662f\u8bf4&#xff1a;<\/p>\n<p>Encoder \u7684\u6838\u5fc3\u4efb\u52a1&#xff0c;\u662f\u8ba9\u6bcf\u4e00\u4e2a Token \u90fd\u83b7\u5f97\u201c\u7ed3\u5408\u4e0a\u4e0b\u6587\u4e4b\u540e\u201d\u7684\u8868\u793a\u3002<\/p>\n<p>\u8fd9\u53e5\u8bdd\u975e\u5e38\u91cd\u8981\u3002<\/p>\n<p>\u5728\u8f93\u5165 Transformer \u4e4b\u524d&#xff1a;<\/p>\n<p>love<\/p>\n<p>\u53ef\u80fd\u53ea\u662f\u201clove\u201d\u8fd9\u4e2a\u8bcd\u672c\u8eab\u7684\u5411\u91cf\u3002<\/p>\n<p>\u4f46\u662f\u7ecf\u8fc7 Encoder \u7684\u591a\u5c42\u5904\u7406\u4e4b\u540e&#xff1a;<\/p>\n<p>love<\/p>\n<p>\u6240\u5bf9\u5e94\u7684\u5411\u91cf&#xff0c;\u5df2\u7ecf\u4e0d\u518d\u53ea\u5305\u542b\u201clove\u201d\u81ea\u8eab\u7684\u4fe1\u606f\u3002<\/p>\n<p>\u5b83\u8fd8\u4f1a\u7ed3\u5408&#xff1a;<\/p>\n<p>\u8c01\u5728 love&#xff1f;<br \/>\nlove \u4ec0\u4e48&#xff1f;<br \/>\n\u5468\u56f4\u8fd8\u6709\u54ea\u4e9b\u8bcd&#xff1f;<br \/>\n\u8fd9\u4e9b\u8bcd\u4e4b\u95f4\u662f\u4ec0\u4e48\u5173\u7cfb&#xff1f;<\/p>\n<p>\u4e8e\u662f&#xff0c;Token \u7684\u8868\u793a\u5c31\u4ece&#xff1a;<\/p>\n<p>\u5b64\u7acb\u7684\u8bcd\u8868\u793a<\/p>\n<p>\u9010\u6e10\u53d8\u6210&#xff1a;<\/p>\n<p>\u5305\u542b\u4e0a\u4e0b\u6587\u4fe1\u606f\u7684\u8868\u793a\u3002<\/p>\n<p>\u8fd9\u5c31\u662f Encoder \u7684\u6838\u5fc3\u4ef7\u503c\u3002<\/p>\n<hr \/>\n<h2>\u56db\u3001Encoder Block&#xff1a;Transformer \u7684\u57fa\u672c\u79ef\u6728<\/h2>\n<p>\u4e00\u4e2a\u5b8c\u6574\u7684 Encoder \u5e76\u4e0d\u662f\u4e00\u6b21\u8ba1\u7b97\u5c31\u7ed3\u675f\u3002<\/p>\n<p>\u5b83\u7531\u591a\u4e2a Encoder Block \u53e0\u52a0\u800c\u6210\u3002<\/p>\n<p>\u539f\u59cb Transformer \u8bba\u6587\u4e2d\u4f7f\u7528\u4e86&#xff1a;<\/p>\n<p>N \u00d7 Encoder Block<\/p>\n<p>\u4e5f\u5c31\u662f\u8bf4&#xff1a;<\/p>\n<p>Input<br \/>\n  \u2193<br \/>\nEncoder Block 1<br \/>\n  \u2193<br \/>\nEncoder Block 2<br \/>\n  \u2193<br \/>\nEncoder Block 3<br \/>\n  \u2193<br \/>\n      \u00b7\u00b7\u00b7<br \/>\n  \u2193<br \/>\nEncoder Block N<\/p>\n<p>\u4e3a\u4ec0\u4e48\u9700\u8981\u591a\u5c42&#xff1f;<\/p>\n<p>\u6211\u4eec\u53ef\u4ee5\u628a\u5b83\u7406\u89e3\u6210&#xff1a;<\/p>\n<p>\u7b2c\u4e00\u5c42\u5efa\u7acb\u6bd4\u8f83\u57fa\u7840\u7684\u5173\u7cfb&#xff0c;\u540e\u9762\u7684\u5c42\u7ee7\u7eed\u5728\u524d\u4e00\u5c42\u7ed3\u679c\u7684\u57fa\u7840\u4e0a\u8fdb\u884c\u66f4\u52a0\u590d\u6742\u7684\u7279\u5f81\u63d0\u53d6\u3002<\/p>\n<p>\u4f8b\u5982&#xff1a;<\/p>\n<p>\u7b2c\u4e00\u5c42&#xff1a;<br \/>\n\u53ef\u80fd\u5173\u6ce8\u76f8\u90bb Token \u4e4b\u95f4\u7684\u5173\u7cfb<\/p>\n<p>\u7b2c\u4e8c\u5c42&#xff1a;<br \/>\n\u5f00\u59cb\u5efa\u7acb\u66f4\u590d\u6742\u7684\u8bed\u4e49\u5173\u7cfb<\/p>\n<p>\u7b2c\u4e09\u5c42&#xff1a;<br \/>\n\u8fdb\u4e00\u6b65\u7ec4\u5408\u8fd9\u4e9b\u4fe1\u606f<\/p>\n<p>\u2026\u2026<\/p>\n<p>\u6700\u540e&#xff1a;<br \/>\n\u5f62\u6210\u66f4\u52a0\u4e30\u5bcc\u7684\u4e0a\u4e0b\u6587\u8868\u793a<\/p>\n<p>\u5f53\u7136&#xff0c;\u771f\u5b9e\u6a21\u578b\u5185\u90e8\u7a76\u7adf\u5b66\u5230\u4e86\u4ec0\u4e48\u5173\u7cfb&#xff0c;\u5e76\u4e0d\u662f\u4eba\u4e3a\u63d0\u524d\u89c4\u5b9a\u597d\u7684\u3002<\/p>\n<p>\u8fd9\u91cc\u7684\u201c\u7b2c\u4e00\u5c42\u5b66\u8bed\u6cd5\u3001\u7b2c\u4e8c\u5c42\u5b66\u8bed\u4e49\u201d\u53ea\u80fd\u5e2e\u52a9\u6211\u4eec\u5efa\u7acb\u76f4\u89c9\u3002<\/p>\n<p>\u771f\u6b63\u7684\u60c5\u51b5\u662f&#xff1a;<\/p>\n<p>\u6a21\u578b\u901a\u8fc7\u8bad\u7ec3\u81ea\u5df1\u5b66\u4e60\u5e94\u8be5\u5982\u4f55\u5229\u7528\u6bcf\u4e00\u5c42\u7684\u53c2\u6570\u3002<\/p>\n<p>\u90a3\u4e48&#xff0c;\u4e00\u4e2a Encoder Block \u5185\u90e8\u5230\u5e95\u6709\u4ec0\u4e48&#xff1f;<\/p>\n<p>\u6211\u4eec\u5148\u770b\u4e00\u4e2a\u7b80\u5316\u7ed3\u6784&#xff1a;<\/p>\n<p>Input<br \/>\n  \u2193<br \/>\nMulti-Head Self-Attention<br \/>\n  \u2193<br \/>\nAdd &amp; Norm<br \/>\n  \u2193<br \/>\nFeed Forward Network<br \/>\n  \u2193<br \/>\nAdd &amp; Norm<br \/>\n  \u2193<br \/>\nOutput<\/p>\n<p>\u8fd9\u91cc\u51fa\u73b0\u4e86\u51e0\u4e2a\u91cd\u8981\u6a21\u5757&#xff1a;<\/p>\n<li>Multi-Head Self-Attention&#xff1b;<\/li>\n<li>Residual Connection&#xff1b;<\/li>\n<li>Layer Normalization&#xff1b;<\/li>\n<li>Feed Forward Network\u3002<\/li>\n<p>\u63a5\u4e0b\u6765&#xff0c;\u6211\u4eec\u5148\u4ece\u6574\u4f53\u804c\u8d23\u7406\u89e3\u5b83\u4eec\u3002<\/p>\n<hr \/>\n<p>\u4e0b\u9762\u662f Encoder Block \u5185\u90e8\u7ed3\u6784\u7684 Mermaid \u56fe&#xff1a;<\/p>\n<p>  #mermaid-svg-pXXaup0xEa4pMhAp{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-pXXaup0xEa4pMhAp .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-pXXaup0xEa4pMhAp .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-pXXaup0xEa4pMhAp .error-icon{fill:#552222;}#mermaid-svg-pXXaup0xEa4pMhAp .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-pXXaup0xEa4pMhAp .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-pXXaup0xEa4pMhAp .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-pXXaup0xEa4pMhAp .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-pXXaup0xEa4pMhAp .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-pXXaup0xEa4pMhAp .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-pXXaup0xEa4pMhAp .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-pXXaup0xEa4pMhAp .marker{fill:#333333;stroke:#333333;}#mermaid-svg-pXXaup0xEa4pMhAp .marker.cross{stroke:#333333;}#mermaid-svg-pXXaup0xEa4pMhAp svg{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-pXXaup0xEa4pMhAp p{margin:0;}#mermaid-svg-pXXaup0xEa4pMhAp .label{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;color:#333;}#mermaid-svg-pXXaup0xEa4pMhAp .cluster-label text{fill:#333;}#mermaid-svg-pXXaup0xEa4pMhAp .cluster-label span{color:#333;}#mermaid-svg-pXXaup0xEa4pMhAp .cluster-label span p{background-color:transparent;}#mermaid-svg-pXXaup0xEa4pMhAp .label text,#mermaid-svg-pXXaup0xEa4pMhAp span{fill:#333;color:#333;}#mermaid-svg-pXXaup0xEa4pMhAp .node rect,#mermaid-svg-pXXaup0xEa4pMhAp .node circle,#mermaid-svg-pXXaup0xEa4pMhAp .node ellipse,#mermaid-svg-pXXaup0xEa4pMhAp .node polygon,#mermaid-svg-pXXaup0xEa4pMhAp .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-pXXaup0xEa4pMhAp .rough-node .label text,#mermaid-svg-pXXaup0xEa4pMhAp .node .label text,#mermaid-svg-pXXaup0xEa4pMhAp .image-shape .label,#mermaid-svg-pXXaup0xEa4pMhAp .icon-shape .label{text-anchor:middle;}#mermaid-svg-pXXaup0xEa4pMhAp .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-pXXaup0xEa4pMhAp .rough-node .label,#mermaid-svg-pXXaup0xEa4pMhAp .node .label,#mermaid-svg-pXXaup0xEa4pMhAp .image-shape .label,#mermaid-svg-pXXaup0xEa4pMhAp .icon-shape .label{text-align:center;}#mermaid-svg-pXXaup0xEa4pMhAp .node.clickable{cursor:pointer;}#mermaid-svg-pXXaup0xEa4pMhAp .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-pXXaup0xEa4pMhAp .arrowheadPath{fill:#333333;}#mermaid-svg-pXXaup0xEa4pMhAp .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-pXXaup0xEa4pMhAp .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-pXXaup0xEa4pMhAp .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-pXXaup0xEa4pMhAp .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-pXXaup0xEa4pMhAp .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-pXXaup0xEa4pMhAp .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-pXXaup0xEa4pMhAp .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-pXXaup0xEa4pMhAp .cluster text{fill:#333;}#mermaid-svg-pXXaup0xEa4pMhAp .cluster span{color:#333;}#mermaid-svg-pXXaup0xEa4pMhAp div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-pXXaup0xEa4pMhAp .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-pXXaup0xEa4pMhAp rect.text{fill:none;stroke-width:0;}#mermaid-svg-pXXaup0xEa4pMhAp .icon-shape,#mermaid-svg-pXXaup0xEa4pMhAp .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-pXXaup0xEa4pMhAp .icon-shape p,#mermaid-svg-pXXaup0xEa4pMhAp .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-pXXaup0xEa4pMhAp .icon-shape .label rect,#mermaid-svg-pXXaup0xEa4pMhAp .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-pXXaup0xEa4pMhAp .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-pXXaup0xEa4pMhAp .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-pXXaup0xEa4pMhAp :root{&#8211;mermaid-font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;}<\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>Input<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>Multi-Head Self-Attention<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>Add &amp; Norm<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>Feed Forward Network<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>Add &amp; Norm<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>Output<\/p>\n<p><\/span><\/p>\n<h2>\u4e94\u3001\u7b2c\u4e00\u90e8\u5206&#xff1a;Multi-Head Self-Attention<\/h2>\n<p>\u8fd9\u662f Transformer \u4e2d\u6700\u6838\u5fc3\u7684\u6a21\u5757\u4e4b\u4e00\u3002<\/p>\n<p>\u867d\u7136\u5177\u4f53\u8ba1\u7b97\u8fc7\u7a0b\u4f1a\u5728\u540e\u9762\u7684\u6587\u7ae0\u4e2d\u8be6\u7ec6\u5c55\u5f00&#xff0c;\u4f46\u73b0\u5728\u6211\u4eec\u5148\u7406\u89e3&#xff1a;<\/p>\n<p>\u5b83\u5728\u6574\u4e2a Transformer \u4e2d\u8d1f\u8d23\u4ec0\u4e48&#xff1f;<\/p>\n<p>\u7b54\u6848\u662f&#xff1a;<\/p>\n<p>\u8ba9 Token \u4e4b\u95f4\u8fdb\u884c\u4fe1\u606f\u4ea4\u4e92\u3002<\/p>\n<p>\u5047\u8bbe\u6709\u4e00\u53e5\u8bdd&#xff1a;<\/p>\n<p>\u6211 \u559c\u6b22 \u5b66\u4e60 \u4eba\u5de5\u667a\u80fd<\/p>\n<p>\u5982\u679c\u6bcf\u4e2a Token \u90fd\u53ea\u662f\u72ec\u7acb\u5b58\u5728&#xff1a;<\/p>\n<p>\u6211<br \/>\n\u559c\u6b22<br \/>\n\u5b66\u4e60<br \/>\n\u4eba\u5de5\u667a\u80fd<\/p>\n<p>\u90a3\u4e48\u6a21\u578b\u5e76\u4e0d\u77e5\u9053&#xff1a;<\/p>\n<p>\u8c01\u559c\u6b22&#xff1f;<br \/>\n\u559c\u6b22\u4ec0\u4e48&#xff1f;<br \/>\n\u5b66\u4e60\u4ec0\u4e48&#xff1f;<\/p>\n<p>\u4e8e\u662f Self-Attention \u4f1a\u8ba9\u6bcf\u4e2a Token \u90fd\u6709\u673a\u4f1a\u53bb\u5173\u6ce8\u5176\u4ed6 Token\u3002<\/p>\n<p>\u4f8b\u5982&#xff1a;<\/p>\n<p>        \u6211<br \/>\n        \u2191<br \/>\n        \u2502<br \/>\n\u559c\u6b22 \u2190 \u5b66\u4e60 \u2192 \u4eba\u5de5\u667a\u80fd<\/p>\n<p>\u66f4\u51c6\u786e\u5730\u8bf4&#xff1a;<\/p>\n<p>\u5bf9\u4e8e\u5f53\u524d Token&#xff0c;\u6a21\u578b\u4f1a\u8ba1\u7b97\u5e8f\u5217\u4e2d\u5176\u4ed6 Token \u5bf9\u5b83\u7684\u91cd\u8981\u7a0b\u5ea6\u3002<\/p>\n<p>\u56e0\u6b64&#xff1a;<\/p>\n<p>Self-Attention<\/p>\n<p>\u89e3\u51b3\u7684\u662f&#xff1a;<\/p>\n<p>Token \u4e0e Token \u4e4b\u95f4\u5982\u4f55\u4ea4\u6362\u4fe1\u606f\u3002<\/p>\n<p>\u8fd9\u91cc\u7684&#xff1a;<\/p>\n<p>Self<\/p>\n<p>\u4e5f\u975e\u5e38\u5173\u952e\u3002<\/p>\n<p>\u5b83\u8868\u793a&#xff1a;<\/p>\n<p>Attention \u7684\u4fe1\u606f\u6765\u6e90\u548c\u67e5\u8be2\u5bf9\u8c61\u6765\u81ea\u540c\u4e00\u4e2a\u5e8f\u5217\u3002<\/p>\n<p>\u4f8b\u5982&#xff1a;<\/p>\n<p>\u6211 \u559c\u6b22 \u5b66\u4e60 \u4eba\u5de5\u667a\u80fd<\/p>\n<p>\u5e8f\u5217\u4e2d\u7684\u6bcf\u4e00\u4e2a Token \u90fd\u53ef\u4ee5\u5173\u6ce8&#xff1a;<\/p>\n<p>\u6211<br \/>\n\u559c\u6b22<br \/>\n\u5b66\u4e60<br \/>\n\u4eba\u5de5\u667a\u80fd<\/p>\n<p>\u5305\u62ec\u81ea\u5df1\u3002<\/p>\n<p>\u56e0\u6b64&#xff1a;<\/p>\n<p>Self-Attention<\/p>\n<p>\u53ef\u4ee5\u7b80\u5355\u7406\u89e3\u6210&#xff1a;<\/p>\n<p>\u5e8f\u5217\u5185\u90e8\u81ea\u5df1\u5173\u6ce8\u81ea\u5df1\u3002<\/p>\n<p>\u81f3\u4e8e\u4e3a\u4ec0\u4e48\u8fd8\u8981&#xff1a;<\/p>\n<p>Multi-Head<\/p>\n<p>\u4e5f\u5c31\u662f\u201c\u591a\u5934\u201d&#xff0c;\u6211\u4eec\u6682\u65f6\u5148\u7559\u4e00\u4e2a\u95ee\u9898&#xff1a;<\/p>\n<p>\u5982\u679c\u4e00\u4e2a Attention \u80fd\u591f\u5efa\u7acb\u5173\u7cfb&#xff0c;\u4e3a\u4ec0\u4e48\u8fd8\u9700\u8981\u591a\u4e2a Attention Head&#xff1f;<\/p>\n<p>\u8fd9\u4e2a\u95ee\u9898\u4f1a\u5728\u540e\u9762\u7684\u6587\u7ae0\u4e2d\u4e13\u95e8\u5c55\u5f00\u3002<\/p>\n<hr \/>\n<h2>\u516d\u3001\u7b2c\u4e8c\u90e8\u5206&#xff1a;\u4e3a\u4ec0\u4e48 Attention \u540e\u9762\u8fd8\u9700\u8981 Add&#xff1f;<\/h2>\n<p>\u5728 Transformer \u7684\u7ed3\u6784\u56fe\u4e2d&#xff0c;\u4f60\u4f1a\u7ecf\u5e38\u770b\u5230&#xff1a;<\/p>\n<p>Add &amp; Norm<\/p>\n<p>\u5176\u4e2d&#xff1a;<\/p>\n<p>Add<\/p>\n<p>\u6307\u7684\u5c31\u662f&#xff1a;<\/p>\n<p>Residual Connection&#xff0c;\u6b8b\u5dee\u8fde\u63a5\u3002<\/p>\n<p>\u5b83\u7684\u7ed3\u6784\u53ef\u4ee5\u7b80\u5355\u7406\u89e3\u4e3a&#xff1a;<\/p>\n<p>x<br \/>\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n\u2502               \u2502<br \/>\n\u2193               \u2502<br \/>\nSublayer(x)     \u2502<br \/>\n\u2502               \u2502<br \/>\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500 &#043; \u2500\u2500\u2500\u2500\u2500\u2518<br \/>\n        \u2193<br \/>\n      Output<\/p>\n<p>\u6570\u5b66\u4e0a\u53ef\u4ee5\u5199\u6210&#xff1a;<\/p>\n<p><span class=\"katex--display\"><span class=\"katex-display\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>         y<\/p>\n<p>         &#061;<\/p>\n<p>         x<\/p>\n<p>         &#043;<\/p>\n<p>         F<\/p>\n<p>         (<\/p>\n<p>         x<\/p>\n<p>         )<\/p>\n<p>         y &#061; x &#043; F(x) <\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.625em;vertical-align: -0.1944em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.0359em\">y<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><span class=\"mrel\">&#061;<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 0.6667em;vertical-align: -0.0833em\"><\/span><span class=\"mord mathnormal\">x<\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><span class=\"mbin\">&#043;<\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1em;vertical-align: -0.25em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.1389em\">F<\/span><span class=\"mopen\">(<\/span><span class=\"mord mathnormal\">x<\/span><span class=\"mclose\">)<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<p>\u5176\u4e2d&#xff1a;<\/p>\n<ul>\n<li><span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\">\n<p>         x<\/p>\n<p>        x<\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.4306em\"><\/span><span class=\"mord mathnormal\">x<\/span><\/span><\/span><\/span><\/span> \u662f\u539f\u59cb\u8f93\u5165&#xff1b;<\/li>\n<li><span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\">\n<p>         F<\/p>\n<p>         (<\/p>\n<p>         x<\/p>\n<p>         )<\/p>\n<p>        F(x)<\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 1em;vertical-align: -0.25em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.1389em\">F<\/span><span class=\"mopen\">(<\/span><span class=\"mord mathnormal\">x<\/span><span class=\"mclose\">)<\/span><\/span><\/span><\/span><\/span> \u662f\u67d0\u4e2a\u5b50\u5c42\u5904\u7406\u4e4b\u540e\u7684\u7ed3\u679c&#xff1b;<\/li>\n<li><span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\">\n<p>         y<\/p>\n<p>        y<\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.625em;vertical-align: -0.1944em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.0359em\">y<\/span><\/span><\/span><\/span><\/span> \u662f\u6700\u7ec8\u8f93\u51fa\u3002<\/li>\n<\/ul>\n<p>\u4f8b\u5982&#xff1a;<\/p>\n<p>Input<br \/>\n  \u2193<br \/>\nAttention<br \/>\n  \u2193<br \/>\n\u5904\u7406\u7ed3\u679c<\/p>\n<p>\u540c\u65f6&#xff1a;<\/p>\n<p>Input \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n                       \u2193<br \/>\nAttention Output &#043; Input<br \/>\n                       \u2193<br \/>\n                    \u65b0\u8f93\u51fa<\/p>\n<p>\u90a3\u4e48\u95ee\u9898\u6765\u4e86&#xff1a;<\/p>\n<p>\u4e3a\u4ec0\u4e48\u4e0d\u76f4\u63a5\u4f7f\u7528 Attention \u7684\u8f93\u51fa&#xff0c;\u800c\u8981\u628a\u539f\u59cb\u8f93\u5165\u518d\u52a0\u56de\u6765&#xff1f;<\/p>\n<p>\u6700\u76f4\u89c2\u7684\u7406\u89e3\u662f&#xff1a;<\/p>\n<p>\u8ba9\u7f51\u7edc\u5728\u5b66\u4e60\u65b0\u4fe1\u606f\u7684\u540c\u65f6&#xff0c;\u4e0d\u8f7b\u6613\u4e22\u5931\u539f\u59cb\u4fe1\u606f\u3002<\/p>\n<p>\u5047\u8bbe\u67d0\u4e2a\u6a21\u5757\u5904\u7406\u4e4b\u540e\u4ea7\u751f&#xff1a;<\/p>\n<p><span class=\"katex--display\"><span class=\"katex-display\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>         F<\/p>\n<p>         (<\/p>\n<p>         x<\/p>\n<p>         )<\/p>\n<p>         F(x) <\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 1em;vertical-align: -0.25em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.1389em\">F<\/span><span class=\"mopen\">(<\/span><span class=\"mord mathnormal\">x<\/span><span class=\"mclose\">)<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<p>\u5982\u679c\u6211\u4eec\u76f4\u63a5\u8f93\u51fa&#xff1a;<\/p>\n<p><span class=\"katex--display\"><span class=\"katex-display\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>         y<\/p>\n<p>         &#061;<\/p>\n<p>         F<\/p>\n<p>         (<\/p>\n<p>         x<\/p>\n<p>         )<\/p>\n<p>         y &#061; F(x) <\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.625em;vertical-align: -0.1944em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.0359em\">y<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><span class=\"mrel\">&#061;<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1em;vertical-align: -0.25em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.1389em\">F<\/span><span class=\"mopen\">(<\/span><span class=\"mord mathnormal\">x<\/span><span class=\"mclose\">)<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<p>\u90a3\u4e48\u4e0b\u4e00\u5c42\u770b\u5230\u7684\u53ea\u6709&#xff1a;<\/p>\n<p>\u5904\u7406\u4e4b\u540e\u7684\u4fe1\u606f<\/p>\n<p>\u800c\u52a0\u5165\u6b8b\u5dee\u8fde\u63a5\u4e4b\u540e&#xff1a;<\/p>\n<p><span class=\"katex--display\"><span class=\"katex-display\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>         y<\/p>\n<p>         &#061;<\/p>\n<p>         x<\/p>\n<p>         &#043;<\/p>\n<p>         F<\/p>\n<p>         (<\/p>\n<p>         x<\/p>\n<p>         )<\/p>\n<p>         y &#061; x &#043; F(x) <\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.625em;vertical-align: -0.1944em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.0359em\">y<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><span class=\"mrel\">&#061;<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 0.6667em;vertical-align: -0.0833em\"><\/span><span class=\"mord mathnormal\">x<\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><span class=\"mbin\">&#043;<\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1em;vertical-align: -0.25em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.1389em\">F<\/span><span class=\"mopen\">(<\/span><span class=\"mord mathnormal\">x<\/span><span class=\"mclose\">)<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<p>\u4e0b\u4e00\u5c42\u540c\u65f6\u4fdd\u7559&#xff1a;<\/p>\n<p>\u539f\u59cb\u4fe1\u606f &#043; \u65b0\u5b66\u4e60\u7684\u4fe1\u606f<\/p>\n<p>\u4ece\u6df1\u5ea6\u7f51\u7edc\u8bad\u7ec3\u7684\u89d2\u5ea6\u6765\u770b&#xff0c;Residual Connection \u8fd8\u6709\u4e00\u4e2a\u975e\u5e38\u91cd\u8981\u7684\u4f5c\u7528&#xff1a;<\/p>\n<p>\u5e2e\u52a9\u4fe1\u606f\u548c\u68af\u5ea6\u5728\u6df1\u5c42\u7f51\u7edc\u4e2d\u4f20\u64ad\u3002<\/p>\n<p>Transformer \u5f80\u5f80\u4f1a\u5806\u53e0\u5f88\u591a\u5c42\u3002<\/p>\n<p>\u5982\u679c\u4fe1\u606f\u5fc5\u987b\u7ecf\u8fc7&#xff1a;<\/p>\n<p>Layer 1<br \/>\n \u2193<br \/>\nLayer 2<br \/>\n \u2193<br \/>\nLayer 3<br \/>\n \u2193<br \/>\n\u2026\u2026<br \/>\n \u2193<br \/>\nLayer N<\/p>\n<p>\u6bcf\u4e00\u5c42\u90fd\u8fdb\u884c\u590d\u6742\u53d8\u6362&#xff0c;\u90a3\u4e48\u7f51\u7edc\u8d8a\u6df1&#xff0c;\u8bad\u7ec3\u5c31\u53ef\u80fd\u8d8a\u56f0\u96be\u3002<\/p>\n<p>Residual Connection \u76f8\u5f53\u4e8e\u63d0\u4f9b\u4e86\u4e00\u6761\u66f4\u52a0\u76f4\u63a5\u7684\u4fe1\u606f\u901a\u9053\u3002<\/p>\n<p>\u6240\u4ee5&#xff1a;<\/p>\n<p>Add<\/p>\n<p>\u5e76\u4e0d\u662f\u4e00\u4e2a\u53ef\u6709\u53ef\u65e0\u7684\u5c0f\u64cd\u4f5c\u3002<\/p>\n<p>\u5b83\u662f\u6df1\u5c42 Transformer \u80fd\u591f\u7a33\u5b9a\u8bad\u7ec3\u7684\u91cd\u8981\u8bbe\u8ba1\u4e4b\u4e00\u3002<\/p>\n<hr \/>\n<p>\u4e0b\u9762\u662f\u6b8b\u5dee\u8fde\u63a5&#xff08;Residual Connection&#xff09;\u7684 Mermaid \u56fe&#xff1a;<\/p>\n<p>  #mermaid-svg-Bj3ezxrbPLqSCgpu{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Bj3ezxrbPLqSCgpu .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Bj3ezxrbPLqSCgpu .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Bj3ezxrbPLqSCgpu .error-icon{fill:#552222;}#mermaid-svg-Bj3ezxrbPLqSCgpu .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Bj3ezxrbPLqSCgpu .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Bj3ezxrbPLqSCgpu .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Bj3ezxrbPLqSCgpu .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Bj3ezxrbPLqSCgpu .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Bj3ezxrbPLqSCgpu .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Bj3ezxrbPLqSCgpu .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Bj3ezxrbPLqSCgpu .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Bj3ezxrbPLqSCgpu .marker.cross{stroke:#333333;}#mermaid-svg-Bj3ezxrbPLqSCgpu svg{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Bj3ezxrbPLqSCgpu p{margin:0;}#mermaid-svg-Bj3ezxrbPLqSCgpu .label{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;color:#333;}#mermaid-svg-Bj3ezxrbPLqSCgpu .cluster-label text{fill:#333;}#mermaid-svg-Bj3ezxrbPLqSCgpu .cluster-label span{color:#333;}#mermaid-svg-Bj3ezxrbPLqSCgpu .cluster-label span p{background-color:transparent;}#mermaid-svg-Bj3ezxrbPLqSCgpu .label text,#mermaid-svg-Bj3ezxrbPLqSCgpu span{fill:#333;color:#333;}#mermaid-svg-Bj3ezxrbPLqSCgpu .node rect,#mermaid-svg-Bj3ezxrbPLqSCgpu .node circle,#mermaid-svg-Bj3ezxrbPLqSCgpu .node ellipse,#mermaid-svg-Bj3ezxrbPLqSCgpu .node polygon,#mermaid-svg-Bj3ezxrbPLqSCgpu .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Bj3ezxrbPLqSCgpu .rough-node .label text,#mermaid-svg-Bj3ezxrbPLqSCgpu .node .label text,#mermaid-svg-Bj3ezxrbPLqSCgpu .image-shape .label,#mermaid-svg-Bj3ezxrbPLqSCgpu .icon-shape .label{text-anchor:middle;}#mermaid-svg-Bj3ezxrbPLqSCgpu .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Bj3ezxrbPLqSCgpu .rough-node .label,#mermaid-svg-Bj3ezxrbPLqSCgpu .node .label,#mermaid-svg-Bj3ezxrbPLqSCgpu .image-shape .label,#mermaid-svg-Bj3ezxrbPLqSCgpu .icon-shape .label{text-align:center;}#mermaid-svg-Bj3ezxrbPLqSCgpu .node.clickable{cursor:pointer;}#mermaid-svg-Bj3ezxrbPLqSCgpu .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Bj3ezxrbPLqSCgpu .arrowheadPath{fill:#333333;}#mermaid-svg-Bj3ezxrbPLqSCgpu .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Bj3ezxrbPLqSCgpu .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Bj3ezxrbPLqSCgpu .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Bj3ezxrbPLqSCgpu .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Bj3ezxrbPLqSCgpu .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Bj3ezxrbPLqSCgpu .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Bj3ezxrbPLqSCgpu .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Bj3ezxrbPLqSCgpu .cluster text{fill:#333;}#mermaid-svg-Bj3ezxrbPLqSCgpu .cluster span{color:#333;}#mermaid-svg-Bj3ezxrbPLqSCgpu div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Bj3ezxrbPLqSCgpu .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Bj3ezxrbPLqSCgpu rect.text{fill:none;stroke-width:0;}#mermaid-svg-Bj3ezxrbPLqSCgpu .icon-shape,#mermaid-svg-Bj3ezxrbPLqSCgpu .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Bj3ezxrbPLqSCgpu .icon-shape p,#mermaid-svg-Bj3ezxrbPLqSCgpu .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Bj3ezxrbPLqSCgpu .icon-shape .label rect,#mermaid-svg-Bj3ezxrbPLqSCgpu .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Bj3ezxrbPLqSCgpu .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Bj3ezxrbPLqSCgpu .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Bj3ezxrbPLqSCgpu :root{&#8211;mermaid-font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;}<\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>x<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>Sublayer(x)<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>&#043;<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>y &#061; x &#043; F(x)<\/p>\n<p><\/span><\/p>\n<h2>\u4e03\u3001\u7b2c\u4e09\u90e8\u5206&#xff1a;Norm \u53c8\u662f\u4ec0\u4e48&#xff1f;<\/h2>\n<p>Add \u540e\u9762\u7d27\u63a5\u7740&#xff1a;<\/p>\n<p>Norm<\/p>\n<p>\u4e5f\u5c31\u662f&#xff1a;<\/p>\n<p>Normalization&#xff0c;\u5f52\u4e00\u5316\u3002<\/p>\n<p>\u5728 Transformer \u4e2d&#xff0c;\u7ecf\u5178\u8bbe\u8ba1\u4f7f\u7528\u7684\u662f&#xff1a;<\/p>\n<p>Layer Normalization&#xff0c;LayerNorm\u3002<\/p>\n<p>\u6211\u4eec\u5148\u4e0d\u6025\u7740\u63a8\u5bfc\u5b83\u7684\u5168\u90e8\u7ec6\u8282&#xff0c;\u53ea\u7406\u89e3\u5b83\u7684\u4f5c\u7528\u3002<\/p>\n<p>\u5047\u8bbe\u67d0\u4e00\u5c42\u8f93\u51fa\u7684\u6570\u636e\u5206\u5e03\u975e\u5e38\u4e0d\u7a33\u5b9a&#xff1a;<\/p>\n<p>\u67d0\u4e9b\u6570\u503c\u975e\u5e38\u5927<br \/>\n\u67d0\u4e9b\u6570\u503c\u975e\u5e38\u5c0f<br \/>\n\u67d0\u4e9b\u6570\u636e\u5206\u5e03\u53d8\u5316\u5f88\u660e\u663e<\/p>\n<p>\u90a3\u4e48\u540e\u9762\u7684\u7f51\u7edc\u5c31\u9700\u8981\u4e0d\u65ad\u9002\u5e94\u8fd9\u4e9b\u53d8\u5316\u3002<\/p>\n<p>\u8fd9\u4f1a\u589e\u52a0\u8bad\u7ec3\u96be\u5ea6\u3002<\/p>\n<p>\u4e8e\u662f&#xff0c;\u6211\u4eec\u5e0c\u671b\u5bf9\u6570\u636e\u8fdb\u884c\u4e00\u5b9a\u7a0b\u5ea6\u7684\u6807\u51c6\u5316\u5904\u7406&#xff0c;\u8ba9\u540e\u7eed\u7f51\u7edc\u80fd\u591f\u5728\u66f4\u52a0\u7a33\u5b9a\u7684\u6570\u636e\u5206\u5e03\u4e0a\u8fdb\u884c\u8bad\u7ec3\u3002<\/p>\n<p>LayerNorm \u7684\u57fa\u672c\u601d\u60f3\u662f&#xff1a;<\/p>\n<p>\u5bf9\u5355\u4e2a\u6837\u672c\u7684\u7279\u5f81\u7ef4\u5ea6\u8fdb\u884c\u5f52\u4e00\u5316\u3002<\/p>\n<p>\u5bf9\u4e8e\u4e00\u4e2a\u8f93\u5165\u5411\u91cf&#xff1a;<\/p>\n<p><span class=\"katex--display\"><span class=\"katex-display\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>         x<\/p>\n<p>         &#061;<\/p>\n<p>         [<\/p>\n<p>          x<\/p>\n<p>          1<\/p>\n<p>         ,<\/p>\n<p>          x<\/p>\n<p>          2<\/p>\n<p>         ,<\/p>\n<p>         \u2026<\/p>\n<p>         ,<\/p>\n<p>          x<\/p>\n<p>          d<\/p>\n<p>         ]<\/p>\n<p>         x &#061; [x_1, x_2, \\\\dots, x_d] <\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.4306em\"><\/span><span class=\"mord mathnormal\">x<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><span class=\"mrel\">&#061;<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1em;vertical-align: -0.25em\"><\/span><span class=\"mopen\">[<\/span><span class=\"mord\"><span class=\"mord mathnormal\">x<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3011em\"><span class=\"\" style=\"top: -2.55em;margin-left: 0em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.1667em\"><\/span><span class=\"mord\"><span class=\"mord mathnormal\">x<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3011em\"><span class=\"\" style=\"top: -2.55em;margin-left: 0em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">2<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.1667em\"><\/span><span class=\"minner\">\u2026<\/span><span class=\"mspace\" style=\"margin-right: 0.1667em\"><\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.1667em\"><\/span><span class=\"mord\"><span class=\"mord mathnormal\">x<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3361em\"><span class=\"\" style=\"top: -2.55em;margin-left: 0em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathnormal mtight\">d<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mclose\">]<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<p>\u9996\u5148\u8ba1\u7b97\u5747\u503c&#xff1a;<\/p>\n<p><span class=\"katex--display\"><span class=\"katex-display\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>         \u03bc<\/p>\n<p>         &#061;<\/p>\n<p>          1<\/p>\n<p>          d<\/p>\n<p>          \u2211<\/p>\n<p>           i<\/p>\n<p>           &#061;<\/p>\n<p>           1<\/p>\n<p>          d<\/p>\n<p>          x<\/p>\n<p>          i<\/p>\n<p>         \\\\mu &#061; \\\\frac{1}{d}\\\\sum_{i&#061;1}^{d}x_i <\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.625em;vertical-align: -0.1944em\"><\/span><span class=\"mord mathnormal\">\u03bc<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><span class=\"mrel\">&#061;<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 3.1138em;vertical-align: -1.2777em\"><\/span><span class=\"mord\"><span class=\"mopen nulldelimiter\"><\/span><span class=\"mfrac\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.3214em\"><span class=\"\" style=\"top: -2.314em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"mord\"><span class=\"mord mathnormal\">d<\/span><\/span><\/span><span class=\"\" style=\"top: -3.23em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"frac-line\" style=\"border-bottom-width: 0.04em\"><\/span><\/span><span class=\"\" style=\"top: -3.677em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"mord\"><span class=\"mord\">1<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.686em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><span class=\"mclose nulldelimiter\"><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.1667em\"><\/span><span class=\"mop op-limits\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.8361em\"><span class=\"\" style=\"top: -1.8723em;margin-left: 0em\"><span class=\"pstrut\" style=\"height: 3.05em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\"><span class=\"mord mathnormal mtight\">i<\/span><span class=\"mrel mtight\">&#061;<\/span><span class=\"mord mtight\">1<\/span><\/span><\/span><\/span><span class=\"\" style=\"top: -3.05em\"><span class=\"pstrut\" style=\"height: 3.05em\"><\/span><span class=\"\"><span class=\"mop op-symbol large-op\">\u2211<\/span><\/span><\/span><span class=\"\" style=\"top: -4.3em;margin-left: 0em\"><span class=\"pstrut\" style=\"height: 3.05em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\"><span class=\"mord mathnormal mtight\">d<\/span><\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.2777em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.1667em\"><\/span><span class=\"mord\"><span class=\"mord mathnormal\">x<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3117em\"><span class=\"\" style=\"top: -2.55em;margin-left: 0em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathnormal mtight\">i<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<p>\u7136\u540e\u8ba1\u7b97\u65b9\u5dee&#xff1a;<\/p>\n<p><span class=\"katex--display\"><span class=\"katex-display\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>          \u03c3<\/p>\n<p>          2<\/p>\n<p>         &#061;<\/p>\n<p>          1<\/p>\n<p>          d<\/p>\n<p>          \u2211<\/p>\n<p>           i<\/p>\n<p>           &#061;<\/p>\n<p>           1<\/p>\n<p>          d<\/p>\n<p>         (<\/p>\n<p>          x<\/p>\n<p>          i<\/p>\n<p>         \u2212<\/p>\n<p>         \u03bc<\/p>\n<p>          )<\/p>\n<p>          2<\/p>\n<p>         \\\\sigma^2 &#061; \\\\frac{1}{d}\\\\sum_{i&#061;1}^{d}(x_i-\\\\mu)^2 <\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.8641em\"><\/span><span class=\"mord\"><span class=\"mord mathnormal\" style=\"margin-right: 0.0359em\">\u03c3<\/span><span class=\"msupsub\"><span class=\"vlist-t\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.8641em\"><span class=\"\" style=\"top: -3.113em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">2<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><span class=\"mrel\">&#061;<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 3.1138em;vertical-align: -1.2777em\"><\/span><span class=\"mord\"><span class=\"mopen nulldelimiter\"><\/span><span class=\"mfrac\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.3214em\"><span class=\"\" style=\"top: -2.314em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"mord\"><span class=\"mord mathnormal\">d<\/span><\/span><\/span><span class=\"\" style=\"top: -3.23em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"frac-line\" style=\"border-bottom-width: 0.04em\"><\/span><\/span><span class=\"\" style=\"top: -3.677em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"mord\"><span class=\"mord\">1<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.686em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><span class=\"mclose nulldelimiter\"><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.1667em\"><\/span><span class=\"mop op-limits\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.8361em\"><span class=\"\" style=\"top: -1.8723em;margin-left: 0em\"><span class=\"pstrut\" style=\"height: 3.05em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\"><span class=\"mord mathnormal mtight\">i<\/span><span class=\"mrel mtight\">&#061;<\/span><span class=\"mord mtight\">1<\/span><\/span><\/span><\/span><span class=\"\" style=\"top: -3.05em\"><span class=\"pstrut\" style=\"height: 3.05em\"><\/span><span class=\"\"><span class=\"mop op-symbol large-op\">\u2211<\/span><\/span><\/span><span class=\"\" style=\"top: -4.3em;margin-left: 0em\"><span class=\"pstrut\" style=\"height: 3.05em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\"><span class=\"mord mathnormal mtight\">d<\/span><\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.2777em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><span class=\"mopen\">(<\/span><span class=\"mord\"><span class=\"mord mathnormal\">x<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3117em\"><span class=\"\" style=\"top: -2.55em;margin-left: 0em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathnormal mtight\">i<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><span class=\"mbin\">\u2212<\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1.1141em;vertical-align: -0.25em\"><\/span><span class=\"mord mathnormal\">\u03bc<\/span><span class=\"mclose\"><span class=\"mclose\">)<\/span><span class=\"msupsub\"><span class=\"vlist-t\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.8641em\"><span class=\"\" style=\"top: -3.113em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">2<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<p>\u6700\u540e\u8fdb\u884c\u5f52\u4e00\u5316&#xff1a;<\/p>\n<p><span class=\"katex--display\"><span class=\"katex-display\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>           x<\/p>\n<p>           ^<\/p>\n<p>          i<\/p>\n<p>         &#061;<\/p>\n<p>            x<\/p>\n<p>            i<\/p>\n<p>           \u2212<\/p>\n<p>           \u03bc<\/p>\n<p>             \u03c3<\/p>\n<p>             2<\/p>\n<p>            &#043;<\/p>\n<p>            \u03f5<\/p>\n<p>         \\\\hat{x}_i &#061; \\\\frac{x_i-\\\\mu}{\\\\sqrt{\\\\sigma^2&#043;\\\\epsilon}} <\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.8444em;vertical-align: -0.15em\"><\/span><span class=\"mord\"><span class=\"mord accent\"><span class=\"vlist-t\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.6944em\"><span class=\"\" style=\"top: -3em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"mord mathnormal\">x<\/span><\/span><span class=\"\" style=\"top: -3em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"accent-body\" style=\"left: -0.2222em\"><span class=\"mord\">^<\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3117em\"><span class=\"\" style=\"top: -2.55em;margin-left: 0em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathnormal mtight\">i<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><span class=\"mrel\">&#061;<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 2.1903em;vertical-align: -0.93em\"><\/span><span class=\"mord\"><span class=\"mopen nulldelimiter\"><\/span><span class=\"mfrac\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.2603em\"><span class=\"\" style=\"top: -2.1966em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"mord\"><span class=\"mord sqrt\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.9134em\"><span class=\"svg-align\" style=\"top: -3em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"mord\" style=\"padding-left: 0.833em\"><span class=\"mord\"><span class=\"mord mathnormal\" style=\"margin-right: 0.0359em\">\u03c3<\/span><span class=\"msupsub\"><span class=\"vlist-t\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.7401em\"><span class=\"\" style=\"top: -2.989em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">2<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><span class=\"mbin\">&#043;<\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><span class=\"mord mathnormal\">\u03f5<\/span><\/span><\/span><span class=\"\" style=\"top: -2.8734em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"hide-tail\" style=\"min-width: 0.853em;height: 1.08em\"><\/p>\n<p>                   <\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.1266em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"\" style=\"top: -3.23em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"frac-line\" style=\"border-bottom-width: 0.04em\"><\/span><\/span><span class=\"\" style=\"top: -3.677em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"mord\"><span class=\"mord\"><span class=\"mord mathnormal\">x<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3117em\"><span class=\"\" style=\"top: -2.55em;margin-left: 0em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathnormal mtight\">i<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><span class=\"mbin\">\u2212<\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><span class=\"mord mathnormal\">\u03bc<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.93em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><span class=\"mclose nulldelimiter\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<p>\u968f\u540e\u518d\u901a\u8fc7\u53ef\u5b66\u4e60\u53c2\u6570\u8fdb\u884c\u7f29\u653e\u548c\u5e73\u79fb&#xff1a;<\/p>\n<p><span class=\"katex--display\"><span class=\"katex-display\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>          y<\/p>\n<p>          i<\/p>\n<p>         &#061;<\/p>\n<p>         \u03b3<\/p>\n<p>           x<\/p>\n<p>           ^<\/p>\n<p>          i<\/p>\n<p>         &#043;<\/p>\n<p>         \u03b2<\/p>\n<p>         y_i &#061; \\\\gamma \\\\hat{x}_i &#043; \\\\beta <\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.625em;vertical-align: -0.1944em\"><\/span><span class=\"mord\"><span class=\"mord mathnormal\" style=\"margin-right: 0.0359em\">y<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3117em\"><span class=\"\" style=\"top: -2.55em;margin-left: -0.0359em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathnormal mtight\">i<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><span class=\"mrel\">&#061;<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 0.8889em;vertical-align: -0.1944em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.0556em\">\u03b3<\/span><span class=\"mord\"><span class=\"mord accent\"><span class=\"vlist-t\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.6944em\"><span class=\"\" style=\"top: -3em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"mord mathnormal\">x<\/span><\/span><span class=\"\" style=\"top: -3em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"accent-body\" style=\"left: -0.2222em\"><span class=\"mord\">^<\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3117em\"><span class=\"\" style=\"top: -2.55em;margin-left: 0em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathnormal mtight\">i<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><span class=\"mbin\">&#043;<\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 0.8889em;vertical-align: -0.1944em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.0528em\">\u03b2<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<p>\u5176\u4e2d&#xff1a;<\/p>\n<ul>\n<li><span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\">\n<p>         \u03b3<\/p>\n<p>        \\\\gamma<\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.625em;vertical-align: -0.1944em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.0556em\">\u03b3<\/span><\/span><\/span><\/span><\/span>&#xff1a;\u7f29\u653e\u53c2\u6570&#xff1b;<\/li>\n<li><span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\">\n<p>         \u03b2<\/p>\n<p>        \\\\beta<\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.8889em;vertical-align: -0.1944em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.0528em\">\u03b2<\/span><\/span><\/span><\/span><\/span>&#xff1a;\u5e73\u79fb\u53c2\u6570&#xff1b;<\/li>\n<li><span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\">\n<p>         \u03f5<\/p>\n<p>        \\\\epsilon<\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.4306em\"><\/span><span class=\"mord mathnormal\">\u03f5<\/span><\/span><\/span><\/span><\/span>&#xff1a;\u9632\u6b62\u5206\u6bcd\u4e3a <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>         0<\/p>\n<p>        0<\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.6444em\"><\/span><span class=\"mord\">0<\/span><\/span><\/span><\/span><\/span> \u7684\u4e00\u4e2a\u5f88\u5c0f\u7684\u5e38\u6570\u3002<\/li>\n<\/ul>\n<p>\u73b0\u5728\u4e0d\u9700\u8981\u5f3a\u884c\u8bb0\u4f4f\u8fd9\u4e9b\u516c\u5f0f\u3002<\/p>\n<p>\u8fd9\u4e00\u9636\u6bb5\u66f4\u91cd\u8981\u7684\u662f\u77e5\u9053&#xff1a;<\/p>\n<p>LayerNorm \u7684\u4f5c\u7528&#xff0c;\u662f\u5e2e\u52a9\u7f51\u7edc\u4fdd\u6301\u66f4\u52a0\u7a33\u5b9a\u7684\u6570\u503c\u5206\u5e03&#xff0c;\u4ece\u800c\u8ba9\u6df1\u5c42\u7f51\u7edc\u8bad\u7ec3\u5f97\u66f4\u52a0\u7a33\u5b9a\u3002<\/p>\n<p>\u56e0\u6b64&#xff1a;<\/p>\n<p>Attention<br \/>\n    \u2193<br \/>\nAdd<br \/>\n    \u2193<br \/>\nLayerNorm<\/p>\n<p>\u5171\u540c\u6784\u6210\u4e86 Transformer \u4e2d\u975e\u5e38\u91cd\u8981\u7684\u4e00\u90e8\u5206\u3002<\/p>\n<hr \/>\n<h2>\u516b\u3001\u7b2c\u56db\u90e8\u5206&#xff1a;Attention \u4e4b\u540e&#xff0c;\u4e3a\u4ec0\u4e48\u8fd8\u9700\u8981 FFN&#xff1f;<\/h2>\n<p>\u5f88\u591a\u521a\u5b66\u4e60 Transformer \u7684\u4eba&#xff0c;\u4f1a\u628a\u6ce8\u610f\u529b\u5168\u90e8\u653e\u5728 Attention \u4e0a\u3002<\/p>\n<p>\u4f46\u5b9e\u9645\u4e0a&#xff0c;\u4e00\u4e2a\u5b8c\u6574\u7684 Transformer Block \u4e2d\u8fd8\u6709\u4e00\u4e2a\u975e\u5e38\u91cd\u8981\u7684\u6a21\u5757&#xff1a;<\/p>\n<p>Feed Forward Network&#xff0c;\u524d\u9988\u795e\u7ecf\u7f51\u7edc\u3002<\/p>\n<p>\u901a\u5e38\u7b80\u79f0&#xff1a;<\/p>\n<p>FFN<\/p>\n<p>\u5b83\u7684\u7ed3\u6784\u5176\u5b9e\u5e76\u4e0d\u590d\u6742\u3002<\/p>\n<p>\u53ef\u4ee5\u7406\u89e3\u6210&#xff1a;<\/p>\n<p>\u8f93\u5165<br \/>\n \u2193<br \/>\nLinear<br \/>\n \u2193<br \/>\nActivation<br \/>\n \u2193<br \/>\nLinear<br \/>\n \u2193<br \/>\n\u8f93\u51fa<\/p>\n<p>\u539f\u59cb Transformer \u4e2d\u53ef\u4ee5\u5199\u6210&#xff1a;<\/p>\n<h2>$$ \\\\operatorname{FFN}(x)<\/h2>\n<p>\\\\max(0, xW_1&#043;b_1)W_2&#043;b_2 $$<\/p>\n<p>\u5176\u4e2d&#xff1a;<\/p>\n<ul>\n<li>\u7b2c\u4e00\u5c42\u7ebf\u6027\u5c42\u901a\u5e38\u5c06\u7ef4\u5ea6\u6269\u5927&#xff1b;<\/li>\n<li>\u4e2d\u95f4\u7ecf\u8fc7\u975e\u7ebf\u6027\u6fc0\u6d3b\u51fd\u6570&#xff1b;<\/li>\n<li>\u7b2c\u4e8c\u5c42\u518d\u5c06\u7ef4\u5ea6\u6620\u5c04\u56de\u6765\u3002<\/li>\n<\/ul>\n<p>\u4f8b\u5982&#xff1a;<\/p>\n<p>d_model<br \/>\n   \u2193<br \/>\nLinear<br \/>\n   \u2193<br \/>\nd_ff<br \/>\n   \u2193<br \/>\nActivation<br \/>\n   \u2193<br \/>\nLinear<br \/>\n   \u2193<br \/>\nd_model<\/p>\n<p>\u90a3\u4e48\u95ee\u9898\u6765\u4e86&#xff1a;<\/p>\n<p>Attention \u4e0d\u662f\u5df2\u7ecf\u5904\u7406\u5b8c\u4fe1\u606f\u4e86\u5417&#xff1f;\u4e3a\u4ec0\u4e48\u8fd8\u9700\u8981 FFN&#xff1f;<\/p>\n<p>\u8fd9\u91cc\u53ef\u4ee5\u4ece\u804c\u8d23\u4e0a\u7406\u89e3\u3002<\/p>\n<p>Attention \u66f4\u64c5\u957f\u505a&#xff1a;<\/p>\n<p>Token \u4e0e Token \u4e4b\u95f4\u7684\u4fe1\u606f\u4ea4\u4e92\u3002<\/p>\n<p>\u4f8b\u5982&#xff1a;<\/p>\n<p>Token A<br \/>\n   \u2194<br \/>\nToken B<br \/>\n   \u2194<br \/>\nToken C<\/p>\n<p>\u800c FFN \u5219\u66f4\u50cf\u662f\u5728&#xff1a;<\/p>\n<p>\u6bcf\u4e2a Token \u83b7\u5f97\u4e0a\u4e0b\u6587\u4fe1\u606f\u4e4b\u540e&#xff0c;\u8fdb\u4e00\u6b65\u5bf9\u81ea\u5df1\u7684\u8868\u793a\u8fdb\u884c\u975e\u7ebf\u6027\u53d8\u6362\u3002<\/p>\n<p>\u6211\u4eec\u53ef\u4ee5\u7b80\u5355\u7406\u89e3\u4e3a&#xff1a;<\/p>\n<p>Attention&#xff1a;<br \/>\n\u201c\u6211\u5e94\u8be5\u4ece\u5176\u4ed6 Token \u83b7\u5f97\u54ea\u4e9b\u4fe1\u606f&#xff1f;\u201d<\/p>\n<p>FFN&#xff1a;<br \/>\n\u201c\u83b7\u5f97\u8fd9\u4e9b\u4fe1\u606f\u4e4b\u540e&#xff0c;\u6211\u5e94\u8be5\u5982\u4f55\u8fdb\u4e00\u6b65\u5904\u7406\u81ea\u5df1&#xff1f;\u201d<\/p>\n<p>\u56e0\u6b64&#xff0c;\u4e00\u4e2a Transformer Block \u5b9e\u9645\u4e0a\u5305\u542b\u4e24\u7c7b\u91cd\u8981\u8ba1\u7b97&#xff1a;<\/p>\n<p>Attention<br \/>\n    \u2193<br \/>\nToken \u4e4b\u95f4\u4ea4\u6362\u4fe1\u606f<\/p>\n<p>FFN<br \/>\n    \u2193<br \/>\n\u6bcf\u4e2a Token \u81ea\u5df1\u8fdb\u884c\u8fdb\u4e00\u6b65\u53d8\u6362<\/p>\n<p>\u4e24\u8005\u7f3a\u4e00\u4e0d\u53ef\u3002<\/p>\n<hr \/>\n<h2>\u4e5d\u3001\u73b0\u5728&#xff0c;\u6211\u4eec\u7ec8\u4e8e\u53ef\u4ee5\u770b\u61c2 Encoder Block \u4e86<\/h2>\n<p>\u7ecf\u8fc7\u524d\u9762\u7684\u62c6\u89e3&#xff0c;\u4e00\u4e2a Encoder Block \u53ef\u4ee5\u91cd\u65b0\u753b\u6210&#xff1a;<\/p>\n<p>Input<br \/>\n  \u2502<br \/>\n  \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n  \u2193                              \u2502<br \/>\nMulti-Head Self-Attention        \u2502<br \/>\n  \u2193                              \u2502<br \/>\n        Add \u25c4\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<br \/>\n  \u2193<br \/>\nLayerNorm<br \/>\n  \u2502<br \/>\n  \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n  \u2193                              \u2502<br \/>\nFeed Forward Network             \u2502<br \/>\n  \u2193                              \u2502<br \/>\n        Add \u25c4\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<br \/>\n  \u2193<br \/>\nLayerNorm<br \/>\n  \u2193<br \/>\nOutput<\/p>\n<p>\u73b0\u5728&#xff0c;\u6211\u4eec\u53ef\u4ee5\u7ed9\u6bcf\u4e2a\u6a21\u5757\u4e00\u53e5\u8bdd\u7684\u5b9a\u4f4d&#xff1a;<\/p>\n<h4>Multi-Head Self-Attention<\/h4>\n<p>\u8ba9\u4e0d\u540c Token \u4e4b\u95f4\u4ea4\u6362\u4fe1\u606f\u3002<\/p>\n<h4>Residual Connection<\/h4>\n<p>\u4fdd\u7559\u539f\u59cb\u4fe1\u606f&#xff0c;\u5e76\u5e2e\u52a9\u6df1\u5c42\u7f51\u7edc\u8bad\u7ec3\u3002<\/p>\n<h4>LayerNorm<\/h4>\n<p>\u7a33\u5b9a\u6570\u636e\u5206\u5e03&#xff0c;\u5e2e\u52a9\u8bad\u7ec3\u8fc7\u7a0b\u66f4\u52a0\u7a33\u5b9a\u3002<\/p>\n<h4>Feed Forward Network<\/h4>\n<p>\u5bf9\u6bcf\u4e2a Token \u7684\u8868\u793a\u8fdb\u4e00\u6b65\u8fdb\u884c\u975e\u7ebf\u6027\u53d8\u6362\u3002<\/p>\n<p>\u4e8e\u662f&#xff0c;\u4e00\u4e2a Encoder Block \u7684\u5b8c\u6574\u5de5\u4f5c\u6d41\u7a0b\u53ef\u4ee5\u7406\u89e3\u6210&#xff1a;<\/p>\n<p>\u539f\u59cb Token \u8868\u793a<br \/>\n      \u2193<br \/>\n\u5148\u770b\u770b\u5176\u4ed6 Token<br \/>\n      \u2193<br \/>\n\u83b7\u5f97\u4e0a\u4e0b\u6587\u4fe1\u606f<br \/>\n      \u2193<br \/>\n\u4fdd\u7559\u539f\u59cb\u4fe1\u606f<br \/>\n      \u2193<br \/>\n\u7a33\u5b9a\u6570\u636e\u5206\u5e03<br \/>\n      \u2193<br \/>\n\u8fdb\u4e00\u6b65\u8fdb\u884c\u975e\u7ebf\u6027\u5904\u7406<br \/>\n      \u2193<br \/>\n\u518d\u6b21\u4fdd\u7559\u539f\u59cb\u4fe1\u606f<br \/>\n      \u2193<br \/>\n\u8f93\u51fa\u65b0\u7684 Token \u8868\u793a<\/p>\n<p>\u8fd9\u5c31\u662f Encoder Block \u7684\u57fa\u672c\u601d\u60f3\u3002<\/p>\n<hr \/>\n<p>\u4e0b\u9762\u662f Encoder Block \u5b8c\u6574\u5de5\u4f5c\u6d41\u7a0b\u7684 Mermaid \u56fe&#xff1a;<\/p>\n<p>  #mermaid-svg-ZjLv1EgqBsceOf01{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ZjLv1EgqBsceOf01 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ZjLv1EgqBsceOf01 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ZjLv1EgqBsceOf01 .error-icon{fill:#552222;}#mermaid-svg-ZjLv1EgqBsceOf01 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ZjLv1EgqBsceOf01 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ZjLv1EgqBsceOf01 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ZjLv1EgqBsceOf01 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ZjLv1EgqBsceOf01 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ZjLv1EgqBsceOf01 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ZjLv1EgqBsceOf01 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ZjLv1EgqBsceOf01 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ZjLv1EgqBsceOf01 .marker.cross{stroke:#333333;}#mermaid-svg-ZjLv1EgqBsceOf01 svg{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ZjLv1EgqBsceOf01 p{margin:0;}#mermaid-svg-ZjLv1EgqBsceOf01 .label{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;color:#333;}#mermaid-svg-ZjLv1EgqBsceOf01 .cluster-label text{fill:#333;}#mermaid-svg-ZjLv1EgqBsceOf01 .cluster-label span{color:#333;}#mermaid-svg-ZjLv1EgqBsceOf01 .cluster-label span p{background-color:transparent;}#mermaid-svg-ZjLv1EgqBsceOf01 .label text,#mermaid-svg-ZjLv1EgqBsceOf01 span{fill:#333;color:#333;}#mermaid-svg-ZjLv1EgqBsceOf01 .node rect,#mermaid-svg-ZjLv1EgqBsceOf01 .node circle,#mermaid-svg-ZjLv1EgqBsceOf01 .node ellipse,#mermaid-svg-ZjLv1EgqBsceOf01 .node polygon,#mermaid-svg-ZjLv1EgqBsceOf01 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-ZjLv1EgqBsceOf01 .rough-node .label text,#mermaid-svg-ZjLv1EgqBsceOf01 .node .label text,#mermaid-svg-ZjLv1EgqBsceOf01 .image-shape .label,#mermaid-svg-ZjLv1EgqBsceOf01 .icon-shape .label{text-anchor:middle;}#mermaid-svg-ZjLv1EgqBsceOf01 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-ZjLv1EgqBsceOf01 .rough-node .label,#mermaid-svg-ZjLv1EgqBsceOf01 .node .label,#mermaid-svg-ZjLv1EgqBsceOf01 .image-shape .label,#mermaid-svg-ZjLv1EgqBsceOf01 .icon-shape .label{text-align:center;}#mermaid-svg-ZjLv1EgqBsceOf01 .node.clickable{cursor:pointer;}#mermaid-svg-ZjLv1EgqBsceOf01 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-ZjLv1EgqBsceOf01 .arrowheadPath{fill:#333333;}#mermaid-svg-ZjLv1EgqBsceOf01 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-ZjLv1EgqBsceOf01 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-ZjLv1EgqBsceOf01 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ZjLv1EgqBsceOf01 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-ZjLv1EgqBsceOf01 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ZjLv1EgqBsceOf01 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-ZjLv1EgqBsceOf01 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-ZjLv1EgqBsceOf01 .cluster text{fill:#333;}#mermaid-svg-ZjLv1EgqBsceOf01 .cluster span{color:#333;}#mermaid-svg-ZjLv1EgqBsceOf01 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-ZjLv1EgqBsceOf01 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-ZjLv1EgqBsceOf01 rect.text{fill:none;stroke-width:0;}#mermaid-svg-ZjLv1EgqBsceOf01 .icon-shape,#mermaid-svg-ZjLv1EgqBsceOf01 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ZjLv1EgqBsceOf01 .icon-shape p,#mermaid-svg-ZjLv1EgqBsceOf01 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-ZjLv1EgqBsceOf01 .icon-shape .label rect,#mermaid-svg-ZjLv1EgqBsceOf01 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ZjLv1EgqBsceOf01 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-ZjLv1EgqBsceOf01 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-ZjLv1EgqBsceOf01 :root{&#8211;mermaid-font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;}<\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"edgeLabel\"><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>\u539f\u59cb Token \u8868\u793a<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>\u5148\u770b\u770b\u5176\u4ed6 Token<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>\u83b7\u5f97\u4e0a\u4e0b\u6587\u4fe1\u606f<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>\u4fdd\u7559\u539f\u59cb\u4fe1\u606f<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>\u7a33\u5b9a\u6570\u636e\u5206\u5e03<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>\u8fdb\u4e00\u6b65\u8fdb\u884c\u975e\u7ebf\u6027\u5904\u7406<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>\u518d\u6b21\u4fdd\u7559\u539f\u59cb\u4fe1\u606f<\/p>\n<p><\/span><\/p>\n<p>         <span class=\"nodeLabel\"><\/p>\n<p>\u8f93\u51fa\u65b0\u7684 Token \u8868\u793a<\/p>\n<p><\/span><\/p>\n<h2>\u5341\u3001Encoder \u5230\u5e95\u8f93\u51fa\u4e86\u4ec0\u4e48&#xff1f;<\/h2>\n<p>Encoder \u5904\u7406\u5b8c\u6210\u4e4b\u540e&#xff0c;\u4f1a\u8f93\u51fa\u4ec0\u4e48&#xff1f;<\/p>\n<p>\u7b54\u6848\u4e0d\u662f&#xff1a;<\/p>\n<p>\u4e00\u4e2a\u53e5\u5b50<\/p>\n<p>\u4e5f\u4e0d\u662f&#xff1a;<\/p>\n<p>\u6700\u7ec8\u5206\u7c7b\u7ed3\u679c<\/p>\n<p>\u800c\u662f&#xff1a;<\/p>\n<p>\u7ecf\u8fc7\u4e0a\u4e0b\u6587\u7f16\u7801\u4e4b\u540e\u7684\u4e00\u7ec4 Token \u8868\u793a\u3002<\/p>\n<p>\u4f8b\u5982\u8f93\u5165&#xff1a;<\/p>\n<p>I love artificial intelligence.<\/p>\n<p>\u6700\u5f00\u59cb&#xff1a;<\/p>\n<p>I<br \/>\nlove<br \/>\nartificial<br \/>\nintelligence<\/p>\n<p>\u6bcf\u4e2a Token \u90fd\u6709\u81ea\u5df1\u7684\u521d\u59cb Embedding\u3002<\/p>\n<p>\u7ecf\u8fc7\u591a\u4e2a Encoder Block \u4e4b\u540e&#xff1a;<\/p>\n<p>I&#039;                \u2190 \u7ed3\u5408\u4e0a\u4e0b\u6587\u4e4b\u540e\u7684\u65b0\u8868\u793a<br \/>\nlove&#039;             \u2190 \u7ed3\u5408\u4e0a\u4e0b\u6587\u4e4b\u540e\u7684\u65b0\u8868\u793a<br \/>\nartificial&#039;       \u2190 \u7ed3\u5408\u4e0a\u4e0b\u6587\u4e4b\u540e\u7684\u65b0\u8868\u793a<br \/>\nintelligence&#039;     \u2190 \u7ed3\u5408\u4e0a\u4e0b\u6587\u4e4b\u540e\u7684\u65b0\u8868\u793a<\/p>\n<p>\u8fd9\u91cc\u7684&#xff1a;<\/p>\n<p>I&#039;<br \/>\nlove&#039;<br \/>\n&#8230;<\/p>\n<p>\u5e76\u4e0d\u662f\u65b0\u7684\u5355\u8bcd\u3002<\/p>\n<p>\u800c\u662f\u8868\u793a&#xff1a;<\/p>\n<p>\u6bcf\u4e2a Token \u5bf9\u5e94\u7684\u5411\u91cf\u5df2\u7ecf\u53d1\u751f\u4e86\u53d8\u5316\u3002<\/p>\n<p>\u4f8b\u5982&#xff1a;<\/p>\n<p>love<\/p>\n<p>\u7ecf\u8fc7 Encoder \u4e4b\u540e&#xff0c;\u5b83\u6240\u5bf9\u5e94\u7684\u5411\u91cf\u4e0d\u518d\u53ea\u8868\u793a&#xff1a;<\/p>\n<p>\u201clove \u8fd9\u4e2a\u8bcd\u672c\u8eab\u201d\u3002<\/p>\n<p>\u5b83\u53ef\u80fd\u5df2\u7ecf\u5305\u542b&#xff1a;<\/p>\n<p>\u8c01\u5728 love&#xff1f;<br \/>\nlove \u7684\u5bf9\u8c61\u662f\u4ec0\u4e48&#xff1f;<br \/>\n\u8fd9\u53e5\u8bdd\u6574\u4f53\u8868\u8fbe\u4ec0\u4e48&#xff1f;<\/p>\n<p>\u8fd9\u6837\u7684\u4e0a\u4e0b\u6587\u4fe1\u606f\u3002<\/p>\n<p>\u6240\u4ee5 Encoder \u7684\u8f93\u51fa\u53ef\u4ee5\u7406\u89e3\u4e3a&#xff1a;<\/p>\n<p>\u4e00\u7ec4\u4e0a\u4e0b\u6587\u5316&#xff08;Contextualized&#xff09;\u7684 Token \u8868\u793a\u3002<\/p>\n<hr \/>\n<h2>\u5341\u4e00\u3001\u63a5\u4e0b\u6765\u8f6e\u5230 Decoder<\/h2>\n<p>\u5982\u679c\u8bf4 Encoder \u8d1f\u8d23&#xff1a;<\/p>\n<p>\u7406\u89e3\u8f93\u5165\u3002<\/p>\n<p>\u90a3\u4e48 Decoder \u8d1f\u8d23\u7684\u5c31\u662f&#xff1a;<\/p>\n<p>\u751f\u6210\u8f93\u51fa\u3002<\/p>\n<p>\u4f8b\u5982\u673a\u5668\u7ffb\u8bd1&#xff1a;<\/p>\n<p>\u8f93\u5165&#xff1a;<\/p>\n<p>I love artificial intelligence.<\/p>\n<p>\u8f93\u51fa&#xff1a;<\/p>\n<p>\u6211\u559c\u6b22\u4eba\u5de5\u667a\u80fd\u3002<\/p>\n<p>Decoder \u5e76\u4e0d\u662f\u4e00\u6b21\u6027\u628a\u6574\u53e5\u8bdd\u5168\u90e8\u751f\u6210\u51fa\u6765\u3002<\/p>\n<p>\u5b83\u901a\u5e38\u6309\u7167\u81ea\u56de\u5f52\u7684\u65b9\u5f0f&#xff1a;<\/p>\n<p>\u5f00\u59cb<br \/>\n \u2193<br \/>\n\u6211<br \/>\n \u2193<br \/>\n\u6211 \u559c\u6b22<br \/>\n \u2193<br \/>\n\u6211 \u559c\u6b22 \u4eba\u5de5\u667a\u80fd<br \/>\n \u2193<br \/>\n\u7ed3\u675f<\/p>\n<p>\u4e5f\u5c31\u662f\u8bf4&#xff1a;<\/p>\n<p>\u751f\u6210\u5f53\u524d Token \u65f6&#xff0c;\u9700\u8981\u53c2\u8003\u4e4b\u524d\u5df2\u7ecf\u751f\u6210\u7684 Token\u3002<\/p>\n<p>\u56e0\u6b64 Decoder \u9762\u4e34\u4e24\u4e2a\u4efb\u52a1&#xff1a;<\/p>\n<li>\u5173\u6ce8\u81ea\u5df1\u4e4b\u524d\u751f\u6210\u7684\u5185\u5bb9&#xff1b;<\/li>\n<li>\u5173\u6ce8 Encoder \u5bf9\u8f93\u5165\u5e8f\u5217\u7684\u7406\u89e3\u3002<\/li>\n<p>\u6240\u4ee5&#xff0c;Decoder \u7684\u7ed3\u6784\u4f1a\u6bd4 Encoder \u66f4\u590d\u6742\u4e00\u4e9b\u3002<\/p>\n<hr \/>\n<h2>\u5341\u4e8c\u3001Decoder Block \u4e3a\u4ec0\u4e48\u6709\u4e24\u4e2a Attention&#xff1f;<\/h2>\n<p>\u4e00\u4e2a\u5178\u578b\u7684 Decoder Block \u53ef\u4ee5\u7b80\u5316\u4e3a&#xff1a;<\/p>\n<p>Input<br \/>\n  \u2193<br \/>\nMasked Multi-Head Self-Attention<br \/>\n  \u2193<br \/>\nAdd &amp; Norm<br \/>\n  \u2193<br \/>\nEncoder-Decoder Attention<br \/>\n  \u2193<br \/>\nAdd &amp; Norm<br \/>\n  \u2193<br \/>\nFeed Forward Network<br \/>\n  \u2193<br \/>\nAdd &amp; Norm<br \/>\n  \u2193<br \/>\nOutput<\/p>\n<p>\u4f60\u4f1a\u53d1\u73b0&#xff1a;<\/p>\n<p>Decoder \u91cc\u9762\u6709\u4e24\u4e2a Attention\u3002<\/p>\n<p>\u4e3a\u4ec0\u4e48&#xff1f;<\/p>\n<p>\u56e0\u4e3a\u5b83\u9700\u8981\u89e3\u51b3\u4e24\u4e2a\u4e0d\u540c\u7684\u95ee\u9898\u3002<\/p>\n<hr \/>\n<h3>12.1 \u7b2c\u4e00\u4e2a Attention&#xff1a;\u770b\u81ea\u5df1\u5df2\u7ecf\u751f\u6210\u7684\u5185\u5bb9<\/h3>\n<p>\u5047\u8bbe Decoder \u6b63\u5728\u751f\u6210&#xff1a;<\/p>\n<p>\u6211 \u559c\u6b22 \u4eba\u5de5\u667a\u80fd<\/p>\n<p>\u5f53\u5b83\u51c6\u5907\u9884\u6d4b\u4e0b\u4e00\u4e2a Token \u65f6&#xff0c;\u5b83\u9700\u8981\u53c2\u8003&#xff1a;<\/p>\n<p>\u6211<br \/>\n\u559c\u6b22<br \/>\n\u4eba\u5de5\u667a\u80fd<\/p>\n<p>\u8fd9\u4e9b\u5df2\u7ecf\u51fa\u73b0\u7684\u4fe1\u606f\u3002<\/p>\n<p>\u56e0\u6b64&#xff0c;\u7b2c\u4e00\u4e2a Attention \u8d1f\u8d23&#xff1a;<\/p>\n<p>\u8ba9 Decoder \u5185\u90e8\u7684 Token \u5f7c\u6b64\u5efa\u7acb\u5173\u7cfb\u3002<\/p>\n<p>\u4f46\u662f&#xff0c;\u8fd9\u91cc\u6709\u4e00\u4e2a\u975e\u5e38\u91cd\u8981\u7684\u9650\u5236\u3002<\/p>\n<p>\u5047\u8bbe\u6211\u4eec\u6b63\u5728\u9884\u6d4b&#xff1a;<\/p>\n<p>Token 3<\/p>\n<p>\u90a3\u4e48\u6a21\u578b\u4e0d\u80fd\u63d0\u524d\u5077\u770b&#xff1a;<\/p>\n<p>Token 4<br \/>\nToken 5<br \/>\nToken 6<\/p>\n<p>\u5426\u5219\u8bad\u7ec3\u548c\u5b9e\u9645\u751f\u6210\u65f6\u5c31\u4f1a\u51fa\u73b0\u95ee\u9898\u3002<\/p>\n<p>\u56e0\u6b64&#xff0c;Decoder \u4f7f\u7528\u7684\u662f&#xff1a;<\/p>\n<p>Masked Self-Attention\u3002<\/p>\n<p>\u6240\u8c13 Mask&#xff0c;\u53ef\u4ee5\u7b80\u5355\u7406\u89e3\u6210&#xff1a;<\/p>\n<p>\u628a\u672a\u6765\u7684\u4fe1\u606f\u906e\u4f4f\u3002<\/p>\n<p>\u4f8b\u5982&#xff1a;<\/p>\n<p>Token 1 \u53ef\u4ee5\u770b&#xff1a;<br \/>\nToken 1<\/p>\n<p>Token 2 \u53ef\u4ee5\u770b&#xff1a;<br \/>\nToken 1\u3001Token 2<\/p>\n<p>Token 3 \u53ef\u4ee5\u770b&#xff1a;<br \/>\nToken 1\u3001Token 2\u3001Token 3<\/p>\n<p>Token 4 \u53ef\u4ee5\u770b&#xff1a;<br \/>\nToken 1\u3001Token 2\u3001Token 3\u3001Token 4<\/p>\n<p>\u4f46\u662f&#xff1a;<\/p>\n<p>Token 2<\/p>\n<p>\u4e0d\u80fd\u5077\u770b&#xff1a;<\/p>\n<p>Token 3\u3001Token 4\u2026\u2026<\/p>\n<p>\u6240\u4ee5\u6ce8\u610f\u529b\u77e9\u9635\u4f1a\u5f62\u6210\u4e00\u79cd\u7c7b\u4f3c&#xff1a;<\/p>\n<p>\u2713 \u00d7 \u00d7 \u00d7<br \/>\n\u2713 \u2713 \u00d7 \u00d7<br \/>\n\u2713 \u2713 \u2713 \u00d7<br \/>\n\u2713 \u2713 \u2713 \u2713<\/p>\n<p>\u7684\u7ed3\u6784\u3002<\/p>\n<p>\u8fd9\u5c31\u662f\u6240\u8c13\u7684&#xff1a;<\/p>\n<p>Causal Mask&#xff0c;\u56e0\u679c\u63a9\u7801\u3002<\/p>\n<p>\u5b83\u4fdd\u8bc1&#xff1a;<\/p>\n<p>\u5f53\u524d Token \u53ea\u80fd\u770b\u5230\u81ea\u5df1\u4ee5\u53ca\u8fc7\u53bb\u7684\u4fe1\u606f&#xff0c;\u4e0d\u80fd\u770b\u5230\u672a\u6765\u3002<\/p>\n<p>\u8fd9\u4e5f\u662f\u540e\u9762\u7406\u89e3 GPT \u7684\u4e00\u4e2a\u975e\u5e38\u91cd\u8981\u7684\u57fa\u7840\u3002<\/p>\n<hr \/>\n<h3>12.2 \u7b2c\u4e8c\u4e2a Attention&#xff1a;\u53bb\u770b Encoder<\/h3>\n<p>Decoder \u53ea\u770b\u81ea\u5df1\u8fd8\u4e0d\u591f\u3002<\/p>\n<p>\u4f8b\u5982\u6211\u4eec\u73b0\u5728\u6b63\u5728\u8fdb\u884c\u7ffb\u8bd1&#xff1a;<\/p>\n<p>\u8f93\u5165&#xff1a;<\/p>\n<p>I love artificial intelligence.<\/p>\n<p>Decoder \u6b63\u5728\u751f\u6210&#xff1a;<\/p>\n<p>\u6211 \u559c\u6b22 \u2026<\/p>\n<p>\u90a3\u4e48\u5f53 Decoder \u60f3\u7ee7\u7eed\u751f\u6210\u65f6&#xff0c;\u5b83\u8fd8\u9700\u8981\u77e5\u9053&#xff1a;<\/p>\n<p>\u8f93\u5165\u7684\u82f1\u6587\u5230\u5e95\u662f\u4ec0\u4e48&#xff1f;<\/p>\n<p>\u6240\u4ee5&#xff0c;\u5b83\u9700\u8981\u53bb\u5173\u6ce8 Encoder \u7684\u8f93\u51fa\u3002<\/p>\n<p>\u4e8e\u662f\u51fa\u73b0\u7b2c\u4e8c\u4e2a Attention&#xff1a;<\/p>\n<p>Encoder-Decoder Attention\u3002<\/p>\n<p>\u6211\u4eec\u53ef\u4ee5\u7406\u89e3\u6210&#xff1a;<\/p>\n<p>Decoder \u5f53\u524d\u72b6\u6001<br \/>\n       \u2193<br \/>\n   \u201c\u6211\u73b0\u5728\u9700\u8981\u4ec0\u4e48\u4fe1\u606f&#xff1f;\u201d<br \/>\n       \u2193<br \/>\n       Query<\/p>\n<p>Encoder \u8f93\u51fa<br \/>\n       \u2193<br \/>\n   \u201c\u8f93\u5165\u4e2d\u6709\u54ea\u4e9b\u4fe1\u606f&#xff1f;\u201d<br \/>\n       \u2193<br \/>\n       Key \/ Value<\/p>\n<p>\u56e0\u6b64&#xff0c;\u8fd9\u4e2a Attention \u7684\u6838\u5fc3\u4f5c\u7528\u662f&#xff1a;<\/p>\n<p>\u8ba9 Decoder \u5728\u751f\u6210\u8f93\u51fa\u65f6&#xff0c;\u80fd\u591f\u53c2\u8003 Encoder \u5bf9\u8f93\u5165\u5e8f\u5217\u7684\u7406\u89e3\u3002<\/p>\n<p>\u4f8b\u5982&#xff1a;<\/p>\n<p>Encoder&#xff1a;<br \/>\n\u7406\u89e3\u82f1\u6587\u8f93\u5165<\/p>\n<p>Decoder&#xff1a;<br \/>\n\u751f\u6210\u4e2d\u6587\u8f93\u51fa<\/p>\n<p>Decoder \u2500\u2500\u2500\u2500\u2500\u2192 \u9700\u8981\u65f6\u5173\u6ce8 Encoder<\/p>\n<p>\u6240\u4ee5\u539f\u59cb Transformer \u7684 Decoder \u5b9e\u9645\u4e0a\u540c\u65f6\u5904\u7406&#xff1a;<\/p>\n<p>\u8fc7\u53bb\u751f\u6210\u7684\u4fe1\u606f<br \/>\n        &#043;<br \/>\n\u8f93\u5165\u5e8f\u5217\u7684\u4fe1\u606f<\/p>\n<p>\u8fd9\u4e5f\u662f\u4e3a\u4ec0\u4e48\u5b83\u9700\u8981\u4e24\u4e2a\u4e0d\u540c\u7684 Attention \u6a21\u5757\u3002<\/p>\n<hr \/>\n<h2>\u5341\u4e09\u3001Encoder \u548c Decoder \u5230\u5e95\u6709\u4ec0\u4e48\u533a\u522b&#xff1f;<\/h2>\n<p>\u73b0\u5728\u6211\u4eec\u53ef\u4ee5\u628a\u4e24\u8005\u653e\u5728\u4e00\u8d77\u6bd4\u8f83\u3002<\/p>\n<h3>Encoder<\/h3>\n<p>Input<br \/>\n \u2193<br \/>\nSelf-Attention<br \/>\n \u2193<br \/>\nAdd &amp; Norm<br \/>\n \u2193<br \/>\nFFN<br \/>\n \u2193<br \/>\nAdd &amp; Norm<\/p>\n<p>\u6838\u5fc3\u76ee\u6807&#xff1a;<\/p>\n<p>\u7406\u89e3\u6574\u4e2a\u8f93\u5165\u5e8f\u5217\u3002<\/p>\n<p>\u56e0\u4e3a Encoder \u5728\u5904\u7406\u8f93\u5165\u65f6&#xff0c;\u901a\u5e38\u53ef\u4ee5\u540c\u65f6\u770b\u5230\u6574\u4e2a\u8f93\u5165&#xff1a;<\/p>\n<p>Token 1<br \/>\nToken 2<br \/>\nToken 3<br \/>\n&#8230;<br \/>\nToken N<\/p>\n<p>\u56e0\u6b64 Encoder \u7684 Self-Attention \u53ef\u4ee5\u5efa\u7acb&#xff1a;<\/p>\n<p>\u4efb\u610f Token \u2194 \u4efb\u610f Token<\/p>\n<p>\u4e4b\u95f4\u7684\u5173\u7cfb\u3002<\/p>\n<hr \/>\n<h3>Decoder<\/h3>\n<p>Output<br \/>\n \u2193<br \/>\nMasked Self-Attention<br \/>\n \u2193<br \/>\nAdd &amp; Norm<br \/>\n \u2193<br \/>\nEncoder-Decoder Attention<br \/>\n \u2193<br \/>\nAdd &amp; Norm<br \/>\n \u2193<br \/>\nFFN<br \/>\n \u2193<br \/>\nAdd &amp; Norm<\/p>\n<p>\u6838\u5fc3\u76ee\u6807&#xff1a;<\/p>\n<p>\u6839\u636e\u5df2\u6709\u4fe1\u606f\u9010\u6b65\u751f\u6210\u8f93\u51fa\u3002<\/p>\n<p>\u56e0\u6b64 Decoder&#xff1a;<\/p>\n<li>\u9700\u8981\u5173\u6ce8\u5df2\u7ecf\u751f\u6210\u7684 Token&#xff1b;<\/li>\n<li>\u4e0d\u80fd\u5077\u770b\u672a\u6765 Token&#xff1b;<\/li>\n<li>\u8fd8\u9700\u8981\u5173\u6ce8 Encoder \u8f93\u51fa\u3002<\/li>\n<p>\u6240\u4ee5 Decoder \u6bd4 Encoder \u591a\u4e86&#xff1a;<\/p>\n<p>Masked Self-Attention<\/p>\n<p>\u548c&#xff1a;<\/p>\n<p>Encoder-Decoder Attention<\/p>\n<p>\u8fd9\u6837\u7684\u7ed3\u6784\u3002<\/p>\n<hr \/>\n<h2>\u5341\u56db\u3001Transformer \u5230\u5e95\u662f\u5982\u4f55\u5b8c\u6210\u4e00\u6b21\u673a\u5668\u7ffb\u8bd1\u7684&#xff1f;<\/h2>\n<p>\u73b0\u5728\u6211\u4eec\u628a\u6574\u4e2a\u8fc7\u7a0b\u4e32\u8d77\u6765\u3002<\/p>\n<p>\u5047\u8bbe&#xff1a;<\/p>\n<p>\u8f93\u5165&#xff1a;<\/p>\n<p>I love artificial intelligence.<\/p>\n<p>\u76ee\u6807&#xff1a;<\/p>\n<p>\u6211\u559c\u6b22\u4eba\u5de5\u667a\u80fd\u3002<\/p>\n<p>\u6574\u4e2a\u6d41\u7a0b\u53ef\u4ee5\u7b80\u5355\u7406\u89e3\u6210&#xff1a;<\/p>\n<p>\u82f1\u6587\u8f93\u5165<br \/>\n   \u2193<br \/>\nTokenization<br \/>\n   \u2193<br \/>\nToken Embedding<br \/>\n   \u2193<br \/>\nPositional Encoding<br \/>\n   \u2193<br \/>\nEncoder<br \/>\n   \u2193<br \/>\nEncoder Output<br \/>\n   \u2193<br \/>\nDecoder<br \/>\n   \u2193<br \/>\n\u9884\u6d4b\u201c\u6211\u201d<br \/>\n   \u2193<br \/>\nDecoder<br \/>\n   \u2193<br \/>\n\u9884\u6d4b\u201c\u559c\u6b22\u201d<br \/>\n   \u2193<br \/>\nDecoder<br \/>\n   \u2193<br \/>\n\u9884\u6d4b\u201c\u4eba\u5de5\u667a\u80fd\u201d<br \/>\n   \u2193<br \/>\nDecoder<br \/>\n   \u2193<br \/>\n\u7ed3\u675f<\/p>\n<p>\u5c55\u5f00\u4e00\u70b9&#xff1a;<\/p>\n<p>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n\u2502          Encoder             \u2502<br \/>\n\u2502                              \u2502<br \/>\n\u2502 I love artificial intelligence \u2502<br \/>\n\u2502              \u2193               \u2502<br \/>\n\u2502      \u4e0a\u4e0b\u6587\u4fe1\u606f\u7406\u89e3           \u2502<br \/>\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<br \/>\n               \u2502<br \/>\n               \u2502 Encoder Output<br \/>\n               \u2193<br \/>\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n\u2502          Decoder             \u2502<br \/>\n\u2502                              \u2502<br \/>\n\u2502 &lt;START&gt;                      \u2502<br \/>\n\u2502     \u2193                        \u2502<br \/>\n\u2502     \u6211                       \u2502<br \/>\n\u2502     \u2193                        \u2502<br \/>\n\u2502     \u6211 \u559c\u6b22                  \u2502<br \/>\n\u2502     \u2193                        \u2502<br \/>\n\u2502     \u6211 \u559c\u6b22 \u4eba\u5de5\u667a\u80fd          \u2502<br \/>\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n<p>\u8fd9\u5c31\u662f\u539f\u59cb Transformer Encoder-Decoder \u67b6\u6784\u7684\u6574\u4f53\u5de5\u4f5c\u65b9\u5f0f\u3002<\/p>\n<hr \/>\n<h2>\u5341\u4e94\u3001\u7b49\u7b49&#xff0c;\u8fd8\u6709\u4e00\u4e2a\u91cd\u8981\u95ee\u9898&#xff1a;Transformer \u600e\u4e48\u77e5\u9053\u987a\u5e8f&#xff1f;<\/h2>\n<p>\u73b0\u5728\u6211\u4eec\u5df2\u7ecf\u77e5\u9053&#xff1a;<\/p>\n<p>Attention<\/p>\n<p>\u53ef\u4ee5\u8ba9\u4e0d\u540c Token \u5efa\u7acb\u8054\u7cfb\u3002<\/p>\n<p>\u4f46\u8fd9\u91cc\u8fd8\u6709\u4e00\u4e2a\u95ee\u9898\u3002<\/p>\n<p>\u5047\u8bbe\u8f93\u5165&#xff1a;<\/p>\n<p>\u6211 \u559c\u6b22 \u4f60<\/p>\n<p>\u548c&#xff1a;<\/p>\n<p>\u4f60 \u559c\u6b22 \u6211<\/p>\n<p>\u8fd9\u4e24\u53e5\u8bdd\u4f7f\u7528\u7684 Token \u5b8c\u5168\u4e00\u6837&#xff1a;<\/p>\n<p>\u6211<br \/>\n\u559c\u6b22<br \/>\n\u4f60<\/p>\n<p>\u4f46\u662f\u987a\u5e8f\u4e0d\u540c&#xff0c;\u542b\u4e49\u4e5f\u5b8c\u5168\u4e0d\u540c\u3002<\/p>\n<p>\u7136\u800c Attention \u672c\u8eab\u53ea\u662f\u5728\u8ba1\u7b97 Token \u4e4b\u95f4\u7684\u5173\u7cfb\u3002<\/p>\n<p>\u90a3\u4e48&#xff1a;<\/p>\n<p>Transformer \u600e\u4e48\u77e5\u9053\u54ea\u4e2a Token \u5728\u524d&#xff0c;\u54ea\u4e2a Token \u5728\u540e&#xff1f;<\/p>\n<p>\u8fd9\u5c31\u662f&#xff1a;<\/p>\n<p>Positional Encoding&#xff0c;\u4f4d\u7f6e\u7f16\u7801\u3002<\/p>\n<p>Transformer \u4e0d\u50cf RNN\u3002<\/p>\n<p>RNN \u5929\u7136\u6309\u7167&#xff1a;<\/p>\n<p>\u7b2c\u4e00\u4e2a<br \/>\n\u2193<br \/>\n\u7b2c\u4e8c\u4e2a<br \/>\n\u2193<br \/>\n\u7b2c\u4e09\u4e2a<\/p>\n<p>\u7684\u987a\u5e8f\u8fdb\u884c\u8ba1\u7b97\u3002<\/p>\n<p>\u56e0\u6b64&#xff1a;<\/p>\n<p>\u987a\u5e8f\u672c\u8eab\u5c31\u5305\u542b\u5728\u8ba1\u7b97\u8fc7\u7a0b\u4e2d\u3002<\/p>\n<p>\u4f46\u662f Transformer \u53ef\u4ee5\u5e76\u884c\u5904\u7406&#xff1a;<\/p>\n<p>Token 1<br \/>\nToken 2<br \/>\nToken 3<br \/>\nToken 4<\/p>\n<p>\u8fd9\u610f\u5473\u7740&#xff1a;<\/p>\n<p>\u6a21\u578b\u672c\u8eab\u5e76\u4e0d\u77e5\u9053\u8fd9\u4e9b Token \u7684\u5148\u540e\u987a\u5e8f\u3002<\/p>\n<p>\u6240\u4ee5\u5fc5\u987b\u989d\u5916\u628a\u4f4d\u7f6e\u4fe1\u606f\u52a0\u5165\u8fdb\u53bb&#xff1a;<\/p>\n<p>Token Embedding<br \/>\n       &#043;<br \/>\nPosition Information<br \/>\n       \u2193<br \/>\nTransformer Input<\/p>\n<p>\u539f\u59cb Transformer \u4f7f\u7528\u7684\u662f&#xff1a;<\/p>\n<p>Sinusoidal Positional Encoding&#xff0c;\u6b63\u5f26\u4f59\u5f26\u4f4d\u7f6e\u7f16\u7801\u3002<\/p>\n<p>\u4f4d\u7f6e\u7f16\u7801\u516c\u5f0f\u5982\u4e0b&#xff1a;<\/p>\n<p><span class=\"katex--display\"><span class=\"katex-display\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>         P<\/p>\n<p>         E<\/p>\n<p>         (<\/p>\n<p>         p<\/p>\n<p>         o<\/p>\n<p>         s<\/p>\n<p>         ,<\/p>\n<p>         2<\/p>\n<p>         i<\/p>\n<p>         )<\/p>\n<p>         &#061;<\/p>\n<p>         sin<\/p>\n<p>         \u2061<\/p>\n<p>          (<\/p>\n<p>            p<\/p>\n<p>            o<\/p>\n<p>            s<\/p>\n<p>            10000<\/p>\n<p>              2<\/p>\n<p>              i<\/p>\n<p>              d<\/p>\n<p>              model<\/p>\n<p>          )<\/p>\n<p>         PE(pos, 2i) &#061; \\\\sin\\\\left(\\\\frac{pos}{10000^{\\\\frac{2i}{d_{\\\\text{model}}}}}\\\\right) <\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 1em;vertical-align: -0.25em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.1389em\">P<\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.0576em\">E<\/span><span class=\"mopen\">(<\/span><span class=\"mord mathnormal\">p<\/span><span class=\"mord mathnormal\">os<\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.1667em\"><\/span><span class=\"mord\">2<\/span><span class=\"mord mathnormal\">i<\/span><span class=\"mclose\">)<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><span class=\"mrel\">&#061;<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 2.4619em;vertical-align: -1.0119em\"><\/span><span class=\"mop\">sin<\/span><span class=\"mspace\" style=\"margin-right: 0.1667em\"><\/span><span class=\"minner\"><span class=\"mopen delimcenter\" style=\"top: 0em\"><span class=\"delimsizing size3\">(<\/span><\/span><span class=\"mord\"><span class=\"mopen nulldelimiter\"><\/span><span class=\"mfrac\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.1076em\"><span class=\"\" style=\"top: -2.11em\"><span class=\"pstrut\" style=\"height: 3.1219em\"><\/span><span class=\"mord\"><span class=\"mord\">1000<\/span><span class=\"mord\"><span class=\"mord\">0<\/span><span class=\"msupsub\"><span class=\"vlist-t\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.1219em\"><span class=\"\" style=\"top: -3.5234em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\"><span class=\"mord mtight\"><span class=\"mopen nulldelimiter sizing reset-size3 size6\"><\/span><span class=\"mfrac\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.8551em\"><span class=\"\" style=\"top: -2.656em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"sizing reset-size3 size1 mtight\"><span class=\"mord mtight\"><span class=\"mord mtight\"><span class=\"mord mathnormal mtight\">d<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3448em\"><span class=\"\" style=\"top: -2.3448em;margin-left: 0em;margin-right: 0.1em\"><span class=\"pstrut\" style=\"height: 2.6944em\"><\/span><span class=\"mord mtight\"><span class=\"mord text mtight\"><span class=\"mord mtight\">model<\/span><\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3496em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"\" style=\"top: -3.2255em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"frac-line mtight\" style=\"border-bottom-width: 0.049em\"><\/span><\/span><span class=\"\" style=\"top: -3.384em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"sizing reset-size3 size1 mtight\"><span class=\"mord mtight\"><span class=\"mord mtight\">2<\/span><span class=\"mord mathnormal mtight\">i<\/span><\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.5937em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><span class=\"mclose nulldelimiter sizing reset-size3 size6\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"\" style=\"top: -3.3519em\"><span class=\"pstrut\" style=\"height: 3.1219em\"><\/span><span class=\"frac-line\" style=\"border-bottom-width: 0.04em\"><\/span><\/span><span class=\"\" style=\"top: -3.7989em\"><span class=\"pstrut\" style=\"height: 3.1219em\"><\/span><span class=\"mord\"><span class=\"mord mathnormal\">p<\/span><span class=\"mord mathnormal\">os<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.0119em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><span class=\"mclose nulldelimiter\"><\/span><\/span><span class=\"mclose delimcenter\" style=\"top: 0em\"><span class=\"delimsizing size3\">)<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<p><span class=\"katex--display\"><span class=\"katex-display\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>         P<\/p>\n<p>         E<\/p>\n<p>         (<\/p>\n<p>         p<\/p>\n<p>         o<\/p>\n<p>         s<\/p>\n<p>         ,<\/p>\n<p>         2<\/p>\n<p>         i<\/p>\n<p>         &#043;<\/p>\n<p>         1<\/p>\n<p>         )<\/p>\n<p>         &#061;<\/p>\n<p>         cos<\/p>\n<p>         \u2061<\/p>\n<p>          (<\/p>\n<p>            p<\/p>\n<p>            o<\/p>\n<p>            s<\/p>\n<p>            10000<\/p>\n<p>              2<\/p>\n<p>              i<\/p>\n<p>              d<\/p>\n<p>              model<\/p>\n<p>          )<\/p>\n<p>         PE(pos, 2i&#043;1) &#061; \\\\cos\\\\left(\\\\frac{pos}{10000^{\\\\frac{2i}{d_{\\\\text{model}}}}}\\\\right) <\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 1em;vertical-align: -0.25em\"><\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.1389em\">P<\/span><span class=\"mord mathnormal\" style=\"margin-right: 0.0576em\">E<\/span><span class=\"mopen\">(<\/span><span class=\"mord mathnormal\">p<\/span><span class=\"mord mathnormal\">os<\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.1667em\"><\/span><span class=\"mord\">2<\/span><span class=\"mord mathnormal\">i<\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><span class=\"mbin\">&#043;<\/span><span class=\"mspace\" style=\"margin-right: 0.2222em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1em;vertical-align: -0.25em\"><\/span><span class=\"mord\">1<\/span><span class=\"mclose\">)<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><span class=\"mrel\">&#061;<\/span><span class=\"mspace\" style=\"margin-right: 0.2778em\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 2.4619em;vertical-align: -1.0119em\"><\/span><span class=\"mop\">cos<\/span><span class=\"mspace\" style=\"margin-right: 0.1667em\"><\/span><span class=\"minner\"><span class=\"mopen delimcenter\" style=\"top: 0em\"><span class=\"delimsizing size3\">(<\/span><\/span><span class=\"mord\"><span class=\"mopen nulldelimiter\"><\/span><span class=\"mfrac\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.1076em\"><span class=\"\" style=\"top: -2.11em\"><span class=\"pstrut\" style=\"height: 3.1219em\"><\/span><span class=\"mord\"><span class=\"mord\">1000<\/span><span class=\"mord\"><span class=\"mord\">0<\/span><span class=\"msupsub\"><span class=\"vlist-t\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.1219em\"><span class=\"\" style=\"top: -3.5234em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\"><span class=\"mord mtight\"><span class=\"mopen nulldelimiter sizing reset-size3 size6\"><\/span><span class=\"mfrac\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.8551em\"><span class=\"\" style=\"top: -2.656em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"sizing reset-size3 size1 mtight\"><span class=\"mord mtight\"><span class=\"mord mtight\"><span class=\"mord mathnormal mtight\">d<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3448em\"><span class=\"\" style=\"top: -2.3448em;margin-left: 0em;margin-right: 0.1em\"><span class=\"pstrut\" style=\"height: 2.6944em\"><\/span><span class=\"mord mtight\"><span class=\"mord text mtight\"><span class=\"mord mtight\">model<\/span><\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3496em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"\" style=\"top: -3.2255em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"frac-line mtight\" style=\"border-bottom-width: 0.049em\"><\/span><\/span><span class=\"\" style=\"top: -3.384em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"sizing reset-size3 size1 mtight\"><span class=\"mord mtight\"><span class=\"mord mtight\">2<\/span><span class=\"mord mathnormal mtight\">i<\/span><\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.5937em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><span class=\"mclose nulldelimiter sizing reset-size3 size6\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"\" style=\"top: -3.3519em\"><span class=\"pstrut\" style=\"height: 3.1219em\"><\/span><span class=\"frac-line\" style=\"border-bottom-width: 0.04em\"><\/span><\/span><span class=\"\" style=\"top: -3.7989em\"><span class=\"pstrut\" style=\"height: 3.1219em\"><\/span><span class=\"mord\"><span class=\"mord mathnormal\">p<\/span><span class=\"mord mathnormal\">os<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.0119em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><span class=\"mclose nulldelimiter\"><\/span><\/span><span class=\"mclose delimcenter\" style=\"top: 0em\"><span class=\"delimsizing size3\">)<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<p>\u5176\u4e2d <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>        p<\/p>\n<p>        o<\/p>\n<p>        s<\/p>\n<p>       pos<\/p>\n<p>    <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.625em;vertical-align: -0.1944em\"><\/span><span class=\"mord mathnormal\">p<\/span><span class=\"mord mathnormal\">os<\/span><\/span><\/span><\/span><\/span> \u4e3a\u4f4d\u7f6e\u7d22\u5f15&#xff0c;<span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>        i<\/p>\n<p>       i<\/p>\n<p>    <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.6595em\"><\/span><span class=\"mord mathnormal\">i<\/span><\/span><\/span><\/span><\/span> \u4e3a\u7ef4\u5ea6\u7d22\u5f15&#xff0c;<span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>         d<\/p>\n<p>         model<\/p>\n<p>       d_{\\\\text{model}}<\/p>\n<p>    <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.8444em;vertical-align: -0.15em\"><\/span><span class=\"mord\"><span class=\"mord mathnormal\">d<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3361em\"><span class=\"\" style=\"top: -2.55em;margin-left: 0em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\"><span class=\"mord text mtight\"><span class=\"mord mtight\">model<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> \u4e3a\u6a21\u578b\u7ef4\u5ea6\u3002<\/p>\n<p>\u73b0\u5728\u4e0d\u9700\u8981\u5f3a\u884c\u7406\u89e3&#xff1a;<\/p>\n<p>\u4e3a\u4ec0\u4e48\u662f 10000&#xff1f;<\/p>\n<p>\u4e3a\u4ec0\u4e48\u4f7f\u7528 sin&#xff1f;<\/p>\n<p>\u4e3a\u4ec0\u4e48\u4f7f\u7528 cos&#xff1f;<\/p>\n<p>\u8fd9\u4e00\u7bc7\u53ea\u9700\u8981\u5148\u5efa\u7acb\u4e00\u4e2a\u6982\u5ff5&#xff1a;<\/p>\n<p>Attention \u672c\u8eab\u4e0d\u5305\u542b\u987a\u5e8f\u4fe1\u606f&#xff0c;\u6240\u4ee5 Transformer \u5fc5\u987b\u989d\u5916\u544a\u8bc9\u6a21\u578b\u6bcf\u4e2a Token \u7684\u4f4d\u7f6e\u3002<\/p>\n<p>\u540e\u9762\u6211\u4eec\u4f1a\u4e13\u95e8\u8ba8\u8bba Position Encoding\u3002<\/p>\n<hr \/>\n<h2>\u5341\u516d\u3001Transformer \u7684\u5b8c\u6574\u6570\u636e\u6d41<\/h2>\n<p>\u5230\u8fd9\u91cc&#xff0c;\u6211\u4eec\u5df2\u7ecf\u53ef\u4ee5\u628a\u539f\u59cb Transformer \u7684\u6574\u4f53\u6d41\u7a0b\u4e32\u8d77\u6765\u4e86\u3002<\/p>\n<p>                    \u8f93\u5165\u5e8f\u5217<br \/>\n                        \u2193<br \/>\n                  Token Embedding<br \/>\n                        \u2193<br \/>\n                Positional Encoding<br \/>\n                        \u2193<br \/>\n              \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n              \u2502 Encoder Block  \u2502<br \/>\n              \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<br \/>\n                        \u2193<br \/>\n              \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n              \u2502 Encoder Block  \u2502<br \/>\n              \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<br \/>\n                        \u2193<br \/>\n                       &#8230;<br \/>\n                        \u2193<br \/>\n                   Encoder Output<br \/>\n                        \u2502<br \/>\n                        \u2502<br \/>\n                        \u2193<br \/>\n\u8f93\u51fa\u5e8f\u5217 \u2192 Token Embedding<br \/>\n                        \u2193<br \/>\n                Positional Encoding<br \/>\n                        \u2193<br \/>\n              Masked Self-Attention<br \/>\n                        \u2193<br \/>\n                   Add &amp; Norm<br \/>\n                        \u2193<br \/>\n             Encoder-Decoder Attention<br \/>\n                        \u2193<br \/>\n                   Add &amp; Norm<br \/>\n                        \u2193<br \/>\n               Feed Forward Network<br \/>\n                        \u2193<br \/>\n                   Add &amp; Norm<br \/>\n                        \u2193<br \/>\n                     Linear<br \/>\n                        \u2193<br \/>\n                     Softmax<br \/>\n                        \u2193<br \/>\n                  \u4e0b\u4e00\u4e2a Token<\/p>\n<p>\u4ece\u804c\u8d23\u4e0a&#xff0c;\u6211\u4eec\u4e5f\u53ef\u4ee5\u628a Transformer \u7b80\u5316\u6210&#xff1a;<\/p>\n<p>Embedding<br \/>\n    \u2193<br \/>\n\u628a Token \u8f6c\u6362\u6210\u5411\u91cf<\/p>\n<p>Positional Encoding<br \/>\n    \u2193<br \/>\n\u544a\u8bc9\u6a21\u578b Token \u7684\u4f4d\u7f6e<\/p>\n<p>Attention<br \/>\n    \u2193<br \/>\n\u5efa\u7acb Token \u4e4b\u95f4\u7684\u5173\u7cfb<\/p>\n<p>FFN<br \/>\n    \u2193<br \/>\n\u8fdb\u4e00\u6b65\u5904\u7406 Token \u8868\u793a<\/p>\n<p>Residual &#043; LayerNorm<br \/>\n    \u2193<br \/>\n\u5e2e\u52a9\u6df1\u5c42\u7f51\u7edc\u7a33\u5b9a\u8bad\u7ec3<\/p>\n<p>Encoder<br \/>\n    \u2193<br \/>\n\u7406\u89e3\u8f93\u5165<\/p>\n<p>Decoder<br \/>\n    \u2193<br \/>\n\u751f\u6210\u8f93\u51fa<\/p>\n<hr \/>\n<h2>\u5341\u4e03\u3001\u4e00\u4e2a\u5bb9\u6613\u4ea7\u751f\u7684\u8bef\u89e3&#xff1a;\u73b0\u5728\u7684 GPT \u8fd8\u662f\u5b8c\u6574 Transformer \u5417&#xff1f;<\/h2>\n<p>\u770b\u5230\u8fd9\u91cc&#xff0c;\u53ef\u80fd\u4f1a\u6709\u4e00\u4e2a\u95ee\u9898&#xff1a;<\/p>\n<p>\u6211\u4eec\u73b0\u5728\u4f7f\u7528\u7684\u5927\u8bed\u8a00\u6a21\u578b&#xff0c;\u4f8b\u5982 GPT&#xff0c;\u771f\u7684\u8fd8\u662f\u4e0a\u9762\u8fd9\u79cd\u5b8c\u6574\u7684 Encoder-Decoder Transformer \u5417&#xff1f;<\/p>\n<p>\u7b54\u6848\u662f&#xff1a;<\/p>\n<p>\u4e0d\u5b8c\u5168\u662f\u3002<\/p>\n<p>Transformer \u6700\u521d\u662f\u4e00\u79cd\u5b8c\u6574\u7684 Encoder-Decoder \u67b6\u6784\u3002<\/p>\n<p>\u4f46\u540e\u6765\u7814\u7a76\u4eba\u5458\u53d1\u73b0&#xff0c;\u4e0d\u540c\u4efb\u52a1\u53ef\u4ee5\u4f7f\u7528 Transformer \u7684\u4e0d\u540c\u90e8\u5206\u3002<\/p>\n<p>\u4e8e\u662f\u9010\u6e10\u51fa\u73b0\u4e86\u4e09\u79cd\u975e\u5e38\u91cd\u8981\u7684\u67b6\u6784&#xff1a;<\/p>\n<p>Encoder-only<br \/>\nDecoder-only<br \/>\nEncoder-Decoder<\/p>\n<p>\u4f8b\u5982&#xff1a;<\/p>\n<h4>Encoder-only<\/h4>\n<p>\u5178\u578b\u4ee3\u8868&#xff1a;<\/p>\n<p>BERT<\/p>\n<p>\u4e3b\u8981\u4f7f\u7528 Transformer Encoder\u3002<\/p>\n<p>\u66f4\u64c5\u957f&#xff1a;<\/p>\n<p>\u6587\u672c\u7406\u89e3<br \/>\n\u6587\u672c\u5206\u7c7b<br \/>\n\u547d\u540d\u5b9e\u4f53\u8bc6\u522b<\/p>\n<hr \/>\n<h4>Decoder-only<\/h4>\n<p>\u5178\u578b\u4ee3\u8868&#xff1a;<\/p>\n<p>GPT<\/p>\n<p>\u4e3b\u8981\u4f7f\u7528 Transformer Decoder \u4e2d\u9002\u5408\u81ea\u56de\u5f52\u751f\u6210\u7684\u7ed3\u6784\u3002<\/p>\n<p>\u66f4\u64c5\u957f&#xff1a;<\/p>\n<p>\u6587\u672c\u751f\u6210<br \/>\n\u5bf9\u8bdd<br \/>\n\u4ee3\u7801\u751f\u6210<\/p>\n<p>\u4eca\u5929\u7684\u5927\u8bed\u8a00\u6a21\u578b&#xff0c;\u5927\u91cf\u91c7\u7528\u7684\u5c31\u662f\u8fd9\u79cd&#xff1a;<\/p>\n<p>Decoder-only Transformer\u3002<\/p>\n<hr \/>\n<h4>Encoder-Decoder<\/h4>\n<p>\u5178\u578b\u4ee3\u8868&#xff1a;<\/p>\n<p>T5<\/p>\n<p>\u66f4\u9002\u5408&#xff1a;<\/p>\n<p>\u673a\u5668\u7ffb\u8bd1<br \/>\n\u6587\u672c\u6458\u8981<br \/>\n\u8f93\u5165\u5e8f\u5217 \u2192 \u8f93\u51fa\u5e8f\u5217<\/p>\n<p>\u6240\u4ee5&#xff1a;<\/p>\n<p>Transformer \u5e76\u4e0d\u662f\u53ea\u6709\u4e00\u79cd\u56fa\u5b9a\u5f62\u6001\u3002<\/p>\n<p>\u5b83\u66f4\u50cf\u662f\u4e00\u5957\u975e\u5e38\u6210\u529f\u7684\u57fa\u7840\u67b6\u6784\u601d\u60f3\u3002<\/p>\n<p>\u4e0d\u540c\u6a21\u578b\u53ef\u4ee5\u6839\u636e\u4efb\u52a1&#xff1a;<\/p>\n<p>\u53ea\u4f7f\u7528 Encoder<\/p>\n<p>\u53ea\u4f7f\u7528 Decoder<\/p>\n<p>\u6216\u8005\u540c\u65f6\u4f7f\u7528 Encoder &#043; Decoder<\/p>\n<p>\u8fd9\u4e5f\u662f Transformer \u540e\u6765\u80fd\u591f\u5e7f\u6cdb\u53d1\u5c55\u7684\u4e00\u4e2a\u91cd\u8981\u539f\u56e0\u3002<\/p>\n<hr \/>\n<h2>\u5341\u516b\u3001\u73b0\u5728&#xff0c;\u6211\u4eec\u5bf9 Transformer \u5e94\u8be5\u6709\u4ec0\u4e48\u6574\u4f53\u8ba4\u8bc6&#xff1f;<\/h2>\n<p>\u7ecf\u8fc7\u8fd9\u4e00\u7bc7\u6587\u7ae0&#xff0c;\u6211\u4eec\u6682\u65f6\u8fd8\u6ca1\u6709\u6df1\u5165&#xff1a;<\/p>\n<p>Q\u3001K\u3001V<\/p>\n<p>\u4e5f\u6ca1\u6709\u771f\u6b63\u8ba1\u7b97&#xff1a;<\/p>\n<p>Attention Score<\/p>\n<p>\u4f46\u73b0\u5728&#xff0c;\u6211\u4eec\u81f3\u5c11\u5e94\u8be5\u5efa\u7acb\u8d77\u8fd9\u6837\u4e00\u5f20\u5730\u56fe&#xff1a;<\/p>\n<p>                     Transformer<br \/>\n                          \u2502<br \/>\n          \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n          \u2502                               \u2502<br \/>\n          \u2193                               \u2193<br \/>\n       Encoder                         Decoder<br \/>\n          \u2502                               \u2502<br \/>\n          \u2502                         Masked Self-Attention<br \/>\n          \u2502                               \u2502<br \/>\n    Self-Attention                         \u2193<br \/>\n          \u2502                       Encoder-Decoder Attention<br \/>\n          \u2193                               \u2502<br \/>\n      Add &amp; Norm                           \u2193<br \/>\n          \u2193                              FFN<br \/>\n         FFN                              \u2502<br \/>\n          \u2193                               \u2193<br \/>\n      Add &amp; Norm                       Add &amp; Norm<\/p>\n<p>\u5e76\u4e14\u77e5\u9053\u6bcf\u4e2a\u6a21\u5757\u7684\u5927\u81f4\u804c\u8d23&#xff1a;<\/p>\n<table>\n<tr>\u6a21\u5757\u6838\u5fc3\u4f5c\u7528<\/tr>\n<tbody>\n<tr>\n<td>Embedding<\/td>\n<td>\u5c06 Token \u8f6c\u6362\u4e3a\u5411\u91cf<\/td>\n<\/tr>\n<tr>\n<td>Positional Encoding<\/td>\n<td>\u63d0\u4f9b\u4f4d\u7f6e\u4fe1\u606f<\/td>\n<\/tr>\n<tr>\n<td>Self-Attention<\/td>\n<td>\u5efa\u7acb Token \u4e4b\u95f4\u7684\u5173\u7cfb<\/td>\n<\/tr>\n<tr>\n<td>Multi-Head Attention<\/td>\n<td>\u4ece\u591a\u4e2a\u89d2\u5ea6\u5efa\u7acb\u5173\u7cfb<\/td>\n<\/tr>\n<tr>\n<td>FFN<\/td>\n<td>\u5bf9 Token \u8868\u793a\u8fdb\u884c\u975e\u7ebf\u6027\u53d8\u6362<\/td>\n<\/tr>\n<tr>\n<td>Residual Connection<\/td>\n<td>\u4fdd\u7559\u4fe1\u606f&#xff0c;\u5e2e\u52a9\u6df1\u5c42\u8bad\u7ec3<\/td>\n<\/tr>\n<tr>\n<td>LayerNorm<\/td>\n<td>\u7a33\u5b9a\u6570\u636e\u5206\u5e03<\/td>\n<\/tr>\n<tr>\n<td>Encoder<\/td>\n<td>\u7406\u89e3\u8f93\u5165<\/td>\n<\/tr>\n<tr>\n<td>Decoder<\/td>\n<td>\u9010\u6b65\u751f\u6210\u8f93\u51fa<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>\u5982\u679c\u80fd\u591f\u5efa\u7acb\u8d77\u8fd9\u5f20\u5730\u56fe&#xff0c;\u90a3\u4e48\u540e\u9762\u5b66\u4e60 Transformer \u7684\u7ec6\u8282\u5c31\u4f1a\u8f7b\u677e\u5f88\u591a\u3002<\/p>\n<hr \/>\n<h2>\u5341\u4e5d\u3001\u5c0f\u7ed3<\/h2>\n<p>\u8fd9\u4e00\u7bc7&#xff0c;\u6211\u4eec\u4e3b\u8981\u5b8c\u6210\u4e86\u4e00\u4ef6\u4e8b\u60c5&#xff1a;<\/p>\n<p>\u4ece\u6574\u4f53\u4e0a\u8ba4\u8bc6 Transformer \u5230\u5e95\u662f\u4ec0\u4e48\u3002<\/p>\n<p>Transformer \u6700\u521d\u91c7\u7528\u7684\u662f&#xff1a;<\/p>\n<p>Encoder-Decoder \u67b6\u6784\u3002<\/p>\n<p>\u5176\u4e2d&#xff1a;<\/p>\n<p>Encoder<\/p>\n<p>\u8d1f\u8d23&#xff1a;<\/p>\n<p>\u7406\u89e3\u8f93\u5165\u5e8f\u5217&#xff0c;\u5e76\u751f\u6210\u5305\u542b\u4e0a\u4e0b\u6587\u4fe1\u606f\u7684\u8868\u793a\u3002<\/p>\n<p>\u800c&#xff1a;<\/p>\n<p>Decoder<\/p>\n<p>\u8d1f\u8d23&#xff1a;<\/p>\n<p>\u5728\u53c2\u8003\u5df2\u6709\u8f93\u51fa\u548c Encoder \u4fe1\u606f\u7684\u57fa\u7840\u4e0a&#xff0c;\u9010\u6b65\u751f\u6210\u65b0\u7684 Token\u3002<\/p>\n<p>\u4e00\u4e2a\u5178\u578b\u7684 Encoder Block \u5305\u542b&#xff1a;<\/p>\n<p>Multi-Head Self-Attention<br \/>\n        \u2193<br \/>\nAdd &amp; Norm<br \/>\n        \u2193<br \/>\nFeed Forward Network<br \/>\n        \u2193<br \/>\nAdd &amp; Norm<\/p>\n<p>\u800c Decoder Block \u5219\u6bd4 Encoder \u591a\u51fa\u4e86&#xff1a;<\/p>\n<p>Masked Self-Attention<\/p>\n<p>\u4ee5\u53ca&#xff1a;<\/p>\n<p>Encoder-Decoder Attention<\/p>\n<p>\u540c\u65f6&#xff0c;\u56e0\u4e3a Transformer \u4e0d\u50cf RNN \u4e00\u6837\u5929\u7136\u6309\u7167\u987a\u5e8f\u5904\u7406\u6570\u636e&#xff0c;\u6240\u4ee5\u8fd8\u9700\u8981&#xff1a;<\/p>\n<p>Positional Encoding\u3002<\/p>\n<p>\u6700\u7ec8&#xff0c;\u6574\u4e2a Transformer \u53ef\u4ee5\u7406\u89e3\u6210&#xff1a;<\/p>\n<p>Token<br \/>\n \u2193<br \/>\nEmbedding<br \/>\n \u2193<br \/>\n\u52a0\u5165\u4f4d\u7f6e\u4fe1\u606f<br \/>\n \u2193<br \/>\nAttention \u5efa\u7acb\u5173\u7cfb<br \/>\n \u2193<br \/>\nFFN \u8fdb\u4e00\u6b65\u5904\u7406<br \/>\n \u2193<br \/>\n\u591a\u5c42 Block \u4e0d\u65ad\u63d0\u53d6\u4fe1\u606f<br \/>\n \u2193<br \/>\n\u5b8c\u6210\u7406\u89e3\u6216\u751f\u6210\u4efb\u52a1<\/p>\n<hr \/>\n<h2>\u4e0b\u4e00\u7bc7&#xff1a;Self-Attention \u5230\u5e95\u662f\u4ec0\u4e48&#xff1f;<\/h2>\n<p>\u73b0\u5728&#xff0c;\u6211\u4eec\u5df2\u7ecf\u770b\u6e05\u4e86 Transformer \u7684\u6574\u4f53\u7ed3\u6784\u3002<\/p>\n<p>\u4f46\u662f\u6574\u5957\u67b6\u6784\u4e2d\u6700\u6838\u5fc3\u7684\u95ee\u9898\u4ecd\u7136\u6ca1\u6709\u771f\u6b63\u89e3\u51b3&#xff1a;<\/p>\n<p>Attention \u5230\u5e95\u662f\u600e\u4e48\u5de5\u4f5c\u7684&#xff1f;<\/p>\n<p>\u4e3a\u4ec0\u4e48 Transformer \u4f1a\u628a\u8f93\u5165\u8f6c\u6362\u6210&#xff1a;<\/p>\n<p>Q<br \/>\nK<br \/>\nV<\/p>\n<p>\u4e3a\u4ec0\u4e48\u8981\u8ba1\u7b97&#xff1a;<\/p>\n<p><span class=\"katex--display\"><span class=\"katex-display\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>         Q<\/p>\n<p>          K<\/p>\n<p>          T<\/p>\n<p>         QK^T <\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 1.0858em;vertical-align: -0.1944em\"><\/span><span class=\"mord mathnormal\">Q<\/span><span class=\"mord\"><span class=\"mord mathnormal\" style=\"margin-right: 0.0715em\">K<\/span><span class=\"msupsub\"><span class=\"vlist-t\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.8913em\"><span class=\"\" style=\"top: -3.113em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathnormal mtight\" style=\"margin-right: 0.1389em\">T<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<p>\u4e3a\u4ec0\u4e48\u8fd8\u8981\u8fdb\u884c&#xff1a;<\/p>\n<p><span class=\"katex--display\"><span class=\"katex-display\"><span class=\"katex\"><span class=\"katex-mathml\"><\/p>\n<p>         softmax<\/p>\n<p>         \u2061<\/p>\n<p>          (<\/p>\n<p>            Q<\/p>\n<p>             K<\/p>\n<p>             T<\/p>\n<p>             d<\/p>\n<p>             k<\/p>\n<p>          )<\/p>\n<p>         \\\\operatorname{softmax} \\\\left( \\\\frac{QK^T}{\\\\sqrt{d_k}} \\\\right) <\/p>\n<p>     <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 2.4684em;vertical-align: -0.95em\"><\/span><span class=\"mop\"><span class=\"mord mathrm\">softmax<\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.1667em\"><\/span><span class=\"minner\"><span class=\"mopen delimcenter\" style=\"top: 0em\"><span class=\"delimsizing size3\">(<\/span><\/span><span class=\"mord\"><span class=\"mopen nulldelimiter\"><\/span><span class=\"mfrac\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 1.5183em\"><span class=\"\" style=\"top: -2.2528em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"mord\"><span class=\"mord sqrt\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.8572em\"><span class=\"svg-align\" style=\"top: -3em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"mord\" style=\"padding-left: 0.833em\"><span class=\"mord\"><span class=\"mord mathnormal\">d<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.3361em\"><span class=\"\" style=\"top: -2.55em;margin-left: 0em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathnormal mtight\" style=\"margin-right: 0.0315em\">k<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"\" style=\"top: -2.8172em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"hide-tail\" style=\"min-width: 0.853em;height: 1.08em\"><\/p>\n<p>                    <\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.1828em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"\" style=\"top: -3.23em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"frac-line\" style=\"border-bottom-width: 0.04em\"><\/span><\/span><span class=\"\" style=\"top: -3.677em\"><span class=\"pstrut\" style=\"height: 3em\"><\/span><span class=\"mord\"><span class=\"mord mathnormal\">Q<\/span><span class=\"mord\"><span class=\"mord mathnormal\" style=\"margin-right: 0.0715em\">K<\/span><span class=\"msupsub\"><span class=\"vlist-t\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.8413em\"><span class=\"\" style=\"top: -3.063em;margin-right: 0.05em\"><span class=\"pstrut\" style=\"height: 2.7em\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathnormal mtight\" style=\"margin-right: 0.1389em\">T<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.93em\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><span class=\"mclose nulldelimiter\"><\/span><\/span><span class=\"mclose delimcenter\" style=\"top: 0em\"><span class=\"delimsizing size3\">)<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n<p>\u8fd9\u4e9b\u64cd\u4f5c\u7a76\u7adf\u5728\u505a\u4ec0\u4e48&#xff1f;<\/p>\n<p>\u4e0b\u4e00\u7bc7&#xff0c;\u6211\u4eec\u5c06\u6b63\u5f0f\u8fdb\u5165 Transformer \u6700\u6838\u5fc3\u7684\u90e8\u5206&#xff1a;<\/p>\n<p>Self-Attention\u3002<\/p>\n<p>\u6211\u4eec\u4f1a\u4ece\u4e00\u4e2a\u5177\u4f53\u7684\u53e5\u5b50\u5f00\u59cb&#xff0c;\u4e00\u6b65\u4e00\u6b65\u7406\u89e3&#xff1a;<\/p>\n<p>Q \u5230\u5e95\u662f\u4ec0\u4e48&#xff1f;<br \/>\nK \u5230\u5e95\u662f\u4ec0\u4e48&#xff1f;<br \/>\nV \u53c8\u662f\u4ec0\u4e48&#xff1f;<\/p>\n<p>\u7136\u540e\u4eb2\u624b\u8d70\u5b8c\u6574\u4e2a Attention \u7684\u8ba1\u7b97\u8fc7\u7a0b\u3002<\/p>\n<p>\u53ea\u6709\u771f\u6b63\u7406\u89e3\u4e86 Self-Attention&#xff0c;Transformer \u7684\u6838\u5fc3\u624d\u7b97\u771f\u6b63\u5f00\u59cb\u5c55\u5f00\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u76ee\u5f55<br \/>\n\u4e00\u3001\u5148\u4ece\u4e00\u5f20\u7ecf\u5178\u7ed3\u6784\u56fe\u5f00\u59cb\u4e8c\u3001Transformer \u4e0d\u662f\u4e00\u4e2a\u6a21\u5757&#xff0c;\u800c\u662f\u4e00\u7ec4\u6a21\u5757\u7684\u7ec4\u5408\u4e09\u3001\u4ec0\u4e48\u662f Encoder&#xff1f;\u56db\u3001Encoder Block&#xff1a;Transformer \u7684\u57fa\u672c\u79ef\u6728\u4e94\u3001\u7b2c\u4e00\u90e8\u5206&#xff1a;Multi-Head Self-Attention\u516d\u3001\u7b2c\u4e8c\u90e8\u5206&#xff1a;\u4e3a\u4ec0\u4e48 Attention \u540e\u9762\u8fd8\u9700\u8981 Add&#xff1f;\u4e03\u3001\u7b2c\u4e09\u90e8\u5206&#xff1a;Norm \u53c8\u662f\u4ec0\u4e48&#xff1f;\u516b\u3001\u7b2c\u56db\u90e8\u5206&amp;#xf<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[99,841,50,190,86],"topic":[],"class_list":["post-97293","post","type-post","status-publish","format-standard","hentry","category-server","tag-java","tag-transformer","tag-50","tag-190","tag-86"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v20.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>\u6df1\u5165\u7406\u89e3 Transformer\uff1aTransformer \u7a76\u7adf\u662f\u4ec0\u4e48\uff1f - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.wsisp.com\/helps\/97293.html\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u6df1\u5165\u7406\u89e3 Transformer\uff1aTransformer \u7a76\u7adf\u662f\u4ec0\u4e48\uff1f - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"og:description\" content=\"\u76ee\u5f55 \u4e00\u3001\u5148\u4ece\u4e00\u5f20\u7ecf\u5178\u7ed3\u6784\u56fe\u5f00\u59cb\u4e8c\u3001Transformer \u4e0d\u662f\u4e00\u4e2a\u6a21\u5757&#xff0c;\u800c\u662f\u4e00\u7ec4\u6a21\u5757\u7684\u7ec4\u5408\u4e09\u3001\u4ec0\u4e48\u662f Encoder&#xff1f;\u56db\u3001Encoder Block&#xff1a;Transformer \u7684\u57fa\u672c\u79ef\u6728\u4e94\u3001\u7b2c\u4e00\u90e8\u5206&#xff1a;Multi-Head Self-Attention\u516d\u3001\u7b2c\u4e8c\u90e8\u5206&#xff1a;\u4e3a\u4ec0\u4e48 Attention \u540e\u9762\u8fd8\u9700\u8981 Add&#xff1f;\u4e03\u3001\u7b2c\u4e09\u90e8\u5206&#xff1a;Norm \u53c8\u662f\u4ec0\u4e48&#xff1f;\u516b\u3001\u7b2c\u56db\u90e8\u5206&amp;#xf\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.wsisp.com\/helps\/97293.html\" \/>\n<meta property=\"og:site_name\" content=\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"article:published_time\" content=\"2026-08-29T00:37:08+00:00\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"21 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/97293.html\",\"url\":\"https:\/\/www.wsisp.com\/helps\/97293.html\",\"name\":\"\u6df1\u5165\u7406\u89e3 Transformer\uff1aTransformer \u7a76\u7adf\u662f\u4ec0\u4e48\uff1f - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"isPartOf\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\"},\"datePublished\":\"2026-08-29T00:37:08+00:00\",\"dateModified\":\"2026-08-29T00:37:08+00:00\",\"author\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/97293.html#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.wsisp.com\/helps\/97293.html\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/97293.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.wsisp.com\/helps\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u6df1\u5165\u7406\u89e3 Transformer\uff1aTransformer \u7a76\u7adf\u662f\u4ec0\u4e48\uff1f\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\",\"url\":\"https:\/\/www.wsisp.com\/helps\/\",\"name\":\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"description\":\"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"contentUrl\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"caption\":\"admin\"},\"sameAs\":[\"http:\/\/wp.wsisp.com\"],\"url\":\"https:\/\/www.wsisp.com\/helps\/author\/admin\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"\u6df1\u5165\u7406\u89e3 Transformer\uff1aTransformer \u7a76\u7adf\u662f\u4ec0\u4e48\uff1f - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.wsisp.com\/helps\/97293.html","og_locale":"zh_CN","og_type":"article","og_title":"\u6df1\u5165\u7406\u89e3 Transformer\uff1aTransformer \u7a76\u7adf\u662f\u4ec0\u4e48\uff1f - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","og_description":"\u76ee\u5f55 \u4e00\u3001\u5148\u4ece\u4e00\u5f20\u7ecf\u5178\u7ed3\u6784\u56fe\u5f00\u59cb\u4e8c\u3001Transformer \u4e0d\u662f\u4e00\u4e2a\u6a21\u5757&#xff0c;\u800c\u662f\u4e00\u7ec4\u6a21\u5757\u7684\u7ec4\u5408\u4e09\u3001\u4ec0\u4e48\u662f Encoder&#xff1f;\u56db\u3001Encoder Block&#xff1a;Transformer \u7684\u57fa\u672c\u79ef\u6728\u4e94\u3001\u7b2c\u4e00\u90e8\u5206&#xff1a;Multi-Head Self-Attention\u516d\u3001\u7b2c\u4e8c\u90e8\u5206&#xff1a;\u4e3a\u4ec0\u4e48 Attention \u540e\u9762\u8fd8\u9700\u8981 Add&#xff1f;\u4e03\u3001\u7b2c\u4e09\u90e8\u5206&#xff1a;Norm \u53c8\u662f\u4ec0\u4e48&#xff1f;\u516b\u3001\u7b2c\u56db\u90e8\u5206&amp;#xf","og_url":"https:\/\/www.wsisp.com\/helps\/97293.html","og_site_name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","article_published_time":"2026-08-29T00:37:08+00:00","author":"admin","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"admin","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"21 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.wsisp.com\/helps\/97293.html","url":"https:\/\/www.wsisp.com\/helps\/97293.html","name":"\u6df1\u5165\u7406\u89e3 Transformer\uff1aTransformer \u7a76\u7adf\u662f\u4ec0\u4e48\uff1f - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","isPartOf":{"@id":"https:\/\/www.wsisp.com\/helps\/#website"},"datePublished":"2026-08-29T00:37:08+00:00","dateModified":"2026-08-29T00:37:08+00:00","author":{"@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41"},"breadcrumb":{"@id":"https:\/\/www.wsisp.com\/helps\/97293.html#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.wsisp.com\/helps\/97293.html"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.wsisp.com\/helps\/97293.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.wsisp.com\/helps"},{"@type":"ListItem","position":2,"name":"\u6df1\u5165\u7406\u89e3 Transformer\uff1aTransformer \u7a76\u7adf\u662f\u4ec0\u4e48\uff1f"}]},{"@type":"WebSite","@id":"https:\/\/www.wsisp.com\/helps\/#website","url":"https:\/\/www.wsisp.com\/helps\/","name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","description":"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41","name":"admin","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/","url":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","contentUrl":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","caption":"admin"},"sameAs":["http:\/\/wp.wsisp.com"],"url":"https:\/\/www.wsisp.com\/helps\/author\/admin"}]}},"_links":{"self":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/97293","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/comments?post=97293"}],"version-history":[{"count":0,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/97293\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media?parent=97293"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/categories?post=97293"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/tags?post=97293"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/topic?post=97293"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}