![\"\"](\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/01\/20260113111659-696629ab52650.png\")
<\/p>\n\u63a5\u524d\u7f6e\u6587\u7ae0：<\/p>\n
\u4ece\u96f6\u6784\u5efa\u5927\u6a21\u578b\u8bfb\u4e66\u8bb0\u5f55(\u4e00)\u2014\u2014\u7406\u89e3\u5927\u8bed\u8a00\u6a21\u578b<\/p>\n
\u4ece\u96f6\u6784\u5efa\u5927\u6a21\u578b\u8bfb\u4e66\u8bb0\u5f55(\u4e8c)\u2014\u2014\u5904\u7406\u6587\u672c\u6570\u636e<\/p>\n
\u4ece\u96f6\u6784\u5efa\u5927\u6a21\u578b\u8bb0\u5f55(\u4e09)\u2014\u2014\u4ece\u96f6\u5b9e\u73b0 Transformer \u6838\u5fc3\u6a21\u5757(\u4e0b)\u5df2\u4e0a\u4f20<\/p>\n
\u672c\u6587\u5c06\u7ee7\u7eed\u8ba8\u8bba\u6ce8\u610f\u529b\u673a\u5236<\/p>\n
<\/p>\n
\u4e3a\u4ec0\u4e48\u8981\u5728\u795e\u7ecf\u7f51\u7edc\u4e2d\u4f7f\u7528\u6ce8\u610f\u529b\u673a\u5236\u5462？<\/p>\n
\u4f20\u7edf\u7684 RNN-based \u673a\u5668\u7ffb\u8bd1\u6a21\u578b\u5728\u751f\u6210\u76ee\u6807\u8bcd\u65f6，\u4f9d\u8d56\u4e8e\u6709\u9650\u7684\u4e0a\u4e0b\u6587\u4fe1\u606f（\u5982\u6700\u540e\u4e00\u4e2a\u9690\u85cf\u72b6\u6001），\u96be\u4ee5\u6709\u6548\u5efa\u6a21\u957f\u8ddd\u79bb\u4f9d\u8d56，\u4e14\u8bad\u7ec3\u8fc7\u7a0b\u4e32\u884c\u3001\u6548\u7387\u4f4e\u3002\u867d\u7136\u540e\u6765\u7684 Seq2Seq+Attention \u6a21\u578b\u901a\u8fc7\u5f15\u5165\u6ce8\u610f\u529b\u673a\u5236\u7f13\u89e3\u4e86\u4fe1\u606f\u74f6\u9888\u95ee\u9898，\u4f46\u4ecd\u53d7\u9650\u4e8e RNN \u7684\u4e32\u884c\u7ed3\u6784\u3002<\/p>\n
Transformer \u67b6\u6784\u63d0\u51fa\u201c\u4ec5\u4f7f\u7528\u6ce8\u610f\u529b\u673a\u5236\u201d（Attention is All You Need），\u901a\u8fc7\u81ea\u6ce8\u610f\u529b\u673a\u5236\u8ba9\u6bcf\u4e2a\u8bcd\u5143\u5728\u7f16\u7801\u6216\u89e3\u7801\u65f6\u90fd\u80fd\u52a8\u6001\u5173\u6ce8\u5e8f\u5217\u4e2d\u6240\u6709\u5176\u4ed6\u8bcd\u5143，\u5e76\u6839\u636e\u8bed\u4e49\u76f8\u5173\u6027\u5206\u914d\u4e0d\u540c\u6743\u91cd\u3002\u8fd9\u4e0d\u4ec5\u663e\u8457\u63d0\u5347\u4e86\u5bf9\u957f\u8ddd\u79bb\u4f9d\u8d56\u7684\u5efa\u6a21\u80fd\u529b，\u8fd8\u4f7f\u5f97\u6574\u4e2a\u7f51\u7edc\u5728\u8bad\u7ec3\u9636\u6bb5\u5b8c\u5168\u53ef\u5e76\u884c\u5316，\u6781\u5927\u52a0\u901f\u4e86\u6a21\u578b\u8bad\u7ec3，\u63a8\u52a8\u4e86\u5927\u8bed\u8a00\u6a21\u578b\u7684\u53d1\u5c55\u3002<\/p>\n
\u4e3a\u76f4\u89c2\u7406\u89e3\u8fd9\u4e00\u673a\u5236，\u6211\u4eec\u4ee5\u4e00\u4e2a\u5177\u4f53\u573a\u666f\u4e3a\u4f8b：\u5c06\u82f1\u6587\u53e5\u5b50 \u201cThe cat sat on the mat.\u201d \u7ffb\u8bd1\u4e3a\u6cd5\u8bed \u201cLe chat s'est assis sur le tapis.\u201d<\/p>\n
![\"\"](\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/01\/20260113111700-696629ac7d242.png\")
<\/p>\n
\u90a3\u4e48Transformer\u5177\u4f53\u662f\u600e\u4e48\u8fd0\u884c\u7684，\u4e00\u4e2a\u57fa\u7840\u7684\u81ea\u6ce8\u610f\u529b\u6846\u67b6\u662f\u600e\u6837\u7684，\u81ea\u6ce8\u610f\u529b\u673a\u5236\u7684\u76ee\u6807\u662f\u4e3a\u6bcf\u4e2a\u8bcd\u5143\u8ba1\u7b97\u4e00\u4e2a\u4e0a\u4e0b\u6587\u5411\u91cf，\u4e0b\u9762\u5c06\u4ee5\u4e00\u4e2a\u4e0d\u5e26\u53ef\u8bad\u7ec3\u6743\u91cd\u7684\u57fa\u7840\u81ea\u6ce8\u610f\u529b\u6846\u67b6\u4e3a\u4f8b\u5c55\u793a\u5176\u8fd0\u884c\u8fc7\u7a0b\u3002<\/p>\n
\u201c\u6bcf\u4e2a\u8bcd\u90fd\u5e94\u5f53\u77e5\u9053\u53e5\u5b50\u91cc\u5176\u4ed6\u8bcd\u5728\u8bf4\u4ec0\u4e48，\u5e76\u6839\u636e\u76f8\u5173\u6027\u52a0\u6743\u878d\u5408\u3002\u5373\u5f97\u5230\u6bcf\u4e2a\u8bcd\u5143\u7684\u4e0a\u4e0b\u6587\u5411\u91cf\u201d<\/p>\n
\u4f8b\u5982，\u5728\u53e5\u5b50 "The cat sat on the mat" \u4e2d：<\/p>\n
\u5f53\u5904\u7406 "sat" \u65f6，\u6a21\u578b\u5e94\u5173\u6ce8 "cat"（\u4e3b\u8bed）\u548c "mat"（\u5730\u70b9）；<\/p>\n<\/li>\n
\u5f53\u5904\u7406 "cat" \u65f6，\u53ef\u80fd\u5173\u6ce8 "The"（\u51a0\u8bcd）\u548c "sat"（\u8c13\u8bed）\u3002<\/p>\n<\/li>\n<\/ul>\n
\u81ea\u6ce8\u610f\u529b\u901a\u8fc7\u67e5\u8be2（Query）\u3001\u952e（Key）\u3001\u503c（Value） \u673a\u5236\u5b9e\u73b0\u8fd9\u4e00\u70b9\u3002<\/p>\n
tips：\u6b64\u90e8\u5206\u4e3a\u65e0\u53c2\u81ea\u6ce8\u610f\u529b\u7684\u7b80\u5316\u6f14\u793a，\u4ec5\u7528\u4e8e\u7406\u89e3\u6ce8\u610f\u529b\u7684\u52a0\u6743\u878d\u5408\u903b\u8f91，\u5e76\u975e Transformer \u7684\u5b9e\u9645\u5b9e\u73b0<\/p>\n
\u4ee5"The cat sat on the mat"\u4e3a\u4f8b，\u5c06\u5176\u5d4c\u5165\u4e3a3\u7ef4\u5411\u91cf<\/p>\n
tensor( [ 2.2315, -0.7460, -0.0614], #The (x^1)
\n [-0.1235, 0.4462, 0.7265], #cat (x^2)
\n [ 1.4900, -2.0396, 1.0440], #sat (x^3)
\n [-0.6735, -0.5763, -0.9291], #on (x^4)
\n [ 0.7707, 0.5180, 0.2458], #the (x^5)
\n [ 0.6508, 0.1164, -1.3904]] #mat (x^6)
\n ) <\/p>\n
\u6bcf\u4e2a\u8bcd\u5143\u7684\u4e0a\u4e0b\u6587\u5411\u91cf\u8ba1\u7b97\u8fc7\u7a0b\u5982\u4e0b<\/p>\n
![\"\"](\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/01\/20260113111704-696629b064965.png\")
<\/p>\n
\u5b9e\u73b0\u6ce8\u610f\u529b\u5206\u6570\u7684\u7b2c\u4e00\u6b65\u662f\u8ba1\u7b97\u6ce8\u610f\u529b\u5206\u6570，\u5373\u8ba1\u7b97\u6bcf\u4e2a\u4f4d\u7f6e i \u7684 Query \u5411\u91cf\u4e0e\u6240\u6709\u4f4d\u7f6e j \u7684 Key \u5411\u91cf\u7684\u70b9\u79ef，\u5f97\u5230\u6ce8\u610f\u529b\u5206\u6570 A[i,j]；\u56e0\u4e3a\u70b9\u79ef\u5728\u5411\u91cf\u957f\u5ea6\u76f8\u8fd1\u65f6\u53ef\u8fd1\u4f3c\u53cd\u6620\u8bed\u4e49\u76f8\u4f3c\u5ea6，\u5e38\u7528\u4e8e\u8861\u91cf\u4e24\u4e2a token \u7684\u76f8\u5173\u6027\u3002\u70b9\u79ef\u8d8a\u5927，\u8bf4\u660e\u4e24\u4e2a\u5411\u91cf\u65b9\u5411\u8d8a\u4e00\u81f4（\u5939\u89d2\u8d8a\u5c0f），\u8bed\u4e49\u76f8\u4f3c\u5ea6\u8d8a\u9ad8，\u6ce8\u610f\u529b\u5206\u6570\u4e5f\u8d8a\u9ad8\u3002\u3002\u4f8b\u5b50\u4e2d\u8bcd\u5143\u4e4b\u95f4\u70b9\u79ef\u8ba1\u7b97\u7ed3\u679c\u4e3a：<\/p>\n
tensor([[ 5.5401, -0.6531, 4.7825, -1.0159, 1.3183, 1.4509], #The\u4e0e\u5176\u4ed6\u8bcd\u5143\u95f4\u7684\u6ce8\u610f\u529b\u5206\u6570，\u8fd9\u4e9b\u6570\u7531\u4e8e\u672a\u7ecf\u8bad\u7ec3，\u662f\u968f\u673a\u751f\u6210\u7684
\n [-0.6531, 0.7421, -0.3357, -0.8489, 0.3145, -1.0385],
\n [ 4.7825, -0.3357, 7.4699, -0.7981, 0.3484, -0.7192],
\n [-1.0159, -0.8489, -0.7981, 1.6490, -1.0459, 0.7864],
\n [ 1.3183, 0.3145, 0.3484, -1.0459, 0.9227, 0.2202],
\n [ 1.4509, -1.0385, -0.7192, 0.7864, 0.2202, 2.3702]],
\n grad_fn=<MmBackward0>)) <\/p>\n
\u5c06\u6ce8\u610f\u529b\u5206\u6570\u8fdb\u884c\u5f52\u4e00\u5316，\u5f52\u4e00\u5316\u540e\u7ed3\u679c\u5982\u4e0b：<\/p>\n
Attention Weight 2: tensor([[6.6504e-01, 1.3588e-03, 3.1176e-01, 9.4538e-04, 9.7572e-03, 1.1141e-02], #\u5355\u8bcdThe\u5f52\u4e00\u5316\u540e\u5206\u6570，\u6c42\u548c\u4e3a1
\n [9.4845e-02, 3.8278e-01, 1.3028e-01, 7.7982e-02, 2.4960e-01, 6.4515e-02],
\n [6.3619e-02, 3.8089e-04, 9.3475e-01, 2.3987e-04, 7.5490e-04, 2.5956e-04],
\n [4.0282e-02, 4.7603e-02, 5.0084e-02, 5.7868e-01, 3.9091e-02, 2.4426e-01],
\n [3.5132e-01, 1.2875e-01, 1.3320e-01, 3.3033e-02, 2.3654e-01, 1.1716e-01],
\n [2.2166e-01, 1.8390e-02, 2.5308e-02, 1.1406e-01, 6.4745e-02, 5.5584e-01]],
\n grad_fn=<SoftmaxBackward0>)
\nAttention Weight 2 Sum: tensor([1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
\n grad_fn=<SumBackward1>) <\/p>\n
\u901a\u8fc7\u5c06\u5d4c\u5165\u8bcd\u5143\u4e0e\u76f8\u5e94\u7684\u6ce8\u610f\u529b\u6743\u91cd\u76f8\u4e58，\u5c06\u5f97\u5230\u7684\u5411\u91cf\u6c42\u548c\u8ba1\u7b97\u4e0a\u4e0b\u6587\u5411\u91cf，\u7ed3\u679c\u5982\u4e0b<\/p>\n
tensor([[ 1.9626, -1.1256, 0.2716], #The\u4e0e\u6ce8\u610f\u529b\u6743\u91cd\u76f8\u4e58\u6c42\u548c\u5f97\u5230\u7684\u4e0a\u4e0b\u6587\u5411\u91cf，\u8fd9\u4e2a\u4e0a\u4e0b\u6587\u5411\u91cf\u5305\u542b\u4e86\u6ce8\u610f\u529b\u5206\u6570\u4fe1\u606f，\u56e0\u6b64\u643a\u5e26\u8be5\u8bcd\u8bed\u4f4d\u7f6e\u7684\u5411\u91cf\u90fd\u878d\u5408\u4e86\u5168\u53e5\u4fe1\u606f\u3002
\n [ 0.5403, -0.0738, 0.3074],
\n [ 1.5353, -1.9535, 0.9718],
\n [-0.0420, -0.3958, -0.7833],
\n [ 1.2028, -0.3592, 0.0755],
\n [ 0.8649, -0.1763, -0.8367]], grad_fn=<MmBackward0>) <\/p>\n
"""
\n\u5982\u4f55\u5f97\u5230\u6bcf\u4e2a\u8bcd\u5143\u7684\u4e0a\u4e0b\u6587\u5411\u91cf（\u65e0\u53c2\u81ea\u6ce8\u610f\u529b\u7b80\u5316\u6f14\u793a）
\n"""<\/p>\n
import torch
\nimport tiktoken<\/p>\n
# ===================== 1. \u5206\u8bcd\u4e0e\u5d4c\u5165 =====================
\n# \u521d\u59cb\u5316\u5206\u8bcd\u5668（GPT2）
\ntokenizer = tiktoken.get_encoding('gpt2')
\ntext = "The cat sat on the mat"
\ntoken_ids = tokenizer.encode(text) # \u5206\u8bcd\u7ed3\u679c: [1169, 398, 6356, 286, 262, 1369]
\nprint(f"\u5206\u8bcd\u540e\u7684token_ids: {token_ids}, shape: {torch.tensor(token_ids).shape}") # shape: [6]<\/p>\n
# \u5b9a\u4e49\u5d4c\u5165\u5c42\u53c2\u6570
\nvocab_size = tokenizer.n_vocab # GPT2\u8bcd\u6c47\u8868\u5927\u5c0f
\nembedding_dim = 3 # \u81ea\u5b9a\u4e49\u5d4c\u5165\u7ef4\u5ea6（\u4ec5\u6f14\u793a\u7528）
\nembedding_layer = torch.nn.Embedding(
\n num_embeddings=vocab_size,
\n embedding_dim=embedding_dim
\n)<\/p>\n
# \u5c06 token IDs \u8f6c\u4e3a PyTorch \u5f20\u91cf
\ninput_ids = torch.tensor(token_ids) # shape: [seq_len] = [6]<\/p>\n
# \u83b7\u53d6\u5d4c\u5165\u5411\u91cf：\u5c06\u79bb\u6563token\u6620\u5c04\u4e3a\u8fde\u7eed\u5411\u91cf
\ninputs = embedding_layer(input_ids) # shape: [seq_len, embedding_dim] = [6, 3]
\nprint("\\\\n=== \u8f93\u5165\u5d4c\u5165\u5411\u91cf ===")
\nprint("inputs shape:", inputs.shape) # \u8f93\u51fa: torch.Size([6, 3])
\nprint("inputs:\\\\n", inputs)<\/p>\n
# ===================== 2. \u8ba1\u7b97\u6ce8\u610f\u529b\u5206\u6570（\u70b9\u79ef） =====================
\n# \u70b9\u79ef\u6ce8\u610f\u529b\u5206\u6570：\u6bcf\u4e2a\u8bcd\u5143\u4e0e\u6240\u6709\u8bcd\u5143\u7684\u76f8\u4f3c\u5ea6
\n# inputs (6,3) @ inputs.T (3,6) \u2192 attn_scores (6,6)
\nattn_scores = inputs @ inputs.T # shape: [seq_len, seq_len] = [6, 6]
\nprint("\\\\n=== \u6ce8\u610f\u529b\u5206\u6570 ===")
\nprint("attn_scores shape:", attn_scores.shape) # \u8f93\u51fa: torch.Size([6, 6])
\nprint("attn_scores:\\\\n", attn_scores)<\/p>\n
# ===================== 3. \u6ce8\u610f\u529b\u5206\u6570\u5f52\u4e00\u5316（Softmax） =====================
\n# dim=-1: \u5bf9\u6700\u540e\u4e00\u7ef4（\u6bcf\u4e2a\u8bcd\u5143\u76846\u4e2a\u5206\u6570）\u505a\u5f52\u4e00\u5316，\u4fdd\u8bc1\u6bcf\u884c\u548c\u4e3a1
\nattn_weights = torch.softmax(attn_scores, dim=-1) # shape: [6, 6]
\nprint("\\\\n=== \u5f52\u4e00\u5316\u6ce8\u610f\u529b\u6743\u91cd ===")
\nprint("Attention Weights shape:", attn_weights.shape) # \u8f93\u51fa: torch.Size([6, 6])
\nprint("Attention Weights:\\\\n", attn_weights)
\n# \u9a8c\u8bc1\u6bcf\u884c\u548c\u4e3a1（\u5f52\u4e00\u5316\u6548\u679c）
\nprint("Attention Weights Sum (\u6bcf\u884c\u548c):\\\\n", attn_weights.sum(dim=-1)) # shape: [6]，\u503c\u5168\u4e3a1.0<\/p>\n
# ===================== 4. \u8ba1\u7b97\u4e0a\u4e0b\u6587\u5411\u91cf（\u52a0\u6743\u6c42\u548c） =====================
\n# attn_weights (6,6) @ inputs (6,3) \u2192 all_context_vecs (6,3)
\nall_context_vecs = attn_weights @ inputs # shape: [seq_len, embedding_dim] = [6, 3]
\nprint("\\\\n=== \u6700\u7ec8\u4e0a\u4e0b\u6587\u5411\u91cf ===")
\nprint("Context Vectors shape:", all_context_vecs.shape) # \u8f93\u51fa: torch.Size([6, 3])
\nprint("Context Vectors:\\\\n", all_context_vecs) <\/p>\n
\u4e0a\u8ff0\u6f14\u793a\u4e86\u81ea\u6ce8\u610f\u529b\u6743\u91cd\u7684\u8ba1\u7b97\u8fc7\u7a0b，\u867d\u7136\u76f4\u89c2，\u4f46\u7f3a\u4e4f\u5b66\u4e60\u80fd\u529b\u3002<\/p>\n
\u771f\u5b9e\u6a21\u578b\u901a\u8fc7\u5f15\u5165\u53ef\u8bad\u7ec3\u7684 Q\/K\/V \u6295\u5f71\u77e9\u9635，\u4f7f\u6ce8\u610f\u529b\u673a\u5236\u80fd\u591f\u6839\u636e\u4efb\u52a1\u52a8\u6001\u8c03\u6574\u5173\u6ce8\u91cd\u70b9\u3002\u201d；\u5728\u771f\u5b9e\u7684 Transformer \u4e2d（\u5982 GPT）：\u5d4c\u5165\u5c42\u662f\u901a\u8fc7\u5927\u91cf\u6587\u672c\u8bad\u7ec3\u5f97\u5230\u7684；\u201cThe\u201d \u548c \u201ccat\u201d \u4f1a\u56e0\u7ecf\u5e38\u5171\u73b0\u800c\u5d4c\u5165\u5411\u91cf\u9760\u8fd1；\u81ea\u6ce8\u610f\u529b\u4f1a\u771f\u6b63\u5b66\u4f1a：\u751f\u6210 \u201ccat\u201d \u65f6\u5173\u6ce8 \u201cThe\u201d，\u751f\u6210 \u201csat\u201d \u65f6\u5173\u6ce8 \u201ccat\u201d \u548c \u201cmat\u201d\u3002\u4f46\u5728\u968f\u673a\u521d\u59cb\u5316\u9636\u6bb5，\u6a21\u578b\u5c1a\u672a\u5b66\u4e60\u5230\u4efb\u4f55\u8bed\u4e49\u6a21\u5f0f，\u6ce8\u610f\u529b\u6743\u91cd\u63a5\u8fd1\u968f\u673a\u5206\u5e03\u3002\u90a3\u4e48\u81ea\u6ce8\u610f\u529b\u673a\u5236\u5982\u4f55\u8fdb\u884c\u8bad\u7ec3\u5462，\u9996\u5148\u9700\u8981\u4e3a\u81ea\u6ce8\u610f\u529b\u673a\u5236\u6dfb\u52a0\u53ef\u8bad\u7ec3\u6743\u91cd，\u901a\u8fc7\u5f15\u5165\u53ef\u8bad\u7ec3\u6743\u91cd，\u53ef\u4ee5\u4f7f\u6a21\u578b\u5b66\u4f1a\u4ea7\u51fa\u201c\u597d\u7684\u201d\u4e0a\u4e0b\u6587\u5411\u91cf\u3002<\/p>\n
\u81ea\u6ce8\u610f\u529b\u7684\u6838\u5fc3\u53ef\u5b66\u4e60\u53c2\u6570\u4e3a W_Q\u3001W_K\u3001W_V \u548c\u8f93\u51fa\u6295\u5f71\u77e9\u9635\u3002\u5728\u7aef\u5230\u7aef\u8bad\u7ec3\u4e2d，\u8fd9\u4e9b\u53c2\u6570\u901a\u8fc7\u53cd\u5411\u4f20\u64ad\u8fed\u4ee3\u66f4\u65b0，\u6700\u7ec8\u8ba9\u6a21\u578b\u5177\u5907*\u4efb\u52a1\u5bfc\u5411\u7684\u52a8\u6001\u805a\u7126\u80fd\u529b\u2014\u2014 \u5373\u9488\u5bf9\u4e0d\u540c\u4efb\u52a1\u548c\u8f93\u5165，\u81ea\u52a8\u8bc6\u522b\u6700\u76f8\u5173\u7684\u8bcd\u5143\u3002<\/p>\n
W_Q, W_K, W_V\u53ef\u4ee5\u7c7b\u6bd4\u4e3a\u53ef\u5b66\u4e60\u7684\u201c\u8f6f\u201d\u6570\u636e\u5e93\u68c0\u7d22\u3002W_K\u7c7b\u6bd4\u4e3a\u6570\u636e\u5e93\u4e2d\u7684\u7d22\u5f15\/\u952e（Index\/Key）；W_V\u7c7b\u6bd4\u4e3a\u6570\u636e\u5e93\u4e2d\u7684\u5b9e\u9645\u6570\u636e\/\u8bb0\u5f55（Value\/Record）W_Q\u7c7b\u6bd4\u4e3a\u7528\u6237\u53d1\u8d77\u7684\u67e5\u8be2\u8bf7\u6c42（Query）；Attention Score\u7c7b\u6bd4\u4e3a\u76f8\u4f3c\u5ea6\u5339\u914d\u5206\u6570；Output\u7c7b\u6bd4\u4e3a\u52a0\u6743\u68c0\u7d22\u7ed3\u679c\u3002<\/p>\n
\u4ee5"The cat sat on the mat"\u4e3a\u4f8b，\u5c06\u5176\u5d4c\u5165\u4e3a3\u7ef4\u5411\u91cf<\/p>\n
tensor( [ 2.2315, -0.7460, -0.0614], #The (x^1)
\n [-0.1235, 0.4462, 0.7265], #cat (x^2)
\n [ 1.4900, -2.0396, 1.0440], #sat (x^3)
\n [-0.6735, -0.5763, -0.9291], #on (x^4)
\n [ 0.7707, 0.5180, 0.2458], #the (x^5)
\n [ 0.6508, 0.1164, -1.3904]] #mat (x^6)
\n ) <\/p>\n
\u6dfb\u52a0\u53ef\u8bad\u7ec3\u6743\u91cd\u7684\u8bcd\u5143\u7684\u4e0a\u4e0b\u6587\u5411\u91cf\u8ba1\u7b97\u8fc7\u7a0b\u5982\u4e0b：<\/p>\n
![\"\"](\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/01\/20260113111704-696629b0ea90b.png\")
<\/p>\n
\u521d\u59cb\u53163\u4e2a\u6743\u91cd\u77e9\u9635：\u7528\u4e8e\u5c06\u539f\u59cb\u5d4c\u5165\u6295\u5f71\u5230\u4e09\u4e2a\u4e0d\u540c\u7684\u8bed\u4e49\u5b50\u7a7a\u95f4\u3002\u8fd9\u4e9b\u77e9\u9635\u662f\u6a21\u578b\u53c2\u6570，\u5728\u540e\u7eed\u8bad\u7ec3\u4e2d\u901a\u8fc7\u53cd\u5411\u4f20\u64ad\u66f4\u65b0<\/p>\n
W_query Parameter containing:
\ntensor([[0.2961, 0.5166],
\n [0.2517, 0.6886],
\n [0.0740, 0.8665]]) # shape (3, 2)
\nW_key Parameter containing:
\ntensor([[0.1366, 0.1025],
\n [0.1841, 0.7264],
\n [0.3153, 0.6871]]) # shape (3, 2)
\nW_value Parameter containing:
\ntensor([[0.0756, 0.1966],
\n [0.3164, 0.4017],
\n [0.1186, 0.8274]]) # shape (3, 2) <\/p>\n
\u4ee5\u7b2c\u4e8c\u4e2a\u8bcd\u5143cat\u4e3a\u4f8b，\u901a\u8fc7\u77e9\u9635\u4e58\u6cd5\u8ba1\u7b97\u67e5\u8be2\u5411\u91cf\u3001\u952e\u5411\u91cf\u548c\u503c\u5411\u91cf，\u5047\u8bbe\u8f93\u5165\u5d4c\u5165\u7ef4\u5ea6\u4e3a3，\u8f93\u51fa\u5d4c\u5165\u7ef4\u5ea6\u4e3a2\u3002\u8ba1\u7b97\u793a\u4f8b：\u7b2c\u4e00\u884c\u7b2c\u4e00\u5217= (-0.1235* 0.2961) + (0.4462 * 0.2517) + (0.7265 * 0.0740)\u2248 -0.0366 + 0.1123 + 0.0538 \u2248 0.1295，\u4ee5\u6b64\u7c7b\u63a8\u3002\u8ba1\u7b97\u7ed3\u679c\u4ee3\u8868cat\u5728\u5f53\u524d\u4e0a\u4e0b\u6587\u4e2d\u53d1\u51fa\u7684\u201c\u67e5\u8be2\u8bf7\u6c42\u201d，\u5b83\u60f3\u77e5\u9053\u54ea\u4e9b\u8bcd\u4e0e\u81ea\u5df1\u76f8\u5173<\/p>\n
query_2 tensor([0.1295, 0.8730])
\nkey_2 tensor([0.2943, 0.8107])
\nvalue_2 tensor([0.2180, 0.7561]) <\/p>\n
\u5f97\u5230\u6240\u6709\u8f93\u5165\u8bcd\u5143\u7684\u952e\u5411\u91cf\u548c\u503c\u5411\u91cf，\u867d\u7136\u8ba1\u7b97\u7684\u662fcat\u7684\u4e0a\u4e0b\u6587\u5411\u91cf，\u4f46\u662f\u4ecd\u7136\u9700\u8981\u6240\u6709\u8bcd\u5143\u7684\u503c\u5411\u91cf\u548c\u952e\u5411\u91cf，\u56e0\u4e3a\u5b83\u4eec\u53c2\u4e0e\u4e86\u8ba1\u7b97\u76f8\u5bf9\u4e8ecat\u7684\u6ce8\u610f\u529b\u6743\u91cd；\u53ef\u4ee5\u901a\u8fc7\u77e9\u9635\u4e58\u6cd5\u8ba1\u7b97\u6240\u6709\u8bcd\u5143\u7684\u7684\u503c\u5411\u91cf\u548c\u952e\u5411\u91cf，\u8ba1\u7b97\u793a\u4f8b：\u7b2c\u4e00\u884c\u7b2c\u4e00\u52172.2315\u00d70.1366+(\u22120.7460)\u00d70.1841+(\u22120.0614)\u00d70.3153k1(1)=2.2315\u00d70.1366+(\u22120.7460)\u00d70.1841+(\u22120.0614)\u00d70.3153=0.1481（\u7b2c\u4e00\u4e2aThe\u7684\u5d4c\u5165\u5411\u91cf\u4e0eW_K\u505a\u77e9\u9635\u4e58\u6cd5）\u3002\u7ed3\u679c\u5982\u4e0b：<\/p>\n
keys tensor([[ 0.1481, -0.3554], #cat\u5bf9\u6240\u6709\u8f93\u5165\u539f\u5c5e\u7684\u952e\u5411\u91cf
\n [ 0.2943, 0.8107],
\n [ 0.1572, -0.6116],
\n [-0.4910, -1.1261],
\n [ 0.2781, 0.6242],
\n [-0.3280, -0.8041]])
\nvalues tensor([[-0.0745, 0.0883], #cat\u5bf9\u6240\u6709\u8f93\u5165\u539f\u5c5e\u7684\u503c\u5411\u91cf
\n [ 0.2180, 0.7561],
\n [-0.4089, 0.3374],
\n [-0.3435, -1.1327],
\n [ 0.2513, 0.5630],
\n [-0.0788, -0.9757]]) <\/p>\n
\u8ba1\u7b97cat\u7684\u6ce8\u610f\u529b\u5206\u6570（\u672a\u7f29\u653e），\u4f7f\u7528cat\u4f5c\u4e3a\u67e5\u8be2\u8bcd\u5143\u548c\u7b2c3\u6b65\u7684\u952e\u5411\u91cf\u8ba1\u7b97\u6ce8\u610f\u529b\u5206\u6570，\u8ba1\u7b97\u793a\u4f8b：\u4ee5cat\u4e3a\u4f8b，\u6ce8\u610f\u529b\u5206\u6570=0.1295\u00d70.2943 + 0.8730\u00d70.8107 \u2248 0.0381 + 0.7078 \u2248 0.7458<\/p>\n
attn_scores_2: tensor([-0.2911, 0.7458, -0.5136, -1.0466, 0.5809, -0.7444]) #cat\u4e0e\u952e\u5411\u91cf\u4e4b\u95f4\u70b9\u51fb\u70b9\u79ef\u8ba1\u7b97\u5f97\u5230\u672a\u7f29\u653e\u7684\u6ce8\u610f\u529b\u5206\u6570
\n#\u5206\u6790\u53ef\u77e5：
\n#\u201ccat\u201d \u4e0e\u81ea\u5df1（i=2）\u5f97\u5206\u6700\u9ad8（0.7458）\u2192 \u81ea\u5173\u6ce8\u5f88\u5e38\u89c1
\n#\u4e0e \u201cthe\u201d（i=5）\u5f97\u5206\u6b21\u9ad8（0.5809）\u2192 \u53ef\u80fd\u56e0\u51a0\u8bcd-\u540d\u8bcd\u5171\u73b0
\n#\u4e0e \u201csat\u201d（i=3）\u5f97\u5206\u5f88\u4f4e（-0.5136）\u2192 \u5f53\u524d\u6743\u91cd\u4e0b\u672a\u6355\u6349\u4e3b\u8c13\u5173\u7cfb（\u56e0\u4e3a\u672a\u8bad\u7ec3！） <\/p>\n
\u5c06\u6ce8\u610f\u529b\u5206\u6570\u8f6c\u6362\u4e3a\u6ce8\u610f\u529b\u6743\u91cd，\u901a\u8fc7\u7f29\u653e\u6ce8\u610f\u529b\u5206\u6570（\u901a\u8fc7\u5d4c\u5165\u7ef4\u5ea6\u7684\u5e73\u65b9\u6839\u8fdb\u884c\u7f29\u653e\u53ef\u4ee5\u9632\u6b62\u5d4c\u5165\u7ef4\u5ea6\u592a\u5927\u5f71\u54cd\u8bad\u7ec3\u6548\u7387，\u5f53\u5d4c\u5165\u7ef4\u5ea6\u8f83\u5927\u65f6，\u70b9\u79ef\u7684\u65b9\u5dee\u4f1a\u589e\u5927，\u5bfc\u81f4 softmax \u8fdb\u5165\u9971\u548c\u533a（\u68af\u5ea6\u63a5\u8fd1 0），\u7f29\u653e\u53ef\u7f13\u89e3\u6b64\u95ee\u9898，\u6240\u4ee5\u53eb\u7f29\u653e\u70b9\u79ef\u6ce8\u610f\u529b\u673a\u5236）\u5e76\u4f7f\u7528softmax\u8ba1\u7b97\u6ce8\u610f\u529b\u6743\u91cd，\u8ba1\u7b97\u793a\u4f8b：\u4ee5cat\u6ce8\u610f\u529b\u5206\u6570\u7f29\u653e\u4e3a\u4f8b，\u7f29\u653e\u5206\u6570=0.7458\/\u6839\u53f72=0.527，\u4e4b\u540e\u8fdb\u884csoftmax\u5f52\u4e00\u5316\u8bad\u7ec3，Softmax \u5c06\u5206\u6570\u8f6c\u6362\u4e3a\u6982\u7387\u5206\u5e03：\u6bcf\u4e2a\u6743\u91cd = exp(score) \/ \u6240\u6709 exp(score) \u4e4b\u548c\u3002<\/p>\n
\u6700\u7ec8\u5f97\u5230\u6743\u91cd\u5f62\u5f0f\u5982\u4e0b：<\/p>\n
attn_weights_2: tensor([0.1408, 0.2932, 0.1203, 0.0825, 0.2609, 0.1022])
\n#\u8fd9\u4e9b\u6743\u91cd\u5b8c\u5168\u7531\u5f53\u524d W_Q\/W_K \u51b3\u5b9a\u3002\u5982\u679c\u6a21\u578b\u7ecf\u8fc7\u8bad\u7ec3，W \u4f1a\u8c03\u6574\u4f7f\u5f97\u201csat\u201d\u5728\u751f\u6210\u65f6\u66f4\u5173\u6ce8\u201ccat\u201d，\u4f46\u73b0\u5728\u662f\u968f\u673a\u7684\u3002 <\/p>\n
\u901a\u8fc7\u5c06\u6ce8\u610f\u529b\u6743\u91cd\u4f5c\u4e3a\u52a0\u6743\u56e0\u5b50\u5bf9\u503c\u5411\u91cf\u8fdb\u884c\u52a0\u6743\u6c42\u548c\u6765\u8ba1\u7b97，\u7528\u4e8e\u8861\u91cf\u6bcf\u4e2a\u503c\u5411\u91cf\u7684\u91cd\u8981\u6027，\u8ba1\u7b97\u793a\u4f8b，0.1408\u00d7(\u22120.0745)+0.2932\u00d70.2180+0.1203\u00d7(\u22120.4089)+0.0825\u00d7(\u22120.3435)+0.2609\u00d70.2513+0.1022\u00d7(\u22120.0788)=0.0334\u3002\u4e0a\u4e0b\u6587\u5411\u91cf\u5f62\u5f0f\u5982\u4e0b：<\/p>\n
context_vec_2 tensor([0.0334, 0.2284])
\n#\u8fd9\u5c31\u662f \u201ccat\u201d \u7684\u4e0a\u4e0b\u6587\u611f\u77e5\u8868\u793a：
\n#\u5b83\u4e0d\u518d\u53ea\u662f\u539f\u59cb\u5d4c\u5165 [-0.1235, 0.4462, 0.7265]\u800c\u662f\u878d\u5408\u4e86\u5168\u53e5\u4fe1\u606f\u76842D\u5411\u91cf，\u540e\u7eed\u4f1a\u9001\u5165 FFN \u6216\u4e0b\u4e00\u5c42，\u8f93\u51fa\u7ef4\u5ea6\u662f\u53ef\u4ee5\u6539\u53d8\u7684，\u4e0d\u4e00\u5b9a\u56fa\u5b9a\u4e3a2 <\/p>\n
"""
\n\u5f15\u5165\u53ef\u8bad\u7ec3\u7684Q\/K\/V\u6743\u91cd\u77e9\u9635\u7684\u81ea\u6ce8\u610f\u529b\u673a\u5236（\u5b8c\u6574\u7248\u672c）
\n\u4ee3\u7801\u4ec5\u4ee5'cat'\u8bcd\u5143（\u7b2c2\u4e2atoken）\u4e3a\u4f8b\u6f14\u793a，\u5b9e\u9645\u4e2d\u6240\u6709\u8bcd\u5143\u5e76\u884c\u8ba1\u7b97
\n"""<\/p>\n
import torch
\ntorch.manual_seed(123) # \u56fa\u5b9a\u968f\u673a\u79cd\u5b50，\u4fdd\u8bc1\u7ed3\u679c\u53ef\u590d\u73b0<\/p>\n
# ===================== 1. \u521d\u59cb\u5316\u8f93\u5165\u5d4c\u5165（\u6a21\u62df\u503c） =====================
\n# \u6a21\u62df6\u4e2a\u8bcd\u5143\u76843\u7ef4\u5d4c\u5165\u5411\u91cf，\u5bf9\u5e94\u53e5\u5b50：The cat sat on the mat
\ninputs = torch.tensor(
\n [[2.2315, -0.7460, -0.0614], # The (token 0)
\n [-0.1235, 0.4462, 0.7265], # cat (token 1，\u91cd\u70b9\u6f14\u793a\u6b64\u8bcd\u5143)
\n [1.4900, -2.0396, 1.0440], # sat (token 2)
\n [-0.6735, -0.5763, -0.9291], # on (token 3)
\n [0.7707, 0.5180, 0.2458], # the (token 4)
\n [0.6508, 0.1164, -1.3904]] # mat (token 5)
\n) # shape: [seq_len, d_in] = [6, 3]
\nprint("\u8f93\u5165\u5d4c\u5165 inputs shape:", inputs.shape) # \u8f93\u51fa: torch.Size([6, 3])<\/p>\n
# \u9009\u53d6\u7b2c2\u4e2a\u8bcd\u5143（cat）\u4f5c\u4e3a\u6f14\u793a\u7684\u67e5\u8be2\u8bcd\u5143
\nx_2 = inputs[1] # shape: [d_in] = [3]
\nprint("\\\\ncat\u8bcd\u5143\u7684\u539f\u59cb\u5d4c\u5165 x_2 shape:", x_2.shape) # \u8f93\u51fa: torch.Size([3])<\/p>\n
# ===================== 2. \u5b9a\u4e49\u7ef4\u5ea6\u4e0e\u521d\u59cb\u5316Q\/K\/V\u6743\u91cd\u77e9\u9635 =====================
\nd_in = inputs.shape[1] # \u8f93\u5165\u5d4c\u5165\u7ef4\u5ea6: 3
\nd_out = 2 # Q\/K\/V\u8f93\u51fa\u7ef4\u5ea6: 2（\u53ef\u81ea\u5b9a\u4e49，\u901a\u5e38d_out=d_model\/h，h\u4e3a\u6ce8\u610f\u529b\u5934\u6570）<\/p>\n
# \u521d\u59cb\u5316\u53ef\u8bad\u7ec3\u6743\u91cd\u77e9\u9635（\u5b9e\u9645\u8bad\u7ec3\u65f6requires_grad=True）
\n# W_query: [d_in, d_out] = [3, 2]
\nW_query = torch.nn.Parameter(torch.rand(d_in, d_out), requires_grad=False)
\n# W_key: [d_in, d_out] = [3, 2]
\nW_key = torch.nn.Parameter(torch.rand(d_in, d_out), requires_grad=False)
\n# W_value: [d_in, d_out] = [3, 2]
\nW_value = torch.nn.Parameter(torch.rand(d_in, d_out), requires_grad=False)<\/p>\n
print("\\\\n=== Q\/K\/V\u6743\u91cd\u77e9\u9635 ===")
\nprint("W_query shape:", W_query.shape, "\\\\n", W_query)
\nprint("W_key shape:", W_key.shape, "\\\\n", W_key)
\nprint("W_value shape:", W_value.shape, "\\\\n", W_value)<\/p>\n
# ===================== 3. \u8ba1\u7b97\u5355\u4e2a\u8bcd\u5143（cat）\u7684Q\/K\/V =====================
\n# query_2: x_2 (3) @ W_query (3,2) \u2192 [2]
\nquery_2 = x_2 @ W_query # shape: [d_out] = [2]
\n# key_2: x_2 (3) @ W_key (3,2) \u2192 [2]
\nkey_2 = x_2 @ W_key # shape: [d_out] = [2]
\n# value_2: x_2 (3) @ W_value (3,2) \u2192 [2]
\nvalue_2 = x_2 @ W_value # shape: [d_out] = [2]<\/p>\n
print("\\\\n=== cat\u8bcd\u5143\u7684Q\/K\/V\u5411\u91cf ===")
\nprint("query_2 shape:", query_2.shape, "\\\\n", query_2)
\nprint("key_2 shape:", key_2.shape, "\\\\n", key_2)
\nprint("value_2 shape:", value_2.shape, "\\\\n", value_2)<\/p>\n
# ===================== 4. \u8ba1\u7b97\u6240\u6709\u8bcd\u5143\u7684K\/V\u77e9\u9635 =====================
\n# keys: inputs (6,3) @ W_key (3,2) \u2192 (6,2)
\nkeys = inputs @ W_key # shape: [seq_len, d_out] = [6, 2]
\n# values: inputs (6,3) @ W_value (3,2) \u2192 (6,2)
\nvalues = inputs @ W_value # shape: [seq_len, d_out] = [6, 2]<\/p>\n
print("\\\\n=== \u6240\u6709\u8bcd\u5143\u7684K\/V\u77e9\u9635 ===")
\nprint("keys shape:", keys.shape, "\\\\n", keys)
\nprint("values shape:", values.shape, "\\\\n", values)<\/p>\n
# ===================== 5. \u8ba1\u7b97cat\u8bcd\u5143\u7684\u6ce8\u610f\u529b\u5206\u6570（\u672a\u7f29\u653e） =====================
\n# attn_scores_2: query_2 (2) @ keys.T (2,6) \u2192 (6)
\nattn_scores_2 = query_2 @ keys.T # shape: [seq_len] = [6]
\nprint("\\\\n=== cat\u8bcd\u5143\u7684\u672a\u7f29\u653e\u6ce8\u610f\u529b\u5206\u6570 ===")
\nprint("attn_scores_2 shape:", attn_scores_2.shape, "\\\\n", attn_scores_2)<\/p>\n
# ===================== 6. \u7f29\u653e+Softmax\u5f97\u5230\u6ce8\u610f\u529b\u6743\u91cd =====================
\nd_k = keys.shape[-1] # \u952e\u5411\u91cf\u7ef4\u5ea6: 2
\n# \u7f29\u653e：\u5206\u6570 \/ \u221ad_k，\u9632\u6b62\u70b9\u79ef\u503c\u8fc7\u5927\u5bfc\u81f4Softmax\u9971\u548c
\nscaled_scores_2 = attn_scores_2 \/ (d_k ** 0.5) # shape: [6]
\n# Softmax\u5f52\u4e00\u5316\u5f97\u5230\u6ce8\u610f\u529b\u6743\u91cd
\nattn_weights_2 = torch.softmax(scaled_scores_2, dim=-1) # shape: [6]<\/p>\n
print("\\\\n=== cat\u8bcd\u5143\u7684\u6ce8\u610f\u529b\u6743\u91cd ===")
\nprint("attn_weights_2 shape:", attn_weights_2.shape, "\\\\n", attn_weights_2)
\nprint("attn_weights_2 sum:", attn_weights_2.sum()) # \u9a8c\u8bc1\u548c\u4e3a1.0<\/p>\n
# ===================== 7. \u8ba1\u7b97cat\u8bcd\u5143\u7684\u4e0a\u4e0b\u6587\u5411\u91cf =====================
\n# context_vec_2: attn_weights_2 (6) @ values (6,2) \u2192 (2)
\ncontext_vec_2 = attn_weights_2 @ values # shape: [d_out] = [2]
\nprint("\\\\n=== cat\u8bcd\u5143\u7684\u6700\u7ec8\u4e0a\u4e0b\u6587\u5411\u91cf ===")
\nprint("context_vec_2 shape:", context_vec_2.shape, "\\\\n", context_vec_2) <\/p>\n
\u672c\u6587\u7cfb\u7edf\u4ecb\u7ecd\u4e86\u81ea\u6ce8\u610f\u529b\u673a\u5236\u7684\u57fa\u672c\u539f\u7406\u4e0e\u5b9e\u73b0\u6d41\u7a0b，\u5e76\u6784\u5efa\u4e86\u4e00\u4e2a\u7b80\u5316\u7684\u81ea\u6ce8\u610f\u529b\u6846\u67b6，\u4ece\u65e0\u53c2\u7248\u672c\u5230\u5f15\u5165\u53ef\u8bad\u7ec3\u6743\u91cd\u7684\u5b8c\u6574\u5f62\u5f0f，\u9010\u6b65\u63ed\u793a\u4e86\u5176\u5982\u4f55\u4e3a\u6bcf\u4e2a\u8bcd\u5143\u751f\u6210\u4e0a\u4e0b\u6587\u611f\u77e5\u7684\u8868\u793a\u3002<\/p>\n
\u7136\u800c，\u57fa\u7840\u81ea\u6ce8\u610f\u529b\u4ecd\u6709\u5c40\u9650：<\/p>\n
\u7f3a\u4e4f\u56e0\u679c\u7ea6\u675f：\u5728\u8bed\u8a00\u751f\u6210\u4efb\u52a1\u4e2d，\u6a21\u578b\u4e0d\u80fd\u201c\u5077\u770b\u201d\u672a\u6765\u8bcd；<\/p>\n<\/li>\n
\u5355\u5934\u8868\u8fbe\u80fd\u529b\u6709\u9650：\u5355\u4e00\u6ce8\u610f\u529b\u5934\u53ef\u80fd\u65e0\u6cd5\u6355\u6349\u591a\u6837\u5316\u7684\u8bed\u4e49\u5173\u7cfb；<\/p>\n<\/li>\n
\u6613\u8fc7\u62df\u5408：\u5c24\u5176\u5728\u5c0f\u6570\u636e\u96c6\u4e0a，\u9ad8\u7ef4\u6ce8\u610f\u529b\u6743\u91cd\u53ef\u80fd\u5bfc\u81f4\u6cdb\u5316\u80fd\u529b\u4e0b\u964d\u3002<\/p>\n<\/li>\n<\/ul>\n
\u4e3a\u6b64，\u73b0\u4ee3\u5927\u8bed\u8a00\u6a21\u578b\u5f15\u5165\u4e86\u591a\u9879\u589e\u5f3a\u8bbe\u8ba1：<\/p>\n
\u56e0\u679c\u6ce8\u610f\u529b（Causal Attention）：\u901a\u8fc7\u63a9\u7801\u673a\u5236\u786e\u4fdd\u89e3\u7801\u65f6\u4ec5\u5173\u6ce8\u5df2\u751f\u6210\u7684\u8bcd\u5143，\u4fdd\u969c\u81ea\u56de\u5f52\u751f\u6210\u7684\u5408\u7406\u6027；<\/p>\n<\/li>\n
\u591a\u5934\u6ce8\u610f\u529b（Multi-Head Attention）：\u5e76\u884c\u4f7f\u7528\u591a\u4e2a\u6ce8\u610f\u529b\u5934，\u5206\u522b\u5b66\u4e60\u4e0d\u540c\u5b50\u7a7a\u95f4\u7684\u8bed\u4e49\u5173\u7cfb（\u5982\u8bed\u6cd5\u3001\u6307\u4ee3\u3001\u4e3b\u9898\u7b49），\u518d\u878d\u5408\u8f93\u51fa，\u663e\u8457\u63d0\u5347\u6a21\u578b\u8868\u8fbe\u80fd\u529b；<\/p>\n<\/li>\n
\u6ce8\u610f\u529b Dropout：\u5728\u8bad\u7ec3\u65f6\u968f\u673a\u7f6e\u96f6\u90e8\u5206\u6ce8\u610f\u529b\u6743\u91cd，\u9632\u6b62\u6a21\u578b\u8fc7\u5ea6\u4f9d\u8d56\u7279\u5b9a\u8bcd\u5bf9，\u589e\u5f3a\u9c81\u68d2\u6027\u3002<\/p>\n<\/li>\n
\u8fd9\u4e9b\u6539\u8fdb\u5171\u540c\u6784\u6210\u4e86 Transformer \u7684\u5f3a\u5927\u57fa\u7840，\u4f7f\u5176\u4e0d\u4ec5\u80fd\u9ad8\u6548\u5904\u7406\u957f\u5e8f\u5217，\u8fd8\u80fd\u5728\u6d77\u91cf\u6587\u672c\u4e2d\u5b66\u4e60\u590d\u6742\u7684\u8bed\u8a00\u89c4\u5f8b\u3002\u4ece\u201cAttention is All You Need\u201d\u5230\u5982\u4eca\u7684\u5343\u4ebf\u53c2\u6570\u5927\u6a21\u578b，\u81ea\u6ce8\u610f\u529b\u673a\u5236\u59cb\u7ec8\u662f\u9a71\u52a8\u81ea\u7136\u8bed\u8a00\u5904\u7406\u8fdb\u6b65\u7684\u6838\u5fc3\u5f15\u64ce\u3002<\/p>\n<\/p>\n
\u540e\u7eed\u6539\u8fdb\u5c06\u5728\u4e0b\u4e00\u7bc7\u6587\u7ae0\u4e2d\u7ee7\u7eed\u5c55\u5f00\u4ecb\u7ecd，\u6b22\u8fce\u5927\u5bb6\u70b9\u8d5e\u6536\u85cf\u5173\u6ce8，\u4e00\u8d77\u4ea4\u6d41\u5b66\u4e60！<\/p>\n
\u4e0b\u4e00\u7bc7\u6587\u7ae0\u5df2\u4e0a\u4f20：\u4ece\u96f6\u6784\u5efa\u5927\u6a21\u578b\u8bb0\u5f55(\u4e09)\u2014\u2014\u4ece\u96f6\u5b9e\u73b0 Transformer \u6838\u5fc3\u6a21\u5757(\u4e0b)<\/p><\/p>\n","protected":false},"excerpt":{"rendered":"
\u63a5\u524d\u7f6e\u6587\u7ae0：
\n\u4ece\u96f6\u6784\u5efa\u5927\u6a21\u578b\u8bfb\u4e66\u8bb0\u5f55(\u4e00)\u2014\u2014\u7406\u89e3\u5927\u8bed\u8a00\u6a21\u578b
\n\u4ece\u96f6\u6784\u5efa\u5927\u6a21\u578b\u8bfb\u4e66\u8bb0\u5f55(\u4e8c)\u2014\u2014\u5904\u7406\u6587\u672c\u6570\u636e
\n\u4ece\u96f6\u6784\u5efa\u5927\u6a21\u578b\u8bb0\u5f55(\u4e09)\u2014\u2014\u4ece\u96f6\u5b9e\u73b0 Transformer \u6838\u5fc3\u6a21\u5757(\u4e0b)\u5df2\u4e0a\u4f20
\n\u672c\u6587\u5c06\u7ee7\u7eed\u8ba8\u8bba\u6ce8\u610f\u529b\u673a\u5236 \u5f15\u8a00
\n\u4e3a\u4ec0\u4e48\u8981\u5728\u795e\u7ecf\u7f51\u7edc\u4e2d\u4f7f\u7528\u6ce8\u610f\u529b\u673a\u5236\u5462？
\n\u4f20\u7edf\u7684 RNN-based \u673a\u5668\u7ffb\u8bd1\u6a21\u578b\u5728\u751f\u6210\u76ee\u6807\u8bcd\u65f6，\u4f9d\u8d56\u4e8e\u6709\u9650\u7684\u4e0a\u4e0b\u6587\u4fe1\u606f（\u5982\u6700\u540e\u4e00\u4e2a\u9690\u85cf\u72b6\u6001），\u96be\u4ee5\u6709\u6548\u5efa\u6a21\u957f\u8ddd\u79bb<\/p>\n","protected":false},"author":2,"featured_media":59365,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[841,50,86],"topic":[],"class_list":["post-59369","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-server","tag-transformer","tag-50","tag-86"],"yoast_head":"\n