{"id":25918,"date":"2025-05-20T09:18:00","date_gmt":"2025-05-20T01:18:00","guid":{"rendered":"https:\/\/aif.amtbbs.org\/?p=25918"},"modified":"2025-05-20T09:18:00","modified_gmt":"2025-05-20T01:18:00","slug":"transformer-%e6%a8%a1%e5%9e%8b%e7%bb%93%e6%9e%84%e8%af%a6%e8%a7%a3%e5%8f%8a%e4%bb%a3%e7%a0%81%e5%ae%9e%e7%8e%b0","status":"publish","type":"post","link":"https:\/\/aif.amtbbs.org\/index.php\/2025\/05\/20\/25918\/","title":{"rendered":"Transformer \u6a21\u578b\u7ed3\u6784\u8be6\u89e3\u53ca\u4ee3\u7801\u5b9e\u73b0!"},"content":{"rendered":"
\n
\"\"
3D Rendering futuristic robot technology development, artificial intelligence AI, and machine learning concept. Global robotic bionic science research for future of human life.<\/figcaption><\/figure>\n<\/div>\n
<\/div>\n
Transformer \u9ed8\u8ba4\u90fd\u662f\u5927\u6a21\u578b\uff0c\u9664\u4e86\u4e00\u4e9b\u7279\u4f8b\uff08\u5982 DistilBERT\uff09\u5916\uff0c\u5b9e\u73b0\u66f4\u597d\u6027\u80fd\u7684\u4e00\u822c\u7b56\u7565\u662f\u589e\u52a0\u6a21\u578b\u7684\u5927\u5c0f\u4ee5\u53ca\u9884\u8bad\u7ec3\u7684\u6570\u636e\u91cf\u3002<\/div>\n
\n
\n

\u4e00\u3001Transformer\u7b80\u8981\u53d1\u5c55\u53f2<\/h3>\n

\u4ee5\u4e0b\u662fTransformer\u6a21\u578b\u53d1\u5c55\u5386\u53f2\u4e2d\u7684\u5173\u952e\u8282\u70b9\uff1a<\/p>\n

 <\/p>\n

<\/p>\n

 <\/p>\n

Transformer\u67b6\u6784\u4e8e2017\u5e746\u6708\u63a8\u51fa\u3002\u539f\u672c\u7814\u7a76\u7684\u91cd\u70b9\u662f\u7ffb\u8bd1\u4efb\u52a1\u3002\u968f\u540e\u63a8\u51fa\u4e86\u51e0\u4e2a\u6709\u5f71\u54cd\u529b\u7684\u6a21\u578b\uff0c\u5305\u62ec\uff1a<\/p>\n\n\n\n\n<\/colgroup>\n\n\n\n\n\n\n\n\n\n
\u65f6\u95f4<\/td>\n\u6a21\u578b<\/td>\n\u7b80\u8981\u8bf4\u660e<\/td>\n<\/tr>\n
2017 \u5e74 6 \u6708<\/td>\n\u300cTransformer\u300d<\/td>\nGoogle \u9996\u6b21\u63d0\u51fa\u57fa\u4e8e Attention \u7684\u6a21\u578b\uff0c\u7528\u4e8e\u673a\u5668\u7ffb\u8bd1\u4efb\u52a1<\/td>\n<\/tr>\n
2018 \u5e74 6 \u6708<\/td>\n\u300cGPT\u300d<\/td>\n\u7b2c\u4e00\u4e2a\u4f7f\u7528 Transformer \u89e3\u7801\u5668\u6a21\u5757\u8fdb\u884c\u9884\u8bad\u7ec3\u7684\u8bed\u8a00\u6a21\u578b\uff0c\u9002\u7528\u4e8e\u591a\u79cd NLP \u4efb\u52a1<\/td>\n<\/tr>\n
2018 \u5e74 10 \u6708<\/td>\n\u300cBERT\u300d<\/td>\n\u4f7f\u7528 Transformer \u7f16\u7801\u5668\u6a21\u5757\uff0c\u901a\u8fc7\u63a9\u7801\u8bed\u8a00\u5efa\u6a21\u751f\u6210\u66f4\u5f3a\u5927\u7684\u53e5\u5b50\u8868\u793a<\/td>\n<\/tr>\n
2019 \u5e74 2 \u6708<\/td>\n\u300cGPT-2\u300d<\/td>\n\u66f4\u5927\u66f4\u5f3a\u7684 GPT \u7248\u672c\uff0c\u7531\u4e8e\u6f5c\u5728\u98ce\u9669\u672a\u7acb\u5373\u53d1\u5e03\uff0c\u5177\u5907\u51fa\u8272\u7684\u6587\u672c\u751f\u6210\u80fd\u529b<\/td>\n<\/tr>\n
2019 \u5e74 10 \u6708<\/td>\n\u300cDistilBERT\u300d<\/td>\nBERT \u7684\u8f7b\u91cf\u5316\u7248\u672c\uff0c\u5728\u4fdd\u7559 97% \u6027\u80fd\u7684\u540c\u65f6\uff0c\u901f\u5ea6\u66f4\u5feb\u3001\u5185\u5b58\u5360\u7528\u66f4\u4f4e<\/td>\n<\/tr>\n
2019 \u5e74 10 \u6708<\/td>\n\u300cBART\u3001T5\u300d<\/td>\n\u4f7f\u7528\u5b8c\u6574\u7684 Encoder-Decoder \u67b6\u6784\uff0c\u5728\u5404\u79cd NLP \u4efb\u52a1\u4e2d\u8868\u73b0\u4f18\u5f02<\/td>\n<\/tr>\n
2020 \u5e74 5 \u6708<\/td>\n\u300cGPT-3\u300d<\/td>\n\u8d85\u5927\u89c4\u6a21\u8bed\u8a00\u6a21\u578b\uff0c\u652f\u6301\u201c\u96f6\u6837\u672c\u5b66\u4e60\u201d\uff0c\u65e0\u9700\u5fae\u8c03\u5373\u53ef\u5b8c\u6210\u65b0\u4efb\u52a1<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

\u8fd9\u4e2a\u5217\u8868\u5e76\u4e0d\u5168\u9762\uff0c\u53ea\u662f\u4e3a\u4e86\u7a81\u51fa\u4e00\u4e9b\u4e0d\u540c\u7c7b\u578b\u7684 Transformer \u6a21\u578b\u3002\u5927\u4f53\u4e0a\uff0c\u5b83\u4eec\u53ef\u4ee5\u5206\u4e3a\u4e09\u7c7b\uff1a<\/p>\n\n\n\n\n\n<\/colgroup>\n\n\n\n\n\n
\u7c7b\u522b<\/td>\n\u6784\u6210<\/td>\n\u7279\u70b9<\/td>\n\u5178\u578b\u6a21\u578b<\/td>\n<\/tr>\n
\u300cGPT-like\u300d<\/p>\n

\uff08\u81ea\u56de\u5f52 Transformer\uff09<\/td>\n

\u53ea\u4f7f\u7528\u89e3\u7801\u5668<\/td>\n\u81ea\u56de\u5f52\u65b9\u5f0f\u9884\u6d4b\u4e0b\u4e00\u4e2a\u8bcd\uff0c\u9002\u5408\u6587\u672c\u751f\u6210\u4efb\u52a1<\/td>\nGPT\u3001GPT-2\u3001GPT-3<\/td>\n<\/tr>\n
\u300cBERT-like\u300d<\/p>\n

\uff08\u81ea\u52a8\u7f16\u7801 Transformer\uff09<\/td>\n

\u53ea\u4f7f\u7528\u7f16\u7801\u5668<\/td>\n\u63a9\u7801\u673a\u5236\u5b66\u4e60\u4e0a\u4e0b\u6587\u8868\u793a\uff0c\u9002\u5408\u7406\u89e3\u7c7b\u4efb\u52a1\u5982\u95ee\u7b54\u3001\u60c5\u611f\u5206\u6790<\/td>\nBERT\u3001RoBERTa\u3001DistilBERT<\/td>\n<\/tr>\n
\u300cBART\/T5-like\u300d<\/p>\n

\uff08\u5e8f\u5217\u5230\u5e8f\u5217 Transformer\uff09<\/td>\n

\u7f16\u7801\u5668 + \u89e3\u7801\u5668<\/td>\n\u5b8c\u6574\u7684 encoder-decoder \u67b6\u6784\uff0c\u9002\u5408\u7ffb\u8bd1\u3001\u6458\u8981\u7b49\u751f\u6210+\u7406\u89e3\u7ed3\u5408\u7684\u4efb\u52a1<\/td>\nBART\u3001T5<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Transformer \u9ed8\u8ba4\u90fd\u662f\u5927\u6a21\u578b\uff0c\u9664\u4e86\u4e00\u4e9b\u7279\u4f8b\uff08\u5982 DistilBERT\uff09\u5916\uff0c\u5b9e\u73b0\u66f4\u597d\u6027\u80fd\u7684\u4e00\u822c\u7b56\u7565\u662f\u589e\u52a0\u6a21\u578b\u7684\u5927\u5c0f\u4ee5\u53ca\u9884\u8bad\u7ec3\u7684\u6570\u636e\u91cf\u3002\u5176\u4e2d\uff0cGPT-2 \u662f\u4f7f\u7528\u300ctransformer \u89e3\u7801\u5668\u6a21\u5757\u300d\u6784\u5efa\u7684\uff0c\u800c BERT \u5219\u662f\u901a\u8fc7\u300ctransformer \u7f16\u7801\u5668\u300d\u6a21\u5757\u6784\u5efa\u7684\u3002<\/p>\n

<\/p>\n

 <\/p>\n

\u4e8c\u3001Transformer \u6574\u4f53\u67b6\u6784<\/h3>\n

\u8bba\u6587\u4e2d\u7ed9\u51fa\u7528\u4e8e\u4e2d\u82f1\u6587\u7ffb\u8bd1\u4efb\u52a1\u7684 Transformer \u6574\u4f53\u67b6\u6784\u5982\u4e0b\u56fe\u6240\u793a\uff1a<\/p>\n

 <\/p>\n

<\/p>\n

\u53ef\u4ee5\u770b\u51faTransformer\u67b6\u6784\u7531Encoder\u548cDecoder\u4e24\u4e2a\u90e8\u5206\u7ec4\u6210\uff1a\u5176\u4e2dEncoder\u548cDecoder\u90fd\u662f\u7531N=6\u4e2a\u76f8\u540c\u7684\u5c42\u5806\u53e0\u800c\u6210\u3002Multi-Head Attention \u7ed3\u6784\u662f Transformer \u67b6\u6784\u7684\u6838\u5fc3\u7ed3\u6784\uff0c\u5176\u7531\u591a\u4e2a Self-Attention \u7ec4\u6210\u7684\u3002\u5176\u4e2d\uff0c<\/p>\n\n\n\n\n\n<\/colgroup>\n\n\n\n\n
\u90e8\u4ef6<\/td>\n\u7ed3\u6784<\/td>\n\u5c42\u6570<\/td>\n\u4e3b\u8981\u6a21\u5757<\/td>\n<\/tr>\n
Encoder<\/td>\n\u7f16\u7801\u5668\u5c42\u5806\u53e0<\/td>\nN=6\u5c42<\/td>\nSelf-Attention+Feed Forward<\/td>\n<\/tr>\n
Decoder<\/td>\n\u89e3\u7801\u5668\u5c42\u5806\u53e0<\/td>\nN=6\u5c42<\/td>\nSelf-Attention+Encoder-Decoder Attention+Feed Forward<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Transformer \u67b6\u6784\u66f4\u8be6\u7ec6\u7684\u53ef\u89c6\u5316\u56fe\u5982\u4e0b\u6240\u793a:<\/p>\n

 <\/p>\n

<\/p>\n

1. \u8f93\u5165\u6a21\u5757<\/h4>\n

(1) Tokenizer\u9884\u5904\u7406<\/strong><\/p>\n

\u5728\u57fa\u4e8eTransformer\u7684\u5927\u6a21\u578bLLM\u4e2d\uff0c\u8f93\u5165\u901a\u5e38\u4e3a\u5b57\u7b26\u4e32\u6587\u672c\u3002\u7531\u4e8e\u6a21\u578b\u65e0\u6cd5\u76f4\u63a5\u5904\u7406\u81ea\u7136\u8bed\u8a00\uff0c\u56e0\u6b64\u9700\u8981\u501f\u52a9Tokenizer\u5bf9\u8f93\u5165\u8fdb\u884c\u9884\u5904\u7406\u3002\u5177\u4f53\u6d41\u7a0b\u5982\u4e0b\uff1a<\/p>\n

    \n
  • \u5206\u8bcd\uff08Tokenization\uff09\uff1a\u5c06\u8f93\u5165\u6587\u672c\u6309\u89c4\u5219\u5207\u5206\u4e3a\u4e00\u4e2a\u4e2a\u8bcd\u5143\uff08token\uff09\uff0c\u5982\u5355\u8bcd\u3001\u5b50\u8bcd\u6216\u7279\u6b8a\u7b26\u53f7\u3002<\/li>\n
  • \u8bcd\u8868\u6620\u5c04\uff08Vocabulary Mapping\uff09\uff1a\u6bcf\u4e2a token \u88ab\u6620\u5c04\u5230\u4e00\u4e2a\u552f\u4e00\u7684\u6574\u6570 ID\uff0c\u8be5 ID \u6765\u81ea\u9884\u8bad\u7ec3\u6a21\u578b\u6240\u4f7f\u7528\u7684\u8bcd\u6c47\u8868\u3002<\/li>\n
  • \u751f\u6210 input_ids \u5411\u91cf\uff08\u77e9\u9635\uff09\uff1a\u6700\u7ec8\u8f93\u51fa\u662f\u4e00\u4e2a\u7531 token ID \u6784\u6210\u7684\u5411\u91cf\uff08\u6216\u77e9\u9635\uff09\uff0c\u4f5c\u4e3a\u6a21\u578b\u8f93\u5165\u3002<\/li>\n<\/ul>\n

    \u4ee5\u4e0b\u662f\u4ee5 Hugging Face \u7684 transformers \u5e93\u4e3a\u4f8b\uff0c\u5c55\u793a\u5982\u4f55\u4f7f\u7528 BertTokenizer \u548c BertModel \u5b8c\u6210\u8f93\u5165\u6587\u672c\u7684\u9884\u5904\u7406\u548c\u7f16\u7801\uff1a<\/p>\n

    \n
    \n