{"id":23936,"date":"2025-01-06T10:33:27","date_gmt":"2025-01-06T02:33:27","guid":{"rendered":"https:\/\/aif.amtbbs.org\/?p=23936"},"modified":"2025-01-06T10:33:27","modified_gmt":"2025-01-06T02:33:27","slug":"openai-%e5%a4%a7%e4%bd%ac%e4%b8%80%e5%8f%a5%e8%af%9d%e9%a2%a0%e8%a6%86%e8%ae%a4%e7%9f%a5%ef%bc%9aai-%e6%90%9e%e5%be%97%e5%a5%bd%e4%b8%8d%e5%a5%bd%ef%bc%8c%e7%9c%9f%e6%ad%a3%e9%87%8d%e8%a6%81%e7%9a%84","status":"publish","type":"post","link":"https:\/\/aif.amtbbs.org\/index.php\/2025\/01\/06\/23936\/","title":{"rendered":"OpenAI \u5927\u4f6c\u4e00\u53e5\u8bdd\u98a0\u8986\u8ba4\u77e5\uff1aAI \u641e\u5f97\u597d\u4e0d\u597d\uff0c\u771f\u6b63\u91cd\u8981\u7684\u662f\u6570\u636e\u96c6\u7684\u9009\u62e9"},"content":{"rendered":"
\"\"<\/div>\n
<\/div>\n
\u5728\u5f97\u51fa\u201cX \u65b9\u6cd5\u65e0\u6548\u201d\u8fd9\u6837\u7684\u7ed3\u8bba\u4e4b\u524d\uff0c\u4f60\u5e94\u8be5\u8c28\u614e\uff0c\u8981\u786e\u4fdd\u7528\u4e8e\u6d4b\u8bd5\u7684\u6570\u636e\u96c6\u786e\u5b9e\u80fd\u591f\u68c0\u9a8c\u8be5\u65b9\u6cd5\u3002<\/div>\n
\n
\n

OpenAI \u7814\u7a76\u5458 Jason Wei \u521a\u521a\u53d1\u8868\u4e86\u4e00\u7bc7\u535a\u6587\uff0c\u63a2\u8ba8\u4e86\u5728\u5f53\u524d AI \u7814\u7a76\u4e2d\u4e00\u9879\u88ab\u4f4e\u4f30\u5374\u81f3\u5173\u91cd\u8981\u7684\u6280\u80fd\uff1a\u627e\u5230\u771f\u6b63\u80fd\u4f53\u73b0\u65b0\u65b9\u6cd5\u6709\u6548\u6027\u7684\u6570\u636e\u96c6<\/strong>\u3002\u8fd9\u9879\u6280\u80fd\u5728\u5341\u5e74\u524d\u8fd8\u4e0d\u5b58\u5728\uff0c\u4f46\u5982\u4eca\u5374\u53ef\u80fd\u6210\u4e3a\u4e00\u9879\u7814\u7a76\u6210\u8d25\u7684\u5173\u952e<\/p>\n

\"\u56fe\u7247\"<\/p>\n

\u4e00\u4e2a\u5e38\u89c1\u7684\u4f8b\u5b50\u662f\u201c\u601d\u7ef4\u94fe (Chain of Thought, CoT) \u5728\u54ea\u4e9b\u6570\u636e\u96c6\u4e0a\u80fd\u63d0\u5347\u6027\u80fd\uff1f\u201d\u3002\u8fd1\u671f\u4e00\u7bc7\u8bba\u6587\u751a\u81f3\u8ba4\u4e3a CoT \u4e3b\u8981\u5bf9\u6570\u5b66\u548c\u903b\u8f91\u4efb\u52a1\u6709\u5e2e\u52a9\u3002Wei \u8ba4\u4e3a\u8fd9\u79cd\u89c2\u70b9\u662f\u7f3a\u4e4f\u60f3\u8c61\u529b\u548c\u591a\u6837\u5316\u8bc4\u4f30\u7684\u8868\u73b0\u3002\u5982\u679c\u6211\u4eec\u7b80\u5355\u5730\u5728 100 \u4e2a\u968f\u673a\u7528\u6237\u804a\u5929\u63d0\u793a\u4e0a\u6d4b\u8bd5 CoT \u6a21\u578b\uff0c\u53ef\u80fd\u770b\u4e0d\u5230\u660e\u663e\u7684\u5dee\u5f02\uff0c\u4f46\u8fd9\u4ec5\u4ec5\u662f\u56e0\u4e3a\u8fd9\u4e9b\u63d0\u793a\u672c\u6765\u5c31\u4e0d\u9700\u8981 CoT \u5c31\u80fd\u89e3\u51b3\u3002\u4e8b\u5b9e\u4e0a\uff0c\u5728\u4e00\u4e9b\u7279\u5b9a\u7684\u6570\u636e\u5b50\u96c6\u4e0a\uff0cCoT \u80fd\u5e26\u6765\u5de8\u5927\u63d0\u5347\u2014\u2014\u4f8b\u5982\u6570\u5b66\u548c\u7f16\u7a0b\u4efb\u52a1\uff0c\u4ee5\u53ca\u4efb\u4f55\u9a8c\u8bc1\u4e0d\u5bf9\u79f0\u7684\u4efb\u52a1<\/p>\n

\u6362\u53e5\u8bdd\u8bf4\uff0c\u5728\u65ad\u8a00\u201cX \u65b9\u6cd5\u65e0\u6548\u201d\u4e4b\u524d\uff0c\u9700\u8981\u786e\u4fdd\u7528\u4e8e\u6d4b\u8bd5\u7684\u6570\u636e\u96c6\u786e\u5b9e\u80fd\u591f\u4f53\u73b0\u8be5\u65b9\u6cd5\u7684\u4ef7\u503c<\/p>\n

Jason Wei \u7684\u8fd9\u7bc7\u535a\u6587\u5f3a\u8c03\u4e86\u5728\u5f53\u524d AI \u7814\u7a76\u4e2d\uff0c\u968f\u7740\u6a21\u578b\u80fd\u529b\u7684\u4e0d\u65ad\u589e\u5f3a\uff0c\u6570\u636e\u96c6\u7684\u9009\u62e9\u53d8\u5f97\u66f4\u52a0\u5fae\u5999\u548c\u5173\u952e\u3002<\/p>\n

\u5168\u6587<\/h2>\n

Jason Wei \u4eba\u5de5\u667a\u80fd\u7814\u7a76\u5458 @OpenAI<\/p>\n

An underrated but occasionally make-or-break skill in AI research (that didn\u2019t really exist ten years ago) is the ability to find a dataset that actually exercises a new method you are working on. Back in the day when the bottleneck in AI was learning, many methods were dataset-agnostic; for example, a better optimizer would be expected to improve on both ImageNet and CIFAR-10. Nowadays language models are so multi-task that the answer to whether something works is almost always \u201cit depends on the dataset\u201d.<\/strong><\/p>\n

\u5728\u4eba\u5de5\u667a\u80fd\u7814\u7a76\u4e2d\uff0c\u4e00\u9879\u88ab\u4f4e\u4f30\u4f46\u5076\u5c14\u80fd\u51b3\u5b9a\u6210\u8d25\u7684\u6280\u80fd\uff08\u5341\u5e74\u524d\u8fd8\u4e0d\u5b58\u5728\uff09\u662f\u627e\u5230\u4e00\u4e2a\u771f\u6b63\u80fd\u68c0\u9a8c\u4f60\u6b63\u5728\u7814\u7a76\u7684\u65b0\u65b9\u6cd5\u7684\u6570\u636e\u96c6\u7684\u80fd\u529b\u3002\u5728\u8fc7\u53bb\uff0c\u4eba\u5de5\u667a\u80fd\u7684\u74f6\u9888\u662f\u5b66\u4e60\uff0c\u8bb8\u591a\u65b9\u6cd5\u4e0e\u6570\u636e\u96c6\u65e0\u5173\uff1b\u4f8b\u5982\uff0c\u4e00\u4e2a\u66f4\u597d\u7684\u4f18\u5316\u5668\u5e94\u8be5\u5728 ImageNet \u548c CIFAR-10 \u4e0a\u90fd\u80fd\u63d0\u9ad8\u6027\u80fd\u3002\u5982\u4eca\uff0c\u8bed\u8a00\u6a21\u578b\u5177\u6709\u5982\u6b64\u5f3a\u5927\u7684\u591a\u4efb\u52a1\u5904\u7406\u80fd\u529b\uff0c\u4ee5\u81f3\u4e8e\u67d0\u4ef6\u4e8b\u662f\u5426\u6709\u6548\uff0c\u7b54\u6848\u51e0\u4e4e\u603b\u662f\u201c\u53d6\u51b3\u4e8e\u6570\u636e\u96c6\u201d\u3002<\/p>\n

A common example of this is the question, \u201con what datasets does chain of thought improve performance?\u201d A recent paper even argued (will link below) that CoT mainly helps on math\/logic, and I think that is both a failure of imagination and a lack of diverse evals. Naively you might try CoT models on 100 random user chat prompts and not see much difference, but this is because the prompts were already solvable without CoT. In fact there is a small and very important slice of data where CoT makes a big difference\u2014the obvious examples are math and coding, but include almost any task with asymmetry of verification. For example, generating a poem that fits a list of constraints is hard on the first try but much easier if you can draft and revise using CoT.<\/strong><\/p>\n

\u4e00\u4e2a\u5e38\u89c1\u7684\u4f8b\u5b50\u662f\u8fd9\u4e2a\u95ee\u9898\uff1a\u201c\u601d\u7ef4\u94fe (Chain of Thought, CoT) \u5728\u54ea\u4e9b\u6570\u636e\u96c6\u4e0a\u80fd\u63d0\u9ad8\u6027\u80fd\uff1f\u201d \u4e00\u7bc7\u6700\u8fd1\u7684\u8bba\u6587\u751a\u81f3\u8ba4\u4e3a\uff08\u94fe\u63a5\u9644\u540e\uff09CoT \u4e3b\u8981\u6709\u52a9\u4e8e\u6570\u5b66\/\u903b\u8f91\uff0c\u6211\u8ba4\u4e3a\u8fd9\u65e2\u662f\u60f3\u8c61\u529b\u7684\u5931\u8d25\uff0c\u4e5f\u662f\u7f3a\u4e4f\u591a\u6837\u5316\u8bc4\u4f30\u7684\u7ed3\u679c\u3002\u4f60\u53ef\u80fd\u4f1a\u7b80\u5355\u5730\u5728 100 \u4e2a\u968f\u673a\u7528\u6237\u804a\u5929\u63d0\u793a\u4e0a\u5c1d\u8bd5 CoT \u6a21\u578b\uff0c\u5374\u770b\u4e0d\u5230\u592a\u5927\u7684\u533a\u522b\uff0c\u4f46\u8fd9\u662f\u56e0\u4e3a\u8fd9\u4e9b\u63d0\u793a\u5728\u6ca1\u6709 CoT \u7684\u60c5\u51b5\u4e0b\u5df2\u7ecf\u53ef\u4ee5\u89e3\u51b3\u3002\u4e8b\u5b9e\u4e0a\uff0c\u5728\u4e00\u5c0f\u90e8\u5206\u975e\u5e38\u91cd\u8981\u7684\u6570\u636e\u4e0a\uff0cCoT \u53ef\u4ee5\u5e26\u6765\u5f88\u5927\u7684\u4e0d\u540c\u2014\u2014\u660e\u663e\u7684\u4f8b\u5b50\u662f\u6570\u5b66\u548c\u7f16\u7801\uff0c\u4f46\u4e5f\u5305\u62ec\u51e0\u4e4e\u4efb\u4f55\u5177\u6709\u9a8c\u8bc1\u4e0d\u5bf9\u79f0\u6027\u7684\u4efb\u52a1\u3002\u4f8b\u5982\uff0c\u751f\u6210\u4e00\u9996\u7b26\u5408\u4e00\u7cfb\u5217\u7ea6\u675f\u6761\u4ef6\u7684\u8bd7\u6b4c\uff0c\u7b2c\u4e00\u6b21\u5c1d\u8bd5\u65f6\u5f88\u56f0\u96be\uff0c\u4f46\u5982\u679c\u4f60\u53ef\u4ee5\u4f7f\u7528 CoT \u8fdb\u884c\u8349\u62df\u548c\u4fee\u6539\uff0c\u5c31\u4f1a\u5bb9\u6613\u5f97\u591a<\/p>\n

As another made-up example, let\u2019s say you want to know if browsing improves performance on geology exams. Maybe using browsing on some random geology dataset didn\u2019t improve performance. The important thing to do here would be to see if the without-browsing model was actually suffering due to lack of world knowledge\u2014if it wasn\u2019t, then this was the wrong dataset to try browsing on.<\/strong><\/p>\n

\u518d\u4e3e\u4e00\u4e2a\u865a\u6784\u7684\u4f8b\u5b50\uff0c\u5047\u8bbe\u4f60\u60f3\u77e5\u9053\u6d4f\u89c8\u7f51\u9875\u662f\u5426\u80fd\u63d0\u9ad8\u5730\u8d28\u5b66\u8003\u8bd5\u7684\u6210\u7ee9\u3002\u4e5f\u8bb8\u5728\u4e00\u4e9b\u968f\u673a\u7684\u5730\u8d28\u5b66\u6570\u636e\u96c6\u4e0a\u4f7f\u7528\u6d4f\u89c8\u5e76\u6ca1\u6709\u63d0\u9ad8\u6027\u80fd\u3002\u8fd9\u91cc\u91cd\u8981\u7684\u662f\u8981\u67e5\u770b\u6ca1\u6709\u6d4f\u89c8\u529f\u80fd\u7684\u6a21\u578b\u662f\u5426\u771f\u7684\u56e0\u4e3a\u7f3a\u4e4f\u4e16\u754c\u77e5\u8bc6\u800c\u8868\u73b0\u4e0d\u4f73\u2014\u2014\u5982\u679c\u4e0d\u662f\uff0c\u90a3\u4e48\u8fd9\u5c31\u4e0d\u662f\u4e00\u4e2a\u6d4b\u8bd5\u6d4f\u89c8\u529f\u80fd\u7684\u6b63\u786e\u6570\u636e\u96c6\u3002<\/p>\n

In other words you should hesitate to draw a conclusion like \u201cX method doesn\u2019t work\u201d without ensuring that the dataset used for testing actually exercises that method. The inertia from five years ago is to take existing benchmarks and try to solve them, but nowadays there is a lot more flexibility and sometimes it even makes sense to create a custom dataset to showcase the initial usefulness of an idea. Obviously the danger with doing this is that a contrived dataset may not represent a substantial portion of user queries. But if the method is in principle general I think this is a good way to start and something people should do more often.<\/strong><\/p>\n

\u6362\u53e5\u8bdd\u8bf4\uff0c\u5728\u5f97\u51fa\u201cX \u65b9\u6cd5\u65e0\u6548\u201d\u8fd9\u6837\u7684\u7ed3\u8bba\u4e4b\u524d\uff0c\u4f60\u5e94\u8be5\u8c28\u614e\uff0c\u8981\u786e\u4fdd\u7528\u4e8e\u6d4b\u8bd5\u7684\u6570\u636e\u96c6\u786e\u5b9e\u80fd\u591f\u68c0\u9a8c\u8be5\u65b9\u6cd5\u3002\u4e94\u5e74\u524d\u7684\u60ef\u6027\u662f\u91c7\u7528\u73b0\u6709\u7684\u57fa\u51c6\u6570\u636e\u96c6\u5e76\u5c1d\u8bd5\u89e3\u51b3\u5b83\u4eec\uff0c\u4f46\u5982\u4eca\u7684\u7075\u6d3b\u6027\u8981\u5927\u5f97\u591a\uff0c\u6709\u65f6\u751a\u81f3\u53ef\u4ee5\u521b\u5efa\u4e00\u4e2a\u81ea\u5b9a\u4e49\u6570\u636e\u96c6\u6765\u5c55\u793a\u4e00\u4e2a\u60f3\u6cd5\u7684\u521d\u6b65\u5b9e\u7528\u6027\u3002\u663e\u7136\uff0c\u8fd9\u6837\u505a\u7684\u5371\u9669\u5728\u4e8e\uff0c\u4eba\u4e3a\u8bbe\u8ba1\u7684\u6570\u636e\u96c6\u53ef\u80fd\u65e0\u6cd5\u4ee3\u8868\u7528\u6237\u67e5\u8be2\u7684\u5f88\u5927\u4e00\u90e8\u5206\u3002\u4f46\u5982\u679c\u8be5\u65b9\u6cd5\u5728\u539f\u5219\u4e0a\u662f\u901a\u7528\u7684\uff0c\u6211\u8ba4\u4e3a\u8fd9\u662f\u4e00\u4e2a\u597d\u7684\u5f00\u59cb\uff0c\u4e5f\u662f\u4eba\u4eec\u5e94\u8be5\u66f4\u591a\u5c1d\u8bd5\u7684\u4e8b\u60c5<\/p>\n

\u6587\u7ae0\u6765\u81ea\uff1a51CTO<\/p>\n<\/div>\n<\/div>\n

<\/div>\n

<\/g><\/svg><\/i> \"Loading\"<\/p>\n

<\/div>\n","protected":false},"excerpt":{"rendered":"

\u5728\u5f97\u51fa\u201cX \u65b9\u6cd5\u65e0\u6548\u201d\u8fd9\u6837\u7684\u7ed3\u8bba\u4e4b\u524d\uff0c\u4f60\u5e94\u8be5\u8c28\u614e\uff0c\u8981\u786e\u4fdd\u7528\u4e8e\u6d4b\u8bd5\u7684\u6570\u636e\u96c6\u786e\u5b9e\u80fd\u591f\u68c0\u9a8c\u8be5\u65b9\u6cd5\u3002 OpenAI \u7814 […]<\/p>\n

<\/div>\n

<\/g><\/svg><\/i> \"Loading\"<\/p>\n

<\/div>\n","protected":false},"author":56,"featured_media":23938,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[23,17,20,80],"tags":[124,887,234],"class_list":["post-23936","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-23","category-17","category-20","category-80","tag-ai","tag-887","tag-234"],"_links":{"self":[{"href":"https:\/\/aif.amtbbs.org\/index.php\/wp-json\/wp\/v2\/posts\/23936","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aif.amtbbs.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aif.amtbbs.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aif.amtbbs.org\/index.php\/wp-json\/wp\/v2\/users\/56"}],"replies":[{"embeddable":true,"href":"https:\/\/aif.amtbbs.org\/index.php\/wp-json\/wp\/v2\/comments?post=23936"}],"version-history":[{"count":1,"href":"https:\/\/aif.amtbbs.org\/index.php\/wp-json\/wp\/v2\/posts\/23936\/revisions"}],"predecessor-version":[{"id":23939,"href":"https:\/\/aif.amtbbs.org\/index.php\/wp-json\/wp\/v2\/posts\/23936\/revisions\/23939"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/aif.amtbbs.org\/index.php\/wp-json\/wp\/v2\/media\/23938"}],"wp:attachment":[{"href":"https:\/\/aif.amtbbs.org\/index.php\/wp-json\/wp\/v2\/media?parent=23936"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aif.amtbbs.org\/index.php\/wp-json\/wp\/v2\/categories?post=23936"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aif.amtbbs.org\/index.php\/wp-json\/wp\/v2\/tags?post=23936"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}