{"topic":{"id":14155,"title":"Zeyuan Zhu ICML 2024 Tutorial: Physics of Language Models","slug":"topic","userId":4668,"categoryId":11,"createdAt":"2025-01-22 09:51:48.935135","bumpedAt":"2025-01-22 09:58:40.66687","views":245,"postsCount":4,"highestPostNumber":4,"likeCount":5,"closed":false,"pinned":false,"pinnedGlobally":false,"pinnedAt":null,"pinnedUntil":null,"isBanner":false,"excerpt":"https://www.youtube.com/watch?v=yBL7J0kgldU 目前看到的最好的 LLM 研究（2h 讲了他们在 Meta 的很多工作）一些截图 LLM 在输出第一句话之前就在内部完成了拓扑排序（逻辑链构造）和人类的逻辑推理习惯不同，LLM 在回答问题前预先计算了所有的依赖，人类是遇到问题后只去寻找问题相关的只看模型参数多少是不对的，推理能力和模型深度有很大关系 GPT 处理语言实际上是在做 2 种动态规划…","deletedAt":null,"isAnonymous":false,"isPrivate":false,"isMembersOnly":false,"category":{"id":11,"name":"深入交流","slug":"distilled","color":"c01c28"},"author":{"id":4668,"username":"IO","name":null,"avatarUrl":null},"realAuthor":{"id":4668,"username":"IO","name":null,"avatarUrl":null},"tags":[]},"posts":[{"id":45704,"postNumber":1,"raw":"https://www.youtube.com/watch?v=yBL7J0kgldU\n目前看到的最好的 LLM 研究（2h 讲了他们在 Meta 的很多工作）\n\n\n一些截图\n![548765b5bcfc6e58b100e35c2a9799af|550x500](/uploads/image/f2/64/f264810b2fff47bffd5e2f2a61a9abbc62ae3d5c8bc45a2298874cc7dfad334f.avif)\nLLM 在输出第一句话之前就在内部完成了拓扑排序（逻辑链构造）\n\n![82a9d821663965a46a0ad70c5371f7c6|690x386](/uploads/image/31/aa/31aa9fdec4b8037c5c498d27f8f5c688730f9aa27c312e199884aaf712efea6b.avif)\n和人类的逻辑推理习惯不同，LLM 在回答问题前预先计算了所有的依赖，人类是遇到问题后只去寻找问题相关的\n\n![8d513cbbfe606a89ba3ec758e6938685|558x499](/uploads/image/1a/db/1adbe558d27b119e674761d58f948281a92a4f470cff765130a608bf90881f03.avif)\n只看模型参数多少是不对的，推理能力和模型深度有很大关系\n\n[grid mode=\"tile\"]\n![b4428b20875e5261862a6e83091fbc83|590x500](/uploads/image/3d/a2/3da2a4368c7a6ce27ba2119a101930ce1229447e969f07526480fead7bfa0727.avif)\n![0f45778aaf37650b96c786f44dc03a8b|544x500](/uploads/image/be/98/be98263dcdd034505da173e02e87cf27dd020b9f49fde6b5a27025a9bd5010c8.avif)\n![image|690x178](/uploads/image/fd/ff/fdffc3b9ad7cec4dc3117e5888fdfb7f764739cd1361187e4c8879cc61c316e6.avif)\n![717bf636406b63862a6ba8f335c49fb9|690x360](/uploads/image/2e/d5/2ed5ac8db669cc93a446001a0b652c4a0845b2db93cff47f694a45edf09bac94.avif)\n[/grid]\n\nGPT 处理语言实际上是在做 2 种动态规划，而这刚好是 IOI 2005 的 R 题，作者刚好打了这场比赛（rk19 Zeyuan Zhu）没有解决\n\n\n还有很多其他有意思的结论，忘了截图","cooked":"\n目前看到的最好的 LLM 研究（2h 讲了他们在 Meta 的很多工作）\n

一些截图
\n $\"548765b5bcfc6e58b100e35c2a9799af\"$
\nLLM 在输出第一句话之前就在内部完成了拓扑排序（逻辑链构造）

$\"82a9d821663965a46a0ad70c5371f7c6\"$
\n和人类的逻辑推理习惯不同，LLM 在回答问题前预先计算了所有的依赖，人类是遇到问题后只去寻找问题相关的

$\"8d513cbbfe606a89ba3ec758e6938685\"$
\n只看模型参数多少是不对的，推理能力和模型深度有很大关系

$\"b4428b20875e5261862a6e83091fbc83\"$
\n $\"0f45778aaf37650b96c786f44dc03a8b\"$
\n $\"image\"$
\n $\"717bf636406b63862a6ba8f335c49fb9\"$

GPT 处理语言实际上是在做 2 种动态规划，而这刚好是 IOI 2005 的 R 题，作者刚好打了这场比赛（rk19 Zeyuan Zhu）没有解决

还有很多其他有意思的结论，忘了截图

\n","createdAt":"2025-01-22 09:51:49.20045","updatedAt":"2025-01-22 09:51:49.20045","replyToPostNumber":null,"replyCount":0,"likeCount":2,"wordCount":204,"liked":false,"bookmarked":false,"polls":[],"reactions":[{"reaction":"❤️","count":2,"users":[{"id":4668,"username":"IO","name":null},{"id":6205,"username":"uika_winwing","name":null}],"hasReacted":false}],"wiki":false,"version":1,"publicVersion":1,"deletedAt":null,"isAnonymous":false,"user":{"id":4668,"username":"IO","name":null,"avatarUrl":null,"admin":true,"moderator":false,"signatureRaw":"","signatureCooked":null}},{"id":45705,"postNumber":2,"raw":"兄弟，你为什么要匿名？","cooked":"

兄弟，你为什么要匿名？

\n","createdAt":"2025-01-22 09:53:16.689975","updatedAt":"2025-01-22 09:53:16.689975","replyToPostNumber":null,"replyCount":0,"likeCount":1,"wordCount":11,"liked":false,"bookmarked":false,"polls":[],"reactions":[{"reaction":"❤️","count":1,"users":[{"id":4668,"username":"IO","name":null}],"hasReacted":false}],"wiki":false,"version":1,"publicVersion":1,"deletedAt":null,"isAnonymous":false,"user":{"id":4668,"username":"IO","name":null,"avatarUrl":null,"admin":true,"moderator":false,"signatureRaw":"","signatureCooked":null}},{"id":45706,"postNumber":3,"raw":"学术交流有匿名审稿环节，为了尽可能避免主观身份的污染","cooked":"

学术交流有匿名审稿环节，为了尽可能避免主观身份的污染

\n","createdAt":"2025-01-22 09:54:38.554382","updatedAt":"2025-01-22 09:54:38.554382","replyToPostNumber":null,"replyCount":0,"likeCount":1,"wordCount":26,"liked":false,"bookmarked":false,"polls":[],"reactions":[{"reaction":"❤️","count":1,"users":[{"id":4668,"username":"IO","name":null}],"hasReacted":false}],"wiki":false,"version":1,"publicVersion":1,"deletedAt":null,"isAnonymous":false,"user":{"id":4668,"username":"IO","name":null,"avatarUrl":null,"admin":true,"moderator":false,"signatureRaw":"","signatureCooked":null}},{"id":45707,"postNumber":4,"raw":"[quote=\"shared_user, post:1, topic:14155, username:anonymous_coward\"]\nGPT 处理语言实际上是在做 2 种动态规划，而这刚好是 IOI 2005 的 R 题\n[/quote]\n注：这里的 GPT2 是作者预训练的，vocabulary 和 dataset 限制在了自己可控的范围，因此是没有教科书上关于 dp 的任何知识，是纯粹从数据学习到的方法","cooked":"

\n\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\t\t\tshared_user:\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\t

\n
GPT 处理语言实际上是在做 2 种动态规划，而这刚好是 IOI 2005 的 R 题
\n

\n\t\t\t\t\t\t

注：这里的 GPT2 是作者预训练的，vocabulary 和 dataset 限制在了自己可控的范围，因此是没有教科书上关于 dp 的任何知识，是纯粹从数据学习到的方法

\n","createdAt":"2025-01-22 09:58:40.66687","updatedAt":"2025-01-22 09:58:40.66687","replyToPostNumber":null,"replyCount":0,"likeCount":1,"wordCount":91,"liked":false,"bookmarked":false,"polls":[],"reactions":[{"reaction":"❤️","count":1,"users":[{"id":4668,"username":"IO","name":null}],"hasReacted":false}],"wiki":false,"version":1,"publicVersion":1,"deletedAt":null,"isAnonymous":false,"user":{"id":4668,"username":"IO","name":null,"avatarUrl":null,"admin":true,"moderator":false,"signatureRaw":"","signatureCooked":null}}],"hasMore":false,"lastReadPostNumber":1,"hasUrlPostNumber":true}