{"topic":{"id":6493,"title":"nlp 任务数据清洗","slug":"topic","categoryId":14,"createdAt":"2023-10-15 02:45:22.235963","bumpedAt":"2023-10-15 15:48:56.994905","views":238,"postsCount":2,"highestPostNumber":2,"likeCount":0,"closed":false,"pinned":false,"pinnedGlobally":false,"pinnedAt":null,"pinnedUntil":null,"isBanner":false,"excerpt":"在一个 nlp 任务中，我现在有一批数据，数据中每个元素包含多个字符串。对于数据清洗有两个问题，还希望相关方向或有相关经验的 uu 给点建议：字符串末尾很多时候会加上出自的地方，比如“--知乎”“--微博”“--观察网”等等，我想对这些无意义的词进行删除，但是并没有头绪，还希望有经验的 uu 给点建议一个元素中多条字符串之间经常会有语义高度重合的情况，这种情况需要进行语义相似度计算并进行筛选吗，如果需要的有没有比较推荐的做法","deletedAt":null,"isAnonymous":false,"category":{"id":14,"name":"问答","slug":"qa","color":"f5c211"},"author":{"id":646,"username":"Mike","name":"Mike","avatarUrl":null},"tags":[]},"posts":[{"id":13847,"postNumber":1,"raw":"在一个 nlp 任务中，我现在有一批数据，数据中每个元素包含多个字符串。对于数据清洗有两个问题，还希望相关方向或有相关经验的 uu 给点建议：\n1. 字符串末尾很多时候会加上出自的地方，比如“--知乎”“--微博”“--观察网”等等，我想对这些无意义的词进行删除，但是并没有头绪，还希望有经验的 uu 给点建议\n2. 一个元素中多条字符串之间经常会有语义高度重合的情况，这种情况需要进行语义相似度计算并进行筛选吗，如果需要的有没有比较推荐的做法","cooked":"

在一个 nlp 任务中，我现在有一批数据，数据中每个元素包含多个字符串。对于数据清洗有两个问题，还希望相关方向或有相关经验的 uu 给点建议：

字符串末尾很多时候会加上出自的地方，比如“--知乎”“--微博”“--观察网”等等，我想对这些无意义的词进行删除，但是并没有头绪，还希望有经验的 uu 给点建议
一个元素中多条字符串之间经常会有语义高度重合的情况，这种情况需要进行语义相似度计算并进行筛选吗，如果需要的有没有比较推荐的做法

\n","createdAt":"2023-10-15 02:45:22.30215","updatedAt":"2023-10-15 02:45:22.30215","replyToPostNumber":null,"replyCount":0,"likeCount":0,"wordCount":198,"liked":false,"bookmarked":false,"polls":[],"reactions":[],"wiki":false,"version":1,"publicVersion":1,"deletedAt":null,"isAnonymous":false,"user":{"id":646,"username":"Mike","name":"Mike","avatarUrl":null,"admin":false,"moderator":false,"signatureRaw":null,"signatureCooked":null}},{"id":13899,"postNumber":2,"raw":"第一个任务很难吗？不是直接 `.removesuffix()` 就完了\n第二条任务可以调用 [OpenAI Embeddings](https://platform.openai.com/docs/guides/embeddings/embeddings)","cooked":"

第一个任务很难吗？不是直接 .removesuffix() 就完了
\n第二条任务可以调用 OpenAI Embeddings

\n","createdAt":"2023-10-15 15:48:56.994905","updatedAt":"2024-03-17 13:07:20.009916","replyToPostNumber":null,"replyCount":0,"likeCount":0,"wordCount":28,"liked":false,"bookmarked":false,"polls":[],"reactions":[],"wiki":false,"version":1,"publicVersion":1,"deletedAt":null,"isAnonymous":false,"user":{"id":1047,"username":"anonymous_coward_old","name":"Anonymous Coward Old","avatarUrl":"/uploads/image/43/32/4332638c11248b8ff07033c11de0759a55205facfbec85e14458245a91991faf.avif","admin":false,"moderator":false,"signatureRaw":null,"signatureCooked":null}}],"hasMore":false,"lastReadPostNumber":1,"hasUrlPostNumber":false}