Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

请问读取数据集内存占用过高的问题 #9

Open
ykallan opened this issue Jul 12, 2023 · 3 comments
Open

请问读取数据集内存占用过高的问题 #9

ykallan opened this issue Jul 12, 2023 · 3 comments

Comments

@ykallan
Copy link

ykallan commented Jul 12, 2023

进行文本多标签分类，数据有90多万，txt文件有不到200m，但是读取数据集占用的内存太多了，不知道是不是bug还是本来就这样，机子32g的内存都不够读取四分之一的数据，
@yongzhuo
Copy link
Owner

文本长度多少？这个是一次加载全量数据的，如果不能，得改成yield的形式

@ykallan
Copy link
Author

ykallan commented Jul 14, 2023

文本长度多少？这个是一次加载全量数据的，如果不能，得改成yield的形式

文本长度大概是64左右，我设置了max_len = 64

晚点我试一下yield

@yongzhuo
Copy link
Owner

yongzhuo commented Jul 14, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yongzhuo @ykallan