Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix CPU Offloading #1159

Merged
merged 1 commit into from
Feb 17, 2025
Merged

Fix CPU Offloading #1159

merged 1 commit into from
Feb 17, 2025

Conversation

dsikka
Copy link
Collaborator

@dsikka dsikka commented Feb 17, 2025

SUMMARY:

  • When updating the training args, place_model_on_device was missed and as a result, when creating the trainer (which we really should not be doing during oneshot...) the default value is left as True and the trainer tries to move the model to a gpu, if it is available.
  • We want this argument to be False as we handle the device map and model initialization based on the calibration needs

TEST PLAN:

  • cpu_offloading_fp8.py ran to completion without issue
  • mult_gpus_int8_device_map made it past the error and is running

@dsikka dsikka changed the title update Fix CPU Offloading Feb 17, 2025
Copy link

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

@dsikka dsikka requested a review from horheynm February 17, 2025 17:08
@dsikka dsikka marked this pull request as ready for review February 17, 2025 17:23
@dsikka dsikka added the ready When a PR is ready for review label Feb 17, 2025
@dsikka dsikka enabled auto-merge (squash) February 17, 2025 17:27
@dsikka dsikka merged commit 2053ee9 into main Feb 17, 2025
8 checks passed
@dsikka dsikka deleted the fix_train_args branch February 17, 2025 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready When a PR is ready for review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants