[e2e] Update vllm tests with additional datasets

[brian-dellabetta]

SUMMARY:
Adding a handful more e2e tests with 3 more datasets

• neuralmagic/LLM_compression_calibration
• garage-bAInd/Open-Platypus
• Open-Orca/slimorca-deduped-cleaned-corrected

and a new SLM:

• Qwen/Qwen2.5-0.5B

I also included an env var flag to skip uploads to HF, defaulting to original behavior. I found this useful for tests.
This adds 15-20 minutes of extra testing (and 1.2GB of HF assets to download) to the nightly runs, which team has said is fine.

To run (you'll have you update your path & device_id):

CADENCE=nightly \
SKIP_HF_UPLOAD=yes \
CUDA_VISIBLE_DEVICES=4 \
TEST_DATA_FILE=~/projects/llm-compressor/tests/e2e/vLLM/configs/fp8_weight_only_channel.yaml \
pytest -s ~/projects/llm-compressor/tests/e2e/vLLM/test_vllm.py

TEST PLAN:
Additional config files for a broader range of datasets and an additional model.

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[dbarbuzzi]

If you could determine and note how much extra time these new additions would take when running the e2e tests, that would be super helpful! It would be ideal if we can preempt any required timeout changes to the nightly runs.

We have a custom test run workflow which you can easily configure to run that specific set of tests against this PR's branch (and choose a different timeout) to do the measurements. I am available to help use that workflow, just ping me on Slack.

[brian-dellabetta]

If you could determine and note how much extra time these new additions would take when running the e2e tests, that would be super helpful! It would be ideal if we can preempt any required timeout changes to the nightly runs.

We have a custom test run workflow which you can easily configure to run that specific set of tests against this PR's branch (and choose a different timeout) to do the measurements. I am available to help use that workflow, just ping me on Slack.

@dbarbuzzi sure thing! Worth discussing in standup Monday to see how to balance coverage and time to complete. Right now it's 6 new configs to 22 pre-existing, not sure if we want/need that many. Probably 15-20 more minutes total as is

[brian-dellabetta]

@dsikka responded an additional 15-20 minutes of nightly run time will not be a concern, so moving this to ready. Just FYI this requires downloading of the following datasets/models.
Open-Orca/slimorca-deduped-cleaned-corrected -- 162MB
neuralmagic/LLM_compression_calibration -- <5MB
garage-bAInd/Open-Platypus -- 15MB
Qwen/Qwen2.5-0.5B -- 1GB

Is there a way to run the nightly tests to confirm, or should we just merge this in today and see how nightly tests go tonight?

[dsikka]

great job!

[kylesayrs]

It'd be nice to move some of these preprocessing and tokenization function definitions to TextGenerationDataset in the future so that others can use it, but no rush there.

LGTM!

[brian-dellabetta]

All tests are passing except tests/e2e/vLLM/configs/sparse2of4_fp8_dynamic_qwen.yaml, which we are removing because of a runtime CUTLASS error. Once resolved, we can bring this back:

cadence: "nightly"
test_type: "regression"
model: Qwen/Qwen2.5-0.5B
recipe: tests/e2e/vLLM/recipes/Sparse_2of4/recipe_sparse_2of4_fp8_dynamic.yaml
scheme: sparse2of4_fp8_dynamic
dataset_id: garage-bAInd/Open-Platypus
dataset_split: train

[rahul-tuli]

LGTM!