Add timer support for testing

[dsikka]

SUMMARY:

• Add timer to our e2e testing to provide support when we see regression across oneshot, compressed saving or vLLM generation
• The csv produced at the end of each test case has the same name as the model produced, apart from the -e2e substring. For example:

For the following model produced from one test case: https://huggingface.co/nm-testing/TinyLlama-1.1B-Chat-v1.0-FP8_DYNAMIC-e2e

The following csv will be produced: TinyLlama-1.1B-Chat-v1.0-FP8_DYNAMIC.csv and the names will be unique as a result.

The contents of the csv:

_load_model_and_tokenizer	_run_oneshot	_save_compressed_model	_run_vllm
1.7816431522369385	1.51918625831604	7.086644649505615	26.71957492828369

• For each function that has its timing recorded, we have a column (e.g in the e2e test, we record timings for _load_model_and_tokenizer, _run_oneshot, _save_compressed_model, and, _run_vllm).
• The values of the columns indicate the time it took to execute the function, each time it was called.
• In our e2e tests, we only call each of the four functions once, so we only have 1 row.
• The csv is just saved to the directory defined by the TIMINGS_DIR env variable

TEST PLAN:

• Ran e2e test cases locally and csv produced expected timing breakdown
• csv saves all logged timings to disk as part of the teardown

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[horheynm]

LGTM

[rahul-tuli]

Great Job!

[dbarbuzzi]

Can you expand on the CSV file contents and location (directory, filename)?

Regarding location, we currently run the script that runs all the various configs for this test in sequence. I assume that this will not cause any existing files to be overwritten (e.g. each test's CSV file(s) will be uniquely named), but it also seems like they might end up scattered across different directories? Since they seem more like a test artifact than a model/etc. one, it seems like it might make more sense to gather them in a consolidated directory (which would also make finding them all programmatically after the tests conclude much simpler.

As far as contents, just an example of the contents after a single test would be helpful.

[dsikka]

Can you expand on the CSV file contents and location (directory, filename)?

Regarding location, we currently run the script that runs all the various configs for this test in sequence. I assume that this will not cause any existing files to be overwritten (e.g. each test's CSV file(s) will be uniquely named), but it also seems like they might end up scattered across different directories? Since they seem more like a test artifact than a model/etc. one, it seems like it might make more sense to gather them in a consolidated directory (which would also make finding them all programmatically after the tests conclude much simpler.

As far as contents, just an example of the contents after a single test would be helpful.

Yup, let me update the PR description to add more colour to what we're producing

[dbarbuzzi]

we can update this to keep one defined location

Customizable (e.g., via env var) would be nice, but even just a dedicated root folder (maybe something like timings) would be ideal, and if we were to expand, we could use subfolders of the root folder (e.g., the test_vllm timings might go into timings/e2e-test_vllm or something — not to get too nested).

I think aside from that, this all looks good to me.

[dbarbuzzi]

Thanks for adding the ability to control the timing CSV output dir via env var, that will be very helpful, especially if we expand the usage of this timer to other tests.

Looks good to me!

[dhuangnm]

This is awesome, thanks for adding it!