[perf] Refact tos downloader in Runtime

[brosoul]

Pull Request Description

• Refact tos downloader in Runtime
• Fix bug that download target could contain directory
• Add env DOWNLOADER_S3_MAX_IO_QUEUE and DOWNLOADER_S3_IO_CHUNKSIZE to control the memory could be used during downloading

TODO: find suitable envs settings for MAX_IO_QUEUE and S3_IO_CHUNKSIZE.

Related Issues

Resolves: #320 , #358

Important: Before submitting, please complete the description above and review the checklist below.

---

Contribution Guidelines (Expand for Details)

We appreciate your contribution to aibrix! To ensure a smooth review process and maintain high code quality, please adhere to the following guidelines:

Pull Request Title Format

Your PR title should start with one of these prefixes to indicate the nature of the change:

• [Bug]: Corrections to existing functionality
• [CI]: Changes to build process or CI pipeline
• [Docs]: Updates or additions to documentation
• [API]: Modifications to aibrix's API or interface
• [CLI]: Changes or additions to the Command Line Interface
• [Misc]: For changes not covered above (use sparingly)

Note: For changes spanning multiple categories, use multiple prefixes in order of importance.

Submission Checklist

• PR title includes appropriate prefix(es)
• Changes are clearly explained in the PR description
• New and existing tests pass successfully
• Code adheres to project style and best practices
• Documentation updated to reflect changes (if applicable)
• Thorough testing completed, no regressions introduced

By submitting this PR, you confirm that you've read these guidelines and your changes align with the project's contribution standards.

[Jeffwan]

BTW, can we keep the tos file and move TOSDownloader to tos.py?

[Jeffwan]

overall looks good to me. I left some comments on legacy codes and file should store TosDownloader

[brosoul]

cc @Jeffwan
The following figure shows the comparison of the download speed from TOS SDK and Boto3 SDK when we set the number of download threads to 32 (DOWNLOADER_NUM_THREADS=32) and the shard size to 64MB (DOWNLOADER_PART_CHUNKSIZE=67108864). We downloaded the safetensors file from deepseek-coder-6.7b-instruct , totaling 12.6 GB.

Note:

• The display of chunk_size in TOS SDK is represented by the chunk_size parameter in utils.copy_and_verify_length function. It is hard coded and cannot be changed directly through environment variables or passing parameters.
• The display of chunk_size in Boto3 SDK is represented by theio_chunksize in TransferConfig. It can be changed through parameter passing.

During the comparison process, we can draw the following conclusions:

• After parameter adjustment, the Boto3 SDK can achieve a bandwidth limit of 10Gbps for TOS (Small model files) that better than the performance of TOS SDK
• Boto3 allows for more convenient adjustment of parameter settings
• During the download process, due to the slower speed of disk IO compared to network IO, the downloaded files will be temporarily stored in buff/cache. Guesswork: So when downloading large model files, when the available memory for caching the downloaded files is exhausted, the download speed will eventually approach disk bandwidth. Screenshots of memory and bandwidth during the download of 200b may illustrate this point.

[Jeffwan]

great. this is excellent!

[Jeffwan]

Seems all comments have been addressed. LGTM