Why Hugging Face Token?

[nitin302]

I see that we need hugging face token to download the model

• Is this the only way models can be used?
• Is there a way to use an AWS S3 folder location?
• What is the process for fine-tuned models?

[ApostaC]

To answer your question: @nitin302

Is this the only way models can be used?

You can also load the model from a local path by passing a file path in modelURL in your values.yaml.
More specifically, the helm chart will try mounting a PersistentVolume to /data in the vLLM pods. Therefore, if you have your model stored in a persistent volume (for example, in the folder my_model/), you can let vLLM load it by setting modelURL: "/data/my_model" in your values.yaml.

Here's a tutorial about load model from persistent volume: https://github.com/vllm-project/production-stack/blob/main/tutorials/03-load-model-from-pv.md

---

Is there a way to use an AWS S3 folder location?

Currently, we haven't supported the S3 folders yet. Adding an init container to download from S3 to local persistent volume when launching the vLLM pods should be a solution.
Can you create a separate issue to request this feature?

---

What is the process for fine-tuned models?

Just like I mentioned above, you can create a persistent volume that contains your fine-tuned models.

[nitin302]

thank you @ApostaC ... do you also have a tutorial for AWS EKS?