[Doc] Add vllm-ascend usage doc & fix doc format #53
Conversation
|
No need to update installation and quick start doc. They will be updated in new PR. |
ok. |
Signed-off-by: Shanshan Shen <[email protected]>
docs/source/tutorials.md
Outdated
| ```bash | ||
| # Use Modelscope mirror to speed up model download | ||
| export VLLM_USE_MODELSCOPE=True | ||
| export MODELSCOPE_CACHE=/root/models/ |
There was a problem hiding this comment.
| export MODELSCOPE_CACHE=/root/models/ |
you can use default cache -v /root/.cache:/root/.cache
docs/source/tutorials.md
Outdated
| -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ | ||
| -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ | ||
| -v /etc/ascend_install.info:/etc/ascend_install.info \ | ||
| -v /root/models:/root/models \ |
There was a problem hiding this comment.
| -v /root/models:/root/models \ | |
| -v /root/.cache:/root/.cache \ |
docs/source/tutorials.md
Outdated
| -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ | ||
| -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ | ||
| -v /etc/ascend_install.info:/etc/ascend_install.info \ | ||
| -v /root/models:/root/models \ |
There was a problem hiding this comment.
| -v /root/models:/root/models \ | |
| -v /root/.cache:/root/.cache \ |
docs/source/tutorials.md
Outdated
| -v /root/models:/root/models \ | ||
| -p 8000:8000 \ | ||
| -e VLLM_USE_MODELSCOPE=True \ | ||
| -e MODELSCOPE_CACHE=/root/models/ \ |
There was a problem hiding this comment.
| -e MODELSCOPE_CACHE=/root/models/ \ |
docs/source/tutorials.md
Outdated
| -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ | ||
| -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ | ||
| -v /etc/ascend_install.info:/etc/ascend_install.info \ | ||
| -v /root/models:/root/models \ |
There was a problem hiding this comment.
| -v /root/models:/root/models \ | |
| -v /root/.cache:/root/.cache \ |
docs/source/tutorials.md
Outdated
| ```bash | ||
| # Use Modelscope mirror to speed up model download | ||
| export VLLM_USE_MODELSCOPE=True | ||
| export MODELSCOPE_CACHE=/root/models/ |
There was a problem hiding this comment.
| export MODELSCOPE_CACHE=/root/models/ |
docs/source/tutorials.md
Outdated
| def clean_up(): | ||
| destroy_model_parallel() | ||
| destroy_distributed_environment() | ||
| gc.collect() | ||
| torch.npu.empty_cache() |
There was a problem hiding this comment.
Looks like a little bit wired, would you mind taking a look? @wangxiyuan
There was a problem hiding this comment.
since this is only a simple example, no need to do
del llm
clean_up()
There was a problem hiding this comment.
When we using mp as distributed_executor_backend, the clean up must be done by hand, otherwise will raise error when exiting process. This is a bug in vLLM.
Signed-off-by: Shanshan Shen <[email protected]>
|
After this PR merged, pls also backport this to v0.7.1 branch. |
What this PR does / why we need it?
docsdir, e.g. format tables, add underline for links, add line feed...Does this PR introduce any user-facing change?
no.
How was this patch tested?
no.