Skip to content

Commit

Permalink
Merge branch 'main' into add-pre-commit
Browse files Browse the repository at this point in the history
Signed-off-by: Harry Mellor <[email protected]>
  • Loading branch information
hmellor committed Jan 29, 2025
2 parents c953ce2 + 240e9c8 commit 7032bb6
Showing 1 changed file with 8 additions and 2 deletions.
10 changes: 8 additions & 2 deletions src/vllm_router/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,14 +52,20 @@ docker build -t <image_name>:<tag> -f docker/Dockerfile .

## Example commands to run the router

You can install the router using the following command:

```bash
pip install -e .
```

**Example 1:** running the router locally at port 8000 in front of multiple serving engines:

```bash
python3 router.py --port 800 \
vllm-router --port 8000 \
--service-discovery static \
--static-backends "http://localhost:9001,http://localhost:9002,http://localhost:9003" \
--static-models "facebook/opt-125m,meta-llama/Llama-3.1-8B-Instruct,facebook/opt-125m" \
--engine-stats-interval 10 \
--log-stats \
--routing-logic roundrobin \
--routing-logic roundrobin
```

0 comments on commit 7032bb6

Please sign in to comment.