[WIP, RFC] Production Stack on Ray Serve

[Hanchenli]

This is an issue explaining the upcoming Production Stack on Ray Serve structure.
The router will be a DeploymentHandle with fastAPI set as ingress for OpenAI API compatibility.

The inference nodes will each initialize with a subprocess running an vllm-lmcache OpenAI compatible server. The current design has the following functions:
report_status which returns the server status including model_name.
streaming_response which returns a streaming response to router for post request such as "v1/completion"

The session based routing will be implemented with a dictionary in the router.

[Hanchenli]

Note that Ray Serve just open-sourced their LLM serving stack at here: https://github.com/ray-project/ray/tree/master/python/ray/llm. The design is similar but comes without smart routing. We will keep the original design but also try to reuse some code from the link.

[thousandhu]

Note that Ray Serve just open-sourced their LLM serving stack at here: https://github.com/ray-project/ray/tree/master/python/ray/llm. The design is similar but comes without smart routing. We will keep the original design but also try to reuse some code from the link.

hi @Hanchenli, do you have the roadmap or design doc for ray llm?

[gaocegege]

By the way, do we have a plan on SkyPilot?

[Hanchenli]

Not currently. We do have plan to support on Terraform to support multiple cloud.