Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature]: Expose a read-only API to check whether engine is sleeping #14311

Open
1 task done
waltforme opened this issue Mar 5, 2025 · 1 comment · May be fixed by #14312
Open
1 task done

[Feature]: Expose a read-only API to check whether engine is sleeping #14311

waltforme opened this issue Mar 5, 2025 · 1 comment · May be fixed by #14312
Labels
feature request New feature or request

Comments

@waltforme
Copy link
Contributor

🚀 The feature, motivation and pitch

The sleep feature works great!

We can do even better by adding a read-only probe to check whether the engine is sleeping or not. Today the /sleep and /wake_up API endpoints POSTs to the API server and mutates the state of the engine. We may introduce an /is_sleeping endpoint which GETs the sleep status.

There are at least two use cases asking for this feature.

  1. We are doing orchestration and optimization among multiple instances of vLLMs, asleep or awake. We need a global state of all the instances' sleeping status. The proposed probe is necessary to compose this state, without having to be stateful, i.e. remember whether each and every engine has been put asleep.
  2. Today a sleeping engine crashes if a request is sent to it. This probe will give a good citizen peace of mind before sending a request.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
@waltforme waltforme added the feature request New feature or request label Mar 5, 2025
@waltforme waltforme linked a pull request Mar 5, 2025 that will close this issue
Open
@youkaichao
Copy link
Member

you should be able to manage the is_sleeping state from an upper level. after you post a /sleep call, you can mark that engine as sleeping.

I don't think we need to explicitly expose this API endpoint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Core] Expose API endpoint /is_sleeping waltforme/vllm
2 participants
@waltforme @youkaichao