Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Chat inputs to AsyncLLMEngine #14289

Open
1 task done
sfc-gh-mkrubinski opened this issue Mar 5, 2025 · 1 comment
Open
1 task done

[Feature]: Chat inputs to AsyncLLMEngine #14289

sfc-gh-mkrubinski opened this issue Mar 5, 2025 · 1 comment
Labels
feature request New feature or request

Comments

@sfc-gh-mkrubinski
Copy link

🚀 The feature, motivation and pitch

Currently, only the LLM class meant for offline inference supports the chat method.
Are there any plans to implement a similar method for AsyncLLMEngine, besides the existing generate?
Alternatively, is there any work on extending the PromptType acceptable by generate to include more prompt variants, such as chat conversations?

Alternatives

No response

Additional context

No response

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
@sfc-gh-mkrubinski sfc-gh-mkrubinski added the feature request New feature or request label Mar 5, 2025
@DarkLight1337
Copy link
Member

I think there is not much reason to do this now because the interface of async engine will change significantly in V1. Maybe after the API is more stable? cc @robertgshaw2-redhat

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants