Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

an endpoint to list all the models deployed in the AiBrix system #800

Open
premdass opened this issue Mar 5, 2025 · 1 comment · May be fixed by #802
Open

an endpoint to list all the models deployed in the AiBrix system #800

premdass opened this issue Mar 5, 2025 · 1 comment · May be fixed by #802
Assignees

Comments

@premdass
Copy link

premdass commented Mar 5, 2025

🚀 Feature Description and Motivation

to discover all the models deployed in the AIBrixsystem, it will be useful to have a single endpoint which displays all the models like the openai /models endpoint

Use Case

to discover all the models deployed in the AIBrixsystem, it will be useful to have a single endpoint which displays all the models like the openai /models endpoint

Proposed Solution

No response

@Jeffwan
Copy link
Collaborator

Jeffwan commented Mar 5, 2025

We get few users report this issue. This would be implemented shortly. We talked about two different options

  1. AIBrix should provides an abstraction to provide all available models. the challenge is current plugin design does not like to manipulate the response. Technically, it should aggregate the models (including lora adapter) from the cache. data plane traffic works out-of-box in multi-model scenarios.
  2. refactor plugin to provide model scope /v1/models return results. That means it only care about single base + all the related adapters. but seems this way need more efforts and can not meet production users's need.

Currently, I prefer the 1st way and we need to do such trick for such api. It brings another problem on the api compatibility and we should revisit it now. #732

@varungup90 varungup90 self-assigned this Mar 5, 2025
@varungup90 varungup90 linked a pull request Mar 6, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants