Support histogram metrics from engine in cache #424
Conversation
1. Refactor the metric fetcher in the cache to support histogram metrics 2. Support histogram metrics fetching way. 3. Add a few histogram metrics like ITL, TTFT, TPOT,E2E etc
1. Use separate const file to host available metrics names 2. Use metrics vs engine metrics name 1:1
Jeffwan
force-pushed
the
jiaxin/routing-support
branch
from
4658c5e to
ffe5d08
Compare
| totalthroughput := 2*promptThroughput + generationThroughput | ||
| klog.V(4).Infof("pod: %v, podIP: %v, promptThroughput: %v, generationThroughput: %v, totalthroughput: %v", | ||
| pod.Name, pod.Status.PodIP, promptThroughput, generationThroughput, totalthroughput) | ||
| total_throughput := 2*promptThroughput.Value + generationThroughput.Value |
There was a problem hiding this comment.
In autoscaler and the gateway, there exist two similar parts of metrics codes. Is this a good design? Shall we need to extract a set of common Metrics?
There was a problem hiding this comment.
minor, it is better to use camel Case naming (totalThroughput) for variable total_throughput
There was a problem hiding this comment.
yeah, golang should use camel Case. my careless.
There was a problem hiding this comment.
Is this a good design? Shall we need to extract a set of common Metrics?
Good point. We should provide an unified solution for both metric consumers.
|
In the current logic of Maybe the function |
that's a very good point. some complex cases are not covered yet. different model labels etc. We should definitely find some existing tools to replace the manual parsing. we actually already have such case because lora model and base model are from same pod and it will create multiple records. Let me cut a separate Pr to improve it. I will merge this one first to meet a basic scenario. |
* Refactor the metrics support in mocked app * Support histogram metrics in cache 1. Refactor the metric fetcher in the cache to support histogram metrics 2. Support histogram metrics fetching way. 3. Add a few histogram metrics like ITL, TTFT, TPOT,E2E etc * Add license headers * Refactor the metric names and files 1. Use separate const file to host available metrics names 2. Use metrics vs engine metrics name 1:1
Pull Request Description
Support histogram metrics from engine in cache
Related Issues
Resolves: phase I of #423
Important: Before submitting, please complete the description above and review the checklist below.
Contribution Guidelines (Expand for Details)
We appreciate your contribution to aibrix! To ensure a smooth review process and maintain high code quality, please adhere to the following guidelines:
Pull Request Title Format
Your PR title should start with one of these prefixes to indicate the nature of the change:
[Bug]: Corrections to existing functionality[CI]: Changes to build process or CI pipeline[Docs]: Updates or additions to documentation[API]: Modifications to aibrix's API or interface[CLI]: Changes or additions to the Command Line Interface[Misc]: For changes not covered above (use sparingly)Note: For changes spanning multiple categories, use multiple prefixes in order of importance.
Submission Checklist
By submitting this PR, you confirm that you've read these guidelines and your changes align with the project's contribution standards.