Is prefix-aware routing implemented in this repo?

[tdoublep]

It says "(WIP) prefix-aware routing" in the README here:
https://github.com/vllm-project/production-stack/tree/main/src/router

A very quick scan through the Python files there, I can't see anything.

Can the results in the blog be reproduced using the code that is currently in this repo?

[gaocegege]

From what I’ve seen in the vllm-dev.slack.com discussions, this feature might roll out in the next few weeks. I’m guessing the performance boost could come from session-id-based routing, which seems similar to how prefix caching works in regular QA tasks—basically, requests from the same session share the same prefix. Just my two cents, though.

[KuntaiDu]

In the current results shown in the blogpost, we use Session-ID-based routing, which can achieve most of the prefix caching potential in general chatting applications. We are working on the prefix-aware routing (it's going to be in the roadmap). cc @ApostaC

[gaocegege]

I'm curious about how to implement this feature. Will we maintain a radix tree in the router to simulate the KV cache status, similar to the SGLang router?

[ApostaC]

I'm curious about how to implement this feature. Will we maintain a radix tree in the router to simulate the KV cache status, similar to the SGLang router?

Good question. I think an alternative way is to simulate the "page" and "eviction" logic at a coarser granularity.
@KuntaiDu is right now working on the design, and there should be an RFC soon.

[gaocegege]

Good question. I think an alternative way is to simulate the "page" and "eviction" logic at a coarser granularity.

Makes a lot of sense. Will keep an eye on it.

[KuntaiDu]

Please check #59 for the initial design of prefix-aware routing

[gaocegege]

I think we could close this issue since we've got the RFC to keep track of it. What do you think?

[tdoublep]

Sure thing.