V1 integration #6
Conversation
Signed-off-by: Joe Runde <[email protected]>
Signed-off-by: Joe Runde <[email protected]>
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
|
The error that you report on AIU with V1, I think Dhruval already observed that with the image built on Friday 28.02, meaning V0. |
|
Interesting, I can confirm though that with this code and vllm@cd1d3c3d the v0 engine works and serves requests using the |
|
We found the problem, and it was us! An erroneous |
|
@joerunde nice! Where was this |
|
@sducouedic we had copied over code from the v1 gpu worker here in this PR for the new This bit here |
|
thanks @joerunde ! |
Signed-off-by: Joe Runde <[email protected]>
Signed-off-by: Joe Runde <[email protected]>
Signed-off-by: Joe Runde <[email protected]>
Signed-off-by: Joe Runde <[email protected]>
Signed-off-by: Joe Runde <[email protected]>
Signed-off-by: Joe Runde <[email protected]>
Signed-off-by: Joe Runde <[email protected]>
|
Alright, the v0 tests are all passing and the linter's happy. I removed the |
Signed-off-by: Yannick Schnider <[email protected]>
| dummy_tensors = lambda v: torch.full( | ||
| (num_reqs, ), v, device=self.device) | ||
| dummy_metadata = SamplingMetadata( | ||
| temperature=dummy_tensors(0.0), |
There was a problem hiding this comment.
Updated the hard coded sampling temperature to 0.0 to do greedy sampling as in most of our test cases.
| # seq_lens = [] | ||
| num_reqs = len(scheduler_output.scheduled_cached_reqs) | ||
|
|
||
| # TODO: Cache the sampling params for the current batch and build this |
There was a problem hiding this comment.
I believe there is some upstream implementation still missing for V1 to correctly fill in the sampling metadata here. So I would say that's okay for now.
| sampled_token_ids=output.sampled_token_ids.tolist(), | ||
| spec_token_ids=None, | ||
| logprobs= | ||
| None, # TODO: add logprobs, needs to be converted from tensor here |
There was a problem hiding this comment.
Is there something hindering us from providing the correct logprobs here? Asking because this is needed to run the tests in vllm-spyre/tests...
| print("\n\n\n FINISHED ITERATION \n\n\n") | ||
| print(self._req_ids2idx) | ||
| print(output.sampled_token_ids) | ||
| print("\n\n") |
There was a problem hiding this comment.
I believe that was for debugging purposes and can be removed.
| # TODO See if we can use `self.execute_model` instead for the warmup | ||
| # It's slightly risky to implement different forward pass logic here, | ||
| # which can go out of sync with the real forward pass and cause problems | ||
| # for torch.compile |
@tjohnson31415 and I spent a while hacking through a new V1-compatible worker , runner, and scheduler. This works on an AIU chip! 🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉
The main blockers here are:
We'll push up a draft PR with some temporary changes for vLLM for the above blockers so we can at least move development forward here on v1