[feature request] Real-time streaming inference load generation by perf_analyzer

[vadimkantorov]

Triton and perf_analyzer has some support for streaming-mode inference:

• https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_analyzer/docs/config.html#shape-input-data-and-streaming
• https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/docs/tutorial.html
• https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/docs/cli.html#streaming

But if I understand well, these still assume that the input to the model is provided in one shot. In some scenarios (e.g. real-time text/speech translation/ or real-time speech recognition), the input should also be continuously supplied in streamed/chunked mode (ideally, reusing a single connection in duplex mode).

I propose to add this scenario to perf_analyzer as well

Thanks!

---

I also wonder if Triton ships any perf_analyzer-like robust Python scripts for correct, low-overhead, zero-gc, concurrent load generation / metrics computation? (so that the load-generation part can be customized)?