[P0] vLLM 0.15.1 #939
(empty)
💬 Comments (3)
Also an MLNode container with vllm 0.15 version should be tested and performance and deployment parameters identified on different cards
ghcr.io/product-science/mlnode:3.0.13-alpha2 https://github.com/gonka-ai/vllm/pull/24
🔄 Auto-synced from Issue #939 every hour.
MLnode should support the vllm 0.15.1 - Processed logprobs flag: we need to output processed logprobs when doing inference to remain compatibple with 0.9 version - A flag for the change to the raw logprobs: processed logprobs are worse for inference validation, than raw logprobs, so we'll need an on-chain parameter and mlnode code, that enables change of the logprobes mode simultaniously on all the mlnodes - Probably we'll need to change the docker file: add some env variables and update the versions of packages.