Optimize token ID handling in vLLM responses #2239
There is one part of the current implementation that works without any changes to vLLM itself, but could likely be improved.
Currently, we enable the return_token_ids parameter so that all token IDs are available when processing logprobs.
The downside is that vLLM also returns the full list of generated token IDs as a separate field in the response. We do not actually need this additional data, so it creates unnecessary response overhead.
We also currently have some workaround logic around trimming / slicing the returned data.
Goal
Investigate whether we can get the token information required for logprobs without returning the redundant full token ID list, or otherwise simplify this flow.
Ideally, we should:
- avoid unnecessary data in the vLLM response;
- remove or simplify the current trimming workaround;
- reduce serialization / response processing overhead;
- potentially save a small amount of latency, likely on the order of a few milliseconds.
Expected outcome
Find and implement a cleaner approach to token ID handling, if one exists, and measure whether it provides a meaningful latency improvement.
Changes to vLLM itself are acceptable to investigate if that turns out to be the cleanest solution, but they are not necessarily required.
🔄 Auto-synced from Issue #2239 every hour.