VLM inference and validation in Gonka #1026
Title
VLM inference and validation in Gonka
Summary
Test the possibility of VLM serving validation, add the necessary tools
Motivation
Gonka already employs validation of token sequences generated by distributed nodes serving large language models (LLMs), which enables the detection of fraudulent behavior and dishonest model hosting. A logical next step is to add visual language models in the set of serving models.
Impact
- inference and validation scripts for visual language models
- tests
Expected outcome
- possibility of the VLM serving in Gonka
- tools for validation
Proposed approach
- add necessary tools in MLNode
- Do image-related tasks on the one machine, save artifacts and validate them on the other machine. Include threshold calibration, using "honest" and "fraud" scenarios with "small" (~2B) and "large" (235B) models
- PR in gonka repo
Intermediate results
Results for "small" visual models - Qwen2-VL-2B-Int4 - Qwen2-VL-2B-Int8 validation are available here For these models F1-score for fraud-detection task is 100%.
Results for large "models" will be added a bit later.
💬 Comments (3)
And here are the results with the latest version of Gonka-VLM
The fraud detection accuracy is about 99%
Models Proposal
This is a proposal to add the Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 multimodal model to the Gonka inference network.
Validation thresholds for all the models were computed using the standard procedure described in the inference validation scripts README.
There is respective notebook with the details of experiments and gdrive folder with raw inference-validation data.
For the inference the test split (1000 images) of the Flickr8K dataset was used.
| Parameter | Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 |
|---|---|
| Notebook | qwen3-VL-235B_thresholds-new.ipynb |
| Validation Data | link |
| Model Len | 128000 |
| Top-K Logprobs | 5 |
| Validation Thresholds | 0,0214 / 0,0224 |
| Fraud Accuracy | 99% |
| Tested Against | Qwen3-VL-235B-A22B-Instruct-AWQ |
| VRAM (example setup) | ~320GB (4xA100 or 4xH100 GPUs) |
All experiments were conducted using MLNode v0.1.0
Qwen3-VL-235B-Instruct is suggested to be deployed with with the following parameters:
additional_args=[
'--max-model-len', '128000',
'--gpu-memory-utilization', '0.95'
]
Script for preparing test set is available here.
An example of the inference
python vlm_inference.py \
--url http://localhost:8801 \
--exp-dir /root/inference \
--prompt "Describe the image." \
--images-dir /root/flickr8k_images/test/ \
--top-logprobs 5 \
--temperature 0.99
An example of the validation
python vlm_validation.py \
--validation-url http://localhost:8001 \
--inference-artifact /root/inference/inference_results.jsonl \
--exp-dir /root/validation \
--images-dir /root/flickr8k_images/test/
🔄 Auto-synced from Issue #1026 every hour.
Here are the results for large model - /Qwen/Qwen3-VL-235B-A22B-Instruct
link