🧪 Evaluation Reproduction

We provide evaluation scripts to reproduce the results of Skywork R1V3.

Most benchmarks can be evaluated using VLMEvalKit by vllm inference engine.

⚙️ Environment Setup

Install all dependencies:

bash ./eval/vlmevalkit/build_env.sh

Set OpenAI API Key and Base URL

Create or edit the .env file in vlmevalkit/ and add the following:

OPENAI_API_KEY=your_api_key_here
OPENAI_API_BASE=https://your_base_url_here

🚀 Evaluation Steps

Step 1: Launch the Model

Start possible setting, and deploy the model

export TORCH_CUDA_ARCH_LIST="8.9+PTX" # optional
bash ./eval/vlmevalkit/eval_shell/launch_vlm_model.sh

Step 2: Run Evaluation Scripts

Evaluate on supported benchmarks:

bash ./vlmevalkit/eval_shell/run_eval.sh

⚠️ Note: Some benchmarks (e.g., MMMU) require post-processing to adjust results using rule-based scripts. For PhyX evaluation, you need to download the TSV dataset from here and place it in the ./eval/vlmevalkit/eval_shell/LMUData directory.

📌 Additional Notes

Ensure the model is properly loaded before running evaluation.
All results will be saved under the outputs/ directory.
For EMMA-mini and MMK12, please refer to EMMA-mini and MMK12 for evaluation instructions.