mirror of
https://github.com/xming521/WeClone.git
synced 2026-08-28 18:07:28 +08:00
xming521
41167f6d29
Refactors dataset management to consistently append '-vl' for vision-language datasets and dynamically name cleaned datasets (e.g., 'dataset-cleaned'). Enforces vLLM as a strict dependency for LLM-based data cleaning, exiting if unavailable. Integrates 'enable_thinking' option for LLM cleaning and enables cleaning by default in relevant test configurations. Adds torchvision dependency for vision models and streamlines the cleaning call in training to centralize decision-making.