Refactors dataset management to consistently append '-vl' for vision-language
datasets and dynamically name cleaned datasets (e.g., 'dataset-cleaned').
Enforces vLLM as a strict dependency for LLM-based data cleaning, exiting if
unavailable. Integrates 'enable_thinking' option for LLM cleaning and enables
cleaning by default in relevant test configurations.
Adds torchvision dependency for vision models and streamlines the cleaning call
in training to centralize decision-making.
Updates the Qwen3 model in test configurations from 4B to the smaller 0.6B version.
This change helps improve test performance and reduce resource requirements.
Ensures the data generation pipeline correctly includes the expected number of <image> tags for multimodal configurations, specifically for Qwen2.5-VL datasets.
Adds a blocked words filter to the dataset creation process.
This prevents undesirable words from appearing in the generated
datasets, enhancing data quality and safety.
A test case is added to verify the filter is working correctly.
Also, skips "text" type in qa_generator to avoid duplicated filter.
Refactors the test environment setup to use pytest fixtures
for session-level setup and teardown. This ensures a clean
and consistent environment before and after each test session.
It also improves the robustness by backing up and restoring
existing 'model_output' and 'dataset/csv' directories.
xming521