Refactors dataset management to consistently append '-vl' for vision-language
datasets and dynamically name cleaned datasets (e.g., 'dataset-cleaned').
Enforces vLLM as a strict dependency for LLM-based data cleaning, exiting if
unavailable. Integrates 'enable_thinking' option for LLM cleaning and enables
cleaning by default in relevant test configurations.
Adds torchvision dependency for vision models and streamlines the cleaning call
in training to centralize decision-making.
Upgrades openai, vllm, torch, transformers, and triton to their
latest versions. This ensures compatibility, incorporates recent bug
fixes, and leverages performance improvements across the project.
Updates the Qwen3 model in test configurations from 4B to the smaller 0.6B version.
This change helps improve test performance and reduce resource requirements.
Ensures that files with a .gif extension are correctly identified and
categorized as animated expressions or stickers based on the platform.
This improves the accuracy of media type classification during data
processing.
Introduces a new 'enable_thinking' configuration for LLM interactions.
This allows dynamic control of thinking behavior in both offline inference
and the data cleaning process. When thinking is enabled for cleaning,
the maximum number of new tokens is increased to accommodate longer
reasoning outputs.
Also reduces the repetition penalty in the cleaning strategy for
improved response diversity and skips data entries containing images
during the LLM cleaning process to avoid irrelevant processing.
Reduces the default LoRA rank in training configuration templates from 16 to 8.
This change aims to optimize resource usage and potentially accelerate training.
Updates pre-commit hooks.
Previously, the cleaning pipeline was entirely disabled for datasets
containing image messages.
Image conversations now receive a default score, bypassing text-focused
LLM cleaning. This enables partial cleaning for mixed datasets and
prevents pipeline failures.
Increases LoRA rank from 4 to 16 in example and default configurations.
This aims to improve model fine-tuning effectiveness.
Adjusts per-device batch size and gradient accumulation steps. This
maintains the same effective batch size while potentially reducing
memory usage.
Standardizes LoRA dropout to 0.25 across configurations.
Updates READMEs to clarify model performance expectations.
Removes the WC-exp Git submodule. This dependency is no longer required for the
project, which simplifies the repository's structure and reduces complexity.
Ensures the data generation pipeline correctly includes the expected number of <image> tags for multimodal configurations, specifically for Qwen2.5-VL datasets.
Adds a blocked words filter to the dataset creation process.
This prevents undesirable words from appearing in the generated
datasets, enhancing data quality and safety.
A test case is added to verify the filter is working correctly.
Also, skips "text" type in qa_generator to avoid duplicated filter.
Implements batch processing for PII detection to improve
performance. Also, fixes a bug in the PII detection logic
and updates the installation instructions in the README.
Initializes the analyzer engines within the
ChinesePIIDetector class, ensuring proper
setup and custom recognizer loading during object
instantiation. This enhances the PII detection
process for Chinese-specific contexts.
Adds retry logic for OpenAI and image description API calls
to improve resilience against transient errors and rate
limiting. Implements exponential backoff with configurable
parameters.
Improves PII detection for Chinese language data by:
- Adding a Chinese-specific PII detector that inherits from
the base PII detector.
- Incorporating custom recognizers for common Chinese PII
patterns (e.g., phone numbers, ID cards).
- Filtering out irrelevant country-specific entities when
processing Chinese text.
- Adding unicode escape id recognizer.
These changes ensure more accurate and comprehensive
identification of sensitive information in Chinese datasets.
Adds a configuration option to include the current
datetime in the system prompt of the generated dataset.
This allows models to be aware of the temporal context
of the conversations.
This commit addresses an issue where newline characters
within text messages were causing problems in downstream
processing. It removes these characters to ensure that
the messages are processed correctly.
Consolidates the separator for combined content to a newline character, removing language-specific branching.
This simplifies the logic and ensures consistency across different languages.
Updates the deepspeed installation command in the README files to specify a maximum version.
Also adds an example of downloading the default model using ModelScope in README_zh.md.
Finally, increments the project version to 0.3.01.
xming521