- Updated the `qa_generator.py` to include a new mechanism for managing chat member relationships, allowing the addition of contextual information about the relationship between users in conversations.
- Refactored the CSV loading function to support loading user relationship data from a `users.json` file, improving the context provided during QA generation.
- Added a new configuration option `add_relation` to the dataset settings, enabling users to toggle this feature.
- Updated the `.gitignore` to exclude additional data directories and cache files for better repository hygiene.
- Bumped version to 0.3.03 in `pyproject.toml` to reflect these changes.
Introduce a ThreadPoolExecutor in the OnlineLLM class to enable
concurrent API calls, significantly boosting throughput for LLM operations.
Refactor the OlineLLMCleaningStrategy to leverage this new batching
capability, allowing multiple data points to be processed in parallel.
Increase the default `clean_batch_size` in settings and enhance
`n_process`/`batch_size` for PII detection to optimize for the new
concurrency. Simplify prompt management by removing a dedicated prompt
for online LLM cleaning. Add context manager support to OnlineLLM for
reliable resource cleanup. Ensure LLM chat responses explicitly request
JSON format.
Refactors dataset management to consistently append '-vl' for vision-language
datasets and dynamically name cleaned datasets (e.g., 'dataset-cleaned').
Enforces vLLM as a strict dependency for LLM-based data cleaning, exiting if
unavailable. Integrates 'enable_thinking' option for LLM cleaning and enables
cleaning by default in relevant test configurations.
Adds torchvision dependency for vision models and streamlines the cleaning call
in training to centralize decision-making.
Reduces the default LoRA rank in training configuration templates from 16 to 8.
This change aims to optimize resource usage and potentially accelerate training.
Increases LoRA rank from 4 to 16 in example and default configurations.
This aims to improve model fine-tuning effectiveness.
Adjusts per-device batch size and gradient accumulation steps. This
maintains the same effective batch size while potentially reducing
memory usage.
Standardizes LoRA dropout to 0.25 across configurations.
Updates READMEs to clarify model performance expectations.
Adds a configuration option to include the current
datetime in the system prompt of the generated dataset.
This allows models to be aware of the temporal context
of the conversations.
xming521