Adds a blocked words filter to the dataset creation process.
This prevents undesirable words from appearing in the generated
datasets, enhancing data quality and safety.
A test case is added to verify the filter is working correctly.
Also, skips "text" type in qa_generator to avoid duplicated filter.
Implements batch processing for PII detection to improve
performance. Also, fixes a bug in the PII detection logic
and updates the installation instructions in the README.
Adds retry logic for OpenAI and image description API calls
to improve resilience against transient errors and rate
limiting. Implements exponential backoff with configurable
parameters.
Improves PII detection for Chinese language data by:
- Adding a Chinese-specific PII detector that inherits from
the base PII detector.
- Incorporating custom recognizers for common Chinese PII
patterns (e.g., phone numbers, ID cards).
- Filtering out irrelevant country-specific entities when
processing Chinese text.
- Adding unicode escape id recognizer.
These changes ensure more accurate and comprehensive
identification of sensitive information in Chinese datasets.
Adds a configuration option to include the current
datetime in the system prompt of the generated dataset.
This allows models to be aware of the temporal context
of the conversations.
This commit addresses an issue where newline characters
within text messages were causing problems in downstream
processing. It removes these characters to ensure that
the messages are processed correctly.
Consolidates the separator for combined content to a newline character, removing language-specific branching.
This simplifies the logic and ensures consistency across different languages.
Adds support for processing stickers in Telegram chat data.
- Includes 'sticker' as a valid DataModality.
- Allows the option to include sticker emojis as message content.
xming521