xming521 a0829d2098 feat(cleaning): add batch processing for online LLM ...

Introduce a ThreadPoolExecutor in the OnlineLLM class to enable
concurrent API calls, significantly boosting throughput for LLM operations.
Refactor the OlineLLMCleaningStrategy to leverage this new batching
capability, allowing multiple data points to be processed in parallel.

Increase the default `clean_batch_size` in settings and enhance
`n_process`/`batch_size` for PII detection to optimize for the new
concurrency. Simplify prompt management by removing a dedicated prompt
for online LLM cleaning. Add context manager support to OnlineLLM for
reliable resource cleanup. Ensure LLM chat responses explicitly request
JSON format.

2025-08-28 21:30:05 +08:00

..

__init__.py

在pyproject.toml中添加langchain依赖，并在strategies.py中引入PromptTemplate以支持数据清洗功能的实现。

2025-05-03 22:57:36 +08:00

clean_data.py

feat(cleaning): add batch processing for online LLM

2025-08-28 21:30:05 +08:00