diff --git a/README.md b/README.md index 095ee91440..ef2ebae081 100644 --- a/README.md +++ b/README.md @@ -32,17 +32,16 @@ PaddleOCR 由 [PMC](https://github.com/PaddlePaddle/PaddleOCR/issues/12122) 监 - 🔥🔥《PaddleOCR 2.9 发布,正式开源文本图像智能分析利器》,文本图像版面解析实现高精度实时预测,低代码全流程开发加速产业应用。集成文本图像矫正、版面区域检测、常规文本检测、印章文本检测、文本识别、表格识别等多功能。6条模型产线一键调用,显著降低开发成本。支持高性能推理、服务化部署和端侧部署等多种部署方式。**10月24日(周四)19:00**直播为您深度解析最新升级亮点。 [报名链接](https://www.wjx.top/vm/PExy7cM.aspx?udsid=896077) - **🔥2024.10.1 添加OCR领域低代码全流程开发能力**: - * 飞桨低代码开发工具PaddleX,依托于PaddleOCR的先进技术,支持了OCR领域的低代码全流程开发能力: - * 🎨 [**模型丰富一键调用**](https://paddlepaddle.github.io/PaddleOCR/latest/paddlex/quick_start.html):将文本图像智能分析、通用OCR、通用版面解析、通用表格识别、公式识别、印章文本识别涉及的**17个模型**整合为6条模型产线,通过极简的**Python API一键调用**,快速体验模型效果。此外,同一套API,也支持图像分类、目标检测、图像分割、时序预测等共计**200+模型**,形成20+单功能模块,方便开发者进行**模型组合**使用。 - * 🚀[**提高效率降低门槛**](https://paddlepaddle.github.io/PaddleOCR/latest/paddlex/overview.html):提供基于**统一命令**和**图形界面**两种方式,实现模型简洁高效的使用、组合与定制。支持**高性能推理、服务化部署和端侧部署**等多种部署方式。此外,对于各种主流硬件如**英伟达GPU、昆仑芯、昇腾、寒武纪和海光**等,进行模型开发时,都可以**无缝切换**。 + - 飞桨低代码开发工具PaddleX,依托于PaddleOCR的先进技术,支持了OCR领域的低代码全流程开发能力: + - 🎨 [**模型丰富一键调用**](https://paddlepaddle.github.io/PaddleOCR/latest/paddlex/quick_start.html):将文本图像智能分析、通用OCR、通用版面解析、通用表格识别、公式识别、印章文本识别涉及的**17个模型**整合为6条模型产线,通过极简的**Python API一键调用**,快速体验模型效果。此外,同一套API,也支持图像分类、目标检测、图像分割、时序预测等共计**200+模型**,形成20+单功能模块,方便开发者进行**模型组合**使用。 + - 🚀[**提高效率降低门槛**](https://paddlepaddle.github.io/PaddleOCR/latest/paddlex/overview.html):提供基于**统一命令**和**图形界面**两种方式,实现模型简洁高效的使用、组合与定制。支持**高性能推理、服务化部署和端侧部署**等多种部署方式。此外,对于各种主流硬件如**英伟达GPU、昆仑芯、昇腾、寒武纪和海光**等,进行模型开发时,都可以**无缝切换**。 + + - 支持文档场景信息抽取v3[PP-ChatOCRv3-doc](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/pipeline_usage/tutorials/information_extraction_pipelines/document_scene_information_extraction.md)、基于RT-DETR的[高精度版面区域检测模型](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/layout_detection.md)和PicoDet的[高效率版面区域检测模型](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/layout_detection.md)、高精度表格结构识别模型[SLANet_Plus](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/table_structure_recognition.md)、文本图像矫正模型[UVDoc](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/text_image_unwarping.md)、公式识别模型[LatexOCR](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/formula_recognition.md)、基于PP-LCNet的[文档图像方向分类模型](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/doc_img_orientation_classification.md) - * 支持文档场景信息抽取v3[PP-ChatOCRv3-doc](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/pipeline_usage/tutorials/information_extraction_pipelines/document_scene_information_extraction.md)、基于RT-DETR的[高精度版面区域检测模型](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/layout_detection.md)和PicoDet的[高效率版面区域检测模型](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/layout_detection.md)、高精度表格结构识别模型[SLANet_Plus](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/table_structure_recognition.md)、文本图像矫正模型[UVDoc](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/text_image_unwarping.md)、公式识别模型[LatexOCR](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/formula_recognition.md)、基于PP-LCNet的[文档图像方向分类模型](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/doc_img_orientation_classification.md) - - **🔥2024.7 添加 PaddleOCR 算法模型挑战赛冠军方案**: - 赛题一:OCR 端到端识别任务冠军方案——[场景文本识别算法-SVTRv2](https://paddlepaddle.github.io/PaddleOCR/latest/algorithm/text_recognition/algorithm_rec_svtrv2.html); - 赛题二:通用表格识别任务冠军方案——[表格识别算法-SLANet-LCNetV2](https://paddlepaddle.github.io/PaddleOCR/latest/algorithm/table_recognition/algorithm_table_slanet.html)。 - ## 🌟 特性 支持多种 OCR 相关前沿算法,在此基础上打造产业级特色模型PP-OCR、PP-Structure和PP-ChatOCR,并打通数据生产、模型训练、压缩、预测部署全流程。 diff --git a/README_en.md b/README_en.md index a0d6028f0d..0bb06c57bf 100644 --- a/README_en.md +++ b/README_en.md @@ -30,13 +30,13 @@ PaddleOCR is being oversight by a [PMC](https://github.com/PaddlePaddle/PaddleOC ## 📣 Recent updates ([more](https://paddlepaddle.github.io/PaddleOCR/latest/en/update.html)) - **🔥 2024.10.18 release PaddleOCR v2.9, including**: - * PaddleX, an All-in-One development tool based on PaddleOCR's advanced technology, supports low-code full-process development capabilities in the OCR field: - * 🎨 [**Rich Model One-Click Call**](https://paddlepaddle.github.io/PaddleOCR/latest/en/paddlex/quick_start.html): Integrates **17 models** related to text image intelligent analysis, general OCR, general layout parsing, table recognition, formula recognition, and seal recognition into 6 pipelines, which can be quickly experienced through a simple **Python API one-click call**. In addition, the same set of APIs also supports a total of **200+ models** in image classification, object detection, image segmentation, and time series forcasting, forming 20+ single-function modules, making it convenient for developers to use **model combinations**. + - PaddleX, an All-in-One development tool based on PaddleOCR's advanced technology, supports low-code full-process development capabilities in the OCR field: + - 🎨 [**Rich Model One-Click Call**](https://paddlepaddle.github.io/PaddleOCR/latest/en/paddlex/quick_start.html): Integrates **17 models** related to text image intelligent analysis, general OCR, general layout parsing, table recognition, formula recognition, and seal recognition into 6 pipelines, which can be quickly experienced through a simple **Python API one-click call**. In addition, the same set of APIs also supports a total of **200+ models** in image classification, object detection, image segmentation, and time series forcasting, forming 20+ single-function modules, making it convenient for developers to use **model combinations**. - * 🚀 [**High Efficiency and Low barrier of entry**](https://paddlepaddle.github.io/PaddleOCR/latest/en/paddlex/overview.html): Provides two methods based on **unified commands** and **GUI** to achieve simple and efficient use, combination, and customization of models. Supports multiple deployment methods such as **high-performance inference, service-oriented deployment, and edge deployment**. Additionally, for various mainstream hardware such as **NVIDIA GPU, Kunlunxin XPU, Ascend NPU, Cambricon MLU, and Haiguang DCU**, models can be developed with **seamless switching**. + - 🚀 [**High Efficiency and Low barrier of entry**](https://paddlepaddle.github.io/PaddleOCR/latest/en/paddlex/overview.html): Provides two methods based on **unified commands** and **GUI** to achieve simple and efficient use, combination, and customization of models. Supports multiple deployment methods such as **high-performance inference, service-oriented deployment, and edge deployment**. Additionally, for various mainstream hardware such as **NVIDIA GPU, Kunlunxin XPU, Ascend NPU, Cambricon MLU, and Haiguang DCU**, models can be developed with **seamless switching**. + + - Supports [PP-ChatOCRv3-doc](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/pipeline_usage/tutorials/information_extraction_pipelines/document_scene_information_extraction_en.md), [high-precision layout detection model based on RT-DETR](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/layout_detection_en.md) and [high-efficiency layout area detection model based on PicoDet](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/layout_detection_en.md), [high-precision table structure recognition model](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/table_structure_recognition_en.md), text image unwarping model [UVDoc](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/text_image_unwarping_en.md), formula recognition model [LatexOCR](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/formula_recognition_en.md), and [document image orientation classification model based on PP-LCNet](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/doc_img_orientation_classification_en.md). - * Supports [PP-ChatOCRv3-doc](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/pipeline_usage/tutorials/information_extraction_pipelines/document_scene_information_extraction_en.md), [high-precision layout detection model based on RT-DETR](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/layout_detection_en.md) and [high-efficiency layout area detection model based on PicoDet](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/layout_detection_en.md), [high-precision table structure recognition model](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/table_structure_recognition_en.md), text image unwarping model [UVDoc](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/text_image_unwarping_en.md), formula recognition model [LatexOCR](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/formula_recognition_en.md), and [document image orientation classification model based on PP-LCNet](https://github.com/PaddlePaddle/PaddleX/blob/release/3.0-beta1/docs/module_usage/tutorials/ocr_modules/doc_img_orientation_classification_en.md). - - **🔥2024.7 Added PaddleOCR Algorithm Model Challenge Champion Solutions**: - Challenge One, OCR End-to-End Recognition Task Champion Solution: [Scene Text Recognition Algorithm-SVTRv2](https://paddlepaddle.github.io/PaddleOCR/algorithm/text_recognition/algorithm_rec_svtrv2.html); - Challenge Two, General Table Recognition Task Champion Solution: [Table Recognition Algorithm-SLANet-LCNetV2](https://paddlepaddle.github.io/PaddleOCR/algorithm/table_recognition/algorithm_table_slanet.html). diff --git a/doc/PaddleOCR_log.png b/doc/PaddleOCR_log.png deleted file mode 100644 index a2df52f856..0000000000 Binary files a/doc/PaddleOCR_log.png and /dev/null differ diff --git a/doc/banner.png b/doc/banner.png deleted file mode 100644 index d72b997a80..0000000000 Binary files a/doc/banner.png and /dev/null differ diff --git a/doc/demo/build.png b/doc/demo/build.png deleted file mode 100644 index 73fb58e24b..0000000000 Binary files a/doc/demo/build.png and /dev/null differ diff --git a/doc/demo/error.png b/doc/demo/error.png deleted file mode 100644 index 6a463262b4..0000000000 Binary files a/doc/demo/error.png and /dev/null differ diff --git a/doc/demo/proxy.png b/doc/demo/proxy.png deleted file mode 100644 index c6d72ce8df..0000000000 Binary files a/doc/demo/proxy.png and /dev/null differ diff --git a/doc/deployment.png b/doc/deployment.png deleted file mode 100644 index afd3cf5110..0000000000 Binary files a/doc/deployment.png and /dev/null differ diff --git a/doc/deployment_en.png b/doc/deployment_en.png deleted file mode 100644 index b9d23e4892..0000000000 Binary files a/doc/deployment_en.png and /dev/null differ diff --git a/doc/doc_ch/FAQ.md b/doc/doc_ch/FAQ.md deleted file mode 100644 index 5c93d1cbd6..0000000000 --- a/doc/doc_ch/FAQ.md +++ /dev/null @@ -1,847 +0,0 @@ -# FAQ - -> 恭喜你发现宝藏! - -PaddleOCR收集整理了自从开源以来在issues和用户群中的常见问题并且给出了简要解答,旨在为OCR的开发者提供一些参考,也希望帮助大家少走一些弯路。 - -其中[通用问题](#1)一般是初次接触OCR相关算法时用户会提出的问题,在[1.5 垂类场景实现思路](#15)中总结了如何在一些具体的场景中确定技术路线进行优化。[PaddleOCR常见问题](#2)是开发者在使用PaddleOCR之后可能会遇到的问题也是PaddleOCR实践过程中的避坑指南。 - -同时PaddleOCR也会在review issue的过程中添加 `good issue`、 `good first issue` 标签,但这些问题可能不会被立刻补充在FAQ文档里,开发者也可对应查看。我们也非常希望开发者能够帮助我们将这些内容补充在FAQ中。 - -OCR领域大佬众多,本文档回答主要依赖有限的项目实践,难免挂一漏万,如有遗漏和不足,也**希望有识之士帮忙补充和修正**,万分感谢。 - -- [FAQ](#faq) - - * [1. 通用问题](#1) - + [1.1 检测](#11) - + [1.2 识别](#12) - + [1.3 端到端](#13) - + [1.4 评估方法](#14) - + [1.5 垂类场景实现思路](#15) - + [1.6 训练过程与模型调优](#16) - + [1.7 补充资料](#17) - - * [2. PaddleOCR实战问题](#2) - + [2.1 PaddleOCR repo](#21) - + [2.2 安装环境](#22) - + [2.3 数据量说明](#23) - + [2.4 数据标注与生成](#24) - + [2.5 预训练模型与微调](#25) - + [2.6 模型超参调整](#26) - + [2.7 模型结构](#27) - + [2.8 PP-OCR系统](#28) - + [2.9 端到端](#29) - + [2.10 模型效果与效果不一致](#210) - + [2.11 训练调试与配置文件](#211) - + [2.12 预测](#212) - + [2.13 推理部署](#213) - - - -## 1. 通用问题 - - - -### 1.1 检测 - -#### Q: 基于深度学习的文字检测方法有哪几种?各有什么优缺点? - -**A**:常用的基于深度学习的文字检测方法一般可以分为基于回归的、基于分割的两大类,当然还有一些将两者进行结合的方法。 - -(1)基于回归的方法分为box回归和像素值回归。a. 采用box回归的方法主要有CTPN、Textbox系列和EAST,这类算法对规则形状文本检测效果较好,但无法准确检测不规则形状文本。 b. 像素值回归的方法主要有CRAFT和SA-Text,这类算法能够检测弯曲文本且对小文本效果优秀但是实时性能不够。 - -(2)基于分割的算法,如PSENet,这类算法不受文本形状的限制,对各种形状的文本都能取得较好的效果,但是往往后处理比较复杂,导致耗时严重。目前也有一些算法专门针对这个问题进行改进,如DB,将二值化进行近似,使其可导,融入训练,从而获取更准确的边界,大大降低了后处理的耗时。 - - - -### 1.2 识别 - -#### Q: PaddleOCR提供的文本识别算法包括哪些? - -A: PaddleOCR主要提供五种文本识别算法,包括CRNN\StarNet\RARE\Rosetta和SRN, 其中CRNN\StarNet和Rosetta是基于ctc的文字识别算法,RARE是基于attention的文字识别算法;SRN为百度自研的文本识别算法,引入了语义信息,显著提升了准确率。 详情可参照如下页面: 文本识别算法 - -#### Q: 文本识别方法CRNN关键技术有哪些? - -A: CRNN 关键技术包括三部分。(1)CNN提取图像卷积特征。(2)深层双向LSTM网络,在卷积特征的基础上继续提取文字序列特征。(3)Connectionist Temporal Classification(CTC),解决训练时字符无法对齐的问题。 - -#### Q: 对于中文行文本识别,CTC和Attention哪种更优? - -**A**:(1)从效果上来看,通用OCR场景CTC的识别效果优于Attention,因为带识别的字典中的字符比较多,常用中文汉字三千字以上,如果训练样本不足的情况下,对于这些字符的序列关系挖掘比较困难。中文场景下Attention模型的优势无法体现。而且Attention适合短语句识别,对长句子识别比较差。 - -(2)从训练和预测速度上,Attention的串行解码结构限制了预测速度,而CTC网络结构更高效,预测速度上更有优势。 - -#### Q: 弯曲形变的文字识别需要怎么处理?TPS应用场景是什么,是否好用? - -**A**:(1)在大多数情况下,如果遇到的场景弯曲形变不是太严重,检测4个顶点,然后直接通过仿射变换转正识别就足够了。 - -(2)如果不能满足需求,可以尝试使用TPS(Thin Plate Spline),即薄板样条插值。TPS是一种插值算法,经常用于图像变形等,通过少量的控制点就可以驱动图像进行变化。一般用在有弯曲形变的文本识别中,当检测到不规则的/弯曲的(如,使用基于分割的方法检测算法)文本区域,往往先使用TPS算法对文本区域矫正成矩形再进行识别,如,STAR-Net、RARE等识别算法中引入了TPS模块。 - -> **Warning**:TPS看起来美好,在实际应用时经常发现并不够鲁棒,并且会增加耗时,需要谨慎使用。 - - - -### 1.3 端到端 - -#### Q: 请问端到端的pgnet相比于DB+CRNN在准确率上有优势吗?或者是pgnet最擅长的场景是什么场景呢? - -A: pgnet是端到端算法,检测识别一步到位,不用分开训练2个模型,也支持弯曲文本的识别,但是在中文上的效果还没有充分验证;db+crnn的验证更充分,应用相对成熟,常规非弯曲的文本都能解的不错。 - -#### Q: 目前OCR普遍是二阶段,端到端的方案在业界落地情况如何? - -**A**:端到端在文字分布密集的业务场景,效率会比较有保证,精度的话看自己业务数据积累情况,如果行级别的识别数据积累比较多的话two-stage会比较好。百度的落地场景,比如工业仪表识别、车牌识别都用到端到端解决方案。 - -#### Q: 二阶段的端到端的场景文本识别方法的不足有哪些? - -A: 这类方法一般需要设计针对ROI提取特征的方法,而ROI操作一般比较耗时。 - -#### Q: AAAI 2021最新的端到端场景文本识别PGNet算法有什么特点? - -A: PGNet不需要字符级别的标注,NMS操作以及ROI操作。同时提出预测文本行内的阅读顺序模块和基于图的修正模块来提升文本识别效果。该算法是百度自研,近期会在PaddleOCR开源。 - - - -### 1.4 评估方法 - -#### Q: OCR领域常用的评估指标是什么? - -**A**:对于两阶段的可以分开来看,分别是检测和识别阶段 - -(1)检测阶段:先按照检测框和标注框的IOU评估,IOU大于某个阈值判断为检测准确。这里检测框和标注框不同于一般的通用目标检测框,是采用多边形进行表示。检测准确率:正确的检测框个数在全部检测框的占比,主要是判断检测指标。检测召回率:正确的检测框个数在全部标注框的占比,主要是判断漏检的指标。 - -(2)识别阶段: -字符识别准确率,即正确识别的文本行占标注的文本行数量的比例,只有整行文本识别对才算正确识别。 - -(3)端到端统计: -端对端召回率:准确检测并正确识别文本行在全部标注文本行的占比; -端到端准确率:准确检测并正确识别文本行在 检测到的文本行数量 的占比; -准确检测的标准是检测框与标注框的IOU大于某个阈值,正确识别的检测框中的文本与标注的文本相同。 - - - -### 1.5 垂类场景实现思路 - -#### Q:背景干扰的文字(如印章盖到落款上,需要识别落款或者印章中的文字),如何识别? - -**A**:(1)在人眼确认可识别的条件下,对于背景有干扰的文字,首先要保证检测框足够准确,如果检测框不准确,需要考虑是否可以通过过滤颜色等方式对图像预处理并且增加更多相关的训练数据;在识别的部分,注意在训练数据中加入背景干扰类的扩增图像。 - -(2)如果MobileNet模型不能满足需求,可以尝试ResNet系列大模型来获得更好的效果。 - -#### Q:请问对于图片中的密集文字,有什么好的处理办法吗? - -A:可以先试用预训练模型测试一下,例如DB+CRNN,判断下密集文字图片中是检测还是识别的问题,然后针对性的改善。还有一种是如果图象中密集文字较小,可以尝试增大图像分辨率,对图像进行一定范围内的拉伸,将文字稀疏化,提高识别效果。 - -#### Q: 文本行较紧密的情况下如何准确检测? - -**A**:使用基于分割的方法,如DB,检测密集文本行时,最好收集一批数据进行训练,并且在训练时,并将生成二值图像的shrink_ratio参数调小一些。 - -#### Q:对于一些在识别时稍微模糊的文本,有没有一些图像增强的方式? - -A:在人类肉眼可以识别的前提下,可以考虑图像处理中的均值滤波、中值滤波或者高斯滤波等模糊算子尝试。也可以尝试从数据扩增扰动来强化模型鲁棒性,另外新的思路有对抗性训练和超分SR思路,可以尝试借鉴。但目前业界尚无普遍认可的最优方案,建议优先在数据采集阶段增加一些限制提升图片质量。 - -#### Q:低像素文字或者字号比较小的文字有什么超分辨率方法吗 - -A:超分辨率方法分为传统方法和基于深度学习的方法。基于深度学习的方法中,比较经典的有SRCNN,另外CVPR2020也有一篇超分辨率的工作可以参考文章:Unpaired Image Super-Resolution using Pseudo-Supervision,但是没有充分的实践验证过,需要看实际场景下的效果。 - -#### Q:对于一些尺寸较大的文档类图片,在检测时会有较多的漏检,怎么避免这种漏检的问题呢? - -A:PaddleOCR中在图像最长边大于960时,将图像等比例缩放为长边960的图像再进行预测,对于这种图像,可以通过修改det_limit_side_len,增大检测的最长边:tools/infer/utility.py#L42 - -#### Q:文档场景中,使用DB模型会出现整行漏检的情况应该怎么解决? - -A:可以在预测时调小 det_db_box_thresh 阈值,默认为0.5, 可调小至0.3观察效果。 - -#### Q: 弯曲文本(如略微形变的文档图像)漏检问题 - -**A**: db后处理中计算文本框平均得分时,是求rectangle区域的平均分数,容易造成弯曲文本漏检,已新增求polygon区域的平均分数,会更准确,但速度有所降低,可按需选择,在相关pr中可查看[可视化对比效果](https://github.com/PaddlePaddle/PaddleOCR/pull/2604)。该功能通过参数 [det_db_score_mode](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/tools/infer/utility.py#L51)进行选择,参数值可选[`fast`(默认)、`slow`],`fast`对应原始的rectangle方式,`slow`对应polygon方式。感谢用户[buptlihang](https://github.com/buptlihang)提[pr](https://github.com/PaddlePaddle/PaddleOCR/pull/2574)帮助解决该问题🌹。 - -#### Q:如何识别文字比较长的文本? - -**A**:在中文识别模型训练时,并不是采用直接将训练样本缩放到[3,32,320]进行训练,而是先等比例缩放图像,保证图像高度为32,宽度不足320的部分补0,宽高比大于10的样本直接丢弃。预测时,如果是单张图像预测,则按上述操作直接对图像缩放,不做宽度320的限制。如果是多张图预测,则采用batch方式预测,每个batch的宽度动态变换,采用这个batch中最长宽度。 - -#### Q:如何识别带空格的英文行文本图像? - -**A**:空格识别可以考虑以下两种方案: - -(1)优化文本检测算法。检测结果在空格处将文本断开。这种方案在检测数据标注时,需要将含有空格的文本行分成好多段。 - -(2)优化文本识别算法。在识别字典里面引入空格字符,然后在识别的训练数据中,如果用空行,进行标注。此外,合成数据时,通过拼接训练数据,生成含有空格的文本。 - -#### Q:弯曲文本有试过opencv的TPS进行弯曲校正吗? - -**A**:opencv的tps需要标出上下边界对应的点,这个点很难通过传统方法或者深度学习方法获取。PaddleOCR里StarNet网络中的tps模块实现了自动学点,自动校正,可以直接尝试这个。 - -#### Q: 如何识别招牌或者广告图中的艺术字? - -A: 招牌或者广告图中的艺术字是文本识别一个非常有挑战性的难题,因为艺术字中的单字和印刷体相比,变化非常大。如果需要识别的艺术字是在一个词典列表内,可以将改每个词典认为是一个待识别图像模板,通过通用图像检索识别系统解决识别问题。可以尝试使用PaddleClas的图像识别系统。 - -#### Q: 印章如何识别 - -A:1. 使用带tps的识别网络或abcnet,2.使用极坐标变换将图片拉平之后使用crnn - -#### Q: 使用预训练模型进行预测,对于特定字符识别识别效果较差,怎么解决? - -A: 由于我们所提供的识别模型是基于通用大规模数据集进行训练的,部分字符可能在训练集中包含较少,因此您可以构建特定场景的数据集,基于我们提供的预训练模型进行微调。建议用于微调的数据集中,每个字符出现的样本数量不低于300,但同时需要注意不同字符的数量均衡。具体可以参考:微调。 - -#### Q: 在使用训练好的识别模型进行预测的时候,发现有很多重复的字,这个怎么解决呢? - -A:可以看下训练的尺度和预测的尺度是否相同,如果训练的尺度为[3, 32, 320],预测的尺度为[3, 64, 640],则会有比较多的重复识别现象。 - -#### Q: 图像正常识别出来的文字是OK的,旋转90度后识别出来的结果就比较差,有什么方法可以优化? - -**A**: 整图旋转90之后效果变差是有可能的,因为目前PPOCR默认输入的图片是正向的; 可以自己训练一个整图的方向分类器,放在预测的最前端(可以参照现有方向分类器的方式),或者可以基于规则做一些预处理,比如判断长宽等等。 - -#### Q: 如何识别竹简上的古文? - -**A**:对于字符都是普通的汉字字符的情况,只要标注足够的数据,finetune模型就可以了。如果数据量不足,您可以尝试[StyleText](https://github.com/PFCCLab/StyleText)工具。 -而如果使用的字符是特殊的古文字、甲骨文、象形文字等,那么首先需要构建一个古文字的字典,之后再进行训练。 - -#### Q: 只想要识别票据中的部分片段,重新训练它的话,只需要训练文本检测模型就可以了吗?问文本识别,方向分类还是用原来的模型这样可以吗? - -**A**:可以的。PaddleOCR的检测、识别、方向分类器三个模型是独立的,在实际使用中可以优化和替换其中任何一个模型。 - -#### Q: 如何用PaddleOCR识别视频中的文字? - -**A**: 目前PaddleOCR主要针对图像做处理,如果需要视频识别,可以先对视频抽帧,然后用PPOCR识别。 - -#### Q: 相机采集的图像为四通道,应该如何处理? - -**A**: 有两种方式处理: - -- 如果没有其他需要,可以在解码数据的时候指定模式为三通道,例如如果使用opencv,可以使用cv::imread(img_path, cv::IMREAD_COLOR)。 -- 如果其他模块需要处理四通道的图像,那也可以在输入PaddleOCR模块之前进行转换,例如使用cvCvtColor(&img,img3chan,CV_RGBA2RGB)。 - -#### Q: 遇到中英文识别模型不支持的字符,该如何对模型做微调? - -**A**:如果希望识别中英文识别模型中不支持的字符,需要更新识别的字典,并完成微调过程。比如说如果希望模型能够进一步识别罗马数字,可以按照以下步骤完成模型微调过程。 - -1. 准备中英文识别数据以及罗马数字的识别数据,用于训练,同时保证罗马数字和中英文识别数字的效果; -2. 修改默认的字典文件,在后面添加罗马数字的字符; -3. 下载PaddleOCR提供的预训练模型,配置预训练模型和数据的路径,开始训练。 - -#### Q:特殊字符(例如一些标点符号)识别效果不好怎么办? - -**A**:首先请您确认要识别的特殊字符是否在字典中。 -如果字符在已经字典中但效果依然不好,可能是由于识别数据较少导致的,您可以增加相应数据finetune模型。 - ---- - -#### Q:单张图上多语种并存识别(如单张图印刷体和手写文字并存),应该如何处理? - -**A**:单张图像中存在多种类型文本的情况很常见,典型的以学生的试卷为代表,一张图像同时存在手写体和印刷体两种文本,这类情况下,可以尝试”1个检测模型+1个N分类模型+N个识别模型”的解决方案。 -其中不同类型文本共用同一个检测模型,N分类模型指额外训练一个分类器,将检测到的文本进行分类,如手写+印刷的情况就是二分类,N种语言就是N分类,在识别的部分,针对每个类型的文本单独训练一个识别模型,如手写+印刷的场景,就需要训练一个手写体识别模型,一个印刷体识别模型,如果一个文本框的分类结果是手写体,那么就传给手写体识别模型进行识别,其他情况同理。 - -#### Q: 多语言的字典里是混合了不同的语种,这个是有什么讲究吗?统一到一个字典里会对精度造成多大的损失? - -**A**:统一到一个字典里,会造成最后一层FC过大,增加模型大小。如果有特殊需求的话,可以把需要的几种语言合并字典训练模型,合并字典之后如果引入过多的形近字,可能会造成精度损失,字符平衡的问题可能也需要考虑一下。在PaddleOCR里暂时将语言字典分开。 - -#### Q:类似泰语这样的小语种,部分字会占用两个字符甚至三个字符,请问如何制作字典。 - -**A**:处理字符的时候,把多字符的当作一个字就行,字典中每行是一个字。 - ---- - -#### Q: 想把简历上的文字识别出来后,能够把关系一一对应起来,比如姓名和它后面的名字组成一对,籍贯、邮箱、学历等等都和各自的内容关联起来,这个应该如何处理,PPOCR目前支持吗? - -**A**: 这样的需求在企业应用中确实比较常见,但往往都是个性化的需求,没有非常规整统一的处理方式。常见的处理方式有如下两种: - -1. 对于单一版式、或者版式差异不大的应用场景,可以基于识别场景的一些先验信息,将识别内容进行配对; 比如运用表单结构信息:常见表单"姓名"关键字的后面,往往紧跟的就是名字信息 -2. 对于版式多样,或者无固定版式的场景, 需要借助于NLP中的NER技术,给识别内容中的某些字段,赋予key值 - -由于这部分需求和业务场景强相关,难以用一个统一的模型去处理,目前PPOCR暂不支持。 如果需要用到NER技术,可以参照Paddle团队的另一个开源套件: [PaddlePaddle/ERNIE](https://github.com/PaddlePaddle/ERNIE), 其提供的预训练模型ERNIE, 可以帮助提升NER任务的准确率。 - - - -### 1.6 训练过程与模型调优 - -#### Q: 增大batch_size模型训练速度没有明显提升 - -A:如果batch_size打得太大,加速效果不明显的话,可以试一下增大初始化内存的值,运行代码前设置环境变量: -export FLAGS_initial_cpu_memory_in_mb=2000 # 设置初始化内存约2G左右 - -#### Q: 预测时提示图像过大,显存、内存溢出了,应该如何处理? - -A: 可以按照这个PR的修改来缓解显存、内存占用 #2230 - -#### Q: 识别训练时,训练集精度已经到达90了,但验证集精度一直在70,涨不上去怎么办? - -A:训练集精度90,测试集70多的话,应该是过拟合了,有两个可尝试的方法:(1)加入更多的增广方式或者调大增广prob的概率,默认为0.4。(2)调大系统的l2 decay值 - - - -### 1.7 补充资料 - -#### Q: 对于小白如何快速入门中文OCR项目实践? - -A:建议可以先了解OCR方向的基础知识,大概了解基础的检测和识别模型算法。然后在Github上可以查看OCR方向相关的repo。目前来看,从内容的完备性来看,PaddleOCR的中英文双语教程文档是有明显优势的,在数据集、模型训练、预测部署文档详实,可以快速入手。而且还有微信用户群答疑,非常适合学习实践。项目地址:PaddleOCR - -AI 快车道课程:https://aistudio.baidu.com/aistudio/course/introduce/1519 - - - -## 2. PaddleOCR实战问题 - - - -### 2.1 PaddleOCR repo - -#### Q: PaddleOCR develop分支和dygraph分支的区别? - -**A**:目前PaddleOCR有四个分支,分别是: - -- develop:基于Paddle静态图开发的分支,推荐使用paddle1.8 或者2.0版本,该分支具备完善的模型训练、预测、推理部署、量化裁剪等功能,领先于release/1.1分支。 -- release/1.1:PaddleOCR 发布的第一个稳定版本,基于静态图开发,具备完善的训练、预测、推理部署、量化裁剪等功能。 -- dygraph:基于Paddle动态图开发的分支,目前仍在开发中,未来将作为主要开发分支,运行要求使用Paddle2.0.0版本。 -- release/2.0-rc1-0:PaddleOCR发布的第二个稳定版本,基于动态图和paddle2.0版本开发,动态图开发的工程更易于调试,目前支,支持模型训练、预测,暂不支持移动端部署。 - -如果您已经上手过PaddleOCR,并且希望在各种环境上部署PaddleOCR,目前建议使用静态图分支,develop或者release/1.1分支。如果您是初学者,想快速训练,调试PaddleOCR中的算法,建议尝鲜PaddleOCR dygraph分支。 - -**注意**:develop和dygraph分支要求的Paddle版本、本地环境有差别,请注意不同分支环境安装部分的差异。 - -#### Q:PaddleOCR与百度的其他OCR产品有什么区别? - -**A**:PaddleOCR主要聚焦通用ocr,如果有垂类需求,您可以用PaddleOCR+垂类数据自己训练; -如果缺少带标注的数据,或者不想投入研发成本,建议直接调用开放的API,开放的API覆盖了目前比较常见的一些垂类。 - - - -### 2.2 安装环境 - -#### Q:OSError: [WinError 126] 找不到指定的模块。mac pro python 3.4 shapely import 问题 - -A:这个问题是因为shapely库安装有误,可以参考 #212 这个issue重新安装一下 - -#### Q:PaddlePaddle怎么指定GPU运行 os.environ["CUDA_VISIBLE_DEVICES"]这种不生效 - -A:通过设置 export CUDA_VISIBLE_DEVICES='0'环境变量 - -#### Q:PaddleOCR是否支持在Windows或Mac系统上运行? - -A:PaddleOCR已完成Windows和Mac系统适配,运行时注意两点: -(1)在快速安装时,如果不想安装docker,可跳过第一步,直接从第二步安装paddle开始。 -(2)inference模型下载时,如果没有安装wget,可直接点击模型链接或将链接地址复制到浏览器进行下载,并解压放置到相应目录。 - - - -### 2.3 数据量说明 - -#### Q:简单的对于精度要求不高的OCR任务,数据集需要准备多少张呢? - -**A**:(1)训练数据的数量和需要解决问题的复杂度有关系。难度越大,精度要求越高,则数据集需求越大,而且一般情况实际中的训练数据越多效果越好。 - -(2)对于精度要求不高的场景,检测任务和识别任务需要的数据量是不一样的。对于检测任务,500张图像可以保证基本的检测效果。对于识别任务,需要保证识别字典中每个字符出现在不同场景的行文本图像数目需要大于200张(举例,如果有字典中有5个字,每个字都需要出现在200张图片以上,那么最少要求的图像数量应该在200-1000张之间),这样可以保证基本的识别效果。 - -#### Q:请问PaddleOCR项目中的中文超轻量和通用模型用了哪些数据集?训练多少样本,gpu什么配置,跑了多少个epoch,大概跑了多久? - -**A**: -(1)检测的话,LSVT街景数据集共3W张图像,超轻量模型,150epoch左右,2卡V100 跑了不到2天;通用模型:2卡V100 150epoch 不到4天。 -(2)识别的话,520W左右的数据集(真实数据26W+合成数据500W)训练,超轻量模型:4卡V100,总共训练了5天左右。通用模型:4卡V100,共训练6天。 - -超轻量模型训练分为2个阶段: -(1)全量数据训练50epoch,耗时3天 -(2)合成数据+真实数据按照1:1数据采样,进行finetune训练200epoch,耗时2天 - -通用模型训练: -真实数据+合成数据,动态采样(1:1)训练,200epoch,耗时 6天左右。 - -#### Q:训练文字识别模型,真实数据有30w,合成数据有500w,需要做样本均衡吗? - -A:需要,一般需要保证一个batch中真实数据样本和合成数据样本的比例是5:1~10:1左右效果比较理想。如果合成数据过大,会过拟合到合成数据,预测效果往往不佳。还有一种启发性的尝试是可以先用大量合成数据训练一个base模型,然后再用真实数据微调,在一些简单场景效果也是会有提升的。 - -#### Q: 当训练数据量少时,如何获取更多的数据? - -A:当训练数据量少时,可以尝试以下三种方式获取更多的数据:(1)人工采集更多的训练数据,最直接也是最有效的方式。(2)基于PIL和opencv基本图像处理或者变换。例如PIL中ImageFont, Image, ImageDraw三个模块将文字写到背景中,opencv的旋转仿射变换,高斯滤波等。(3)利用数据生成算法合成数据,例如pix2pix等算法。 - - - -### 2.4 数据标注与生成 - -> [!NOTE] -> StyleText 已经移动到 [PFCCLab/StyleText](https://github.com/PFCCLab/StyleText) - -#### Q: Style-Text 如何不文字风格迁移,就像普通文本生成程序一样默认字体直接输出到分割的背景图? - -**A**:使用image_synth模式会输出fake_bg.jpg,即为背景图。如果想要批量提取背景,可以稍微修改一下代码,将fake_bg保存下来即可。要修改的位置: -https://github.com/PaddlePaddle/PaddleOCR/blob/de3e2e7cd3b8b65ee02d7a41e570fa5b511a3c1d/StyleText/engine/synthesisers.py#L68 - -#### Q: 能否修改StyleText配置文件中的分辨率? - -**A**:StyleText目前的训练数据主要是高度32的图片,建议不要改变高度。未来我们会支持更丰富的分辨率。 - -#### Q: StyleText是否可以更换字体文件? - -**A**:StyleText项目中的字体文件为标准字体,主要用作模型的输入部分,不能够修改。 -StyleText的用途主要是:提取style_image中的字体、背景等style信息,根据语料生成同样style的图片。 - -#### Q: StyleText批量生成图片为什么没有输出? - -**A**:需要检查以下您配置文件中的路径是否都存在。尤其要注意的是[label_file配置](https://github.com/PFCCLab/StyleText/blob/main/README_ch.md#%E4%B8%89%E5%BF%AB%E9%80%9F%E4%B8%8A%E6%89%8B)。 -如果您使用的style_image输入没有label信息,您依然需要提供一个图片文件列表。 - -#### Q:使用StyleText进行数据合成时,文本(TextInput)的长度远超StyleInput的长度,该怎么处理与合成呢? - -**A**:在使用StyleText进行数据合成的时候,建议StyleInput的长度长于TextInput的长度。有2种方法可以处理上述问题: - -1. 将StyleInput按列的方向进行复制与扩充,直到其超过TextInput的长度。 -2. 将TextInput进行裁剪,保证每段TextInput都稍短于StyleInput,分别合成之后,再拼接在一起。 - -实际使用中发现,使用第2种方法的效果在长文本合成的场景中的合成效果更好,StyleText中提供的也是第2种数据合成的逻辑。 - -#### Q: StyleText 合成数据效果不好? - -**A**:StyleText模型生成的数据主要用于OCR识别模型的训练。PaddleOCR目前识别模型的输入为32 x N,因此当前版本模型主要适用高度为32的数据。 -建议要合成的数据尺寸设置为32 x N。尺寸相差不多的数据也可以生成,尺寸很大或很小的数据效果确实不佳。 - - - -### 2.5 预训练模型与微调 - -#### Q:如何更换文本检测/识别的backbone? - -A:无论是文字检测,还是文字识别,骨干网络的选择是预测效果和预测效率的权衡。一般,选择更大规模的骨干网络,例如ResNet101_vd,则检测或识别更准确,但预测耗时相应也会增加。而选择更小规模的骨干网络,例如MobileNetV3_small_x0_35,则预测更快,但检测或识别的准确率会大打折扣。幸运的是不同骨干网络的检测或识别效果与在ImageNet数据集图像1000分类任务效果正相关。飞桨图像分类套件PaddleClas汇总了ResNet_vd、Res2Net、HRNet、MobileNetV3、GhostNet等23种系列的分类网络结构,在上述图像分类任务的top1识别准确率,GPU(V100和T4)和CPU(骁龙855)的预测耗时以及相应的117个预训练模型下载地址。 - -(1)文字检测骨干网络的替换,主要是确定类似于ResNet的4个stages,以方便集成后续的类似FPN的检测头。此外,对于文字检测问题,使用ImageNet训练的分类预训练模型,可以加速收敛和效果提升。 - -(2)文字识别的骨干网络的替换,需要注意网络宽高stride的下降位置。由于文本识别一般宽高比例很大,因此高度下降频率少一些,宽度下降频率多一些。可以参考PaddleOCR中MobileNetV3骨干网络的改动。 - -#### Q: 参照文档做实际项目时,是重新训练还是在官方训练的基础上进行训练?具体如何操作? - -**A**: 基于官方提供的模型,进行finetune的话,收敛会更快一些。 具体操作上,以识别模型训练为例:如果修改了字符文件,可以设置pretraind_model为官方提供的预训练模型 - -#### Q: 下载的识别模型解压后缺失文件,没有期望的inference.pdiparams, inference.pdmodel等文件 - -A:用解压软件解压可能会出现这个问题,建议二次解压下或者用命令行解压tar xf - -#### Q: 为什么在checkpoints中load下载的预训练模型会报错? - -A: 这里有两个不同的概念: - -pretrained_model:指预训练模型,是已经训练完成的模型。这时会load预训练模型的参数,但并不会load学习率、优化器以及训练状态等。如果需要finetune,应该使用pretrained。 -checkpoints:指之前训练的中间结果,例如前一次训练到了100个epoch,想接着训练。这时会load尝试所有信息,包括模型的参数,之前的状态等。 - -#### Q: 如何对检测模型finetune,比如冻结前面的层或某些层使用小的学习率学习? - -**A**:如果是冻结某些层,可以将变量的stop_gradient属性设置为True,这样计算这个变量之前的所有参数都不会更新了,参考:https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/faq/train_cn.html#id4 - -如果对某些层使用更小的学习率学习,静态图里还不是很方便,一个方法是在参数初始化的时候,给权重的属性设置固定的学习率,参考:https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/api/paddle/fluid/param_attr/ParamAttr_cn.html#paramattr - -实际上我们实验发现,直接加载模型去fine-tune,不设置某些层不同学习率,效果也都不错 - - - -### 2.6 模型超参调整 - -#### Q: DB检测训练输入尺寸640,可以改大一些吗? - -A:不建议改大。检测模型训练输入尺寸是预处理中random crop后的尺寸,并非直接将原图进行resize,多数场景下这个尺寸并不小了,改大后可能反而并不合适,而且训练会变慢。另外,代码里可能有的地方参数按照预设输入尺寸适配的,改大后可能有隐藏风险。 - -#### Q: 预处理部分,图片的长和宽为什么要处理成32的倍数? - -A:以检测中的resnet骨干网络为例,图像输入网络之后,需要经过5次2倍降采样,共32倍,因此建议输入的图像尺寸为32的倍数。 - -#### Q: 在识别模型中,为什么降采样残差结构的stride为(2, 1)? - -**A**: stride为(2, 1),表示在图像y方向(高度方向)上stride为2,x方向(宽度方向)上为1。由于待识别的文本图像通常为长方形,这样只在高度方向做下采样,尽量保留宽度方向的序列信息,避免宽度方向下采样后丢失过多的文字信息。 - -#### Q:训练识别时,如何选择合适的网络输入shape? - -**A**:一般高度采用32,最长宽度的选择,有两种方法: - -(1)统计训练样本图像的宽高比分布。最大宽高比的选取考虑满足80%的训练样本。 - -(2)统计训练样本文字数目。最长字符数目的选取考虑满足80%的训练样本。然后中文字符长宽比近似认为是1,英文认为3:1,预估一个最长宽度。 - -#### Q:识别模型框出来的位置太紧凑,会丢失边缘的文字信息,导致识别错误 - -A:可以在命令中加入 --det_db_unclip_ratio ,参数定义位置,这个参数是检测后处理时控制文本框大小的,默认1.6,可以尝试改成2.5或者更大,反之,如果觉得文本框不够紧凑,也可以把该参数调小。 - - - -### 2.7 模型结构 - -#### Q:文本识别训练不加LSTM是否可以收敛? - -**A**:理论上是可以收敛的,加上LSTM模块主要是为了挖掘文字之间的序列关系,提升识别效果。对于有明显上下文语义的场景效果会比较明显。 - -#### Q:文本识别中LSTM和GRU如何选择? - -**A**:从项目实践经验来看,序列模块采用LSTM的识别效果优于GRU,但是LSTM的计算量比GRU大一些,可以根据自己实际情况选择。 - -#### Q:对于CRNN模型,backbone采用DenseNet和ResNet_vd,哪种网络结构更好? - -**A**:Backbone的识别效果在CRNN模型上的效果,与Imagenet 1000 图像分类任务上识别效果和效率一致。在图像分类任务上ResnNet_vd(79%+)的识别精度明显优于DenseNet(77%+),此外对于GPU,Nvidia针对ResNet系列模型做了优化,预测效率更高,所以相对而言,resnet_vd是较好选择。如果是移动端,可以优先考虑MobileNetV3系列。 - -#### Q: 如何根据不同的硬件平台选用不同的backbone? - -**A**:在不同的硬件上,不同的backbone的速度优势不同,可以根据不同平台的速度-精度图来确定backbone,这里可以参考[PaddleClas模型速度-精度图](https://github.com/PaddlePaddle/PaddleClas/tree/release/2.0/docs/zh_CN/models)。 - - - -### 2.8 PP-OCR系统 - -#### Q: 在PP-OCR系统中,文本检测的骨干网络为什么没有使用SE模块? - -**A**:SE模块是MobileNetV3网络一个重要模块,目的是估计特征图每个特征通道重要性,给特征图每个特征分配权重,提高网络的表达能力。但是,对于文本检测,输入网络的分辨率比较大,一般是640\*640,利用SE模块估计特征图每个特征通道重要性比较困难,网络提升能力有限,但是该模块又比较耗时,因此在PP-OCR系统中,文本检测的骨干网络没有使用SE模块。实验也表明,当去掉SE模块,超轻量模型大小可以减小40%,文本检测效果基本不受影响。详细可以参考PP-OCR技术文章,https://arxiv.org/abs/2009.09941. - -#### Q: PP-OCR系统中,文本检测的结果有置信度吗? - -**A**:文本检测的结果有置信度,由于推理过程中没有使用,所以没有显示的返回到最终结果中。如果需要文本检测结果的置信度,可以在[文本检测DB的后处理代码](../../ppocr/postprocess/db_postprocess.py)的155行,添加scores信息。这样,在[检测预测代码](../../tools/infer/predict_det.py)的197行,就可以拿到文本检测的scores信息。 - -#### Q: DB文本检测,特征提取网络金字塔构建的部分代码在哪儿? - -**A**:特征提取网络金字塔构建的部分:[代码位置](../../ppocr/modeling/necks/db_fpn.py)。ppocr/modeling文件夹里面是组网相关的代码,其中architectures是文本检测或者文本识别整体流程代码;backbones是骨干网络相关代码;necks是类似与FPN的颈函数代码;heads是提取文本检测或者文本识别预测结果相关的头函数;transforms是类似于TPS特征预处理模块。更多的信息可以参考[代码组织结构](./tree.md)。 - -#### Q:PaddleOCR如何做到横排和竖排同时支持的? - -**A**:合成了一批竖排文字,逆时针旋转90度后加入训练集与横排一起训练。预测时根据图片长宽比判断是否为竖排,若为竖排则将crop出的文本逆时针旋转90度后送入识别网络。 - -#### Q: 目前知识蒸馏有哪些主要的实践思路? - -**A**:知识蒸馏即利用教师模型指导学生模型的训练,目前有3种主要的蒸馏思路: - -1. 基于输出结果的蒸馏,即让学生模型学习教师模型的软标签(分类或者OCR识别等任务中)或者概率热度图(分割等任务中)。 -2. 基于特征图的蒸馏,即让学生模型学习教师模型中间层的特征图,拟合中间层的一些特征。 -3. 基于关系的蒸馏,针对不同的样本(假设个数为N),教师模型会有不同的输出,那么可以基于不同样本的输出,计算一个NxN的相关性矩阵,可以让学生模型去学习教师模型关于不同样本的相关性矩阵。 - -当然,知识蒸馏方法日新月异,也欢迎大家提出更多的总结与建议。 - -#### Q: 文字识别模型模型的输出矩阵需要进行解码才能得到识别的文本。代码中实现为preds_idx = preds.argmax(axis=2),也就是最佳路径解码法。这是一种贪心算法,是每一个时间步只将最大概率的字符作为当前时间步的预测输出,但得到的结果不一定是最好的。为什么不使用beam search这种方式进行解码呢? - -**A**:实验发现,使用贪心的方法去做解码,识别精度影响不大,但是速度方面的优势比较明显,因此PaddleOCR中使用贪心算法去做识别的解码。 - - - -### 2.9 端到端 - -#### Q: 端到端算法PGNet是否支持中文识别,速度会很慢嘛? - -**A**:目前开源的PGNet算法模型主要是用于检测英文数字,对于中文的识别需要自己训练,大家可以使用开源的端到端中文数据集,而对于复杂文本(弯曲文本)的识别,也可以自己构造一批数据集针对进行训练,对于推理速度,可以先将模型转换为inference再进行预测,速度应该会相当可观。 - -#### Q: 端到端算法PGNet提供了两种后处理方式,两者之间有什么区别呢? - -**A**: 两种后处理的区别主要在于速度的推理,config中PostProcess有fast/slow两种模式,slow模式的后处理速度慢,精度相对较高,fast模式的后处理速度快,精度也在可接受的范围之内。建议使用速度快的后处理方式。 - -#### Q: 使用PGNet进行eval报错? - -**A**: 需要注意,我们目前在release/2.1更新了评测代码,目前支持A,B两种评测模式: - -* A模式:该模式主要为了方便用户使用,与训练集一样的标注文件就可以正常进行eval操作, 代码中默认是A模式。 -* B模式:该模式主要为了保证我们的评测代码可以和Total Text官方的评测方式对齐,该模式下直接加载官方提供的mat文件进行eval。 - -#### Q: PGNet有中文预训练模型吗? - -**A**: 目前我们尚未提供针对中文的预训练模型,如有需要,可以尝试自己训练。具体需要修改的地方有: - - 1. [config文件中](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/configs/e2e/e2e_r50_vd_pg.yml#L23-L24),字典文件路径及语种设置; - 1. [网络结构中](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/ppocr/modeling/heads/e2e_pg_head.py#L181),`out_channels`修改为字典中的字符数目+1(考虑到空格); - 1. [loss中](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/ppocr/losses/e2e_pg_loss.py#L93),修改`37`为字典中的字符数目+1(考虑到空格); - -#### Q: 用于PGNet的训练集,文本框的标注有要求吗? - -**A**: PGNet支持多点标注,比如4点、8点、14点等。但需要注意的是,标注点尽可能分布均匀(相邻标注点间隔距离均匀一致),且label文件中的标注点需要从标注框的左上角开始,按标注点顺时针顺序依次编写,以上问题都可能对训练精度造成影响。 -我们提供的,基于Total Text数据集的PGNet预训练模型使用了14点标注方式。 - -#### Q: 用PGNet做进行端到端训练时,数据集标注的点的个数必须都是统一一样的吗? 能不能随意标点数,只要能够按顺时针从左上角开始标这样? - -**A**: 目前代码要求标注为统一的点数。 - - - -### 2.10 模型效果与效果不一致 - -#### Q: PP-OCR检测效果不好,该如何优化? - -A: 具体问题具体分析: -如果在你的场景上检测效果不可用,首选是在你的数据上做finetune训练; -如果图像过大,文字过于密集,建议不要过度压缩图像,可以尝试修改检测预处理的resize逻辑,防止图像被过度压缩; -检测框大小过于紧贴文字或检测框过大,可以调整db_unclip_ratio这个参数,加大参数可以扩大检测框,减小参数可以减小检测框大小; -检测框存在很多漏检问题,可以减小DB检测后处理的阈值参数det_db_box_thresh,防止一些检测框被过滤掉,也可以尝试设置det_db_score_mode为'slow'; -其他方法可以选择use_dilation为True,对检测输出的feature map做膨胀处理,一般情况下,会有效果改善; - -#### Q:同一张图通用检测出21个条目,轻量级检测出26个 ,难道不是轻量级的好吗? - -**A**:可以主要参考可视化效果,通用模型更倾向于检测一整行文字,轻量级可能会有一行文字被分成两段检测的情况,不是数量越多,效果就越好。 - -#### Q: DB有些框太贴文本了反而去掉了一些文本的边角影响识别,这个问题有什么办法可以缓解吗? - -**A**:可以把后处理的参数unclip_ratio适当调大一点。 - -#### Q: 使用合成数据精调小模型后,效果可以,但是还没开源的小infer模型效果好,这是为什么呢? - -**A**:(1)要保证使用的配置文件和pretrain weights是对应的; - -(2)在微调时,一般都需要真实数据,如果使用合成数据,效果反而可能会有下降,PaddleOCR中放出的识别inference模型也是基于预训练模型在真实数据上微调得到的,效果提升比较明显; - -(3)在训练的时候,文本长度超过25的训练图像都会被丢弃,因此需要看下真正参与训练的图像有多少,太少的话也容易过拟合。 - -#### Q: 表格识别中,如何提高单字的识别结果? - -**A**: 首先需要确认一下检测模型有没有有效的检测出单个字符,如果没有的话,需要在训练集当中添加相应的单字数据集。 - -#### Q: 动态图分支(dygraph,release/2.0),训练模型和推理模型效果不一致 - -A:当前问题表现为:使用训练完的模型直接测试结果较好,但是转换为inference model后,预测结果不一致;出现这个问题一般是两个原因: - -1. 预处理函数设置的不一致 -2. 后处理参数不一致 repo中config.yml文件的前后处理参数和inference预测默认的超参数有不一致的地方,建议排查下训练模型预测和inference预测的前后处理, 参考issue。 - -#### Q: 自己训练的det模型,在同一张图片上,inference模型与eval模型结果差别很大,为什么? - -A:这是由于图片预处理不同造成的。如果训练的det模型图片输入并不是默认的shape[600, 600],eval的程序中图片预处理方式与train时一致 (由xxx_reader.yml中的test_image_shape参数决定缩放大小,但predict_eval.py中的图片预处理方式由程序里的preprocess_params决定, 最好不要传入max_side_len,而是传入和训练时一样大小的test_image_shape。 - -#### Q: 训练模型和测试模型的检测结果差距较大 - -**A**:1. 检查两个模型使用的后处理参数是否是一样的,训练的后处理参数在配置文件中的PostProcess部分,测试模型的后处理参数在tools/infer/utility.py中,最新代码中两个后处理参数已保持一致。 - -#### Q: PaddleOCR模型Python端预测和C++预测结果不一致? - -A:正常来说,python端预测和C++预测文本是一致的,如果预测结果差异较大, 建议首先排查diff出现在检测模型还是识别模型,或者尝试换其他模型是否有类似的问题。 其次,检查python端和C++端数据处理部分是否存在差异,建议保存环境,更新PaddleOCR代码再试下。 如果更新代码或者更新代码都没能解决,建议在PaddleOCR微信群里或者issue中抛出您的问题。 - -用户总结的排查步骤:https://github.com/PaddlePaddle/PaddleOCR/issues/2470 - - - -### 2.11 训练调试与配置文件 - -#### Q: 某个类别的样本比较少,通过增加训练的迭代次数或者是epoch,变相增加小样本的数目,这样能缓解这个问题么? - -A: 尽量保证类别均衡, 某些类别样本少,可以通过补充合成数据的方式处理;实验证明训练集中出现频次较少的字符,识别效果会比较差,增加迭代次数不能改变样本量少的问题。 - -#### Q:文本检测换成自己的数据没法训练,有一些”###”是什么意思? - -**A**:数据格式有问题,”###” 表示要被忽略的文本区域,所以你的数据都被跳过了,可以换成其他任意字符或者就写个空的。 - -#### Q:如何调试数据读取程序? - -A:tools/train.py中有一个test_reader()函数用于调试数据读取。 - -#### Q:中文文本检测、文本识别构建训练集的话,大概需要多少数据量 - -A:检测需要的数据相对较少,在PaddleOCR模型的基础上进行Fine-tune,一般需要500张可达到不错的效果。 识别分英文和中文,一般英文场景需要几十万数据可达到不错的效果,中文则需要几百万甚至更多。 - -#### Q: config yml文件中的ratio_list参数的作用是什么? - -**A**: 在动态图中,ratio_list在有多个数据源的情况下使用,ratio_list中的每个值是每个epoch从对应数据源采样数据的比例。如ratio_list=[0.3,0.2],label_file_list=['data1','data2'],代表每个epoch的训练数据包含data1 30%的数据,和data2里 20%的数据,ratio_list中数值的和不需要等于1。ratio_list和label_file_list的长度必须一致。 - -静态图检测数据采样的逻辑与动态图不同,但基本不影响训练精度。 - -在静态图中,使用 检测 dataloader读取数据时,会先设置每个epoch的数据量,比如这里设置为1000,ratio_list中的值表示在1000中的占比,比如ratio_list是[0.3, 0.7],则表示使用两个数据源,每个epoch从第一个数据源采样1000*0.3=300张图,从第二个数据源采样700张图。ratio_list的值的和也不需要等于1。 - -#### Q: iaa里面添加的数据增强方式,是每张图像训练都会做增强还是随机的?如何添加一个数据增强方法? - -**A**:iaa增强的训练配置参考:[这里](https://github.com/PaddlePaddle/PaddleOCR/blob/0ccc1720c252beb277b9e522a1b228eb6abffb8a/configs/det/ch_ppocr_v2.0/ch_det_mv3_db_v2.0.yml#L82)。其中{ 'type': Fliplr, 'args': { 'p': 0.5 } } p是概率。新增数据增强,可以参考[这个方法](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.1/doc/doc_ch/add_new_algorithm.md#%E6%95%B0%E6%8D%AE%E5%8A%A0%E8%BD%BD%E5%92%8C%E5%A4%84%E7%90%86) - -#### Q: 怎么加速训练过程呢? - -**A**:OCR模型训练过程中一般包含大量的数据增广,这些数据增广是比较耗时的,因此可以离线生成大量增广后的图像,直接送入网络进行训练,机器资源充足的情况下,也可以使用分布式训练的方法,可以参考[分布式训练教程文档](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/doc/doc_ch/distributed_training.md)。 - -#### Q: 一些特殊场景的数据识别效果差,但是数据量很少,不够用来finetune怎么办? - -**A**:您可以合成一些接近使用场景的数据用于训练。 -我们计划推出基于特定场景的文本数据合成工具,请您持续关注PaddleOCR的近期更新。 - -#### Q: PaddleOCR可以识别灰度图吗? - -**A**:PaddleOCR的模型均为三通道输入。如果您想使用灰度图作为输入,建议直接用3通道的模式读入灰度图, -或者将单通道图像转换为三通道图像再识别。例如,opencv的cvtColor函数就可以将灰度图转换为RGB三通道模式。 - -#### Q: 如何合成手写中文数据集? - -**A**: 手写数据集可以通过手写单字数据集合成得到。随机选取一定数量的单字图片和对应的label,将图片高度resize为随机的统一高度后拼接在一起,即可得到合成数据集。对于需要添加文字背景的情况,建议使用阈值化将单字图片的白色背景处理为透明背景,再与真实背景图进行合成。具体可以参考文档[手写数据集](https://github.com/PaddlePaddle/PaddleOCR/blob/a72d6f23be9979e0c103d911a9dca3e4613e8ccf/doc/doc_ch/handwritten_datasets.md)。 - -#### Q:PaddleOCR默认不是200个step保存一次模型吗?为啥文件夹下面都没有生成 - -**A**:因为默认保存的起始点不是0,而是4000,将eval_batch_step [4000, 5000]改为[0, 2000] 就是从第0次迭代开始,每2000迭代保存一次模型 - -#### Q: PaddleOCR在训练的时候一直使用cosine_decay的学习率下降策略,这是为什么呢? - -**A**:cosine_decay表示在训练的过程中,学习率按照cosine的变化趋势逐渐下降至0,在迭代轮数更长的情况下,比常量的学习率变化策略会有更好的收敛效果,因此在实际训练的时候,均采用了cosine_decay,来获得精度更高的模型。 - -#### Q: Cosine学习率的更新策略是怎样的?训练过程中为什么会在一个值上停很久? - -**A**: Cosine学习率的说明可以参考[这里](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/optimizer/lr/CosineAnnealingDecay_cn.html#cosineannealingdecay) - -在PaddleOCR中,为了让学习率更加平缓,我们将其中的epoch调整成了iter。 -学习率的更新会和总的iter数量有关。当iter比较大时,会经过较多iter才能看出学习率的值有变化。 - -#### Q: 之前的CosineWarmup方法为什么不见了? - -**A**: 我们对代码结构进行了调整,目前的Cosine可以覆盖原有的CosineWarmup的功能,只需要在配置文件中增加相应配置即可。 -例如下面的代码,可以设置warmup为2个epoch: - -``` -lr: - name: Cosine - learning_rate: 0.001 - warmup_epoch: 2 -``` - -#### Q: 训练识别和检测时学习率要加上warmup,目的是什么? - -**A**: Warmup机制先使学习率从一个较小的值逐步升到一个较大的值,而不是直接就使用较大的学习率,这样有助于模型的稳定收敛。在OCR检测和OCR识别中,一般会带来精度~0.5%的提升。 - -#### Q: 关于dygraph分支中,文本识别模型训练,要使用数据增强应该如何设置? - -**A**:可以参考[配置文件](../../configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml)在Train['dataset']['transforms']添加RecAug字段,使数据增强生效。可以通过添加对aug_prob设置,表示每种数据增强采用的概率。aug_prob默认是0.4。详细设置可以参考[ISSUE 1744](https://github.com/PaddlePaddle/PaddleOCR/issues/1744)。 - -#### Q: 训练过程中,训练程序意外退出/挂起,应该如何解决? - -**A**: 考虑内存,显存(使用GPU训练的话)是否不足,可在配置文件中,将训练和评估的batch size调小一些。需要注意,训练batch size调小时,学习率learning rate也要调小,一般可按等比例调整。 - -#### Q: 训练程序启动后直到结束,看不到训练过程log? - -**A**: 可以从以下三方面考虑: - - 1. 检查训练进程是否正常退出、显存占用是否释放、是否有残留进程,如果确定是训练程序卡死,可以检查环境配置,遇到环境问题建议使用docker,可以参考说明文档[安装](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/doc/doc_ch/installation.md)。 - 2. 检查数据集的数据量是否太小,可调小batch size从而增加一个epoch中的训练step数量,或在训练config文件中,将参数print_batch_step改为1,即每一个step打印一次log信息。 - 3. 如果使用私有数据集训练,可先用PaddleOCR提供/推荐的数据集进行训练,排查私有数据集是否存在问题。 - -#### Q: 配置文件中的参数num workers是什么意思,应该如何设置? - -**A**: 训练数据的读取需要硬盘IO,而硬盘IO速度远小于GPU运算速度,为了避免数据读取成为训练速度瓶颈,可以使用多进程读取数据,num workers表示数据读取的进程数量,0表示不使用多进程读取。在Linux系统下,多进程读取数据时,进程间通信需要基于共享内存,因此使用多进程读取数据时,建议设置共享内存不低于2GB,最好可以达到8GB,此时,num workers可以设置为CPU核心数。如果机器硬件配置较低,或训练进程卡死、dataloader报错,可以将num workers设置为0,即不使用多进程读取数据。 - - - -### 2.12 预测 - -#### Q: 为什么PaddleOCR检测预测是只支持一张图片测试?即test_batch_size_per_card=1 - -A:测试的时候,对图像等比例缩放,最长边960,不同图像等比例缩放后长宽不一致,无法组成batch,所以设置为test_batch_size为1。 - -#### Q: PaddleOCR支持tensorrt推理吗? - -A: 支持的,需要在编译的时候将CMakeLists.txt文件当中,将相关代码option(WITH_TENSORRT "Compile demo with TensorRT." OFF)的OFF改成ON。关于服务器端部署的更多设置,可以参考飞桨官网 - -#### Q: 如何使用TensorRT加速PaddleOCR预测? - -**A**: 目前paddle的dygraph分支已经支持了python和C++ TensorRT预测的代码,python端inference预测时把参数[--use_tensorrt=True](https://github.com/PaddlePaddle/PaddleOCR/blob/3ec57e8df9263de6fa897e33d2d91bc5d0849ef3/tools/infer/utility.py#L37)即可, -C++TensorRT预测需要使用支持TRT的预测库并在编译时打开[-DWITH_TENSORRT=ON](https://github.com/PaddlePaddle/PaddleOCR/blob/3ec57e8df9263de6fa897e33d2d91bc5d0849ef3/deploy/cpp_infer/tools/build.sh#L15)。 -如果想修改其他分支代码支持TensorRT预测,可以参考[PR](https://github.com/PaddlePaddle/PaddleOCR/pull/2921)。 - -注:建议使用TensorRT大于等于6.1.0.5以上的版本。 - -#### Q: 为什么识别模型做预测的时候,预测图片的数量数量还会影响预测的精度 -**A**: 推理时识别模型默认的batch_size=6, 如预测图片长度变化大,可能影响预测效果。如果出现上述问题可在推理的时候设置识别bs=1,命令如下: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/ch/word_4.jpg" --rec_model_dir="./ch_PP-OCRv3_rec_infer/" --rec_batch_num=1 -``` - - - -### 2.13 推理部署 - -#### Q:PaddleOCR模型推理方式有几种?各自的优缺点是什么 - -**A**:目前推理方式支持基于训练引擎推理和基于预测引擎推理。 - -(1)基于训练引擎推理不需要转换模型,但是需要先组网再load参数,语言只支持python,不适合系统集成。 - -(2)基于预测引擎的推理需要先转换模型为inference格式,然后可以进行不需要组网的推理,语言支持c++和python,适合系统集成。 - -#### Q:PaddleOCR中,对于模型预测加速,CPU加速的途径有哪些?基于TenorRT加速GPU对输入有什么要求? - -**A**:(1)CPU可以使用mkldnn进行加速;对于python inference的话,可以把enable_mkldnn改为true,[参考代码](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/tools/infer/utility.py#L99),对于cpp inference的话,可参考[文档](https://github.com/PaddlePaddle/PaddleOCR/tree/dygraph/deploy/cpp_infer) - -(2)GPU需要注意变长输入问题等,TRT6 之后才支持变长输入 - -#### Q:hubserving、pdserving这两种部署方式区别是什么? - -A:hubserving原本是paddlehub的配套服务部署工具,可以很方便的将paddlehub内置的模型部署为服务,paddleocr使用了这个功能,并将模型路径等参数暴露出来方便用户自定义修改。paddle serving是面向所有paddle模型的部署工具,文档中可以看到我们提供了快速版和标准版,其中快速版和hubserving的本质是一样的,而标准版基于rpc,更稳定,更适合分布式部署。 - -#### Q: 目前paddle hub serving 只支持 imgpath,如果我想用imgurl 去哪里改呢? - -A:图片是在[这里](https://github.com/PaddlePaddle/PaddleOCR/blob/67ef25d593c4eabfaaceb22daade4577f53bed81/deploy/hubserving/ocr_system/module.py#L55)读取的, 可以参考下面的写法,将url path转化为np array - -``` -response = request.urlopen('http://i1.whymtj.com/uploads/tu/201902/9999/52491ae4ba.jpg') -img_array = np.array(bytearray(response.read()), dtype=np.uint8) -img = cv.imdecode(img_array, -1) -``` - -#### Q: C++ 端侧部署可以只对OCR的检测部署吗? - -A:可以的,识别和检测模块是解耦的。如果想对检测部署,需要自己修改一下main函数, 只保留检测相关就可以: [参考](https://github.com/PaddlePaddle/PaddleOCR/blob/de3e2e7cd3b8b65ee02d7a41e570fa5b511a3c1d/deploy/cpp_infer/src/main.cpp#L72) - -#### Q:服务部署可以只发布文本识别,而不带文本检测模型么? - -A:可以的。默认的服务部署是检测和识别串联预测的。也支持单独发布文本检测或文本识别模型,比如使用PaddleHUBPaddleOCR 模型时,deploy下有三个文件夹,分别是 -ocr_det:检测预测 -ocr_rec: 识别预测 -ocr_system: 检测识别串联预测 - -#### Q: lite预测库和nb模型版本不匹配,该如何解决? - -**A**: 如果可以正常预测就不用管,如果这个问题导致无法正常预测,可以尝试使用同一个commit的Paddle Lite代码编译预测库和opt文件,可以参考[移动端部署教程](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.1/deploy/lite/readme.md)。 - -#### Q:如何将PaddleOCR预测模型封装成SDK - -**A**:如果是Python的话,可以使用tools/infer/predict_system.py中的TextSystem进行sdk封装,如果是c++的话,可以使用deploy/cpp_infer/src下面的DBDetector和CRNNRecognizer完成封装 - -#### Q:为什么PaddleOCR检测预测是只支持一张图片测试?即test_batch_size_per_card=1 - -**A**:测试的时候,对图像等比例缩放,最长边960,不同图像等比例缩放后长宽不一致,无法组成batch,所以设置为test_batch_size为1。 - -#### Q:为什么第一张张图预测时间很长,第二张之后预测时间会降低? - -**A**:第一张图需要显存资源初始化,耗时较多。完成模型加载后,之后的预测时间会明显缩短。 - -#### Q: 采用Paddle-Lite进行端侧部署,出现问题,环境没问题。 - -**A**:如果你的预测库是自己编译的,那么你的nb文件也要自己编译,用同一个lite版本。不能直接用下载的nb文件,因为版本不同。 - -#### Q: 如何多进程运行paddleocr? - -**A**:实例化多个paddleocr服务,然后将服务注册到注册中心,之后通过注册中心统一调度即可,关于注册中心,可以搜索eureka了解一下具体使用,其他的注册中心也行。 - -#### Q: 如何多进程预测? - -**A**: 近期PaddleOCR新增了[多进程预测控制参数](https://github.com/PaddlePaddle/PaddleOCR/blob/a312647be716776c1aac33ff939ae358a39e8188/tools/infer/utility.py#L103),`use_mp`表示是否使用多进程,`total_process_num`表示在使用多进程时的进程数。具体使用方式请参考[文档](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/doc/doc_ch/inference.md#1-%E8%B6%85%E8%BD%BB%E9%87%8F%E4%B8%AD%E6%96%87ocr%E6%A8%A1%E5%9E%8B%E6%8E%A8%E7%90%86)。 - -#### Q: 怎么解决paddleOCR在T4卡上有越预测越慢的情况? - -**A**: - -1. T4 GPU没有主动散热,因此在测试的时候需要在每次infer之后需要sleep 30ms,否则机器容易因为过热而降频(inference速度会变慢),温度过高也有可能会导致宕机。 -2. T4在不使用的时候,也有可能会降频,因此在做benchmark的时候需要锁频,下面这两条命令可以进行锁频。 - -``` -nvidia-smi -i 0 -pm ENABLED -nvidia-smi --lock-gpu-clocks=1590 -i 0 -``` - -#### Q: 在windows上进行cpp inference的部署时,总是提示找不到`paddle_fluid.dll`和`opencv_world346.dll`, - -**A**:有2种方法可以解决这个问题: - -1. 将paddle预测库和opencv库的地址添加到系统环境变量中。 -2. 将提示缺失的dll文件拷贝到编译产出的`ocr_system.exe`文件夹中。 - -#### Q: win下C++部署中文识别乱码的解决方法 - -**A**: win下编码格式不是utf8,而ppocr_keys_v1.txt的编码格式的utf8,将ppocr_keys_v1.txt 的编码从utf-8修改为 Ansi 编码格式就行了。 - -#### Q: windows 3060显卡GPU模式启动 加载模型慢。 - -**A**: 30系列的显卡需要使用cuda11。 - - -#### Q:想在Mac上部署,从哪里下载预测库呢? - -**A**:Mac上的Paddle预测库可以从这里下载:[https://paddle-inference-lib.bj.bcebos.com/mac/2.0.0/cpu_avx_openblas/paddle_inference.tgz](https://paddle-inference-lib.bj.bcebos.com/mac/2.0.0/cpu_avx_openblas/paddle_inference.tgz) - - -#### Q:内网环境如何进行服务化部署呢? - -**A**:仍然可以使用PaddleServing或者HubServing进行服务化部署,保证内网地址可以访问即可。 - -#### Q: 使用hub_serving部署,延时较高,可能的原因是什么呀? - -**A**: 首先,测试的时候第一张图延时较高,可以多测试几张然后观察后几张图的速度;其次,如果是在cpu端部署serving端模型(如backbone为ResNet34),耗时较慢,建议在cpu端部署mobile(如backbone为MobileNetV3)模型。 - -#### Q: 在使用PaddleLite进行预测部署时,启动预测后卡死/手机死机? - -**A**: 请检查模型转换时所用PaddleLite的版本,和预测库的版本是否对齐。即PaddleLite版本为2.8,则预测库版本也要为2.8。 - -#### Q: 预测时显存爆炸、内存泄漏问题? - -**A**: 打开显存/内存优化开关`enable_memory_optim`可以解决该问题,相关代码已合入,[查看详情](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/tools/infer/utility.py#L153)。 diff --git a/doc/doc_ch/PP-OCRv3_introduction.md b/doc/doc_ch/PP-OCRv3_introduction.md deleted file mode 100644 index 446af23e4e..0000000000 --- a/doc/doc_ch/PP-OCRv3_introduction.md +++ /dev/null @@ -1,222 +0,0 @@ -[English](../doc_en/PP-OCRv3_introduction_en.md) | 简体中文 - -# PP-OCRv3 - -- [1. 简介](#1) -- [2. 检测优化](#2) -- [3. 识别优化](#3) -- [4. 端到端评估](#4) - - - -## 1. 简介 - -PP-OCRv3在PP-OCRv2的基础上进一步升级。整体的框架图保持了与PP-OCRv2相同的pipeline,针对检测模型和识别模型进行了优化。其中,检测模块仍基于DB算法优化,而识别模块不再采用CRNN,换成了IJCAI 2022最新收录的文本识别算法[SVTR](https://arxiv.org/abs/2205.00159),并对其进行产业适配。PP-OCRv3系统框图如下所示(粉色框中为PP-OCRv3新增策略): - -
- -
- - -从算法改进思路上看,分别针对检测和识别模型,进行了共9个方面的改进: - -- 检测模块: - - LK-PAN:大感受野的PAN结构; - - DML:教师模型互学习策略; - - RSE-FPN:残差注意力机制的FPN结构; - - -- 识别模块: - - SVTR_LCNet:轻量级文本识别网络; - - GTC:Attention指导CTC训练策略; - - TextConAug:挖掘文字上下文信息的数据增广策略; - - TextRotNet:自监督的预训练模型; - - UDML:联合互学习策略; - - UIM:无标注数据挖掘方案。 - -从效果上看,速度可比情况下,多种场景精度均有大幅提升: -- 中文场景,相对于PP-OCRv2中文模型提升超5%; -- 英文数字场景,相比于PP-OCRv2英文模型提升11%; -- 多语言场景,优化80+语种识别效果,平均准确率提升超5%。 - - - -## 2. 检测优化 - -PP-OCRv3检测模型是对PP-OCRv2中的[CML](https://arxiv.org/pdf/2109.03144.pdf)(Collaborative Mutual Learning) 协同互学习文本检测蒸馏策略进行了升级。如下图所示,CML的核心思想结合了①传统的Teacher指导Student的标准蒸馏与 ②Students网络之间的DML互学习,可以让Students网络互学习的同时,Teacher网络予以指导。PP-OCRv3分别针对教师模型和学生模型进行进一步效果优化。其中,在对教师模型优化时,提出了大感受野的PAN结构LK-PAN和引入了DML(Deep Mutual Learning)蒸馏策略;在对学生模型优化时,提出了残差注意力机制的FPN结构RSE-FPN。 - -
- -
- -消融实验如下: - -|序号|策略|模型大小|hmean|速度(cpu + mkldnn)| -|-|-|-|-|-| -|baseline teacher|PP-OCR server|49.0M|83.20%|171ms| -|teacher1|DB-R50-LK-PAN|124.0M|85.00%|396ms| -|teacher2|DB-R50-LK-PAN-DML|124.0M|86.00%|396ms| -|baseline student|PP-OCRv2|3.0M|83.20%|117ms| -|student0|DB-MV3-RSE-FPN|3.6M|84.50%|124ms| -|student1|DB-MV3-CML(teacher2)|3.0M|84.30%|117ms| -|student2|DB-MV3-RSE-FPN-CML(teacher2)|3.60M|85.40%|124ms| - -测试环境: Intel Gold 6148 CPU,预测时开启MKLDNN加速。 - -**(1)LK-PAN:大感受野的PAN结构** - -LK-PAN (Large Kernel PAN) 是一个具有更大感受野的轻量级[PAN](https://arxiv.org/pdf/1803.01534.pdf)结构,核心是将PAN结构的path augmentation中卷积核从`3*3`改为`9*9`。通过增大卷积核,提升特征图每个位置覆盖的感受野,更容易检测大字体的文字以及极端长宽比的文字。使用LK-PAN结构,可以将教师模型的hmean从83.2%提升到85.0%。 - -
- -
- -**(2)DML:教师模型互学习策略** - -[DML](https://arxiv.org/abs/1706.00384) (Deep Mutual Learning)互学习蒸馏方法,如下图所示,通过两个结构相同的模型互相学习,可以有效提升文本检测模型的精度。教师模型采用DML策略,hmean从85%提升到86%。将PP-OCRv2中CML的教师模型更新为上述更高精度的教师模型,学生模型的hmean可以进一步从83.2%提升到84.3%。 - -
- -
- -**(3)RSE-FPN:残差注意力机制的FPN结构** - -RSE-FPN(Residual Squeeze-and-Excitation FPN)如下图所示,引入残差结构和通道注意力结构,将FPN中的卷积层更换为通道注意力结构的RSEConv层,进一步提升特征图的表征能力。考虑到PP-OCRv2的检测模型中FPN通道数非常小,仅为96,如果直接用SEblock代替FPN中卷积会导致某些通道的特征被抑制,精度会下降。RSEConv引入残差结构会缓解上述问题,提升文本检测效果。进一步将PP-OCRv2中CML的学生模型的FPN结构更新为RSE-FPN,学生模型的hmean可以进一步从84.3%提升到85.4%。 - -
- -
- - - -## 3. 识别优化 - -PP-OCRv3的识别模块是基于文本识别算法[SVTR](https://arxiv.org/abs/2205.00159)优化。SVTR不再采用RNN结构,通过引入Transformers结构更加有效地挖掘文本行图像的上下文信息,从而提升文本识别能力。直接将PP-OCRv2的识别模型,替换成SVTR_Tiny,识别准确率从74.8%提升到80.1%(+5.3%),但是预测速度慢了将近11倍,CPU上预测一条文本行,将近100ms。因此,如下图所示,PP-OCRv3采用如下6个优化策略进行识别模型加速。 - -
- -
- -基于上述策略,PP-OCRv3识别模型相比PP-OCRv2,在速度可比的情况下,精度进一步提升4.6%。 具体消融实验如下所示: - -| ID | 策略 | 模型大小 | 精度 | 预测耗时(CPU + MKLDNN)| -|-----|-----|--------|----| --- | -| 01 | PP-OCRv2 | 8.0M | 74.80% | 8.54ms | -| 02 | SVTR_Tiny | 21.0M | 80.10% | 97.00ms | -| 03 | SVTR_LCNet(h32) | 12.0M | 71.90% | 6.60ms | -| 04 | SVTR_LCNet(h48) | 12.0M | 73.98% | 7.60ms | -| 05 | + GTC | 12.0M | 75.80% | 7.60ms | -| 06 | + TextConAug | 12.0M | 76.30% | 7.60ms | -| 07 | + TextRotNet | 12.0M | 76.90% | 7.60ms | -| 08 | + UDML | 12.0M | 78.40% | 7.60ms | -| 09 | + UIM | 12.0M | 79.40% | 7.60ms | - -注: 测试速度时,实验01-03输入图片尺寸均为(3,32,320),04-08输入图片尺寸均为(3,48,320)。在实际预测时,图像为变长输入,速度会有所变化。测试环境: Intel Gold 6148 CPU,预测时开启MKLDNN加速。 - -**(1)SVTR_LCNet:轻量级文本识别网络** - -SVTR_LCNet是针对文本识别任务,将基于Transformer的[SVTR](https://arxiv.org/abs/2205.00159)网络和轻量级CNN网络[PP-LCNet](https://arxiv.org/abs/2109.15099) 融合的一种轻量级文本识别网络。使用该网络,预测速度优于PP-OCRv2的识别模型20%,但是由于没有采用蒸馏策略,该识别模型效果略差。此外,进一步将输入图片规范化高度从32提升到48,预测速度稍微变慢,但是模型效果大幅提升,识别准确率达到73.98%(+2.08%),接近PP-OCRv2采用蒸馏策略的识别模型效果。 - -SVTR_Tiny 网络结构如下所示: - -
- -
- - -由于 MKLDNN 加速库支持的模型结构有限,SVTR 在 CPU+MKLDNN 上相比 PP-OCRv2 慢了10倍。PP-OCRv3 期望在提升模型精度的同时,不带来额外的推理耗时。通过分析发现,SVTR_Tiny 结构的主要耗时模块为 Mixing Block,因此我们对 SVTR_Tiny 的结构进行了一系列优化(详细速度数据请参考下方消融实验表格): - - -1. 将 SVTR 网络前半部分替换为 PP-LCNet 的前三个stage,保留4个 Global Mixing Block ,精度为76%,加速69%,网络结构如下所示: -
- -
-2. 将4个 Global Mixing Block 减小到2个,精度为72.9%,加速69%,网络结构如下所示: -
- -
-3. 实验发现 Global Mixing Block 的预测速度与输入其特征的shape有关,因此后移 Global Mixing Block 的位置到池化层之后,精度下降为71.9%,速度超越基于CNN结构的PP-OCRv2-baseline 22%,网络结构如下所示: -
- -
- -具体消融实验如下所示: - -| ID | 策略 | 模型大小 | 精度 | 速度(CPU + MKLDNN)| -|-----|-----|--------|----| --- | -| 01 | PP-OCRv2-baseline | 8.0M | 69.30% | 8.54ms | -| 02 | SVTR_Tiny | 21.0M | 80.10% | 97.00ms | -| 03 | SVTR_LCNet(G4) | 9.2M | 76.00% | 30.00ms | -| 04 | SVTR_LCNet(G2) | 13.0M | 72.98% | 9.37ms | -| 05 | SVTR_LCNet(h32) | 12.0M | 71.90% | 6.60ms | -| 06 | SVTR_LCNet(h48) | 12.0M | 73.98% | 7.60ms | - -注: 测试速度时,01-05输入图片尺寸均为(3,32,320); PP-OCRv2-baseline 代表没有借助蒸馏方法训练得到的模型 - -**(2)GTC:Attention指导CTC训练策略** - -[GTC](https://arxiv.org/pdf/2002.01276.pdf)(Guided Training of CTC),利用Attention模块CTC训练,融合多种文本特征的表达,是一种有效的提升文本识别的策略。使用该策略,预测时完全去除 Attention 模块,在推理阶段不增加任何耗时,识别模型的准确率进一步提升到75.8%(+1.82%)。训练流程如下所示: -
- -
- -**(3)TextConAug:挖掘文字上下文信息的数据增广策略** - -TextConAug是一种挖掘文字上下文信息的数据增广策略,主要思想来源于论文[ConCLR](https://www.cse.cuhk.edu.hk/~byu/papers/C139-AAAI2022-ConCLR.pdf),作者提出ConAug数据增广,在一个batch内对2张不同的图像进行联结,组成新的图像并进行自监督对比学习。PP-OCRv3将此方法应用到有监督的学习任务中,设计了TextConAug数据增强方法,可以丰富训练数据上下文信息,提升训练数据多样性。使用该策略,识别模型的准确率进一步提升到76.3%(+0.5%)。TextConAug示意图如下所示: - -
- -
- - -**(4)TextRotNet:自监督的预训练模型** - -TextRotNet是使用大量无标注的文本行数据,通过自监督方式训练的预训练模型,参考于论文[STR-Fewer-Labels](https://github.com/ku21fan/STR-Fewer-Labels)。该模型可以初始化SVTR_LCNet的初始权重,从而帮助文本识别模型收敛到更佳位置。使用该策略,识别模型的准确率进一步提升到76.9%(+0.6%)。TextRotNet训练流程如下图所示: - -
- -
- - -**(5)UDML:联合互学习策略** - -UDML(Unified-Deep Mutual Learning)联合互学习是PP-OCRv2中就采用的对于文本识别非常有效的提升模型效果的策略。在PP-OCRv3中,针对两个不同的SVTR_LCNet和Attention结构,对他们之间的PP-LCNet的特征图、SVTR模块的输出和Attention模块的输出同时进行监督训练。使用该策略,识别模型的准确率进一步提升到78.4%(+1.5%)。 - - -**(6)UIM:无标注数据挖掘方案** - -UIM(Unlabeled Images Mining)是一种非常简单的无标注数据挖掘方案。核心思想是利用高精度的文本识别大模型对无标注数据进行预测,获取伪标签,并且选择预测置信度高的样本作为训练数据,用于训练小模型。使用该策略,识别模型的准确率进一步提升到79.4%(+1%)。实际操作中,我们使用全量数据集训练高精度SVTR-Tiny模型(acc=82.5%)进行数据挖掘,点击获取[模型下载地址和使用教程](../../applications/高精度中文识别模型.md)。 - -
- -
- - - -## 4. 端到端评估 - -经过以上优化,最终PP-OCRv3在速度可比情况下,中文场景端到端Hmean指标相比于PP-OCRv2提升5%,效果大幅提升。具体指标如下表所示: - -| Model | Hmean | Model Size (M) | Time Cost (CPU, ms) | Time Cost (T4 GPU, ms) | -|-----|-----|--------|----| --- | -| PP-OCR mobile | 50.30% | 8.1 | 356.00 | 116.00 | -| PP-OCR server | 57.00% | 155.1 | 1056.00 | 200.00 | -| PP-OCRv2 | 57.60% | 11.6 | 330.00 | 111.00 | -| PP-OCRv3 | 62.90% | 15.6 | 331.00 | 86.64 | - -测试环境:CPU型号为Intel Gold 6148,CPU预测时开启MKLDNN加速。 - - -除了更新中文模型,本次升级也同步优化了英文数字模型,端到端效果提升11%,如下表所示: - -| Model | Recall | Precision | Hmean | -|-----|-----|--------|----| -| PP-OCR_en | 38.99% | 45.91% | 42.17% | -| PP-OCRv3_en | 50.95% | 55.53% | 53.14% | - -同时,也对已支持的80余种语言识别模型进行了升级更新,在有评估集的四种语系识别准确率平均提升5%以上,如下表所示: - -| Model | 拉丁语系 | 阿拉伯语系 | 日语 | 韩语 | -|-----|-----|--------|----| --- | -| PP-OCR_mul | 69.60% | 40.50% | 38.50% | 55.40% | -| PP-OCRv3_mul | 75.20%| 45.37% | 45.80% | 60.10% | diff --git a/doc/doc_ch/PP-OCRv4_introduction.md b/doc/doc_ch/PP-OCRv4_introduction.md deleted file mode 100644 index a6694af6d9..0000000000 --- a/doc/doc_ch/PP-OCRv4_introduction.md +++ /dev/null @@ -1,178 +0,0 @@ -# PP-OCRv4 - -- [1. 简介](#1) -- [2. 检测优化](#2) -- [3. 识别优化](#3) -- [4. 端到端评估](#4) - - - -## 1. 简介 - -PP-OCRv4在PP-OCRv3的基础上进一步升级。整体的框架图保持了与PP-OCRv3相同的pipeline,针对检测模型和识别模型进行了数据、网络结构、训练策略等多个模块的优化。 PP-OCRv4系统框图如下所示: - -
- -
- - -从算法改进思路上看,分别针对检测和识别模型,进行了共10个方面的改进: -* 检测模块: - * LCNetV3:精度更高的骨干网络 - * PFHead:并行head分支融合结构 - * DSR: 训练中动态增加shrink ratio - * CML:添加Student和Teacher网络输出的KL div loss -* 识别模块: - * SVTR_LCNetV3:精度更高的骨干网络 - * Lite-Neck:精简的Neck结构 - * GTC-NRTR:稳定的Attention指导分支 - * Multi-Scale:多尺度训练策略 - * DF: 数据挖掘方案 - * DKD :DKD蒸馏策略 - -从效果上看,速度可比情况下,多种场景精度均有大幅提升: -* 中文场景,相对于PP-OCRv3中文模型提升超4%; -* 英文数字场景,相比于PP-OCRv3英文模型提升6%; -* 多语言场景,优化80个语种识别效果,平均准确率提升超8%。 - - - -## 2. 检测优化 - -PP-OCRv4检测模型在PP-OCRv3检测模型的基础上,在网络结构,训练策略,蒸馏策略三个方面做了优化。首先,PP-OCRv4检测模型使用PP-LCNetV3替换MobileNetv3,并提出并行分支融合的PFhead结构;其次,训练时动态调整shrink ratio的比例;最后,PP-OCRv4对CML的蒸馏loss进行优化,进一步提升文字检测效果。 - -消融实验如下: - -|序号|策略|模型大小|hmean|速度(cpu + mkldnn)| -|-|-|-|-|-| -|baseline|PP-OCRv3|3.4M|78.84%|69ms| -|baseline student|PP-OCRv3 student|3.4M|76.22%|69ms| -|01|+PFHead|3.6M|76.97%|96ms| -|02|+Dynamic Shrink Ratio|3.6M|78.24%|96ms| -|03|+PP-LCNetv3|4.8M|79.08%|94ms| -|03|+CML|4.8M|79.87%|67ms| - -测试环境: Intel Gold 6148 CPU,预测引擎使用openvino。 - -**(1)PFhead:多分支融合Head结构** - -PFhead结构如下图所示,PFHead在经过第一个转置卷积后,分别进行上采样和转置卷积,上采样的输出通过3x3卷积得到输出结果,然后和转置卷积的分支的结果级联并经过1x1卷积层,最后1x1卷积的结果和转置卷积的结果相加得到最后输出的概率图。PP-OCRv4学生检测模型使用PFhead,hmean从76.22%增加到76.97%。 - -
- -
- -**(2)DSR: 收缩比例动态调整策略** - -动态shrink ratio(dynamic shrink ratio): 在训练中,shrink ratio由固定值调整为动态变化,随着训练epoch的增加,shrink ratio从0.4线性增加到0.6。该策略在PP-OCRv4学生检测模型上,hmean从76.97%提升到78.24%。 - -**(3) PP-LCNetV3:精度更高的骨干网络** - -PP-LCNetV3系列模型是PP-LCNet系列模型的延续,覆盖了更大的精度范围,能够适应不同下游任务的需要。PP-LCNetV3系列模型从多个方面进行了优化,提出了可学习仿射变换模块,对重参数化策略、激活函数进行了改进,同时调整了网络深度与宽度。最终,PP-LCNetV3系列模型能够在性能与效率之间达到最佳的平衡,在不同精度范围内取得极致的推理速度。使用PP-LCNetV3替换MobileNetv3 backbone,PP-OCRv4学生检测模型hmean从78.24%提升到79.08%。 - -**(4)CML: 融合KD的互学习策略** - -PP-OCRv4检测模型对PP-OCRv3中的CML(Collaborative Mutual Learning) 协同互学习文本检测蒸馏策略进行了优化。如下图所示,在计算Student Model和Teacher Model的distill Loss时,额外添加KL div loss,让两者输出的response maps分布接近,由此进一步提升Student网络的精度,检测Hmean从79.08%增加到79.56%,端到端指标从61.31%增加到61.87%。 - -
- -
- - -## 3. 识别优化 - -PP-OCRv4识别模型在PP-OCRv3的基础上进一步升级。如下图所示,整体的框架图保持了与PP-OCRv3识别模型相同的pipeline,分别进行了数据、网络结构、训练策略等方面的优化。 - -
- -
- -经过如图所示的策略优化,PP-OCRv4识别模型相比PP-OCRv3,在速度可比的情况下,精度进一步提升4%。 具体消融实验如下所示: - -| ID | 策略 | 模型大小 | 精度 | 预测耗时(CPU openvino)| -|-----|-----|--------|----| --- | -| 01 | PP-OCRv3 | 12M | 71.50% | 8.54ms | -| 02 | +DF | 12M | 72.70% | 8.54ms | -| 03 | + LiteNeck + GTC | 9.6M | 73.21% | 9.09ms | -| 04 | + PP-LCNetV3 | 11M | 74.18% | 9.8ms | -| 05 | + multi-scale | 11M | 74.20% | 9.8ms | -| 06 | + TextConAug | 11M | 74.72% | 9.8ms | -| 08 | + UDML | 11M | 75.45% | 9.8ms | - -注: 测试速度时,输入图片尺寸均为(3,48,320)。在实际预测时,图像为变长输入,速度会有所变化。测试环境: Intel Gold 6148 CPU,预测时使用Openvino预测引擎。 - -**(1)DF:数据挖掘方案** - -DF(Data Filter) 是一种简单有效的数据挖掘方案。核心思想是利用已有模型预测训练数据,通过置信度和预测结果等信息,对全量的训练数据进行筛选。具体的:首先使用少量数据快速训练得到一个低精度模型,使用该低精度模型对千万级的数据进行预测,去除置信度大于0.95的样本,该部分被认为是对提升模型精度无效的冗余样本。其次使用PP-OCRv3作为高精度模型,对剩余数据进行预测,去除置信度小于0.15的样本,该部分被认为是难以识别或质量很差的样本。 -使用该策略,千万级别训练数据被精简至百万级,模型训练时间从2周减少到5天,显著提升了训练效率,同时精度提升至72.7%(+1.2%)。 - - -
- -
- - -**(2)PP-LCNetV3:精度更优的骨干网络** - -PP-LCNetV3系列模型是PP-LCNet系列模型的延续,覆盖了更大的精度范围,能够适应不同下游任务的需要。PP-LCNetV3系列模型从多个方面进行了优化,提出了可学习仿射变换模块,对重参数化策略、激活函数进行了改进,同时调整了网络深度与宽度。最终,PP-LCNetV3系列模型能够在性能与效率之间达到最佳的平衡,在不同精度范围内取得极致的推理速度。 - -**(3)Lite-Neck:精简参数的Neck结构** - -Lite-Neck整体结构沿用PP-OCRv3版本的结构,在参数上稍作精简,识别模型整体的模型大小可从12M降低到8.5M,而精度不变;在CTCHead中,将Neck输出特征的维度从64提升到120,此时模型大小从8.5M提升到9.6M。 - - -**(4)GTC-NRTR:Attention指导CTC训练策略** - -GTC(Guided Training of CTC),是PP-OCRv3识别模型的最有效的策略之一,融合多种文本特征的表达,有效的提升文本识别精度。在PP-OCRv4中使用训练更稳定的Transformer模型NRTR作为指导分支,相比V3版本中的SAR基于循环神经网络的结构,NRTR基于Transformer实现解码过程泛化能力更强,能有效指导CTC分支学习,解决简单场景下快速过拟合的问题。使用Lite-Neck和GTC-NRTR两个策略,识别精度提升至73.21%(+0.5%)。 - -
- -
- - -**(5)Multi-Scale:多尺度训练策略** - -动态尺度训练策略,是在训练过程中随机resize输入图片的高度,以增强识别模型在端到端串联使用时的鲁棒性。在训练时,每个iter从(32,48,64)三种高度中随机选择一种高度进行resize。实验证明,使用该策略,尽管在识别测试集上准确率没有提升,但在端到端串联评估时,指标提升0.5%。 - -
- -
- - -**(6)DKD:蒸馏策略** - -识别模型的蒸馏包含两个部分,NRTRhead蒸馏和CTCHead蒸馏; - -对于NRTR head,使用了DKD loss蒸馏,拉近学生模型和教师模型的NRTR head logits。最终NRTR head的loss是学生与教师间的DKD loss和与ground truth的cross entropy loss的加权和,用于监督学生模型的backbone训练。通过实验,我们发现加入DKD loss后,计算与ground truth的cross entropy loss时去除label smoothing可以进一步提高精度,因此我们在这里使用的是不带label smoothing的cross entropy loss。 - -对于CTCHead,由于CTC的输出中存在Blank位,即使教师模型和学生模型的预测结果一样,二者的输出的logits分布也会存在差异,影响教师模型向学生模型的知识传递。PP-OCRv4识别模型蒸馏策略中,将CTC输出logits沿着文本长度维度计算均值,将多字符识别问题转换为多字符分类问题,用于监督CTC Head的训练。使用该策略融合NRTRhead DKD蒸馏策略,指标从74.72%提升到75.45%。 - - - - -## 4. 端到端评估 - -经过以上优化,最终PP-OCRv4在速度可比情况下,中文场景端到端Hmean指标相比于PP-OCRv3提升4.5%,效果大幅提升。具体指标如下表所示: - -| Model | Hmean | Model Size (M) | Time Cost (CPU, ms) | -|-----|-----|--------|----| -| PP-OCRv3 | 57.99% | 15.6 | 78 | -| PP-OCRv4 | 62.24% | 15.8 | 76 | - -测试环境:CPU型号为Intel Gold 6148,CPU预测时使用openvino。 - -除了更新中文模型,本次升级也优化了英文数字模型,在自有评估集上文本识别准确率提升6%,如下表所示: - -| Model | ACC | -|-----|-----| -| PP-OCR_en | 54.38% | -| PP-OCRv3_en | 64.04% | -| PP-OCRv4_en | 70.1% | - -同时,对已支持的80余种语言识别模型进行了升级更新,在有评估集的四种语系识别准确率平均提升8%以上,如下表所示: - -| Model | 拉丁语系 | 阿拉伯语系 | 日语 | 韩语 | -|-----|-----|--------|----| --- | -| PP-OCR_mul | 69.60% | 40.50% | 38.50% | 55.40% | -| PP-OCRv3_mul | 71.57%| 72.90% | 45.85% | 77.23% | -| PP-OCRv4_mul | 80.00%| 75.48% | 56.50% | 83.25% | diff --git a/doc/doc_ch/PPOCRv3_det_train.md b/doc/doc_ch/PPOCRv3_det_train.md deleted file mode 100644 index 616d77f697..0000000000 --- a/doc/doc_ch/PPOCRv3_det_train.md +++ /dev/null @@ -1,250 +0,0 @@ - -# PP-OCRv3 文本检测模型训练 - -- [1. 简介](#1) -- [2. PP-OCRv3检测训练](#2) -- [3. 基于PP-OCRv3检测的finetune训练](#3) - - -## 1. 简介 - -PP-OCRv3在PP-OCRv2的基础上进一步升级。本节介绍PP-OCRv3检测模型的训练步骤。有关PP-OCRv3策略介绍参考[文档](./PP-OCRv3_introduction.md)。 - - - -## 2. 检测训练 - -PP-OCRv3检测模型是对PP-OCRv2中的[CML](https://arxiv.org/pdf/2109.03144.pdf)(Collaborative Mutual Learning) 协同互学习文本检测蒸馏策略进行了升级。PP-OCRv3分别针对检测教师模型和学生模型进行进一步效果优化。其中,在对教师模型优化时,提出了大感受野的PAN结构LK-PAN和引入了DML(Deep Mutual Learning)蒸馏策略;在对学生模型优化时,提出了残差注意力机制的FPN结构RSE-FPN。 - -PP-OCRv3检测训练包括两个步骤: -- 步骤1:采用DML蒸馏方法训练检测教师模型 -- 步骤2:使用步骤1得到的教师模型采用CML方法训练出轻量学生模型 - - -### 2.1 准备数据和运行环境 - -训练数据采用icdar2015数据,准备训练集步骤参考[ocr_dataset](./dataset/ocr_datasets.md). - -运行环境准备参考[文档](./installation.md)。 - - -### 2.2 训练教师模型 - -教师模型训练的配置文件是[ch_PP-OCRv3_det_dml.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.5/configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_dml.yml)。教师模型模型结构的Backbone、Neck、Head分别为Resnet50, LKPAN, DBHead,采用DML的蒸馏方法训练。有关配置文件的详细介绍参考[文档](./knowledge_distillation.md)。 - - -下载ImageNet预训练模型: -``` -# 下载ResNet50_vd的预训练模型 -wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet50_vd_ssld_pretrained.pdparams -``` - -**启动训练** -``` -# 单卡训练 -python3 tools/train.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_dml.yml \ - -o Architecture.Models.Student.pretrained=./pretrain_models/ResNet50_vd_ssld_pretrained \ - Architecture.Models.Student2.pretrained=./pretrain_models/ResNet50_vd_ssld_pretrained \ - Global.save_model_dir=./output/ -# 如果要使用多GPU分布式训练,请使用如下命令: -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_dml.yml \ - -o Architecture.Models.Student.pretrained=./pretrain_models/ResNet50_vd_ssld_pretrained \ - Architecture.Models.Student2.pretrained=./pretrain_models/ResNet50_vd_ssld_pretrained \ - Global.save_model_dir=./output/ -``` - -训练过程中保存的模型在output目录下,包含以下文件: -``` -best_accuracy.states -best_accuracy.pdparams # 默认保存最优精度的模型参数 -best_accuracy.pdopt # 默认保存最优精度的优化器相关参数 -latest.states -latest.pdparams # 默认保存的最新模型参数 -latest.pdopt # 默认保存的最新模型的优化器相关参数 -``` -其中,best_accuracy是保存的精度最高的模型参数,可以直接使用该模型评估。 - -模型评估命令如下: -``` -python3 tools/eval.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_dml.yml -o Global.checkpoints=./output/best_accuracy -``` - -训练的教师模型结构更大,精度更高,用于提升学生模型的精度。 - -**提取教师模型参数** -best_accuracy包含两个模型的参数,分别对应配置文件中的Student,Student2。提取Student的参数方法如下: - -``` -import paddle -# 加载预训练模型 -all_params = paddle.load("output/best_accuracy.pdparams") -# 查看权重参数的keys -print(all_params.keys()) -# 模型的权重提取 -s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key} -# 查看模型权重参数的keys -print(s_params.keys()) -# 保存 -paddle.save(s_params, "./pretrain_models/dml_teacher.pdparams") -``` - -提取出来的模型参数可以用于模型进一步的finetune训练或者蒸馏训练。 - -### 2.3 训练学生模型 - -训练学生模型的配置文件是[ch_PP-OCRv3_det_cml.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.5/configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml) -上一节训练得到的教师模型作为监督,采用CML方式训练得到轻量的学生模型。 - -下载学生模型的ImageNet预训练模型: -``` -# 下载MobileNetV3的预训练模型 -wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/MobileNetV3_large_x0_5_pretrained.pdparams -``` - -**启动训练** - -``` -# 单卡训练 -python3 tools/train.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml \ - -o Architecture.Models.Student.pretrained=./pretrain_models/MobileNetV3_large_x0_5_pretrained \ - Architecture.Models.Student2.pretrained=./pretrain_models/MobileNetV3_large_x0_5_pretrained \ - Architecture.Models.Teacher.pretrained=./pretrain_models/dml_teacher \ - Global.save_model_dir=./output/ -# 如果要使用多GPU分布式训练,请使用如下命令: -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml \ - -o Architecture.Models.Student.pretrained=./pretrain_models/MobileNetV3_large_x0_5_pretrained \ - Architecture.Models.Student2.pretrained=./pretrain_models/MobileNetV3_large_x0_5_pretrained \ - Architecture.Models.Teacher.pretrained=./pretrain_models/dml_teacher \ - Global.save_model_dir=./output/ -``` - -训练过程中保存的模型在output目录下, -模型评估命令如下: -``` -python3 tools/eval.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml -o Global.checkpoints=./output/best_accuracy -``` - -best_accuracy包含三个模型的参数,分别对应配置文件中的Student,Student2,Teacher。提取Student参数的方法如下: - -``` -import paddle -# 加载预训练模型 -all_params = paddle.load("output/best_accuracy.pdparams") -# 查看权重参数的keys -print(all_params.keys()) -# 模型的权重提取 -s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key} -# 查看模型权重参数的keys -print(s_params.keys()) -# 保存 -paddle.save(s_params, "./pretrain_models/cml_student.pdparams") -``` - -提取出来的Student的参数可用于模型部署或者做进一步的finetune训练。 - - - - -## 3. 基于PP-OCRv3检测finetune训练 - -本节介绍如何使用PP-OCRv3检测模型在其他场景上的finetune训练。 - -finetune训练适用于三种场景: -- 基于CML蒸馏方法的finetune训练,适用于教师模型在使用场景上精度高于PP-OCRv3检测模型,且希望得到一个轻量检测模型。 -- 基于PP-OCRv3轻量检测模型的finetune训练,无需训练教师模型,希望在PP-OCRv3检测模型基础上提升使用场景上的精度。 -- 基于DML蒸馏方法的finetune训练,适用于采用DML方法进一步提升精度的场景。 - - -**基于CML蒸馏方法的finetune训练** - -下载PP-OCRv3训练模型: -``` -wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar -tar xf ch_PP-OCRv3_det_distill_train.tar -``` -ch_PP-OCRv3_det_distill_train/best_accuracy.pdparams包含CML配置文件中Student、Student2、Teacher模型的参数。 - -启动训练: - -``` -# 单卡训练 -python3 tools/train.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml \ - -o Global.pretrained_model=./ch_PP-OCRv3_det_distill_train/best_accuracy \ - Global.save_model_dir=./output/ -# 如果要使用多GPU分布式训练,请使用如下命令: -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml \ - -o Global.pretrained_model=./ch_PP-OCRv3_det_distill_train/best_accuracy \ - Global.save_model_dir=./output/ -``` - -**基于PP-OCRv3轻量检测模型的finetune训练** - - -下载PP-OCRv3训练模型,并提取Student结构的模型参数: -``` -wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar -tar xf ch_PP-OCRv3_det_distill_train.tar -``` - -提取Student参数的方法如下: - -``` -import paddle -# 加载预训练模型 -all_params = paddle.load("output/best_accuracy.pdparams") -# 查看权重参数的keys -print(all_params.keys()) -# 模型的权重提取 -s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key} -# 查看模型权重参数的keys -print(s_params.keys()) -# 保存 -paddle.save(s_params, "./student.pdparams") -``` - -使用配置文件[ch_PP-OCRv3_det_student.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.5/configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_student.yml)训练。 - -**启动训练** - -``` -# 单卡训练 -python3 tools/train.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_student.yml \ - -o Global.pretrained_model=./student \ - Global.save_model_dir=./output/ -# 如果要使用多GPU分布式训练,请使用如下命令: -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_student.yml \ - -o Global.pretrained_model=./student \ - Global.save_model_dir=./output/ -``` - - -**基于DML蒸馏方法的finetune训练** - -以ch_PP-OCRv3_det_distill_train中的Teacher模型为例,首先提取Teacher结构的参数,方法如下: -``` -import paddle -# 加载预训练模型 -all_params = paddle.load("ch_PP-OCRv3_det_distill_train/best_accuracy.pdparams") -# 查看权重参数的keys -print(all_params.keys()) -# 模型的权重提取 -s_params = {key[len("Teacher."):]: all_params[key] for key in all_params if "Teacher." in key} -# 查看模型权重参数的keys -print(s_params.keys()) -# 保存 -paddle.save(s_params, "./teacher.pdparams") -``` - -**启动训练** -``` -# 单卡训练 -python3 tools/train.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_dml.yml \ - -o Architecture.Models.Student.pretrained=./teacher \ - Architecture.Models.Student2.pretrained=./teacher \ - Global.save_model_dir=./output/ -# 如果要使用多GPU分布式训练,请使用如下命令: -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_dml.yml \ - -o Architecture.Models.Student.pretrained=./teacher \ - Architecture.Models.Student2.pretrained=./teacher \ - Global.save_model_dir=./output/ -``` diff --git a/doc/doc_ch/add_new_algorithm.md b/doc/doc_ch/add_new_algorithm.md deleted file mode 100644 index bb97e00aa6..0000000000 --- a/doc/doc_ch/add_new_algorithm.md +++ /dev/null @@ -1,315 +0,0 @@ -# 添加新算法 - -PaddleOCR将一个算法分解为以下几个部分,并对各部分进行模块化处理,方便快速组合出新的算法。 - -* [1. 数据加载和处理](#1) -* [2. 网络](#2) -* [3. 后处理](#3) -* [4. 损失函数](#4) -* [5. 指标评估](#5) -* [6. 优化器](#6) - -下面将分别对每个部分进行介绍,并介绍如何在该部分里添加新算法所需模块。 - - - -## 1. 数据加载和处理 - -数据加载和处理由不同的模块(module)组成,其完成了图片的读取、数据增强和label的制作。这一部分在[ppocr/data](../../ppocr/data)下。 各个文件及文件夹作用说明如下: - -```bash -ppocr/data/ -├── imaug # 图片的读取、数据增强和label制作相关的文件 -│ ├── label_ops.py # 对label进行变换的modules -│ ├── operators.py # 对image进行变换的modules -│ ├──..... -├── __init__.py -├── lmdb_dataset.py # 读取lmdb的数据集的dataset -└── simple_dataset.py # 读取以`image_path\tgt`形式保存的数据集的dataset -``` - -PaddleOCR内置了大量图像操作相关模块,对于没有没有内置的模块可通过如下步骤添加: - -1. 在 [ppocr/data/imaug](../../ppocr/data/imaug) 文件夹下新建文件,如my_module.py。 -2. 在 my_module.py 文件内添加相关代码,示例代码如下: - -```python -class MyModule: - def __init__(self, *args, **kwargs): - # your init code - pass - - def __call__(self, data): - img = data['image'] - label = data['label'] - # your process code - - data['image'] = img - data['label'] = label - return data -``` - -3. 在 [ppocr/data/imaug/\__init\__.py](../../ppocr/data/imaug/__init__.py) 文件内导入添加的模块。 - -数据处理的所有处理步骤由不同的模块顺序执行而成,在config文件中按照列表的形式组合并执行。如: - -```yaml -# angle class data process -transforms: - - DecodeImage: # load image - img_mode: BGR - channel_first: False - - MyModule: - args1: args1 - args2: args2 - - KeepKeys: - keep_keys: [ 'image', 'label' ] # dataloader will return list in this order -``` - - - -## 2. 网络 - -网络部分完成了网络的组网操作,PaddleOCR将网络划分为四部分,这一部分在[ppocr/modeling](../../ppocr/modeling)下。 进入网络的数据将按照顺序(transforms->backbones-> -necks->heads)依次通过这四个部分。 - -```bash -├── architectures # 网络的组网代码 -├── transforms # 网络的图像变换模块 -├── backbones # 网络的特征提取模块 -├── necks # 网络的特征增强模块 -└── heads # 网络的输出模块 -``` - -PaddleOCR内置了DB,EAST,SAST,CRNN和Attention等算法相关的常用模块,对于没有内置的模块可通过如下步骤添加,四个部分添加步骤一致,以backbones为例: - -1. 在 [ppocr/modeling/backbones](../../ppocr/modeling/backbones) 文件夹下新建文件,如my_backbone.py。 -2. 在 my_backbone.py 文件内添加相关代码,示例代码如下: - -```python -import paddle -import paddle.nn as nn -import paddle.nn.functional as F - - -class MyBackbone(nn.Layer): - def __init__(self, *args, **kwargs): - super(MyBackbone, self).__init__() - # your init code - self.conv = nn.xxxx - - def forward(self, inputs): - # your network forward - y = self.conv(inputs) - return y -``` - -3. 在 [ppocr/modeling/backbones/\__init\__.py](../../ppocr/modeling/backbones/__init__.py)文件内导入添加的模块。 - -在完成网络的四部分模块添加之后,只需要配置文件中进行配置即可使用,如: - -```yaml -Architecture: - model_type: rec - algorithm: CRNN - Transform: - name: MyTransform - args1: args1 - args2: args2 - Backbone: - name: MyBackbone - args1: args1 - Neck: - name: MyNeck - args1: args1 - Head: - name: MyHead - args1: args1 -``` - - - -## 3. 后处理 - -后处理实现解码网络输出获得文本框或者识别到的文字。这一部分在[ppocr/postprocess](../../ppocr/postprocess)下。 -PaddleOCR内置了DB,EAST,SAST,CRNN和Attention等算法相关的后处理模块,对于没有内置的组件可通过如下步骤添加: - -1. 在 [ppocr/postprocess](../../ppocr/postprocess) 文件夹下新建文件,如 my_postprocess.py。 -2. 在 my_postprocess.py 文件内添加相关代码,示例代码如下: - -```python -import paddle - - -class MyPostProcess: - def __init__(self, *args, **kwargs): - # your init code - pass - - def __call__(self, preds, label=None, *args, **kwargs): - if isinstance(preds, paddle.Tensor): - preds = preds.numpy() - # you preds decode code - preds = self.decode_preds(preds) - if label is None: - return preds - # you label decode code - label = self.decode_label(label) - return preds, label - - def decode_preds(self, preds): - # you preds decode code - pass - - def decode_label(self, preds): - # you label decode code - pass -``` - -3. 在 [ppocr/postprocess/\__init\__.py](../../ppocr/postprocess/__init__.py)文件内导入添加的模块。 - -在后处理模块添加之后,只需要配置文件中进行配置即可使用,如: - -```yaml -PostProcess: - name: MyPostProcess - args1: args1 - args2: args2 -``` - - - -## 4. 损失函数 - -损失函数用于计算网络输出和label之间的距离。这一部分在[ppocr/losses](../../ppocr/losses)下。 -PaddleOCR内置了DB,EAST,SAST,CRNN和Attention等算法相关的损失函数模块,对于没有内置的模块可通过如下步骤添加: - -1. 在 [ppocr/losses](../../ppocr/losses) 文件夹下新建文件,如 my_loss.py。 -2. 在 my_loss.py 文件内添加相关代码,示例代码如下: - -```python -import paddle -from paddle import nn - - -class MyLoss(nn.Layer): - def __init__(self, **kwargs): - super(MyLoss, self).__init__() - # you init code - pass - - def __call__(self, predicts, batch): - label = batch[1] - # your loss code - loss = self.loss(input=predicts, label=label) - return {'loss': loss} -``` - -3. 在 [ppocr/losses/\__init\__.py](../../ppocr/losses/__init__.py)文件内导入添加的模块。 - -在损失函数添加之后,只需要配置文件中进行配置即可使用,如: - -```yaml -Loss: - name: MyLoss - args1: args1 - args2: args2 -``` - - - -## 5. 指标评估 - -指标评估用于计算网络在当前batch上的性能。这一部分在[ppocr/metrics](../../ppocr/metrics)下。 PaddleOCR内置了检测,分类和识别等算法相关的指标评估模块,对于没有内置的模块可通过如下步骤添加: - -1. 在 [ppocr/metrics](../../ppocr/metrics) 文件夹下新建文件,如my_metric.py。 -2. 在 my_metric.py 文件内添加相关代码,示例代码如下: - -```python - -class MyMetric(object): - def __init__(self, main_indicator='acc', **kwargs): - # main_indicator is used for select best model - self.main_indicator = main_indicator - self.reset() - - def __call__(self, preds, batch, *args, **kwargs): - # preds is out of postprocess - # batch is out of dataloader - labels = batch[1] - cur_correct_num = 0 - cur_all_num = 0 - # you metric code - self.correct_num += cur_correct_num - self.all_num += cur_all_num - return {'acc': cur_correct_num / cur_all_num, } - - def get_metric(self): - """ - return metrics { - 'acc': 0, - 'norm_edit_dis': 0, - } - """ - acc = self.correct_num / self.all_num - self.reset() - return {'acc': acc} - - def reset(self): - # reset metric - self.correct_num = 0 - self.all_num = 0 - -``` - -3. 在 [ppocr/metrics/\__init\__.py](../../ppocr/metrics/__init__.py)文件内导入添加的模块。 - -在指标评估模块添加之后,只需要配置文件中进行配置即可使用,如: - -```yaml -Metric: - name: MyMetric - main_indicator: acc -``` - - - -## 6. 优化器 - -优化器用于训练网络。优化器内部还包含了网络正则化和学习率衰减模块。 这一部分在[ppocr/optimizer](../../ppocr/optimizer)下。 PaddleOCR内置了`Momentum`,`Adam` -和`RMSProp`等常用的优化器模块,`Linear`,`Cosine`,`Step`和`Piecewise`等常用的正则化模块与`L1Decay`和`L2Decay`等常用的学习率衰减模块。 -对于没有内置的模块可通过如下步骤添加,以`optimizer`为例: - -1. 在 [ppocr/optimizer/optimizer.py](../../ppocr/optimizer/optimizer.py) 文件内创建自己的优化器,示例代码如下: - -```python -from paddle import optimizer as optim - - -class MyOptim(object): - def __init__(self, learning_rate=0.001, *args, **kwargs): - self.learning_rate = learning_rate - - def __call__(self, parameters): - # It is recommended to wrap the built-in optimizer of paddle - opt = optim.XXX( - learning_rate=self.learning_rate, - parameters=parameters) - return opt - -``` - -在优化器模块添加之后,只需要配置文件中进行配置即可使用,如: - -```yaml -Optimizer: - name: MyOptim - args1: args1 - args2: args2 - lr: - name: Cosine - learning_rate: 0.001 - regularizer: - name: 'L2' - factor: 0 -``` diff --git a/doc/doc_ch/algorithm_det_ct.md b/doc/doc_ch/algorithm_det_ct.md deleted file mode 100644 index ea3522b7bf..0000000000 --- a/doc/doc_ch/algorithm_det_ct.md +++ /dev/null @@ -1,95 +0,0 @@ -# CT - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [CentripetalText: An Efficient Text Instance Representation for Scene Text Detection](https://arxiv.org/abs/2107.05945) -> Tao Sheng, Jie Chen, Zhouhui Lian -> NeurIPS, 2021 - - -在Total-Text文本检测公开数据集上,算法复现效果如下: - -|模型|骨干网络|配置文件|precision|recall|Hmean|下载链接| -| --- | --- | --- | --- | --- | --- | --- | -|CT|ResNet18_vd|[configs/det/det_r18_vd_ct.yml](../../configs/det/det_r18_vd_ct.yml)|88.68%|81.70%|85.05%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r18_ct_train.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -CT模型使用Total-Text文本检测公开数据集训练得到,数据集下载可参考 [Total-Text-Dataset](https://github.com/cs-chan/Total-Text-Dataset/tree/master/Dataset), 我们将标签文件转成了paddleocr格式,转换好的标签文件下载参考[train.txt](https://paddleocr.bj.bcebos.com/dataset/ct_tipc/train.txt), [text.txt](https://paddleocr.bj.bcebos.com/dataset/ct_tipc/test.txt)。 - -请参考[文本检测训练教程](./detection.md)。PaddleOCR对代码进行了模块化,训练不同的检测模型只需要**更换配置文件**即可。 - - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将CT文本检测训练过程中保存的模型,转换成inference model。以基于Resnet18_vd骨干网络,在Total-Text英文数据集训练的模型为例( [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r18_ct_train.tar) ),可以使用如下命令进行转换: - -```shell -python3 tools/export_model.py -c configs/det/det_r18_vd_ct.yml -o Global.pretrained_model=./det_r18_ct_train/best_accuracy Global.save_inference_dir=./inference/det_ct -``` - -CT文本检测模型推理,可以执行如下命令: - -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img623.jpg" --det_model_dir="./inference/det_ct/" --det_algorithm="CT" -``` - -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_img623_ct.jpg) - - - -### 4.2 C++推理 - -暂不支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@inproceedings{sheng2021centripetaltext, - title={CentripetalText: An Efficient Text Instance Representation for Scene Text Detection}, - author={Tao Sheng and Jie Chen and Zhouhui Lian}, - booktitle={Thirty-Fifth Conference on Neural Information Processing Systems}, - year={2021} -} -``` diff --git a/doc/doc_ch/algorithm_det_db.md b/doc/doc_ch/algorithm_det_db.md deleted file mode 100644 index 5401132061..0000000000 --- a/doc/doc_ch/algorithm_det_db.md +++ /dev/null @@ -1,119 +0,0 @@ -# DB与DB++ - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [Real-time Scene Text Detection with Differentiable Binarization](https://arxiv.org/abs/1911.08947) -> Liao, Minghui and Wan, Zhaoyi and Yao, Cong and Chen, Kai and Bai, Xiang -> AAAI, 2020 - -> [Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion](https://arxiv.org/abs/2202.10304) -> Liao, Minghui and Zou, Zhisheng and Wan, Zhaoyi and Yao, Cong and Bai, Xiang -> TPAMI, 2022 - - -在ICDAR2015文本检测公开数据集上,算法复现效果如下: - -|模型|骨干网络|配置文件|precision|recall|Hmean|下载链接| -| --- | --- | --- | --- | --- | --- | --- | -|DB|ResNet50_vd|[configs/det/det_r50_vd_db.yml](../../configs/det/det_r50_vd_db.yml)|86.41%|78.72%|82.38%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_db_v2.0_train.tar)| -|DB|MobileNetV3|[configs/det/det_mv3_db.yml](../../configs/det/det_mv3_db.yml)|77.29%|73.08%|75.12%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_db_v2.0_train.tar)| -|DB++|ResNet50|[configs/det/det_r50_db++_icdar15.yml](../../configs/det/det_r50_db++_icdar15.yml)|90.89%|82.66%|86.58%|[合成数据预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/ResNet50_dcn_asf_synthtext_pretrained.pdparams)/[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_r50_db%2B%2B_icdar15_train.tar)| - -在TD_TR文本检测公开数据集上,算法复现效果如下: - -|模型|骨干网络|配置文件|precision|recall|Hmean|下载链接| -| --- | --- | --- | --- | --- | --- | --- | -|DB++|ResNet50|[configs/det/det_r50_db++_td_tr.yml](../../configs/det/det_r50_db++_td_tr.yml)|92.92%|86.48%|89.58%|[合成数据预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/ResNet50_dcn_asf_synthtext_pretrained.pdparams)/[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_r50_db%2B%2B_td_tr_train.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本检测训练教程](./detection.md)。PaddleOCR对代码进行了模块化,训练不同的检测模型只需要**更换配置文件**即可。 - - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将DB文本检测训练过程中保存的模型,转换成inference model。以基于Resnet50_vd骨干网络,在ICDAR2015英文数据集训练的模型为例( [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_db_v2.0_train.tar) ),可以使用如下命令进行转换: - -```shell -python3 tools/export_model.py -c configs/det/det_r50_vd_db.yml -o Global.pretrained_model=./det_r50_vd_db_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_db -``` - -DB文本检测模型推理,可以执行如下命令: - -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_db/" --det_algorithm="DB" -``` - -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_img_10_db.jpg) - -**注意**:由于ICDAR2015数据集只有1000张训练图像,且主要针对英文场景,所以上述模型对中文文本图像检测效果会比较差。 - - -### 4.2 C++推理 - -准备好推理模型后,参考[cpp infer](../../deploy/cpp_infer/)教程进行操作即可。 - - -### 4.3 Serving服务化部署 - -准备好推理模型后,参考[pdserving](../../deploy/pdserving/)教程进行Serving服务化部署,包括Python Serving和C++ Serving两种模式。 - - -### 4.4 更多推理部署 - -DB模型还支持以下推理部署方式: - -- Paddle2ONNX推理:准备好推理模型后,参考[paddle2onnx](../../deploy/paddle2onnx/)教程操作。 - - -## 5. FAQ - - -## 引用 - -```bibtex -@inproceedings{liao2020real, - title={Real-time scene text detection with differentiable binarization}, - author={Liao, Minghui and Wan, Zhaoyi and Yao, Cong and Chen, Kai and Bai, Xiang}, - booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, - volume={34}, - number={07}, - pages={11474--11481}, - year={2020} -} - -@article{liao2022real, - title={Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion}, - author={Liao, Minghui and Zou, Zhisheng and Wan, Zhaoyi and Yao, Cong and Bai, Xiang}, - journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, - year={2022}, - publisher={IEEE} -} -``` diff --git a/doc/doc_ch/algorithm_det_drrg.md b/doc/doc_ch/algorithm_det_drrg.md deleted file mode 100644 index 8e08d01d1a..0000000000 --- a/doc/doc_ch/algorithm_det_drrg.md +++ /dev/null @@ -1,78 +0,0 @@ -# DRRG - -- [1. 算法简介](#1-算法简介) -- [2. 环境配置](#2-环境配置) -- [3. 模型训练、评估、预测](#3-模型训练评估预测) -- [4. 推理部署](#4-推理部署) - - [4.1 Python推理](#41-python推理) - - [4.2 C++推理](#42-c推理) - - [4.3 Serving服务化部署](#43-serving服务化部署) - - [4.4 更多推理部署](#44-更多推理部署) -- [5. FAQ](#5-faq) -- [引用](#引用) - - -## 1. 算法简介 - -论文信息: -> [Deep Relational Reasoning Graph Network for Arbitrary Shape Text Detection](https://arxiv.org/abs/2003.07493) -> Zhang, Shi-Xue and Zhu, Xiaobin and Hou, Jie-Bo and Liu, Chang and Yang, Chun and Wang, Hongfa and Yin, Xu-Cheng -> CVPR, 2020 - -在CTW1500文本检测公开数据集上,算法复现效果如下: - -| 模型 |骨干网络|配置文件|precision|recall|Hmean|下载链接| -|-----| --- | --- | --- | --- | --- | --- | -| DRRG | ResNet50_vd | [configs/det/det_r50_drrg_ctw.yml](../../configs/det/det_r50_drrg_ctw.yml)| 89.92%|80.91%|85.18%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/det_r50_drrg_ctw_train.tar)| - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -上述DRRG模型使用CTW1500文本检测公开数据集训练得到,数据集下载可参考 [ocr_datasets](./dataset/ocr_datasets.md)。 - -数据下载完成后,请参考[文本检测训练教程](./detection.md)进行训练。PaddleOCR对代码进行了模块化,训练不同的检测模型只需要**更换配置文件**即可。 - - - -## 4. 推理部署 - - -### 4.1 Python推理 - -由于模型前向运行时需要多次转换为Numpy数据进行运算,因此DRRG的动态图转静态图暂未支持。 - - -### 4.2 C++推理 - -暂未支持 - - -### 4.3 Serving服务化部署 - -暂未支持 - - -### 4.4 更多推理部署 - -暂未支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@inproceedings{zhang2020deep, - title={Deep relational reasoning graph network for arbitrary shape text detection}, - author={Zhang, Shi-Xue and Zhu, Xiaobin and Hou, Jie-Bo and Liu, Chang and Yang, Chun and Wang, Hongfa and Yin, Xu-Cheng}, - booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, - pages={9699--9708}, - year={2020} -} -``` diff --git a/doc/doc_ch/algorithm_det_east.md b/doc/doc_ch/algorithm_det_east.md deleted file mode 100644 index ef60e1e075..0000000000 --- a/doc/doc_ch/algorithm_det_east.md +++ /dev/null @@ -1,96 +0,0 @@ -# EAST - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [EAST: An Efficient and Accurate Scene Text Detector](https://arxiv.org/abs/1704.03155) -> Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, Jiajun Liang -> CVPR, 2017 - - -在ICDAR2015文本检测公开数据集上,算法复现效果如下: - -|模型|骨干网络|配置文件|precision|recall|Hmean|下载链接| -| --- | --- | --- | --- | --- | --- | --- | -|EAST|ResNet50_vd| [det_r50_vd_east.yml](../../configs/det/det_r50_vd_east.yml)|88.71%| 81.36%| 84.88%| [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_east_v2.0_train.tar)| -|EAST|MobileNetV3|[det_mv3_east.yml](../../configs/det/det_mv3_east.yml) | 78.20%| 79.10%| 78.65%| [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_east_v2.0_train.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -上表中的EAST训练模型使用ICDAR2015文本检测公开数据集训练得到,数据集下载可参考 [ocr_datasets](./dataset/ocr_datasets.md)。 - -数据下载完成后,请参考[文本检测训练教程](./detection.md)进行训练。PaddleOCR对代码进行了模块化,训练不同的检测模型只需要**更换配置文件**即可。 - - - -## 4. 推理部署 - - -### 4.1 Python推理 - -首先将EAST文本检测训练过程中保存的模型,转换成inference model。以基于Resnet50_vd骨干网络,在ICDAR2015英文数据集训练的模型为例([训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_east_v2.0_train.tar)),可以使用如下命令进行转换: - -```shell -python3 tools/export_model.py -c configs/det/det_r50_vd_east.yml -o Global.pretrained_model=./det_r50_vd_east_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_r50_east/ -``` - -EAST文本检测模型推理,需要设置参数--det_algorithm="EAST",执行预测: -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_r50_east/" --det_algorithm="EAST" -``` - -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。 - -![](../imgs_results/det_res_img_10_east.jpg) - - -### 4.2 C++推理 - -由于后处理暂未使用CPP编写,EAST文本检测模型暂不支持CPP推理。 - - -### 4.3 Serving服务化部署 - -暂未支持 - - -### 4.4 更多推理部署 - -暂未支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@inproceedings{zhou2017east, - title={East: an efficient and accurate scene text detector}, - author={Zhou, Xinyu and Yao, Cong and Wen, He and Wang, Yuzhi and Zhou, Shuchang and He, Weiran and Liang, Jiajun}, - booktitle={Proceedings of the IEEE conference on Computer Vision and Pattern Recognition}, - pages={5551--5560}, - year={2017} -} -``` diff --git a/doc/doc_ch/algorithm_det_fcenet.md b/doc/doc_ch/algorithm_det_fcenet.md deleted file mode 100644 index a70caa29fb..0000000000 --- a/doc/doc_ch/algorithm_det_fcenet.md +++ /dev/null @@ -1,102 +0,0 @@ -# FCENet - -- [1. 算法简介](#1-算法简介) -- [2. 环境配置](#2-环境配置) -- [3. 模型训练、评估、预测](#3-模型训练评估预测) -- [4. 推理部署](#4-推理部署) - - [4.1 Python推理](#41-python推理) - - [4.2 C++推理](#42-c推理) - - [4.3 Serving服务化部署](#43-serving服务化部署) - - [4.4 更多推理部署](#44-更多推理部署) -- [5. FAQ](#5-faq) -- [引用](#引用) - - -## 1. 算法简介 - -论文信息: -> [Fourier Contour Embedding for Arbitrary-Shaped Text Detection](https://arxiv.org/abs/2104.10442) -> Yiqin Zhu and Jianyong Chen and Lingyu Liang and Zhanghui Kuang and Lianwen Jin and Wayne Zhang -> CVPR, 2021 - -在CTW1500文本检测公开数据集上,算法复现效果如下: - -| 模型 |骨干网络|配置文件|precision|recall|Hmean|下载链接| -|-----| --- | --- | --- | --- | --- | --- | -| FCE | ResNet50_dcn | [configs/det/det_r50_vd_dcn_fce_ctw.yml](../../configs/det/det_r50_vd_dcn_fce_ctw.yml)| 88.39%|82.18%|85.27%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/det_r50_dcn_fce_ctw_v2.0_train.tar)| - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -上述FCE模型使用CTW1500文本检测公开数据集训练得到,数据集下载可参考 [ocr_datasets](./dataset/ocr_datasets.md)。 - -数据下载完成后,请参考[文本检测训练教程](./detection.md)进行训练。PaddleOCR对代码进行了模块化,训练不同的检测模型只需要**更换配置文件**即可。 - - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将FCE文本检测训练过程中保存的模型,转换成inference model。以基于Resnet50_vd_dcn骨干网络,在CTW1500英文数据集训练的模型为例( [模型下载地址](https://paddleocr.bj.bcebos.com/contribution/det_r50_dcn_fce_ctw_v2.0_train.tar) ),可以使用如下命令进行转换: - -```shell -python3 tools/export_model.py -c configs/det/det_r50_vd_dcn_fce_ctw.yml -o Global.pretrained_model=./det_r50_dcn_fce_ctw_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_fce -``` - -FCE文本检测模型推理,执行非弯曲文本检测,可以执行如下命令: - -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_fce/" --det_algorithm="FCE" --det_fce_box_type=quad -``` - -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_img_10_fce.jpg) - -如果想执行弯曲文本检测,可以执行如下命令: - -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img623.jpg" --det_model_dir="./inference/det_fce/" --det_algorithm="FCE" --det_fce_box_type=poly -``` - -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_img623_fce.jpg) - -**注意**:由于CTW1500数据集只有1000张训练图像,且主要针对英文场景,所以上述模型对中文文本图像检测效果会比较差。 - - -### 4.2 C++推理 - -由于后处理暂未使用CPP编写,FCE文本检测模型暂不支持CPP推理。 - - -### 4.3 Serving服务化部署 - -暂未支持 - - -### 4.4 更多推理部署 - -暂未支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@InProceedings{zhu2021fourier, - title={Fourier Contour Embedding for Arbitrary-Shaped Text Detection}, - author={Yiqin Zhu and Jianyong Chen and Lingyu Liang and Zhanghui Kuang and Lianwen Jin and Wayne Zhang}, - year={2021}, - booktitle = {CVPR} -} -``` diff --git a/doc/doc_ch/algorithm_det_psenet.md b/doc/doc_ch/algorithm_det_psenet.md deleted file mode 100644 index 58d8ccf972..0000000000 --- a/doc/doc_ch/algorithm_det_psenet.md +++ /dev/null @@ -1,106 +0,0 @@ -# PSENet - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [Shape robust text detection with progressive scale expansion network](https://arxiv.org/abs/1903.12473) -> Wang, Wenhai and Xie, Enze and Li, Xiang and Hou, Wenbo and Lu, Tong and Yu, Gang and Shao, Shuai -> CVPR, 2019 - -在ICDAR2015文本检测公开数据集上,算法复现效果如下: - -|模型|骨干网络|配置文件|precision|recall|Hmean|下载链接| -| --- | --- | --- | --- | --- | --- | --- | -|PSE| ResNet50_vd | [configs/det/det_r50_vd_pse.yml](../../configs/det/det_r50_vd_pse.yml)| 85.81% |79.53%|82.55%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_r50_vd_pse_v2.0_train.tar)| -|PSE| MobileNetV3| [configs/det/det_mv3_pse.yml](../../configs/det/det_mv3_pse.yml) | 82.20% |70.48%|75.89%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_mv3_pse_v2.0_train.tar)| - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -上述PSE模型使用ICDAR2015文本检测公开数据集训练得到,数据集下载可参考 [ocr_datasets](./dataset/ocr_datasets.md)。 - -数据下载完成后,请参考[文本检测训练教程](./detection.md)进行训练。PaddleOCR对代码进行了模块化,训练不同的检测模型只需要**更换配置文件**即可。 - - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将PSE文本检测训练过程中保存的模型,转换成inference model。以基于Resnet50_vd骨干网络,在ICDAR2015英文数据集训练的模型为例( [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_r50_vd_pse_v2.0_train.tar) ),可以使用如下命令进行转换: - -```shell -python3 tools/export_model.py -c configs/det/det_r50_vd_pse.yml -o Global.pretrained_model=./det_r50_vd_pse_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_pse -``` - -PSE文本检测模型推理,执行非弯曲文本检测,可以执行如下命令: - -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_pse/" --det_algorithm="PSE" --det_pse_box_type=quad -``` - -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_img_10_pse.jpg) - -如果想执行弯曲文本检测,可以执行如下命令: - -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_pse/" --det_algorithm="PSE" --det_pse_box_type=poly -``` - -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_img_10_pse_poly.jpg) - -**注意**:由于ICDAR2015数据集只有1000张训练图像,且主要针对英文场景,所以上述模型对中文或弯曲文本图像检测效果会比较差。 - - -### 4.2 C++推理 - -由于后处理暂未使用CPP编写,PSE文本检测模型暂不支持CPP推理。 - - -### 4.3 Serving服务化部署 - -暂未支持 - - -### 4.4 更多推理部署 - -暂未支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@inproceedings{wang2019shape, - title={Shape robust text detection with progressive scale expansion network}, - author={Wang, Wenhai and Xie, Enze and Li, Xiang and Hou, Wenbo and Lu, Tong and Yu, Gang and Shao, Shuai}, - booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, - pages={9336--9345}, - year={2019} -} -``` diff --git a/doc/doc_ch/algorithm_det_sast.md b/doc/doc_ch/algorithm_det_sast.md deleted file mode 100644 index 2ac8eeaa8f..0000000000 --- a/doc/doc_ch/algorithm_det_sast.md +++ /dev/null @@ -1,115 +0,0 @@ -# SAST - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [A Single-Shot Arbitrarily-Shaped Text Detector based on Context Attended Multi-Task Learning](https://arxiv.org/abs/1908.05498) -> Wang, Pengfei and Zhang, Chengquan and Qi, Fei and Huang, Zuming and En, Mengyi and Han, Junyu and Liu, Jingtuo and Ding, Errui and Shi, Guangming -> ACM MM, 2019 - -在ICDAR2015文本检测公开数据集上,算法复现效果如下: - -|模型|骨干网络|配置文件|precision|recall|Hmean|下载链接| -| --- | --- | --- | --- | --- | --- | --- | -|SAST|ResNet50_vd|[configs/det/det_r50_vd_sast_icdar15.yml](../../configs/det/det_r50_vd_sast_icdar15.yml)|91.39%|83.77%|87.42%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_icdar15_v2.0_train.tar)| - - -在Total-text文本检测公开数据集上,算法复现效果如下: - -|模型|骨干网络|配置文件|precision|recall|Hmean|下载链接| -| --- | --- | --- | --- | --- | --- | --- | -|SAST|ResNet50_vd|[configs/det/det_r50_vd_sast_totaltext.yml](../../configs/det/det_r50_vd_sast_totaltext.yml)|89.63%|78.44%|83.66%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_totaltext_v2.0_train.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本检测训练教程](./detection.md)。PaddleOCR对代码进行了模块化,训练不同的检测模型只需要**更换配置文件**即可。 - - - -## 4. 推理部署 - - -### 4.1 Python推理 -#### (1). 四边形文本检测模型(ICDAR2015) -首先将SAST文本检测训练过程中保存的模型,转换成inference model。以基于Resnet50_vd骨干网络,在ICDAR2015英文数据集训练的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_icdar15_v2.0_train.tar)),可以使用如下命令进行转换: -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_sast_icdar15.yml -o Global.pretrained_model=./det_r50_vd_sast_icdar15_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_sast_ic15 - -``` -**SAST文本检测模型推理,需要设置参数`--det_algorithm="SAST"`**,可以执行如下命令: -``` -python3 tools/infer/predict_det.py --det_algorithm="SAST" --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_sast_ic15/" -``` -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_img_10_sast.jpg) - -#### (2). 弯曲文本检测模型(Total-Text) -首先将SAST文本检测训练过程中保存的模型,转换成inference model。以基于Resnet50_vd骨干网络,在Total-Text英文数据集训练的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_totaltext_v2.0_train.tar)),可以使用如下命令进行转换: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_sast_totaltext.yml -o Global.pretrained_model=./det_r50_vd_sast_totaltext_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_sast_tt - -``` - -SAST文本检测模型推理,需要设置参数`--det_algorithm="SAST"`,同时,还需要增加参数`--det_box_type=poly`,可以执行如下命令: -``` -python3 tools/infer/predict_det.py --det_algorithm="SAST" --image_dir="./doc/imgs_en/img623.jpg" --det_model_dir="./inference/det_sast_tt/" --det_box_type='poly' -``` -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_img623_sast.jpg) - -**注意**:本代码库中,SAST后处理Locality-Aware NMS有python和c++两种版本,c++版速度明显快于python版。由于c++版本nms编译版本问题,只有python3.5环境下会调用c++版nms,其他情况将调用python版nms。 - - -### 4.2 C++推理 - -暂未支持 - - -### 4.3 Serving服务化部署 - -暂未支持 - - -### 4.4 更多推理部署 - -暂未支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@inproceedings{wang2019single, - title={A Single-Shot Arbitrarily-Shaped Text Detector based on Context Attended Multi-Task Learning}, - author={Wang, Pengfei and Zhang, Chengquan and Qi, Fei and Huang, Zuming and En, Mengyi and Han, Junyu and Liu, Jingtuo and Ding, Errui and Shi, Guangming}, - booktitle={Proceedings of the 27th ACM International Conference on Multimedia}, - pages={1277--1285}, - year={2019} -} -``` diff --git a/doc/doc_ch/algorithm_e2e_pgnet.md b/doc/doc_ch/algorithm_e2e_pgnet.md deleted file mode 100644 index 78d710fdb9..0000000000 --- a/doc/doc_ch/algorithm_e2e_pgnet.md +++ /dev/null @@ -1,187 +0,0 @@ -# 端对端OCR算法-PGNet -- [一、简介](#简介) -- [二、环境配置](#环境配置) -- [三、快速使用](#快速使用) -- [四、模型训练、评估、推理](#模型训练、评估、推理) - - -## 一、简介 -OCR算法可以分为两阶段算法和端对端的算法。二阶段OCR算法一般分为两个部分,文本检测和文本识别算法,文件检测算法从图像中得到文本行的检测框,然后识别算法去识别文本框中的内容。而端对端OCR算法可以在一个算法中完成文字检测和文字识别,其基本思想是设计一个同时具有检测单元和识别模块的模型,共享其中两者的CNN特征,并联合训练。由于一个算法即可完成文字识别,端对端模型更小,速度更快。 - -### PGNet算法介绍 -近些年来,端对端OCR算法得到了良好的发展,包括MaskTextSpotter系列、TextSnake、TextDragon、PGNet系列等算法。在这些算法中,PGNet算法具备其他算法不具备的优势,包括: -- 设计PGNet loss指导训练,不需要字符级别的标注 -- 不需要NMS和ROI相关操作,加速预测 -- 提出预测文本行内的阅读顺序模块; -- 提出基于图的修正模块(GRM)来进一步提高模型识别性能 -- 精度更高,预测速度更快 - -PGNet算法细节详见[论文](https://www.aaai.org/AAAI21Papers/AAAI-2885.WangP.pdf) ,算法原理图如下所示: -![](../pgnet_framework.png) -输入图像经过特征提取送入四个分支,分别是:文本边缘偏移量预测TBO模块,文本中心线预测TCL模块,文本方向偏移量预测TDO模块,以及文本字符分类图预测TCC模块。 -其中TBO以及TCL的输出经过后处理后可以得到文本的检测结果,TCL、TDO、TCC负责文本识别。 - -其检测识别效果图如下: - -![](../imgs_results/e2e_res_img293_pgnet.png) -![](../imgs_results/e2e_res_img295_pgnet.png) - -### 性能指标 - -#### 测试集: Total Text - -#### 测试环境: NVIDIA Tesla V100-SXM2-16GB - -|PGNetA|det_precision|det_recall|det_f_score|e2e_precision|e2e_recall|e2e_f_score|FPS|下载| -| --- | --- | --- | --- | --- | --- | --- | --- | --- | -|Paper|85.30|86.80|86.10|-|-|61.70|38.20 (size=640)|-| -|Ours|87.03|82.48|84.69|61.71|58.43|60.03|48.73 (size=768)|[下载链接](https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/en_server_pgnetA.tar)| - -*note:PaddleOCR里的PGNet实现针对预测速度做了优化,在精度下降可接受范围内,可以显著提升端对端预测速度* - - - - -## 二、环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目 - - -## 三、快速使用 -### inference模型下载 -本节以训练好的端到端模型为例,快速使用模型预测,首先下载训练好的端到端inference模型[下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/e2e_server_pgnetA_infer.tar) -``` -mkdir inference && cd inference -# 下载英文端到端模型并解压 -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/e2e_server_pgnetA_infer.tar && tar xf e2e_server_pgnetA_infer.tar -``` -* windows 环境下如果没有安装wget,下载模型时可将链接复制到浏览器中下载,并解压放置在相应目录下 - -解压完毕后应有如下文件结构: -``` -├── e2e_server_pgnetA_infer -│ ├── inference.pdiparams -│ ├── inference.pdiparams.info -│ └── inference.pdmodel -``` -### 单张图像或者图像集合预测 -```bash -# 预测image_dir指定的单张图像 -python3 tools/infer/predict_e2e.py --e2e_algorithm="PGNet" --image_dir="./doc/imgs_en/img623.jpg" --e2e_model_dir="./inference/e2e_server_pgnetA_infer/" --e2e_pgnet_valid_set="totaltext" - -# 预测image_dir指定的图像集合 -python3 tools/infer/predict_e2e.py --e2e_algorithm="PGNet" --image_dir="./doc/imgs_en/" --e2e_model_dir="./inference/e2e_server_pgnetA_infer/" --e2e_pgnet_valid_set="totaltext" - -# 如果想使用CPU进行预测,需设置use_gpu参数为False -python3 tools/infer/predict_e2e.py --e2e_algorithm="PGNet" --image_dir="./doc/imgs_en/img623.jpg" --e2e_model_dir="./inference/e2e_server_pgnetA_infer/" --e2e_pgnet_valid_set="totaltext" --use_gpu=False -``` -### 可视化结果 -可视化文本检测结果默认保存到./inference_results文件夹里面,结果文件的名称前缀为'e2e_res'。结果示例如下: -![](../imgs_results/e2e_res_img623_pgnet.jpg) - - -## 四、模型训练、评估、推理 -本节以totaltext数据集为例,介绍PaddleOCR中端到端模型的训练、评估与测试。 - -### 准备数据 -下载解压[totaltext](https://paddleocr.bj.bcebos.com/dataset/total_text.tar) 数据集到PaddleOCR/train_data/目录,数据集组织结构: -``` -/PaddleOCR/train_data/total_text/train/ - |- rgb/ # total_text数据集的训练数据 - |- img11.jpg - | ... - |- train.txt # total_text数据集的训练标注 -``` - -train.txt标注文件格式如下,文件名和标注信息中间用"\t"分隔: -``` -" 图像文件名 json.dumps编码的图像标注信息" -rgb/img11.jpg [{"transcription": "ASRAMA", "points": [[214.0, 325.0], [235.0, 308.0], [259.0, 296.0], [286.0, 291.0], [313.0, 295.0], [338.0, 305.0], [362.0, 320.0], [349.0, 347.0], [330.0, 337.0], [310.0, 329.0], [290.0, 324.0], [269.0, 328.0], [249.0, 336.0], [231.0, 346.0]]}, {...}] -``` -json.dumps编码前的图像标注信息是包含多个字典的list,字典中的 `points` 表示文本框的四个点的坐标(x, y),从左上角的点开始顺时针排列。 -`transcription` 表示当前文本框的文字,**当其内容为“###”时,表示该文本框无效,在训练时会跳过。** -如果您想在其他数据集上训练,可以按照上述形式构建标注文件。 - -### 启动训练 - -PGNet训练分为两个步骤:step1: 在合成数据上训练,得到预训练模型,此时模型精度依然较低;step2: 加载预训练模型,在totaltext数据集上训练;为快速训练,我们直接提供了step1的预训练模型。 -```shell -cd PaddleOCR/ -下载step1 预训练模型 -wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/train_step1.tar -可以得到以下的文件格式 -./pretrain_models/train_step1/ - └─ best_accuracy.pdopt - └─ best_accuracy.states - └─ best_accuracy.pdparams -``` -*如果您安装的是cpu版本,请将配置文件中的 `use_gpu` 字段修改为false* - -```shell -# 单机单卡训练 e2e 模型 -python3 tools/train.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.pretrained_model=./pretrain_models/train_step1/best_accuracy Global.load_static_weights=False -# 单机多卡训练,通过 --gpus 参数设置使用的GPU ID -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.pretrained_model=./pretrain_models/train_step1/best_accuracy Global.load_static_weights=False -``` - -上述指令中,通过-c 选择训练使用configs/e2e/e2e_r50_vd_pg.yml配置文件。 -有关配置文件的详细解释,请参考[链接](./config.md)。 - -您也可以通过-o参数在不需要修改yml文件的情况下,改变训练的参数,比如,调整训练的学习率为0.0001 -```shell -python3 tools/train.py -c configs/e2e/e2e_r50_vd_pg.yml -o Optimizer.base_lr=0.0001 -``` - -#### 断点训练 -如果训练程序中断,如果希望加载训练中断的模型从而恢复训练,可以通过指定Global.checkpoints指定要加载的模型路径: -```shell -python3 tools/train.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.checkpoints=./your/trained/model -``` - -**注意**:`Global.checkpoints`的优先级高于`Global.pretrain_weights`的优先级,即同时指定两个参数时,优先加载`Global.checkpoints`指定的模型,如果`Global.checkpoints`指定的模型路径有误,会加载`Global.pretrain_weights`指定的模型。 - -PaddleOCR计算三个OCR端到端相关的指标,分别是:Precision、Recall、Hmean。 - -运行如下代码,根据配置文件`e2e_r50_vd_pg.yml`中`save_res_path`指定的测试集检测结果文件,计算评估指标。 - -评估时设置后处理参数`max_side_len=768`,使用不同数据集、不同模型训练,可调整参数进行优化 -训练中模型参数默认保存在`Global.save_model_dir`目录下。在评估指标时,需要设置`Global.checkpoints`指向保存的参数文件。 -```shell -python3 tools/eval.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.checkpoints="{path/to/weights}/best_accuracy" -``` - -### 模型预测 -测试单张图像的端到端识别效果 -```shell -python3 tools/infer_e2e.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.infer_img="./doc/imgs_en/img_10.jpg" Global.pretrained_model="./output/e2e_pgnet/best_accuracy" Global.load_static_weights=false -``` - -测试文件夹下所有图像的端到端识别效果 -```shell -python3 tools/infer_e2e.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.infer_img="./doc/imgs_en/" Global.pretrained_model="./output/e2e_pgnet/best_accuracy" Global.load_static_weights=false -``` - -### 预测推理 -#### (1). 四边形文本检测模型(ICDAR2015) -首先将PGNet端到端训练过程中保存的模型,转换成inference model。以基于Resnet50_vd骨干网络,以英文数据集训练的模型为例[模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/en_server_pgnetA.tar) ,可以使用如下命令进行转换: -``` -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/en_server_pgnetA.tar && tar xf en_server_pgnetA.tar -python3 tools/export_model.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.pretrained_model=./en_server_pgnetA/best_accuracy Global.load_static_weights=False Global.save_inference_dir=./inference/e2e -``` -**PGNet端到端模型推理,需要设置参数`--e2e_algorithm="PGNet"` and `--e2e_pgnet_valid_set="partvgg"`**,可以执行如下命令: -``` -python3 tools/infer/predict_e2e.py --e2e_algorithm="PGNet" --image_dir="./doc/imgs_en/img_10.jpg" --e2e_model_dir="./inference/e2e/" --e2e_pgnet_valid_set="partvgg" --e2e_pgnet_valid_set="totaltext" -``` -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'e2e_res'。结果示例如下: - -![](../imgs_results/e2e_res_img_10_pgnet.jpg) - -#### (2). 弯曲文本检测模型(Total-Text) -对于弯曲文本样例 - -**PGNet端到端模型推理,需要设置参数`--e2e_algorithm="PGNet"`,同时,还需要增加参数`--e2e_pgnet_valid_set="totaltext"`,**可以执行如下命令: -``` -python3 tools/infer/predict_e2e.py --e2e_algorithm="PGNet" --image_dir="./doc/imgs_en/img623.jpg" --e2e_model_dir="./inference/e2e/" --e2e_pgnet_valid_set="totaltext" -``` -可视化文本端到端结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'e2e_res'。结果示例如下: - -![](../imgs_results/e2e_res_img623_pgnet.jpg) diff --git a/doc/doc_ch/algorithm_inference.md b/doc/doc_ch/algorithm_inference.md deleted file mode 100755 index f4858a9a0d..0000000000 --- a/doc/doc_ch/algorithm_inference.md +++ /dev/null @@ -1,536 +0,0 @@ - -# 基于Python预测引擎推理 - -inference 模型(`paddle.jit.save`保存的模型) -一般是模型训练,把模型结构和模型参数保存在文件中的固化模型,多用于预测部署场景。 -训练过程中保存的模型是checkpoints模型,保存的只有模型的参数,多用于恢复训练等。 -与checkpoints模型相比,inference 模型会额外保存模型的结构信息,在预测部署、加速推理上性能优越,灵活方便,适合于实际系统集成。 - -接下来首先介绍如何将训练的模型转换成inference模型,然后将依次介绍文本检测、文本角度分类器、文本识别以及三者串联在CPU、GPU上的预测方法。 - - -- [一、训练模型转inference模型](#训练模型转inference模型) - - [检测模型转inference模型](#检测模型转inference模型) - - [识别模型转inference模型](#识别模型转inference模型) - - [方向分类模型转inference模型](#方向分类模型转inference模型) - - -- [二、文本检测模型推理](#文本检测模型推理) - - [1. 超轻量中文检测模型推理](#超轻量中文检测模型推理) - - [2. DB文本检测模型推理](#DB文本检测模型推理) - - [3. EAST文本检测模型推理](#EAST文本检测模型推理) - - [4. SAST文本检测模型推理](#SAST文本检测模型推理) - - -- [三、文本识别模型推理](#文本识别模型推理) - - [1. 超轻量中文识别模型推理](#超轻量中文识别模型推理) - - [2. 基于CTC损失的识别模型推理](#基于CTC损失的识别模型推理) - - [3. 基于SRN损失的识别模型推理](#基于SRN损失的识别模型推理) - - [4. 自定义文本识别字典的推理](#自定义文本识别字典的推理) - - [5. 多语言模型的推理](#多语言模型的推理) - - -- [四、方向分类模型推理](#方向识别模型推理) - - [1. 方向分类模型推理](#方向分类模型推理) - - -- [五、文本检测、方向分类和文字识别串联推理](#文本检测、方向分类和文字识别串联推理) - - [1. 超轻量中文OCR模型推理](#超轻量中文OCR模型推理) - - [2. 其他模型推理](#其他模型推理) - - -- [六、参数解释](#参数解释) - - -- [七、FAQ](#FAQ) - - - -## 一、训练模型转inference模型 - -### 检测模型转inference模型 - -下载超轻量级中文检测模型: -``` -wget -P ./ch_lite/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_train.tar && tar xf ./ch_lite/ch_ppocr_mobile_v2.0_det_train.tar -C ./ch_lite/ -``` -上述模型是以MobileNetV3为backbone训练的DB算法,将训练好的模型转换成inference模型只需要运行如下命令: -``` -# -c 后面设置训练算法的yml配置文件 -# -o 配置可选参数 -# Global.pretrained_model 参数设置待转换的训练模型地址,不用添加文件后缀 .pdmodel,.pdopt或.pdparams。 -# Global.save_inference_dir参数设置转换的模型将保存的地址。 - -python3 tools/export_model.py -c configs/det/ch_ppocr_v2.0/ch_det_mv3_db_v2.0.yml -o Global.pretrained_model=./ch_lite/ch_ppocr_mobile_v2.0_det_train/best_accuracy Global.save_inference_dir=./inference/det_db/ -``` -转inference模型时,使用的配置文件和训练时使用的配置文件相同。另外,还需要设置配置文件中的`Global.pretrained_model`参数,其指向训练中保存的模型参数文件。 -转换成功后,在模型保存目录下有三个文件: -``` -inference/det_db/ - ├── inference.pdiparams # 检测inference模型的参数文件 - ├── inference.pdiparams.info # 检测inference模型的参数信息,可忽略 - └── inference.pdmodel # 检测inference模型的program文件 -``` - - -### 识别模型转inference模型 - -下载超轻量中文识别模型: -``` -wget -P ./ch_lite/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_train.tar && tar xf ./ch_lite/ch_ppocr_mobile_v2.0_rec_train.tar -C ./ch_lite/ -``` - -识别模型转inference模型与检测的方式相同,如下: -``` -# -c 后面设置训练算法的yml配置文件 -# -o 配置可选参数 -# Global.pretrained_model 参数设置待转换的训练模型地址,不用添加文件后缀 .pdmodel,.pdopt或.pdparams。 -# Global.save_inference_dir参数设置转换的模型将保存的地址。 - -python3 tools/export_model.py -c configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml -o Global.pretrained_model=./ch_lite/ch_ppocr_mobile_v2.0_rec_train/best_accuracy Global.save_inference_dir=./inference/rec_crnn/ -``` - -**注意:**如果您是在自己的数据集上训练的模型,并且调整了中文字符的字典文件,请注意修改配置文件中的`character_dict_path`是否是所需要的字典文件。 - -转换成功后,在目录下有三个文件: -``` -/inference/rec_crnn/ - ├── inference.pdiparams # 识别inference模型的参数文件 - ├── inference.pdiparams.info # 识别inference模型的参数信息,可忽略 - └── inference.pdmodel # 识别inference模型的program文件 -``` - - -### 方向分类模型转inference模型 - -下载方向分类模型: -``` -wget -P ./ch_lite/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar && tar xf ./ch_lite/ch_ppocr_mobile_v2.0_cls_train.tar -C ./ch_lite/ -``` - -方向分类模型转inference模型与检测的方式相同,如下: -``` -# -c 后面设置训练算法的yml配置文件 -# -o 配置可选参数 -# Global.pretrained_model 参数设置待转换的训练模型地址,不用添加文件后缀 .pdmodel,.pdopt或.pdparams。 -# Global.save_inference_dir参数设置转换的模型将保存的地址。 - -python3 tools/export_model.py -c configs/cls/cls_mv3.yml -o Global.pretrained_model=./ch_lite/ch_ppocr_mobile_v2.0_cls_train/best_accuracy Global.save_inference_dir=./inference/cls/ -``` - -转换成功后,在目录下有三个文件: -``` -/inference/cls/ - ├── inference.pdiparams # 分类inference模型的参数文件 - ├── inference.pdiparams.info # 分类inference模型的参数信息,可忽略 - └── inference.pdmodel # 分类inference模型的program文件 -``` - - -## 二、文本检测模型推理 - -文本检测模型推理,默认使用DB模型的配置参数。当不使用DB模型时,在推理时,需要通过传入相应的参数进行算法适配,细节参考下文。 - - -### 1. 超轻量中文检测模型推理 - -超轻量中文检测模型推理,可以执行如下命令: - -``` -# 下载超轻量中文检测模型: -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_infer.tar -tar xf ch_ppocr_mobile_v2.0_det_infer.tar -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./ch_ppocr_mobile_v2.0_det_infer/" -``` - -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_00018069.jpg) - -通过参数`limit_type`和`det_limit_side_len`来对图片的尺寸进行限制, -`limit_type`可选参数为[`max`, `min`], -`det_limit_size_len` 为正整数,一般设置为32 的倍数,比如960。 - -参数默认设置为`limit_type='max', det_limit_side_len=960`。表示网络输入图像的最长边不能超过960, -如果超过这个值,会对图像做等宽比的resize操作,确保最长边为`det_limit_side_len`。 -设置为`limit_type='min', det_limit_side_len=960` 则表示限制图像的最短边为960。 - -如果输入图片的分辨率比较大,而且想使用更大的分辨率预测,可以设置det_limit_side_len 为想要的值,比如1216: -``` -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./inference/det_db/" --det_limit_type=max --det_limit_side_len=1216 -``` - -如果想使用CPU进行预测,执行命令如下 -``` -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./inference/det_db/" --use_gpu=False -``` - - -### 2. DB文本检测模型推理 - -首先将DB文本检测训练过程中保存的模型,转换成inference model。以基于Resnet50_vd骨干网络,在ICDAR2015英文数据集训练的模型为例( [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_db_v2.0_train.tar) ),可以使用如下命令进行转换: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_db.yml -o Global.pretrained_model=./det_r50_vd_db_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_db -``` - -DB文本检测模型推理,可以执行如下命令: - -``` -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_db/" -``` - -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_img_10_db.jpg) - -**注意**:由于ICDAR2015数据集只有1000张训练图像,且主要针对英文场景,所以上述模型对中文文本图像检测效果会比较差。 - - -### 3. EAST文本检测模型推理 - -首先将EAST文本检测训练过程中保存的模型,转换成inference model。以基于Resnet50_vd骨干网络,在ICDAR2015英文数据集训练的模型为例( [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_east_v2.0_train.tar) ),可以使用如下命令进行转换: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_east.yml -o Global.pretrained_model=./det_r50_vd_east_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_east -``` - -**EAST文本检测模型推理,需要设置参数`--det_algorithm="EAST"`**,可以执行如下命令: - -``` -python3 tools/infer/predict_det.py --det_algorithm="EAST" --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_east/" -``` -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_img_10_east.jpg) - -**注意**:本代码库中,EAST后处理Locality-Aware NMS有python和c++两种版本,c++版速度明显快于python版。由于c++版本nms编译版本问题,只有python3.5环境下会调用c++版nms,其他情况将调用python版nms。 - - - -### 4. SAST文本检测模型推理 -#### (1). 四边形文本检测模型(ICDAR2015) -首先将SAST文本检测训练过程中保存的模型,转换成inference model。以基于Resnet50_vd骨干网络,在ICDAR2015英文数据集训练的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_icdar15_v2.0_train.tar)),可以使用如下命令进行转换: -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_sast_icdar15.yml -o Global.pretrained_model=./det_r50_vd_sast_icdar15_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_sast_ic15 - -``` -**SAST文本检测模型推理,需要设置参数`--det_algorithm="SAST"`**,可以执行如下命令: -``` -python3 tools/infer/predict_det.py --det_algorithm="SAST" --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_sast_ic15/" -``` -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_img_10_sast.jpg) - -#### (2). 弯曲文本检测模型(Total-Text) -首先将SAST文本检测训练过程中保存的模型,转换成inference model。以基于Resnet50_vd骨干网络,在Total-Text英文数据集训练的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_totaltext_v2.0_train.tar)),可以使用如下命令进行转换: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_sast_totaltext.yml -o Global.pretrained_model=./det_r50_vd_sast_totaltext_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_sast_tt - -``` - -SAST文本检测模型推理,需要设置参数`--det_algorithm="SAST"`,同时,还需要增加参数`--det_sast_polygon=True`,可以执行如下命令: -``` -python3 tools/infer/predict_det.py --det_algorithm="SAST" --image_dir="./doc/imgs_en/img623.jpg" --det_model_dir="./inference/det_sast_tt/" --det_sast_polygon=True -``` -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_img623_sast.jpg) - -**注意**:本代码库中,SAST后处理Locality-Aware NMS有python和c++两种版本,c++版速度明显快于python版。由于c++版本nms编译版本问题,只有python3.5环境下会调用c++版nms,其他情况将调用python版nms。 - - - -## 三、文本识别模型推理 - -下面将介绍超轻量中文识别模型推理、基于CTC损失的识别模型推理和基于Attention损失的识别模型推理。对于中文文本识别,建议优先选择基于CTC损失的识别模型,实践中也发现基于Attention损失的效果不如基于CTC损失的识别模型。此外,如果训练时修改了文本的字典,请参考下面的自定义文本识别字典的推理。 - - - -### 1. 超轻量中文识别模型推理 - -超轻量中文识别模型推理,可以执行如下命令: - -``` -# 下载超轻量中文识别模型: -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_infer.tar -tar xf ch_ppocr_mobile_v2.0_rec_infer.tar -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/ch/word_4.jpg" --rec_model_dir="ch_ppocr_mobile_v2.0_rec_infer" -``` - -![](../imgs_words/ch/word_4.jpg) - -执行命令后,上面图像的预测结果(识别的文本和得分)会打印到屏幕上,示例如下: - -```bash -Predicts of ./doc/imgs_words/ch/word_4.jpg:('实力活力', 0.98458153) -``` - - -### 2. 基于CTC损失的识别模型推理 - -我们以 CRNN 为例,介绍基于CTC损失的识别模型推理。 Rosetta 使用方式类似,不用设置识别算法参数rec_algorithm。 - -首先将 CRNN 文本识别训练过程中保存的模型,转换成inference model。以基于Resnet34_vd骨干网络,使用MJSynth和SynthText两个英文文本识别合成数据集训练 -的模型为例( [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar) ),可以使用如下命令进行转换: - -``` -python3 tools/export_model.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model=./rec_r34_vd_none_bilstm_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_crnn -``` - -CRNN 文本识别模型推理,可以执行如下命令: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./inference/rec_crnn/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" -``` - -![](../imgs_words_en/word_336.png) - -执行命令后,上面图像的识别结果如下: - -```bash -Predicts of ./doc/imgs_words_en/word_336.png:('super', 0.9999073) -``` - -**注意**:由于上述模型是参考[DTRB](https://arxiv.org/abs/1904.01906)文本识别训练和评估流程,与超轻量级中文识别模型训练有两方面不同: - -- 训练时采用的图像分辨率不同,训练上述模型采用的图像分辨率是[3,32,100],而中文模型训练时,为了保证长文本的识别效果,训练时采用的图像分辨率是[3, 32, 320]。预测推理程序默认的形状参数是训练中文采用的图像分辨率,即[3, 32, 320]。因此,这里推理上述英文模型时,需要通过参数rec_image_shape设置识别图像的形状。 - -- 字符列表,DTRB论文中实验只是针对26个小写英文本母和10个数字进行实验,总共36个字符。所有大小字符都转成了小写字符,不在上面列表的字符都忽略,认为是空格。因此这里没有输入字符字典,而是通过如下命令生成字典.因此在推理时需要设置参数rec_char_dict_path,指定为英文字典"./ppocr/utils/ic15_dict.txt"。 - -``` -self.character_str = "0123456789abcdefghijklmnopqrstuvwxyz" -dict_character = list(self.character_str) -``` - -### 3. 基于SRN损失的识别模型推理 -基于SRN损失的识别模型,需要额外设置识别算法参数 --rec_algorithm="SRN"。 -同时需要保证预测shape与训练时一致,如: --rec_image_shape="1, 64, 256" - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" \ - --rec_model_dir="./inference/srn/" \ - --rec_image_shape="1, 64, 256" \ - --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" \ - --rec_algorithm="SRN" -``` - -### 4. 自定义文本识别字典的推理 -如果训练时修改了文本的字典,在使用inference模型预测时,需要通过`--rec_char_dict_path`指定使用的字典路径 - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./your inference model" --rec_image_shape="3, 32, 100" --rec_char_dict_path="your text dict path" -``` - - -### 5. 多语言模型的推理 -如果您需要预测的是其他语言模型,在使用inference模型预测时,需要通过`--rec_char_dict_path`指定使用的字典路径, 同时为了得到正确的可视化结果, -需要通过 `--vis_font_path` 指定可视化的字体路径,`doc/fonts/` 路径下有默认提供的小语种字体,例如韩文识别: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/korean/1.jpg" --rec_model_dir="./your inference model" --rec_char_dict_path="ppocr/utils/dict/korean_dict.txt" --vis_font_path="doc/fonts/korean.ttf" -``` -![](../imgs_words/korean/1.jpg) - -执行命令后,上图的预测结果为: -``` text -Predicts of ./doc/imgs_words/korean/1.jpg:('바탕으로', 0.9948904) -``` - - -## 四、方向分类模型推理 - -下面将介绍方向分类模型推理。 - - -### 1. 方向分类模型推理 - -方向分类模型推理,可以执行如下命令: - -``` -# 下载超轻量中文方向分类器模型: -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar -tar xf ch_ppocr_mobile_v2.0_cls_infer.tar -python3 tools/infer/predict_cls.py --image_dir="./doc/imgs_words/ch/word_4.jpg" --cls_model_dir="ch_ppocr_mobile_v2.0_cls_infer" -``` - -![](../imgs_words/ch/word_1.jpg) - -执行命令后,上面图像的预测结果(分类的方向和得分)会打印到屏幕上,示例如下: - -``` -Predicts of ./doc/imgs_words/ch/word_4.jpg:['0', 0.9999982] -``` - - -## 五、文本检测、方向分类和文字识别串联推理 - -### 1. 超轻量中文OCR模型推理 - -在执行预测时,需要通过参数`image_dir`指定单张图像或者图像集合的路径、参数`det_model_dir`,`cls_model_dir`和`rec_model_dir`分别指定检测,方向分类和识别的inference模型路径。参数`use_angle_cls`用于控制是否启用方向分类模型。`use_mp`表示是否使用多进程。`total_process_num`表示在使用多进程时的进程数。可视化识别结果默认保存到 ./inference_results 文件夹里面。 - -```shell -# 使用方向分类器 -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./inference/det_db/" --cls_model_dir="./inference/cls/" --rec_model_dir="./inference/rec_crnn/" --use_angle_cls=true - -# 不使用方向分类器 -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./inference/det_db/" --rec_model_dir="./inference/rec_crnn/" --use_angle_cls=false - -# 使用多进程 -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./inference/det_db/" --rec_model_dir="./inference/rec_crnn/" --use_angle_cls=false --use_mp=True --total_process_num=6 -``` - - - -执行命令后,识别结果图像如下: - -![](../imgs_results/system_res_00018069.jpg) - - -### 2. 其他模型推理 - -如果想尝试使用其他检测算法或者识别算法,请参考上述文本检测模型推理和文本识别模型推理,更新相应配置和模型。 - -**注意:由于检测框矫正逻辑的局限性,暂不支持使用SAST弯曲文本检测模型(即,使用参数`--det_sast_polygon=True`时)进行模型串联。** - -下面给出基于EAST文本检测和STAR-Net文本识别执行命令: - -``` -python3 tools/infer/predict_system.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_east/" --det_algorithm="EAST" --rec_model_dir="./inference/starnet/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" -``` - -执行命令后,识别结果图像如下: - -![](../imgs_results/img_10_east_starnet.jpg) - - - - -# 六、参数解释 - -更多关于预测过程的参数解释如下所示。 - -* 全局信息 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| image_dir | str | 无,必须显式指定 | 图像或者文件夹路径 | -| vis_font_path | str | "./doc/fonts/simfang.ttf" | 用于可视化的字体路径 | -| drop_score | float | 0.5 | 识别得分小于该值的结果会被丢弃,不会作为返回结果 | -| use_pdserving | bool | False | 是否使用Paddle Serving进行预测 | -| warmup | bool | False | 是否开启warmup,在统计预测耗时的时候,可以使用这种方法 | -| draw_img_save_dir | str | "./inference_results" | 系统串联预测OCR结果的保存文件夹 | -| save_crop_res | bool | False | 是否保存OCR的识别文本图像 | -| crop_res_save_dir | str | "./output" | 保存OCR识别出来的文本图像路径 | -| use_mp | bool | False | 是否开启多进程预测 | -| total_process_num | int | 6 | 开启的进城数,`use_mp`为`True`时生效 | -| process_id | int | 0 | 当前进程的id号,无需自己修改 | -| benchmark | bool | False | 是否开启benchmark,对预测速度、显存占用等进行统计 | -| save_log_path | str | "./log_output/" | 开启`benchmark`时,日志结果的保存文件夹 | -| show_log | bool | True | 是否显示预测中的日志信息 | -| use_onnx | bool | False | 是否开启onnx预测 | - - -* 预测引擎相关 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| use_gpu | bool | True | 是否使用GPU进行预测 | -| ir_optim | bool | True | 是否对计算图进行分析与优化,开启后可以加速预测过程 | -| use_tensorrt | bool | False | 是否开启tensorrt | -| min_subgraph_size | int | 15 | tensorrt中最小子图size,当子图的size大于该值时,才会尝试对该子图使用trt engine计算 | -| precision | str | fp32 | 预测的精度,支持`fp32`, `fp16`, `int8` 3种输入 | -| enable_mkldnn | bool | True | 是否开启mkldnn | -| cpu_threads | int | 10 | 开启mkldnn时,cpu预测的线程数 | - -* 文本检测模型相关 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| det_algorithm | str | "DB" | 文本检测算法名称,目前支持`DB`, `EAST`, `SAST`, `PSE` | -| det_model_dir | str | xx | 检测inference模型路径 | -| det_limit_side_len | int | 960 | 检测的图像边长限制 | -| det_limit_type | str | "max" | 检测的变成限制类型,目前支持`min`, `max`,`min`表示保证图像最短边不小于`det_limit_side_len`,`max`表示保证图像最长边不大于`det_limit_side_len` | - -其中,DB算法相关参数如下 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| det_db_thresh | float | 0.3 | DB输出的概率图中,得分大于该阈值的像素点才会被认为是文字像素点 | -| det_db_box_thresh | float | 0.6 | 检测结果边框内,所有像素点的平均得分大于该阈值时,该结果会被认为是文字区域 | -| det_db_unclip_ratio | float | 1.5 | `Vatti clipping`算法的扩张系数,使用该方法对文字区域进行扩张 | -| max_batch_size | int | 10 | 预测的batch size | -| use_dilation | bool | False | 是否对分割结果进行膨胀以获取更优检测效果 | -| det_db_score_mode | str | "fast" | DB的检测结果得分计算方法,支持`fast`和`slow`,`fast`是根据polygon的外接矩形边框内的所有像素计算平均得分,`slow`是根据原始polygon内的所有像素计算平均得分,计算速度相对较慢一些,但是更加准确一些。 | - -EAST算法相关参数如下 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| det_east_score_thresh | float | 0.8 | EAST后处理中score map的阈值 | -| det_east_cover_thresh | float | 0.1 | EAST后处理中文本框的平均得分阈值 | -| det_east_nms_thresh | float | 0.2 | EAST后处理中nms的阈值 | - -SAST算法相关参数如下 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| det_sast_score_thresh | float | 0.5 | SAST后处理中的得分阈值 | -| det_sast_nms_thresh | float | 0.5 | SAST后处理中nms的阈值 | -| det_sast_polygon | bool | False | 是否多边形检测,弯曲文本场景(如Total-Text)设置为True | - -PSE算法相关参数如下 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| det_pse_thresh | float | 0.0 | 对输出图做二值化的阈值 | -| det_pse_box_thresh | float | 0.85 | 对box进行过滤的阈值,低于此阈值的丢弃 | -| det_pse_min_area | float | 16 | box的最小面积,低于此阈值的丢弃 | -| det_pse_box_type | str | "box" | 返回框的类型,box:四点坐标,poly: 弯曲文本的所有点坐标 | -| det_pse_scale | int | 1 | 输入图像相对于进后处理的图的比例,如`640*640`的图像,网络输出为`160*160`,scale为2的情况下,进后处理的图片shape为`320*320`。这个值调大可以加快后处理速度,但是会带来精度的下降 | - -* 文本识别模型相关 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| rec_algorithm | str | "CRNN" | 文本识别算法名称,目前支持`CRNN`, `SRN`, `RARE`, `NETR`, `SAR` | -| rec_model_dir | str | 无,如果使用识别模型,该项是必填项 | 识别inference模型路径 | -| rec_image_shape | list | [3, 32, 320] | 识别时的图像尺寸, | -| rec_batch_num | int | 6 | 识别的batch size | -| max_text_length | int | 25 | 识别结果最大长度,在`SRN`中有效 | -| rec_char_dict_path | str | "./ppocr/utils/ppocr_keys_v1.txt" | 识别的字符字典文件 | -| use_space_char | bool | True | 是否包含空格,如果为`True`,则会在最后字符字典中补充`空格`字符 | - - -* 端到端文本检测与识别模型相关 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| e2e_algorithm | str | "PGNet" | 端到端算法名称,目前支持`PGNet` | -| e2e_model_dir | str | 无,如果使用端到端模型,该项是必填项 | 端到端模型inference模型路径 | -| e2e_limit_side_len | int | 768 | 端到端的输入图像边长限制 | -| e2e_limit_type | str | "max" | 端到端的边长限制类型,目前支持`min`, `max`,`min`表示保证图像最短边不小于`e2e_limit_side_len`,`max`表示保证图像最长边不大于`e2e_limit_side_len` | -| e2e_pgnet_score_thresh | float | 0.5 | 端到端得分阈值,小于该阈值的结果会被丢弃 | -| e2e_char_dict_path | str | "./ppocr/utils/ic15_dict.txt" | 识别的字典文件路径 | -| e2e_pgnet_valid_set | str | "totaltext" | 验证集名称,目前支持`totaltext`, `partvgg`,不同数据集对应的后处理方式不同,与训练过程保持一致即可 | -| e2e_pgnet_mode | str | "fast" | PGNet的检测结果得分计算方法,支持`fast`和`slow`,`fast`是根据polygon的外接矩形边框内的所有像素计算平均得分,`slow`是根据原始polygon内的所有像素计算平均得分,计算速度相对较慢一些,但是更加准确一些。 | - - -* 方向分类器模型相关 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| use_angle_cls | bool | False | 是否使用方向分类器 | -| cls_model_dir | str | 无,如果需要使用,则必须显式指定路径 | 方向分类器inference模型路径 | -| cls_image_shape | list | [3, 48, 192] | 预测尺度 | -| label_list | list | ['0', '180'] | class id对应的角度值 | -| cls_batch_num | int | 6 | 方向分类器预测的batch size | -| cls_thresh | float | 0.9 | 预测阈值,模型预测结果为180度,且得分大于该阈值时,认为最终预测结果为180度,需要翻转 | - - - -# 七、FAQ - -* 如果是使用paddle2.0之前版本的代码导出的`inference模型`,则其文件名为`model`与`params`,分别对应paddle2.0或者之后版本导出的`inference.pdmodel`与`inference.pdiparams`;不过目前PaddleOCR的release分支已经不支持paddle2.0之前版本导出的inference 模型,如果希望使用,需要使用develop分支(静态图分支)的代码与文档。 diff --git a/doc/doc_ch/algorithm_kie_layoutxlm.md b/doc/doc_ch/algorithm_kie_layoutxlm.md deleted file mode 100644 index 0cbcad2501..0000000000 --- a/doc/doc_ch/algorithm_kie_layoutxlm.md +++ /dev/null @@ -1,200 +0,0 @@ -# 关键信息抽取算法-LayoutXLM - -- [1. 算法简介](#1-算法简介) -- [2. 环境配置](#2-环境配置) -- [3. 模型训练、评估、预测](#3-模型训练评估预测) -- [4. 推理部署](#4-推理部署) - - [4.1 Python推理](#41-python推理) - - [4.2 C++推理部署](#42-推理部署) - - [4.3 Serving服务化部署](#43-serving服务化部署) - - [4.4 更多推理部署](#44-更多推理部署) -- [5. FAQ](#5-faq) -- [引用](#引用) - - - - -## 1. 算法简介 - - -论文信息: - -> [LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding](https://arxiv.org/abs/2104.08836) -> -> Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei -> -> 2021 - -在XFUND_zh数据集上,算法复现效果如下: - -|模型|骨干网络|任务|配置文件|hmean|下载链接| -| --- | --- |--|--- | --- | --- | -|LayoutXLM|LayoutXLM-base|SER |[ser_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml)|90.38%|[训练模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)/[推理模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh_infer.tar)| -|LayoutXLM|LayoutXLM-base|RE | [re_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml)|74.83%|[训练模型](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar)/[推理模型](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh_infer.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - - -## 3. 模型训练、评估、预测 - -请参考[关键信息抽取教程](./kie.md)。PaddleOCR对代码进行了模块化,训练不同的关键信息抽取模型只需要**更换配置文件**即可。 - - - -## 4. 推理部署 - - - -### 4.1 Python推理 - -- SER - -首先将训练得到的模型转换成inference model。LayoutXLM模型在XFUND_zh数据集上训练的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)),可以使用下面的命令进行转换。 - -``` bash -wget https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar -tar -xf ser_LayoutXLM_xfun_zh.tar -python3 tools/export_model.py -c configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./ser_LayoutXLM_xfun_zh Global.save_inference_dir=./inference/ser_layoutxlm_infer -``` - -LayoutXLM模型基于SER任务进行推理,可以执行如下命令: - -```bash -cd ppstructure -python3 kie/predict_kie_token_ser.py \ - --kie_algorithm=LayoutXLM \ - --ser_model_dir=../inference/ser_layoutxlm_infer \ - --image_dir=./docs/kie/input/zh_val_42.jpg \ - --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \ - --vis_font_path=../doc/fonts/simfang.ttf -``` - -SER可视化结果默认保存到`./output`文件夹里面,结果示例如下: - -
- -
- -- RE - -首先将训练得到的模型转换成inference model。LayoutXLM模型在XFUND_zh数据集上训练的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar)),可以使用下面的命令进行转换。 - -``` bash -wget https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar -tar -xf re_LayoutXLM_xfun_zh.tar -python3 tools/export_model.py -c configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./re_LayoutXLM_xfun_zh Global.save_inference_dir=./inference/ser_layoutxlm_infer -``` - -LayoutXLM模型基于RE任务进行推理,可以执行如下命令: - -```bash -cd ppstructure -python3 kie/predict_kie_token_ser_re.py \ - --kie_algorithm=LayoutXLM \ - --re_model_dir=../inference/re_layoutxlm_infer \ - --ser_model_dir=../inference/ser_layoutxlm_infer \ - --image_dir=./docs/kie/input/zh_val_42.jpg \ - --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \ - --vis_font_path=../doc/fonts/simfang.ttf -``` - -RE可视化结果默认保存到`./output`文件夹里面,结果示例如下: - -
- -
- - -### 4.2 C++推理部署 - -暂不支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - - -## 5. FAQ - -## 引用 - - -```bibtex -@article{DBLP:journals/corr/abs-2104-08836, - author = {Yiheng Xu and - Tengchao Lv and - Lei Cui and - Guoxin Wang and - Yijuan Lu and - Dinei Flor{\^{e}}ncio and - Cha Zhang and - Furu Wei}, - title = {LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich - Document Understanding}, - journal = {CoRR}, - volume = {abs/2104.08836}, - year = {2021}, - url = {https://arxiv.org/abs/2104.08836}, - eprinttype = {arXiv}, - eprint = {2104.08836}, - timestamp = {Thu, 14 Oct 2021 09:17:23 +0200}, - biburl = {https://dblp.org/rec/journals/corr/abs-2104-08836.bib}, - bibsource = {dblp computer science bibliography, https://dblp.org} -} - -@article{DBLP:journals/corr/abs-1912-13318, - author = {Yiheng Xu and - Minghao Li and - Lei Cui and - Shaohan Huang and - Furu Wei and - Ming Zhou}, - title = {LayoutLM: Pre-training of Text and Layout for Document Image Understanding}, - journal = {CoRR}, - volume = {abs/1912.13318}, - year = {2019}, - url = {http://arxiv.org/abs/1912.13318}, - eprinttype = {arXiv}, - eprint = {1912.13318}, - timestamp = {Mon, 01 Jun 2020 16:20:46 +0200}, - biburl = {https://dblp.org/rec/journals/corr/abs-1912-13318.bib}, - bibsource = {dblp computer science bibliography, https://dblp.org} -} - -@article{DBLP:journals/corr/abs-2012-14740, - author = {Yang Xu and - Yiheng Xu and - Tengchao Lv and - Lei Cui and - Furu Wei and - Guoxin Wang and - Yijuan Lu and - Dinei A. F. Flor{\^{e}}ncio and - Cha Zhang and - Wanxiang Che and - Min Zhang and - Lidong Zhou}, - title = {LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding}, - journal = {CoRR}, - volume = {abs/2012.14740}, - year = {2020}, - url = {https://arxiv.org/abs/2012.14740}, - eprinttype = {arXiv}, - eprint = {2012.14740}, - timestamp = {Tue, 27 Jul 2021 09:53:52 +0200}, - biburl = {https://dblp.org/rec/journals/corr/abs-2012-14740.bib}, - bibsource = {dblp computer science bibliography, https://dblp.org} -} -``` diff --git a/doc/doc_ch/algorithm_kie_sdmgr.md b/doc/doc_ch/algorithm_kie_sdmgr.md deleted file mode 100644 index 86b44f6d4a..0000000000 --- a/doc/doc_ch/algorithm_kie_sdmgr.md +++ /dev/null @@ -1,144 +0,0 @@ - -# 关键信息抽取算法-SDMGR - -- [1. 算法简介](#1-算法简介) -- [2. 环境配置](#2-环境配置) -- [3. 模型训练、评估、预测](#3-模型训练评估预测) - - [3.1 模型训练](#31-模型训练) - - [3.2 模型评估](#32-模型评估) - - [3.3 模型预测](#33-模型预测) -- [4. 推理部署](#4-推理部署) - - [4.1 Python推理](#41-python推理) - - [4.2 C++推理部署](#42-c推理部署) - - [4.3 Serving服务化部署](#43-serving服务化部署) - - [4.4 更多推理部署](#44-更多推理部署) -- [5. FAQ](#5-faq) -- [引用](#引用) - - - - -## 1. 算法简介 - - -论文信息: - -> [Spatial Dual-Modality Graph Reasoning for Key Information Extraction](https://arxiv.org/abs/2103.14470) -> -> Hongbin Sun and Zhanghui Kuang and Xiaoyu Yue and Chenhao Lin and Wayne Zhang -> -> 2021 - -在wildreceipt发票公开数据集上,算法复现效果如下: - -|模型|骨干网络|配置文件|hmean|下载链接| -| --- | --- | --- | --- | --- | -|SDMGR|VGG6|[configs/kie/sdmgr/kie_unet_sdmgr.yml](../../configs/kie/sdmgr/kie_unet_sdmgr.yml)|86.70%|[训练模型]( https://paddleocr.bj.bcebos.com/dygraph_v2.1/kie/kie_vgg16.tar)/[推理模型(coming soon)]()| - - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - - -## 3. 模型训练、评估、预测 - -SDMGR是一个关键信息提取算法,将每个检测到的文本区域分类为预定义的类别,如订单ID、发票号码,金额等。 - -训练和测试的数据采用wildreceipt数据集,通过如下指令下载数据集: - -```bash -wget https://paddleocr.bj.bcebos.com/ppstructure/dataset/wildreceipt.tar && tar xf wildreceipt.tar -``` - -创建数据集软链到PaddleOCR/train_data目录下: -``` -cd PaddleOCR/ && mkdir train_data && cd train_data - -ln -s ../../wildreceipt ./ -``` - - -### 3.1 模型训练 - -训练采用的配置文件是`configs/kie/sdmgr/kie_unet_sdmgr.yml`,配置文件中默认训练数据路径是`train_data/wildreceipt`,准备好数据后,可以通过如下指令执行训练: - -``` -python3 tools/train.py -c configs/kie/sdmgr/kie_unet_sdmgr.yml -o Global.save_model_dir=./output/kie/ -``` - -### 3.2 模型评估 - -执行下面的命令进行模型评估 - -```bash -python3 tools/eval.py -c configs/kie/sdmgr/kie_unet_sdmgr.yml -o Global.checkpoints=./output/kie/best_accuracy -``` - -输出信息示例如下所示。 - -```py -[2022/08/10 05:22:23] ppocr INFO: metric eval *************** -[2022/08/10 05:22:23] ppocr INFO: hmean:0.8670120239257812 -[2022/08/10 05:22:23] ppocr INFO: fps:10.18816520530961 -``` - -### 3.3 模型预测 - -执行下面的命令进行模型预测,预测的时候需要预先加载存储图片路径以及OCR信息的文本文件,使用`Global.infer_img`进行指定。 - -```bash -python3 tools/infer_kie.py -c configs/kie/kie_unet_sdmgr.yml -o Global.checkpoints=kie_vgg16/best_accuracy Global.infer_img=./train_data/wildreceipt/1.txt -``` - -执行预测后的结果保存在`./output/sdmgr_kie/predicts_kie.txt`文件中,可视化结果保存在`/output/sdmgr_kie/kie_results/`目录下。 - -可视化结果如下图所示: - -
- -
- - -## 4. 推理部署 - - -### 4.1 Python推理 - -暂不支持 - - -### 4.2 C++推理部署 - -暂不支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - - -## 5. FAQ - -## 引用 - - -```bibtex -@misc{sun2021spatial, - title={Spatial Dual-Modality Graph Reasoning for Key Information Extraction}, - author={Hongbin Sun and Zhanghui Kuang and Xiaoyu Yue and Chenhao Lin and Wayne Zhang}, - year={2021}, - eprint={2103.14470}, - archivePrefix={arXiv}, - primaryClass={cs.CV} -} -``` diff --git a/doc/doc_ch/algorithm_kie_vi_layoutxlm.md b/doc/doc_ch/algorithm_kie_vi_layoutxlm.md deleted file mode 100644 index 1ec778a899..0000000000 --- a/doc/doc_ch/algorithm_kie_vi_layoutxlm.md +++ /dev/null @@ -1,196 +0,0 @@ -# 关键信息抽取算法-VI-LayoutXLM - -- [1. 算法简介](#1-算法简介) -- [2. 环境配置](#2-环境配置) -- [3. 模型训练、评估、预测](#3-模型训练评估预测) -- [4. 推理部署](#4-推理部署) - - [4.1 Python推理](#41-python推理) - - [4.2 C++推理部署](#42-c推理部署) - - [4.3 Serving服务化部署](#43-serving服务化部署) - - [4.4 更多推理部署](#44-更多推理部署) -- [5. FAQ](#5-faq) -- [引用](#引用) - - - - -## 1. 算法简介 - -VI-LayoutXLM基于LayoutXLM进行改进,在下游任务训练过程中,去除视觉骨干网络模块,最终精度基本无损的情况下,模型推理速度进一步提升。 - -在XFUND_zh数据集上,算法复现效果如下: - -|模型|骨干网络|任务|配置文件|hmean|下载链接| -| --- | --- |---| --- | --- | --- | -|VI-LayoutXLM |VI-LayoutXLM-base | SER |[ser_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml)|93.19%|[训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)/[推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar)| -|VI-LayoutXLM |VI-LayoutXLM-base |RE | [re_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml)|83.92%|[训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar)/[推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_infer.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - - -## 3. 模型训练、评估、预测 - -请参考[关键信息抽取教程](./kie.md)。PaddleOCR对代码进行了模块化,训练不同的关键信息抽取模型只需要**更换配置文件**即可。 - - - -## 4. 推理部署 - - - -### 4.1 Python推理 - -- SER - -首先将训练得到的模型转换成inference model。以VI-LayoutXLM模型在XFUND_zh数据集上训练的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)),可以使用下面的命令进行转换。 - -``` bash -wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar -tar -xf ser_vi_layoutxlm_xfund_pretrained.tar -python3 tools/export_model.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./ser_vi_layoutxlm_xfund_pretrained/best_accuracy Global.save_inference_dir=./inference/ser_vi_layoutxlm_infer -``` - -VI-LayoutXLM模型基于SER任务进行推理,可以执行如下命令: - -```bash -cd ppstructure -python3 kie/predict_kie_token_ser.py \ - --kie_algorithm=LayoutXLM \ - --ser_model_dir=../inference/ser_vi_layoutxlm_infer \ - --image_dir=./docs/kie/input/zh_val_42.jpg \ - --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \ - --vis_font_path=../doc/fonts/simfang.ttf \ - --ocr_order_method="tb-yx" -``` - -SER可视化结果默认保存到`./output`文件夹里面,结果示例如下: - -
- -
- -- RE - -首先将训练得到的模型转换成inference model。以VI-LayoutXLM模型在XFUND_zh数据集上训练的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar)),可以使用下面的命令进行转换。 - -``` bash -wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar -tar -xf re_vi_layoutxlm_xfund_pretrained.tar -python3 tools/export_model.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./re_vi_layoutxlm_xfund_pretrained/best_accuracy Global.save_inference_dir=./inference/re_vi_layoutxlm_infer -``` - -VI-LayoutXLM模型基于RE任务进行推理,可以执行如下命令: - -```bash -cd ppstructure -python3 kie/predict_kie_token_ser_re.py \ - --kie_algorithm=LayoutXLM \ - --re_model_dir=../inference/re_vi_layoutxlm_infer \ - --ser_model_dir=../inference/ser_vi_layoutxlm_infer \ - --use_visual_backbone=False \ - --image_dir=./docs/kie/input/zh_val_42.jpg \ - --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \ - --vis_font_path=../doc/fonts/simfang.ttf \ - --ocr_order_method="tb-yx" -``` - -RE可视化结果默认保存到`./output`文件夹里面,结果示例如下: - -
- -
- - -### 4.2 C++推理部署 - -暂不支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - - -## 5. FAQ - -## 引用 - - -```bibtex -@article{DBLP:journals/corr/abs-2104-08836, - author = {Yiheng Xu and - Tengchao Lv and - Lei Cui and - Guoxin Wang and - Yijuan Lu and - Dinei Flor{\^{e}}ncio and - Cha Zhang and - Furu Wei}, - title = {LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich - Document Understanding}, - journal = {CoRR}, - volume = {abs/2104.08836}, - year = {2021}, - url = {https://arxiv.org/abs/2104.08836}, - eprinttype = {arXiv}, - eprint = {2104.08836}, - timestamp = {Thu, 14 Oct 2021 09:17:23 +0200}, - biburl = {https://dblp.org/rec/journals/corr/abs-2104-08836.bib}, - bibsource = {dblp computer science bibliography, https://dblp.org} -} - -@article{DBLP:journals/corr/abs-1912-13318, - author = {Yiheng Xu and - Minghao Li and - Lei Cui and - Shaohan Huang and - Furu Wei and - Ming Zhou}, - title = {LayoutLM: Pre-training of Text and Layout for Document Image Understanding}, - journal = {CoRR}, - volume = {abs/1912.13318}, - year = {2019}, - url = {http://arxiv.org/abs/1912.13318}, - eprinttype = {arXiv}, - eprint = {1912.13318}, - timestamp = {Mon, 01 Jun 2020 16:20:46 +0200}, - biburl = {https://dblp.org/rec/journals/corr/abs-1912-13318.bib}, - bibsource = {dblp computer science bibliography, https://dblp.org} -} - -@article{DBLP:journals/corr/abs-2012-14740, - author = {Yang Xu and - Yiheng Xu and - Tengchao Lv and - Lei Cui and - Furu Wei and - Guoxin Wang and - Yijuan Lu and - Dinei A. F. Flor{\^{e}}ncio and - Cha Zhang and - Wanxiang Che and - Min Zhang and - Lidong Zhou}, - title = {LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding}, - journal = {CoRR}, - volume = {abs/2012.14740}, - year = {2020}, - url = {https://arxiv.org/abs/2012.14740}, - eprinttype = {arXiv}, - eprint = {2012.14740}, - timestamp = {Tue, 27 Jul 2021 09:53:52 +0200}, - biburl = {https://dblp.org/rec/journals/corr/abs-2012-14740.bib}, - bibsource = {dblp computer science bibliography, https://dblp.org} -} -``` diff --git a/doc/doc_ch/algorithm_overview.md b/doc/doc_ch/algorithm_overview.md deleted file mode 100755 index 90b4b62013..0000000000 --- a/doc/doc_ch/algorithm_overview.md +++ /dev/null @@ -1,204 +0,0 @@ -# 前沿算法与模型 - -- [1. 两阶段OCR算法](#1) - - [1.1 文本检测算法](#11) - - [1.2 文本识别算法](#12) - - [1.3 文本超分辨率算法](#13) - - [1.4 公式识别算法](#14) -- [2. 端到端OCR算法](#2) -- [3. 表格识别算法](#3) -- [4. 关键信息抽取算法](#4) - -本文给出了PaddleOCR已支持的OCR算法列表,以及每个算法在**英文公开数据集**上的模型和指标,主要用于算法简介和算法性能对比,更多包括中文在内的其他数据集上的模型请参考[PP-OCRv3 系列模型下载](./models_list.md)。 - ->> -PaddleOCR将**持续新增**支持OCR领域前沿算法与模型,**欢迎广大开发者合作共建,贡献更多算法,合入有奖🎁!具体可查看[社区常规赛](https://github.com/PaddlePaddle/PaddleOCR/issues/4982)。** ->> -新增算法可参考教程:[使用PaddleOCR架构添加新算法](./add_new_algorithm.md) - - - - -## 1. 两阶段算法 - - - -### 1.1 文本检测算法 - -已支持的文本检测算法列表(戳链接获取使用教程): -- [x] [DB与DB++](./algorithm_det_db.md) -- [x] [EAST](./algorithm_det_east.md) -- [x] [SAST](./algorithm_det_sast.md) -- [x] [PSENet](./algorithm_det_psenet.md) -- [x] [FCENet](./algorithm_det_fcenet.md) -- [x] [DRRG](./algorithm_det_drrg.md) -- [x] [CT](./algorithm_det_ct.md) - -在ICDAR2015文本检测公开数据集上,算法效果如下: - -|模型|骨干网络|precision|recall|Hmean|下载链接| -| --- | --- | --- | --- | --- | --- | -|EAST|ResNet50_vd|88.71%|81.36%|84.88%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_east_v2.0_train.tar)| -|EAST|MobileNetV3|78.20%|79.10%|78.65%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_east_v2.0_train.tar)| -|DB|ResNet50_vd|86.41%|78.72%|82.38%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_db_v2.0_train.tar)| -|DB|MobileNetV3|77.29%|73.08%|75.12%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_db_v2.0_train.tar)| -|SAST|ResNet50_vd|91.39%|83.77%|87.42%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_icdar15_v2.0_train.tar)| -|PSE|ResNet50_vd|85.81%|79.53%|82.55%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_r50_vd_pse_v2.0_train.tar)| -|PSE|MobileNetV3|82.20%|70.48%|75.89%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_mv3_pse_v2.0_train.tar)| -|DB++|ResNet50|90.89%|82.66%|86.58%|[合成数据预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/ResNet50_dcn_asf_synthtext_pretrained.pdparams)/[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_r50_db%2B%2B_icdar15_train.tar)| - -在Total-text文本检测公开数据集上,算法效果如下: - -|模型|骨干网络|precision|recall|Hmean|下载链接| -| --- | --- | --- | --- | --- | --- | -|SAST|ResNet50_vd|89.63%|78.44%|83.66%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_totaltext_v2.0_train.tar)| -|CT|ResNet18_vd|88.68%|81.70%|85.05%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r18_ct_train.tar)| - -在CTW1500文本检测公开数据集上,算法效果如下: - -|模型|骨干网络|precision|recall|Hmean|下载链接| -| --- | --- | --- | --- | --- | --- | -|FCE|ResNet50_dcn|88.39%|82.18%|85.27%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/det_r50_dcn_fce_ctw_v2.0_train.tar)| -|DRRG|ResNet50_vd|89.92%|80.91%|85.18%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/det_r50_drrg_ctw_train.tar)| - -**说明:** SAST模型训练额外加入了icdar2013、icdar2017、COCO-Text、ArT等公开数据集进行调优。PaddleOCR用到的经过整理格式的英文公开数据集下载: -* [百度云地址](https://pan.baidu.com/s/12cPnZcVuV1zn5DOd4mqjVw) (提取码: 2bpi) -* [Google Drive下载地址](https://drive.google.com/drive/folders/1ll2-XEVyCQLpJjawLDiRlvo_i4BqHCJe?usp=sharing) - - - - -### 1.2 文本识别算法 - -已支持的文本识别算法列表(戳链接获取使用教程): -- [x] [CRNN](./algorithm_rec_crnn.md) -- [x] [Rosetta](./algorithm_rec_rosetta.md) -- [x] [STAR-Net](./algorithm_rec_starnet.md) -- [x] [RARE](./algorithm_rec_rare.md) -- [x] [SRN](./algorithm_rec_srn.md) -- [x] [NRTR](./algorithm_rec_nrtr.md) -- [x] [SAR](./algorithm_rec_sar.md) -- [x] [SEED](./algorithm_rec_seed.md) -- [x] [SVTR](./algorithm_rec_svtr.md) -- [x] [ViTSTR](./algorithm_rec_vitstr.md) -- [x] [ABINet](./algorithm_rec_abinet.md) -- [x] [VisionLAN](./algorithm_rec_visionlan.md) -- [x] [SPIN](./algorithm_rec_spin.md) -- [x] [RobustScanner](./algorithm_rec_robustscanner.md) -- [x] [RFL](./algorithm_rec_rfl.md) -- [x] [ParseQ](./algorithm_rec_parseq.md) -- [x] [CPPD](./algorithm_rec_cppd.md) -- [x] [SATRN](./algorithm_rec_satrn.md) - -参考[DTRB](https://arxiv.org/abs/1904.01906)[3]文字识别训练和评估流程,使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估,算法效果如下: - -|模型|骨干网络|Avg Accuracy|模型存储命名|下载链接| -|---|---|---|---|---| -|Rosetta|Resnet34_vd|79.11%|rec_r34_vd_none_none_ctc|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_none_ctc_v2.0_train.tar)| -|Rosetta|MobileNetV3|75.80%|rec_mv3_none_none_ctc|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_none_none_ctc_v2.0_train.tar)| -|CRNN|Resnet34_vd|81.04%|rec_r34_vd_none_bilstm_ctc|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar)| -|CRNN|MobileNetV3|77.95%|rec_mv3_none_bilstm_ctc|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_none_bilstm_ctc_v2.0_train.tar)| -|StarNet|Resnet34_vd|82.85%|rec_r34_vd_tps_bilstm_ctc|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_ctc_v2.0_train.tar)| -|StarNet|MobileNetV3|79.28%|rec_mv3_tps_bilstm_ctc|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_tps_bilstm_ctc_v2.0_train.tar)| -|RARE|Resnet34_vd|83.98%|rec_r34_vd_tps_bilstm_att |[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_att_v2.0_train.tar)| -|RARE|MobileNetV3|81.76%|rec_mv3_tps_bilstm_att |[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_tps_bilstm_att_v2.0_train.tar)| -|SRN|Resnet50_vd_fpn| 86.31% | rec_r50fpn_vd_none_srn | [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r50_vd_srn_train.tar) | -|NRTR|NRTR_MTB| 84.21% | rec_mtb_nrtr | [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mtb_nrtr_train.tar) | -|SAR|Resnet31| 87.20% | rec_r31_sar | [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_r31_sar_train.tar) | -|SEED|Aster_Resnet| 85.35% | rec_resnet_stn_bilstm_att | [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_resnet_stn_bilstm_att.tar) | -|SVTR|SVTR-Tiny| 89.25% | rec_svtr_tiny_none_ctc_en | [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar) | -|ViTSTR|ViTSTR| 79.82% | rec_vitstr_none_ce | [训练模型](https://paddleocr.bj.bcebos.com/rec_vitstr_none_ce_train.tar) | -|ABINet|Resnet45| 90.75% | rec_r45_abinet | [训练模型](https://paddleocr.bj.bcebos.com/rec_r45_abinet_train.tar) | -|VisionLAN|Resnet45| 90.30% | rec_r45_visionlan | [训练模型](https://paddleocr.bj.bcebos.com/VisionLAN/rec_r45_visionlan_train.tar) | -|SPIN|ResNet32| 90.00% | rec_r32_gaspin_bilstm_att | [训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_r32_gaspin_bilstm_att.tar) | -|RobustScanner|ResNet31| 87.77% | rec_r31_robustscanner | [训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_r31_robustscanner.tar)| -|RFL|ResNetRFL| 88.63% | rec_resnet_rfl_att | [训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl_att_train.tar) | -|ParseQ|VIT| 91.24% | rec_vit_parseq_synth | [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_synth.tgz) | -|CPPD|SVTR-Base| 93.8% | rec_svtrnet_cppd_base_en | [训练模型](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar) | -|SATRN|ShallowCNN| 88.05% | rec_satrn | [训练模型](https://pan.baidu.com/s/10J-Bsd881bimKaclKszlaQ?pwd=lk8a) | - - - -### 1.3 文本超分辨率算法 -已支持的文本超分辨率算法列表(戳链接获取使用教程): -- [x] [Text Gestalt](./algorithm_sr_gestalt.md) -- [x] [Text Telescope](./algorithm_sr_telescope.md) - -在TextZoom公开数据集上,算法效果如下: - -|模型|骨干网络|PSNR_Avg|SSIM_Avg|配置文件|下载链接| -|---|---|---|---|---|---| -|Text Gestalt|tsrn|19.28|0.6560| [configs/sr/sr_tsrn_transformer_strock.yml](../../configs/sr/sr_tsrn_transformer_strock.yml)|[训练模型](https://paddleocr.bj.bcebos.com/sr_tsrn_transformer_strock_train.tar)| -|Text Telescope|tbsrn|21.56|0.7411| [configs/sr/sr_telescope.yml](../../configs/sr/sr_telescope.yml)|[训练模型](https://paddleocr.bj.bcebos.com/contribution/sr_telescope_train.tar)| - - - -### 1.4 公式识别算法 - -已支持的公式识别算法列表(戳链接获取使用教程): -- [x] [CAN](./algorithm_rec_can.md) -- [x] [LaTeX-OCR](./algorithm_rec_latex_ocr.md) - -在CROHME手写公式数据集上,算法效果如下: - -|模型 |骨干网络|配置文件|ExpRate|下载链接| -| ----- | ----- | ----- | ----- | ----- | -|CAN|DenseNet|[rec_d28_can.yml](../../configs/rec/rec_d28_can.yml)|51.72%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_d28_can_train.tar)| - -在LaTeX-OCR印刷公式数据集上,算法效果如下: - -| 模型 | 骨干网络 |配置文件 | BLEU score | normed edit distance | ExpRate |下载链接| -|-----------|------------| ----- |:-----------:|:---------------------:|:---------:| ----- | -| LaTeX-OCR | Hybrid ViT |[rec_latex_ocr.yml](../../configs/rec/rec_latex_ocr.yml)| 0.8821 | 0.0823 | 40.01% |[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_latex_ocr_train.tar)| - - - - -## 2. 端到端算法 - -已支持的端到端OCR算法列表(戳链接获取使用教程): -- [x] [PGNet](./algorithm_e2e_pgnet.md) - - - -## 3. 表格识别算法 - -已支持的表格识别算法列表(戳链接获取使用教程): -- [x] [TableMaster](./algorithm_table_master.md) - -在PubTabNet表格识别公开数据集上,算法效果如下: - -|模型|骨干网络|配置文件|acc|下载链接| -|---|---|---|---|---| -|TableMaster|TableResNetExtra|[configs/table/table_master.yml](../../configs/table/table_master.yml)|77.47%|[训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/tablemaster/table_structure_tablemaster_train.tar) / [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/tablemaster/table_structure_tablemaster_infer.tar)| - - - -## 4. 关键信息抽取算法 - -已支持的关键信息抽取算法列表(戳链接获取使用教程): - -- [x] [VI-LayoutXLM](./algorithm_kie_vi_layoutxlm.md) -- [x] [LayoutLM](./algorithm_kie_layoutxlm.md) -- [x] [LayoutLMv2](./algorithm_kie_layoutxlm.md) -- [x] [LayoutXLM](./algorithm_kie_layoutxlm.md) -- [x] [SDMGR](././algorithm_kie_sdmgr.md) - -在wildreceipt发票公开数据集上,算法复现效果如下: - -|模型|骨干网络|配置文件|hmean|下载链接| -| --- | --- | --- | --- | --- | -|SDMGR|VGG6|[configs/kie/sdmgr/kie_unet_sdmgr.yml](../../configs/kie/sdmgr/kie_unet_sdmgr.yml)|86.70%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/kie/kie_vgg16.tar)| - - -在XFUND_zh公开数据集上,算法效果如下: - -|模型|骨干网络|任务|配置文件|hmean|下载链接| -| --- | --- | --- | --- | --- | --- | -|VI-LayoutXLM| VI-LayoutXLM-base | SER | [ser_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml)|**93.19%**|[训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)| -|LayoutXLM| LayoutXLM-base | SER | [ser_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml)|90.38%|[训练模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)| -|LayoutLM| LayoutLM-base | SER | [ser_layoutlm_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutlm_xfund_zh.yml)|77.31%|[训练模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLM_xfun_zh.tar)| -|LayoutLMv2| LayoutLMv2-base | SER | [ser_layoutlmv2_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutlmv2_xfund_zh.yml)|85.44%|[训练模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLMv2_xfun_zh.tar)| -|VI-LayoutXLM| VI-LayoutXLM-base | RE | [re_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml)|**83.92%**|[训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar)| -|LayoutXLM| LayoutXLM-base | RE | [re_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml)|74.83%|[训练模型](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar)| -|LayoutLMv2| LayoutLMv2-base | RE | [re_layoutlmv2_xfund_zh.yml](../../configs/kie/layoutlm_series/re_layoutlmv2_xfund_zh.yml)|67.77%|[训练模型](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutLMv2_xfun_zh.tar)| diff --git a/doc/doc_ch/algorithm_rec-satrn.md b/doc/doc_ch/algorithm_rec-satrn.md deleted file mode 100644 index f59b61c2d4..0000000000 --- a/doc/doc_ch/algorithm_rec-satrn.md +++ /dev/null @@ -1,112 +0,0 @@ -# SATRN - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention](https://arxiv.org/abs/1910.04396) -> Junyeop Lee, Sungrae Park, Jeonghun Baek, Seong Joon Oh, Seonghyeon Kim, Hwalsuk Lee -> CVPR, 2020 -参考[DTRB](https://arxiv.org/abs/1904.01906) 文字识别训练和评估流程,使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估,算法效果如下: - -|模型|骨干网络|Avg Accuracy|配置文件|下载链接| -|---|---|---|---|---| -|SATRN|ShallowCNN|88.05%|[configs/rec/rec_satrn.yml](../../configs/rec/rec_satrn.yml)|[训练模型](https://pan.baidu.com/s/10J-Bsd881bimKaclKszlaQ?pwd=lk8a)| - - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练不同的识别模型只需要**更换配置文件**即可。 - -- 训练 - -在完成数据准备后,便可以启动训练,训练命令如下: - -``` -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_satrn.yml -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c rec_satrn.yml -``` - -- 评估 - -``` -# GPU 评估, Global.pretrained_model 为待测权重 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_satrn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -- 预测: - -``` -# 预测使用的配置文件必须与训练一致 -python3 tools/infer_rec.py -c configs/rec/rec_satrn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将SATRN文本识别训练过程中保存的模型,转换成inference model。( [模型下载地址](https://pan.baidu.com/s/10J-Bsd881bimKaclKszlaQ?pwd=lk8a) ),可以使用如下命令进行转换: - -``` -python3 tools/export_model.py -c configs/rec/rec_satrn.yml -o Global.pretrained_model=./rec_satrn/best_accuracy Global.save_inference_dir=./inference/rec_satrn -``` - -SATRN文本识别模型推理,可以执行如下命令: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_satrn/" --rec_image_shape="3, 48, 48, 160" --rec_algorithm="SATRN" --rec_char_dict_path="ppocr/utils/dict90.txt" --max_text_length=30 --use_space_char=False -``` - - -### 4.2 C++推理 - -由于C++预处理后处理还未支持SATRN,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - -## 引用 - -```bibtex -@article{lee2019recognizing, - title={On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention}, - author={Junyeop Lee and Sungrae Park and Jeonghun Baek and Seong Joon Oh and Seonghyeon Kim and Hwalsuk Lee}, - year={2019}, - eprint={1910.04396}, - archivePrefix={arXiv}, - primaryClass={cs.CV} -} -``` diff --git a/doc/doc_ch/algorithm_rec_abinet.md b/doc/doc_ch/algorithm_rec_abinet.md deleted file mode 100644 index 47507c36c7..0000000000 --- a/doc/doc_ch/algorithm_rec_abinet.md +++ /dev/null @@ -1,155 +0,0 @@ -# 场景文本识别算法-ABINet - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [ABINet: Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://openaccess.thecvf.com/content/CVPR2021/papers/Fang_Read_Like_Humans_Autonomous_Bidirectional_and_Iterative_Language_Modeling_for_CVPR_2021_paper.pdf) -> Shancheng Fang and Hongtao Xie and Yuxin Wang and Zhendong Mao and Yongdong Zhang -> CVPR, 2021 - - - -`ABINet`使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估,算法复现效果如下: - -|模型|骨干网络|配置文件|Acc|下载链接| -| --- | --- | --- | --- | --- | -|ABINet|ResNet45|[rec_r45_abinet.yml](../../configs/rec/rec_r45_abinet.yml)|90.75%|[预训练、训练模型](https://paddleocr.bj.bcebos.com/rec_r45_abinet_train.tar)| - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - - -### 3.1 模型训练 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练`ABINet`识别模型时需要**更换配置文件**为`ABINet`的[配置文件](../../configs/rec/rec_r45_abinet.yml)。 - -#### 启动训练 - - -具体地,在完成数据准备后,便可以启动训练,训练命令如下: -```shell -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_r45_abinet.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r45_abinet.yml -``` - - -### 3.2 评估 - -可下载已训练完成的[模型文件](#model),使用如下命令进行评估: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r45_abinet.yml -o Global.pretrained_model=./rec_r45_abinet_train/best_accuracy -``` - - -### 3.3 预测 - -使用如下命令进行单张图片预测: -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/infer_rec.py -c configs/rec/rec_r45_abinet.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_r45_abinet_train/best_accuracy -# 预测文件夹下所有图像时,可修改infer_img为文件夹,如 Global.infer_img='./doc/imgs_words_en/'。 -``` - - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将训练得到best模型,转换成inference model。这里以训练完成的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/rec_r45_abinet_train.tar) ),可以使用如下命令进行转换: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/export_model.py -c configs/rec/rec_r45_abinet.yml -o Global.pretrained_model=./rec_r45_abinet_train/best_accuracy Global.save_inference_dir=./inference/rec_r45_abinet/ -``` -**注意:** -- 如果您是在自己的数据集上训练的模型,并且调整了字典文件,请注意修改配置文件中的`character_dict_path`是否是所需要的字典文件。 -- 如果您修改了训练时的输入大小,请修改`tools/export_model.py`文件中的对应ABINet的`infer_shape`。 - -转换成功后,在目录下有三个文件: -``` -/inference/rec_r45_abinet/ - ├── inference.pdiparams # 识别inference模型的参数文件 - ├── inference.pdiparams.info # 识别inference模型的参数信息,可忽略 - └── inference.pdmodel # 识别inference模型的program文件 -``` - -执行如下命令进行模型推理: - -```shell -python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_r45_abinet/' --rec_algorithm='ABINet' --rec_image_shape='3,32,128' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' -# 预测文件夹下所有图像时,可修改image_dir为文件夹,如 --image_dir='./doc/imgs_words_en/'。 -``` - -![](../imgs_words_en/word_10.png) - -执行命令后,上面图像的预测结果(识别的文本和得分)会打印到屏幕上,示例如下: -结果如下: -```shell -Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9999995231628418) -``` - -**注意**: - -- 训练上述模型采用的图像分辨率是[3,32,128],需要通过参数`rec_image_shape`设置为您训练时的识别图像形状。 -- 在推理时需要设置参数`rec_char_dict_path`指定字典,如果您修改了字典,请修改该参数为您的字典文件。 -- 如果您修改了预处理方法,需修改`tools/infer/predict_rec.py`中ABINet的预处理为您的预处理方法。 - - - -### 4.2 C++推理部署 - -由于C++预处理后处理还未支持ABINet,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - -1. MJSynth和SynthText两种数据集来自于[ABINet源repo](https://github.com/FangShancheng/ABINet) 。 -2. 我们使用ABINet作者提供的预训练模型进行finetune训练。 - -## 引用 - -```bibtex -@article{Fang2021ABINet, - title = {ABINet: Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition}, - author = {Shancheng Fang and Hongtao Xie and Yuxin Wang and Zhendong Mao and Yongdong Zhang}, - booktitle = {CVPR}, - year = {2021}, - url = {https://arxiv.org/abs/2103.06495}, - pages = {7098-7107} -} -``` diff --git a/doc/doc_ch/algorithm_rec_can.md b/doc/doc_ch/algorithm_rec_can.md deleted file mode 100644 index 13e868e505..0000000000 --- a/doc/doc_ch/algorithm_rec_can.md +++ /dev/null @@ -1,172 +0,0 @@ -# 手写数学公式识别算法-CAN - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition](https://arxiv.org/abs/2207.11463) -> Bohan Li, Ye Yuan, Dingkang Liang, Xiao Liu, Zhilong Ji, Jinfeng Bai, Wenyu Liu, Xiang Bai -> ECCV, 2022 - - - -`CAN`使用CROHME手写公式数据集进行训练,在对应测试集上的精度如下: - -|模型 |骨干网络|配置文件|ExpRate|下载链接| -| ----- | ----- | ----- | ----- | ----- | -|CAN|DenseNet|[rec_d28_can.yml](../../configs/rec/rec_d28_can.yml)|51.72%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_d28_can_train.tar)| - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - - -### 3.1 模型训练 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练`CAN`识别模型时需要**更换配置文件**为`CAN`的[配置文件](../../configs/rec/rec_d28_can.yml)。 - -#### 启动训练 - - -具体地,在完成数据准备后,便可以启动训练,训练命令如下: -```shell -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_d28_can.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_d28_can.yml -``` - -**注意:** -- 我们提供的数据集,即[`CROHME数据集`](https://paddleocr.bj.bcebos.com/dataset/CROHME.tar)将手写公式存储为黑底白字的格式,若您自行准备的数据集与之相反,即以白底黑字模式存储,请在训练时做出如下修改 -``` -python3 tools/train.py -c configs/rec/rec_d28_can.yml -o Train.dataset.transforms.GrayImageChannelFormat.inverse=False -``` -- 默认每训练1个epoch(1105次iteration)进行1次评估,若您更改训练的batch_size,或更换数据集,请在训练时作出如下修改 -``` -python3 tools/train.py -c configs/rec/rec_d28_can.yml -o Global.eval_batch_step=[0, {length_of_dataset//batch_size}] -``` - -# - -### 3.2 评估 - -可下载已训练完成的[模型文件](https://paddleocr.bj.bcebos.com/contribution/rec_d28_can_train.tar),使用如下命令进行评估: - -```shell -# 注意将pretrained_model的路径设置为本地路径。若使用自行训练保存的模型,请注意修改路径和文件名为{path/to/weights}/{model_name}。 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_d28_can.yml -o Global.pretrained_model=./rec_d28_can_train/best_accuracy.pdparams -``` - - -### 3.3 预测 - -使用如下命令进行单张图片预测: -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/infer_rec.py -c configs/rec/rec_d28_can.yml -o Architecture.Head.attdecoder.is_train=False Global.infer_img='./doc/datasets/crohme_demo/hme_00.jpg' Global.pretrained_model=./rec_d28_can_train/best_accuracy.pdparams - -# 预测文件夹下所有图像时,可修改infer_img为文件夹,如 Global.infer_img='./doc/datasets/crohme_demo/'。 -``` - - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将训练得到best模型,转换成inference model。这里以训练完成的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/contribution/rec_d28_can_train.tar) ),可以使用如下命令进行转换: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/export_model.py -c configs/rec/rec_d28_can.yml -o Global.pretrained_model=./rec_d28_can_train/best_accuracy.pdparams Global.save_inference_dir=./inference/rec_d28_can/ Architecture.Head.attdecoder.is_train=False - -# 目前的静态图模型默认的输出长度最大为36,如果您需要预测更长的序列,请在导出模型时指定其输出序列为合适的值,例如 Architecture.Head.max_text_length=72 -``` -**注意:** -- 如果您是在自己的数据集上训练的模型,并且调整了字典文件,请注意修改配置文件中的`character_dict_path`是否是所需要的字典文件。 - -转换成功后,在目录下有三个文件: -``` -/inference/rec_d28_can/ - ├── inference.pdiparams # 识别inference模型的参数文件 - ├── inference.pdiparams.info # 识别inference模型的参数信息,可忽略 - └── inference.pdmodel # 识别inference模型的program文件 -``` - -执行如下命令进行模型推理: - -```shell -python3 tools/infer/predict_rec.py --image_dir="./doc/datasets/crohme_demo/hme_00.jpg" --rec_algorithm="CAN" --rec_batch_num=1 --rec_model_dir="./inference/rec_d28_can/" --rec_char_dict_path="./ppocr/utils/dict/latex_symbol_dict.txt" - -# 预测文件夹下所有图像时,可修改image_dir为文件夹,如 --image_dir='./doc/datasets/crohme_demo/'。 - -# 如果您需要在白底黑字的图片上进行预测,请设置 --rec_image_inverse=False -``` - -![测试图片样例](../datasets/crohme_demo/hme_00.jpg) - -执行命令后,上面图像的预测结果(识别的文本)会打印到屏幕上,示例如下: -```shell -Predicts of ./doc/imgs_hme/hme_00.jpg:['x _ { k } x x _ { k } + y _ { k } y x _ { k }', []] -``` - - -**注意**: - -- 需要注意预测图像为**黑底白字**,即手写公式部分为白色,背景为黑色的图片。 -- 在推理时需要设置参数`rec_char_dict_path`指定字典,如果您修改了字典,请修改该参数为您的字典文件。 -- 如果您修改了预处理方法,需修改`tools/infer/predict_rec.py`中CAN的预处理为您的预处理方法。 - - - -### 4.2 C++推理部署 - -由于C++预处理后处理还未支持CAN,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - -1. CROHME数据集来自于[CAN源repo](https://github.com/LBH1024/CAN) 。 - -## 引用 - -```bibtex -@misc{https://doi.org/10.48550/arxiv.2207.11463, - doi = {10.48550/ARXIV.2207.11463}, - url = {https://arxiv.org/abs/2207.11463}, - author = {Li, Bohan and Yuan, Ye and Liang, Dingkang and Liu, Xiao and Ji, Zhilong and Bai, Jinfeng and Liu, Wenyu and Bai, Xiang}, - keywords = {Computer Vision and Pattern Recognition (cs.CV), Artificial Intelligence (cs.AI), FOS: Computer and information sciences, FOS: Computer and information sciences}, - title = {When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition}, - publisher = {arXiv}, - year = {2022}, - copyright = {arXiv.org perpetual, non-exclusive license} -} -``` diff --git a/doc/doc_ch/algorithm_rec_cppd.md b/doc/doc_ch/algorithm_rec_cppd.md deleted file mode 100644 index 4fde62e549..0000000000 --- a/doc/doc_ch/algorithm_rec_cppd.md +++ /dev/null @@ -1,198 +0,0 @@ -# 场景文本识别算法-CPPD - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) - - -## 1. 算法简介 - -论文信息: -> [Context Perception Parallel Decoder for Scene Text Recognition](https://arxiv.org/abs/2307.12270) -> Yongkun Du and Zhineng Chen and Caiyan Jia and Xiaoting Yin and Chenxia Li and Yuning Du and Yu-Gang Jiang - - -### CPPD算法简介 - -基于深度学习的场景文本识别模型通常是Encoder-Decoder结构,其中decoder可以分为两种:(1)CTC,(2)Attention-based。目前SOTA模型大多使用Attention-based的decoder,而attention-based可以分为AR和PD两种,一般来说,AR解码器识别精度优于PD,而PD解码速度快于AR,CPPD通过精心设计的CO和CC模块,达到了“AR的精度,PD的速度”的效果。 - - - -CPPD在场景文本识别公开数据集上的精度(%)和模型文件如下: - -* 英文训练集和测试集来自于[PARSeq](https://github.com/baudm/parseq)。 - - -| 模型 |IC13
857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg | 下载链接 | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:| -| CPPD Tiny | 97.1 | 94.4 | 96.6 | 86.6 | 88.5 | 90.3 | 92.25 | [英文](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_tiny_en_train.tar) | -| CPPD Base | 98.2 | 95.5 | 97.6 | 87.9 | 90.0 | 92.7 | 93.80 | [英文](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar)| -| CPPD Base 48*160 | 97.5 | 95.5 | 97.7 | 87.7 | 92.4 | 93.7 | 94.10 | [英文](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_48_160_en_train.tar) | - -* 英文合成数据集(MJ+ST)训练,英文Union14M-L benchmark测试结果[U14m](https://github.com/Mountchicken/Union14M/)。 - -| 模型 |Curve | Multi-
Oriented |Artistic |Contextless| Salient | Multi-
word | General | Avg | 下载链接 | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:| -| CPPD Tiny | 52.4 | 12.3 | 48.2 | 54.4 | 61.5 | 53.4 | 61.4 | 49.10 | 同上表 | -| CPPD Base | 65.5 | 18.6 | 56.0 | 61.9 | 71.0 | 57.5 | 65.8 | 56.63 | 同上表 | -| CPPD Base 48*160 | 71.9 | 22.1 | 60.5 | 67.9 | 78.3 | 63.9 | 67.1 | 61.69 | 同上表 | - -* Union14M-L 训练集From scratch训练,英文测试结果。 - -| 模型 |IC13
857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg | 下载链接 | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:| -| CPPD Base 32*128 | 98.5 | 97.7 | 99.2 | 90.3 | 94.6 | 98.3 | 96.42 | Coming soon | - -| 模型 |Curve | Multi-
Oriented |Artistic |Contextless| Salient | Multi-
word | General | Avg | 下载链接 | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:| -| CPPD Base 32*128 | 83.0 | 71.2 | 75.1 | 80.9 | 79.4 | 82.6 | 83.7 | 79.41 | Coming soon | - -* 加载合成数据集预训练模型,Union14M-L 训练集微调训练,英文测试结果。 - -| 模型 |IC13
857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg | 下载链接 | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:| -| CPPD Base 32*128 | 98.7 | 98.5 | 99.4 | 91.7 | 96.7 | 99.7 | 97.44 | [英文](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_u14m_train.tar) | - -| 模型 |Curve | Multi-
Oriented |Artistic |Contextless| Salient | Multi-
word | General | Avg | 下载链接 | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:| -| CPPD Base 32*128 | 87.5 | 70.7 | 78.2 | 82.9 | 85.5 | 85.4 | 84.3 | 82.08 | 同上表 | - -* 中文训练集和测试集来自于[Chinese Benckmark](https://github.com/FudanVI/benchmarking-chinese-text-recognition)。 - -| 模型 | Scene | Web | Document | Handwriting | Avg | 下载链接 | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:| -| CPPD Base | 74.4 | 76.1 | 98.6 | 55.3 | 76.10 | [中文](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_ch_train.tar) | -| CPPD Base + STN | 78.4 | 79.3 | 98.9 | 57.6 | 78.55 | [中文](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_stn_ch_train.tar) | - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - - -### 3.1 模型训练 - -#### 数据集准备 - -[英文数据集下载](https://github.com/baudm/parseq) - -[Union14M-L 下载](https://github.com/Mountchicken/Union14M) - -[中文数据集下载](https://github.com/fudanvi/benchmarking-chinese-text-recognition#download) - -#### 启动训练 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练`CPPD`识别模型时需要**更换配置文件**为`CPPD`的[配置文件](../../configs/rec/rec_svtrnet_cppd_base_en.yml)。 - -具体地,在完成数据准备后,便可以启动训练,训练命令如下: -```shell -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_svtrnet_cppd_base_en.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_svtrnet_cppd_base_en.yml -``` - - -### 3.2 评估 - -可下载`CPPD`提供的模型文件和配置文件:[下载地址](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar) ,以`CPPD-B`为例,使用如下命令进行评估: - -```shell -# 下载包含CPPD-B的模型文件和配置文件的tar压缩包并解压 -wget https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar && tar xf rec_svtr_cppd_base_en_train.tar -# 注意将pretrained_model的路径设置为本地路径。 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c ./rec_svtr_cppd_base_en_train/rec_svtrnet_cppd_base_en.yml -o Global.pretrained_model=./rec_svtr_cppd_base_en_train/best_model -``` - - -### 3.3 预测 - -使用如下命令进行单张图片预测: -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/infer_rec.py -c ./rec_svtr_cppd_base_en_train/rec_svtrnet_cppd_base_en.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_svtr_cppd_base_en_train/best_model -# 预测文件夹下所有图像时,可修改infer_img为文件夹,如 Global.infer_img='./doc/imgs_words_en/'。 -``` - - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将训练得到best模型,转换成inference model。下面以基于`CPPD-B`,在英文数据集训练的模型为例([模型和配置文件下载地址](https://paddleocr.bj.bcebos.com/CPPD/rec_svtr_cppd_base_en_train.tar),可以使用如下命令进行转换: - -**注意:** -- 如果您是在自己的数据集上训练的模型,并且调整了字典文件,请注意修改配置文件中的`character_dict_path`是否为所正确的字典文件。 - -执行如下命令进行模型导出和推理: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -# export model -# en -python3 tools/export_model.py -c configs/rec/rec_svtrnet_cppd_base_en.yml -o Global.pretrained_model=./rec_svtr_cppd_base_en_train/best_model.pdparams Global.save_inference_dir=./rec_svtr_cppd_base_en_infer -# ch -python3 tools/export_model.py -c configs/rec/rec_svtrnet_cppd_base_ch.yml -o Global.pretrained_model=./rec_svtr_cppd_base_ch_train/best_model.pdparams Global.save_inference_dir=./rec_svtr_cppd_base_ch_infer - -# speed test -# docker image https://hub.docker.com/r/paddlepaddle/paddle/tags/: sudo docker pull paddlepaddle/paddle:2.4.2-gpu-cuda11.2-cudnn8.2-trt8.0 -# install auto_log: pip install https://paddleocr.bj.bcebos.com/libs/auto_log-1.2.0-py3-none-any.whl -# en -python3 tools/infer/predict_rec.py --image_dir='../iiik' --rec_model_dir='./rec_svtr_cppd_base_en_infer/' --rec_algorithm='CPPD' --rec_image_shape='3,32,100' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' --warmup=True --benchmark=True --rec_batch_num=1 --use_tensorrt=True -# ch -python3 tools/infer/predict_rec.py --image_dir='../iiik' --rec_model_dir='./rec_svtr_cppd_base_ch_infer/' --rec_algorithm='CPPDPadding' --rec_image_shape='3,32,256' --warmup=True --benchmark=True --rec_batch_num=1 --use_tensorrt=True -# stn_ch -python3 tools/infer/predict_rec.py --image_dir='../iiik' --rec_model_dir='./rec_svtr_cppd_base_stn_ch_infer/' --rec_algorithm='CPPD' --rec_image_shape='3,64,256' --warmup=True --benchmark=True --rec_batch_num=1 --use_tensorrt=True -``` - -导出成功后,在目录下有三个文件: -``` -/inference/rec_svtr_cppd_base_en_infer/ - ├── inference.pdiparams # 识别inference模型的参数文件 - ├── inference.pdiparams.info # 识别inference模型的参数信息,可忽略 - └── inference.pdmodel # 识别inference模型的program文件 -``` - - - - -### 4.2 C++推理部署 - -由于C++预处理后处理还未支持CPPD,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - - -## 引用 - -```bibtex -@article{Du2023CPPD, - title = {Context Perception Parallel Decoder for Scene Text Recognition}, - author = {Du, Yongkun and Chen, Zhineng and Jia, Caiyan and Yin, Xiaoting and Li, Chenxia and Du, Yuning and Jiang, Yu-Gang}, - booktitle = {Arxiv}, - year = {2023}, - url = {https://arxiv.org/abs/2307.12270} -} -``` diff --git a/doc/doc_ch/algorithm_rec_crnn.md b/doc/doc_ch/algorithm_rec_crnn.md deleted file mode 100644 index 490485ec10..0000000000 --- a/doc/doc_ch/algorithm_rec_crnn.md +++ /dev/null @@ -1,140 +0,0 @@ -# CRNN - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://arxiv.org/abs/1507.05717) - -> Baoguang Shi, Xiang Bai, Cong Yao - -> IEEE, 2015 - -参考[DTRB](https://arxiv.org/abs/1904.01906) 文字识别训练和评估流程,使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估,算法效果如下: - -|模型|骨干网络|Avg Accuracy|配置文件|下载链接| -|---|---|---|---|---| -|CRNN|Resnet34_vd|81.04%|[configs/rec/rec_r34_vd_none_bilstm_ctc.yml](../../configs/rec/rec_r34_vd_none_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar)| -|CRNN|MobileNetV3|77.95%|[configs/rec/rec_mv3_none_bilstm_ctc.yml](../../configs/rec/rec_mv3_none_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_none_bilstm_ctc_v2.0_train.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练不同的识别模型只需要**更换配置文件**即可。 - -- 训练 - -在完成数据准备后,便可以启动训练,训练命令如下: - -``` -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c rec_r34_vd_none_bilstm_ctc.yml - -``` - -- 评估 - -``` -# GPU 评估, Global.pretrained_model 为待测权重 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -- 预测: - -``` -# 预测使用的配置文件必须与训练一致 -python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. 推理部署 - - -### 4.1 Python推理 - -首先将 CRNN 文本识别训练过程中保存的模型,转换成inference model。以基于Resnet34_vd骨干网络,使用MJSynth和SynthText两个英文文本识别合成数据集训练的[模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar) 为例,可以使用如下命令进行转换: -```shell -python3 tools/export_model.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model=./rec_r34_vd_none_bilstm_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_crnn -``` -CRNN 文本识别模型推理,可以执行如下命令: - -```shell -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./inference/rec_crnn/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" -``` - -![](../imgs_words_en/word_336.png) - -执行命令后,上面图像的识别结果如下: - -```bash -Predicts of ./doc/imgs_words_en/word_336.png:('super', 0.9999073) -``` - -**注意**:由于上述模型是参考[DTRB](https://arxiv.org/abs/1904.01906)文本识别训练和评估流程,与超轻量级中文识别模型训练有两方面不同: - -- 训练时采用的图像分辨率不同,训练上述模型采用的图像分辨率是[3,32,100],而中文模型训练时,为了保证长文本的识别效果,训练时采用的图像分辨率是[3, 32, 320]。预测推理程序默认的形状参数是训练中文采用的图像分辨率,即[3, 32, 320]。因此,这里推理上述英文模型时,需要通过参数rec_image_shape设置识别图像的形状。 - -- 字符列表,DTRB论文中实验只是针对26个小写英文本母和10个数字进行实验,总共36个字符。所有大小字符都转成了小写字符,不在上面列表的字符都忽略,认为是空格。因此这里没有输入字符字典,而是通过如下命令生成字典.因此在推理时需要设置参数rec_char_dict_path,指定为英文字典"./ppocr/utils/ic15_dict.txt"。 - -``` -self.character_str = "0123456789abcdefghijklmnopqrstuvwxyz" -dict_character = list(self.character_str) -``` - - - -### 4.2 C++推理 - -准备好推理模型后,参考[cpp infer](../../deploy/cpp_infer/)教程进行操作即可。 - - -### 4.3 Serving服务化部署 - -准备好推理模型后,参考[pdserving](../../deploy/pdserving/)教程进行Serving服务化部署,包括Python Serving和C++ Serving两种模式。 - - -### 4.4 更多推理部署 - -CRNN模型还支持以下推理部署方式: - -- Paddle2ONNX推理:准备好推理模型后,参考[paddle2onnx](../../deploy/paddle2onnx/)教程操作。 - - -## 5. FAQ - - -## 引用 - -```bibtex -@ARTICLE{7801919, - author={Shi, Baoguang and Bai, Xiang and Yao, Cong}, - journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, - title={An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition}, - year={2017}, - volume={39}, - number={11}, - pages={2298-2304}, - doi={10.1109/TPAMI.2016.2646371}} -``` diff --git a/doc/doc_ch/algorithm_rec_nrtr.md b/doc/doc_ch/algorithm_rec_nrtr.md deleted file mode 100644 index c619ac1dbc..0000000000 --- a/doc/doc_ch/algorithm_rec_nrtr.md +++ /dev/null @@ -1,290 +0,0 @@ -# 场景文本识别算法-NRTR - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) -- [6. 发行公告](#6) - - -## 1. 算法简介 - -论文信息: -> [NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition](https://arxiv.org/abs/1806.00926) -> Fenfen Sheng and Zhineng Chen and Bo Xu -> ICDAR, 2019 - - - -`NRTR`使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估,算法复现效果如下: - -|模型|骨干网络|配置文件|Acc|下载链接| -| --- | --- | --- | --- | --- | -|NRTR|MTB|[rec_mtb_nrtr.yml](../../configs/rec/rec_mtb_nrtr.yml)|84.21%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mtb_nrtr_train.tar)| - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - - -### 3.1 模型训练 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练`NRTR`识别模型时需要**更换配置文件**为`NRTR`的[配置文件](../../configs/rec/rec_mtb_nrtr.yml)。 - -#### 启动训练 - - -具体地,在完成数据准备后,便可以启动训练,训练命令如下: -```shell -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_mtb_nrtr.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_mtb_nrtr.yml -``` - - -### 3.2 评估 - -可下载已训练完成的[模型文件](#model),使用如下命令进行评估: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_mtb_nrtr.yml -o Global.pretrained_model=./rec_mtb_nrtr_train/best_accuracy -``` - - -### 3.3 预测 - -使用如下命令进行单张图片预测: -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/infer_rec.py -c configs/rec/rec_mtb_nrtr.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_mtb_nrtr_train/best_accuracy -# 预测文件夹下所有图像时,可修改infer_img为文件夹,如 Global.infer_img='./doc/imgs_words_en/'。 -``` - - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将训练得到best模型,转换成inference model。这里以训练完成的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mtb_nrtr_train.tar) ),可以使用如下命令进行转换: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/export_model.py -c configs/rec/rec_mtb_nrtr.yml -o Global.pretrained_model=./rec_mtb_nrtr_train/best_accuracy Global.save_inference_dir=./inference/rec_mtb_nrtr/ -``` -**注意:** -- 如果您是在自己的数据集上训练的模型,并且调整了字典文件,请注意修改配置文件中的`character_dict_path`是否是所需要的字典文件。 -- 如果您修改了训练时的输入大小,请修改`tools/export_model.py`文件中的对应NRTR的`infer_shape`。 - -转换成功后,在目录下有三个文件: -``` -/inference/rec_mtb_nrtr/ - ├── inference.pdiparams # 识别inference模型的参数文件 - ├── inference.pdiparams.info # 识别inference模型的参数信息,可忽略 - └── inference.pdmodel # 识别inference模型的program文件 -``` - -执行如下命令进行模型推理: - -```shell -python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_mtb_nrtr/' --rec_algorithm='NRTR' --rec_image_shape='1,32,100' --rec_char_dict_path='./ppocr/utils/EN_symbol_dict.txt' -# 预测文件夹下所有图像时,可修改image_dir为文件夹,如 --image_dir='./doc/imgs_words_en/'。 -``` - -![](../imgs_words_en/word_10.png) - -执行命令后,上面图像的预测结果(识别的文本和得分)会打印到屏幕上,示例如下: -结果如下: -```shell -Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9465042352676392) -``` - -**注意**: - -- 训练上述模型采用的图像分辨率是[1,32,100],需要通过参数`rec_image_shape`设置为您训练时的识别图像形状。 -- 在推理时需要设置参数`rec_char_dict_path`指定字典,如果您修改了字典,请修改该参数为您的字典文件。 -- 如果您修改了预处理方法,需修改`tools/infer/predict_rec.py`中NRTR的预处理为您的预处理方法。 - - - -### 4.2 C++推理部署 - -由于C++预处理后处理还未支持NRTR,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - -1. `NRTR`论文中使用Beam搜索进行解码字符,但是速度较慢,这里默认未使用Beam搜索,以贪婪搜索进行解码字符。 - - -## 6. 发行公告 - -1. release/2.6更新NRTR代码结构,新版NRTR可加载旧版(release/2.5及之前)模型参数,使用下面示例代码将旧版模型参数转换为新版模型参数: - -```python - - params = paddle.load('path/' + '.pdparams') # 旧版本参数 - state_dict = model.state_dict() # 新版模型参数 - new_state_dict = {} - - for k1, v1 in state_dict.items(): - - k = k1 - if 'encoder' in k and 'self_attn' in k and 'qkv' in k and 'weight' in k: - - k_para = k[:13] + 'layers.' + k[13:] - q = params[k_para.replace('qkv', 'conv1')].transpose((1, 0, 2, 3)) - k = params[k_para.replace('qkv', 'conv2')].transpose((1, 0, 2, 3)) - v = params[k_para.replace('qkv', 'conv3')].transpose((1, 0, 2, 3)) - - new_state_dict[k1] = np.concatenate([q[:, :, 0, 0], k[:, :, 0, 0], v[:, :, 0, 0]], -1) - - elif 'encoder' in k and 'self_attn' in k and 'qkv' in k and 'bias' in k: - - k_para = k[:13] + 'layers.' + k[13:] - q = params[k_para.replace('qkv', 'conv1')] - k = params[k_para.replace('qkv', 'conv2')] - v = params[k_para.replace('qkv', 'conv3')] - - new_state_dict[k1] = np.concatenate([q, k, v], -1) - - elif 'encoder' in k and 'self_attn' in k and 'out_proj' in k: - - k_para = k[:13] + 'layers.' + k[13:] - new_state_dict[k1] = params[k_para] - - elif 'encoder' in k and 'norm3' in k: - k_para = k[:13] + 'layers.' + k[13:] - new_state_dict[k1] = params[k_para.replace('norm3', 'norm2')] - - elif 'encoder' in k and 'norm1' in k: - k_para = k[:13] + 'layers.' + k[13:] - new_state_dict[k1] = params[k_para] - - - elif 'decoder' in k and 'self_attn' in k and 'qkv' in k and 'weight' in k: - k_para = k[:13] + 'layers.' + k[13:] - q = params[k_para.replace('qkv', 'conv1')].transpose((1, 0, 2, 3)) - k = params[k_para.replace('qkv', 'conv2')].transpose((1, 0, 2, 3)) - v = params[k_para.replace('qkv', 'conv3')].transpose((1, 0, 2, 3)) - new_state_dict[k1] = np.concatenate([q[:, :, 0, 0], k[:, :, 0, 0], v[:, :, 0, 0]], -1) - - elif 'decoder' in k and 'self_attn' in k and 'qkv' in k and 'bias' in k: - k_para = k[:13] + 'layers.' + k[13:] - q = params[k_para.replace('qkv', 'conv1')] - k = params[k_para.replace('qkv', 'conv2')] - v = params[k_para.replace('qkv', 'conv3')] - new_state_dict[k1] = np.concatenate([q, k, v], -1) - - elif 'decoder' in k and 'self_attn' in k and 'out_proj' in k: - - k_para = k[:13] + 'layers.' + k[13:] - new_state_dict[k1] = params[k_para] - - elif 'decoder' in k and 'cross_attn' in k and 'q' in k and 'weight' in k: - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('cross_attn', 'multihead_attn') - q = params[k_para.replace('q', 'conv1')].transpose((1, 0, 2, 3)) - new_state_dict[k1] = q[:, :, 0, 0] - - elif 'decoder' in k and 'cross_attn' in k and 'q' in k and 'bias' in k: - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('cross_attn', 'multihead_attn') - q = params[k_para.replace('q', 'conv1')] - new_state_dict[k1] = q - - elif 'decoder' in k and 'cross_attn' in k and 'kv' in k and 'weight' in k: - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('cross_attn', 'multihead_attn') - k = params[k_para.replace('kv', 'conv2')].transpose((1, 0, 2, 3)) - v = params[k_para.replace('kv', 'conv3')].transpose((1, 0, 2, 3)) - new_state_dict[k1] = np.concatenate([k[:, :, 0, 0], v[:, :, 0, 0]], -1) - - elif 'decoder' in k and 'cross_attn' in k and 'kv' in k and 'bias' in k: - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('cross_attn', 'multihead_attn') - k = params[k_para.replace('kv', 'conv2')] - v = params[k_para.replace('kv', 'conv3')] - new_state_dict[k1] = np.concatenate([k, v], -1) - - elif 'decoder' in k and 'cross_attn' in k and 'out_proj' in k: - - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('cross_attn', 'multihead_attn') - new_state_dict[k1] = params[k_para] - elif 'decoder' in k and 'norm' in k: - k_para = k[:13] + 'layers.' + k[13:] - new_state_dict[k1] = params[k_para] - elif 'mlp' in k and 'weight' in k: - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('fc', 'conv') - k_para = k_para.replace('mlp.', '') - w = params[k_para].transpose((1, 0, 2, 3)) - new_state_dict[k1] = w[:, :, 0, 0] - elif 'mlp' in k and 'bias' in k: - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('fc', 'conv') - k_para = k_para.replace('mlp.', '') - w = params[k_para] - new_state_dict[k1] = w - - else: - new_state_dict[k1] = params[k1] - - if list(new_state_dict[k1].shape) != list(v1.shape): - print(k1) - - - for k, v1 in state_dict.items(): - if k not in new_state_dict.keys(): - print(1, k) - elif list(new_state_dict[k].shape) != list(v1.shape): - print(2, k) - - - - model.set_state_dict(new_state_dict) - paddle.save(model.state_dict(), 'nrtrnew_from_old_params.pdparams') - -``` - -2. 新版相比与旧版,代码结构简洁,推理速度有所提高。 - - -## 引用 - -```bibtex -@article{Sheng2019NRTR, - title = {NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition}, - author = {Fenfen Sheng and Zhineng Chen and Bo Xu}, - booktitle = {ICDAR}, - year = {2019}, - url = {http://arxiv.org/abs/1806.00926}, - pages = {781-786} -} -``` diff --git a/doc/doc_ch/algorithm_rec_parseq.md b/doc/doc_ch/algorithm_rec_parseq.md deleted file mode 100644 index 7853a9df8d..0000000000 --- a/doc/doc_ch/algorithm_rec_parseq.md +++ /dev/null @@ -1,124 +0,0 @@ -# ParseQ - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://arxiv.org/abs/2207.06966) -> Darwin Bautista, Rowel Atienza -> ECCV, 2021 - -原论文分别使用真实文本识别数据集(Real)和合成文本识别数据集(Synth)进行训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估。 -其中: -- 真实文本识别数据集(Real)包含COCO-Text, RCTW17, Uber-Text, ArT, LSVT, MLT19, ReCTS, TextOCR, OpenVINO数据集 -- 合成文本识别数据集(Synth)包含MJSynth和SynthText数据集 - -在不同数据集上训练的算法的复现效果如下: - -|数据集|模型|骨干网络|配置文件|Acc|下载链接| -| --- | --- | --- | --- | --- | --- | -|Synth|ParseQ|VIT|[rec_vit_parseq.yml](../../configs/rec/rec_vit_parseq.yml)|91.24%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_synth.tgz)| -|Real|ParseQ|VIT|[rec_vit_parseq.yml](../../configs/rec/rec_vit_parseq.yml)|94.74%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_real.tgz)| -||||||| - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本识别教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练不同的识别模型只需要**更换配置文件**即可。 - -训练 - -具体地,在完成数据准备后,便可以启动训练,训练命令如下: - -``` -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_vit_parseq.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_vit_parseq.yml -``` - -评估 - -``` -# GPU 评估, Global.pretrained_model 为待测权重 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_vit_parseq.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -预测: - -``` -# 预测使用的配置文件必须与训练一致 -python3 tools/infer_rec.py -c configs/rec/rec_vit_parseq.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将ParseQ文本识别训练过程中保存的模型,转换成inference model。( [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_real.tgz) ),可以使用如下命令进行转换: - -``` -python3 tools/export_model.py -c configs/rec/rec_vit_parseq.yml -o Global.pretrained_model=./rec_vit_parseq_real/best_accuracy Global.save_inference_dir=./inference/rec_parseq -``` - -ParseQ文本识别模型推理,可以执行如下命令: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_parseq/" --rec_image_shape="3, 32, 128" --rec_algorithm="ParseQ" --rec_char_dict_path="ppocr/utils/dict/parseq_dict.txt" --max_text_length=25 --use_space_char=False -``` - - -### 4.2 C++推理 - -由于C++预处理后处理还未支持ParseQ,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@InProceedings{bautista2022parseq, - title={Scene Text Recognition with Permuted Autoregressive Sequence Models}, - author={Bautista, Darwin and Atienza, Rowel}, - booktitle={European Conference on Computer Vision}, - pages={178--196}, - month={10}, - year={2022}, - publisher={Springer Nature Switzerland}, - address={Cham}, - doi={10.1007/978-3-031-19815-1_11}, - url={https://doi.org/10.1007/978-3-031-19815-1_11} -} -``` diff --git a/doc/doc_ch/algorithm_rec_rare.md b/doc/doc_ch/algorithm_rec_rare.md deleted file mode 100644 index 9476c2e69a..0000000000 --- a/doc/doc_ch/algorithm_rec_rare.md +++ /dev/null @@ -1,121 +0,0 @@ -# RARE - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [Robust Scene Text Recognition with Automatic Rectification](https://arxiv.org/abs/1603.03915v2) -> Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, Xiang Bai∗ -> CVPR, 2016 - -使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估,算法复现效果如下: - -|模型|骨干网络|配置文件|Avg Accuracy|下载链接| -| --- | --- | --- | --- | --- | -|RARE|Resnet34_vd|[configs/rec/rec_r34_vd_tps_bilstm_att.yml](../../configs/rec/rec_r34_vd_tps_bilstm_att.yml)|83.60%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_att_v2.0_train.tar)| -|RARE|MobileNetV3|[configs/rec/rec_mv3_tps_bilstm_att.yml](../../configs/rec/rec_mv3_tps_bilstm_att.yml)|82.50%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_tps_bilstm_att_v2.0_train.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - -## 3. 模型训练、评估、预测 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练不同的识别模型只需要**更换配置文件**即可。以基于Resnet34_vd骨干网络为例: - - -### 3.1 训练 - -``` -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -``` - - -### 3.2 评估 - -``` -# GPU评估, Global.pretrained_model为待评估模型 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - - -### 3.3 预测 - -``` -python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将RARE文本识别训练过程中保存的模型,转换成inference model。以基于Resnet34_vd骨干网络,在MJSynth和SynthText两个文字识别数据集训练得到的模型为例( [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_att_v2.0_train.tar) ),可以使用如下命令进行转换: - -```shell -python3 tools/export_model.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -o Global.pretrained_model=./rec_r34_vd_tps_bilstm_att_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_rare -``` - -RARE文本识别模型推理,可以执行如下命令: - -```shell -python3 tools/infer/predict_rec.py --image_dir="doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_rare/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" -``` -推理结果如下所示: - -![](../../doc/imgs_words/en/word_1.png) - -``` -Predicts of doc/imgs_words/en/word_1.png:('joint ', 0.9999969601631165) -``` - - - -### 4.2 C++推理 - -暂不支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -RARE模型还支持以下推理部署方式: - -- Paddle2ONNX推理:准备好推理模型后,参考[paddle2onnx](../../deploy/paddle2onnx/)教程操作。 - - -## 5. FAQ - - -## 引用 - -```bibtex -@inproceedings{2016Robust, - title={Robust Scene Text Recognition with Automatic Rectification}, - author={ Shi, B. and Wang, X. and Lyu, P. and Cong, Y. and Xiang, B. }, - booktitle={2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, - year={2016}, -} -``` diff --git a/doc/doc_ch/algorithm_rec_rfl.md b/doc/doc_ch/algorithm_rec_rfl.md deleted file mode 100644 index 547fab3458..0000000000 --- a/doc/doc_ch/algorithm_rec_rfl.md +++ /dev/null @@ -1,161 +0,0 @@ -# 场景文本识别算法-RFL - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [Reciprocal Feature Learning via Explicit and Implicit Tasks in Scene Text Recognition](https://arxiv.org/abs/2105.06229.pdf) -> Hui Jiang, Yunlu Xu, Zhanzhan Cheng, Shiliang Pu, Yi Niu, Wenqi Ren, Fei Wu, and Wenming Tan -> ICDAR, 2021 - - - -`RFL`使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估,算法复现效果如下: - -|模型|骨干网络|配置文件|Acc|下载链接| -| --- | --- | --- | --- | --- | -|RFL-CNT|ResNetRFL|[rec_resnet_rfl_visual.yml](../../configs/rec/rec_resnet_rfl_visual.yml)|93.40%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl_visual_train.tar)| -|RFL-Att|ResNetRFL|[rec_resnet_rfl_att.yml](../../configs/rec/rec_resnet_rfl_att.yml)|88.63%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl_att_train.tar)| - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - - -### 3.1 模型训练 - -PaddleOCR对代码进行了模块化,训练`RFL`识别模型时需要**更换配置文件**为`RFL`的[配置文件](../../configs/rec/rec_resnet_rfl_att.yml)。 - -#### 启动训练 - - -具体地,在完成数据准备后,便可以启动训练,训练命令如下: -```shell -#step1:训练CNT分支 -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_resnet_rfl_visual.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_resnet_rfl_visual.yml - -#step2:联合训练CNT和Att分支,注意将pretrained_model的路径设置为本地路径。 -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model=./output/rec/rec_resnet_rfl_visual/best_accuracy - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model=./output/rec/rec_resnet_rfl_visual/best_accuracy -``` - - -### 3.2 评估 - -可下载已训练完成的[模型文件](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl.tar),使用如下命令进行评估: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model=./output/rec/rec_resnet_rfl_att/best_accuracy -``` - - -### 3.3 预测 - -使用如下命令进行单张图片预测: -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/infer_rec.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./output/rec/rec_resnet_rfl_att/best_accuracy -# 预测文件夹下所有图像时,可修改infer_img为文件夹,如 Global.infer_img='./doc/imgs_words_en/'。 -``` - - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将训练得到best模型,转换成inference model。这里以训练完成的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl.tar) ),可以使用如下命令进行转换: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/export_model.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model=./output/rec/rec_resnet_rfl_att/best_accuracy Global.save_inference_dir=./inference/rec_resnet_rfl_att/ -``` -**注意:** -- 如果您是在自己的数据集上训练的模型,并且调整了字典文件,请注意修改配置文件中的`character_dict_path`是否是所需要的字典文件。 -- 如果您修改了训练时的输入大小,请修改`tools/export_model.py`文件中的对应RFL的`infer_shape`。 - -转换成功后,在目录下有三个文件: -``` -/inference/rec_resnet_rfl_att/ - ├── inference.pdiparams # 识别inference模型的参数文件 - ├── inference.pdiparams.info # 识别inference模型的参数信息,可忽略 - └── inference.pdmodel # 识别inference模型的program文件 -``` - -执行如下命令进行模型推理: - -```shell -python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_resnet_rfl_att/' --rec_algorithm='RFL' --rec_image_shape='1,32,100' -# 预测文件夹下所有图像时,可修改image_dir为文件夹,如 --image_dir='./doc/imgs_words_en/'。 -``` - -![](../imgs_words_en/word_10.png) - -执行命令后,上面图像的预测结果(识别的文本和得分)会打印到屏幕上,示例如下: -结果如下: -```shell -Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9999927282333374) -``` - -**注意**: - -- 训练上述模型采用的图像分辨率是[1,32,100],需要通过参数`rec_image_shape`设置为您训练时的识别图像形状。 -- 在推理时需要设置参数`rec_char_dict_path`指定字典,如果您修改了字典,请修改该参数为您的字典文件。 -- 如果您修改了预处理方法,需修改`tools/infer/predict_rec.py`中RFL的预处理为您的预处理方法。 - - - -### 4.2 C++推理部署 - -由于C++预处理后处理还未支持RFL,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@article{2021Reciprocal, - title = {Reciprocal Feature Learning via Explicit and Implicit Tasks in Scene Text Recognition}, - author = {Jiang, H. and Xu, Y. and Cheng, Z. and Pu, S. and Niu, Y. and Ren, W. and Wu, F. and Tan, W. }, - booktitle = {ICDAR}, - year = {2021}, - url = {https://arxiv.org/abs/2105.06229} -} -``` diff --git a/doc/doc_ch/algorithm_rec_robustscanner.md b/doc/doc_ch/algorithm_rec_robustscanner.md deleted file mode 100644 index a1ab3baf03..0000000000 --- a/doc/doc_ch/algorithm_rec_robustscanner.md +++ /dev/null @@ -1,113 +0,0 @@ -# RobustScanner - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition](https://arxiv.org/pdf/2007.07542.pdf) -> Xiaoyu Yue, Zhanghui Kuang, Chenhao Lin, Hongbin Sun, Wayne -Zhang -> ECCV, 2020 - -使用MJSynth和SynthText两个合成文字识别数据集训练,在IIIT, SVT, IC13, IC15, SVTP, CUTE数据集上进行评估,算法复现效果如下: - -|模型|骨干网络|配置文件|Acc|下载链接| -| --- | --- | --- | --- | --- | -|RobustScanner|ResNet31|[rec_r31_robustscanner.yml](../../configs/rec/rec_r31_robustscanner.yml)|87.77%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_r31_robustscanner.tar)| - -注:除了使用MJSynth和SynthText两个文字识别数据集外,还加入了[SynthAdd](https://pan.baidu.com/share/init?surl=uV0LtoNmcxbO-0YA7Ch4dg)数据(提取码:627x),和部分真实数据,具体数据细节可以参考论文。 - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本识别教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练不同的识别模型只需要**更换配置文件**即可。 - -训练 - -具体地,在完成数据准备后,便可以启动训练,训练命令如下: - -``` -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_r31_robustscanner.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r31_robustscanner.yml -``` - -评估 - -``` -# GPU 评估, Global.pretrained_model 为待测权重 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r31_robustscanner.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -预测: - -``` -# 预测使用的配置文件必须与训练一致 -python3 tools/infer_rec.py -c configs/rec/rec_r31_robustscanner.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将RobustScanner文本识别训练过程中保存的模型,转换成inference model。可以使用如下命令进行转换: - -``` -python3 tools/export_model.py -c configs/rec/rec_r31_robustscanner.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/rec_r31_robustscanner -``` -RobustScanner文本识别模型推理,可以执行如下命令: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_r31_robustscanner/" --rec_image_shape="3, 48, 48, 160" --rec_algorithm="RobustScanner" --rec_char_dict_path="ppocr/utils/dict90.txt" --use_space_char=False -``` - - -### 4.2 C++推理 - -由于C++预处理后处理还未支持RobustScanner,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@article{2020RobustScanner, - title={RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition}, - author={Xiaoyu Yue and Zhanghui Kuang and Chenhao Lin and Hongbin Sun and Wayne Zhang}, - journal={ECCV2020}, - year={2020}, -} -``` diff --git a/doc/doc_ch/algorithm_rec_rosetta.md b/doc/doc_ch/algorithm_rec_rosetta.md deleted file mode 100644 index c1784cac8c..0000000000 --- a/doc/doc_ch/algorithm_rec_rosetta.md +++ /dev/null @@ -1,123 +0,0 @@ -# Rosetta - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [Rosetta: Large Scale System for Text Detection and Recognition in Images](https://arxiv.org/abs/1910.05085) -> Borisyuk F , Gordo A , V Sivakumar -> KDD, 2018 - -使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估, 算法复现效果如下: - -|模型|骨干网络|配置文件|Avg Accuracy|下载链接| -| --- | --- | --- | --- | --- | -|Rosetta|Resnet34_vd|[configs/rec/rec_r34_vd_none_none_ctc.yml](../../configs/rec/rec_r34_vd_none_none_ctc.yml)|79.11%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_none_ctc_v2.0_train.tar)| -|Rosetta|MobileNetV3|[configs/rec/rec_mv3_none_none_ctc.yml](../../configs/rec/rec_mv3_none_none_ctc.yml)|75.80%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_none_none_ctc_v2.0_train.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练不同的识别模型只需要**更换配置文件**即可。 以基于Resnet34_vd骨干网络为例: - - -### 3.1 训练 - -``` -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -``` - - -### 3.2 评估 - -``` -# GPU评估, Global.pretrained_model为待评估模型 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - - -### 3.3 预测 - -``` -python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将Rosetta文本识别训练过程中保存的模型,转换成inference model。以基于Resnet34_vd骨干网络,在MJSynth和SynthText两个文字识别数据集训练得到的模型为例( [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_none_ctc_v2.0_train.tar) ),可以使用如下命令进行转换: - -```shell -python3 tools/export_model.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -o Global.pretrained_model=./rec_r34_vd_none_none_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_rosetta -``` - -Rosetta文本识别模型推理,可以执行如下命令: - -```shell -python3 tools/infer/predict_rec.py --image_dir="doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_rosetta/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" -``` - -推理结果如下所示: - -![](../../doc/imgs_words/en/word_1.png) - -``` -Predicts of doc/imgs_words/en/word_1.png:('joint', 0.9999982714653015) -``` - - -### 4.2 C++推理 - -暂不支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -Rosetta模型还支持以下推理部署方式: - -- Paddle2ONNX推理:准备好推理模型后,参考[paddle2onnx](../../deploy/paddle2onnx/)教程操作。 - - -## 5. FAQ - - -## 引用 - -```bibtex -@inproceedings{2018Rosetta, - title={Rosetta: Large Scale System for Text Detection and Recognition in Images}, - author={ Borisyuk, Fedor and Gordo, Albert and Sivakumar, Viswanath }, - booktitle={the 24th ACM SIGKDD International Conference}, - year={2018}, -} -``` diff --git a/doc/doc_ch/algorithm_rec_sar.md b/doc/doc_ch/algorithm_rec_sar.md deleted file mode 100644 index cfb1de2539..0000000000 --- a/doc/doc_ch/algorithm_rec_sar.md +++ /dev/null @@ -1,114 +0,0 @@ -# SAR - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition](https://arxiv.org/abs/1811.00751) -> Hui Li, Peng Wang, Chunhua Shen, Guyu Zhang -> AAAI, 2019 - -使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估,算法复现效果如下: - -|模型|骨干网络|配置文件|Acc|下载链接| -| --- | --- | --- | --- | --- | -|SAR|ResNet31|[rec_r31_sar.yml](../../configs/rec/rec_r31_sar.yml)|87.20%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_r31_sar_train.tar)| - -注:除了使用MJSynth和SynthText两个文字识别数据集外,还加入了[SynthAdd](https://pan.baidu.com/share/init?surl=uV0LtoNmcxbO-0YA7Ch4dg)数据(提取码:627x),和部分真实数据,具体数据细节可以参考论文。 - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本识别教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练不同的识别模型只需要**更换配置文件**即可。 - -训练 - -具体地,在完成数据准备后,便可以启动训练,训练命令如下: - -``` -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_r31_sar.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r31_sar.yml -``` - -评估 - -``` -# GPU 评估, Global.pretrained_model 为待测权重 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r31_sar.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -预测: - -``` -# 预测使用的配置文件必须与训练一致 -python3 tools/infer_rec.py -c configs/rec/rec_r31_sar.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将SAR文本识别训练过程中保存的模型,转换成inference model。( [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_r31_sar_train.tar) ),可以使用如下命令进行转换: - -``` -python3 tools/export_model.py -c configs/rec/rec_r31_sar.yml -o Global.pretrained_model=./rec_r31_sar_train/best_accuracy Global.save_inference_dir=./inference/rec_sar -``` - -SAR文本识别模型推理,可以执行如下命令: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_sar/" --rec_image_shape="3, 48, 48, 160" --rec_algorithm="SAR" --rec_char_dict_path="ppocr/utils/dict90.txt" --max_text_length=30 --use_space_char=False -``` - - -### 4.2 C++推理 - -由于C++预处理后处理还未支持SAR,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@article{Li2019ShowAA, - title={Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition}, - author={Hui Li and Peng Wang and Chunhua Shen and Guyu Zhang}, - journal={ArXiv}, - year={2019}, - volume={abs/1811.00751} -} -``` diff --git a/doc/doc_ch/algorithm_rec_seed.md b/doc/doc_ch/algorithm_rec_seed.md deleted file mode 100644 index 6d59c9fee2..0000000000 --- a/doc/doc_ch/algorithm_rec_seed.md +++ /dev/null @@ -1,113 +0,0 @@ -# SEED - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://arxiv.org/pdf/2005.10977.pdf) - -> Qiao, Zhi and Zhou, Yu and Yang, Dongbao and Zhou, Yucan and Wang, Weiping - -> CVPR, 2020 - -参考[DTRB](https://arxiv.org/abs/1904.01906) 文字识别训练和评估流程,使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估,算法效果如下: - -|模型|骨干网络|Avg Accuracy|配置文件|下载链接| -|---|---|---|---|---| -|SEED|Aster_Resnet| 85.20% | [configs/rec/rec_resnet_stn_bilstm_att.yml](../../configs/rec/rec_resnet_stn_bilstm_att.yml) | [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_resnet_stn_bilstm_att.tar) | - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练不同的识别模型只需要**更换配置文件**即可。 - -- 训练 - -SEED模型需要额外加载FastText训练好的[语言模型](https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.bin.gz) ,并且安装 fasttext 依赖: - -``` -python3 -m pip install fasttext==0.9.1 -``` - -然后,在完成数据准备后,便可以启动训练,训练命令如下: - -``` -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_resnet_stn_bilstm_att.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c rec_resnet_stn_bilstm_att.yml - -``` - -- 评估 - -``` -# GPU 评估, Global.pretrained_model 为待测权重 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_resnet_stn_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -- 预测: - -``` -# 预测使用的配置文件必须与训练一致 -python3 tools/infer_rec.py -c configs/rec/rec_resnet_stn_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. 推理部署 - - -### 4.1 Python推理 - -coming soon - - - -### 4.2 C++推理 - -coming soon - - -### 4.3 Serving服务化部署 - -coming soon - - -### 4.4 更多推理部署 - -coming soon - - -## 5. FAQ - - -## 引用 - -```bibtex -@inproceedings{qiao2020seed, - title={Seed: Semantics enhanced encoder-decoder framework for scene text recognition}, - author={Qiao, Zhi and Zhou, Yu and Yang, Dongbao and Zhou, Yucan and Wang, Weiping}, - booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, - pages={13528--13537}, - year={2020} -} -``` diff --git a/doc/doc_ch/algorithm_rec_spin.md b/doc/doc_ch/algorithm_rec_spin.md deleted file mode 100644 index 2b9c04abca..0000000000 --- a/doc/doc_ch/algorithm_rec_spin.md +++ /dev/null @@ -1,112 +0,0 @@ -# SPIN: Structure-Preserving Inner Offset Network for Scene Text Recognition - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [SPIN: Structure-Preserving Inner Offset Network for Scene Text Recognition](https://arxiv.org/abs/2005.13117) -> Chengwei Zhang, Yunlu Xu, Zhanzhan Cheng, Shiliang Pu, Yi Niu, Fei Wu, Futai Zou -> AAAI, 2020 - -SPIN收录于AAAI2020。主要用于OCR识别任务。在任意形状文本识别中,矫正网络是一种较为常见的前置处理模块,但诸如RARE\ASTER\ESIR等只考虑了空间变换,并没有考虑色度变换。本文提出了一种结构Structure-Preserving Inner Offset Network (SPIN),可以在色彩空间上进行变换。该模块是可微分的,可以加入到任意识别器中。 -使用MJSynth和SynthText两个合成文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估,算法复现效果如下: - -|模型|骨干网络|配置文件|Acc|下载链接| -| --- | --- | --- | --- | --- | -|SPIN|ResNet32|[rec_r32_gaspin_bilstm_att.yml](../../configs/rec/rec_r32_gaspin_bilstm_att.yml)|90.00%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_r32_gaspin_bilstm_att.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本识别教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练不同的识别模型只需要**更换配置文件**即可。 - -训练 - -具体地,在完成数据准备后,便可以启动训练,训练命令如下: - -``` -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -``` - -评估 - -``` -# GPU 评估, Global.pretrained_model 为待测权重 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -预测: - -``` -# 预测使用的配置文件必须与训练一致 -python3 tools/infer_rec.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将SPIN文本识别训练过程中保存的模型,转换成inference model。可以使用如下命令进行转换: - -``` -python3 tools/export_model.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/rec_r32_gaspin_bilstm_att -``` -SPIN文本识别模型推理,可以执行如下命令: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_r32_gaspin_bilstm_att/" --rec_image_shape="3, 32, 100" --rec_algorithm="SPIN" --rec_char_dict_path="/ppocr/utils/dict/spin_dict.txt" --use_space_char=Falsee -``` - - -### 4.2 C++推理 - -由于C++预处理后处理还未支持SPIN,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@article{2020SPIN, - title={SPIN: Structure-Preserving Inner Offset Network for Scene Text Recognition}, - author={Chengwei Zhang and Yunlu Xu and Zhanzhan Cheng and Shiliang Pu and Yi Niu and Fei Wu and Futai Zou}, - journal={AAAI2020}, - year={2020}, -} -``` diff --git a/doc/doc_ch/algorithm_rec_srn.md b/doc/doc_ch/algorithm_rec_srn.md deleted file mode 100644 index dd61a388c7..0000000000 --- a/doc/doc_ch/algorithm_rec_srn.md +++ /dev/null @@ -1,113 +0,0 @@ -# SRN - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [Towards Accurate Scene Text Recognition with Semantic Reasoning Networks](https://arxiv.org/abs/2003.12294#) -> Deli Yu, Xuan Li, Chengquan Zhang, Junyu Han, Jingtuo Liu, Errui Ding -> CVPR,2020 - -使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估,算法复现效果如下: - -|模型|骨干网络|配置文件|Acc|下载链接| -| --- | --- | --- | --- | --- | -|SRN|Resnet50_vd_fpn|[rec_r50_fpn_srn.yml](../../configs/rec/rec_r50_fpn_srn.yml)|86.31%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r50_vd_srn_train.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本识别教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练不同的识别模型只需要**更换配置文件**即可。 - -训练 - -具体地,在完成数据准备后,便可以启动训练,训练命令如下: - -``` -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_r50_fpn_srn.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r50_fpn_srn.yml -``` - -评估 - -``` -# GPU 评估, Global.pretrained_model 为待测权重 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r50_fpn_srn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -预测: - -``` -# 预测使用的配置文件必须与训练一致 -python3 tools/infer_rec.py -c configs/rec/rec_r50_fpn_srn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将SRN文本识别训练过程中保存的模型,转换成inference model。( [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r50_vd_srn_train.tar) ),可以使用如下命令进行转换: - -``` -python3 tools/export_model.py -c configs/rec/rec_r50_fpn_srn.yml -o Global.pretrained_model=./rec_r50_vd_srn_train/best_accuracy Global.save_inference_dir=./inference/rec_srn -``` - -SRN文本识别模型推理,可以执行如下命令: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_srn/" --rec_image_shape="1,64,256" --rec_algorithm="SRN" --rec_char_dict_path=./ppocr/utils/ic15_dict.txt --use_space_char=False -``` - - -### 4.2 C++推理 - -由于C++预处理后处理还未支持SRN,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@article{Yu2020TowardsAS, - title={Towards Accurate Scene Text Recognition With Semantic Reasoning Networks}, - author={Deli Yu and Xuan Li and Chengquan Zhang and Junyu Han and Jingtuo Liu and Errui Ding}, - journal={2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, - year={2020}, - pages={12110-12119} -} -``` diff --git a/doc/doc_ch/algorithm_rec_starnet.md b/doc/doc_ch/algorithm_rec_starnet.md deleted file mode 100644 index c1ca761a07..0000000000 --- a/doc/doc_ch/algorithm_rec_starnet.md +++ /dev/null @@ -1,139 +0,0 @@ -# STAR-Net - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [STAR-Net: a spatial attention residue network for scene text recognition.](http://www.bmva.org/bmvc/2016/papers/paper043/paper043.pdf) - -> Wei Liu, Chaofeng Chen, Kwan-Yee K. Wong, Zhizhong Su and Junyu Han. - -> BMVC, pages 43.1-43.13, 2016 - -参考[DTRB](https://arxiv.org/abs/1904.01906) 文字识别训练和评估流程,使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估,算法效果如下: - -|模型|骨干网络|Avg Accuracy|配置文件|下载链接| -|---|---|---|---|---| -|StarNet|Resnet34_vd|84.44%|[configs/rec/rec_r34_vd_tps_bilstm_ctc.yml](../../configs/rec/rec_r34_vd_tps_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_ctc_v2.0_train.tar)| -|StarNet|MobileNetV3|81.42%|[configs/rec/rec_mv3_tps_bilstm_ctc.yml](../../configs/rec/rec_mv3_tps_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_tps_bilstm_ctc_v2.0_train.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练不同的识别模型只需要**更换配置文件**即可。 - -- 训练 - -在完成数据准备后,便可以启动训练,训练命令如下: - -``` -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c rec_r34_vd_tps_bilstm_ctc.yml - -``` - -- 评估 - -``` -# GPU 评估, Global.pretrained_model 为待测权重 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -- 预测: - -``` -# 预测使用的配置文件必须与训练一致 -python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. 推理部署 - - -### 4.1 Python推理 - -首先将 STAR-Net 文本识别训练过程中保存的模型,转换成inference model。以基于Resnet34_vd骨干网络,使用MJSynth和SynthText两个英文文本识别合成数据集训练的[模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar) 为例,可以使用如下命令进行转换: -```shell -python3 tools/export_model.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model=./rec_r34_vd_tps_bilstm_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_starnet -``` -STAR-Net 文本识别模型推理,可以执行如下命令: - -```shell -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./inference/rec_starnet/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" -``` - -![](../imgs_words_en/word_336.png) - -执行命令后,上面图像的识别结果如下: - -```bash -Predicts of ./doc/imgs_words_en/word_336.png:('super', 0.9999073) -``` - -**注意**:由于上述模型是参考[DTRB](https://arxiv.org/abs/1904.01906)文本识别训练和评估流程,与超轻量级中文识别模型训练有两方面不同: - -- 训练时采用的图像分辨率不同,训练上述模型采用的图像分辨率是[3,32,100],而中文模型训练时,为了保证长文本的识别效果,训练时采用的图像分辨率是[3, 32, 320]。预测推理程序默认的形状参数是训练中文采用的图像分辨率,即[3, 32, 320]。因此,这里推理上述英文模型时,需要通过参数rec_image_shape设置识别图像的形状。 - -- 字符列表,DTRB论文中实验只是针对26个小写英文本母和10个数字进行实验,总共36个字符。所有大小字符都转成了小写字符,不在上面列表的字符都忽略,认为是空格。因此这里没有输入字符字典,而是通过如下命令生成字典.因此在推理时需要设置参数rec_char_dict_path,指定为英文字典"./ppocr/utils/ic15_dict.txt"。 - -``` -self.character_str = "0123456789abcdefghijklmnopqrstuvwxyz" -dict_character = list(self.character_str) -``` - - - -### 4.2 C++推理 - -准备好推理模型后,参考[cpp infer](../../deploy/cpp_infer/)教程进行操作即可。 - - -### 4.3 Serving服务化部署 - -准备好推理模型后,参考[pdserving](../../deploy/pdserving/)教程进行Serving服务化部署,包括Python Serving和C++ Serving两种模式。 - - -### 4.4 更多推理部署 - -STAR-Net模型还支持以下推理部署方式: - -- Paddle2ONNX推理:准备好推理模型后,参考[paddle2onnx](../../deploy/paddle2onnx/)教程操作。 - - -## 5. FAQ - - -## 引用 - -```bibtex -@inproceedings{liu2016star, - title={STAR-Net: a spatial attention residue network for scene text recognition.}, - author={Liu, Wei and Chen, Chaofeng and Wong, Kwan-Yee K and Su, Zhizhong and Han, Junyu}, - booktitle={BMVC}, - volume={2}, - pages={7}, - year={2016} -} -``` diff --git a/doc/doc_ch/algorithm_rec_svtr.md b/doc/doc_ch/algorithm_rec_svtr.md deleted file mode 100644 index 34881c1146..0000000000 --- a/doc/doc_ch/algorithm_rec_svtr.md +++ /dev/null @@ -1,191 +0,0 @@ -# 场景文本识别算法-SVTR - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [SVTR: Scene Text Recognition with a Single Visual Model](https://arxiv.org/abs/2205.00159) -> Yongkun Du and Zhineng Chen and Caiyan Jia and Xiaoting Yin and Tianlun Zheng and Chenxia Li and Yuning Du and Yu-Gang Jiang -> IJCAI, 2022 - -场景文本识别旨在将自然图像中的文本转录为数字字符序列,从而传达对场景理解至关重要的高级语义。这项任务由于文本变形、字体、遮挡、杂乱背景等方面的变化具有一定的挑战性。先前的方法为提高识别精度做出了许多工作。然而文本识别器除了准确度外,还因为实际需求需要考虑推理速度等因素。 - -### SVTR算法简介 - -主流的场景文本识别模型通常包含两个模块:用于特征提取的视觉模型和用于文本转录的序列模型。这种架构虽然准确,但复杂且效率较低,限制了在实际场景中的应用。SVTR提出了一种用于场景文本识别的单视觉模型,该模型在patch-wise image tokenization框架内,完全摒弃了序列建模,在精度具有竞争力的前提下,模型参数量更少,速度更快,主要有以下几点贡献: -1. 首次发现单视觉模型可以达到与视觉语言模型相媲美甚至更高的准确率,并且其具有效率高和适应多语言的优点,在实际应用中很有前景。 -2. SVTR从字符组件的角度出发,逐渐的合并字符组件,自下而上地完成字符的识别。 -3. SVTR引入了局部和全局Mixing,分别用于提取字符组件特征和字符间依赖关系,与多尺度的特征一起,形成多粒度特征描述。 - - - -SVTR在场景文本识别公开数据集上的精度(%)和模型文件如下: - -* 中文数据集来自于[Chinese Benckmark](https://arxiv.org/abs/2112.15093) ,SVTR的中文训练评估策略遵循该论文。 - -| 模型 |IC13
857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg_6 |IC15
2077 |IC13
1015 |IC03
867|IC03
860|Avg_10 | Chinese
scene_test| 下载链接 | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:|:-----:|:-----:|:---------------------------------------------:|:-----:|:-----:| -| SVTR Tiny | 96.85 | 91.34 | 94.53 | 83.99 | 85.43 | 89.24 | 90.87 | 80.55 | 95.37 | 95.27 | 95.70 | 90.13 | 67.90 | [英文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar) / [中文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_ch_train.tar) | -| SVTR Small | 95.92 | 93.04 | 95.03 | 84.70 | 87.91 | 92.01 | 91.63 | 82.72 | 94.88 | 96.08 | 96.28 | 91.02 | 69.00 | [英文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_small_none_ctc_en_train.tar) / [中文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_small_none_ctc_ch_train.tar) | -| SVTR Base | 97.08 | 91.50 | 96.03 | 85.20 | 89.92 | 91.67 | 92.33 | 83.73 | 95.66 | 95.62 | 95.81 | 91.61 | 71.40 | [英文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_base_none_ctc_en_train.tar) / - | -| SVTR Large | 97.20 | 91.65 | 96.30 | 86.58 | 88.37 | 95.14 | 92.82 | 84.54 | 96.35 | 96.54 | 96.74 | 92.24 | 72.10 | [英文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_large_none_ctc_en_train.tar) / [中文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_large_none_ctc_ch_train.tar) | - - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - - -### 3.1 模型训练 - -#### 数据集准备 - -[英文数据集下载](https://github.com/clovaai/deep-text-recognition-benchmark#download-lmdb-dataset-for-traininig-and-evaluation-from-here) -[中文数据集下载](https://github.com/fudanvi/benchmarking-chinese-text-recognition#download) - -#### 启动训练 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练`SVTR`识别模型时需要**更换配置文件**为`SVTR`的[配置文件](../../configs/rec/rec_svtrnet.yml)。 - -具体地,在完成数据准备后,便可以启动训练,训练命令如下: -```shell -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_svtrnet.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_svtrnet.yml -``` - - -### 3.2 评估 - -可下载`SVTR`提供的模型文件和配置文件:[下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar) ,以`SVTR-T`为例,使用如下命令进行评估: - -```shell -# 下载包含SVTR-T的模型文件和配置文件的tar压缩包并解压 -wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar && tar xf rec_svtr_tiny_none_ctc_en_train.tar -# 注意将pretrained_model的路径设置为本地路径。 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c ./rec_svtr_tiny_none_ctc_en_train/rec_svtr_tiny_6local_6global_stn_en.yml -o Global.pretrained_model=./rec_svtr_tiny_none_ctc_en_train/best_accuracy -``` - - -### 3.3 预测 - -使用如下命令进行单张图片预测: -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/infer_rec.py -c ./rec_svtr_tiny_none_ctc_en_train/rec_svtr_tiny_6local_6global_stn_en.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_svtr_tiny_none_ctc_en_train/best_accuracy -# 预测文件夹下所有图像时,可修改infer_img为文件夹,如 Global.infer_img='./doc/imgs_words_en/'。 -``` - - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将训练得到best模型,转换成inference model。下面以`SVTR-T`在英文数据集训练的模型为例([模型和配置文件下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar) ),可以使用如下命令进行转换: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/export_model.py -c ./rec_svtr_tiny_none_ctc_en_train/rec_svtr_tiny_6local_6global_stn_en.yml -o Global.pretrained_model=./rec_svtr_tiny_none_ctc_en_train/best_accuracy Global.save_inference_dir=./inference/rec_svtr_tiny_stn_en -``` - -**注意:** -- 如果您是在自己的数据集上训练的模型,并且调整了字典文件,请注意修改配置文件中的`character_dict_path`是否为所正确的字典文件。 - -转换成功后,在目录下有三个文件: -``` -/inference/rec_svtr_tiny_stn_en/ - ├── inference.pdiparams # 识别inference模型的参数文件 - ├── inference.pdiparams.info # 识别inference模型的参数信息,可忽略 - └── inference.pdmodel # 识别inference模型的program文件 -``` - - -执行如下命令进行模型推理: - -```shell -python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_svtr_tiny_stn_en/' --rec_algorithm='SVTR' --rec_image_shape='3,64,256' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' -# 预测文件夹下所有图像时,可修改image_dir为文件夹,如 --image_dir='./doc/imgs_words_en/'。 -``` -![](../imgs_words_en/word_10.png) - -执行命令后,上面图像的预测结果(识别的文本和得分)会打印到屏幕上,示例如下: -结果如下: -```shell -Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9999998807907104) -``` - -**注意**: - -- 如果您调整了训练时的输入分辨率,需要通过参数`rec_image_shape`设置为您需要的识别图像形状。 -- 在推理时需要设置参数`rec_char_dict_path`指定字典,如果您修改了字典,请修改该参数为您的字典文件。 -- 如果您修改了预处理方法,需修改`tools/infer/predict_rec.py`中SVTR的预处理为您的预处理方法。 - - -### 4.2 C++推理部署 - -由于C++预处理后处理还未支持SVTR,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - -- 1. GPU和CPU速度对比 - - 由于`SVTR`使用的算子大多为矩阵相乘,在GPU环境下,速度具有优势,但在CPU开启mkldnn加速环境下,`SVTR`相比于被优化的卷积网络没有优势。 -- 2. SVTR模型转ONNX失败 - - 保证`paddle2onnx`和`onnxruntime`版本最新,转onnx命令参考[SVTR模型转onnx步骤实例](https://github.com/PaddlePaddle/PaddleOCR/issues/7821#issuecomment-1271214273)。 -- 3. SVTR转ONNX成功但是推理结果不正确 - - 可能的原因模型参数`out_char_num`设置不正确,应设置为W//4、W//8或者W//12,可以参考[高精度中文场景文本识别模型SVTR的3.3.3章节](https://aistudio.baidu.com/aistudio/projectdetail/5073182?contributionType=1)。 -- 4. 长文本识别优化 - - 参考[高精度中文场景文本识别模型SVTR的3.3章节](https://aistudio.baidu.com/aistudio/projectdetail/5073182?contributionType=1)。 -- 5. 论文结果复现注意事项 - - 数据集使用[ABINet](https://github.com/FangShancheng/ABINet)提供的数据集; - - 默认使用4卡GPU训练,单卡Batchsize默认为512,总Batchsize为2048,对应的学习率为0.0005,当修改Batchsize或者改变GPU卡数,学习率应等比例修改。 -- 6. 进一步优化的探索点 - - 学习率调整:可以调整为默认的两倍保持Batchsize不变;或者将Batchsize减小为默认的1/2,保持学习率不变; - - 数据增强策略:可选`RecConAug`和`RecAug`; - - 如果不使用STN时,可以将`mixer`的`Local`替换为`Conv`、`local_mixer`全部修改为`[5, 5]`; - - 网格搜索最优的`embed_dim`、`depth`、`num_heads`配置; - - 使用`后Normalization策略`,即是将模型配置`prenorm`修改为`True`。 - - -## 引用 - -```bibtex -@article{Du2022SVTR, - title = {SVTR: Scene Text Recognition with a Single Visual Model}, - author = {Du, Yongkun and Chen, Zhineng and Jia, Caiyan and Yin, Xiaoting and Zheng, Tianlun and Li, Chenxia and Du, Yuning and Jiang, Yu-Gang}, - booktitle = {IJCAI}, - year = {2022}, - url = {https://arxiv.org/abs/2205.00159} -} -``` diff --git a/doc/doc_ch/algorithm_rec_visionlan.md b/doc/doc_ch/algorithm_rec_visionlan.md deleted file mode 100644 index eb58942c24..0000000000 --- a/doc/doc_ch/algorithm_rec_visionlan.md +++ /dev/null @@ -1,154 +0,0 @@ -# 场景文本识别算法-VisionLAN - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network](https://arxiv.org/abs/2108.09661) -> Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang, Shenggao Zhu, Yongdong Zhang -> ICCV, 2021 - - - -`VisionLAN`使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC13, IC15, SVTP, CUTE数据集上进行评估,算法复现效果如下: - -|模型|骨干网络|配置文件|Acc|下载链接| -| --- | --- | --- | --- | --- | -|VisionLAN|ResNet45|[rec_r45_visionlan.yml](../../configs/rec/rec_r45_visionlan.yml)|90.30%|[预训练、训练模型](https://paddleocr.bj.bcebos.com/VisionLAN/rec_r45_visionlan_train.tar)| - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - - -### 3.1 模型训练 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练`VisionLAN`识别模型时需要**更换配置文件**为`VisionLAN`的[配置文件](../../configs/rec/rec_r45_visionlan.yml)。 - -#### 启动训练 - - -具体地,在完成数据准备后,便可以启动训练,训练命令如下: -```shell -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_r45_visionlan.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r45_visionlan.yml -``` - - -### 3.2 评估 - -可下载已训练完成的[模型文件](#model),使用如下命令进行评估: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/eval.py -c configs/rec/rec_r45_visionlan.yml -o Global.pretrained_model=./rec_r45_visionlan_train/best_accuracy -``` - - -### 3.3 预测 - -使用如下命令进行单张图片预测: -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/infer_rec.py -c configs/rec/rec_r45_visionlan.yml -o Global.infer_img='./doc/imgs_words/en/word_2.png' Global.pretrained_model=./rec_r45_visionlan_train/best_accuracy -# 预测文件夹下所有图像时,可修改infer_img为文件夹,如 Global.infer_img='./doc/imgs_words_en/'。 -``` - - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将训练得到best模型,转换成inference model。这里以训练完成的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/VisionLAN/rec_r45_visionlan_train.tar)),可以使用如下命令进行转换: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/export_model.py -c configs/rec/rec_r45_visionlan.yml -o Global.pretrained_model=./rec_r45_visionlan_train/best_accuracy Global.save_inference_dir=./inference/rec_r45_visionlan/ -``` -**注意:** -- 如果您是在自己的数据集上训练的模型,并且调整了字典文件,请注意修改配置文件中的`character_dict_path`是否是所需要的字典文件。 -- 如果您修改了训练时的输入大小,请修改`tools/export_model.py`文件中的对应VisionLAN的`infer_shape`。 - -转换成功后,在目录下有三个文件: -``` -./inference/rec_r45_visionlan/ - ├── inference.pdiparams # 识别inference模型的参数文件 - ├── inference.pdiparams.info # 识别inference模型的参数信息,可忽略 - └── inference.pdmodel # 识别inference模型的program文件 -``` - -执行如下命令进行模型推理: - -```shell -python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words/en/word_2.png' --rec_model_dir='./inference/rec_r45_visionlan/' --rec_algorithm='VisionLAN' --rec_image_shape='3,64,256' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' --use_space_char=False -# 预测文件夹下所有图像时,可修改image_dir为文件夹,如 --image_dir='./doc/imgs_words_en/'。 -``` - -![](../imgs_words/en/word_2.png) - -执行命令后,上面图像的预测结果(识别的文本和得分)会打印到屏幕上,示例如下: -结果如下: -```shell -Predicts of ./doc/imgs_words/en/word_2.png:('yourself', 0.9999493) -``` - -**注意**: - -- 训练上述模型采用的图像分辨率是[3,64,256],需要通过参数`rec_image_shape`设置为您训练时的识别图像形状。 -- 在推理时需要设置参数`rec_char_dict_path`指定字典,如果您修改了字典,请修改该参数为您的字典文件。 -- 如果您修改了预处理方法,需修改`tools/infer/predict_rec.py`中VisionLAN的预处理为您的预处理方法。 - - - -### 4.2 C++推理部署 - -由于C++预处理后处理还未支持VisionLAN,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - -1. MJSynth和SynthText两种数据集来自于[VisionLAN源repo](https://github.com/wangyuxin87/VisionLAN) 。 -2. 我们使用VisionLAN作者提供的预训练模型进行finetune训练,预训练模型配套字典为'ppocr/utils/ic15_dict.txt'。 - -## 引用 - -```bibtex -@inproceedings{wang2021two, - title={From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network}, - author={Wang, Yuxin and Xie, Hongtao and Fang, Shancheng and Wang, Jing and Zhu, Shenggao and Zhang, Yongdong}, - booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision}, - pages={14194--14203}, - year={2021} -} -``` diff --git a/doc/doc_ch/algorithm_rec_vitstr.md b/doc/doc_ch/algorithm_rec_vitstr.md deleted file mode 100644 index ab12be7202..0000000000 --- a/doc/doc_ch/algorithm_rec_vitstr.md +++ /dev/null @@ -1,154 +0,0 @@ -# 场景文本识别算法-ViTSTR - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [Vision Transformer for Fast and Efficient Scene Text Recognition](https://arxiv.org/abs/2105.08582) -> Rowel Atienza -> ICDAR, 2021 - - - -`ViTSTR`使用MJSynth和SynthText两个文字识别数据集训练,在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估,算法复现效果如下: - -|模型|骨干网络|配置文件|Acc|下载链接| -| --- | --- | --- | --- | --- | -|ViTSTR|ViTSTR|[rec_vitstr_none_ce.yml](../../configs/rec/rec_vitstr_none_ce.yml)|79.82%|[训练模型](https://paddleocr.bj.bcebos.com/rec_vitstr_none_ce_train.tar)| - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - - -### 3.1 模型训练 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练`ViTSTR`识别模型时需要**更换配置文件**为`ViTSTR`的[配置文件](../../configs/rec/rec_vitstr_none_ce.yml)。 - -#### 启动训练 - - -具体地,在完成数据准备后,便可以启动训练,训练命令如下: -```shell -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/rec_vitstr_none_ce.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_vitstr_none_ce.yml -``` - - -### 3.2 评估 - -可下载已训练完成的[模型文件](#model),使用如下命令进行评估: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_vitstr_none_ce.yml -o Global.pretrained_model=./rec_vitstr_none_ce_train/best_accuracy -``` - - -### 3.3 预测 - -使用如下命令进行单张图片预测: -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/infer_rec.py -c configs/rec/rec_vitstr_none_ce.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_vitstr_none_ce_train/best_accuracy -# 预测文件夹下所有图像时,可修改infer_img为文件夹,如 Global.infer_img='./doc/imgs_words_en/'。 -``` - - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将训练得到best模型,转换成inference model。这里以训练完成的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/rec_vitstr_none_ce_train.tar) ),可以使用如下命令进行转换: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/export_model.py -c configs/rec/rec_vitstr_none_ce.yml -o Global.pretrained_model=./rec_vitstr_none_ce_train/best_accuracy Global.save_inference_dir=./inference/rec_vitstr/ -``` -**注意:** -- 如果您是在自己的数据集上训练的模型,并且调整了字典文件,请注意修改配置文件中的`character_dict_path`是否是所需要的字典文件。 -- 如果您修改了训练时的输入大小,请修改`tools/export_model.py`文件中的对应ViTSTR的`infer_shape`。 - -转换成功后,在目录下有三个文件: -``` -/inference/rec_vitstr/ - ├── inference.pdiparams # 识别inference模型的参数文件 - ├── inference.pdiparams.info # 识别inference模型的参数信息,可忽略 - └── inference.pdmodel # 识别inference模型的program文件 -``` - -执行如下命令进行模型推理: - -```shell -python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_vitstr/' --rec_algorithm='ViTSTR' --rec_image_shape='1,224,224' --rec_char_dict_path='./ppocr/utils/EN_symbol_dict.txt' -# 预测文件夹下所有图像时,可修改image_dir为文件夹,如 --image_dir='./doc/imgs_words_en/'。 -``` - -![](../imgs_words_en/word_10.png) - -执行命令后,上面图像的预测结果(识别的文本和得分)会打印到屏幕上,示例如下: -结果如下: -```shell -Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9998350143432617) -``` - -**注意**: - -- 训练上述模型采用的图像分辨率是[1,224,224],需要通过参数`rec_image_shape`设置为您训练时的识别图像形状。 -- 在推理时需要设置参数`rec_char_dict_path`指定字典,如果您修改了字典,请修改该参数为您的字典文件。 -- 如果您修改了预处理方法,需修改`tools/infer/predict_rec.py`中ViTSTR的预处理为您的预处理方法。 - - - -### 4.2 C++推理部署 - -由于C++预处理后处理还未支持ViTSTR,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - -1. 在`ViTSTR`论文中,使用在ImageNet1k上的预训练权重进行初始化训练,我们在训练未采用预训练权重,最终精度没有变化甚至有所提高。 -2. 我们仅仅复现了`ViTSTR`中的tiny版本,如果需要使用small、base版本,可将[ViTSTR源repo](https://github.com/roatienza/deep-text-recognition-benchmark) 中的预训练权重转为Paddle权重使用。 - -## 引用 - -```bibtex -@article{Atienza2021ViTSTR, - title = {Vision Transformer for Fast and Efficient Scene Text Recognition}, - author = {Rowel Atienza}, - booktitle = {ICDAR}, - year = {2021}, - url = {https://arxiv.org/abs/2105.08582} -} -``` diff --git a/doc/doc_ch/algorithm_sr_gestalt.md b/doc/doc_ch/algorithm_sr_gestalt.md deleted file mode 100644 index aac82b1b62..0000000000 --- a/doc/doc_ch/algorithm_sr_gestalt.md +++ /dev/null @@ -1,127 +0,0 @@ -# Text Gestalt - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [Text Gestalt: Stroke-Aware Scene Text Image Super-Resolution](https://arxiv.org/pdf/2112.08171.pdf) - -> Chen, Jingye and Yu, Haiyang and Ma, Jianqi and Li, Bin and Xue, Xiangyang - -> AAAI, 2022 - -参考[FudanOCR](https://github.com/FudanVI/FudanOCR/tree/main/text-gestalt) 数据下载说明,在TextZoom测试集合上超分算法效果如下: - -|模型|骨干网络|PSNR_Avg|SSIM_Avg|配置文件|下载链接| -|---|---|---|---|---|---| -|Text Gestalt|tsrn|19.28|0.6560| [configs/sr/sr_tsrn_transformer_strock.yml](../../configs/sr/sr_tsrn_transformer_strock.yml)|[训练模型](https://paddleocr.bj.bcebos.com/sr_tsrn_transformer_strock_train.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练不同的识别模型只需要**更换配置文件**即可。 - -- 训练 - -在完成数据准备后,便可以启动训练,训练命令如下: - -``` -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/sr/sr_tsrn_transformer_strock.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/sr/sr_tsrn_transformer_strock.yml - -``` - -- 评估 - -``` -# GPU 评估, Global.pretrained_model 为待测权重 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/sr/sr_tsrn_transformer_strock.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -- 预测: - -``` -# 预测使用的配置文件必须与训练一致 -python3 tools/infer_sr.py -c configs/sr/sr_tsrn_transformer_strock.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words_en/word_52.png -``` - -![](../imgs_words_en/word_52.png) - -执行命令后,上面图像的超分结果如下: - -![](../imgs_results/sr_word_52.png) - - -## 4. 推理部署 - - -### 4.1 Python推理 - -首先将文本超分训练过程中保存的模型,转换成inference model。以 Text-Gestalt 训练的[模型](https://paddleocr.bj.bcebos.com/sr_tsrn_transformer_strock_train.tar) 为例,可以使用如下命令进行转换: -```shell -python3 tools/export_model.py -c configs/sr/sr_tsrn_transformer_strock.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/sr_out -``` -Text-Gestalt 文本超分模型推理,可以执行如下命令: -``` -python3 tools/infer/predict_sr.py --sr_model_dir=./inference/sr_out --image_dir=doc/imgs_words_en/word_52.png --sr_image_shape=3,32,128 - -``` - -执行命令后,图像的超分结果如下: - -![](../imgs_results/sr_word_52.png) - - -### 4.2 C++推理 - -暂未支持 - - -### 4.3 Serving服务化部署 - -暂未支持 - - -### 4.4 更多推理部署 - -暂未支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@inproceedings{chen2022text, - title={Text gestalt: Stroke-aware scene text image super-resolution}, - author={Chen, Jingye and Yu, Haiyang and Ma, Jianqi and Li, Bin and Xue, Xiangyang}, - booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, - volume={36}, - number={1}, - pages={285--293}, - year={2022} -} -``` diff --git a/doc/doc_ch/algorithm_sr_telescope.md b/doc/doc_ch/algorithm_sr_telescope.md deleted file mode 100644 index e2351be72f..0000000000 --- a/doc/doc_ch/algorithm_sr_telescope.md +++ /dev/null @@ -1,128 +0,0 @@ -# Text Telescope - -- [1. 算法简介](#1) -- [2. 环境配置](#2) -- [3. 模型训练、评估、预测](#3) - - [3.1 训练](#3-1) - - [3.2 评估](#3-2) - - [3.3 预测](#3-3) -- [4. 推理部署](#4) - - [4.1 Python推理](#4-1) - - [4.2 C++推理](#4-2) - - [4.3 Serving服务化部署](#4-3) - - [4.4 更多推理部署](#4-4) -- [5. FAQ](#5) - - -## 1. 算法简介 - -论文信息: -> [Scene Text Telescope: Text-Focused Scene Image Super-Resolution](https://openaccess.thecvf.com/content/CVPR2021/papers/Chen_Scene_Text_Telescope_Text-Focused_Scene_Image_Super-Resolution_CVPR_2021_paper.pdf) - -> Chen, Jingye, Bin Li, and Xiangyang Xue - -> CVPR, 2021 - -参考[FudanOCR](https://github.com/FudanVI/FudanOCR/tree/main/scene-text-telescope) 数据下载说明,在TextZoom测试集合上超分算法效果如下: - -|模型|骨干网络|PSNR_Avg|SSIM_Avg|配置文件|下载链接| -|---|---|---|---|---|---| -|Text Telescope|tbsrn|21.56|0.7411| [configs/sr/sr_telescope.yml](../../configs/sr/sr_telescope.yml)|[训练模型](https://paddleocr.bj.bcebos.com/contribution/sr_telescope_train.tar)| - -[TextZoom数据集](https://paddleocr.bj.bcebos.com/dataset/TextZoom.tar) 来自两个超分数据集RealSR和SR-RAW,两个数据集都包含LR-HR对,TextZoom有17367对训数据和4373对测试数据。 - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -请参考[文本识别训练教程](./recognition.md)。PaddleOCR对代码进行了模块化,训练不同的识别模型只需要**更换配置文件**即可。 - -- 训练 - -在完成数据准备后,便可以启动训练,训练命令如下: - -``` -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/sr/sr_telescope.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/sr/sr_telescope.yml - -``` - -- 评估 - -``` -# GPU 评估, Global.pretrained_model 为待测权重 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/sr/sr_telescope.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -- 预测: - -``` -# 预测使用的配置文件必须与训练一致 -python3 tools/infer_sr.py -c configs/sr/sr_telescope.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words_en/word_52.png -``` - -![](../imgs_words_en/word_52.png) - -执行命令后,上面图像的超分结果如下: - -![](../imgs_results/sr_word_52.png) - - -## 4. 推理部署 - - -### 4.1 Python推理 - -首先将文本超分训练过程中保存的模型,转换成inference model。以 Text-Telescope 训练的[模型](https://paddleocr.bj.bcebos.com/contribution/Telescope_train.tar.gz) 为例,可以使用如下命令进行转换: -```shell -python3 tools/export_model.py -c configs/sr/sr_telescope.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/sr_out -``` -Text-Telescope 文本超分模型推理,可以执行如下命令: -``` -python3 tools/infer/predict_sr.py --sr_model_dir=./inference/sr_out --image_dir=doc/imgs_words_en/word_52.png --sr_image_shape=3,32,128 - -``` - -执行命令后,图像的超分结果如下: - -![](../imgs_results/sr_word_52.png) - - -### 4.2 C++推理 - -暂未支持 - - -### 4.3 Serving服务化部署 - -暂未支持 - - -### 4.4 更多推理部署 - -暂未支持 - - -## 5. FAQ - - -## 引用 - -```bibtex -@INPROCEEDINGS{9578891, - author={Chen, Jingye and Li, Bin and Xue, Xiangyang}, - booktitle={2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, - title={Scene Text Telescope: Text-Focused Scene Image Super-Resolution}, - year={2021}, - volume={}, - number={}, - pages={12021-12030}, - doi={10.1109/CVPR46437.2021.01185}} -``` diff --git a/doc/doc_ch/algorithm_table_master.md b/doc/doc_ch/algorithm_table_master.md deleted file mode 100644 index 36455ed9f9..0000000000 --- a/doc/doc_ch/algorithm_table_master.md +++ /dev/null @@ -1,114 +0,0 @@ -# 表格识别算法-TableMASTER - -- [1. 算法简介](#1-算法简介) -- [2. 环境配置](#2-环境配置) -- [3. 模型训练、评估、预测](#3-模型训练评估预测) -- [4. 推理部署](#4-推理部署) - - [4.1 Python推理](#41-python推理) - - [4.2 C++推理部署](#42-c推理部署) - - [4.3 Serving服务化部署](#43-serving服务化部署) - - [4.4 更多推理部署](#44-更多推理部署) -- [5. FAQ](#5-faq) -- [引用](#引用) - - -## 1. 算法简介 - -论文信息: -> [TableMaster: PINGAN-VCGROUP’S SOLUTION FOR ICDAR 2021 COMPETITION ON SCIENTIFIC LITERATURE PARSING TASK B: TABLE RECOGNITION TO HTML](https://arxiv.org/pdf/2105.01848.pdf) -> Ye, Jiaquan and Qi, Xianbiao and He, Yelin and Chen, Yihao and Gu, Dengyi and Gao, Peng and Xiao, Rong -> 2021 - -在PubTabNet表格识别公开数据集上,算法复现效果如下: - -|模型|骨干网络|配置文件|acc|下载链接| -| --- | --- | --- | --- | --- | -|TableMaster|TableResNetExtra|[configs/table/table_master.yml](../../configs/table/table_master.yml)|77.47%|[训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/tablemaster/table_structure_tablemaster_train.tar)/[推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/tablemaster/table_structure_tablemaster_infer.tar)| - - - -## 2. 环境配置 -请先参考[《运行环境准备》](./environment.md)配置PaddleOCR运行环境,参考[《项目克隆》](./clone.md)克隆项目代码。 - - - -## 3. 模型训练、评估、预测 - -上述TableMaster模型使用PubTabNet表格识别公开数据集训练得到,数据集下载可参考 [table_datasets](./dataset/table_datasets.md)。 - -数据下载完成后,请参考[文本识别教程](./recognition.md)进行训练。PaddleOCR对代码进行了模块化,训练不同的模型只需要**更换配置文件**即可。 - - -## 4. 推理部署 - - -### 4.1 Python推理 -首先将训练得到best模型,转换成inference model。以基于TableResNetExtra骨干网络,在PubTabNet数据集训练的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/contribution/table_master.tar)),可以使用如下命令进行转换: - -```shell -# 注意将pretrained_model的路径设置为本地路径。 -python3 tools/export_model.py -c configs/table/table_master.yml -o Global.pretrained_model=output/table_master/best_accuracy Global.save_inference_dir=./inference/table_master -``` - -**注意:** -- 如果您是在自己的数据集上训练的模型,并且调整了字典文件,请注意修改配置文件中的`character_dict_path`是否为所正确的字典文件。 - -转换成功后,在目录下有三个文件: -``` -./inference/table_master/ - ├── inference.pdiparams # 识别inference模型的参数文件 - ├── inference.pdiparams.info # 识别inference模型的参数信息,可忽略 - └── inference.pdmodel # 识别inference模型的program文件 -``` - - -执行如下命令进行模型推理: - -```shell -cd ppstructure/ -python3.7 table/predict_structure.py --table_model_dir=../output/table_master/table_structure_tablemaster_infer/ --table_algorithm=TableMaster --table_char_dict_path=../ppocr/utils/dict/table_master_structure_dict.txt --table_max_len=480 --image_dir=docs/table/table.jpg -# 预测文件夹下所有图像时,可修改image_dir为文件夹,如 --image_dir='docs/table'。 -``` - -执行命令后,上面图像的预测结果(结构信息和表格中每个单元格的坐标)会打印到屏幕上,同时会保存单元格坐标的可视化结果。示例如下: -结果如下: -```shell -[2022/06/16 13:06:54] ppocr INFO: result: ['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '
', '', ''], [[72.17591094970703, 10.759100914001465, 60.29658508300781, 16.6805362701416], [161.85562133789062, 10.884308815002441, 14.9495210647583, 16.727018356323242], [277.79876708984375, 29.54340362548828, 31.490320205688477, 18.143272399902344], -... -[336.11724853515625, 280.3601989746094, 39.456939697265625, 18.121286392211914]] -[2022/06/16 13:06:54] ppocr INFO: save vis result to ./output/table.jpg -[2022/06/16 13:06:54] ppocr INFO: Predict time of docs/table/table.jpg: 17.36806297302246 -``` - -**注意**: - -- TableMaster在推理时比较慢,建议使用GPU进行使用。 - - -### 4.2 C++推理部署 - -由于C++预处理后处理还未支持TableMaster,所以暂未支持 - - -### 4.3 Serving服务化部署 - -暂不支持 - - -### 4.4 更多推理部署 - -暂不支持 - - -## 5. FAQ - -## 引用 - -```bibtex -@article{ye2021pingan, - title={PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Literature Parsing Task B: Table Recognition to HTML}, - author={Ye, Jiaquan and Qi, Xianbiao and He, Yelin and Chen, Yihao and Gu, Dengyi and Gao, Peng and Xiao, Rong}, - journal={arXiv preprint arXiv:2105.01848}, - year={2021} -} -``` diff --git a/doc/doc_ch/angle_class.md b/doc/doc_ch/angle_class.md deleted file mode 100644 index 723d0d2ce5..0000000000 --- a/doc/doc_ch/angle_class.md +++ /dev/null @@ -1,154 +0,0 @@ -# 文本方向分类器 - -- [1.方法介绍](#方法介绍) -- [2.数据准备](#数据准备) -- [3.启动训练](#启动训练) -- [4.训练](#训练) -- [5.评估](#评估) -- [6.预测](#预测) - - -## 1. 方法介绍 -文本方向分类器主要用于图片非0度的场景下,在这种场景下需要对图片里检测到的文本行进行一个转正的操作。在PaddleOCR系统内, -文字检测之后得到的文本行图片经过仿射变换之后送入识别模型,此时只需要对文字进行一个0和180度的角度分类,因此PaddleOCR内置的 -文本方向分类器**只支持了0和180度的分类**。如果想支持更多角度,可以自己修改算法进行支持。 - -0和180度数据样本例子: - -![](../imgs_results/angle_class_example.jpg) - - -## 2. 数据准备 - -请按如下步骤设置数据集: - -训练数据的默认存储路径是 `PaddleOCR/train_data/cls`,如果您的磁盘上已有数据集,只需创建软链接至数据集目录: - -``` -ln -sf /train_data/cls/dataset -``` - -请参考下文组织您的数据。 -- 训练集 - -首先建议将训练图片放入同一个文件夹,并用一个txt文件(cls_gt_train.txt)记录图片路径和标签。 - -**注意:** 默认请将图片路径和图片标签用 `\t` 分割,如用其他方式分割将造成训练报错 - -0和180分别表示图片的角度为0度和180度 - -``` -" 图像文件名 图像标注信息 " -train/cls/train/word_001.jpg 0 -train/cls/train/word_002.jpg 180 -``` - -最终训练集应有如下文件结构: -``` -|-train_data - |-cls - |- cls_gt_train.txt - |- train - |- word_001.png - |- word_002.jpg - |- word_003.jpg - | ... -``` - -- 测试集 - -同训练集类似,测试集也需要提供一个包含所有图片的文件夹(test)和一个cls_gt_test.txt,测试集的结构如下所示: - -``` -|-train_data - |-cls - |- cls_gt_test.txt - |- test - |- word_001.jpg - |- word_002.jpg - |- word_003.jpg - | ... -``` - -## 3. 启动训练 - -将准备好的txt文件和图片文件夹路径分别写入配置文件的 `Train/Eval.dataset.label_file_list` 和 `Train/Eval.dataset.data_dir` 字段下,`Train/Eval.dataset.data_dir`字段下的路径和文件里记载的图片名构成了图片的绝对路径。 - -PaddleOCR提供了训练脚本、评估脚本和预测脚本。 - -开始训练: - -*如果您安装的是cpu版本,请将配置文件中的 `use_gpu` 字段修改为false* - -``` -# GPU训练 支持单卡,多卡训练,通过 '--gpus' 指定卡号。 -# 启动训练,下面的命令已经写入train.sh文件中,只需修改文件里的配置文件路径即可 -python3 -m paddle.distributed.launch --gpus '0,1,2,3,4,5,6,7' tools/train.py -c configs/cls/cls_mv3.yml -``` - -- 数据增强 - -PaddleOCR提供了多种数据增强方式,如果您希望在训练时加入扰动,请在配置文件中取消`Train.dataset.transforms`下的`RecAug`和`RandAugment`字段的注释。 - -默认的扰动方式有:颜色空间转换(cvtColor)、模糊(blur)、抖动(jitter)、噪声(Gasuss noise)、随机切割(random crop)、透视(perspective)、颜色反转(reverse),随机数据增强(RandAugment)。 - -训练过程中除随机数据增强外每种扰动方式以50%的概率被选择,具体代码实现请参考: -[rec_img_aug.py](../../ppocr/data/imaug/rec_img_aug.py) -[randaugment.py](../../ppocr/data/imaug/randaugment.py) - -*由于OpenCV的兼容性问题,扰动操作暂时只支持linux* - - -## 4. 训练 - -PaddleOCR支持训练和评估交替进行, 可以在 `configs/cls/cls_mv3.yml` 中修改 `eval_batch_step` 设置评估频率,默认每1000个iter评估一次。训练过程中将会保存如下内容: -```bash -├── best_accuracy.pdopt # 最佳模型的优化器参数 -├── best_accuracy.pdparams # 最佳模型的参数 -├── best_accuracy.states # 最佳模型的指标和epoch等信息 -├── config.yml # 本次实验的配置文件 -├── latest.pdopt # 最新模型的优化器参数 -├── latest.pdparams # 最新模型的参数 -├── latest.states # 最新模型的指标和epoch等信息 -└── train.log # 训练日志 -``` - -如果验证集很大,测试将会比较耗时,建议减少评估次数,或训练完再进行评估。 - -**注意,预测/评估时的配置文件请务必与训练一致。** - - -## 5. 评估 - -评估数据集可以通过修改`configs/cls/cls_mv3.yml`文件里的`Eval.dataset.label_file_list` 字段设置。 - -``` -export CUDA_VISIBLE_DEVICES=0 -# GPU 评估, Global.checkpoints 为待测权重 -python3 tools/eval.py -c configs/cls/cls_mv3.yml -o Global.checkpoints={path/to/weights}/best_accuracy -``` - - -## 6. 预测 - -* 训练引擎的预测 - -使用 PaddleOCR 训练好的模型,可以通过以下脚本进行快速预测。 - -通过 `Global.infer_img` 指定预测图片或文件夹路径,通过 `Global.checkpoints` 指定权重: - -``` -# 预测分类结果 -python3 tools/infer_cls.py -c configs/cls/cls_mv3.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.load_static_weights=false Global.infer_img=doc/imgs_words/ch/word_1.jpg -``` - -预测图片: - -![](../imgs_words/ch/word_1.jpg) - -得到输入图像的预测结果: - -``` -infer_img: doc/imgs_words/ch/word_1.jpg - result: ('0', 0.9998784) -``` diff --git a/doc/doc_ch/application.md b/doc/doc_ch/application.md deleted file mode 100644 index 9105a87854..0000000000 --- a/doc/doc_ch/application.md +++ /dev/null @@ -1,41 +0,0 @@ -# 场景应用 - -PaddleOCR场景应用覆盖通用,制造、金融、交通行业的主要OCR垂类应用,在PP-OCR、PP-Structure的通用能力基础之上,以notebook的形式展示利用场景数据微调、模型优化方法、数据增广等内容,为开发者快速落地OCR应用提供示范与启发。 - -> 如需下载全部垂类模型,可以扫描下方二维码,关注公众号填写问卷后,加入PaddleOCR官方交流群获取20G OCR学习大礼包(内含《动手学OCR》电子书、课程回放视频、前沿论文等重磅资料) - -
- -
- - -> 如果您是企业开发者且未在下述场景中找到合适的方案,可以填写[OCR应用合作调研问卷](https://paddle.wjx.cn/vj/QwF7GKw.aspx),免费与官方团队展开不同层次的合作,包括但不限于问题抽象、确定技术方案、项目答疑、共同研发等。如果您已经使用PaddleOCR落地项目,也可以填写此问卷,与飞桨平台共同宣传推广,提升企业技术品宣。期待您的提交! - -## 通用 - -| 类别 | 亮点 | 类别 | 亮点 | -| ---------------------- | -------- | ---------- | ------------ | -| 高精度中文识别模型SVTR | 新增模型 | 手写体识别 | 新增字形支持 | - -## 制造 - -| 类别 | 亮点 | 类别 | 亮点 | -| -------------- | ------------------------------ | -------------- | -------------------- | -| 数码管识别 | 数码管数据合成、漏识别调优 | 电表识别 | 大分辨率图像检测调优 | -| 液晶屏读数识别 | 检测模型蒸馏、Serving部署 | PCB文字识别 | 小尺寸文本检测与识别 | -| 包装生产日期 | 点阵字符合成、过曝过暗文字识别 | 液晶屏缺陷检测 | 非文字形态识别 | - -## 金融 - -| 类别 | 亮点 | 类别 | 亮点 | -| -------------- | ------------------------ | ------------ | --------------------- | -| 表单VQA | 多模态通用表单结构化提取 | 通用卡证识别 | 通用结构化提取 | -| 增值税发票 | 敬请期待 | 身份证识别 | 结构化提取、图像阴影 | -| 印章检测与识别 | 端到端弯曲文本识别 | 合同比对 | 密集文本检测、NLP串联 | - -## 交通 - -| 类别 | 亮点 | 类别 | 亮点 | -| ----------------- | ------------------------------ | ---------- | -------- | -| 车牌识别 | 多角度图像、轻量模型、端侧部署 | 快递单识别 | 敬请期待 | -| 驾驶证/行驶证识别 | 敬请期待 | | | diff --git a/doc/doc_ch/benchmark.md b/doc/doc_ch/benchmark.md deleted file mode 100644 index da82d30ec2..0000000000 --- a/doc/doc_ch/benchmark.md +++ /dev/null @@ -1,38 +0,0 @@ -# Benchmark - -本文给出了中英文OCR系列模型精度指标和在各平台预测耗时的benchmark。 - -## 测试数据 -针对OCR实际应用场景,包括合同,车牌,铭牌,火车票,化验单,表格,证书,街景文字,名片,数码显示屏等,收集的300张图像,每张图平均有17个文本框,下图给出了一些图像示例。 - -
- -
- -## 评估指标 - -说明: - -- 检测输入图像的长边尺寸是960。 -- 评估耗时阶段为图像预测耗时,不包括图像的预处理和后处理。 -- `Intel至强6148`为服务器端CPU型号,测试中使用Intel MKL-DNN 加速。 -- `骁龙855`为移动端处理平台型号。 - -预测模型大小和整体识别精度对比 - -| 模型名称 | 整体模型
大小\(M\) | 检测模型
大小\(M\) | 方向分类器
模型大小\(M\) | 识别模型
大小\(M\) | 整体识别
F\-score | -|:-:|:-:|:-:|:-:|:-:|:-:| -| PP-OCRv2 | 11\.6 | 3\.0 | 0\.9 | 8\.6 | 0\.5224 | -| PP-OCR mobile | 8\.1 | 2\.6 | 0\.9 | 4\.6 | 0\.503 | -| PP-OCR server | 155\.1 | 47\.2 | 0\.9 | 107 | 0\.570 | - - -预测模型在CPU和GPU上的速度对比,单位ms - -| 模型名称 | CPU | T4 GPU | -|:-:|:-:|:-:| -| PP-OCRv2 | 330 | 111 | -| PP-OCR mobile | 356 | 11 6| -| PP-OCR server | 1056 | 200 | - -更多 PP-OCR 系列模型的预测指标可以参考[PP-OCR Benchmark](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.2/doc/doc_ch/benchmark.md) diff --git a/doc/doc_ch/clone.md b/doc/doc_ch/clone.md deleted file mode 100644 index 98a63d19a2..0000000000 --- a/doc/doc_ch/clone.md +++ /dev/null @@ -1,22 +0,0 @@ -# 项目克隆 - -## 1. 克隆PaddleOCR repo代码 - -``` -【推荐】git clone https://github.com/PaddlePaddle/PaddleOCR -``` - -如果因为网络问题无法pull成功,也可选择使用码云上的托管: - -``` -git clone https://gitee.com/paddlepaddle/PaddleOCR -``` - -注:码云托管代码可能无法实时同步本github项目更新,存在3~5天延时,请优先使用推荐方式。 - -## 2. 安装第三方库 - -``` -cd PaddleOCR -pip3 install -r requirements.txt -``` diff --git a/doc/doc_ch/code_and_doc.md b/doc/doc_ch/code_and_doc.md deleted file mode 100644 index ce11015795..0000000000 --- a/doc/doc_ch/code_and_doc.md +++ /dev/null @@ -1,337 +0,0 @@ -# 附录 - -本附录包含了Python、文档规范以及Pull Request流程,请各位开发者遵循相关内容 - -- [附录1:Python代码规范](#附录1) - -- [附录2:文档规范](#附录2) - -- [附录3:Pull Request说明](#附录3) - - - -## 附录1:Python代码规范 - -PaddleOCR的Python代码遵循 [PEP8规范](https://www.python.org/dev/peps/pep-0008/),其中一些关注的重点包括如下内容 - -- 空格 - - - 空格应该加在逗号、分号、冒号后,而非他们的前面 - - ```python - # 正确: - print(x, y) - - # 错误: - print(x , y) - ``` - - - 在函数中指定关键字参数或默认参数值时, 不要在其两侧使用空格 - - ```python - # 正确: - def complex(real, imag=0.0) - # 错误: - def complex(real, imag = 0.0) - ``` - -- 注释 - - - 行内注释:行内注释使用 `#` 号表示,在代码与 `#` 之间需要空两个空格, `#` 与注释之间应当空一个空格,例如 - - ```python - x = x + 1 # Compensate for border - ``` - - - 函数和方法:每个函数的定义后的描述应该包括以下内容: - - - 函数描述:函数的作用,输入输出的 - - - Args:每个参数的名字以及对该参数的描述 - - Returns:返回值的含义和类型 - - ```python - def fetch_bigtable_rows(big_table, keys, other_silly_variable=None): - """Fetches rows from a Bigtable. - - Retrieves rows pertaining to the given keys from the Table instance - represented by big_table. Silly things may happen if - other_silly_variable is not None. - - Args: - big_table: An open Bigtable Table instance. - keys: A sequence of strings representing the key of each table row - to fetch. - other_silly_variable: Another optional variable, that has a much - longer name than the other args, and which does nothing. - - Returns: - A dict mapping keys to the corresponding table row data - fetched. Each row is represented as a tuple of strings. For - example: - - {'Serak': ('Rigel VII', 'Preparer'), - 'Zim': ('Irk', 'Invader'), - 'Lrrr': ('Omicron Persei 8', 'Emperor')} - - If a key from the keys argument is missing from the dictionary, - then that row was not found in the table. - """ - pass - ``` - - - -## 附录2:文档规范 - -### 2.1 总体说明 - -- 文档位置:如果您增加的新功能可以补充在原有的Markdown文件中,请**不要重新新建**一个文件。如果您对添加的位置不清楚,可以先PR代码,然后在commit中询问官方人员。 - -- 新增Markdown文档名称:使用英文描述文档内容,一般由小写字母与下划线组合而成,例如 `add_new_algorithm.md` - -- 新增Markdown文档格式:目录 - 正文 - FAQ - - > 目录生成方法可以使用 [此网站](https://ecotrust-canada.github.io/markdown-toc/) 将md内容复制之后自动提取目录,然后在md文件的每个标题前添加 `` - -- 中英双语:任何对文档的改动或新增都需要分别在中文和英文文档上进行。 - -### 2.2 格式规范 - -- 标题格式:文档标题格式按照:阿拉伯数字小数点组合 - 空格 - 标题的格式(例如 `2.1 XXXX` , `2. XXXX`) - -- 代码块:通过代码块格式展示需要运行的代码,在代码块前描述命令参数的含义。例如: - - > 检测+方向分类器+识别全流程:设置方向分类器参数 `--use_angle_cls true` 后可对竖排文本进行识别。 - > - > ``` - > paddleocr --image_dir ./imgs/11.jpg --use_angle_cls true - > ``` - -- 变量引用:如果在行内引用到代码变量或命令参数,需要用行内代码表示,例如上方 `--use_angle_cls true` ,并在前后各空一格 - -- 统一命名:如PP-OCRv2、PP-OCR mobile、`paddleocr` whl包、PPOCRLabel、Paddle Lite等 - -- 补充说明:通过引用格式 `>` 补充说明,或对注意事项进行说明 - -- 图片:如果在说明文档中增加了图片,请规范图片的命名形式(描述图片内容),并将图片添加在 `doc/` 下 - - - -## 附录3:Pull Request说明 - -### 3.1 PaddleOCR分支说明 - -PaddleOCR未来将维护2种分支,分别为: - -- release/x.x系列分支:为稳定的发行版本分支,也是默认分支。PaddleOCR会根据功能更新情况发布新的release分支,同时适配Paddle的release版本。随着版本迭代,release/x.x系列分支会越来越多,默认维护最新版本的release分支。 -- dygraph分支:为开发分支,适配Paddle动态图的dygraph版本,主要用于开发新功能。如果有同学需要进行二次开发,请选择dygraph分支。为了保证dygraph分支能在需要的时候拉出release/x.x分支,dygraph分支的代码只能使用Paddle最新release分支中有效的api。也就是说,如果Paddle dygraph分支中开发了新的api,但尚未出现在release分支代码中,那么请不要在PaddleOCR中使用。除此之外,对于不涉及api的性能优化、参数调整、策略更新等,都可以正常进行开发。 - -PaddleOCR的历史分支,未来将不再维护。考虑到一些同学可能仍在使用,这些分支还会继续保留: - -- develop分支:这个分支曾用于静态图的开发与测试,目前兼容>=1.7版本的Paddle。如果有特殊需求,要适配旧版本的Paddle,那还可以使用这个分支,但除了修复bug外不再更新代码。 - -PaddleOCR欢迎大家向repo中积极贡献代码,下面给出一些贡献代码的基本流程。 - -### 3.2 PaddleOCR代码提交流程与规范 - -> 如果你熟悉Git使用,可以直接跳转到 [3.2.10 提交代码的一些约定](#提交代码的一些约定) - -#### 3.2.1 创建你的 `远程仓库` - -- 在PaddleOCR的 [GitHub首页](https://github.com/PaddlePaddle/PaddleOCR),点击左上角 `Fork` 按钮,在你的个人目录下创建 `远程仓库`,比如`https://github.com/{your_name}/PaddleOCR`。 - -![banner](../banner.png) - -- 将 `远程仓库` Clone到本地 - -``` -# 拉取dygraph分支的代码 -git clone https://github.com/{your_name}/PaddleOCR.git -b dygraph -cd PaddleOCR -``` - -> 多数情况下clone失败是由于网络原因,请稍后重试或配置代理 - -#### 3.2.2 通过Token方式登录与建立连接 - -首先查看当前 `远程仓库` 的信息。 - -``` -git remote -v -# origin https://github.com/{your_name}/PaddleOCR.git (fetch) -# origin https://github.com/{your_name}/PaddleOCR.git (push) -``` - -只有clone的 `远程仓库` 的信息,也就是自己用户名下的 PaddleOCR。由于Github的登录方式变化,需要通过Token的方式重新配置 `远程仓库` 的地址。生成Token的方式如下: - -1. 找到个人访问令牌(token):在Github页面右上角点击自己的头像,然后依次选择 Settings --> Developer settings --> Personal access tokens -2. 点击 Generate new token:在Note中填入token名称,例如’paddle‘。在Select scopes选择repo(必选)、admin:repo_hook、delete_repo等,可根据自身需要勾选。然后点击Generate token生成token。最后复制生成的token。 - -删除原始的origin配置 - -``` -git remote rm origin -``` - -将remote分支改成 `https://oauth2:{token}@github.com/{your_name}/PaddleOCR.git`。例如:如果token值为12345,你的用户名为PPOCR,则运行下方命令 - -``` -git remote add origin https://oauth2:12345@github.com/PPOCR/PaddleOCR.git -``` - -这样我们就与自己的 `远程仓库` 建立了连接。接下来我们创建一个原始 PaddleOCR 仓库的远程主机,命名为 upstream。 - -``` -git remote add upstream https://github.com/PaddlePaddle/PaddleOCR.git -``` - -使用 `git remote -v` 查看当前 `远程仓库` 的信息,输出如下,发现包括了origin和upstream 2个 `远程仓库` 。 - -``` -origin https://oauth2:{token}@github.com/{your_name}/PaddleOCR.git (fetch) -origin https://oauth2:{token}@github.com/{your_name}/PaddleOCR.git (push) -upstream https://github.com/PaddlePaddle/PaddleOCR.git (fetch) -upstream https://github.com/PaddlePaddle/PaddleOCR.git (push) -``` - -这主要是为了后续在提交pull request(PR)时,始终保持本地仓库最新。 - -#### 3.2.3 创建本地分支 - -首先获取 upstream 的最新代码,然后基于上游仓库 (upstream)的dygraph创建new_branch分支。 - -``` -git fetch upstream -git checkout -b new_branch upstream/dygraph -``` - -> 如果对于新Fork的PaddleOCR项目,用户远程仓库(origin)与上游(upstream)仓库的分支更新情况相同,也可以基于origin仓库的默认分支或指定分支创建新的本地分支,命令如下。 -> -> ``` -> # 基于用户远程仓库(origin)的dygraph创建new_branch分支 -> git checkout -b new_branch origin/dygraph -> -> # 基于用户远程仓库(origin)的默认分支创建new_branch分支 -> git checkout -b new_branch -> ``` - -最终会显示切换到新的分支,输出信息如下 - -``` -Branch new_branch set up to track remote branch develop from upstream. -Switched to a new branch 'new_branch' -``` - -切换分支之后即可在此分支上进行文件改动 - -#### 3.2.4 使用pre-commit勾子 - -Paddle 开发人员使用 pre-commit 工具来管理 Git 预提交钩子。 它可以帮助我们格式化源代码(C++,Python),在提交(commit)前自动检查一些基本事宜(如每个文件只有一个 EOL,Git 中不要添加大文件等)。 - -pre-commit测试是 Travis-CI 中单元测试的一部分,不满足钩子的 PR 不能被提交到 PaddleOCR,首先安装并在当前目录运行它: - -``` -pip install pre-commit -pre-commit install -``` - - > 1. Paddle 使用 clang-format 来调整 C/C++ 源代码格式,请确保 `clang-format` 版本在 3.8 以上。 - > - > 2. 通过pip install pre-commit和conda install -c conda-forge pre-commit安装的yapf稍有不同的,PaddleOCR 开发人员使用的是 `pip install pre-commit`。 - -#### 3.2.5 修改与提交代码 - - 假设对PaddleOCR的 `README.md` 做了一些修改,可以通过 `git status` 查看改动的文件,然后使用 `git add` 添加改动文件。 - -``` -git status # 查看改动文件 -git add README.md -pre-commit -``` - -重复上述步骤,直到pre-comit格式检查不报错。如下所示。 - -![img](../precommit_pass.png) - -提交修改,并写明修改内容("your commit info") - -``` -git commit -m "your commit info" -``` - -#### 3.2.6 Push到远程仓库 - -使用push命令将修改的commit提交到 `远程仓库` - -``` -git push origin new_branch -``` - -#### 3.2.7 提交Pull Request - -打开自己的远程仓库界面,选择提交的分支。点击new pull request或contribute进入PR界面。选择本地分支和目标分支,如下图所示。在PR的描述说明中,填写该PR所完成的功能。接下来等待review,如果有需要修改的地方,参照上述步骤更新 origin 中的对应分支即可。 - -![banner](../pr.png) - -#### 3.2.8 签署CLA协议和通过单元测试 - -- 签署CLA 在首次向PaddlePaddle提交Pull Request时,您需要您签署一次CLA(Contributor License Agreement)协议,以保证您的代码可以被合入,具体签署方式如下: - - 1. 请您查看PR中的Check部分,找到license/cla,并点击右侧detail,进入CLA网站 - - 2. 点击CLA网站中的“Sign in with GitHub to agree”,点击完成后将会跳转回您的Pull Request页面 - -#### 3.2.9 删除分支 - -- 删除远程分支 - - 在 PR 被 merge 进主仓库后,我们可以在 PR 的页面删除远程仓库的分支。 - - 也可以使用 `git push origin :分支名` 删除远程分支,如: - - ``` - git push origin :new_branch - ``` - -- 删除本地分支 - - ``` - # 切换到dygraph分支,否则无法删除当前分支 - git checkout dygraph - - # 删除new_branch分支 - git branch -D new_branch - ``` - - - -#### 3.2.10 提交代码的一些约定 - -为了使官方维护人员在评审代码时更好地专注于代码本身,请您每次提交代码时,遵守以下约定: - -1)请保证Travis-CI 中单元测试能顺利通过。如果没过,说明提交的代码存在问题,官方维护人员一般不做评审。 - -2)提交Pull Request前: - -- 请注意commit的数量。 - - 原因:如果仅仅修改一个文件但提交了十几个commit,每个commit只做了少量的修改,这会给评审人带来很大困扰。评审人需要逐一查看每个commit才能知道做了哪些修改,且不排除commit之间的修改存在相互覆盖的情况。 - - 建议:每次提交时,保持尽量少的commit,可以通过git commit --amend补充上次的commit。对已经Push到远程仓库的多个commit,可以参考[squash commits after push](https://stackoverflow.com/questions/5667884/how-to-squash-commits-in-git-after-they-have-been-pushed)。 - -- 请注意每个commit的名称:应能反映当前commit的内容,不能太随意。 - -3)如果解决了某个Issue的问题,请在该Pull Request的第一个评论框中加上:fix #issue_number,这样当该Pull Request被合并后,会自动关闭对应的Issue。关键词包括:close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved,请选择合适的词汇。详细可参考[Closing issues via commit messages](https://help.github.com/articles/closing-issues-via-commit-messages)。 - -此外,在回复评审人意见时,请您遵守以下约定: - -1)官方维护人员的每一个review意见都希望得到回复,这样会更好地提升开源社区的贡献。 - -- 对评审意见同意且按其修改完的,给个简单的Done即可; -- 对评审意见不同意的,请给出您自己的反驳理由。 - -2)如果评审意见比较多: - -- 请给出总体的修改情况。 -- 请采用`start a review`进行回复,而非直接回复的方式。原因是每个回复都会发送一封邮件,会造成邮件灾难。 diff --git a/doc/doc_ch/config.md b/doc/doc_ch/config.md deleted file mode 100644 index 5ca1f68e6f..0000000000 --- a/doc/doc_ch/config.md +++ /dev/null @@ -1,226 +0,0 @@ -# 配置文件内容与生成 - -* [1. 可选参数列表](#1) -* [2. 配置文件参数介绍](#2) -* [3. 多语言配置文件生成](#3) - - - -## 1. 可选参数列表 - -以下列表可以通过`--help`查看 - -| FLAG | 支持脚本 | 用途 | 默认值 | 备注 | -| :----------------------: | :------------: | :---------------: | :--------------: | :-----------------: | -| -c | ALL | 指定配置文件 | None | **配置模块说明请参考 参数介绍** | -| -o | ALL | 设置配置文件里的参数内容 | None | 使用-o配置相较于-c选择的配置文件具有更高的优先级。例如:`-o Global.use_gpu=false` | - - - -## 2. 配置文件参数介绍 - -以 `rec_chinese_lite_train_v2.0.yml ` 为例 -### Global - -| 字段 | 用途 | 默认值 | 备注 | -| :----------------------: | :---------------------: | :--------------: | :--------------------: | -| use_gpu | 设置代码是否在gpu运行 | true | \ | -| epoch_num | 最大训练epoch数 | 500 | \ | -| log_smooth_window | log队列长度,每次打印输出队列里的中间值 | 20 | \ | -| print_batch_step | 设置打印log间隔 | 10 | \ | -| save_model_dir | 设置模型保存路径 | output/{算法名称} | \ | -| save_epoch_step | 设置模型保存间隔 | 3 | \ | -| eval_batch_step | 设置模型评估间隔 | 2000 或 [1000, 2000] | 2000 表示每2000次迭代评估一次,[1000, 2000]表示从1000次迭代开始,每2000次评估一次 | -| cal_metric_during_train | 设置是否在训练过程中评估指标,此时评估的是模型在当前batch下的指标 | true | \ | -| load_static_weights | 设置预训练模型是否是静态图模式保存(目前仅检测算法需要) | true | \ | -| pretrained_model | 设置加载预训练模型路径 | ./pretrain_models/CRNN/best_accuracy | \ | -| checkpoints | 加载模型参数路径 | None | 用于中断后加载参数继续训练 | -| use_visualdl | 设置是否启用visualdl进行可视化log展示 | False | [教程地址](https://www.paddlepaddle.org.cn/paddle/visualdl) | -| infer_img | 设置预测图像路径或文件夹路径 | ./infer_img | \|| -| character_dict_path | 设置字典路径 | ./ppocr/utils/ppocr_keys_v1.txt | 如果为空,则默认使用小写字母+数字作为字典 | -| max_text_length | 设置文本最大长度 | 25 | \ | -| use_space_char | 设置是否识别空格 | True | \| | -| label_list | 设置方向分类器支持的角度 | ['0','180'] | 仅在方向分类器中生效 | -| save_res_path | 设置检测模型的结果保存地址 | ./output/det_db/predicts_db.txt | 仅在检测模型中生效 | - -### Optimizer ([ppocr/optimizer](../../ppocr/optimizer)) - -| 字段 | 用途 | 默认值 | 备注 | -| :---------------------: |:-------------:|:-------------:| :--------------------: | -| name | 优化器类名 | Adam | 目前支持`Momentum`,`Adam`,`RMSProp`, 见[ppocr/optimizer/optimizer.py](../../ppocr/optimizer/optimizer.py) | -| beta1 | 设置一阶矩估计的指数衰减率 | 0.9 | \ | -| beta2 | 设置二阶矩估计的指数衰减率 | 0.999 | \ | -| clip_norm | 所允许的二范数最大值 | | \ | -| **lr** | 设置学习率decay方式 | - | \ | -| name | 学习率decay类名 | Cosine | 目前支持`Linear`,`Cosine`,`Step`,`Piecewise`, 见[ppocr/optimizer/learning_rate.py](../../ppocr/optimizer/learning_rate.py) | -| learning_rate | 基础学习率 | 0.001 | \ | -| **regularizer** | 设置网络正则化方式 | - | \ | -| name | 正则化类名 | L2 | 目前支持`L1`,`L2`, 见[ppocr/optimizer/regularizer.py](../../ppocr/optimizer/regularizer.py) | -| factor | 正则化系数 | 0.00001 | \ | - - -### Architecture ([ppocr/modeling](../../ppocr/modeling)) -在PaddleOCR中,网络被划分为Transform,Backbone,Neck和Head四个阶段 - -| 字段 | 用途 | 默认值 | 备注 | -| :---------------------: | :---------------------: | :--------------: | :--------------------: | -| model_type | 网络类型 | rec | 目前支持`rec`,`det`,`cls` | -| algorithm | 模型名称 | CRNN | 支持列表见[algorithm_overview](./algorithm_overview.md) | -| **Transform** | 设置变换方式 | - | 目前仅rec类型的算法支持, 具体见[ppocr/modeling/transforms](../../ppocr/modeling/transforms) | -| name | 变换方式类名 | TPS | 目前支持`TPS` | -| num_fiducial | TPS控制点数 | 20 | 上下边各十个 | -| loc_lr | 定位网络学习率 | 0.1 | \ | -| model_name | 定位网络大小 | small | 目前支持`small`,`large` | -| **Backbone** | 设置网络backbone类名 | - | 具体见[ppocr/modeling/backbones](../../ppocr/modeling/backbones) | -| name | backbone类名 | ResNet | 目前支持`MobileNetV3`,`ResNet` | -| layers | resnet层数 | 34 | 支持18,34,50,101,152,200 | -| model_name | MobileNetV3 网络大小 | small | 支持`small`,`large` | -| **Neck** | 设置网络neck | - | 具体见[ppocr/modeling/necks](../../ppocr/modeling/necks) | -| name | neck类名 | SequenceEncoder | 目前支持`SequenceEncoder`,`DBFPN` | -| encoder_type | SequenceEncoder编码器类型 | rnn | 支持`reshape`,`fc`,`rnn` | -| hidden_size | rnn内部单元数 | 48 | \ | -| out_channels | DBFPN输出通道数 | 256 | \ | -| **Head** | 设置网络Head | - | 具体见[ppocr/modeling/heads](../../ppocr/modeling/heads) | -| name | head类名 | CTCHead | 目前支持`CTCHead`,`DBHead`,`ClsHead` | -| fc_decay | CTCHead正则化系数 | 0.0004 | \ | -| k | DBHead二值化系数 | 50 | \ | -| class_dim | ClsHead输出分类数 | 2 | \ | - - -### Loss ([ppocr/losses](../../ppocr/losses)) - -| 字段 | 用途 | 默认值 | 备注 | -| :---------------------: | :---------------------: | :--------------: | :--------------------: | -| name | 网络loss类名 | CTCLoss | 目前支持`CTCLoss`,`DBLoss`,`ClsLoss` | -| balance_loss | DBLossloss中是否对正负样本数量进行均衡(使用OHEM) | True | \ | -| ohem_ratio | DBLossloss中的OHEM的负正样本比例 | 3 | \ | -| main_loss_type | DBLossloss中shrink_map所采用的loss | DiceLoss | 支持`DiceLoss`,`BCELoss` | -| alpha | DBLossloss中shrink_map_loss的系数 | 5 | \ | -| beta | DBLossloss中threshold_map_loss的系数 | 10 | \ | - -### PostProcess ([ppocr/postprocess](../../ppocr/postprocess)) - -| 字段 | 用途 | 默认值 | 备注 | -| :---------------------: | :---------------------: | :--------------: | :--------------------: | -| name | 后处理类名 | CTCLabelDecode | 目前支持`CTCLoss`,`AttnLabelDecode`,`DBPostProcess`,`ClsPostProcess` | -| thresh | DBPostProcess中分割图进行二值化的阈值 | 0.3 | \ | -| box_thresh | DBPostProcess中对输出框进行过滤的阈值,低于此阈值的框不会输出 | 0.7 | \ | -| max_candidates | DBPostProcess中输出的最大文本框数量 | 1000 | | -| unclip_ratio | DBPostProcess中对文本框进行放大的比例 | 2.0 | \ | - -### Metric ([ppocr/metrics](../../ppocr/metrics)) - -| 字段 | 用途 | 默认值 | 备注 | -| :---------------------: | :---------------------: | :--------------: | :--------------------: | -| name | 指标评估方法名称 | CTCLabelDecode | 目前支持`DetMetric`,`RecMetric`,`ClsMetric` | -| main_indicator | 主要指标,用于选取最优模型 | acc | 对于检测方法为hmean,识别和分类方法为acc | - -### Dataset ([ppocr/data](../../ppocr/data)) -| 字段 | 用途 | 默认值 | 备注 | -| :---------------------: | :---------------------: | :--------------: | :--------------------: | -| **dataset** | 每次迭代返回一个样本 | - | - | -| name | dataset类名 | SimpleDataSet | 目前支持`SimpleDataSet`和`LMDBDataSet` | -| data_dir | 数据集图片存放路径 | ./train_data | \ | -| label_file_list | 数据标签路径 | ["./train_data/train_list.txt"] | dataset为LMDBDataSet时不需要此参数 | -| ratio_list | 数据集的比例 | [1.0] | 若label_file_list中有两个train_list,且ratio_list为[0.4,0.6],则从train_list1中采样40%,从train_list2中采样60%组合整个dataset | -| transforms | 对图片和标签进行变换的方法列表 | [DecodeImage,CTCLabelEncode,RecResizeImg,KeepKeys] | 见[ppocr/data/imaug](../../ppocr/data/imaug) | -| **loader** | dataloader相关 | - | | -| shuffle | 每个epoch是否将数据集顺序打乱 | True | \ | -| batch_size_per_card | 训练时单卡batch size | 256 | \ | -| drop_last | 是否丢弃因数据集样本数不能被 batch_size 整除而产生的最后一个不完整的mini-batch | True | \ | -| num_workers | 用于加载数据的子进程个数,若为0即为不开启子进程,在主进程中进行数据加载 | 8 | \ | - - - -## 3. 多语言配置文件生成 - -PaddleOCR目前已支持80种(除中文外)语种识别,`configs/rec/multi_languages` 路径下提供了一个多语言的配置文件模版: [rec_multi_language_lite_train.yml](../../configs/rec/multi_language/rec_multi_language_lite_train.yml)。 - -您有两种方式创建所需的配置文件: - -1. 通过脚本自动生成 - -[generate_multi_language_configs.py](../../configs/rec/multi_language/generate_multi_language_configs.py) 可以帮助您生成多语言模型的配置文件 - -- 以意大利语为例,如果您的数据是按如下格式准备的: - ``` - |-train_data - |- it_train.txt # 训练集标签 - |- it_val.txt # 验证集标签 - |- data - |- word_001.jpg - |- word_002.jpg - |- word_003.jpg - | ... - ``` - - 可以使用默认参数,生成配置文件: - - ```bash - # 该代码需要在指定目录运行 - cd PaddleOCR/configs/rec/multi_language/ - # 通过-l或者--language参数设置需要生成的语种的配置文件,该命令会将默认参数写入配置文件 - python3 generate_multi_language_configs.py -l it - ``` - -- 如果您的数据放置在其他位置,或希望使用自己的字典,可以通过指定相关参数来生成配置文件: - - ```bash - # -l或者--language字段是必须的 - # --train修改训练集,--val修改验证集,--data_dir修改数据集目录,--dict修改字典路径, -o修改对应默认参数 - cd PaddleOCR/configs/rec/multi_language/ - python3 generate_multi_language_configs.py -l it \ # 语种 - --train {path/of/train_label.txt} \ # 训练标签文件的路径 - --val {path/of/val_label.txt} \ # 验证集标签文件的路径 - --data_dir {train_data/path} \ # 训练数据的根目录 - --dict {path/of/dict} \ # 字典文件路径 - -o Global.use_gpu=False # 是否使用gpu - ... - - ``` - -意大利文由拉丁字母组成,因此执行完命令后会得到名为 rec_latin_lite_train.yml 的配置文件。 - -2. 手动修改配置文件 - - 您也可以手动修改模版中的以下几个字段得到配置文件: - - ``` - Global: - use_gpu: True - epoch_num: 500 - ... - character_dict_path: {path/of/dict} # 字典文件所在路径 - - Train: - dataset: - name: SimpleDataSet - data_dir: train_data/ # 数据存放根目录 - label_file_list: ["./train_data/train_list.txt"] # 训练集label路径 - ... - - Eval: - dataset: - name: SimpleDataSet - data_dir: train_data/ # 数据存放根目录 - label_file_list: ["./train_data/val_list.txt"] # 验证集label路径 - ... - - ``` - -目前PaddleOCR支持的多语言算法有: - -| 配置文件 | 算法名称 | backbone | trans | seq | pred | language | -| :--------: | :-------: | :-------: | :-------: | :-----: | :-----: | :-----: | -| rec_chinese_cht_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 中文繁体 | -| rec_en_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 英语(区分大小写) | -| rec_french_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 法语 | -| rec_ger_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 德语 | -| rec_japan_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 日语 | -| rec_korean_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 韩语 | -| rec_latin_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 拉丁字母 | -| rec_arabic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 阿拉伯字母 | -| rec_cyrillic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 斯拉夫字母 | -| rec_devanagari_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 梵文字母 | - -更多支持语种请参考: [多语言模型](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/doc/doc_ch/multi_languages.md) diff --git a/doc/doc_ch/customize.md b/doc/doc_ch/customize.md deleted file mode 100644 index 3da61ab44b..0000000000 --- a/doc/doc_ch/customize.md +++ /dev/null @@ -1,30 +0,0 @@ -# 如何生产自定义超轻量模型? - -生产自定义的超轻量模型可分为三步:训练文本检测模型、训练文本识别模型、模型串联预测。 - -## step1:训练文本检测模型 - -PaddleOCR提供了EAST、DB两种文本检测算法,均支持MobileNetV3、ResNet50_vd两种骨干网络,根据需要选择相应的配置文件,启动训练。例如,训练使用MobileNetV3作为骨干网络的DB检测模型(即超轻量模型使用的配置): -``` -python3 tools/train.py -c configs/det/det_mv3_db.yml 2>&1 | tee det_db.log -``` -更详细的数据准备和训练教程参考文档教程中[文本检测模型训练/评估/预测](./detection.md)。 - -## step2:训练文本识别模型 - -PaddleOCR提供了CRNN、Rosetta、STAR-Net、RARE四种文本识别算法,均支持MobileNetV3、ResNet34_vd两种骨干网络,根据需要选择相应的配置文件,启动训练。例如,训练使用MobileNetV3作为骨干网络的CRNN识别模型(即超轻量模型使用的配置): -``` -python3 tools/train.py -c configs/rec/rec_chinese_lite_train.yml 2>&1 | tee rec_ch_lite.log -``` -更详细的数据准备和训练教程参考文档教程中[文本识别模型训练/评估/预测](./recognition.md)。 - -## step3:模型串联预测 - -PaddleOCR提供了检测和识别模型的串联工具,可以将训练好的任一检测模型和任一识别模型串联成两阶段的文本识别系统。输入图像经过文本检测、检测框矫正、文本识别、得分过滤四个主要阶段输出文本位置和识别结果,同时可选择对结果进行可视化。 - -在执行预测时,需要通过参数image_dir指定单张图像或者图像集合的路径、参数det_model_dir指定检测inference模型的路径和参数rec_model_dir指定识别inference模型的路径。可视化识别结果默认保存到 ./inference_results 文件夹里面。 - -``` -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/11.jpg" --det_model_dir="./inference/det/" --rec_model_dir="./inference/rec/" -``` -更多的文本检测、识别串联推理使用方式请参考文档教程中的[基于预测引擎推理](./algorithm_inference.md)。 diff --git a/doc/doc_ch/data_annotation.md b/doc/doc_ch/data_annotation.md deleted file mode 100644 index bd578f7283..0000000000 --- a/doc/doc_ch/data_annotation.md +++ /dev/null @@ -1,27 +0,0 @@ -## 数据标注工具 -这里整理了常用的数据标注工具,持续更新中,欢迎各位小伙伴贡献工具~ - - -### 1. labelImg -- 工具描述:矩形标注 -- 工具地址:https://github.com/tzutalin/labelImg -- 示意图: - ![](../datasets/labelimg.jpg) - -### 2. roLabelImg -- 工具描述:基于labelImg重写的标注工具,支持旋转矩形标注 -- 工具地址:https://github.com/cgvict/roLabelImg -- 示意图: - ![](../datasets/roLabelImg.png) - -### 3. labelme -- 工具描述:支持四点、多边形、圆形等多种标注 -- 工具地址:https://github.com/wkentaro/labelme -- 示意图: - ![](../datasets/labelme.jpg) - -### 4. Vott -- 工具描述:支持矩形,多边形等图片标注.支持视频标注.方便使用的快捷键以及比较好看的界面.同时支持导出多种标签格式. -- 工具地址:https://github.com/microsoft/VoTT -- 示意图: - ![](../datasets/VoTT.jpg) diff --git a/doc/doc_ch/data_synthesis.md b/doc/doc_ch/data_synthesis.md deleted file mode 100644 index 0b2b95af16..0000000000 --- a/doc/doc_ch/data_synthesis.md +++ /dev/null @@ -1,9 +0,0 @@ -# 数据合成工具 -除了开源数据,用户还可使用合成工具自行合成。这里整理了常用的数据合成工具,持续更新中,欢迎各位小伙伴贡献工具~ -- [text_renderer](https://github.com/Sanster/text_renderer) -- [SynthText](https://github.com/ankush-me/SynthText) -- [SynthText_Chinese_version](https://github.com/JarveeLee/SynthText_Chinese_version) -- [TextRecognitionDataGenerator](https://github.com/Belval/TextRecognitionDataGenerator) -- [SynthText3D](https://github.com/MhLiao/SynthText3D) -- [UnrealText](https://github.com/Jyouhou/UnrealText/) -- [SynthTIGER](https://github.com/clovaai/synthtiger) diff --git a/doc/doc_ch/dataset/datasets.md b/doc/doc_ch/dataset/datasets.md deleted file mode 100644 index 84fd9650b3..0000000000 --- a/doc/doc_ch/dataset/datasets.md +++ /dev/null @@ -1,88 +0,0 @@ -## 通用中英文OCR数据集 -这里整理了常用中文数据集,持续更新中,欢迎各位小伙伴贡献数据集~ -- [ICDAR2019-LSVT](#ICDAR2019-LSVT) -- [ICDAR2017-RCTW-17](#ICDAR2017-RCTW-17) -- [中文街景文字识别](#中文街景文字识别) -- [中文文档文字识别](#中文文档文字识别) -- [ICDAR2019-ArT](#ICDAR2019-ArT) -- [电子印章数据集](#电子印章数据集) - -除了开源数据,用户还可使用合成工具自行合成,可参考[数据合成工具](../data_synthesis.md); - -如果需要标注自己的数据,可参考[数据标注工具](../data_annotation.md)。 - - -#### 1、ICDAR2019-LSVT -- **数据来源**:https://ai.baidu.com/broad/introduction?dataset=lsvt -- **数据简介**: 共45w中文街景图像,包含5w(2w测试+3w训练)全标注数据(文本坐标+文本内容),40w弱标注数据(仅文本内容),如下图所示: - ![](../../datasets/LSVT_1.jpg) - (a) 全标注数据 - ![](../../datasets/LSVT_2.jpg) - (b) 弱标注数据 -- **下载地址**:https://ai.baidu.com/broad/download?dataset=lsvt -- **说明**:其中,test数据集的label目前没有开源,如要评估结果,可以去官网提交:https://rrc.cvc.uab.es/?ch=16 - - -#### 2、ICDAR2017-RCTW-17 -- **数据来源**:https://rctw.vlrlab.net/ -- **数据简介**:共包含12,000+图像,大部分图片是通过手机摄像头在野外采集的。有些是截图。这些图片展示了各种各样的场景,包括街景、海报、菜单、室内场景和手机应用程序的截图。 - ![](../../datasets/rctw.jpg) -- **下载地址**:https://rctw.vlrlab.net/dataset/ - - -#### 3、中文街景文字识别 -- **数据来源**:https://aistudio.baidu.com/aistudio/competition/detail/8 -- **数据简介**:ICDAR2019-LSVT行识别任务,共包括29万张图片,其中21万张图片作为训练集(带标注),8万张作为测试集(无标注)。数据集采自中国街景,并由街景图片中的文字行区域(例如店铺标牌、地标等等)截取出来而形成。所有图像都经过一些预处理,将文字区域利用仿射变化,等比映射为一张高为48像素的图片,如图所示: - ![](../../datasets/ch_street_rec_1.png) - (a) 标注:魅派集成吊顶 - ![](../../datasets/ch_street_rec_2.png) - (b) 标注:母婴用品连锁 -- **下载地址** -https://aistudio.baidu.com/aistudio/datasetdetail/8429 - - -#### 4、中文文档文字识别 -- **数据来源**:https://github.com/YCG09/chinese_ocr -- **数据简介**: - - 共约364万张图片,按照99:1划分成训练集和验证集。 - - 数据利用中文语料库(新闻 + 文言文),通过字体、大小、灰度、模糊、透视、拉伸等变化随机生成 - - 包含汉字、英文字母、数字和标点共5990个字符(字符集合:https://github.com/YCG09/chinese_ocr/blob/master/train/char_std_5990.txt ) - - 每个样本固定10个字符,字符随机截取自语料库中的句子 - - 图片分辨率统一为280x32 - ![](../../datasets/ch_doc1.jpg) - ![](../../datasets/ch_doc3.jpg) -- **下载地址**:https://pan.baidu.com/s/1QkI7kjah8SPHwOQ40rS1Pw (密码:lu7m) - - -#### 5、ICDAR2019-ArT -- **数据来源**:https://ai.baidu.com/broad/introduction?dataset=art -- **数据简介**:共包含10,166张图像,训练集5603图,测试集4563图。由Total-Text、SCUT-CTW1500、Baidu Curved Scene Text (ICDAR2019-LSVT部分弯曲数据) 三部分组成,包含水平、多方向和弯曲等多种形状的文本。 - ![](../../datasets/ArT.jpg) -- **下载地址**:https://ai.baidu.com/broad/download?dataset=art - - -#### 6、电子印章数据集 -- **数据来源**:https://aistudio.baidu.com/aistudio/datasetdetail/154271/0 -- **数据简介**:共包含10000张图像,训练集8000图,测试集2000图。数据集是用程序合成的,并不涉及隐私安全,主要用于印章弯曲文本的训练与检测。由开发者[jingsongliujing](https://github.com/jingsongliujing)贡献 -- **下载地址**:https://aistudio.baidu.com/aistudio/datasetdetail/154271/0 - -## 参考文献 -**ICDAR 2019-LSVT Challenge** -``` -@article{sun2019icdar, - title={ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling--RRC-LSVT}, - author={Sun, Yipeng and Ni, Zihan and Chng, Chee-Kheng and Liu, Yuliang and Luo, Canjie and Ng, Chun Chet and Han, Junyu and Ding, Errui and Liu, Jingtuo and Karatzas, Dimosthenis and others}, - journal={arXiv preprint arXiv:1909.07741}, - year={2019} -} -``` - -**ICDAR 2019-ArT Challenge** -``` -@article{chng2019icdar2019, - title={ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)}, - author={Chng, Chee-Kheng and Liu, Yuliang and Sun, Yipeng and Ng, Chun Chet and Luo, Canjie and Ni, Zihan and Fang, ChuanMing and Zhang, Shuaitao and Han, Junyu and Ding, Errui and others}, - journal={arXiv preprint arXiv:1909.07145}, - year={2019} -} -``` diff --git a/doc/doc_ch/dataset/handwritten_datasets.md b/doc/doc_ch/dataset/handwritten_datasets.md deleted file mode 100644 index 6485870cdf..0000000000 --- a/doc/doc_ch/dataset/handwritten_datasets.md +++ /dev/null @@ -1,28 +0,0 @@ -# 手写OCR数据集 -这里整理了常用手写数据集,持续更新中,欢迎各位小伙伴贡献数据集~ -- [中科院自动化研究所-手写中文数据集](#中科院自动化研究所-手写中文数据集) -- [NIST手写单字数据集-英文](#NIST手写单字数据集-英文) - - -## 中科院自动化研究所-手写中文数据集 -- **数据来源**:http://www.nlpr.ia.ac.cn/databases/handwriting/Download.html -- **数据简介**: - * 包含在线和离线两类手写数据,`HWDB1.0~1.2`总共有3895135个手写单字样本,分属7356类(7185个汉字和171个英文字母、数字、符号);`HWDB2.0~2.2`总共有5091页图像,分割为52230个文本行和1349414个文字。所有文字和文本样本均存为灰度图像。部分单字样本图片如下所示。 - - ![](../../datasets/CASIA_0.jpg) - -- **下载地址**:http://www.nlpr.ia.ac.cn/databases/handwriting/Download.html -- **使用建议**:数据为单字,白色背景,可以大量合成文字行进行训练。白色背景可以处理成透明状态,方便添加各种背景。对于需要语义的情况,建议从真实语料出发,抽取单字组成文字行 - - - -## NIST手写单字数据集-英文(NIST Handprinted Forms and Characters Database) - -- **数据来源**: [https://www.nist.gov/srd/nist-special-database-19](https://www.nist.gov/srd/nist-special-database-19) - -- **数据简介**: NIST19数据集适用于手写文档和字符识别的模型训练,从3600位作者的手写样本表格中提取得到,总共包含81万张字符图片。其中9张图片示例如下。 - - ![](../../datasets/nist_demo.png) - - -- **下载地址**: [https://www.nist.gov/srd/nist-special-database-19](https://www.nist.gov/srd/nist-special-database-19) diff --git a/doc/doc_ch/dataset/kie_datasets.md b/doc/doc_ch/dataset/kie_datasets.md deleted file mode 100644 index be5624dbf2..0000000000 --- a/doc/doc_ch/dataset/kie_datasets.md +++ /dev/null @@ -1,53 +0,0 @@ -# 关键信息抽取数据集 - -这里整理了常见的关键信息抽取数据集,持续更新中,欢迎各位小伙伴贡献数据集~ - -- [FUNSD数据集](#funsd) -- [XFUND数据集](#xfund) -- [wildreceipt数据集](#wildreceipt) - - - -## 1. FUNSD数据集 - -- **数据来源**:https://guillaumejaume.github.io/FUNSD/ -- **数据简介**:FUNSD数据集是一个用于表单理解的数据集,它包含199张真实的、完全标注的扫描版图片,类型包括市场报告、广告以及学术报告等,并分为149张训练集以及50张测试集。FUNSD数据集适用于多种类型的DocVQA任务,如字段级实体分类、字段级实体连接等。部分图像以及标注框可视化如下所示: -
- - -
- 图中,橙色区域代表`header`,淡蓝色区域代表`question`, 绿色区域表`answer`,粉红色代区域表`other`。 - -- **下载地址**:https://guillaumejaume.github.io/FUNSD/download/ - - - -## 2. XFUND数据集 -- **数据来源**:https://github.com/doc-analysis/XFUND -- **数据简介**:XFUND是一个多语种表单理解数据集,它包含7种不同语种的表单数据,并且全部用人工进行了键-值对形式的标注。其中每个语种的数据都包含了199张表单数据,并分为149张训练集以及50张测试集。部分图像以及标注框可视化如下所示: - -
- - -
- -- **下载地址**:https://github.com/doc-analysis/XFUND/releases/tag/v1.0 - - - - -## 3. wildreceipt数据集 - -- **数据来源**:https://arxiv.org/abs/2103.14470 -- **数据简介**:wildreceipt数据集是英文发票数据集,包含26个类别(此处类别体系包含`Ignore`类别),共标注了50000个文本框。其中训练集包含1267张图片,测试集包含472张图片。部分图像以及标注框可视化如下所示: - -
- - -
- -**注:** 这里对于类别为`Ignore`或者`Others`的文本,没有进行可视化。 - -- **下载地址**: - - 原始数据下载地址:[链接](https://download.openmmlab.com/mmocr/data/wildreceipt.tar) - - 数据格式转换后适配于PaddleOCR训练的数据下载地址:[链接](https://paddleocr.bj.bcebos.com/ppstructure/dataset/wildreceipt.tar) diff --git a/doc/doc_ch/dataset/layout_datasets.md b/doc/doc_ch/dataset/layout_datasets.md deleted file mode 100644 index 728a9be5fd..0000000000 --- a/doc/doc_ch/dataset/layout_datasets.md +++ /dev/null @@ -1,53 +0,0 @@ -## 版面分析数据集 - -这里整理了常用版面分析数据集,持续更新中,欢迎各位小伙伴贡献数据集~ -- [publaynet数据集](#publaynet) -- [CDLA数据集](#CDLA) -- [TableBank数据集](#TableBank) - -版面分析数据集多为目标检测数据集,除了开源数据,用户还可使用合成工具自行合成,如[labelme](https://github.com/wkentaro/labelme)等。 - - - - -#### 1、publaynet数据集 -- **数据来源**:https://github.com/ibm-aur-nlp/PubLayNet -- **数据简介**:publaynet数据集的训练集合中包含35万张图像,验证集合中包含1.1万张图像。总共包含5个类别,分别是: `text, title, list, table, figure`。部分图像以及标注框可视化如下所示。 - -
- - -
- -- **下载地址**:https://developer.ibm.com/exchanges/data/all/publaynet/ -- **说明**:使用该数据集时,需要遵守[CDLA-Permissive](https://cdla.io/permissive-1-0/)协议。 - - - - -#### 2、CDLA数据集 -- **数据来源**:https://github.com/buptlihang/CDLA -- **数据简介**:CDLA据集的训练集合中包含5000张图像,验证集合中包含1000张图像。总共包含10个类别,分别是: `Text, Title, Figure, Figure caption, Table, Table caption, Header, Footer, Reference, Equation`。部分图像以及标注框可视化如下所示。 - -
- - -
- -- **下载地址**:https://github.com/buptlihang/CDLA -- **说明**:基于[PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection/tree/develop)套件,在该数据集上训练目标检测模型时,在转换label时,需要将`label.txt`中的`__ignore__`与`_background_`去除。 - - - - -#### 3、TableBank数据集 -- **数据来源**:https://doc-analysis.github.io/tablebank-page/index.html -- **数据简介**:TableBank数据集包含Latex(训练集187199张,验证集7265张,测试集5719张)与Word(训练集73383张,验证集2735张,测试集2281张)两种类别的文档。仅包含`Table` 1个类别。部分图像以及标注框可视化如下所示。 - -
- - -
- -- **下载地址**:https://doc-analysis.github.io/tablebank-page/index.html -- **说明**:使用该数据集时,需要遵守[Apache-2.0](https://github.com/doc-analysis/TableBank/blob/master/LICENSE)协议。 diff --git a/doc/doc_ch/dataset/ocr_datasets.md b/doc/doc_ch/dataset/ocr_datasets.md deleted file mode 100644 index b7666fd63e..0000000000 --- a/doc/doc_ch/dataset/ocr_datasets.md +++ /dev/null @@ -1,165 +0,0 @@ -# OCR数据集 - -- [1. 文本检测](#1-文本检测) - - [1.1 PaddleOCR 文字检测数据格式](#11-paddleocr-文字检测数据格式) - - [1.2 公开数据集](#12-公开数据集) - - [1.2.1 ICDAR 2015](#121-icdar-2015) -- [2. 文本识别](#2-文本识别) - - [2.1 PaddleOCR 文字识别数据格式](#21-paddleocr-文字识别数据格式) - - [2.2 公开数据集](#22-公开数据集) - - [2.1 ICDAR 2015](#21-icdar-2015) -- [3. 数据存放路径](#3-数据存放路径) - -这里整理了OCR中常用的公开数据集,持续更新中,欢迎各位小伙伴贡献数据集~ - -## 1. 文本检测 - -### 1.1 PaddleOCR 文字检测数据格式 - - -PaddleOCR 中的文本检测算法支持的标注文件格式如下,中间用"\t"分隔: -``` -" 图像文件名 json.dumps编码的图像标注信息" -ch4_test_images/img_61.jpg [{"transcription": "MASA", "points": [[310, 104], [416, 141], [418, 216], [312, 179]]}, {...}] -``` -json.dumps编码前的图像标注信息是包含多个字典的list,字典中的 `points` 表示文本框的四个点的坐标(x, y),从左上角的点开始顺时针排列。 -`transcription` 表示当前文本框的文字,**当其内容为“###”时,表示该文本框无效,在训练时会跳过。** - -如果您想在我们未提供的数据集上训练,可以按照上述形式构建标注文件。 - -### 1.2 公开数据集 - -| 数据集名称 |图片下载地址| PaddleOCR 标注下载地址 | -|---|---|---| -| ICDAR 2015 |https://rrc.cvc.uab.es/?ch=4&com=downloads| [train](https://paddleocr.bj.bcebos.com/dataset/train_icdar2015_label.txt) / [test](https://paddleocr.bj.bcebos.com/dataset/test_icdar2015_label.txt) | -| ctw1500 |https://paddleocr.bj.bcebos.com/dataset/ctw1500.zip| 图片下载地址中已包含 | -| total text |https://paddleocr.bj.bcebos.com/dataset/total_text.tar| 图片下载地址中已包含 | -| td tr |https://paddleocr.bj.bcebos.com/dataset/TD_TR.tar| 图片下载地址中已包含 | - -#### 1.2.1 ICDAR 2015 -ICDAR 2015 数据集包含1000张训练图像和500张测试图像。ICDAR 2015 数据集可以从上表中链接下载,首次下载需注册。 -注册完成登陆后,下载下图中红色框标出的部分,其中, `Training Set Images`下载的内容保存在`icdar_c4_train_imgs`文件夹下,`Test Set Images` 下载的内容保存早`ch4_test_images`文件夹下 - -

- -

- -将下载到的数据集解压到工作目录下,假设解压在 PaddleOCR/train_data/下。然后从上表中下载转换好的标注文件。 - -PaddleOCR 也提供了数据格式转换脚本,可以将官网 label 转换支持的数据格式。 数据转换工具在 `ppocr/utils/gen_label.py`, 这里以训练集为例: - -``` -# 将官网下载的标签文件转换为 train_icdar2015_label.txt -python gen_label.py --mode="det" --root_path="/path/to/icdar_c4_train_imgs/" \ - --input_path="/path/to/ch4_training_localization_transcription_gt" \ - --output_label="/path/to/train_icdar2015_label.txt" -``` - -解压数据集和下载标注文件后,PaddleOCR/train_data/ 有两个文件夹和两个文件,按照如下方式组织icdar2015数据集: -``` -/PaddleOCR/train_data/icdar2015/text_localization/ - └─ icdar_c4_train_imgs/ icdar 2015 数据集的训练数据 - └─ ch4_test_images/ icdar 2015 数据集的测试数据 - └─ train_icdar2015_label.txt icdar 2015 数据集的训练标注 - └─ test_icdar2015_label.txt icdar 2015 数据集的测试标注 -``` - -## 2. 文本识别 - -### 2.1 PaddleOCR 文字识别数据格式 - -PaddleOCR 中的文字识别算法支持两种数据格式: - - - `lmdb` 用于训练以lmdb格式存储的数据集,使用 [lmdb_dataset.py](../../../ppocr/data/lmdb_dataset.py) 进行读取; - - `通用数据` 用于训练以文本文件存储的数据集,使用 [simple_dataset.py](../../../ppocr/data/simple_dataset.py)进行读取。 - -下面以通用数据集为例, 介绍如何准备数据集: - -* 训练集 - -建议将训练图片放入同一个文件夹,并用一个txt文件(rec_gt_train.txt)记录图片路径和标签,txt文件里的内容如下: - -**注意:** txt文件中默认请将图片路径和图片标签用 \t 分割,如用其他方式分割将造成训练报错。 - -``` -" 图像文件名 图像标注信息 " - -train_data/rec/train/word_001.jpg 简单可依赖 -train_data/rec/train/word_002.jpg 用科技让复杂的世界更简单 -... -``` - -最终训练集应有如下文件结构: -``` -|-train_data - |-rec - |- rec_gt_train.txt - |- train - |- word_001.png - |- word_002.jpg - |- word_003.jpg - | ... -``` - -除上述单张图像为一行格式之外,PaddleOCR也支持对离线增广后的数据进行训练,为了防止相同样本在同一个batch中被多次采样,我们可以将相同标签对应的图片路径写在一行中,以列表的形式给出,在训练中,PaddleOCR会随机选择列表中的一张图片进行训练。对应地,标注文件的格式如下。 - -``` -["11.jpg", "12.jpg"] 简单可依赖 -["21.jpg", "22.jpg", "23.jpg"] 用科技让复杂的世界更简单 -3.jpg ocr -``` - -上述示例标注文件中,"11.jpg"和"12.jpg"的标签相同,都是`简单可依赖`,在训练的时候,对于该行标注,会随机选择其中的一张图片进行训练。 - - -- 验证集 - -同训练集类似,验证集也需要提供一个包含所有图片的文件夹(test)和一个rec_gt_test.txt,验证集的结构如下所示: - -``` -|-train_data - |-rec - |- rec_gt_test.txt - |- test - |- word_001.jpg - |- word_002.jpg - |- word_003.jpg - | ... -``` - - -### 2.2 公开数据集 - -| 数据集名称 | 图片下载地址 | PaddleOCR 标注下载地址 | -|---|---|---------------------------------------------------------------------| -| en benchmark(MJ, SJ, IIIT, SVT, IC03, IC13, IC15, SVTP, and CUTE.) | [DTRB](https://github.com/clovaai/deep-text-recognition-benchmark#download-lmdb-dataset-for-traininig-and-evaluation-from-here) | LMDB格式,可直接用[lmdb_dataset.py](../../../ppocr/data/lmdb_dataset.py)加载 | -|ICDAR 2015| http://rrc.cvc.uab.es/?ch=4&com=downloads | [train](https://paddleocr.bj.bcebos.com/dataset/rec_gt_train.txt)/ [test](https://paddleocr.bj.bcebos.com/dataset/rec_gt_test.txt) | -| 多语言数据集 |[百度网盘](https://pan.baidu.com/s/1bS_u207Rm7YbY33wOECKDA) 提取码:frgi
[google drive](https://drive.google.com/file/d/18cSWX7wXSy4G0tbKJ0d9PuIaiwRLHpjA/view) | 图片下载地址中已包含 | - -#### 2.1 ICDAR 2015 - -ICDAR 2015 数据集可以在上表中链接下载,用于快速验证。也可以从上表中下载 en benchmark 所需的lmdb格式数据集。 - -下载完图片后从上表中下载转换好的标注文件。 - -PaddleOCR 也提供了数据格式转换脚本,可以将ICDAR官网 label 转换为PaddleOCR支持的数据格式。 数据转换工具在 `ppocr/utils/gen_label.py`, 这里以训练集为例: - -``` -# 将官网下载的标签文件转换为 rec_gt_label.txt -python gen_label.py --mode="rec" --input_path="{path/of/origin/label}" --output_label="rec_gt_label.txt" -``` - -数据样式格式如下,(a)为原始图片,(b)为每张图片对应的 Ground Truth 文本文件: -![](../../datasets/icdar_rec.png) - - -## 3. 数据存放路径 - -PaddleOCR训练数据的默认存储路径是 `PaddleOCR/train_data`,如果您的磁盘上已有数据集,只需创建软链接至数据集目录: - -``` -# linux and mac os -ln -sf /train_data/dataset -# windows -mklink /d /train_data/dataset -``` diff --git a/doc/doc_ch/dataset/table_datasets.md b/doc/doc_ch/dataset/table_datasets.md deleted file mode 100644 index 58f4cf4705..0000000000 --- a/doc/doc_ch/dataset/table_datasets.md +++ /dev/null @@ -1,44 +0,0 @@ -# 表格识别数据集 - -- [数据集汇总](#数据集汇总) -- [1. PubTabNet数据集](#1-pubtabnet数据集) -- [2. 好未来表格识别竞赛数据集](#2-好未来表格识别竞赛数据集) -- [3. 好未来表格识别竞赛数据集](#2-WTW中文场景表格数据集) - -这里整理了常用表格识别数据集,持续更新中,欢迎各位小伙伴贡献数据集~ - -## 数据集汇总 - -| 数据集名称 |图片下载地址| PPOCR标注下载地址 | -|---|---|---| -| PubTabNet |https://github.com/ibm-aur-nlp/PubTabNet| jsonl格式,可直接用[pubtab_dataset.py](../../../ppocr/data/pubtab_dataset.py)加载 | -| 好未来表格识别竞赛数据集 |https://ai.100tal.com/dataset| jsonl格式,可直接用[pubtab_dataset.py](../../../ppocr/data/pubtab_dataset.py)加载 | -| WTW中文场景表格数据集 |https://github.com/wangwen-whu/WTW-Dataset| 需要进行转换后才能用[pubtab_dataset.py](../../../ppocr/data/pubtab_dataset.py)加载 | - -## 1. PubTabNet数据集 -- **数据简介**:PubTabNet数据集的训练集合中包含50万张图像,验证集合中包含0.9万张图像。部分图像可视化如下所示。 - - -

- - -
- -- **说明**:使用该数据集时,需要遵守[CDLA-Permissive](https://cdla.io/permissive-1-0/)协议。 - -## 2. 好未来表格识别竞赛数据集 -- **数据简介**:好未来表格识别竞赛数据集的训练集合中包含1.6万张图像。验证集未给出可训练的标注。 - -
- - -
- -## 3. WTW中文场景表格数据集 -- **数据简介**:WTW中文场景表格数据集包含表格检测和表格数据两部分数据,数据集中同时包含扫描和拍照两张场景的图像。 - -https://github.com/wangwen-whu/WTW-Dataset/blob/main/demo/20210816_210413.gif - -
- -
diff --git a/doc/doc_ch/dataset/vertical_and_multilingual_datasets.md b/doc/doc_ch/dataset/vertical_and_multilingual_datasets.md deleted file mode 100644 index 095b7713e1..0000000000 --- a/doc/doc_ch/dataset/vertical_and_multilingual_datasets.md +++ /dev/null @@ -1,79 +0,0 @@ -# 垂类多语言OCR数据集 -这里整理了常用垂类和多语言OCR数据集,持续更新中,欢迎各位小伙伴贡献数据集~ -- [中国城市车牌数据集](#中国城市车牌数据集) -- [银行信用卡数据集](#银行信用卡数据集) -- [验证码数据集-Captcha](#验证码数据集-Captcha) -- [多语言数据集](#多语言数据集) - - - -## 中国城市车牌数据集 - -- **数据来源**:[https://github.com/detectRecog/CCPD](https://github.com/detectRecog/CCPD) - -- **数据简介**: 包含超过25万张中国城市车牌图片及车牌检测、识别信息的标注。包含以下几种不同场景中的车牌图片信息。 - * CCPD-Base: 通用车牌图片 - * CCPD-DB: 车牌区域亮度较亮、较暗或者不均匀 - * CCPD-FN: 车牌离摄像头拍摄位置相对更远或者更近 - * CCPD-Rotate: 车牌包含旋转(水平20\~50度,竖直-10\~10度) - * CCPD-Tilt: 车牌包含旋转(水平15\~45度,竖直15\~45度) - * CCPD-Blur: 车牌包含由于摄像机镜头抖动导致的模糊情况 - * CCPD-Weather: 车牌在雨天、雪天或者雾天拍摄得到 - * CCPD-Challenge: 至今在车牌检测识别任务中最有挑战性的一些图片 - * CCPD-NP: 没有安装车牌的新车图片。 - - ![](../../datasets/ccpd_demo.png) - - -- **下载地址** - * 百度云下载地址(提取码是hm0U): [https://pan.baidu.com/s/1i5AOjAbtkwb17Zy-NQGqkw](https://pan.baidu.com/s/1i5AOjAbtkwb17Zy-NQGqkw) - * Google drive下载地址:[https://drive.google.com/file/d/1rdEsCUcIUaYOVRkx5IMTRNA7PcGMmSgc/view](https://drive.google.com/file/d/1rdEsCUcIUaYOVRkx5IMTRNA7PcGMmSgc/view) - - - -## 银行信用卡数据集 - -- **数据来源**: [https://www.kesci.com/home/dataset/5954cf1372ead054a5e25870](https://www.kesci.com/home/dataset/5954cf1372ead054a5e25870) - -- **数据简介**: 训练数据共提供了三类数据 - * 1.招行样卡数据: 包括卡面图片数据及标注数据,总共618张图片 - * 2.单字符数据: 包括图片及标注数据,总共37张图片。 - * 3.仅包含其他银行卡面,不具有更细致的信息,总共50张图片。 - - * demo图片展示如下,标注信息存储在excel表格中,下面的demo图片标注为 - * 前8位卡号:62257583 - * 卡片种类:本行卡 - * 有效期结束:07/41 - * 卡用户拼音:MICHAEL - - ![](../../datasets/cmb_demo.jpg) - -- **下载地址**: [https://cdn.kesci.com/cmb2017-2.zip](https://cdn.kesci.com/cmb2017-2.zip) - - - - -## 验证码数据集-Captcha - -- **数据来源**: [https://github.com/lepture/captcha](https://github.com/lepture/captcha) - -- **数据简介**: 这是一个数据合成的工具包,可以根据输入的文本,输出验证码图片,使用该工具包生成几张demo图片如下。 - - ![](../../datasets/captcha_demo.png) - -- **下载地址**: 该数据集是生成得到,无下载地址。 - - - - -## 多语言数据集(Multi-lingual scene text detection and recognition) - -- **数据来源**: [https://rrc.cvc.uab.es/?ch=15&com=downloads](https://rrc.cvc.uab.es/?ch=15&com=downloads) - -- **数据简介**: 多语言检测数据集MLT同时包含了语种识别和检测任务。 - * 在检测任务中,训练集包含10000张图片,共有10种语言,每种语言包含1000张训练图片。测试集包含10000张图片。 - * 在识别任务中,训练集包含111998个样本。 - - -- **下载地址**: 训练集较大,分2部分下载,需要在网站上注册之后才能下载: -[https://rrc.cvc.uab.es/?ch=15&com=downloads](https://rrc.cvc.uab.es/?ch=15&com=downloads) diff --git a/doc/doc_ch/detection.md b/doc/doc_ch/detection.md deleted file mode 100644 index 8271f100a9..0000000000 --- a/doc/doc_ch/detection.md +++ /dev/null @@ -1,265 +0,0 @@ -# 文字检测 - -本节以icdar2015数据集为例,介绍PaddleOCR中检测模型训练、评估、测试的使用方式。 - -- [1. 准备数据和模型](#1-准备数据和模型) - - [1.1 准备数据集](#11-准备数据集) - - [1.2 下载预训练模型](#12-下载预训练模型) -- [2. 开始训练](#2-开始训练) - - [2.1 启动训练](#21-启动训练) - - [2.2 断点训练](#22-断点训练) - - [2.3 更换Backbone 训练](#23-更换backbone-训练) - - [2.4 混合精度训练](#24-混合精度训练) - - [2.5 分布式训练](#25-分布式训练) - - [2.6 知识蒸馏训练](#26-知识蒸馏训练) - - [2.7 其他训练环境](#27-其他训练环境) - - [2.8 模型微调](#28-模型微调) -- [3. 模型评估与预测](#3-模型评估与预测) - - [3.1 指标评估](#31-指标评估) - - [3.2 测试检测效果](#32-测试检测效果) -- [4. 模型导出与预测](#4-模型导出与预测) -- [5. FAQ](#5-faq) - - -# 1. 准备数据和模型 - -## 1.1 准备数据集 - -准备数据集可参考 [ocr_datasets](./dataset/ocr_datasets.md) 。 - - - -## 1.2 下载预训练模型 - -首先下载模型backbone的pretrain model,PaddleOCR的检测模型目前支持两种backbone,分别是MobileNetV3、ResNet_vd系列, -您可以根据需求使用[PaddleClas](https://github.com/PaddlePaddle/PaddleClas/tree/release/2.0/ppcls/modeling/architectures)中的模型更换backbone, -对应的backbone预训练模型可以从[PaddleClas repo 主页中找到下载链接](https://github.com/PaddlePaddle/PaddleClas/blob/release%2F2.0/README_cn.md#resnet%E5%8F%8A%E5%85%B6vd%E7%B3%BB%E5%88%97)。 - -```shell -cd PaddleOCR/ -# 根据backbone的不同选择下载对应的预训练模型 -# 下载MobileNetV3的预训练模型 -wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/MobileNetV3_large_x0_5_pretrained.pdparams -# 或,下载ResNet18_vd的预训练模型 -wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet18_vd_pretrained.pdparams -# 或,下载ResNet50_vd的预训练模型 -wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet50_vd_ssld_pretrained.pdparams -``` - - -# 2. 开始训练 - - -## 2.1 启动训练 - -*如果您安装的是cpu版本,请将配置文件中的 `use_gpu` 字段修改为false* - -```shell -# 单机单卡训练 mv3_db 模型 -python3 tools/train.py -c configs/det/det_mv3_db.yml \ - -o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained - -# 单机多卡训练,通过 --gpus 参数设置使用的GPU ID -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/det_mv3_db.yml \ - -o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained - -``` - -上述指令中,通过-c 选择训练使用configs/det/det_mv3_db.yml配置文件。 -有关配置文件的详细解释,请参考[链接](./config.md)。 - -您也可以通过-o参数在不需要修改yml文件的情况下,改变训练的参数,比如,调整训练的学习率为0.0001 -```shell -python3 tools/train.py -c configs/det/det_mv3_db.yml -o Optimizer.base_lr=0.0001 -``` - - -## 2.2 断点训练 - -如果训练程序中断,如果希望加载训练中断的模型从而恢复训练,可以通过指定Global.checkpoints指定要加载的模型路径: -```shell -python3 tools/train.py -c configs/det/det_mv3_db.yml -o Global.checkpoints=./your/trained/model -``` - -**注意**:`Global.checkpoints`的优先级高于`Global.pretrained_model`的优先级,即同时指定两个参数时,优先加载`Global.checkpoints`指定的模型,如果`Global.checkpoints`指定的模型路径有误,会加载`Global.pretrained_model`指定的模型。 - - -## 2.3 更换Backbone 训练 - -PaddleOCR将网络划分为四部分,分别在[ppocr/modeling](../../ppocr/modeling)下。 进入网络的数据将按照顺序(transforms->backbones-> -necks->heads)依次通过这四个部分。 - -```bash -├── architectures # 网络的组网代码 -├── transforms # 网络的图像变换模块 -├── backbones # 网络的特征提取模块 -├── necks # 网络的特征增强模块 -└── heads # 网络的输出模块 -``` -如果要更换的Backbone 在PaddleOCR中有对应实现,直接修改配置yml文件中`Backbone`部分的参数即可。 - -如果要使用新的Backbone,更换backbones的例子如下: - -1. 在 [ppocr/modeling/backbones](../../ppocr/modeling/backbones) 文件夹下新建文件,如my_backbone.py。 -2. 在 my_backbone.py 文件内添加相关代码,示例代码如下: - -```python -import paddle -import paddle.nn as nn -import paddle.nn.functional as F - - -class MyBackbone(nn.Layer): - def __init__(self, *args, **kwargs): - super(MyBackbone, self).__init__() - # your init code - self.conv = nn.xxxx - - def forward(self, inputs): - # your network forward - y = self.conv(inputs) - return y -``` - -3. 在 [ppocr/modeling/backbones/\__init\__.py](../../ppocr/modeling/backbones/__init__.py)文件内导入添加的`MyBackbone`模块,然后修改配置文件中Backbone进行配置即可使用,格式如下: - -```yaml -Backbone: -name: MyBackbone -args1: args1 -``` - -**注意**:如果要更换网络的其他模块,可以参考[文档](./add_new_algorithm.md)。 - - -## 2.4 混合精度训练 - -如果您想进一步加快训练速度,可以使用[自动混合精度训练](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/01_paddle2.0_introduction/basic_concept/amp_cn.html), 以单机单卡为例,命令如下: - -```shell -python3 tools/train.py -c configs/det/det_mv3_db.yml \ - -o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained \ - Global.use_amp=True Global.scale_loss=1024.0 Global.use_dynamic_loss_scaling=True - ``` -**注意** -文本检测模型使用AMP时可能遇到训练不收敛问题,可以参考[discussions](https://github.com/PaddlePaddle/PaddleOCR/discussions/12445)中的临时解决方案进行使用。 - - - -## 2.5 分布式训练 - -多机多卡训练时,通过 `--ips` 参数设置使用的机器IP地址,通过 `--gpus` 参数设置使用的GPU ID: - -```bash -python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/det/det_mv3_db.yml \ - -o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained -``` - -**注意:** (1)采用多机多卡训练时,需要替换上面命令中的ips值为您机器的地址,机器之间需要能够相互ping通;(2)训练时需要在多个机器上分别启动命令。查看机器ip地址的命令为`ifconfig`;(3)更多关于分布式训练的性能优势等信息,请参考:[分布式训练教程](./distributed_training.md)。 - - - - -## 2.6 知识蒸馏训练 - -PaddleOCR支持了基于知识蒸馏的检测模型训练过程,更多内容可以参考[知识蒸馏说明文档](./knowledge_distillation.md)。 - -**注意:** 知识蒸馏训练目前只支持PP-OCR使用的`DB`和`CRNN`算法。 - - - -## 2.7 其他训练环境 - -- Windows GPU/CPU -在Windows平台上与Linux平台略有不同: -Windows平台只支持`单卡`的训练与预测,指定GPU进行训练`set CUDA_VISIBLE_DEVICES=0` -在Windows平台,DataLoader只支持单进程模式,因此需要设置 `num_workers` 为0; - -- macOS -不支持GPU模式,需要在配置文件中设置`use_gpu`为False,其余训练评估预测命令与Linux GPU完全相同。 - -- Linux DCU -DCU设备上运行需要设置环境变量 `export HIP_VISIBLE_DEVICES=0,1,2,3`,其余训练评估预测命令与Linux GPU完全相同。 - - - -## 2.8 模型微调 - -实际使用过程中,建议加载官方提供的预训练模型,在自己的数据集中进行微调,关于检测模型的微调方法,请参考:[模型微调教程](./finetune.md)。 - - - -# 3. 模型评估与预测 - - -## 3.1 指标评估 - -PaddleOCR计算三个OCR检测相关的指标,分别是:Precision、Recall、Hmean(F-Score)。 - -训练中模型参数默认保存在`Global.save_model_dir`目录下。在评估指标时,需要设置`Global.checkpoints`指向保存的参数文件。 - -```shell -python3 tools/eval.py -c configs/det/det_mv3_db.yml -o Global.checkpoints="{path/to/weights}/best_accuracy" -``` - - - -## 3.2 测试检测效果 - -测试单张图像的检测效果: - -```shell -python3 tools/infer_det.py -c configs/det/det_mv3_db.yml -o Global.infer_img="./doc/imgs_en/img_10.jpg" Global.pretrained_model="./output/det_db/best_accuracy" -``` - -测试DB模型时,调整后处理阈值: -```shell -python3 tools/infer_det.py -c configs/det/det_mv3_db.yml -o Global.infer_img="./doc/imgs_en/img_10.jpg" Global.pretrained_model="./output/det_db/best_accuracy" PostProcess.box_thresh=0.6 PostProcess.unclip_ratio=2.0 -``` -* 注:`box_thresh`、`unclip_ratio`是DB后处理参数,其他检测模型不支持。 - -测试文件夹下所有图像的检测效果: -```shell -python3 tools/infer_det.py -c configs/det/det_mv3_db.yml -o Global.infer_img="./doc/imgs_en/" Global.pretrained_model="./output/det_db/best_accuracy" -``` - - -# 4. 模型导出与预测 - -inference 模型(`paddle.jit.save`保存的模型) -一般是模型训练,把模型结构和模型参数保存在文件中的固化模型,多用于预测部署场景。 -训练过程中保存的模型是checkpoints模型,保存的只有模型的参数,多用于恢复训练等。 -与checkpoints模型相比,inference 模型会额外保存模型的结构信息,在预测部署、加速推理上性能优越,灵活方便,适合于实际系统集成。 - -检测模型转inference 模型方式: -```shell -# 加载配置文件`det_mv3_db.yml`,从`output/det_db`目录下加载`best_accuracy`模型,inference模型保存在`./output/det_db_inference`目录下 -python3 tools/export_model.py -c configs/det/det_mv3_db.yml -o Global.pretrained_model="./output/det_db/best_accuracy" Global.save_inference_dir="./output/det_db_inference/" -``` - -DB检测模型inference 模型预测: - -```shell -python3 tools/infer/predict_det.py --det_algorithm="DB" --det_model_dir="./output/det_db_inference/" --image_dir="./doc/imgs/" --use_gpu=True -``` -如果是其他检测,比如EAST模型,det_algorithm参数需要修改为EAST,默认为DB算法: - -```shell -python3 tools/infer/predict_det.py --det_algorithm="EAST" --det_model_dir="./output/det_db_inference/" --image_dir="./doc/imgs/" --use_gpu=True -``` - -更多关于推理超参数的配置与解释,请参考:[模型推理超参数解释教程](./inference_args.md)。 - - - -# 5. FAQ - -Q1: 训练模型转inference 模型之后预测效果不一致? - -**A**:此类问题出现较多,问题多是trained model预测时候的预处理、后处理参数和inference model预测的时候的预处理、后处理参数不一致导致的。以det_mv3_db.yml配置文件训练的模型为例,训练模型、inference模型预测结果不一致问题解决方式如下: -- 检查[trained model预处理](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/configs/det/det_mv3_db.yml#L116),和[inference model的预测预处理](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/tools/infer/predict_det.py#L42)函数是否一致。算法在评估的时候,输入图像大小会影响精度,为了和论文保持一致,训练icdar15配置文件中将图像resize到[736, 1280],但是在inference model预测的时候只有一套默认参数,会考虑到预测速度问题,默认限制图像最长边为960做resize的。训练模型预处理和inference模型的预处理函数位于[ppocr/data/imaug/operators.py](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/ppocr/data/imaug/operators.py#L147) -- 检查[trained model后处理](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/configs/det/det_mv3_db.yml#L51),和[inference 后处理参数](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/tools/infer/utility.py#L50)是否一致。 - -Q1: 训练EAST模型提示找不到lanms库? - -**A**:执行pip3 install lanms-nova 即可。 diff --git a/doc/doc_ch/distributed_training.md b/doc/doc_ch/distributed_training.md deleted file mode 100644 index dbbc4dc8b7..0000000000 --- a/doc/doc_ch/distributed_training.md +++ /dev/null @@ -1,70 +0,0 @@ -# 分布式训练 - -## 简介 - -* 分布式训练的高性能,是飞桨的核心优势技术之一,在分类任务上,分布式训练可以达到几乎线性的加速比。OCR训练任务中往往包含大量训练数据,以识别为例,ppocrv2.0模型在训练时使用了1800W数据,如果使用单机训练,会非常耗时。因此,PaddleOCR中使用分布式训练接口完成训练任务,同时支持单机训练与多机训练。更多关于分布式训练的方法与文档可以参考:[分布式训练快速开始教程](https://fleet-x.readthedocs.io/en/latest/paddle_fleet_rst/parameter_server/ps_quick_start.html)。 - -## 使用方法 - -### 单机训练 - -* 以识别为例,本地准备好数据之后,使用`paddle.distributed.launch`的接口启动训练任务即可。下面为运行代码示例。 - -```shell -python3 -m paddle.distributed.launch \ - --log_dir=./log/ \ - --gpus "0,1,2,3,4,5,6,7" \ - tools/train.py \ - -c configs/rec/rec_mv3_none_bilstm_ctc.yml -``` - -### 多机训练 - -* 相比单机训练,多机训练时,只需要添加`--ips`的参数,该参数表示需要参与分布式训练的机器的ip列表,不同机器的ip用逗号隔开。下面为运行代码示例。 - - -```shell -ip_list="192.168.0.1,192.168.0.2" -python3 -m paddle.distributed.launch \ - --log_dir=./log/ \ - --ips="${ip_list}" \ - --gpus="0,1,2,3,4,5,6,7" \ - tools/train.py \ - -c configs/rec/rec_mv3_none_bilstm_ctc.yml -``` - -**注:** -* 不同机器的ip信息需要用逗号隔开,可以通过`ifconfig`或者`ipconfig`查看。 -* 不同机器之间需要做免密设置,且可以直接ping通,否则无法完成通信。 -* 不同机器之间的代码、数据与运行命令或脚本需要保持一致,且所有的机器上都需要运行设置好的训练命令或者脚本。最终`ip_list`中的第一台机器的第一块设备是trainer0,以此类推。 - - -## 性能效果测试 - -* 在2机8卡P40的机器上进行模型训练,不同模型的精度、训练耗时、多机加速比情况如下所示。 - -| 模型 | 配置 | 数据集 | 单机8卡耗时/精度 | 2机8卡耗时/精度 | 加速比 | -|:------:|:-----:|:--------:|:--------:|:--------:|:-----:| -| CRNN | [rec_chinese_lite_train_v2.0.yml](../../configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml) | 26W中文数据集 | 2.50d/66.7% | 1.67d/67.0% | **1.5** | - - -* 在3机8卡V100的机器上进行模型训练,不同模型的精度、训练耗时、多机加速比情况如下所示。 - -| 模型 | 配置 | 数据集 | 单机8卡耗时/精度 | 3机8卡耗时/精度 | 加速比 | -|:------:|:-----:|:--------:|:--------:|:--------:|:-----:| -| SLANet | [SLANet.yml](../../configs/table/SLANet.yml) | PubTabNet | 49.8h/76.2% | 19.75h/74.77% | **2.52** | - - - > 注意:这里3机8卡训练时,单卡batch size相比于单机8卡不变,学习率乘以2 (默认乘以3的话,精度仅有73.42%) - - -* 在4机8卡V100的机器上进行模型训练,不同模型的精度、训练耗时、多机加速比情况如下所示。 - - -| 模型 | 配置 | 数据集 | 单机8卡耗时/精度 | 4机8卡耗时/精度 | 加速比 | -|:------:|:-----:|:--------:|:--------:|:--------:|:-----:| -| SVTR | [ch_PP-OCRv3_rec_distillation.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml) | PP-OCRv3_rec data | 10d/- | 2.84d/74.0% | **3.5** | - - -* **注意** - * 在训练的GPU卡数过多时,精度会稍微有所损失(1%左右),此时可以尝试通过添加warmup或者适当增加迭代轮数来弥补精度损失。 diff --git a/doc/doc_ch/enhanced_ctc_loss.md b/doc/doc_ch/enhanced_ctc_loss.md deleted file mode 100644 index 4be16f3220..0000000000 --- a/doc/doc_ch/enhanced_ctc_loss.md +++ /dev/null @@ -1,78 +0,0 @@ -# Enhanced CTC Loss - -在OCR识别中, CRNN是一种在工业界广泛使用的文字识别算法。 在训练阶段,其采用CTCLoss来计算网络损失; 在推理阶段,其采用CTCDecode来获得解码结果。虽然CRNN算法在实际业务中被证明能够获得很好的识别效果, 然而用户对识别准确率的要求却是无止境的,如何进一步提升文字识别的准确率呢? 本文以CTCLoss为切人点,分别从难例挖掘、 多任务学习、 Metric Learning 3个不同的角度探索了CTCLoss的改进融合方案,提出了EnhancedCTCLoss,其包括如下3个组成部分: Focal-CTC Loss,A-CTC Loss, C-CTC Loss。 - -## 1. Focal-CTC Loss -Focal Loss 出自论文《Focal Loss for Dense Object Detection》, 该loss最先提出的时候主要是为了解决one-stage目标检测中正负样本比例严重失衡的问题。该损失函数降低了大量简单负样本在训练中所占的权重,也可理解为一种困难样本挖掘。 -其损失函数形式如下: -
- -
- -其中, y' 是经过激活函数的输出,取值在0-1之间。其在原始的交叉熵损失的基础上加了一个调制系数(1 – y’)^ γ和平衡因子α。 当α = 1,y=1时,其损失函数与交叉熵损失的对比如下图所示: -
- -
- -从上图可以看到, 当γ> 0时,调整系数(1-y’)^γ 赋予易分类样本损失一个更小的权重,使得网络更关注于困难的、错分的样本。 调整因子γ用于调节简单样本权重降低的速率,当γ为0时即为交叉熵损失函数,当γ增加时,调整因子的影响也会随之增大。实验发现γ为2是最优。平衡因子α用来平衡正负样本本身的比例不均,文中α取0.25。 - -对于经典的CTC算法,假设某个特征序列(f1, f2, ......ft), 经过CTC解码之后结果等于label的概率为y’, 则CTC解码结果不为label的概率即为(1-y’);不难发现, CTCLoss值和y’有如下关系: -
- -
- -结合Focal Loss的思想,赋予困难样本较大的权重,简单样本较小的权重,可以使网络更加聚焦于对困难样本的挖掘,进一步提升识别的准确率,由此我们提出了Focal-CTC Loss; 其定义如下所示: -
- -
- -实验中,γ取值为2, α= 1, 具体实现见: [rec_ctc_loss.py](../../ppocr/losses/rec_ctc_loss.py) - -## 2. A-CTC Loss -A-CTC Loss是CTC Loss + ACE Loss的简称。 其中ACE Loss出自论文< Aggregation Cross-Entropy for Sequence Recognition>. ACE Loss相比于CTCLoss,主要有如下两点优势: -+ ACE Loss能够解决2-D文本的识别问题; CTCLoss只能够处理1-D文本 -+ ACE Loss 在时间复杂度和空间复杂度上优于CTC loss - -前人总结的OCR识别算法的优劣如下图所示: -
- -
- -虽然ACELoss确实如上图所说,可以处理2D预测,在内存占用及推理速度方面具备优势,但在实践过程中,我们发现单独使用ACE Loss, 识别效果并不如CTCLoss. 因此,我们尝试将CTCLoss和ACELoss进行结合,同时以CTCLoss为主,将ACELoss 定位为一个辅助监督loss。 这一尝试收到了效果,在我们内部的实验数据集上,相比单独使用CTCLoss,识别准确率可以提升1%左右。 -A_CTC Loss定义如下: -
- -
- -实验中,λ = 0.1. ACE loss实现代码见: [ace_loss.py](../../ppocr/losses/ace_loss.py) - -## 3. C-CTC Loss -C-CTC Loss是CTC Loss + Center Loss的简称。 其中Center Loss出自论文 < A Discriminative Feature Learning Approach for Deep Face Recognition>. 最早用于人脸识别任务,用于增大类间距离,减小类内距离, 是Metric Learning领域一种较早的、也比较常用的一种算法。 -在中文OCR识别任务中,通过对badcase分析, 我们发现中文识别的一大难点是相似字符多,容易误识。 由此我们想到是否可以借鉴Metric Learing的想法, 增大相似字符的类间距,从而提高识别准确率。然而,MetricLearning主要用于图像识别领域,训练数据的标签为一个固定的值;而对于OCR识别来说,其本质上是一个序列识别任务,特征和label之间并不具有显式的对齐关系,因此两者如何结合依然是一个值得探索的方向。 -通过尝试Arcmargin, Cosmargin等方法, 我们最终发现Centerloss 有助于进一步提升识别的准确率。C_CTC Loss定义如下: -
- -
- -实验中,我们设置λ=0.25. center_loss实现代码见: [center_loss.py](../../ppocr/losses/center_loss.py) - -值得一提的是, 在C-CTC Loss中,选择随机初始化Center并不能够带来明显的提升. 我们的Center初始化方法如下: -+ 基于原始的CTCLoss, 训练得到一个网络N -+ 挑选出训练集中,识别完全正确的部分, 组成集合G -+ 将G中的每个样本送入网络,进行前向计算, 提取最后一个FC层的输入(即feature)及其经过argmax计算的结果(即index)之间的对应关系 -+ 将相同index的feature进行聚合,计算平均值,得到各自字符的初始center. - -以配置文件`configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml`为例, center提取命令如下所示: -``` -python tools/export_center.py -c configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml -o Global.pretrained_model="./output/rec_mobile_pp-OCRv2/best_accuracy" -``` -运行完后,会在PaddleOCR主目录下生成`train_center.pkl`. - -## 4. 实验 -对于上述的三种方案,我们基于百度内部数据集进行了训练、评测,实验情况如下表所示: -|algorithm| Focal_CTC | A_CTC | C-CTC | -|:------| :------| ------: | :------: | -|gain| +0.3% | +0.7% | +1.7% | - -基于上述实验结论,我们在PP-OCRv2中,采用了C-CTC的策略。 值得一提的是,由于PP-OCRv2 处理的是6625个中文字符的识别任务,字符集比较大,形似字较多,所以在该任务上C-CTC 方案带来的提升较大。 但如果换做其他OCR识别任务,结论可能会有所不同。大家可以尝试Focal-CTC,A-CTC, C-CTC以及组合方案EnhancedCTC,相信会带来不同程度的提升效果。 -统一的融合方案见如下文件: [rec_enhanced_ctc_loss.py](../../ppocr/losses/rec_enhanced_ctc_loss.py) diff --git a/doc/doc_ch/environment.md b/doc/doc_ch/environment.md deleted file mode 100644 index 23bec4b978..0000000000 --- a/doc/doc_ch/environment.md +++ /dev/null @@ -1,312 +0,0 @@ -# 运行环境准备 - -Windows和Mac用户推荐使用Anaconda搭建Python环境,Linux用户建议使用docker搭建Python环境。 - -推荐环境: -- PaddlePaddle >= 2.1.2 -- Python 3.7 -- CUDA10.1 / CUDA10.2 -- CUDNN 7.6 - -> 如果您已经安装Python环境,可以直接参考[PaddleOCR快速开始](./quickstart.md) - -* [1. Python环境搭建](#1) - + [1.1 Windows](#1.1) - + [1.2 Mac](#1.2) - + [1.3 Linux](#1.3) - - - -## 1. Python环境搭建 - - - -### 1.1 Windows - -#### 1.1.1 安装Anaconda - -- 说明:使用paddlepaddle需要先安装python环境,这里我们选择python集成环境Anaconda工具包 - - Anaconda是1个常用的python包管理程序 - - 安装完Anaconda后,可以安装python环境,以及numpy等所需的工具包环境。 -- Anaconda下载: - - 地址:https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/?C=M&O=D - - 大部分win10电脑均为64位操作系统,选择x86_64版本;若电脑为32位操作系统,则选择x86.exe - - anaconda download - - 下载完成后,双击安装程序进入图形界面 - - 默认安装位置为C盘,建议将安装位置更改到D盘: - - install config - - 勾选conda加入环境变量,忽略警告: - - add conda to path - -#### 1.1.2 打开终端并创建conda环境 - -- 打开Anaconda Prompt终端:左下角Windows Start Menu -> Anaconda3 -> Anaconda Prompt启动控制台 - - anaconda download - - -- 创建新的conda环境 - - ```shell - # 在命令行输入以下命令,创建名为paddle_env的环境 - # 此处为加速下载,使用清华源 - conda create --name paddle_env python=3.8 --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/ # 这是一行命令 - ``` - - 该命令会创建1个名为paddle_env、python版本为3.8的可执行环境,根据网络状态,需要花费一段时间 - - 之后命令行中会输出提示信息,输入y并回车继续安装 - - conda create - -- 激活刚创建的conda环境,在命令行中输入以下命令: - - ```shell - # 激活paddle_env环境 - conda activate paddle_env - # 查看当前python的位置 - where python - ``` - - create environment - - - - -以上anaconda环境和python环境安装完毕 - - - -### 1.2 Mac - -#### 1.2.1 安装Anaconda - -- 说明:使用paddlepaddle需要先安装python环境,这里我们选择python集成环境Anaconda工具包 - - Anaconda是1个常用的python包管理程序 - - 安装完Anaconda后,可以安装python环境,以及numpy等所需的工具包环境 -- Anaconda下载: - - 地址:https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/?C=M&O=D - - anaconda download - - - 选择最下方的`Anaconda3-2021.05-MacOSX-x86_64.pkg`下载 -- 下载完成后,双击.pkg文件进入图形界面 - - 按默认设置即可,安装需要花费一段时间 -- 建议安装vscode或pycharm等代码编辑器 - -#### 1.2.2 打开终端并创建conda环境 - -- 打开终端 - - - 同时按下command键和空格键,在聚焦搜索中输入"终端",双击进入终端 - -- **将conda加入环境变量** - - - 加入环境变量是为了让系统能识别conda命令 - - - 输入以下命令,在终端中打开`~/.bash_profile`: - - ```shell - vim ~/.bash_profile - ``` - - - 在`~/.bash_profile`中将conda添加为环境变量: - - ```shell - # 先按i进入编辑模式 - # 在第一行输入: - export PATH="~/opt/anaconda3/bin:$PATH" - # 若安装时自定义了安装位置,则将~/opt/anaconda3/bin改为自定义的安装目录下的bin文件夹 - ``` - - ```shell - # 修改后的~/.bash_profile文件应如下(其中xxx为用户名): - export PATH="~/opt/anaconda3/bin:$PATH" - # >>> conda initialize >>> - # !! Contents within this block are managed by 'conda init' !! - __conda_setup="$('/Users/xxx/opt/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)" - if [ $? -eq 0 ]; then - eval "$__conda_setup" - else - if [ -f "/Users/xxx/opt/anaconda3/etc/profile.d/conda.sh" ]; then - . "/Users/xxx/opt/anaconda3/etc/profile.d/conda.sh" - else - export PATH="/Users/xxx/opt/anaconda3/bin:$PATH" - fi - fi - unset __conda_setup - # <<< conda initialize <<< - ``` - - - 修改完成后,先按`esc`键退出编辑模式,再输入`:wq!`并回车,以保存退出 - - - 验证是否能识别conda命令: - - - 在终端中输入`source ~/.bash_profile`以更新环境变量 - - 再在终端输入`conda info --envs`,若能显示当前有base环境,则conda已加入环境变量 - -- 创建新的conda环境 - - ```shell - # 在命令行输入以下命令,创建名为paddle_env的环境 - # 此处为加速下载,使用清华源 - conda create --name paddle_env python=3.8 --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/ - ``` - - - 该命令会创建1个名为paddle_env、python版本为3.8的可执行环境,根据网络状态,需要花费一段时间 - - - 之后命令行中会输出提示信息,输入y并回车继续安装 - - - conda_create - -- 激活刚创建的conda环境,在命令行中输入以下命令: - - ```shell - # 激活paddle_env环境 - conda activate paddle_env - # 查看当前python的位置 - where python - ``` - - conda_actviate - -以上anaconda环境和python环境安装完毕 - - - -### 1.3 Linux - -Linux用户可选择Anaconda或Docker两种方式运行。如果你熟悉Docker且需要训练PaddleOCR模型,推荐使用Docker环境,PaddleOCR的开发流程均在Docker环境下运行。如果你不熟悉Docker,也可以使用Anaconda来运行项目。 - -#### 1.3.1 Anaconda环境配置 - -- 说明:使用paddlepaddle需要先安装python环境,这里我们选择python集成环境Anaconda工具包 - - Anaconda是1个常用的python包管理程序 - - 安装完Anaconda后,可以安装python环境,以及numpy等所需的工具包环境 - -- **下载Anaconda**: - - - 下载地址:https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/?C=M&O=D - - - - 选择适合您操作系统的版本 - - 可在终端输入`uname -m`查询系统所用的指令集 - -- 下载法1:本地下载,再将安装包传到linux服务器上 - -- 下载法2:直接使用linux命令行下载 - - ```shell - # 首先安装wget - sudo apt-get install wget # Ubuntu - sudo yum install wget # CentOS - ``` - - ```shell - # 然后使用wget从清华源上下载 - # 如要下载Anaconda3-2021.05-Linux-x86_64.sh,则下载命令如下: - wget https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/Anaconda3-2021.05-Linux-x86_64.sh - - # 若您要下载其他版本,需要将最后1个/后的文件名改成您希望下载的版本 - ``` - -- 安装Anaconda: - - - 在命令行输入`sh Anaconda3-2021.05-Linux-x86_64.sh` - - 若您下载的是其它版本,则将该命令的文件名替换为您下载的文件名 - - 按照安装提示安装即可 - - 查看许可时可输入q来退出 - -- **将conda加入环境变量** - - - 加入环境变量是为了让系统能识别conda命令,若您在安装时已将conda加入环境变量path,则可跳过本步 - - - 在终端中打开`~/.bashrc`: - - ```shell - # 在终端中输入以下命令: - vim ~/.bashrc - ``` - - - 在`~/.bashrc`中将conda添加为环境变量: - - ```shell - # 先按i进入编辑模式 - # 在第一行输入: - export PATH="~/anaconda3/bin:$PATH" - # 若安装时自定义了安装位置,则将~/anaconda3/bin改为自定义的安装目录下的bin文件夹 - ``` - - ```shell - # 修改后的~/.bash_profile文件应如下(其中xxx为用户名): - export PATH="~/opt/anaconda3/bin:$PATH" - # >>> conda initialize >>> - # !! Contents within this block are managed by 'conda init' !! - __conda_setup="$('/Users/xxx/opt/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)" - if [ $? -eq 0 ]; then - eval "$__conda_setup" - else - if [ -f "/Users/xxx/opt/anaconda3/etc/profile.d/conda.sh" ]; then - . "/Users/xxx/opt/anaconda3/etc/profile.d/conda.sh" - else - export PATH="/Users/xxx/opt/anaconda3/bin:$PATH" - fi - fi - unset __conda_setup - # <<< conda initialize <<< - ``` - - - 修改完成后,先按`esc`键退出编辑模式,再输入`:wq!`并回车,以保存退出 - - - 验证是否能识别conda命令: - - - 在终端中输入`source ~/.bash_profile`以更新环境变量 - - 再在终端输入`conda info --envs`,若能显示当前有base环境,则conda已加入环境变量 - -- 创建新的conda环境 - - ```shell - # 在命令行输入以下命令,创建名为paddle_env的环境 - # 此处为加速下载,使用清华源 - conda create --name paddle_env python=3.8 --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/ - ``` - - - 该命令会创建1个名为paddle_env、python版本为3.8的可执行环境,根据网络状态,需要花费一段时间 - - - 之后命令行中会输出提示信息,输入y并回车继续安装 - - conda_create - -- 激活刚创建的conda环境,在命令行中输入以下命令: - - ```shell - # 激活paddle_env环境 - conda activate paddle_env - ``` - - -以上anaconda环境和python环境安装完毕 - -#### 1.3.2 Docker环境配置 - -**注意:第一次使用这个镜像,会自动下载该镜像,请耐心等待。您也可以访问[DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/)获取与您机器适配的镜像。** - -```bash -# 切换到工作目录下 -cd /home/Projects -# 首次运行需创建一个docker容器,再次运行时不需要运行当前命令 -# 创建一个名字为ppocr的docker容器,并将当前目录映射到容器的/paddle目录下 - -#如果您希望在CPU环境下使用docker,使用docker而不是nvidia-docker创建docker -sudo docker run --name ppocr -v $PWD:/paddle --network=host -it registry.baidubce.com/paddlepaddle/paddle:2.1.3-gpu-cuda10.2-cudnn7 /bin/bash - -#如果使用CUDA10,请运行以下命令创建容器,设置docker容器共享内存shm-size为64G,建议设置32G以上 -# 如果是CUDA11+CUDNN8,推荐使用镜像registry.baidubce.com/paddlepaddle/paddle:2.1.3-gpu-cuda11.2-cudnn8 -sudo nvidia-docker run --name ppocr -v $PWD:/paddle --shm-size=64G --network=host -it registry.baidubce.com/paddlepaddle/paddle:2.1.3-gpu-cuda10.2-cudnn7 /bin/bash - -# ctrl+P+Q可退出docker 容器,重新进入docker 容器使用如下命令 -sudo docker container exec -it ppocr /bin/bash -``` diff --git a/doc/doc_ch/equation_a_ctc.png b/doc/doc_ch/equation_a_ctc.png deleted file mode 100644 index ae097610d3..0000000000 Binary files a/doc/doc_ch/equation_a_ctc.png and /dev/null differ diff --git a/doc/doc_ch/equation_c_ctc.png b/doc/doc_ch/equation_c_ctc.png deleted file mode 100644 index 67207a9937..0000000000 Binary files a/doc/doc_ch/equation_c_ctc.png and /dev/null differ diff --git a/doc/doc_ch/equation_ctcloss.png b/doc/doc_ch/equation_ctcloss.png deleted file mode 100644 index 33ad92c9e4..0000000000 Binary files a/doc/doc_ch/equation_ctcloss.png and /dev/null differ diff --git a/doc/doc_ch/equation_focal_ctc.png b/doc/doc_ch/equation_focal_ctc.png deleted file mode 100644 index 6ba1e8715d..0000000000 Binary files a/doc/doc_ch/equation_focal_ctc.png and /dev/null differ diff --git a/doc/doc_ch/finetune.md b/doc/doc_ch/finetune.md deleted file mode 100644 index ec4bd06590..0000000000 --- a/doc/doc_ch/finetune.md +++ /dev/null @@ -1,231 +0,0 @@ -# 模型微调 - -## 1. 模型微调背景与意义 - -PaddleOCR提供的PP-OCR系列模型在通用场景中性能优异,能够解决绝大多数情况下的检测与识别问题。在垂类场景中,如果希望获取更优的模型效果,可以通过模型微调的方法,进一步提升PP-OCR系列检测与识别模型的精度。 - -本文主要介绍文本检测与识别模型在模型微调时的一些注意事项,最终希望您在自己的场景中,通过模型微调,可以获取精度更高的文本检测与识别模型。 - -本文核心要点如下所示。 - -1. PP-OCR提供的预训练模型有较好的泛化能力 -2. 加入少量真实数据(检测任务>=500张, 识别任务>=5000张),会大幅提升垂类场景的检测与识别效果 -3. 在模型微调时,加入真实通用场景数据,可以进一步提升模型精度与泛化性能 -4. 在图像检测任务中,增大图像的预测尺度,能够进一步提升较小文字区域的检测效果 -5. 在模型微调时,需要适当调整超参数(学习率,batch size最为重要),以获得更优的微调效果。 - -更多详细内容,请参考第2章与第3章。 - -## 2. 文本检测模型微调 - -### 2.1 数据选择 - -* 数据量:建议至少准备500张的文本检测数据集用于模型微调。 - -* 数据标注:单行文本标注格式,建议标注的检测框与实际语义内容一致。如在火车票场景中,姓氏与名字可能离得较远,但是它们在语义上属于同一个检测字段,这里也需要将整个姓名标注为1个检测框。 - -### 2.2 模型选择 - -建议选择PP-OCRv3模型(配置文件:[ch_PP-OCRv3_det_student.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_student.yml),预训练模型:[ch_PP-OCRv3_det_distill_train.tar](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar))进行微调,其精度与泛化性能是目前提供的最优预训练模型。 - -更多PP-OCR系列模型,请参考[PP-OCR 系列模型库](./models_list.md)。 - -注意:在使用上述预训练模型的时候,需要使用文件夹中的`student.pdparams`文件作为预训练模型,即,仅使用学生模型。 - - -### 2.3 训练超参选择 - -在模型微调的时候,最重要的超参就是预训练模型路径`pretrained_model`, 学习率`learning_rate`与`batch_size`,部分配置文件如下所示。 - -```yaml -Global: - pretrained_model: ./ch_PP-OCRv3_det_distill_train/student.pdparams # 预训练模型路径 -Optimizer: - lr: - name: Cosine - learning_rate: 0.001 # 学习率 - warmup_epoch: 2 - regularizer: - name: 'L2' - factor: 0 - -Train: - loader: - shuffle: True - drop_last: False - batch_size_per_card: 8 # 单卡batch size - num_workers: 4 -``` - -上述配置文件中,首先需要将`pretrained_model`字段指定为`student.pdparams`文件路径。 - -PaddleOCR提供的配置文件是在8卡训练(相当于总的batch size是`8*8=64`)、且没有加载预训练模型情况下的配置文件,因此您的场景中,学习率与总的batch size需要对应线性调整,例如 - -* 如果您的场景中是单卡训练,单卡batch_size=8,则总的batch_size=8,建议将学习率调整为`1e-4`左右。 -* 如果您的场景中是单卡训练,由于显存限制,只能设置单卡batch_size=4,则总的batch_size=4,建议将学习率调整为`5e-5`左右。 - -### 2.4 预测超参选择 - -对训练好的模型导出并进行推理时,可以通过进一步调整预测的图像尺度,来提升小面积文本的检测效果,下面是DBNet推理时的一些超参数,可以通过适当调整,提升效果。 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| det_db_thresh | float | 0.3 | DB输出的概率图中,得分大于该阈值的像素点才会被认为是文字像素点 | -| det_db_box_thresh | float | 0.6 | 检测结果边框内,所有像素点的平均得分大于该阈值时,该结果会被认为是文字区域 | -| det_db_unclip_ratio | float | 1.5 | `Vatti clipping`算法的扩张系数,使用该方法对文字区域进行扩张 | -| max_batch_size | int | 10 | 预测的batch size | -| use_dilation | bool | False | 是否对分割结果进行膨胀以获取更优检测效果 | -| det_db_score_mode | str | "fast" | DB的检测结果得分计算方法,支持`fast`和`slow`,`fast`是根据polygon的外接矩形边框内的所有像素计算平均得分,`slow`是根据原始polygon内的所有像素计算平均得分,计算速度相对较慢一些,但是更加准确一些。 | - - -更多关于推理方法的介绍可以参考[Paddle Inference推理教程](././inference_ppocr.md)。 - - -## 3. 文本识别模型微调 - - -### 3.1 数据选择 - -* 数据量:不更换字典的情况下,建议至少准备5000张的文本识别数据集用于模型微调;如果更换了字典(不建议),需要的数量更多。 - -* 数据分布:建议分布与实测场景尽量一致。如果实测场景包含大量短文本,则训练数据中建议也包含较多短文本,如果实测场景对于空格识别效果要求较高,则训练数据中建议也包含较多带空格的文本内容。 - -* 数据合成:针对部分字符识别有误的情况,建议获取一批特定字符数据,加入到原数据中使用小学习率微调。其中原始数据与新增数据比例可尝试 10:1 ~ 5:1, 避免单一场景数据过多导致模型过拟合,同时尽量平衡语料词频,确保常用字的出现频率不会过低。 - - 特定字符生成可以使用 TextRenderer 工具,合成例子可参考 [数码管数据合成](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/applications/%E5%85%89%E5%8A%9F%E7%8E%87%E8%AE%A1%E6%95%B0%E7%A0%81%E7%AE%A1%E5%AD%97%E7%AC%A6%E8%AF%86%E5%88%AB/%E5%85%89%E5%8A%9F%E7%8E%87%E8%AE%A1%E6%95%B0%E7%A0%81%E7%AE%A1%E5%AD%97%E7%AC%A6%E8%AF%86%E5%88%AB.md#31-%E6%95%B0%E6%8D%AE%E5%87%86%E5%A4%87) - ,合成数据语料尽量来自真实使用场景,在贴近真实场景的基础上保持字体、背景的丰富性,有助于提升模型效果。 - -* 通用中英文数据:在训练的时候,可以在训练集中添加通用真实数据(如在不更换字典的微调场景中,建议添加LSVT、RCTW、MTWI等真实数据),进一步提升模型的泛化性能。 - -### 3.2 模型选择 - -建议选择PP-OCRv3模型(配置文件:[ch_PP-OCRv3_rec_distillation.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml),预训练模型:[ch_PP-OCRv3_rec_train.tar](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_train.tar))进行微调,其精度与泛化性能是目前提供的最优预训练模型。 - -更多PP-OCR系列模型,请参考[PP-OCR 系列模型库](./models_list.md)。 - -PP-OCRv3 模型使用了GTC策略,其中SAR分支参数量大,当训练数据为简单场景时模型容易过拟合,导致微调效果不佳,建议去除GTC策略,模型结构部分配置文件修改如下: - -```yaml -Architecture: - model_type: rec - algorithm: SVTR - Transform: - Backbone: - name: MobileNetV1Enhance - scale: 0.5 - last_conv_stride: [1, 2] - last_pool_type: avg - Neck: - name: SequenceEncoder - encoder_type: svtr - dims: 64 - depth: 2 - hidden_dims: 120 - use_guide: False - Head: - name: CTCHead - fc_decay: 0.00001 -Loss: - name: CTCLoss - -Train: - dataset: - ...... - transforms: - # 去除 RecConAug 增广 - # - RecConAug: - # prob: 0.5 - # ext_data_num: 2 - # image_shape: [48, 320, 3] - # max_text_length: *max_text_length - - RecAug: - # 修改 Encode 方式 - - CTCLabelEncode: - - KeepKeys: - keep_keys: - - image - - label - - length -... - -Eval: - dataset: - ... - transforms: - ... - - CTCLabelEncode: - - KeepKeys: - keep_keys: - - image - - label - - length -... - - -``` - -### 3.3 训练超参选择 - -与文本检测任务微调相同,在识别模型微调的时候,最重要的超参就是预训练模型路径`pretrained_model`, 学习率`learning_rate`与`batch_size`,部分默认配置文件如下所示。 - -```yaml -Global: - pretrained_model: # 预训练模型路径 -Optimizer: - lr: - name: Piecewise - decay_epochs : [700, 800] - values : [0.001, 0.0001] # 学习率 - warmup_epoch: 5 - regularizer: - name: 'L2' - factor: 0 - -Train: - dataset: - name: SimpleDataSet - data_dir: ./train_data/ - label_file_list: - - ./train_data/train_list.txt - ratio_list: [1.0] # 采样比例,默认值是[1.0] - loader: - shuffle: True - drop_last: False - batch_size_per_card: 128 # 单卡batch size - num_workers: 8 - -``` - - -上述配置文件中,首先需要将`pretrained_model`字段指定为3.2章节中解压得到的`ch_PP-OCRv3_rec_train/best_accuracy.pdparams`文件路径。 - -PaddleOCR提供的配置文件是在8卡训练(相当于总的batch size是`8*128=1024`)、且没有加载预训练模型情况下的配置文件,因此您的场景中,学习率与总的batch size需要对应线性调整,例如: - -* 如果您的场景中是单卡训练,单卡batch_size=128,则总的batch_size=128,在加载预训练模型的情况下,建议将学习率调整为`[1e-4, 2e-5]`左右(piecewise学习率策略,需设置2个值,下同)。 -* 如果您的场景中是单卡训练,因为显存限制,只能设置单卡batch_size=64,则总的batch_size=64,在加载预训练模型的情况下,建议将学习率调整为`[5e-5, 1e-5]`左右。 - - -如果有通用真实场景数据加进来,建议每个epoch中,垂类场景数据与真实场景的数据量保持在1:1左右。 - -比如:您自己的垂类场景识别数据量为1W,数据标签文件为`vertical.txt`,收集到的通用场景识别数据量为10W,数据标签文件为`general.txt`, - - -那么,可以设置`label_file_list`和`ratio_list`参数如下所示。每个epoch中,`vertical.txt`中会进行全采样(采样比例为1.0),包含1W条数据;`general.txt`中会按照0.1的采样比例进行采样,包含`10W*0.1=1W`条数据,最终二者的比例为`1:1`。 - -```yaml -Train: - dataset: - name: SimpleDataSet - data_dir: ./train_data/ - label_file_list: - - vertical.txt - - general.txt - ratio_list: [1.0, 0.1] -``` - - -### 3.4 训练调优 - -训练过程并非一蹴而就的,完成一个阶段的训练评估后,建议收集分析当前模型在真实场景中的 badcase,有针对性的调整训练数据比例,或者进一步新增合成数据。通过多次迭代训练,不断优化模型效果。 - -如果在训练时修改了自定义字典,由于无法加载最后一层FC的参数,在迭代初期acc=0是正常的情况,不必担心,加载预训练模型依然可以加快模型收敛。 diff --git a/doc/doc_ch/focal_loss_formula.png b/doc/doc_ch/focal_loss_formula.png deleted file mode 100644 index 971cebcd08..0000000000 Binary files a/doc/doc_ch/focal_loss_formula.png and /dev/null differ diff --git a/doc/doc_ch/focal_loss_image.png b/doc/doc_ch/focal_loss_image.png deleted file mode 100644 index 430550a732..0000000000 Binary files a/doc/doc_ch/focal_loss_image.png and /dev/null differ diff --git a/doc/doc_ch/framework.png b/doc/doc_ch/framework.png deleted file mode 100644 index db151a0e16..0000000000 Binary files a/doc/doc_ch/framework.png and /dev/null differ diff --git a/doc/doc_ch/hardware/install_other_devices.md b/doc/doc_ch/hardware/install_other_devices.md deleted file mode 100644 index ccb1fbadfb..0000000000 --- a/doc/doc_ch/hardware/install_other_devices.md +++ /dev/null @@ -1,44 +0,0 @@ -# 多硬件安装飞桨 -本文档主要针对昇腾 NPU 硬件平台,介绍如何安装飞桨。 -## 1. 昇腾 NPU 飞桨安装 -### 1.1 环境准备 -当前 PaddleOCR 支持昇腾 910B 芯片,昇腾驱动版本为 23.0.3。考虑到环境差异性,我们推荐使用飞桨官方提供的标准镜像完成环境准备。 -- 1. 拉取镜像,此镜像仅为开发环境,镜像中不包含预编译的飞桨安装包,镜像中已经默认安装了昇腾算子库 CANN-8.0.RC1。 - -``` -# 适用于 X86 架构,暂时不提供 Arch64 架构镜像 -docker pull registry.baidubce.com/device/paddle-npu:cann80RC1-ubuntu20-x86_64-gcc84-py39 -``` - -- 2. 参考如下命令启动容器,ASCEND_RT_VISIBLE_DEVICES 指定可见的 NPU 卡号 -``` -docker run -it --name paddle-npu-dev -v $(pwd):/work \ - --privileged --network=host --shm-size=128G -w=/work \ - -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ - -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ - -v /usr/local/dcmi:/usr/local/dcmi \ - -e ASCEND_RT_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" \ - registry.baidubce.com/device/paddle-npu:cann80RC1-ubuntu20-x86_64-gcc84-py39 /bin/bash -``` -### 1.2 安装 paddle 包 -当前提供 Python3.9 的 wheel 安装包。如有其他 Python 版本需求,可以参考[飞桨官方文档](https://www.paddlepaddle.org.cn/install/quick)自行编译安装。 - -- 1. 下载安装 Python3.9 的 wheel 安装包 - -``` -# 注意需要先安装飞桨 cpu 版本 -pip install https://paddle-model-ecology.bj.bcebos.com/paddlex/whl/paddle-device/npu/paddlepaddle-0.0.0-cp39-cp39-linux_x86_64.whl -pip install https://paddle-model-ecology.bj.bcebos.com/paddlex/whl/paddle-device/npu/paddle_custom_npu-0.0.0-cp39-cp39-linux_x86_64.whl -``` -- 2. 验证安装包 -安装完成之后,运行如下命令。 -``` -python -c "import paddle; paddle.utils.run_check()" -``` -预期得到如下输出结果 -``` -Running verify PaddlePaddle program ... -PaddlePaddle works well on 1 npu. -PaddlePaddle works well on 8 npus. -PaddlePaddle is installed successfully! Let's start deep learning with PaddlePaddle now. -``` diff --git a/doc/doc_ch/hardware/supported_models.md b/doc/doc_ch/hardware/supported_models.md deleted file mode 100644 index 10b5d240d5..0000000000 --- a/doc/doc_ch/hardware/supported_models.md +++ /dev/null @@ -1,7 +0,0 @@ -# PaddleOCR模型列表 - -*多硬件安装方式请参考[多硬件安装文档](install_other_devices.md)* - -| 模型名称 | 昇腾NPU | -| ---------------- | -------- | -| PP-OCRv4 | √ | diff --git a/doc/doc_ch/inference.md b/doc/doc_ch/inference.md deleted file mode 100644 index b73245a410..0000000000 --- a/doc/doc_ch/inference.md +++ /dev/null @@ -1,405 +0,0 @@ - -# 基于Python预测引擎的推理 - -推理模型(通过 paddle.jit.save 保存的模型)一般是模型训练完成后保存的固化模型,多用于部署中的预测。 - -训练过程中保存的模型是检查点模型,保存的是模型的参数,多用于恢复训练。 - -与检查点模型相比,推理模型会额外保存模型的结构信息。因此,它更容易部署,因为模型结构和模型参数已经固化在推理模型文件中,适合与实际系统集成。更多细节请参考文档[Classification Framework](https://github.com/PaddlePaddle/PaddleClas/blob/release%2F2.0/docs/zh_CN/extension/paddle_mobile_inference.md). - -接下来,我们首先介绍如何将训练好的模型转换为推理模型,然后介绍基于推理模型的文本检测、文本识别、角度分类及其串联。 - -- [基于Python预测引擎的推理](#基于python预测引擎的推理) - - [1. 将训练模型转换为推理模型](#1-将训练模型转换为推理模型) - - [1.1 将检测模型转换为推理模型](#11-将检测模型转换为推理模型) - - [1.2 将识别模型转换为推理模型](#12-将识别模型转换为推理模型) - - [1.3 将角度分类模型转换为推理模型](#13-将角度分类模型转换为推理模型) - - [2. 文本检测模型推理](#2-文本检测模型推理) - - [2.1 轻量级中文检测模型推理](#21-轻量级中文检测模型推理) - - [2.2 DB文本检测模型推理](#22-db文本检测模型推理) - - [2.3 EAST文本检测模型推理](#23-east文本检测模型推理) - - [2.4 Sast文本检测模型推理](#24-sast文本检测模型推理) - - [(1). 四边形文本检测模型 (ICDAR2015)](#1-四边形文本检测模型-icdar2015) - - [(2). 曲线文本检测模型 (Total-Text)](#2-曲线文本检测模型-total-text) - - [3. 文本识别模型推理](#3-文本识别模型推理) - - [3.1 轻量级中文文本识别模型推理](#31-轻量级中文文本识别模型推理) - - [3.2 基于CTC的文本识别模型推理](#32-基于ctc的文本识别模型推理) - - [3.3 基于SRN的文本识别模型推理](#33-基于srn的文本识别模型推理) - - [3.4 使用自定义字符字典的文本识别模型推理](#34-使用自定义字符字典的文本识别模型推理) - - [3.5 多语言模型推理](#35-多语言模型推理) - - [4. 角度分类模型推理](#4-角度分类模型推理) - - [5. 文本检测、角度分类与识别推理串联](#5-文本检测角度分类与识别推理串联) - - [5.1 轻量级中文模型](#51-轻量级中文模型) - - [5.2 其他模型](#52-其他模型) - - -## 1. 将训练模型转换为推理模型 - - -### 1.1 将检测模型转换为推理模型 - -下载轻量级中文检测模型: -``` -wget -P ./ch_lite/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_train.tar && tar xf ./ch_lite/ch_ppocr_mobile_v2.0_det_train.tar -C ./ch_lite/ -``` - -上述模型是使用 MobileNetV3 作为backbone训练的 DB 算法。要将训练好的模型转换为推理模型,只需运行以下命令: -``` -# -c Set the training algorithm yml configuration file -# -o Set optional parameters -# Global.pretrained_model parameter Set the training model address to be converted without adding the file suffix .pdmodel, .pdopt or .pdparams. -# Global.save_inference_dir Set the address where the converted model will be saved. - -python3 tools/export_model.py -c configs/det/ch_ppocr_v2.0/ch_det_mv3_db_v2.0.yml -o Global.pretrained_model=./ch_lite/ch_ppocr_mobile_v2.0_det_train/best_accuracy Global.save_inference_dir=./inference/det_db/ -``` - -在转换为推理模型时,使用的配置文件与训练时使用的配置文件相同。此外,还需要在配置文件中设置 `Global.pretrained_model` 参数。 -转换成功后,模型保存目录中会有三个文件: -``` -inference/det_db/ - ├── inference.pdiparams # 检测推理模型的参数文件 - ├── inference.pdiparams.info # 检测推理模型的参数信息文件,可以忽略 - └── inference.pdmodel # 检测推理模型的程序文件 -``` - - -### 1.2 将识别模型转换为推理模型 - -下载轻量级中文识别模型: -``` -wget -P ./ch_lite/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_train.tar && tar xf ./ch_lite/ch_ppocr_mobile_v2.0_rec_train.tar -C ./ch_lite/ -``` - -识别模型的转换方式与检测模型相同,如下所示: -``` -# -c 设置训练算法的 yml 配置文件 -# -o 设置可选参数 -# Global.pretrained_model 参数设置要转换的训练模型地址,不需要添加文件后缀 .pdmodel, .pdopt 或 .pdparams。 -# Global.save_inference_dir 设置转换后模型保存的地址。 - -python3 tools/export_model.py -c configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml -o Global.pretrained_model=./ch_lite/ch_ppocr_mobile_v2.0_rec_train/best_accuracy Global.save_inference_dir=./inference/rec_crnn/ -``` - -如果您使用自己的数据集训练了一个模型,并且使用了不同的字典文件,请确保在配置文件中将 `character_dict_path` 修改为您的字典文件路径。 - -转换成功后,模型保存目录中会有三个文件: -``` -inference/det_db/ - ├── inference.pdiparams # The parameter file of recognition inference model - ├── inference.pdiparams.info # The parameter information of recognition inference model, which can be ignored - └── inference.pdmodel # The program file of recognition model -``` - - -### 1.3 将角度分类模型转换为推理模型 - -下载角度分类模型: -``` -wget -P ./ch_lite/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar && tar xf ./ch_lite/ch_ppocr_mobile_v2.0_cls_train.tar -C ./ch_lite/ -``` - -角度分类模型的转换方式与检测模型相同,如下所示: -``` -# -c Set the training algorithm yml configuration file -# -o Set optional parameters -# Global.pretrained_model parameter Set the training model address to be converted without adding the file suffix .pdmodel, .pdopt or .pdparams. -# Global.save_inference_dir Set the address where the converted model will be saved. - -python3 tools/export_model.py -c configs/cls/cls_mv3.yml -o Global.pretrained_model=./ch_lite/ch_ppocr_mobile_v2.0_cls_train/best_accuracy Global.save_inference_dir=./inference/cls/ -``` - -转换成功后,目录中会有两个文件: -``` -inference/det_db/ - ├── inference.pdiparams # The parameter file of angle class inference model - ├── inference.pdiparams.info # The parameter information of angle class inference model, which can be ignored - └── inference.pdmodel # The program file of angle class model -``` - - - -## 2. 文本检测模型推理 - -以下将介绍轻量级中文检测模型推理、DB文本检测模型推理和EAST文本检测模型推理。默认配置基于DB文本检测模型的推理设置。 -由于EAST和DB算法有很大的不同,在推理时,需要**通过传入相应的参数来适配EAST文本检测算法**。 - - -### 2.1 轻量级中文检测模型推理 - -对于轻量级中文检测模型推理,可以执行以下命令: - -``` -# download DB text detection inference model -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_infer.tar -tar xf ch_ppocr_mobile_v2.0_det_infer.tar -# predict -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./inference/det_db/" -``` - -可视化的文本检测结果默认保存在 `./inference_results` 文件夹中,结果文件的名称以 `det_res` 为前缀。结果示例如下: - -![](../imgs_results/det_res_00018069.jpg) - -可以使用参数 `limit_type` 和 `det_limit_side_len` 来限制输入图像的大小, -`limit_type` 的可选参数是 [`max`, `min`], -而 `det_limit_side_len` 是一个正整数,通常设置为32的倍数,例如960。 - -这些参数的默认设置是 `limit_type='max', det_limit_side_len=960`。表示网络输入图像的最长边不能超过960, -如果超过这个值,图像将按相同比例调整尺寸,以确保最长边为 `det_limit_side_len`。 -设置为 `limit_type='min', det_limit_side_len=960`,则表示图像的最短边被限制为960。 - -如果输入图片的分辨率较大,并且希望使用更大的分辨率进行预测,可以将 `det_limit_side_len` 设置为所需的值,例如1216: - -``` -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./inference/det_db/" --det_limit_type=max --det_limit_side_len=1216 -``` - -如果希望使用CPU进行预测,请执行以下命令: -``` -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./inference/det_db/" --use_gpu=False -``` - - -### 2.2 DB文本检测模型推理 - -首先,将DB文本检测训练过程中保存的模型转换为推理模型。以基于Resnet50_vd骨干网络并在ICDAR2015英文数据集上训练的模型为例([模型下载链接](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_db_v2.0_train.tar)),可以使用以下命令进行转换: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_db.yml -o Global.pretrained_model=./det_r50_vd_db_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_db -``` - -DB文本检测模型推理,可以执行以下命令: - -``` -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_db/" -``` - -可视化的文本检测结果默认保存在 `./inference_results` 文件夹中,结果文件的名称以 `det_res` 为前缀。结果示例如下: - -![](../imgs_results/det_res_img_10_db.jpg) - -**注意**:由于ICDAR2015数据集只有1000张训练图像,主要针对英文场景,上述模型在中文文本图像上的检测结果非常差。 - - - -### 2.3 EAST文本检测模型推理 - -首先,将EAST文本检测训练过程中保存的模型转换为推理模型。以基于Resnet50_vd骨干网络并在ICDAR2015英文数据集上训练的模型为例([模型下载链接](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_east_v2.0_train.tar)),可以使用以下命令进行转换: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_east.yml -o Global.pretrained_model=./det_r50_vd_east_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_east -``` -**对于EAST文本检测模型推理,需要设置参数 `--det_algorithm="EAST"`**,运行以下命令: - -``` -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_east/" --det_algorithm="EAST" -``` - -可视化的文本检测结果默认保存在 `./inference_results` 文件夹中,结果文件的名称以 `det_res` 为前缀。结果示例如下: - -![](../imgs_results/det_res_img_10_east.jpg) - -**注意**:EAST后处理的局部感知NMS有两个版本:Python和C++。C++版本的速度明显快于Python版本。由于C++版本NMS的编译版本问题,只有在Python 3.5环境中才会调用C++版本NMS,在其他情况下将调用Python版本NMS。 - - -### 2.4 Sast文本检测模型推理 - -#### (1). 四边形文本检测模型(ICDAR2015) - -首先,将SAST文本检测训练过程中保存的模型转换为推理模型。以基于Resnet50_vd骨干网络并在ICDAR2015英文数据集上训练的模型为例([模型下载链接](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_icdar15_v2.0_train.tar)),可以使用以下命令进行转换: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_sast_icdar15.yml -o Global.pretrained_model=./det_r50_vd_sast_icdar15_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_sast_ic15 -``` - -**对于SAST四边形文本检测模型推理,需要设置参数 `--det_algorithm="SAST"`**,运行以下命令: - -``` -python3 tools/infer/predict_det.py --det_algorithm="SAST" --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_sast_ic15/" -``` - -可视化的文本检测结果默认保存在 `./inference_results` 文件夹中,结果文件的名称以 `det_res` 为前缀。结果示例如下: - -![](../imgs_results/det_res_img_10_sast.jpg) - -#### (2). 曲线文本检测模型(Total-Text) - -首先,将SAST文本检测训练过程中保存的模型转换为推理模型。以基于Resnet50_vd骨干网络并在Total-Text英文数据集上训练的模型为例([模型下载链接](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_totaltext_v2.0_train.tar)),可以使用以下命令进行转换: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_sast_totaltext.yml -o Global.pretrained_model=./det_r50_vd_sast_totaltext_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_sast_tt -``` - -对于SAST曲线文本检测模型推理,需要设置参数 `--det_algorithm="SAST"` 和 `--det_sast_polygon=True`,运行以下命令: - -``` -python3 tools/infer/predict_det.py --det_algorithm="SAST" --image_dir="./doc/imgs_en/img623.jpg" --det_model_dir="./inference/det_sast_tt/" --det_sast_polygon=True -``` - -可视化的文本检测结果默认保存在 `./inference_results` 文件夹中,结果文件的名称以 `det_res` 为前缀。结果示例如下: - -![](../imgs_results/det_res_img623_sast.jpg) - -**注意**:SAST后处理的局部感知NMS有两个版本:Python和C++。C++版本的速度明显快于Python版本。由于C++版本NMS的编译版本问题,只有在Python 3.5环境中才会调用C++版本NMS,在其他情况下将调用Python版本NMS。 - - -## 3. 文本识别模型推理 - -以下将介绍轻量级中文识别模型推理、其他基于CTC和Attention的文本识别模型推理。对于中文文本识别,建议选择基于CTC损失的识别模型。实践中也发现,基于Attention损失的模型效果不如基于CTC损失的模型。此外,如果在训练过程中修改了字符字典,请确保在推理过程中使用相同的字符集。详情请见下文。 - - -### 3.1 轻量级中文文本识别模型推理 - -对于轻量级中文识别模型推理,可以执行以下命令: - -``` -# download CRNN text recognition inference model -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_infer.tar -tar xf ch_ppocr_mobile_v2.0_rec_infer.tar -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_10.png" --rec_model_dir="ch_ppocr_mobile_v2.0_rec_infer" -``` - -![](../imgs_words_en/word_10.png) - -执行命令后,上述图像的预测结果(识别的文本和分数)将打印在屏幕上。 - -```bash -Predicts of ./doc/imgs_words_en/word_10.png:('PAIN', 0.9897658) -``` - - -### 3.2 基于CTC的文本识别模型推理 - -以CRNN为例,介绍基于CTC损失的识别模型推理。Rosetta和Star-Net的使用方式类似,无需设置识别算法参数 `rec_algorithm`。 - -首先,将CRNN文本识别训练过程中保存的模型转换为推理模型。以基于Resnet34_vd骨干网络,使用MJSynth和SynthText(两个英文文本识别合成数据集)进行训练的模型为例([模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar))。可以按如下方式进行转换: - -``` -python3 tools/export_model.py -c configs/det/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model=./rec_r34_vd_none_bilstm_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_crnn -``` - -对于CRNN文本识别模型推理,执行以下命令: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./inference/starnet/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" -``` - -![](../imgs_words_en/word_336.png) - -执行命令后,上述图像的识别结果如下: - -```bash -Predicts of ./doc/imgs_words_en/word_336.png:('super', 0.9999073) -``` - -**注意**:由于上述模型参考了 [DTRB](https://arxiv.org/abs/1904.01906) 文本识别的训练和评估过程,与轻量级中文识别模型的训练在两个方面有所不同: - -- **训练时使用的图像分辨率不同**:上述模型训练时使用的图像分辨率为 [3,32,100],而我们的中文模型训练时,为了确保对长文本的识别效果,使用的图像分辨率为 [3, 32, 320]。推理阶段的默认形状参数是训练阶段使用的图像分辨率,即 [3, 32, 320]。因此,在此处运行上述英文模型的推理时,需要通过参数 `rec_image_shape` 设置识别图像的形状。 - -- **字符列表**:DTRB 论文中的实验仅针对 26 个小写英文字母和 10 个数字,共 36 个字符。所有大写和小写字符都转换为小写字符,不在上述列表中的字符被忽略并视为空格。因此,这里不使用字符字典文件,而是通过以下命令生成字典。 - -``` -self.character_str = "0123456789abcdefghijklmnopqrstuvwxyz" -dict_character = list(self.character_str) -``` - - -### 3.3 基于SRN的文本识别模型推理 - -基于SRN的识别模型需要额外设置识别算法参数 `--rec_algorithm="SRN"`。同时,需要确保预测形状与训练一致,例如:`--rec_image_shape="1, 64, 256"`。 - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" \ - --rec_model_dir="./inference/srn/" \ - --rec_image_shape="1, 64, 256" \ - --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" \ - --rec_algorithm="SRN" -``` - - -### 3.4 使用自定义字符字典的文本识别模型推理 - -如果在训练过程中修改了文本字典,在使用推理模型进行预测时,需要通过 `--rec_char_dict_path` 指定字典路径。 - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./your inference model" --rec_image_shape="3, 32, 100" --rec_char_dict_path="your text dict path" -``` - - - -### 3.5 多语言模型推理 - -如果需要预测其他语言的模型,在使用推理模型进行预测时,需要通过 `--rec_char_dict_path` 指定字典路径。同时,为了获得正确的可视化结果,需要通过 `--vis_font_path` 指定可视化字体路径。在 `doc/fonts` 路径下默认提供了一些小语种字体,例如韩文识别: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/korean/1.jpg" --rec_model_dir="./your inference model" --rec_char_dict_path="ppocr/utils/dict/korean_dict.txt" --vis_font_path="doc/fonts/korean.ttf" -``` -![](../imgs_words/korean/1.jpg) - -执行命令后,上述图像的预测结果是: - -``` text -Predicts of ./doc/imgs_words/korean/1.jpg:('바탕으로', 0.9948904) -``` - - -## 4. 角度分类模型推理 - -对于角度分类模型推理,可以执行以下命令: - -``` -python3 tools/infer/predict_cls.py --image_dir="./doc/imgs_words_en/word_10.png" --cls_model_dir="./inference/cls/" -``` -``` -# download text angle class inference model: -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar -tar xf ch_ppocr_mobile_v2.0_cls_infer.tar -python3 tools/infer/predict_cls.py --image_dir="./doc/imgs_words_en/word_10.png" --cls_model_dir="ch_ppocr_mobile_v2.0_cls_infer" -``` -![](../imgs_words_en/word_10.png) - -执行命令后,上述图像的预测结果(分类角度和分数)将打印在屏幕上。 - -``` - Predicts of ./doc/imgs_words_en/word_10.png:['0', 0.9999995] -``` - -## 5. 文本检测、角度分类和识别推理串联 - - -### 5.1 轻量级中文模型 - -在进行预测时,需要通过参数 `image_dir` 指定单张图像或图像文件夹的路径,参数 `det_model_dir` 指定检测推理模型的路径,参数 `cls_model_dir` 指定角度分类推理模型的路径,参数 `rec_model_dir` 指定识别推理模型的路径。参数 `use_angle_cls` 用于控制是否启用角度分类模型。参数 `use_mp` 指定是否使用多进程推理,`total_process_num` 指定使用多进程时的进程数量(Paddle Inference 不是线程安全的,建议使用多进程)。可视化的识别结果默认保存在 `./inference_results` 文件夹中。 - -```shell -# use direction classifier -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./inference/det_db/" --cls_model_dir="./inference/cls/" --rec_model_dir="./inference/rec_crnn/" --use_angle_cls=true - -# not use use direction classifier -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./inference/det_db/" --rec_model_dir="./inference/rec_crnn/" - -# use multi-process -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./inference/det_db/" --rec_model_dir="./inference/rec_crnn/" --use_angle_cls=false --use_mp=True --total_process_num=6 -``` - - -执行命令后,识别结果图像如下: - -![](../imgs_results/system_res_00018069.jpg) - - -### 5.2 其他模型 - -如果你想尝试其他检测算法或识别算法,请参考上述文本检测模型推理和文本识别模型推理,更新相应的配置和模型。 - -**注意:由于检测框旋转逻辑的限制,目前不支持将 SAST 曲线文本检测模型(使用参数 `det_sast_polygon=True`)与其他模型组合使用。** - -以下命令使用 EAST 文本检测和 STAR-Net 文本识别的组合: - -``` -python3 tools/infer/predict_system.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_east/" --det_algorithm="EAST" --rec_model_dir="./inference/starnet/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" -``` - -执行命令后,识别结果图像如下: - -![](../imgs_results/img_10_east_starnet.jpg) diff --git a/doc/doc_ch/inference_args.md b/doc/doc_ch/inference_args.md deleted file mode 100644 index aad7973b16..0000000000 --- a/doc/doc_ch/inference_args.md +++ /dev/null @@ -1,121 +0,0 @@ -# PaddleOCR模型推理参数解释 - -在使用PaddleOCR进行模型推理时,可以自定义修改参数,来修改模型、数据、预处理、后处理等内容(参数文件:[utility.py](../../tools/infer/utility.py)),详细的参数解释如下所示。 - -* 全局信息 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| image_dir | str | 无,必须显式指定 | 图像或者文件夹路径 | -| page_num | int | 0 | 当输入类型为pdf文件时有效,指定预测前面page_num页,默认预测所有页 | -| vis_font_path | str | "./doc/fonts/simfang.ttf" | 用于可视化的字体路径 | -| drop_score | float | 0.5 | 识别得分小于该值的结果会被丢弃,不会作为返回结果 | -| use_pdserving | bool | False | 是否使用Paddle Serving进行预测 | -| warmup | bool | False | 是否开启warmup,在统计预测耗时的时候,可以使用这种方法 | -| draw_img_save_dir | str | "./inference_results" | 系统串联预测OCR结果的保存文件夹 | -| save_crop_res | bool | False | 是否保存OCR的识别文本图像 | -| crop_res_save_dir | str | "./output" | 保存OCR识别出来的文本图像路径 | -| use_mp | bool | False | 是否开启多进程预测 | -| total_process_num | int | 6 | 开启的进程数,`use_mp`为`True`时生效 | -| process_id | int | 0 | 当前进程的id号,无需自己修改 | -| benchmark | bool | False | 是否开启benchmark,对预测速度、显存占用等进行统计 | -| save_log_path | str | "./log_output/" | 开启`benchmark`时,日志结果的保存文件夹 | -| show_log | bool | True | 是否显示预测中的日志信息 | -| use_onnx | bool | False | 是否开启onnx预测 | - - -* 预测引擎相关 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| use_gpu | bool | True | 是否使用GPU进行预测 | -| ir_optim | bool | True | 是否对计算图进行分析与优化,开启后可以加速预测过程 | -| use_tensorrt | bool | False | 是否开启tensorrt | -| min_subgraph_size | int | 15 | tensorrt中最小子图size,当子图的size大于该值时,才会尝试对该子图使用trt engine计算 | -| precision | str | fp32 | 预测的精度,支持`fp32`, `fp16`, `int8` 3种输入 | -| enable_mkldnn | bool | True | 是否开启mkldnn | -| cpu_threads | int | 10 | 开启mkldnn时,cpu预测的线程数 | - -* 文本检测模型相关 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| det_algorithm | str | "DB" | 文本检测算法名称,目前支持`DB`, `EAST`, `SAST`, `PSE`, `DB++`, `FCE` | -| det_model_dir | str | xx | 检测inference模型路径 | -| det_limit_side_len | int | 960 | 检测的图像边长限制 | -| det_limit_type | str | "max" | 检测的边长限制类型,目前支持`min`和`max`,`min`表示保证图像最短边不小于`det_limit_side_len`,`max`表示保证图像最长边不大于`det_limit_side_len` | - -其中,DB算法相关参数如下 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| det_db_thresh | float | 0.3 | DB输出的概率图中,得分大于该阈值的像素点才会被认为是文字像素点 | -| det_db_box_thresh | float | 0.6 | 检测结果边框内,所有像素点的平均得分大于该阈值时,该结果会被认为是文字区域 | -| det_db_unclip_ratio | float | 1.5 | `Vatti clipping`算法的扩张系数,使用该方法对文字区域进行扩张 | -| max_batch_size | int | 10 | 预测的batch size | -| use_dilation | bool | False | 是否对分割结果进行膨胀以获取更优检测效果 | -| det_db_score_mode | str | "fast" | DB的检测结果得分计算方法,支持`fast`和`slow`,`fast`是根据polygon的外接矩形边框内的所有像素计算平均得分,`slow`是根据原始polygon内的所有像素计算平均得分,计算速度相对较慢一些,但是更加准确一些。 | - -EAST算法相关参数如下 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| det_east_score_thresh | float | 0.8 | EAST后处理中score map的阈值 | -| det_east_cover_thresh | float | 0.1 | EAST后处理中文本框的平均得分阈值 | -| det_east_nms_thresh | float | 0.2 | EAST后处理中nms的阈值 | - -SAST算法相关参数如下 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| det_sast_score_thresh | float | 0.5 | SAST后处理中的得分阈值 | -| det_sast_nms_thresh | float | 0.5 | SAST后处理中nms的阈值 | -| det_box_type | str | quad | 是否多边形检测,弯曲文本场景(如Total-Text)设置为'poly' | - -PSE算法相关参数如下 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| det_pse_thresh | float | 0.0 | 对输出图做二值化的阈值 | -| det_pse_box_thresh | float | 0.85 | 对box进行过滤的阈值,低于此阈值的丢弃 | -| det_pse_min_area | float | 16 | box的最小面积,低于此阈值的丢弃 | -| det_box_type | str | "quad" | 返回框的类型,quad:四点坐标,poly: 弯曲文本的所有点坐标 | -| det_pse_scale | int | 1 | 输入图像相对于进后处理的图的比例,如`640*640`的图像,网络输出为`160*160`,scale为2的情况下,进后处理的图片shape为`320*320`。这个值调大可以加快后处理速度,但是会带来精度的下降 | - -* 文本识别模型相关 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| rec_algorithm | str | "CRNN" | 文本识别算法名称,目前支持`CRNN`, `SRN`, `RARE`, `NETR`, `SAR`, `ViTSTR`, `ABINet`, `VisionLAN`, `SPIN`, `RobustScanner`, `SVTR`, `SVTR_LCNet` | -| rec_model_dir | str | 无,如果使用识别模型,该项是必填项 | 识别inference模型路径 | -| rec_image_shape | str | "3,48,320" | 识别时的图像尺寸 | -| rec_batch_num | int | 6 | 识别的batch size | -| max_text_length | int | 25 | 识别结果最大长度,在`SRN`中有效 | -| rec_char_dict_path | str | "./ppocr/utils/ppocr_keys_v1.txt" | 识别的字符字典文件 | -| use_space_char | bool | True | 是否包含空格,如果为`True`,则会在最后字符字典中补充`空格`字符 | - - -* 端到端文本检测与识别模型相关 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| e2e_algorithm | str | "PGNet" | 端到端算法名称,目前支持`PGNet` | -| e2e_model_dir | str | 无,如果使用端到端模型,该项是必填项 | 端到端模型inference模型路径 | -| e2e_limit_side_len | int | 768 | 端到端的输入图像边长限制 | -| e2e_limit_type | str | "max" | 端到端的边长限制类型,目前支持`min`, `max`,`min`表示保证图像最短边不小于`e2e_limit_side_len`,`max`表示保证图像最长边不大于`e2e_limit_side_len` | -| e2e_pgnet_score_thresh | float | 0.5 | 端到端得分阈值,小于该阈值的结果会被丢弃 | -| e2e_char_dict_path | str | "./ppocr/utils/ic15_dict.txt" | 识别的字典文件路径 | -| e2e_pgnet_valid_set | str | "totaltext" | 验证集名称,目前支持`totaltext`, `partvgg`,不同数据集对应的后处理方式不同,与训练过程保持一致即可 | -| e2e_pgnet_mode | str | "fast" | PGNet的检测结果得分计算方法,支持`fast`和`slow`,`fast`是根据polygon的外接矩形边框内的所有像素计算平均得分,`slow`是根据原始polygon内的所有像素计算平均得分,计算速度相对较慢一些,但是更加准确一些。 | - - -* 方向分类器模型相关 - -| 参数名称 | 类型 | 默认值 | 含义 | -| :--: | :--: | :--: | :--: | -| use_angle_cls | bool | False | 是否使用方向分类器 | -| cls_model_dir | str | 无,如果需要使用,则必须显式指定路径 | 方向分类器inference模型路径 | -| cls_image_shape | str | "3,48,192" | 预测尺度 | -| label_list | list | ['0', '180'] | class id对应的角度值 | -| cls_batch_num | int | 6 | 方向分类器预测的batch size | -| cls_thresh | float | 0.9 | 预测阈值,模型预测结果为180度,且得分大于该阈值时,认为最终预测结果为180度,需要翻转 | diff --git a/doc/doc_ch/inference_ppocr.md b/doc/doc_ch/inference_ppocr.md deleted file mode 100644 index 085802fc2c..0000000000 --- a/doc/doc_ch/inference_ppocr.md +++ /dev/null @@ -1,194 +0,0 @@ -# 基于Python引擎的PP-OCR模型库推理 - -本文介绍针对PP-OCR模型库的Python推理引擎使用方法,内容依次为文本检测、文本识别、方向分类器以及三者串联在CPU、GPU上的预测方法。 - - -- [基于Python引擎的PP-OCR模型库推理](#基于python引擎的pp-ocr模型库推理) - - [1. 文本检测模型推理](#1-文本检测模型推理) - - [2. 文本识别模型推理](#2-文本识别模型推理) - - [2.1 超轻量中文识别模型推理](#21-超轻量中文识别模型推理) - - [2.2 英文识别模型推理](#22-英文识别模型推理) - - [2.3 多语言模型的推理](#23-多语言模型的推理) - - [3. 方向分类模型推理](#3-方向分类模型推理) - - [4. 文本检测、方向分类和文字识别串联推理](#4-文本检测方向分类和文字识别串联推理) - - [5. TensorRT推理](#5-tensorrt推理) - - - -## 1. 文本检测模型推理 - -文本检测模型推理,默认使用DB模型的配置参数。超轻量中文检测模型推理,可以执行如下命令: - -``` -# 下载超轻量中文检测模型: -wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar -tar xf ch_PP-OCRv3_det_infer.tar -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" - -``` - -可视化文本检测结果默认保存到`./inference_results`文件夹里面,结果文件的名称前缀为'det_res'。结果示例如下: - -![](../imgs_results/det_res_00018069.jpg) - -通过参数`limit_type`和`det_limit_side_len`来对图片的尺寸进行限制, -`limit_type`可选参数为[`max`, `min`], -`det_limit_size_len` 为正整数,一般设置为32 的倍数,比如960。 - -参数默认设置为`limit_type='max', det_limit_side_len=960`。表示网络输入图像的最长边不能超过960, -如果超过这个值,会对图像做等宽比的resize操作,确保最长边为`det_limit_side_len`。 -设置为`limit_type='min', det_limit_side_len=960` 则表示限制图像的最短边为960。 - -如果输入图片的分辨率比较大,而且想使用更大的分辨率预测,可以设置det_limit_side_len 为想要的值,比如1216: - -```bash -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --det_limit_type=max --det_limit_side_len=1216 -``` - -如果想使用CPU进行预测,执行命令如下 - -```bash -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --use_gpu=False -``` - - - - -## 2. 文本识别模型推理 - - - -### 2.1 超轻量中文识别模型推理 - -**注意** `PP-OCRv3`的识别模型使用的输入shape为`3,48,320`, 如果使用其他识别模型,则需根据模型设置参数`--rec_image_shape`。此外,`PP-OCRv3`的识别模型默认使用的`rec_algorithm`为`SVTR_LCNet`,注意和原始`SVTR`的区别。 - -超轻量中文识别模型推理,可以执行如下命令: - -``` -# 下载超轻量中文识别模型: -wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar -tar xf ch_PP-OCRv3_rec_infer.tar -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/ch/word_4.jpg" --rec_model_dir="./ch_PP-OCRv3_rec_infer/" -``` - -![](../imgs_words/ch/word_4.jpg) - -执行命令后,上面图像的预测结果(识别的文本和得分)会打印到屏幕上,示例如下: - -```bash -Predicts of ./doc/imgs_words/ch/word_4.jpg:('实力活力', 0.9956803321838379) -``` - - - -### 2.2 英文识别模型推理 - -英文识别模型推理,可以执行如下命令, 注意修改字典路径: - -``` -# 下载英文数字识别模型: -wget https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_infer.tar -tar xf en_PP-OCRv3_rec_infer.tar -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./en_PP-OCRv3_rec_infer/" --rec_char_dict_path="ppocr/utils/en_dict.txt" -``` - -![](../imgs_words/en/word_1.png) - -执行命令后,上图的预测结果为: - -``` -Predicts of ./doc/imgs_words/en/word_1.png: ('JOINT', 0.998160719871521) -``` - - -### 2.3 多语言模型的推理 - -如果您需要预测的是其他语言模型,可以在[此链接](./models_list.md#%E5%A4%9A%E8%AF%AD%E8%A8%80%E8%AF%86%E5%88%AB%E6%A8%A1%E5%9E%8B)中找到对应语言的inference模型,在使用inference模型预测时,需要通过`--rec_char_dict_path`指定使用的字典路径, 同时为了得到正确的可视化结果,需要通过 `--vis_font_path` 指定可视化的字体路径,`doc/fonts/` 路径下有默认提供的小语种字体,例如韩文识别: -``` -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/multilingual/korean_mobile_v2.0_rec_infer.tar -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/korean/1.jpg" --rec_model_dir="./your inference model" --rec_char_dict_path="ppocr/utils/dict/korean_dict.txt" --vis_font_path="doc/fonts/korean.ttf" -``` - -![](../imgs_words/korean/1.jpg) - -执行命令后,上图的预测结果为: - -``` text -Predicts of ./doc/imgs_words/korean/1.jpg:('바탕으로', 0.9948904) -``` - - - -## 3. 方向分类模型推理 - -方向分类模型推理,可以执行如下命令: - -``` -# 下载超轻量中文方向分类器模型: -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar -tar xf ch_ppocr_mobile_v2.0_cls_infer.tar -python3 tools/infer/predict_cls.py --image_dir="./doc/imgs_words/ch/word_4.jpg" --cls_model_dir="ch_ppocr_mobile_v2.0_cls_infer" -``` - -![](../imgs_words/ch/word_1.jpg) - -执行命令后,上面图像的预测结果(分类的方向和得分)会打印到屏幕上,示例如下: - -``` -Predicts of ./doc/imgs_words/ch/word_4.jpg:['0', 0.9999982] -``` - - - -## 4. 文本检测、方向分类和文字识别串联推理 - -**注意** `PP-OCRv3`的识别模型使用的输入shape为`3,48,320`, 如果使用其他识别模型,则需根据模型设置参数`--rec_image_shape`。此外,`PP-OCRv3`的识别模型默认使用的`rec_algorithm`为`SVTR_LCNet`,注意和原始`SVTR`的区别。 - -以超轻量中文OCR模型推理为例,在执行预测时,需要通过参数`image_dir`指定单张图像或者图像集合的路径,也支持PDF文件、参数`det_model_dir`,`cls_model_dir`和`rec_model_dir`分别指定检测,方向分类和识别的inference模型路径。参数`use_angle_cls`用于控制是否启用方向分类模型。`use_mp`表示是否使用多进程(Paddle Inference并不是线程安全,建议使用多进程)。`total_process_num`表示在使用多进程时的进程数。可视化识别结果默认保存到 ./inference_results 文件夹里面。 - -```shell -# 使用方向分类器 -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --cls_model_dir="./cls/" --rec_model_dir="./ch_PP-OCRv3_rec_infer/" --use_angle_cls=true -# 不使用方向分类器 -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --rec_model_dir="./ch_PP-OCRv3_rec_infer/" --use_angle_cls=false -# 使用多进程 -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --rec_model_dir="./ch_PP-OCRv3_rec_infer/" --use_angle_cls=false --use_mp=True --total_process_num=6 -# 使用PDF文件,可以通过使用`page_num`参数来控制推理前几页,默认为0,表示推理所有页 -python3 tools/infer/predict_system.py --image_dir="./xxx.pdf" --det_model_dir="./ch_PP-OCRv3_det_infer/" --cls_model_dir="./cls/" --rec_model_dir="./ch_PP-OCRv3_rec_infer/" --use_angle_cls=true --page_num=2 -``` - - -执行命令后,识别结果图像如下: - -![](../imgs_results/system_res_00018069_v3.jpg) - -更多关于推理超参数的配置与解释,请参考:[模型推理超参数解释教程](./inference_args.md)。 - - -## 5. TensorRT推理 - -Paddle Inference 采用子图的形式集成 TensorRT,针对 GPU 推理场景,TensorRT 可对一些子图进行优化,包括 OP 的横向和纵向融合,过滤冗余的 OP,并为 OP 自动选择最优的 kernel,加快推理速度。 - -如果希望使用Paddle Inference进行TRT推理,一般需要2个步骤。 - -* (1)收集该模型关于特定数据集的动态shape信息,并存储到文件中。 -* (2)加载动态shape信息文件,进行TRT推理。 - -以文本检测模型为例,首先使用下面的命令,生成动态shape文件,最终会在`ch_PP-OCRv3_det_infer`目录下面生成`det_trt_dynamic_shape.txt`的文件,该文件即存储了动态shape信息的文件。 - -```bash -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --use_tensorrt=True -``` - -上面的推理过程仅用于收集动态shape信息,没有用TRT进行推理。 - -运行完成以后,再使用下面的命令,进行TRT推理。 - -```bash -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --use_tensorrt=True -``` - -**注意:** - -* 如果在第一步中,已经存在动态shape信息文件,则无需重新收集,直接预测,即使用TRT推理;如果希望重新生成动态shape信息文件,则需要先将模型目录下的动态shape信息文件删掉,再重新生成。 -* 动态shape信息文件一般情况下仅需生成一次。在实际部署过程中,建议首先在线下验证集或者测试集合上生成好,之后可以直接加载该文件进行线上TRT推理。 diff --git a/doc/doc_ch/installation.md b/doc/doc_ch/installation.md deleted file mode 100644 index 962a926911..0000000000 --- a/doc/doc_ch/installation.md +++ /dev/null @@ -1,62 +0,0 @@ -## 快速安装 - -经测试PaddleOCR可在glibc 2.23上运行,您也可以测试其他glibc版本或安装glic 2.23 -PaddleOCR 工作环境 -- PaddlePaddle 2.0.0 -- python3.7 -- glibc 2.23 -- cuDNN 7.6+ (GPU) - -建议使用我们提供的docker运行PaddleOCR,有关docker、nvidia-docker使用请参考[链接](https://www.runoob.com/docker/docker-tutorial.html/)。 - -*如您希望使用 mac 或 windows直接运行预测代码,可以从第2步开始执行。* - -**1. (建议)准备docker环境。第一次使用这个镜像,会自动下载该镜像,请耐心等待。** -``` -# 切换到工作目录下 -cd /home/Projects -# 首次运行需创建一个docker容器,再次运行时不需要运行当前命令 -# 创建一个名字为ppocr的docker容器,并将当前目录映射到容器的/paddle目录下 - -如果您希望在CPU环境下使用docker,使用docker而不是nvidia-docker创建docker -sudo docker run --name ppocr -v $PWD:/paddle --network=host -it paddlepaddle/paddle:latest-dev-cuda10.1-cudnn7-gcc82 /bin/bash - -如果使用CUDA10,请运行以下命令创建容器,设置docker容器共享内存shm-size为64G,建议设置32G以上 -sudo nvidia-docker run --name ppocr -v $PWD:/paddle --shm-size=64G --network=host -it paddlepaddle/paddle:latest-dev-cuda10.1-cudnn7-gcc82 /bin/bash - -您也可以访问[DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/)获取与您机器适配的镜像。 - -# ctrl+P+Q可退出docker 容器,重新进入docker 容器使用如下命令 -sudo docker container exec -it ppocr /bin/bash -``` - -**2. 安装PaddlePaddle 2.0** -``` -pip3 install --upgrade pip - -如果您的机器安装的是CUDA9或CUDA10,请运行以下命令安装 -python3 -m pip install paddlepaddle-gpu==2.0.0 -i https://mirror.baidu.com/pypi/simple - -如果您的机器是CPU,请运行以下命令安装 - -python3 -m pip install paddlepaddle==2.0.0 -i https://mirror.baidu.com/pypi/simple - -更多的版本需求,请参照[安装文档](https://www.paddlepaddle.org.cn/install/quick)中的说明进行操作。 -``` - -**3. 克隆PaddleOCR repo代码** -``` -【推荐】git clone https://github.com/PaddlePaddle/PaddleOCR - -如果因为网络问题无法pull成功,也可选择使用码云上的托管: - -git clone https://gitee.com/paddlepaddle/PaddleOCR - -注:码云托管代码可能无法实时同步本github项目更新,存在3~5天延时,请优先使用推荐方式。 -``` - -**4. 安装第三方库** -``` -cd PaddleOCR -pip3 install -r requirements.txt -``` diff --git a/doc/doc_ch/kie.md b/doc/doc_ch/kie.md deleted file mode 100644 index 0ebb05d0e1..0000000000 --- a/doc/doc_ch/kie.md +++ /dev/null @@ -1,507 +0,0 @@ -# 关键信息抽取 - -本文提供了PaddleOCR关键信息抽取的全流程指南,包括语义实体识别 (Semantic Entity Recognition) 以及关系抽取 (Relation Extraction, RE) 任务的数据准备、模型训练、调优、评估、预测,各个阶段的详细说明。 - -- [1. 数据准备](#1-数据准备) - - [1.1. 准备数据集](#11-准备数据集) - - [1.2. 自定义数据集](#12-自定义数据集) - - [1.3. 数据下载](#13-数据下载) -- [2. 开始训练](#2-开始训练) - - [2.1. 启动训练](#21-启动训练) - - [2.2. 断点训练](#22-断点训练) - - [2.3. 混合精度训练](#24-混合精度训练) - - [2.4. 分布式训练](#25-分布式训练) - - [2.5. 知识蒸馏训练](#26-知识蒸馏训练) - - [2.6. 其他训练环境](#27-其他训练环境) -- [3. 模型评估与预测](#3-模型评估与预测) - - [3.1. 指标评估](#31-指标评估) - - [3.2. 测试信息抽取效果](#32-测试识别效果) -- [4. 模型导出与预测](#4-模型导出与预测) -- [5. FAQ](#5-faq) - -# 1. 数据准备 - -## 1.1. 准备数据集 - -在训练信息抽取相关模型时,PaddleOCR支持以下数据格式。 - - `通用数据` 用于训练以文本文件存储的数据集(SimpleDataSet); - -训练数据的默认存储路径是 `PaddleOCR/train_data`,如果您的磁盘上已有数据集,只需创建软链接至数据集目录: - -``` -# linux and mac os -ln -sf /train_data/dataset -# windows -mklink /d /train_data/dataset -``` - -## 1.2. 自定义数据集 - -训练过程中一般包含训练集与验证集,二者数据格式相同,下面介绍如何自定义数据集。 - -**(1)训练集** - -建议将训练图片放入同一个文件夹,并用一个文本文件记录图片路径和标签,文本文件里的内容如下: - -```py -" 图像文件名 图像标注信息 " -zh_train_0.jpg [{"transcription": "汇丰晋信", "label": "other", "points": [[104, 114], [530, 114], [530, 175], [104, 175]], "id": 1, "linking": []}, {"transcription": "受理时间:", "label": "question", "points": [[126, 267], [266, 267], [266, 305], [126, 305]], "id": 7, "linking": [[7, 13]]}, {"transcription": "2020.6.15", "label": "answer", "points": [[321, 239], [537, 239], [537, 285], [321, 285]], "id": 13, "linking": [[7, 13]]}] -zh_train_1.jpg [{"transcription": "中国人体器官捐献", "label": "other", "points": [[544, 459], [954, 459], [954, 517], [544, 517]], "id": 1, "linking": []}, {"transcription": ">编号:MC545715483585", "label": "other", "points": [[1462, 470], [2054, 470], [2054, 543], [1462, 543]], "id": 10, "linking": []}, {"transcription": "CHINAORGANDONATION", "label": "other", "points": [[543, 516], [958, 516], [958, 551], [543, 551]], "id": 14, "linking": []}, {"transcription": "中国人体器官捐献志愿登记表", "label": "header", "points": [[635, 793], [1892, 793], [1892, 904], [635, 904]], "id": 18, "linking": []}] -... -``` - -**注意:** 文本文件中默认请将图片路径和图片标签用 `\t` 分割,如用其他方式分割将造成训练报错。 - -其中图像标注信息字符串经过json解析之后可以得到一个列表信息,列表中每个元素是一个字典,存储了每个文本行的需要信息,各个字段的含义如下。 - -- transcription: 存储了文本行的文字内容 -- label: 该文本行内容所属的类别 -- points: 存储文本行的四点位置信息 -- id: 存储文本行的id信息,用于RE任务的训练 -- linking: 存储文本行的之间的连接信息,用于RE任务的训练 - -**(2)验证集** - -验证集构建方式与训练集相同。 - -**(3)字典文件** - -训练集与验证集中的文本行包含标签信息,所有标签的列表存在字典文件中(如`class_list.txt`),字典文件中的每一行表示为一个类别名称。 - -以XFUND_zh数据为例,共包含4个类别,字典文件内容如下所示。 - -``` -OTHER -QUESTION -ANSWER -HEADER -``` - -在标注文件中,每个标注的文本行内容的`label`字段标注信息需要属于字典内容。 - -最终数据集应有如下文件结构: - -``` -|-train_data - |-data_name - |- train.json - |- train - |- zh_train_0.png - |- zh_train_1.jpg - | ... - |- val.json - |- val - |- zh_val_0.png - |- zh_val_1.jpg - | ... -``` - -**注:** - -- 标注文件中的类别信息不区分大小写,如`HEADER`与`header`会被解析为相同的类别id,因此在标注的时候,不能使用小写处理后相同的字符串表示不同的类别。 -- 在整理标注文件的时候,建议将other这个类别(其他,无需关注的文本行可以标注为other)放在第一行,在解析的时候,会将`other`类别的类别id解析为0,后续不会对该类进行可视化。 - -## 1.3. 数据下载 - -如果你没有本地数据集,可以从[XFUND](https://github.com/doc-analysis/XFUND)或者[FUNSD](https://guillaumejaume.github.io/FUNSD/)官网下载数据,然后使用XFUND与FUNSD的处理脚本([XFUND](../../ppstructure/kie/tools/trans_xfun_data.py), [FUNSD](../../ppstructure/kie/tools/trans_funsd_label.py)),生成用于PaddleOCR训练的数据格式,并使用公开数据集快速体验关键信息抽取的流程。 - -更多关于公开数据集的介绍,请参考[关键信息抽取数据集说明文档](./dataset/kie_datasets.md)。 - -PaddleOCR也支持了关键信息抽取模型的标注,具体使用方法请参考:[PPOCRLabel使用文档](https://github.com/PFCCLab/PPOCRLabel/blob/main/README_ch.md)。 - - -# 2. 开始训练 - -PaddleOCR提供了训练脚本、评估脚本和预测脚本,本节将以 VI-LayoutXLM 多模态预训练模型为例进行讲解。 - -> 如果希望使用基于SDMGR的关键信息抽取算法,请参考:[SDMGR使用](./algorithm_kie_sdmgr.md)。 - -## 2.1. 启动训练 - -如果你没有使用自定义数据集,可以使用PaddleOCR中已经处理好的XFUND_zh数据集进行快速体验。 - -```bash -mkdir train_data -cd train_data -wget https://paddleocr.bj.bcebos.com/ppstructure/dataset/XFUND.tar && tar -xf XFUND.tar -cd .. -``` - -如果不希望训练,直接体验后面的模型评估、预测、动转静、推理的流程,可以下载PaddleOCR中提供的预训练模型,并跳过2.1部分。 - -使用下面的方法,下载基于XFUND数据的SER与RE任务预训练模型。 - -```bash -mkdir pretrained_model -cd pretrained_model -# 下载并解压SER预训练模型 -wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar & tar -xf ser_vi_layoutxlm_xfund_pretrained.tar - -# 下载并解压RE预训练模型 -wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar & tar -xf re_vi_layoutxlm_xfund_pretrained.tar -``` - -开始训练: - -- 如果您安装的是cpu版本,请将配置文件中的 `use_gpu` 字段修改为false -- PaddleOCR在训练时,会默认下载VI-LayoutXLM预训练模型,这里无需预先下载。 - -```bash -# GPU训练 支持单卡,多卡训练 -# 训练日志会自动保存到 配置文件中"{Global.save_model_dir}" 下的train.log文件中 - -# SER单卡训练 -python3 tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml - -# SER多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml - -# RE任务单卡训练 -python3 tools/train.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml -``` - -以SER任务为例,正常启动训练后,会看到以下log输出: - -``` -[2022/08/08 16:28:28] ppocr INFO: epoch: [1/200], global_step: 10, lr: 0.000006, loss: 1.871535, avg_reader_cost: 0.28200 s, avg_batch_cost: 0.82318 s, avg_samples: 8.0, ips: 9.71838 samples/s, eta: 0:51:59 -[2022/08/08 16:28:33] ppocr INFO: epoch: [1/200], global_step: 19, lr: 0.000018, loss: 1.461939, avg_reader_cost: 0.00042 s, avg_batch_cost: 0.32037 s, avg_samples: 6.9, ips: 21.53773 samples/s, eta: 0:37:55 -[2022/08/08 16:28:39] ppocr INFO: cur metric, precision: 0.11526348939743859, recall: 0.19776657060518732, hmean: 0.14564265817747712, fps: 34.008392345050055 -[2022/08/08 16:28:45] ppocr INFO: save best model is to ./output/ser_vi_layoutxlm_xfund_zh/best_accuracy -[2022/08/08 16:28:45] ppocr INFO: best metric, hmean: 0.14564265817747712, precision: 0.11526348939743859, recall: 0.19776657060518732, fps: 34.008392345050055, best_epoch: 1 -[2022/08/08 16:28:51] ppocr INFO: save model in ./output/ser_vi_layoutxlm_xfund_zh/latest -``` - -log 中自动打印如下信息: - -| 字段 | 含义 | -| :----: | :------: | -| epoch | 当前迭代轮次 | -| iter | 当前迭代次数 | -| lr | 当前学习率 | -| loss | 当前损失函数 | -| reader_cost | 当前 batch 数据处理耗时 | -| batch_cost | 当前 batch 总耗时 | -| samples | 当前 batch 内的样本数 | -| ips | 每秒处理图片的数量 | - - -PaddleOCR支持训练和评估交替进行, 可以在 `configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` 中修改 `eval_batch_step` 设置评估频率,默认每19个iter评估一次。评估过程中默认将最佳hmean模型,保存为 `output/ser_vi_layoutxlm_xfund_zh/best_accuracy/` 。 - -如果验证集很大,测试将会比较耗时,建议减少评估次数,或训练完再进行评估。 - -**提示:** 可通过 -c 参数选择 `configs/kie/` 路径下的多种模型配置进行训练,PaddleOCR支持的信息抽取算法可以参考[前沿算法列表](./algorithm_overview.md)。 - - -如果你希望训练自己的数据集,需要修改配置文件中的数据配置、字典文件以及类别数。 - - -以 `configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` 为例,修改的内容如下所示。 - -```yaml - -Architecture: - # ... - Backbone: - name: LayoutXLMForSer - pretrained: True - mode: vi - # 由于采用BIO标注,假设字典中包含n个字段(包含other)时,则类别数为2n-1; 假设字典中包含n个字段(不含other)时,则类别数为2n+1。否则在train过程会报:IndexError: (OutOfRange) label value should less than the shape of axis dimension 。 - num_classes: &num_classes 7 - -PostProcess: - name: kieSerTokenLayoutLMPostProcess - # 修改字典文件的路径为你自定义的数据集的字典路径 - class_path: &class_path train_data/XFUND/class_list_xfun.txt - -Train: - dataset: - name: SimpleDataSet - # 修改为你自己的训练数据目录 - data_dir: train_data/XFUND/zh_train/image - # 修改为你自己的训练数据标签文件 - label_file_list: - - train_data/XFUND/zh_train/train.json - ... - loader: - # 训练时的单卡batch_size - batch_size_per_card: 8 - ... - -Eval: - dataset: - name: SimpleDataSet - # 修改为你自己的验证数据目录 - data_dir: train_data/XFUND/zh_val/image - # 修改为你自己的验证数据标签文件 - label_file_list: - - train_data/XFUND/zh_val/val.json - ... - loader: - # 验证时的单卡batch_size - batch_size_per_card: 8 -``` - -**注意,预测/评估时的配置文件请务必与训练一致。** - -## 2.2. 断点训练 - -如果训练程序中断,如果希望加载训练中断的模型从而恢复训练,可以通过指定` Architecture.Backbone.checkpoints`指定要加载的模型路径: - -```bash -python3 tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy -``` - -**注意**: - -- `Architecture.Backbone.checkpoints`的优先级高于`Architecture.Backbone.pretrained`,需要加载之前训练好的训练模型进行模型微调、恢复训练、模型评估时,需要使用`Architecture.Backbone.checkpoints`指定模型参数路径;如果需要使用默认提供的通用预训练模型进行训练,则需要指定`Architecture.Backbone.pretrained`为`True`,同时指定`Architecture.Backbone.checkpoints`为空(`null`)。 -- LayoutXLM系列模型均是调用了PaddleNLP中的预训练模型,模型加载与保存的逻辑与PaddleNLP基本一致,因此在这里不需要指定`Global.pretrained_model`或者`Global.checkpoints`参数;此外,LayoutXLM系列模型的蒸馏训练目前不支持断点训练。 - - -## 2.3. 混合精度训练 - -coming soon! - -## 2.4. 分布式训练 - -多机多卡训练时,通过 `--ips` 参数设置使用的机器IP地址,通过 `--gpus` 参数设置使用的GPU ID: - -```bash -python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -``` - -**注意:** (1)采用多机多卡训练时,需要替换上面命令中的ips值为您机器的地址,机器之间需要能够相互ping通;(2)训练时需要在多个机器上分别启动命令。查看机器ip地址的命令为`ifconfig`;(3)更多关于分布式训练的性能优势等信息,请参考:[分布式训练教程](./distributed_training.md)。 - -## 2.5. 知识蒸馏训练 - -PaddleOCR支持了基于U-DML知识蒸馏的关键信息抽取模型训练过程,配置文件请参考:[ser_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml),更多关于知识蒸馏的说明文档请参考:[知识蒸馏说明文档](./knowledge_distillation.md)。 - -**注意**: PaddleOCR中LayoutXLM系列关键信息抽取模型的保存与加载逻辑与PaddleNLP保持一致,因此在蒸馏的过程中仅保存了学生模型的参数,如果希望使用保存的模型进行评估,需要使用学生模型的配置(上面的蒸馏文件对应的学生模型为[ser_vi_layoutxlm_xfund_zh.yml](../../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml)) - - -## 2.6. 其他训练环境 - -- Windows GPU/CPU -在Windows平台上与Linux平台略有不同: -Windows平台只支持`单卡`的训练与预测,指定GPU进行训练`set CUDA_VISIBLE_DEVICES=0` -在Windows平台,DataLoader只支持单进程模式,因此需要设置 `num_workers` 为0; - -- macOS -不支持GPU模式,需要在配置文件中设置`use_gpu`为False,其余训练评估预测命令与Linux GPU完全相同。 - -- Linux DCU -DCU设备上运行需要设置环境变量 `export HIP_VISIBLE_DEVICES=0,1,2,3`,其余训练评估预测命令与Linux GPU完全相同。 - - -# 3. 模型评估与预测 - -## 3.1. 指标评估 - -训练中模型参数默认保存在`Global.save_model_dir`目录下。在评估指标时,需要设置`Architecture.Backbone.checkpoints`指向保存的参数文件。评估数据集可以通过 `configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` 修改Eval中的 `label_file_path` 设置。 - -```bash -# GPU 评估, Global.checkpoints 为待测权重 -python3 tools/eval.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy -``` - -会输出以下信息,打印出precision、recall、hmean等信息。 - -```py -[2022/08/09 07:59:28] ppocr INFO: metric eval *************** -[2022/08/09 07:59:28] ppocr INFO: precision:0.697476609016161 -[2022/08/09 07:59:28] ppocr INFO: recall:0.8861671469740634 -[2022/08/09 07:59:28] ppocr INFO: hmean:0.7805806758686339 -[2022/08/09 07:59:28] ppocr INFO: fps:17.367364606899105 -``` - - -## 3.2. 测试信息抽取结果 - -使用 PaddleOCR 训练好的模型,可以通过以下脚本进行快速预测。 - -默认预测的图片存储在 `infer_img` 里,通过 `-o Architecture.Backbone.checkpoints` 加载训练好的参数文件: - -根据配置文件中设置的 `save_model_dir` 和 `save_epoch_step` 字段,会有以下几种参数被保存下来: - -``` -output/ser_vi_layoutxlm_xfund_zh/ -├── best_accuracy - ├── metric.states - ├── model_config.json - ├── model_state.pdparams -├── best_accuracy.pdopt -├── config.yml -├── train.log -├── latest - ├── metric.states - ├── model_config.json - ├── model_state.pdparams -├── latest.pdopt -``` - -其中 best_accuracy.* 是评估集上的最优模型;latest.* 是最新保存的一个模型。 - - -预测使用的配置文件必须与训练一致,如您通过 `python3 tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` 完成了模型的训练过程。 - -您可以使用如下命令进行中文模型预测。 - - -```bash -python3 tools/infer_kie_token_ser.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.infer_img=./ppstructure/docs/kie/input/zh_val_42.jpg -``` - -预测图片如下所示,图片会存储在`Global.save_res_path`路径中。 - -
- -
- -预测过程中,默认会加载PP-OCRv3的检测识别模型,用于OCR的信息抽取,如果希望加载预先获取的OCR结果,可以使用下面的方式进行预测,指定`Global.infer_img`为标注文件,其中包含图片路径以及OCR信息,同时指定`Global.infer_mode`为False,表示此时不使用OCR预测引擎。 - -```bash -python3 tools/infer_kie_token_ser.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.infer_img=./train_data/XFUND/zh_val/val.json Global.infer_mode=False -``` - -对于上述图片,如果使用标注的OCR结果进行信息抽取,预测结果如下。 - -
- -
- -可以看出,部分检测框信息更加准确,但是整体信息抽取识别结果基本一致。 - - -在RE任务模型预测时,需要先给出模型SER结果,因此需要同时加载SER的配置文件与模型权重,示例如下。 - - -```bash -python3 ./tools/infer_kie_token_ser_re.py \ - -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \ - -o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_udml_xfund_zh/best_accuracy/ \ - Global.infer_img=./train_data/XFUND/zh_val/image/ \ - -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \ - -o_ser Architecture.Backbone.checkpoints=pretrain_models/ \ - ser_vi_layoutxlm_udml_xfund_zh/best_accuracy/ -``` - -预测结果如下所示。 - -
- -
- - -如果希望使用标注或者预先获取的OCR信息进行关键信息抽取,同上,可以指定`Global.infer_mode`为False,指定`Global.infer_img`为标注文件。 - -```bash -python3 ./tools/infer_kie_token_ser_re.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_udml_xfund_zh/re_layoutxlm_xfund_zh_v4_udml/best_accuracy/ Global.infer_img=./train_data/XFUND/zh_val/val.json Global.infer_mode=False -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o_ser Architecture.Backbone.checkpoints=pretrain_models/ser_vi_layoutxlm_udml_xfund_zh/best_accuracy/ -``` - -其中`c_ser`表示SER的配置文件,`o_ser` 后面需要加上待修改的SER模型与配置文件,如预训练权重等。 - - -预测结果如下所示。 - -
- -
- -可以看出,直接使用标注的OCR结果的RE预测结果要更加准确一些。 - -# 4. 模型导出与预测 - - -## 4.1 模型导出 - -inference 模型(`paddle.jit.save`保存的模型) -一般是模型训练,把模型结构和模型参数保存在文件中的固化模型,多用于预测部署场景。 -训练过程中保存的模型是checkpoints模型,保存的只有模型的参数,多用于恢复训练等。 -与checkpoints模型相比,inference 模型会额外保存模型的结构信息,在预测部署、加速推理上性能优越,灵活方便,适合于实际系统集成。 - -信息抽取模型中的SER任务转inference模型步骤如下: - -```bash -# -c 后面设置训练算法的yml配置文件 -# -o 配置可选参数 -# Architecture.Backbone.checkpoints 参数设置待转换的训练模型地址 -# Global.save_inference_dir 参数设置转换的模型将保存的地址 - -python3 tools/export_model.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.save_inference_dir=./inference/ser_vi_layoutxlm -``` - -转换成功后,在目录下有三个文件: - -``` -inference/ser_vi_layoutxlm/ - ├── inference.pdiparams # inference模型的参数文件 - ├── inference.pdiparams.info # inference模型的参数信息,可忽略 - └── inference.pdmodel # inference模型的模型结构文件 -``` - -信息抽取模型中的RE任务转inference模型步骤如下: - -``` bash -# -c 后面设置训练算法的yml配置文件 -# -o 配置可选参数 -# Architecture.Backbone.checkpoints 参数设置待转换的训练模型地址 -# Global.save_inference_dir 参数设置转换的模型将保存的地址 - -python3 tools/export_model.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/re_vi_layoutxlm_xfund_zh/best_accuracy Global.save_inference_dir=./inference/re_vi_layoutxlm -``` - -转换成功后,在目录下有三个文件: - -``` -inference/re_vi_layoutxlm/ - ├── inference.pdiparams # inference模型的参数文件 - ├── inference.pdiparams.info # inference模型的参数信息,可忽略 - └── inference.pdmodel # inference模型的模型结构文件 -``` - -## 4.2 模型推理 - -VI-LayoutXLM模型基于SER任务进行推理,可以执行如下命令: - -```bash -cd ppstructure -python3 kie/predict_kie_token_ser.py \ - --kie_algorithm=LayoutXLM \ - --ser_model_dir=../inference/ser_vi_layoutxlm \ - --image_dir=./docs/kie/input/zh_val_42.jpg \ - --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \ - --vis_font_path=../doc/fonts/simfang.ttf \ - --ocr_order_method="tb-yx" -``` - -可视化SER结果结果默认保存到`./output`文件夹里面。结果示例如下: - -
- -
- -VI-LayoutXLM模型基于RE任务进行推理,可以执行如下命令: - -```bash -cd ppstructure -python3 kie/predict_kie_token_ser_re.py \ - --kie_algorithm=LayoutXLM \ - --re_model_dir=../inference/re_vi_layoutxlm \ - --ser_model_dir=../inference/ser_vi_layoutxlm \ - --use_visual_backbone=False \ - --image_dir=./docs/kie/input/zh_val_42.jpg \ - --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \ - --vis_font_path=../doc/fonts/simfang.ttf \ - --ocr_order_method="tb-yx" -``` - -RE可视化结果默认保存到`./output`文件夹里面,结果示例如下: - -
- -
- -# 5. FAQ - -Q1: 训练模型转inference 模型之后预测效果不一致? - -**A**:该问题多是trained model预测时候的预处理、后处理参数和inference model预测的时候的预处理、后处理参数不一致导致的。可以对比训练使用的配置文件中的预处理、后处理和预测时是否存在差异。 diff --git a/doc/doc_ch/knowledge_distillation.md b/doc/doc_ch/knowledge_distillation.md deleted file mode 100644 index 8c6cca5d1c..0000000000 --- a/doc/doc_ch/knowledge_distillation.md +++ /dev/null @@ -1,622 +0,0 @@ - -# 知识蒸馏 - -- [知识蒸馏](#知识蒸馏) - - [1. 简介](#1-简介) - - [1.1 知识蒸馏介绍](#11-知识蒸馏介绍) - - [1.2 PaddleOCR知识蒸馏简介](#12-paddleocr知识蒸馏简介) - - [2. 配置文件解析](#2-配置文件解析) - - [2.1 识别配置文件解析](#21-识别配置文件解析) - - [2.1.1 模型结构](#211-模型结构) - - [2.1.2 损失函数](#212-损失函数) - - [2.1.3 后处理](#213-后处理) - - [2.1.4 指标计算](#214-指标计算) - - [2.1.5 蒸馏模型微调](#215-蒸馏模型微调) - - [2.2 检测配置文件解析](#22-检测配置文件解析) - - [2.2.1 模型结构](#221-模型结构) - - [2.2.2 损失函数](#222-损失函数) - - [2.2.3 后处理](#223-后处理) - - [2.2.4 蒸馏指标计算](#224-蒸馏指标计算) - - [2.2.5 检测蒸馏模型finetune](#225-检测蒸馏模型finetune) - - -## 1. 简介 - -### 1.1 知识蒸馏介绍 - -近年来,深度神经网络在计算机视觉、自然语言处理等领域被验证是一种极其有效的解决问题的方法。通过构建合适的神经网络,加以训练,最终网络模型的性能指标基本上都会超过传统算法。 - -在数据量足够大的情况下,通过合理构建网络模型的方式增加其参数量,可以显著改善模型性能,但是这又带来了模型复杂度急剧提升的问题。大模型在实际场景中使用的成本较高。 - -深度神经网络一般有较多的参数冗余,目前有几种主要的方法对模型进行压缩,减小其参数量。如裁剪、量化、知识蒸馏等,其中知识蒸馏是指使用教师模型(teacher model)去指导学生模型(student model)学习特定任务,保证小模型在参数量不变的情况下,得到比较大的性能提升。 - -此外,在知识蒸馏任务中,也衍生出了互学习的模型训练方法,论文[Deep Mutual Learning](https://arxiv.org/abs/1706.00384)中指出,使用两个完全相同的模型在训练的过程中互相监督,可以达到比单个模型训练更好的效果。 - - -### 1.2 PaddleOCR知识蒸馏简介 - -无论是大模型蒸馏小模型,还是小模型之间互相学习,更新参数,他们本质上是都是不同模型之间输出或者特征图(feature map)之间的相互监督,区别仅在于 (1) 模型是否需要固定参数。(2) 模型是否需要加载预训练模型。 - -对于大模型蒸馏小模型的情况,大模型一般需要加载预训练模型并固定参数;对于小模型之间互相蒸馏的情况,小模型一般都不加载预训练模型,参数也都是可学习的状态。 - -在知识蒸馏任务中,不只有2个模型之间进行蒸馏的情况,多个模型之间互相学习的情况也非常普遍。因此在知识蒸馏代码框架中,也有必要支持该种类别的蒸馏方法。 - -PaddleOCR中集成了知识蒸馏的算法,具体地,有以下几个主要的特点: -- 支持任意网络的互相学习,不要求子网络结构完全一致或者具有预训练模型;同时子网络数量也没有任何限制,只需要在配置文件中添加即可。 -- 支持loss函数通过配置文件任意配置,不仅可以使用某种loss,也可以使用多种loss的组合 -- 支持知识蒸馏训练、预测、评估与导出等所有模型相关的环境,方便使用与部署。 - - -通过知识蒸馏,在中英文通用文字识别任务中,不增加任何预测耗时的情况下,可以给模型带来3%以上的精度提升,结合学习率调整策略以及模型结构微调策略,最终提升提升超过5%。 - - - -## 2. 配置文件解析 - -在知识蒸馏训练的过程中,数据预处理、优化器、学习率、全局的一些属性没有任何变化。模型结构、损失函数、后处理、指标计算等模块的配置文件需要进行微调。 - -下面以识别与检测的知识蒸馏配置文件为例,对知识蒸馏的训练与配置进行解析。 - - -### 2.1 识别配置文件解析 - -配置文件在[ch_PP-OCRv3_rec_distillation.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml)。 - - -#### 2.1.1 模型结构 - -知识蒸馏任务中,模型结构配置如下所示。 - -```yaml -Architecture: - model_type: &model_type "rec" # 模型类别,rec、det等,每个子网络的模型类别 - name: DistillationModel # 结构名称,蒸馏任务中,为DistillationModel,用于构建对应的结构 - algorithm: Distillation # 算法名称 - Models: # 模型,包含子网络的配置信息 - Teacher: # 子网络名称,至少需要包含`pretrained`与`freeze_params`信息,其他的参数为子网络的构造参数 - pretrained: # 该子网络是否需要加载预训练模型 - freeze_params: false # 是否需要固定参数 - return_all_feats: true # 子网络的参数,表示是否需要返回所有的features,如果为False,则只返回最后的输出 - model_type: *model_type # 模型类别 - algorithm: SVTR # 子网络的算法名称,该子网络其余参数均为构造参数,与普通的模型训练配置一致 - Transform: - Backbone: - name: MobileNetV1Enhance - scale: 0.5 - last_conv_stride: [1, 2] - last_pool_type: avg - Head: - name: MultiHead - head_list: - - CTCHead: - Neck: - name: svtr - dims: 64 - depth: 2 - hidden_dims: 120 - use_guide: True - Head: - fc_decay: 0.00001 - - SARHead: - enc_dim: 512 - max_text_length: *max_text_length - Student: - pretrained: - freeze_params: false - return_all_feats: true - model_type: *model_type - algorithm: SVTR - Transform: - Backbone: - name: MobileNetV1Enhance - scale: 0.5 - last_conv_stride: [1, 2] - last_pool_type: avg - Head: - name: MultiHead - head_list: - - CTCHead: - Neck: - name: svtr - dims: 64 - depth: 2 - hidden_dims: 120 - use_guide: True - Head: - fc_decay: 0.00001 - - SARHead: - enc_dim: 512 - max_text_length: *max_text_length -``` - -当然,这里如果希望添加更多的子网络进行训练,也可以按照`Student`与`Teacher`的添加方式,在配置文件中添加相应的字段。比如说如果希望有3个模型互相监督,共同训练,那么`Architecture`可以写为如下格式。 - -```yaml -Architecture: - model_type: &model_type "rec" - name: DistillationModel - algorithm: Distillation - Models: - Teacher: - pretrained: - freeze_params: false - return_all_feats: true - model_type: *model_type - algorithm: SVTR - Transform: - Backbone: - name: MobileNetV1Enhance - scale: 0.5 - last_conv_stride: [1, 2] - last_pool_type: avg - Head: - name: MultiHead - head_list: - - CTCHead: - Neck: - name: svtr - dims: 64 - depth: 2 - hidden_dims: 120 - use_guide: True - Head: - fc_decay: 0.00001 - - SARHead: - enc_dim: 512 - max_text_length: *max_text_length - Student: - pretrained: - freeze_params: false - return_all_feats: true - model_type: *model_type - algorithm: SVTR - Transform: - Backbone: - name: MobileNetV1Enhance - scale: 0.5 - last_conv_stride: [1, 2] - last_pool_type: avg - Head: - name: MultiHead - head_list: - - CTCHead: - Neck: - name: svtr - dims: 64 - depth: 2 - hidden_dims: 120 - use_guide: True - Head: - fc_decay: 0.00001 - - SARHead: - enc_dim: 512 - max_text_length: *max_text_length - Student2: - pretrained: - freeze_params: false - return_all_feats: true - model_type: *model_type - algorithm: SVTR - Transform: - Backbone: - name: MobileNetV1Enhance - scale: 0.5 - last_conv_stride: [1, 2] - last_pool_type: avg - Head: - name: MultiHead - head_list: - - CTCHead: - Neck: - name: svtr - dims: 64 - depth: 2 - hidden_dims: 120 - use_guide: True - Head: - fc_decay: 0.00001 - - SARHead: - enc_dim: 512 - max_text_length: *max_text_length -``` - -最终该模型训练时,包含3个子网络:`Teacher`, `Student`, `Student2`。 - -蒸馏模型`DistillationModel`类的具体实现代码可以参考[distillation_model.py](../../ppocr/modeling/architectures/distillation_model.py)。 - -最终模型`forward`输出为一个字典,key为所有的子网络名称,例如这里为`Student`与`Teacher`,value为对应子网络的输出,可以为`Tensor`(只返回该网络的最后一层)和`dict`(也返回了中间的特征信息)。 - -在识别任务中,为了添加更多损失函数,保证蒸馏方法的可扩展性,将每个子网络的输出保存为`dict`,其中包含子模块输出。以该识别模型为例,每个子网络的输出结果均为`dict`,key包含`backbone_out`,`neck_out`, `head_out`,`value`为对应模块的tensor,最终对于上述配置文件,`DistillationModel`的输出格式如下。 - -```json -{ - "Teacher": { - "backbone_out": tensor, - "neck_out": tensor, - "head_out": tensor, - }, - "Student": { - "backbone_out": tensor, - "neck_out": tensor, - "head_out": tensor, - } -} -``` - - -#### 2.1.2 损失函数 - -知识蒸馏任务中,损失函数配置如下所示。 - -```yaml -Loss: - name: CombinedLoss - loss_config_list: - - DistillationDMLLoss: # 蒸馏的DML损失函数,继承自标准的DMLLoss - weight: 1.0 # 权重 - act: "softmax" # 激活函数,对输入使用激活函数处理,可以为softmax, sigmoid或者为None,默认为None - use_log: true # 对输入计算log,如果函数已经 - model_name_pairs: # 用于计算DML loss的子网络名称对,如果希望计算其他子网络的DML loss,可以在列表下面继续填充 - - ["Student", "Teacher"] - key: head_out # 取子网络输出dict中,该key对应的tensor - multi_head: True # 是否为多头结构 - dis_head: ctc # 指定用于计算损失函数的head - name: dml_ctc # 蒸馏loss的前缀名称,避免不同loss之间的命名冲突 - - DistillationDMLLoss: # 蒸馏的DML损失函数,继承自标准的DMLLoss - weight: 0.5 # 权重 - act: "softmax" # 激活函数,对输入使用激活函数处理,可以为softmax, sigmoid或者为None,默认为None - use_log: true # 对输入计算log,如果函数已经 - model_name_pairs: # 用于计算DML loss的子网络名称对,如果希望计算其他子网络的DML loss,可以在列表下面继续填充 - - ["Student", "Teacher"] - key: head_out # 取子网络输出dict中,该key对应的tensor - multi_head: True # 是否为多头结构 - dis_head: sar # 指定用于计算损失函数的head - name: dml_sar # 蒸馏loss的前缀名称,避免不同loss之间的命名冲突 - - DistillationDistanceLoss: # 蒸馏的距离损失函数 - weight: 1.0 # 权重 - mode: "l2" # 距离计算方法,目前支持l1, l2, smooth_l1 - model_name_pairs: # 用于计算distance loss的子网络名称对 - - ["Student", "Teacher"] - key: backbone_out # 取子网络输出dict中,该key对应的tensor - - DistillationCTCLoss: # 基于蒸馏的CTC损失函数,继承自标准的CTC loss - weight: 1.0 # 损失函数的权重,loss_config_list中,每个损失函数的配置都必须包含该字段 - model_name_list: ["Student", "Teacher"] # 对于蒸馏模型的预测结果,提取这两个子网络的输出,与gt计算CTC loss - key: head_out # 取子网络输出dict中,该key对应的tensor - - DistillationSARLoss: # 基于蒸馏的SAR损失函数,继承自标准的SARLoss - weight: 1.0 # 损失函数的权重,loss_config_list中,每个损失函数的配置都必须包含该字段 - model_name_list: ["Student", "Teacher"] # 对于蒸馏模型的预测结果,提取这两个子网络的输出,与gt计算CTC loss - key: head_out # 取子网络输出dict中,该key对应的tensor - multi_head: True # 是否为多头结构,为true时,取出其中的SAR分支计算损失函数 -``` - -上述损失函数中,所有的蒸馏损失函数均继承自标准的损失函数类,主要功能为: 对蒸馏模型的输出进行解析,找到用于计算损失的中间节点(tensor),再使用标准的损失函数类去计算。 - -以上述配置为例,最终蒸馏训练的损失函数包含下面5个部分。 - -- `Student`和`Teacher`最终输出(`head_out`)的CTC分支与gt的CTC loss,权重为1。在这里因为2个子网络都需要更新参数,因此2者都需要计算与g的loss。 -- `Student`和`Teacher`最终输出(`head_out`)的SAR分支与gt的SAR loss,权重为1.0。在这里因为2个子网络都需要更新参数,因此2者都需要计算与g的loss。 -- `Student`和`Teacher`最终输出(`head_out`)的CTC分支之间的DML loss,权重为1。 -- `Student`和`Teacher`最终输出(`head_out`)的SAR分支之间的DML loss,权重为0.5。 -- `Student`和`Teacher`的骨干网络输出(`backbone_out`)之间的l2 loss,权重为1。 - - -关于`CombinedLoss`更加具体的实现可以参考: [combined_loss.py](../../ppocr/losses/combined_loss.py#L23)。关于`DistillationCTCLoss`等蒸馏损失函数更加具体的实现可以参考[distillation_loss.py](../../ppocr/losses/distillation_loss.py)。 - - -#### 2.1.3 后处理 - -知识蒸馏任务中,后处理配置如下所示。 - -```yaml -PostProcess: - name: DistillationCTCLabelDecode # 蒸馏任务的CTC解码后处理,继承自标准的CTCLabelDecode类 - model_name: ["Student", "Teacher"] # 对于蒸馏模型的预测结果,提取这两个子网络的输出,进行解码 - key: head_out # 取子网络输出dict中,该key对应的tensor - multi_head: True # 多头结构时,会取出其中的CTC分支进行计算 -``` - -以上述配置为例,最终会同时计算`Student`和`Teahcer` 2个子网络的CTC解码输出,返回一个`dict`,`key`为用于处理的子网络名称,`value`为用于处理的子网络列表。 - -关于`DistillationCTCLabelDecode`更加具体的实现可以参考: [rec_postprocess.py](../../ppocr/postprocess/rec_postprocess.py#L128) - - -#### 2.1.4 指标计算 - -知识蒸馏任务中,指标计算配置如下所示。 - -```yaml -Metric: - name: DistillationMetric # 蒸馏任务的CTC解码后处理,继承自标准的CTCLabelDecode类 - base_metric_name: RecMetric # 指标计算的基类,对于模型的输出,会基于该类,计算指标 - main_indicator: acc # 指标的名称 - key: "Student" # 选取该子网络的 main_indicator 作为作为保存保存best model的判断标准 - ignore_space: False # 评估时是否忽略空格的影响 -``` - -以上述配置为例,最终会使用`Student`子网络的acc指标作为保存best model的判断指标,同时,日志中也会打印出所有子网络的acc指标。 - -关于`DistillationMetric`更加具体的实现可以参考: [distillation_metric.py](../../ppocr/metrics/distillation_metric.py#L24)。 - - -#### 2.1.5 蒸馏模型微调 - -对蒸馏得到的识别蒸馏进行微调有2种方式。 - -(1)基于知识蒸馏的微调:这种情况比较简单,下载预训练模型,在[ch_PP-OCRv3_rec_distillation.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml)中配置好预训练模型路径以及自己的数据路径,即可进行模型微调训练。 - -(2)微调时不使用知识蒸馏:这种情况,需要首先将预训练模型中的学生模型参数提取出来,具体步骤如下。 - -* 首先下载预训练模型并解压。 -```shell -# 下面预训练模型并解压 -wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_train.tar -tar -xf ch_PP-OCRv3_rec_train.tar -``` - -* 然后使用python,对其中的学生模型参数进行提取 - -```python -import paddle -# 加载预训练模型 -all_params = paddle.load("ch_PP-OCRv3_rec_train/best_accuracy.pdparams") -# 查看权重参数的keys -print(all_params.keys()) -# 学生模型的权重提取 -s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key} -# 查看学生模型权重参数的keys -print(s_params.keys()) -# 保存 -paddle.save(s_params, "ch_PP-OCRv3_rec_train/student.pdparams") -``` - -转化完成之后,使用[ch_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec.yml),修改预训练模型的路径(为导出的`student.pdparams`模型路径)以及自己的数据路径,即可进行模型微调。 - - -### 2.2 检测配置文件解析 - -检测模型蒸馏的配置文件在PaddleOCR/configs/det/ch_PP-OCRv3/目录下,包含两个个蒸馏配置文件: -- ch_PP-OCRv3_det_cml.yml,采用cml蒸馏,采用一个大模型蒸馏两个小模型,且两个小模型互相学习的方法 -- ch_PP-OCRv3_det_dml.yml,采用DML的蒸馏,两个Student模型互蒸馏的方法 - - -#### 2.2.1 模型结构 - -知识蒸馏任务中,模型结构配置如下所示: - -``` -Architecture: - name: DistillationModel # 结构名称,蒸馏任务中,为DistillationModel,用于构建对应的结构 - algorithm: Distillation # 算法名称 - Models: # 模型,包含子网络的配置信息 - Student: # 子网络名称,至少需要包含`pretrained`与`freeze_params`信息,其他的参数为子网络的构造参数 - freeze_params: false # 是否需要固定参数 - return_all_feats: false # 子网络的参数,表示是否需要返回所有的features,如果为False,则只返回最后的输出 - model_type: det - algorithm: DB - Backbone: - name: ResNet - in_channels: 3 - layers: 50 - Neck: - name: LKPAN - out_channels: 256 - Head: - name: DBHead - kernel_list: [7,2,2] - k: 50 - Teacher: # 另外一个子网络,这里给的是DML蒸馏示例, - freeze_params: true - return_all_feats: false - model_type: det - algorithm: DB - Transform: - Backbone: - name: ResNet - in_channels: 3 - layers: 50 - Neck: - name: LKPAN - out_channels: 256 - Head: - name: DBHead - kernel_list: [7,2,2] - k: 50 - -``` - -如果是采用DML,即两个小模型互相学习的方法,上述配置文件里的Teacher网络结构需要设置为Student模型一样的配置,具体参考配置文件[ch_PP-OCRv3_det_dml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_dml.yml)。 - -下面介绍[ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml)的配置文件参数: - -``` -Architecture: - name: DistillationModel - algorithm: Distillation - model_type: det - Models: - Teacher: # CML蒸馏的Teacher模型配置 - pretrained: ./pretrain_models/ch_ppocr_server_v2.0_det_train/best_accuracy - freeze_params: true # Teacher 不训练 - return_all_feats: false - model_type: det - algorithm: DB - Transform: - Backbone: - name: ResNet - in_channels: 3 - layers: 50 - Neck: - name: LKPAN - out_channels: 256 - Head: - name: DBHead - kernel_list: [7,2,2] - k: 50 - Student: # CML蒸馏的Student模型配置 - pretrained: ./pretrain_models/MobileNetV3_large_x0_5_pretrained - freeze_params: false - return_all_feats: false - model_type: det - algorithm: DB - Backbone: - name: MobileNetV3 - scale: 0.5 - model_name: large - disable_se: true - Neck: - name: RSEFPN - out_channels: 96 - shortcut: True - Head: - name: DBHead - k: 50 - Student2: # CML蒸馏的Student2模型配置 - pretrained: ./pretrain_models/MobileNetV3_large_x0_5_pretrained - freeze_params: false - return_all_feats: false - model_type: det - algorithm: DB - Transform: - Backbone: - name: MobileNetV3 - scale: 0.5 - model_name: large - disable_se: true - Neck: - name: RSEFPN - out_channels: 96 - shortcut: True - Head: - name: DBHead - k: 50 - -``` - - -蒸馏模型`DistillationModel`类的具体实现代码可以参考[distillation_model.py](../../ppocr/modeling/architectures/distillation_model.py)。 - -最终模型`forward`输出为一个字典,key为所有的子网络名称,例如这里为`Student`与`Teacher`,value为对应子网络的输出,可以为`Tensor`(只返回该网络的最后一层)和`dict`(也返回了中间的特征信息)。 - -在蒸馏任务中,为了方便添加蒸馏损失函数,每个网络的输出保存为`dict`,其中包含子模块输出。每个子网络的输出结果均为`dict`,key包含`backbone_out`,`neck_out`, `head_out`,`value`为对应模块的tensor,最终对于上述配置文件,`DistillationModel`的输出格式如下。 - -```json -{ - "Teacher": { - "backbone_out": tensor, - "neck_out": tensor, - "head_out": tensor, - }, - "Student": { - "backbone_out": tensor, - "neck_out": tensor, - "head_out": tensor, - } -} -``` - - -#### 2.2.2 损失函数 - -检测ch_PP-OCRv3_det_cml.yml蒸馏损失函数配置如下所示。 -```yaml -Loss: - name: CombinedLoss - loss_config_list: - - DistillationDilaDBLoss: - weight: 1.0 - model_name_pairs: - - ["Student", "Teacher"] - - ["Student2", "Teacher"] # 改动1,计算两个Student和Teacher的损失 - key: maps - balance_loss: true - main_loss_type: DiceLoss - alpha: 5 - beta: 10 - ohem_ratio: 3 - - DistillationDMLLoss: # 改动2,增加计算两个Student之间的损失 - model_name_pairs: - - ["Student", "Student2"] - maps_name: "thrink_maps" - weight: 1.0 - # act: None - key: maps - - DistillationDBLoss: - weight: 1.0 - model_name_list: ["Student", "Student2"] # 改动3,计算两个Student和GT之间的损失 - balance_loss: true - main_loss_type: DiceLoss - alpha: 5 - beta: 10 - ohem_ratio: 3 - -``` - -关于`DistillationDilaDBLoss`更加具体的实现可以参考: [distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.4/ppocr/losses/distillation_loss.py#L185)。关于`DistillationDBLoss`等蒸馏损失函数更加具体的实现可以参考[distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/04c44974b13163450dfb6bd2c327863f8a194b3c/ppocr/losses/distillation_loss.py?_pjax=%23js-repo-pjax-container%2C%20div%5Bitemtype%3D%22http%3A%2F%2Fschema.org%2FSoftwareSourceCode%22%5D%20main%2C%20%5Bdata-pjax-container%5D#L148)。 - - -#### 2.2.3 后处理 - -知识蒸馏任务中,检测蒸馏后处理配置如下所示。 - -```yaml -PostProcess: - name: DistillationDBPostProcess # DB检测蒸馏任务的CTC解码后处理,继承自标准的DBPostProcess类 - model_name: ["Student", "Student2", "Teacher"] # 对于蒸馏模型的预测结果,提取多个子网络的输出,进行解码,不需要后处理的网络可以不在model_name中设置 - thresh: 0.3 - box_thresh: 0.6 - max_candidates: 1000 - unclip_ratio: 1.5 -``` - -以上述配置为例,最终会同时计算`Student`,`Student2`和`Teacher` 3个子网络的输出做后处理计算。同时,由于有多个输入,后处理返回的输出也有多个, - -关于`DistillationDBPostProcess`更加具体的实现可以参考: [db_postprocess.py](../../ppocr/postprocess/db_postprocess.py#L195) - - -#### 2.2.4 蒸馏指标计算 - -知识蒸馏任务中,检测蒸馏指标计算配置如下所示。 - -```yaml -Metric: - name: DistillationMetric - base_metric_name: DetMetric - main_indicator: hmean - key: "Student" -``` - -由于蒸馏需要包含多个网络,甚至多个Student网络,在计算指标的时候只需要计算一个Student网络的指标即可,`key`字段设置为`Student`则表示只计算`Student`网络的精度。 - - -#### 2.2.5 检测蒸馏模型finetune - -PP-OCRv3检测蒸馏有两种方式: -- 采用ch_PP-OCRv3_det_cml.yml,采用cml蒸馏,同样Teacher模型设置为PaddleOCR提供的模型或者您训练好的大模型 -- 采用ch_PP-OCRv3_det_dml.yml,采用DML的蒸馏,两个Student模型互蒸馏的方法,在PaddleOCR采用的数据集上相比单独训练Student模型有1%-2%的提升。 - -在具体fine-tune时,需要在网络结构的`pretrained`参数中设置要加载的预训练模型。 - -在精度提升方面,cml的精度>dml的精度蒸馏方法的精度。当数据量不足或者Teacher模型精度与Student精度相差不大的时候,这个结论或许会改变。 - - -另外,由于PaddleOCR提供的蒸馏预训练模型包含了多个模型的参数,如果您希望提取Student模型的参数,可以参考如下代码: -``` -# 下载蒸馏训练模型的参数 -wget https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv3_det_distill_train.tar -``` - -```python -import paddle -# 加载预训练模型 -all_params = paddle.load("ch_PP-OCRv3_det_distill_train/best_accuracy.pdparams") -# 查看权重参数的keys -print(all_params.keys()) -# 学生模型的权重提取 -s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key} -# 查看学生模型权重参数的keys -print(s_params.keys()) -# 保存 -paddle.save(s_params, "ch_PP-OCRv3_det_distill_train/student.pdparams") -``` - -最终`Student`模型的参数将会保存在`ch_PP-OCRv3_det_distill_train/student.pdparams`中,用于模型的fine-tune。 diff --git a/doc/doc_ch/models.md b/doc/doc_ch/models.md deleted file mode 100644 index ffe646c167..0000000000 --- a/doc/doc_ch/models.md +++ /dev/null @@ -1,46 +0,0 @@ - -# PP-OCR模型库 -PP-OCR模型一节主要补充一些OCR模型的基本概念以及如何快速运用PP-OCR模型库中的模型。 - -本节包含两个部分,首先在[PP-OCR模型下载](./models_list.md)中解释PP-OCR模型的类型概念,并提供所有模型的下载链接。然后在[基于Python引擎的PP-OCR模型库推理](./inference_ppocr.md)中介绍PP-OCR模型库的使用方法,可以通过Python推理引擎快速利用丰富的模型库模型获得测试结果。 - ------- - -下面我们首先了解一些OCR相关的基本概念: - -- [1. OCR 简要介绍](#1-ocr-----) - * [1.1 OCR 检测模型基本概念](#11-ocr---------) - * [1.2 OCR 识别模型基本概念](#12-ocr---------) - * [1.3 PP-OCR模型](#13-pp-ocr--) - - -## 1. OCR 简要介绍 -本节简要介绍OCR检测模型、识别模型的基本概念,并介绍PaddleOCR的PP-OCR模型。 - -OCR(Optical Character Recognition,光学字符识别)目前是文字识别的统称,已不限于文档或书本文字识别,更包括识别自然场景下的文字,又可以称为STR(Scene Text Recognition)。 - -OCR文字识别一般包括两个部分,文本检测和文本识别;文本检测首先利用检测算法检测到图像中的文本行;然后检测到的文本行用识别算法去识别到具体文字。 - - -### 1.1 OCR 检测模型基本概念 - -文本检测就是要定位图像中的文字区域,然后通常以边界框的形式将单词或文本行标记出来。传统的文字检测算法多是通过手工提取特征的方式,特点是速度快,简单场景效果好,但是面对自然场景,效果会大打折扣。当前多是采用深度学习方法来做。 - -基于深度学习的文本检测算法可以大致分为以下几类: -1. 基于目标检测的方法;一般是预测得到文本框后,通过NMS筛选得到最终文本框,多是四点文本框,对弯曲文本场景效果不理想。典型算法为EAST、Text Box等方法。 -2. 基于分割的方法;将文本行当成分割目标,然后通过分割结果构建外接文本框,可以处理弯曲文本,对于文本交叉场景问题效果不理想。典型算法为DB、PSENet等方法。 -3. 混合目标检测和分割的方法; - - -### 1.2 OCR 识别模型基本概念 - -OCR识别算法的输入数据一般是文本行,背景信息不多,文字占据主要部分,识别算法目前可以分为两类算法: -1. 基于CTC的方法;即识别算法的文字预测模块是基于CTC的,常用的算法组合为CNN+RNN+CTC。目前也有一些算法尝试在网络中加入transformer模块等等。 -2. 基于Attention的方法;即识别算法的文字预测模块是基于Attention的,常用算法组合是CNN+RNN+Attention。 - - -### 1.3 PP-OCR模型 - -PaddleOCR 中集成了很多OCR算法,文本检测算法有DB、EAST、SAST等等,文本识别算法有CRNN、RARE、StarNet、Rosetta、SRN等算法。 - -其中PaddleOCR针对中英文自然场景通用OCR,推出了PP-OCR系列模型,PP-OCR模型由DB+CRNN算法组成,利用海量中文数据训练加上模型调优方法,在中文场景上具备较高的文本检测识别能力。并且PaddleOCR推出了高精度超轻量PP-OCRv2模型,检测模型仅3M,识别模型仅8.5M,利用[PaddleSlim](https://github.com/PaddlePaddle/PaddleSlim)的模型量化方法,可以在保持精度不降低的情况下,将检测模型压缩到0.8M,识别压缩到3M,更加适用于移动端部署场景。 diff --git a/doc/doc_ch/models_list.md b/doc/doc_ch/models_list.md deleted file mode 100644 index 730dcb5f56..0000000000 --- a/doc/doc_ch/models_list.md +++ /dev/null @@ -1,155 +0,0 @@ -# PP-OCR系列模型列表(V4,2023年8月1日更新) - -> **说明** -> 1. V4版模型相比V3版模型,在模型精度上有进一步提升 -> 2. V3版模型相比V2版模型,在模型精度上有进一步提升 -> 3. 2.0+版模型和[1.1版模型](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/doc/doc_ch/models_list.md) 的主要区别在于动态图训练vs.静态图训练,模型性能上无明显差距。 -> 4. 本文档提供的是PPOCR自研模型列表,更多基于公开数据集的算法介绍与预训练模型可以参考:[算法概览文档](./algorithm_overview.md)。 - - -- PP-OCR系列模型列表(V4,2023年8月1日更新) - - [1. 文本检测模型](#1-文本检测模型) - - [1.1 中文检测模型](#1.1) - - [2.2 英文检测模型](#1.2) - - [1.3 多语言检测模型](#1.3) - - [2. 文本识别模型](#2-文本识别模型) - - [2.1 中文识别模型](#21-中文识别模型) - - [2.2 英文识别模型](#22-英文识别模型) - - [2.3 多语言识别模型(更多语言持续更新中...)](#23-多语言识别模型更多语言持续更新中) - - [3. 文本方向分类模型](#3-文本方向分类模型) - - [4. Paddle-Lite 模型](#4-paddle-lite-模型) - -PaddleOCR提供的可下载模型包括`推理模型`、`训练模型`、`预训练模型`、`nb模型`,模型区别说明如下: - -|模型类型|模型格式|简介| -|--- | --- | --- | -|推理模型|inference.pdmodel、inference.pdiparams|用于预测引擎推理,[详情](./inference_ppocr.md)| -|训练模型、预训练模型|\*.pdparams、\*.pdopt、\*.states |训练过程中保存的模型的参数、优化器状态和训练中间信息,多用于模型指标评估和恢复训练| -|nb模型|\*.nb|经过飞桨Paddle-Lite工具优化后的模型,适用于移动端/IoT端等端侧部署场景(需使用飞桨Paddle Lite部署)。| - - -各个模型的关系如下面的示意图所示。 - -![](../imgs/model_prod_flow_ch.png) - - - -## 1. 文本检测模型 - - - -### 1.1 中文检测模型 - -|模型名称|模型简介|配置文件|推理模型大小|下载地址| -| --- | --- | --- | --- | --- | -|ch_PP-OCRv4_det| 【最新】原始超轻量模型,支持中英文、多语种文本检测 |[ch_PP-OCRv4_det_cml.yml](../../configs/det/ch_PP-OCRv4/ch_PP-OCRv4_det_cml.yml)| 4.70M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_det_train.tar)| -|ch_PP-OCRv4_server_det| 【最新】原始高精度模型,支持中英文、多语种文本检测 |[ch_PP-OCRv4_det_teacher.yml](../../configs/det/ch_PP-OCRv4/ch_PP-OCRv4_det_teacher.yml)| 110M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_det_server_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_det_server_train.tar)| -|ch_PP-OCRv3_det_slim|slim量化+蒸馏版超轻量模型,支持中英文、多语种文本检测|[ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml)| 1.1M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_slim_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_slim_distill_train.tar) / [nb模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_slim_infer.nb)| -|ch_PP-OCRv3_det| 原始超轻量模型,支持中英文、多语种文本检测 |[ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml)| 3.80M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar)| -|ch_PP-OCRv2_det_slim| slim量化+蒸馏版超轻量模型,支持中英文、多语种文本检测|[ch_PP-OCRv2_det_cml.yml](../../configs/det/ch_PP-OCRv2/ch_PP-OCRv2_det_cml.yml)| 3.0M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_slim_quant_infer.tar)| -|ch_PP-OCRv2_det| 原始超轻量模型,支持中英文、多语种文本检测|[ch_PP-OCRv2_det_cml.yml](../../configs/det/ch_PP-OCRv2/ch_PP-OCRv2_det_cml.yml)|3.0M|[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_distill_train.tar)| -|ch_ppocr_mobile_slim_v2.0_det|slim裁剪版超轻量模型,支持中英文、多语种文本检测|[ch_det_mv3_db_v2.0.yml](../../configs/det/ch_ppocr_v2.0/ch_det_mv3_db_v2.0.yml)| 2.60M |[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/slim/ch_ppocr_mobile_v2.0_det_prune_infer.tar)| -|ch_ppocr_mobile_v2.0_det|原始超轻量模型,支持中英文、多语种文本检测|[ch_det_mv3_db_v2.0.yml](../../configs/det/ch_ppocr_v2.0/ch_det_mv3_db_v2.0.yml)|3.0M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_train.tar)| -|ch_ppocr_server_v2.0_det|通用模型,支持中英文、多语种文本检测,比超轻量模型更大,但效果更好|[ch_det_res18_db_v2.0.yml](../../configs/det/ch_ppocr_v2.0/ch_det_res18_db_v2.0.yml)|47.0M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_det_train.tar)| - - - -### 1.2 英文检测模型 - -|模型名称|模型简介|配置文件|推理模型大小|下载地址| -| --- | --- | --- | --- | --- | -|en_PP-OCRv3_det_slim |【最新】slim量化版超轻量模型,支持英文、数字检测 | [ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml) | 1.1M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_slim_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_slim_distill_train.tar) / [nb模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_slim_infer.nb) | -|en_PP-OCRv3_det |【最新】原始超轻量模型,支持英文、数字检测|[ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml)| 3.8M | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_distill_train.tar) | - -* 注:英文检测模型与中文检测模型结构完全相同,只有训练数据不同,在此仅提供相同的配置文件。 - - - -### 1.3 多语言检测模型 - -|模型名称|模型简介|配置文件|推理模型大小|下载地址| -| --- | --- | --- | --- | --- | -| ml_PP-OCRv3_det_slim |【最新】slim量化版超轻量模型,支持多语言检测 | [ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml) | 1.1M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/Multilingual_PP-OCRv3_det_slim_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/Multilingual_PP-OCRv3_det_slim_distill_train.tar) / [nb模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/Multilingual_PP-OCRv3_det_slim_infer.nb) | -| ml_PP-OCRv3_det |【最新】原始超轻量模型,支持多语言检测 | [ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml)| 3.8M | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/Multilingual_PP-OCRv3_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/Multilingual_PP-OCRv3_det_distill_train.tar) | - -* 注:多语言检测模型与中文检测模型结构完全相同,只有训练数据不同,在此仅提供相同的配置文件。 - - - -## 2. 文本识别模型 - - - -### 2.1 中文识别模型 - -|模型名称|模型简介|配置文件|推理模型大小|下载地址| -| --- | --- | --- | --- | --- | -|ch_PP-OCRv4_rec|【最新】超轻量模型,支持中英文、数字识别|[ch_PP-OCRv4_rec_distill.yml](../../configs/rec/PP-OCRv4/ch_PP-OCRv4_rec_distill.yml)| 10M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_rec_train.tar) | -|ch_PP-OCRv4_server_rec|【最新】高精度模型,支持中英文、数字识别|[ch_PP-OCRv4_rec_hgnet.yml](../../configs/rec/PP-OCRv4/ch_PP-OCRv4_rec_hgnet.yml)| 88M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_rec_server_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_rec_server_train.tar) | -|ch_PP-OCRv3_rec_slim |slim量化版超轻量模型,支持中英文、数字识别|[ch_PP-OCRv3_rec_distillation.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml)| 4.9M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_slim_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_slim_train.tar) / [nb模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_slim_infer.nb) | -|ch_PP-OCRv3_rec|原始超轻量模型,支持中英文、数字识别|[ch_PP-OCRv3_rec_distillation.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml)| 12.4M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_train.tar) | -|ch_PP-OCRv2_rec_slim| slim量化版超轻量模型,支持中英文、数字识别|[ch_PP-OCRv2_rec.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml)| 9.0M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_slim_quant_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_slim_quant_train.tar) | -|ch_PP-OCRv2_rec| 原始超轻量模型,支持中英文、数字识别|[ch_PP-OCRv2_rec_distillation.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec_distillation.yml)|8.50M|[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_train.tar) | -|ch_ppocr_mobile_slim_v2.0_rec|slim裁剪量化版超轻量模型,支持中英文、数字识别|[rec_chinese_lite_train_v2.0.yml](../../configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml)| 6.0M |[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_slim_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_slim_train.tar) | -|ch_ppocr_mobile_v2.0_rec|原始超轻量模型,支持中英文、数字识别|[rec_chinese_lite_train_v2.0.yml](../../configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml)|5.20M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_train.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_pre.tar) | -|ch_ppocr_server_v2.0_rec|通用模型,支持中英文、数字识别|[rec_chinese_common_train_v2.0.yml](../../configs/rec/ch_ppocr_v2.0/rec_chinese_common_train_v2.0.yml)|94.8M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_rec_train.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_rec_pre.tar) | - -**说明:** `训练模型`是基于预训练模型在真实数据与竖排合成文本数据上finetune得到的模型,在真实应用场景中有着更好的表现,`预训练模型`则是直接基于全量真实数据与合成数据训练得到,更适合用于在自己的数据集上finetune。 - - -### 2.2 英文识别模型 - -|模型名称|模型简介|配置文件|推理模型大小|下载地址| -| --- | --- | --- | --- | --- | -|en_PP-OCRv4_rec |【最新】原始超轻量模型,支持英文、数字识别|[en_PP-OCRv4_rec.yml](../../configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml)| 9.7M | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/english/en_PP-OCRv4_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/english/en_PP-OCRv4_rec_train.tar) | -|en_PP-OCRv3_rec_slim |slim量化版超轻量模型,支持英文、数字识别 | [en_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/en_PP-OCRv3_rec.yml)| 3.2M |[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_slim_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_slim_train.tar) / [nb模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_slim_infer.nb) | -|en_PP-OCRv3_rec |原始超轻量模型,支持英文、数字识别|[en_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/en_PP-OCRv3_rec.yml)| 9.6M | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_train.tar) | -|en_number_mobile_slim_v2.0_rec|slim裁剪量化版超轻量模型,支持英文、数字识别|[rec_en_number_lite_train.yml](../../configs/rec/multi_language/rec_en_number_lite_train.yml)| 2.7M | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/en_number_mobile_v2.0_rec_slim_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/en_number_mobile_v2.0_rec_slim_train.tar) | -|en_number_mobile_v2.0_rec|原始超轻量模型,支持英文、数字识别|[rec_en_number_lite_train.yml](../../configs/rec/multi_language/rec_en_number_lite_train.yml)|2.6M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/multilingual/en_number_mobile_v2.0_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/multilingual/en_number_mobile_v2.0_rec_train.tar) | - -**注意:** 所有英文识别模型的字典文件均为`ppocr/utils/en_dict.txt` - - -### 2.3 多语言识别模型(更多语言持续更新中...) - -|模型名称|字典文件|模型简介|配置文件|推理模型大小|下载地址| -| --- | --- | --- | --- |--- | --- | -| korean_PP-OCRv3_rec | ppocr/utils/dict/korean_dict.txt |韩文识别|[korean_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/korean_PP-OCRv3_rec.yml)|11.0M|[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/korean_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/korean_PP-OCRv3_rec_train.tar) | -| japan_PP-OCRv3_rec | ppocr/utils/dict/japan_dict.txt |日文识别|[japan_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/japan_PP-OCRv3_rec.yml)|11.0M|[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/japan_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/japan_PP-OCRv3_rec_train.tar) | -| chinese_cht_PP-OCRv3_rec | ppocr/utils/dict/chinese_cht_dict.txt | 中文繁体识别|[chinese_cht_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/chinese_cht_PP-OCRv3_rec.yml)|12.0M|[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/chinese_cht_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/chinese_cht_PP-OCRv3_rec_train.tar) | -| te_PP-OCRv3_rec | ppocr/utils/dict/te_dict.txt | 泰卢固文识别|[te_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/te_PP-OCRv3_rec.yml)|9.6M|[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/te_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/te_PP-OCRv3_rec_train.tar) | -| ka_PP-OCRv3_rec | ppocr/utils/dict/ka_dict.txt |卡纳达文识别|[ka_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/ka_PP-OCRv3_rec.yml)|9.9M|[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/ka_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/ka_PP-OCRv3_rec_train.tar) | -| ta_PP-OCRv3_rec | ppocr/utils/dict/ta_dict.txt |泰米尔文识别|[ta_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/ta_PP-OCRv3_rec.yml)|9.6M|[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/ta_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/ta_PP-OCRv3_rec_train.tar) | -| latin_PP-OCRv3_rec | ppocr/utils/dict/latin_dict.txt | 拉丁文识别 | [latin_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/latin_PP-OCRv3_rec.yml) |9.7M|[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/latin_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/latin_PP-OCRv3_rec_train.tar) | -| arabic_PP-OCRv3_rec | ppocr/utils/dict/arabic_dict.txt | 阿拉伯字母 | [arabic_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/arabic_PP-OCRv3_rec.yml) |9.6M|[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/arabic_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/arabic_PP-OCRv3_rec_train.tar) | -| cyrillic_PP-OCRv3_rec | ppocr/utils/dict/cyrillic_dict.txt | 斯拉夫字母 | [cyrillic_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/cyrillic_PP-OCRv3_rec.yml) |9.6M|[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/cyrillic_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/cyrillic_PP-OCRv3_rec_train.tar) | -| devanagari_PP-OCRv3_rec | ppocr/utils/dict/devanagari_dict.txt |梵文字母 | [devanagari_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/devanagari_PP-OCRv3_rec.yml) |9.9M|[推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/devanagari_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/devanagari_PP-OCRv3_rec_train.tar) | - -查看完整语种列表与使用教程请参考: [多语言模型](./multi_languages.md) - - - -## 3. 文本方向分类模型 - -|模型名称|模型简介|配置文件|推理模型大小|下载地址| -| --- | --- | --- | --- | --- | -|ch_ppocr_mobile_slim_v2.0_cls|slim量化版模型,对检测到的文本行文字角度分类|[cls_mv3.yml](../../configs/cls/cls_mv3.yml)| 2.1M |[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_slim_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_slim_infer.tar) / [nb模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_ppocr_mobile_v2.0_cls_infer_opt.nb) | -|ch_ppocr_mobile_v2.0_cls|原始分类器模型,对检测到的文本行文字角度分类|[cls_mv3.yml](../../configs/cls/cls_mv3.yml)|1.38M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | - - - -## 4. Paddle-Lite 模型 - -Paddle-Lite 是一个高性能、轻量级、灵活性强且易于扩展的深度学习推理框架,它可以对inference模型进一步优化,得到适用于移动端/IoT端等端侧部署场景的`nb模型`。一般建议基于量化模型进行转换,因为可以将模型以INT8形式进行存储与推理,从而进一步减小模型大小,提升模型速度。 - -本节主要列出PP-OCRv2以及更早版本的检测与识别nb模型,最新版本的nb模型可以直接从上面的模型列表中获得。 - - -|模型版本|模型简介|模型大小|检测模型|文本方向分类模型|识别模型|Paddle-Lite版本| -|---|---|---|---|---|---|---| -|PP-OCRv2|蒸馏版超轻量中文OCR移动端模型|11.0M|[下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_PP-OCRv2_det_infer_opt.nb)|[下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_ppocr_mobile_v2.0_cls_infer_opt.nb)|[下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_PP-OCRv2_rec_infer_opt.nb)|v2.10| -|PP-OCRv2(slim)|蒸馏版超轻量中文OCR移动端模型|4.6M|[下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_PP-OCRv2_det_slim_opt.nb)|[下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_ppocr_mobile_v2.0_cls_slim_opt.nb)|[下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_PP-OCRv2_rec_slim_opt.nb)|v2.10| -|PP-OCRv2|蒸馏版超轻量中文OCR移动端模型|11.0M|[下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_infer_opt.nb)|[下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_cls_opt.nb)|[下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_infer_opt.nb)|v2.9| -|PP-OCRv2(slim)|蒸馏版超轻量中文OCR移动端模型|4.9M|[下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_slim_opt.nb)|[下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_cls_slim_opt.nb)|[下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_slim_opt.nb)|v2.9| -|V2.0|ppocr_v2.0超轻量中文OCR移动端模型|7.8M|[下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_det_opt.nb)|[下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_cls_opt.nb)|[下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_rec_opt.nb)|v2.9| -|V2.0(slim)|ppocr_v2.0超轻量中文OCR移动端模型|3.3M|[下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_det_slim_opt.nb)|[下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_cls_slim_opt.nb)|[下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_rec_slim_opt.nb)|v2.9| diff --git a/doc/doc_ch/multi_languages.md b/doc/doc_ch/multi_languages.md deleted file mode 100644 index 4a6539fe8e..0000000000 --- a/doc/doc_ch/multi_languages.md +++ /dev/null @@ -1,292 +0,0 @@ -# 多语言模型 - -**近期更新** - -- 2022.5.8 更新`PP-OCRv3`版 多语言检测和识别模型,平均识别准确率提升5%以上。 -- 2021.4.9 支持**80种**语言的检测和识别 -- 2021.4.9 支持**轻量高精度**英文模型检测识别 - -PaddleOCR 旨在打造一套丰富、领先、且实用的OCR工具库,不仅提供了通用场景下的中英文模型,也提供了专门在英文场景下训练的模型, -和覆盖[80个语言](#语种缩写)的小语种模型。 - -其中英文模型支持,大小写字母和常见标点的检测识别,并优化了空格字符的识别: - -
- -
- -小语种模型覆盖了拉丁语系、阿拉伯语系、中文繁体、韩语、日语等等: - -
- - - - -
- - -本文档将简要介绍小语种模型的使用方法。 - -- [1 安装](#安装) - - [1.1 paddle 安装](#paddle安装) - - [1.2 paddleocr package 安装](#paddleocr_package_安装) - -- [2 快速使用](#快速使用) - - [2.1 命令行运行](#命令行运行) - - [2.2 python 脚本运行](#python_脚本运行) -- [3 自定义训练](#自定义训练) -- [4 预测部署](#预测部署) -- [4 支持语种及缩写](#语种缩写) - - -## 1 安装 - - -### 1.1 paddle 安装 -``` -# cpu -pip install paddlepaddle - -# gpu -pip install paddlepaddle-gpu -``` - - -### 1.2 paddleocr package 安装 - - -pip 安装 -``` -pip install paddleocr -``` - -本地构建并安装 -``` -python3 -m build -pip3 install dist/paddleocr-x.x.x-py3-none-any.whl # x.x.x是paddleocr的版本号 -``` - - -## 2 快速使用 - - -### 2.1 命令行运行 - -查看帮助信息 - -``` -paddleocr -h -``` - -* 整图预测(检测+识别) - -Paddleocr目前支持80个语种,可以通过修改--lang参数进行切换,具体支持的[语种](#语种缩写)可查看表格。 - -``` bash - -paddleocr --image_dir doc/imgs_en/254.jpg --lang=en -``` - -
- - -
- - -结果是一个list,每个item包含了文本框,文字和识别置信度 -```text -[('PHO CAPITAL', 0.95723116), [[66.0, 50.0], [327.0, 44.0], [327.0, 76.0], [67.0, 82.0]]] -[('107 State Street', 0.96311164), [[72.0, 90.0], [451.0, 84.0], [452.0, 116.0], [73.0, 121.0]]] -[('Montpelier Vermont', 0.97389287), [[69.0, 132.0], [501.0, 126.0], [501.0, 158.0], [70.0, 164.0]]] -[('8022256183', 0.99810505), [[71.0, 175.0], [363.0, 170.0], [364.0, 202.0], [72.0, 207.0]]] -[('REG 07-24-201706:59 PM', 0.93537045), [[73.0, 299.0], [653.0, 281.0], [654.0, 318.0], [74.0, 336.0]]] -[('045555', 0.99346405), [[509.0, 331.0], [651.0, 325.0], [652.0, 356.0], [511.0, 362.0]]] -[('CT1', 0.9988654), [[535.0, 367.0], [654.0, 367.0], [654.0, 406.0], [535.0, 406.0]]] -...... -``` - -* 识别预测 - -```bash -paddleocr --image_dir doc/imgs_words_en/word_308.png --det false --lang=en -``` - -结果是一个tuple,返回识别结果和识别置信度 - -```text -(0.99879867, 'LITTLE') -``` - -* 检测预测 - -``` -paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg --rec false -``` - -结果是一个list,每个item只包含文本框 - -``` -[[26.0, 457.0], [137.0, 457.0], [137.0, 477.0], [26.0, 477.0]] -[[25.0, 425.0], [372.0, 425.0], [372.0, 448.0], [25.0, 448.0]] -[[128.0, 397.0], [273.0, 397.0], [273.0, 414.0], [128.0, 414.0]] -...... -``` - - -### 2.2 python 脚本运行 - -ppocr 也支持在python脚本中运行,便于嵌入到您自己的代码中 : - -* 整图预测(检测+识别) - -``` -from paddleocr import PaddleOCR, draw_ocr - -# 同样也是通过修改 lang 参数切换语种 -ocr = PaddleOCR(lang="korean") # 首次执行会自动下载模型文件 -img_path = 'doc/imgs/korean_1.jpg ' -result = ocr.ocr(img_path) -# 可通过参数控制单独执行识别、检测 -# result = ocr.ocr(img_path, det=False) 只执行识别 -# result = ocr.ocr(img_path, rec=False) 只执行检测 -# 打印检测框和识别结果 -for line in result: - print(line) - -# 可视化 -from PIL import Image -image = Image.open(img_path).convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/korean.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` - -结果可视化: - -
- -
- -ppocr 还支持方向分类, 更多使用方式请参考:[whl包使用说明](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.0/doc/doc_ch/whl.md) - - -## 3 自定义训练 - -ppocr 支持使用自己的数据进行自定义训练或finetune, 其中识别模型可以参考 [法语配置文件](../../configs/rec/multi_language/rec_french_lite_train.yml) -修改训练数据路径、字典等参数。 - -详细数据准备、训练过程可参考:[文本识别](../doc_ch/recognition.md)、[文本检测](../doc_ch/detection.md)。 - -假设已经准备好了训练数据,可根据以下步骤快速启动训练: - -- 修改配置文件 - -以 `rec_french_lite_train.yml` 为例: - -``` -Global: - ... - # 添加自定义字典,如修改字典请将路径指向新字典 - character_dict_path: ./ppocr/utils/dict/french_dict.txt - ... - # 识别空格 - use_space_char: True - -... - -Train: - dataset: - # 数据集格式,支持LMDBDataSet以及SimpleDataSet - name: SimpleDataSet - # 数据集路径 - data_dir: ./train_data/ - # 训练集标签文件 - label_file_list: ["./train_data/french_train.txt"] - ... - -Eval: - dataset: - # 数据集格式,支持LMDBDataSet以及SimpleDataSet - name: SimpleDataSet - # 数据集路径 - data_dir: ./train_data - # 验证集标签文件 - label_file_list: ["./train_data/french_val.txt"] - ... -``` - -- 启动训练: - -``` -# 下载预训练模型 -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/multilingual/french_mobile_v2.0_rec_train.tar -tar -xf french_mobile_v2.0_rec_train.tar - -#加载预训练模型 单卡训练 -python3 tools/train.py -c configs/rec/rec_french_lite_train.yml -o Global.pretrained_model=french_mobile_v2.0_rec_train/best_accuracy - -#加载预训练模型 多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_french_lite_train.yml -o Global.pretrained_model=french_mobile_v2.0_rec_train/best_accuracy -``` - - -更多功能如预测部署、数据标注等功能可以阅读完整的[文档教程](../../README_ch.md)。 - - -## 4 预测部署 - -除了安装whl包进行快速预测,ppocr 也提供了多种预测部署方式,如有需求可阅读相关文档: -- [基于Python脚本预测引擎推理](./inference_ppocr.md) -- [基于C++预测引擎推理](../../deploy/cpp_infer/readme_ch.md) -- [服务化部署](../../deploy/hubserving/readme.md) -- [端侧部署](../../deploy/lite/readme_ch.md) -- [Benchmark](./benchmark.md) - - - - -## 5 支持语种及缩写 - -| 语种 | 描述 | 缩写 | | 语种 | 描述 | 缩写 | -| --- | --- | --- | ---|--- | --- | --- | -|中文|chinese and english|ch| |保加利亚文|Bulgarian |bg| -|英文|english|en| |乌克兰文|Ukranian|uk| -|法文|french|fr| |白俄罗斯文|Belarusian|be| -|德文|german|german| |泰卢固文|Telugu |te| -|日文|japan|japan| | 阿巴扎文 |Abaza | abq | -|韩文|korean|korean| |泰米尔文|Tamil |ta| -|中文繁体|chinese traditional |chinese_cht| |南非荷兰文 |Afrikaans |af| -|意大利文| Italian |it| |阿塞拜疆文 |Azerbaijani |az| -|西班牙文|Spanish |es| |波斯尼亚文|Bosnian|bs| -|葡萄牙文| Portuguese|pt| |捷克文|Czech|cs| -|俄罗斯文|Russia|ru| |威尔士文 |Welsh |cy| -|阿拉伯文|Arabic|ar| |丹麦文 |Danish|da| -|印地文|Hindi|hi| |爱沙尼亚文 |Estonian |et| -|维吾尔|Uyghur|ug| |爱尔兰文 |Irish |ga| -|波斯文|Persian|fa| |克罗地亚文|Croatian |hr| -|乌尔都文|Urdu|ur| |匈牙利文|Hungarian |hu| -|塞尔维亚文(latin)| Serbian(latin) |rs_latin| |印尼文|Indonesian|id| -|欧西坦文|Occitan |oc| |冰岛文 |Icelandic|is| -|马拉地文|Marathi|mr| |库尔德文 |Kurdish|ku| -|尼泊尔文|Nepali|ne| |立陶宛文|Lithuanian |lt| -|塞尔维亚文(cyrillic)|Serbian(cyrillic)|rs_cyrillic| |拉脱维亚文 |Latvian |lv| -|毛利文|Maori|mi| | 达尔瓦文|Dargwa |dar| -|马来文 |Malay|ms| | 因古什文|Ingush |inh| -|马耳他文 |Maltese |mt| | 拉克文|Lak |lbe| -|荷兰文 |Dutch |nl| | 莱兹甘文|Lezghian |lez| -|挪威文 |Norwegian |no| |塔巴萨兰文 |Tabassaran |tab| -|波兰文|Polish |pl| | 比尔哈文|Bihari |bh| -| 罗马尼亚文|Romanian |ro| | 迈蒂利文|Maithili |mai| -| 斯洛伐克文|Slovak |sk| | 昂加文|Angika |ang| -| 斯洛文尼亚文|Slovenian |sl| | 孟加拉文|Bhojpuri |bho| -| 阿尔巴尼亚文|Albanian |sq| | 摩揭陀文 |Magahi |mah| -| 瑞典文|Swedish |sv| | 那格浦尔文|Nagpur |sck| -| 西瓦希里文|Swahili |sw| | 尼瓦尔文|Newari |new| -| 塔加洛文|Tagalog |tl| | 保加利亚文 |Goan Konkani|gom| -| 土耳其文|Turkish |tr| | 沙特阿拉伯文|Saudi Arabia|sa| -| 乌兹别克文|Uzbek |uz| | 阿瓦尔文|Avar |ava| -| 越南文|Vietnamese |vi| | 阿瓦尔文|Avar |ava| -| 蒙古文|Mongolian |mn| | 阿迪赫文|Adyghe |ady| diff --git a/doc/doc_ch/ocr_book.md b/doc/doc_ch/ocr_book.md deleted file mode 100644 index 420bc8860a..0000000000 --- a/doc/doc_ch/ocr_book.md +++ /dev/null @@ -1,25 +0,0 @@ -# 《动手学OCR》电子书 - -《动手学OCR》是PaddleOCR团队携手华中科技大学博导/教授,IAPR Fellow 白翔、复旦大学青年研究员陈智能、中国移动研究院视觉领域资深专家黄文辉、中国工商银行大数据人工智能实验室研究员等产学研同仁,以及OCR开发者共同打造的结合OCR前沿理论与代码实践的教材。主要特色如下: - -- 覆盖从文本检测识别到文档分析的OCR全栈技术 -- 紧密结合理论实践,跨越代码实现鸿沟,并配套教学视频 -- Notebook交互式学习,灵活修改代码,即刻获得结果 - - -## 本书结构 - -![](https://ai-studio-static-online.cdn.bcebos.com/5e612d9079b84958940614d9613eb928f1a50fe21ba6446cb99186bf2d76fe3d) - -- 第一部分是本书的推荐序、序言与预备知识,包含本书的定位与使用书籍内容的过程中需要用到的知识索引、资源链接等 -- 第二部分是本书的4-8章,介绍与OCR核心的检测、识别能力相关的概念、应用与产业实践。在“OCR技术导论”中总括性的解释OCR的应用场景和挑战、技术基本概念以及在产业应用中的痛点问题。然后在 -“文本检测”与“文本识别”两章中介绍OCR的两个基本任务,并在每章中配套一个算法展开代码详解与实战练习。第6、7章是关于PP-OCR系列模型的详细介绍,PP-OCR是一套面向产业应用的OCR系统,在 -基础检测和识别模型的基础之上经过一系列优化策略达到通用领域的产业级SOTA模型,同时打通多种预测部署方案,赋能企业快速落地OCR应用。 -- 第三部分是本书的9-12章,介绍两阶段OCR引擎之外的应用,包括数据合成、预处理算法、端到端模型,重点展开了OCR在文档场景下的版面分析、表格识别、视觉文档问答的能力,同样通过算法与代码结 -合的方式使得读者能够深入理解并应用。 - - -## 资料地址 -- 中文版电子书下载请扫描首页二维码入群后领取 -- [notebook教程](https://github.com/PaddleOCR-Community/Dive-into-OCR) -- [教学视频](https://aistudio.baidu.com/aistudio/education/group/info/25207) diff --git a/doc/doc_ch/ppocr_introduction.md b/doc/doc_ch/ppocr_introduction.md deleted file mode 100644 index ce0736d91b..0000000000 --- a/doc/doc_ch/ppocr_introduction.md +++ /dev/null @@ -1,156 +0,0 @@ -[English](../doc_en/ppocr_introduction_en.md) | 简体中文 - -# PP-OCR - -- [1. 简介](#1) -- [2. 特性](#2) -- [3. benchmark](#3) -- [4. 效果展示](#4) -- [5. 使用教程](#5) - - [5.1 快速体验](#51) - - [5.2 模型训练、压缩、推理部署](#52) -- [6. 模型库](#6) - - - -## 1. 简介 - -PP-OCR是PaddleOCR自研的实用的超轻量OCR系统。在实现[前沿算法](algorithm.md)的基础上,考虑精度与速度的平衡,进行**模型瘦身**和**深度优化**,使其尽可能满足产业落地需求。 - -#### PP-OCR - -PP-OCR是一个两阶段的OCR系统,其中文本检测算法选用[DB](algorithm_det_db.md),文本识别算法选用[CRNN](algorithm_rec_crnn.md),并在检测和识别模块之间添加[文本方向分类器](angle_class.md),以应对不同方向的文本识别。 - -PP-OCR系统pipeline如下: - -
- -
- - -PP-OCR系统在持续迭代优化,目前已发布PP-OCR和PP-OCRv2两个版本: - -PP-OCR从骨干网络选择和调整、预测头部的设计、数据增强、学习率变换策略、正则化参数选择、预训练模型使用以及模型自动裁剪量化8个方面,采用19个有效策略,对各个模块的模型进行效果调优和瘦身(如绿框所示),最终得到整体大小为3.5M的超轻量中英文OCR和2.8M的英文数字OCR。更多细节请参考[PP-OCR技术报告](https://arxiv.org/abs/2009.09941)。 - -#### PP-OCRv2 - -PP-OCRv2在PP-OCR的基础上,进一步在5个方面重点优化,检测模型采用CML协同互学习知识蒸馏策略和CopyPaste数据增广策略;识别模型采用LCNet轻量级骨干网络、UDML 改进知识蒸馏策略和[Enhanced CTC loss](./enhanced_ctc_loss.md)损失函数改进(如上图红框所示),进一步在推理速度和预测效果上取得明显提升。更多细节请参考[PP-OCRv2技术报告](https://arxiv.org/abs/2109.03144)。 - -#### PP-OCRv3 - -PP-OCRv3在PP-OCRv2的基础上,针对检测模型和识别模型,进行了共计9个方面的升级: -- PP-OCRv3检测模型对PP-OCRv2中的CML协同互学习文本检测蒸馏策略进行了升级,分别针对教师模型和学生模型进行进一步效果优化。其中,在对教师模型优化时,提出了大感受野的PAN结构LK-PAN和引入了DML蒸馏策略;在对学生模型优化时,提出了残差注意力机制的FPN结构RSE-FPN。 -- PP-OCRv3的识别模块是基于文本识别算法[SVTR](https://arxiv.org/abs/2205.00159)优化。SVTR不再采用RNN结构,通过引入Transformers结构更加有效地挖掘文本行图像的上下文信息,从而提升文本识别能力。PP-OCRv3通过轻量级文本识别网络SVTR_LCNet、Attention损失指导CTC损失训练策略、挖掘文字上下文信息的数据增广策略TextConAug、TextRotNet自监督预训练模型、UDML联合互学习策略、UIM无标注数据挖掘方案,6个方面进行模型加速和效果提升。 - -PP-OCRv3系统pipeline如下: - -
- -
-更多细节请参考[PP-OCRv3技术报告](https://arxiv.org/abs/2206.03001v2) 👉[中文简洁版](./PP-OCRv3_introduction.md) - -PP-OCRv4在PP-OCRv3的基础上进一步升级。整体的框架图保持了与PP-OCRv3相同的pipeline,针对检测模型和识别模型进行了数据、网络结构、训练策略等多个模块的优化。 PP-OCRv4系统框图如下所示: - -
- -
- -从算法改进思路上看,分别针对检测和识别模型,进行了共10个方面的改进: - -- 检测模块: - - LCNetV3:精度更高的骨干网络 - - PFHead:并行head分支融合结构 - - DSR: 训练中动态增加shrink ratio - - CML:添加Student和Teacher网络输出的KL div loss -- 识别模块: - - SVTR_LCNetV3:精度更高的骨干网络 - - Lite-Neck:精简的Neck结构 - - GTC-NRTR:稳定的Attention指导分支 - - Multi-Scale:多尺度训练策略 - - DF: 数据挖掘方案 - - DKD :DKD蒸馏策略 - -从效果上看,速度可比情况下,多种场景精度均有大幅提升: - -- 中文场景,相对于PP-OCRv3中文模型提升超4%; -- 英文数字场景,相比于PP-OCRv3英文模型提升6%; -- 多语言场景,优化80个语种识别效果,平均准确率提升超8%。 -- 更多细节请参考 👉[中文简洁版](./PP-OCRv4_introduction.md) - - - -## 2. 特性 - -- 超轻量PP-OCRv4系列:检测(4.7M)+ 方向分类器(1.4M)+ 识别(10M)= 16.1M -- 超轻量PP-OCRv3系列:检测(3.6M)+ 方向分类器(1.4M)+ 识别(12M)= 17.0M -- 超轻量PP-OCRv2系列:检测(3.1M)+ 方向分类器(1.4M)+ 识别(8.5M)= 13.0M -- 超轻量PP-OCR mobile移动端系列:检测(3.0M)+方向分类器(1.4M)+ 识别(5.0M)= 9.4M -- 通用PP-OCR server系列:检测(47.1M)+方向分类器(1.4M)+ 识别(94.9M)= 143.4M -- 支持中英文数字组合识别、竖排文本识别、长文本识别 -- 支持多语言识别:韩语、日语、德语、法语等约80种语言 - - -## 3. benchmark - -关于PP-OCR系列模型之间的性能对比,请查看[benchmark](./benchmark.md)文档。 - - - -## 4. 效果展示 [more](./visualization.md) - -
-PP-OCRv3 中文模型 -
- - - -
-
- -
-PP-OCRv3 英文模型 -
- - -
-
- -
-PP-OCRv3 多语言模型 -
- - -
-
- - - -## 5. 使用教程 - - -### 5.1 快速体验 - -- 在线网站体验:超轻量PP-OCR mobile模型体验地址:https://www.paddlepaddle.org.cn/hub/scene/ocr -- 移动端demo体验:[安装包DEMO下载地址](https://ai.baidu.com/easyedge/app/openSource?from=paddlelite)(基于EasyEdge和Paddle-Lite, 支持iOS和Android系统) -- 一行命令快速使用:[快速开始(中英文/多语言)](./doc/doc_ch/quickstart.md) - - -### 5.2 模型训练、压缩、推理部署 - -更多教程,包括模型训练、模型压缩、推理部署等,请参考[文档教程](../../README.md#文档教程)。 - - -## 6. 模型库 - -PP-OCR中英文模型列表如下: - -| 模型简介 | 模型名称 | 推荐场景 | 检测模型 | 方向分类器 | 识别模型 | -| ------------------------------------- | ----------------------- | --------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | -| 中英文超轻量 PP-OCRv4 模型(15.8M) | ch_PP-OCRv4_xx | 移动端&服务器端 | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_det_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_rec_train.tar) | -| 中英文超轻量PP-OCRv3模型(16.2M) | ch_PP-OCRv3_xx | 移动端&服务器端 | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_train.tar) | -| 英文超轻量PP-OCRv3模型(13.4M) | en_PP-OCRv3_xx | 移动端&服务器端 | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_distill_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_train.tar) | -| 中英文超轻量PP-OCRv2模型(13.0M) | ch_PP-OCRv2_xx | 移动端&服务器端 | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_distill_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_train.tar) | -| 中英文超轻量PP-OCR mobile模型(9.4M) | ch_ppocr_mobile_v2.0_xx | 移动端&服务器端 | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_infer.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_infer.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_pre.tar) | -| 中英文通用PP-OCR server模型(143.4M) | ch_ppocr_server_v2.0_xx | 服务器端 | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_det_infer.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_det_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_rec_infer.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_rec_pre.tar) | - -更多模型下载(包括英文数字模型、多语言模型、Paddle-Lite模型等),可以参考[PP-OCR 系列模型下载](./models_list.md)。 diff --git a/doc/doc_ch/quickstart.md b/doc/doc_ch/quickstart.md deleted file mode 100644 index 65b29b1879..0000000000 --- a/doc/doc_ch/quickstart.md +++ /dev/null @@ -1,297 +0,0 @@ -# PaddleOCR 快速开始 - -**说明:** 本文主要介绍PaddleOCR wheel包对PP-OCR系列模型的快速使用,如要体验文档分析相关功能,请参考[PP-Structure快速使用教程](../../ppstructure/docs/quickstart.md)。 - -- [1. 安装](#1) - - [1.1 安装PaddlePaddle](#11) - - [1.2 安装PaddleOCR whl包](#12) -- [2. 便捷使用](#2) - - [2.1 命令行使用](#21) - - [2.1.1 中英文模型](#211) - - [2.1.2 多语言模型](#212) - - [2.2 Python脚本使用](#22) - - [2.2.1 中英文与多语言使用](#221) -- [3.小结](#3) - - - -## 1. 安装 - - -### 1.1 安装PaddlePaddle - -> 如果您没有基础的Python运行环境,请参考[运行环境准备](./environment.md)。 - -- 您的机器安装的是CUDA 11,请运行以下命令安装 - - ```bash - pip install paddlepaddle-gpu - ``` - -- 您的机器是CPU,请运行以下命令安装 - - ```bash - pip install paddlepaddle - ``` - -更多的版本需求,请参照[飞桨官网安装文档](https://www.paddlepaddle.org.cn/install/quick)中的说明进行操作。 - - -### 1.2 安装PaddleOCR whl包 - -```bash -pip install paddleocr -``` - - -## 2. 便捷使用 - -### 2.1 命令行使用 - -PaddleOCR提供了一系列测试图片,点击[这里](https://paddleocr.bj.bcebos.com/dygraph_v2.1/ppocr_img.zip)下载并解压,然后在终端中切换到相应目录 - -``` -cd /path/to/ppocr_img -``` - -如果不使用提供的测试图片,可以将下方`--image_dir`参数替换为相应的测试图片路径。 - - -#### 2.1.1 中英文模型 - -* 检测+方向分类器+识别全流程:`--use_angle_cls true`设置使用方向分类器识别180度旋转文字,`--use_gpu false`设置不使用GPU - - ```bash - paddleocr --image_dir ./imgs/11.jpg --use_angle_cls true --use_gpu false - ``` - - 结果是一个list,每个item包含了文本框,文字和识别置信度 - - ```bash - [[[28.0, 37.0], [302.0, 39.0], [302.0, 72.0], [27.0, 70.0]], ('纯臻营养护发素', 0.9658738374710083)] - ...... - ``` - - 此外,paddleocr也支持输入pdf文件,并且可以通过指定参数`page_num`来控制推理前面几页,默认为0,表示推理所有页。 - ```bash - paddleocr --image_dir ./xxx.pdf --use_angle_cls true --use_gpu false --page_num 2 - ``` - -- 单独使用检测:设置`--rec`为`false` - - ```bash - paddleocr --image_dir ./imgs/11.jpg --rec false - ``` - - 结果是一个list,每个item只包含文本框 - - ```bash - [[27.0, 459.0], [136.0, 459.0], [136.0, 479.0], [27.0, 479.0]] - [[28.0, 429.0], [372.0, 429.0], [372.0, 445.0], [28.0, 445.0]] - ...... - ``` - -- 单独使用识别:设置`--det`为`false` - - ```bash - paddleocr --image_dir ./imgs_words/ch/word_1.jpg --det false - ``` - - 结果是一个list,每个item只包含识别结果和识别置信度 - - ```bash - ['韩国小馆', 0.994467] - ``` - -**版本说明** -paddleocr默认使用PP-OCRv4模型(`--ocr_version PP-OCRv4`),如需使用其他版本可通过设置参数`--ocr_version`,具体版本说明如下: -| 版本名称 | 版本说明 | -| --- | --- | -| PP-OCRv4 | 支持中、英文检测和识别,方向分类器,支持多语种识别 | -| PP-OCRv3 | 支持中、英文检测和识别,方向分类器,支持多语种识别 | -| PP-OCRv2 | 支持中英文的检测和识别,方向分类器,多语言暂未更新 | -| PP-OCR | 支持中、英文检测和识别,方向分类器,支持多语种识别 | - -如需新增自己训练的模型,可以在[paddleocr](../../paddleocr.py)中增加模型链接和字段,重新编译即可。 - -更多whl包使用可参考[whl包文档](./whl.md) - - - -#### 2.1.2 多语言模型 - -PaddleOCR目前支持80个语种,可以通过修改`--lang`参数进行切换,对于英文模型,指定`--lang=en`。 - -``` bash -paddleocr --image_dir ./imgs_en/254.jpg --lang=en -``` - -
- - -
- -结果是一个list,每个item包含了文本框,文字和识别置信度 - -```text -[[[67.0, 51.0], [327.0, 46.0], [327.0, 74.0], [68.0, 80.0]], ('PHOCAPITAL', 0.9944712519645691)] -[[[72.0, 92.0], [453.0, 84.0], [454.0, 114.0], [73.0, 122.0]], ('107 State Street', 0.9744491577148438)] -[[[69.0, 135.0], [501.0, 125.0], [501.0, 156.0], [70.0, 165.0]], ('Montpelier Vermont', 0.9357033967971802)] -...... -``` - -常用的多语言简写包括 - -| 语种 | 缩写 | | 语种 | 缩写 | | 语种 | 缩写 | -| -------- | ----------- | ---- | -------- | ------ | ---- | -------- | ------ | -| 中文 | ch | | 法文 | fr | | 日文 | japan | -| 英文 | en | | 德文 | german | | 韩文 | korean | -| 繁体中文 | chinese_cht | | 意大利文 | it | | 俄罗斯文 | ru | - -全部语种及其对应的缩写列表可查看[多语言模型教程](./multi_languages.md) - - - -### 2.2 Python脚本使用 - -#### 2.2.1 中英文与多语言使用 - -通过Python脚本使用PaddleOCR whl包,whl包会自动下载ppocr轻量级模型作为默认模型。 - -* 检测+方向分类器+识别全流程 - -```python -from paddleocr import PaddleOCR, draw_ocr - -# Paddleocr目前支持的多语言语种可以通过修改lang参数进行切换 -# 例如`ch`, `en`, `fr`, `german`, `korean`, `japan` -ocr = PaddleOCR(use_angle_cls=True, lang="ch") # need to run only once to download and load model into memory -img_path = './imgs/11.jpg' -result = ocr.ocr(img_path, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# 显示结果 -from PIL import Image -result = result[0] -image = Image.open(img_path).convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='./fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` - -结果是一个list,每个item包含了文本框,文字和识别置信度 - -```bash -[[[28.0, 37.0], [302.0, 39.0], [302.0, 72.0], [27.0, 70.0]], ('纯臻营养护发素', 0.9658738374710083)] -...... -``` - -结果可视化 - -
- -
- - - - -如果输入是PDF文件,那么可以参考下面代码进行可视化 - -```python -from paddleocr import PaddleOCR, draw_ocr - -# Paddleocr目前支持的多语言语种可以通过修改lang参数进行切换 -# 例如`ch`, `en`, `fr`, `german`, `korean`, `japan` -PAGE_NUM = 10 # 将识别页码前置作为全局,防止后续打开pdf的参数和前文识别参数不一致 / Set the recognition page number -pdf_path = 'default.pdf' -ocr = PaddleOCR(use_angle_cls=True, lang="ch", page_num=PAGE_NUM) # need to run only once to download and load model into memory -# ocr = PaddleOCR(use_angle_cls=True, lang="ch", page_num=PAGE_NUM,use_gpu=0) # 如果需要使用GPU,请取消此行的注释 并注释上一行 / To Use GPU,uncomment this line and comment the above one. -result = ocr.ocr(pdf_path, cls=True) -for idx in range(len(result)): - res = result[idx] - if res == None: # 识别到空页就跳过,防止程序报错 / Skip when empty result detected to avoid TypeError:NoneType - print(f"[DEBUG] Empty page {idx+1} detected, skip it.") - continue - for line in res: - print(line) -# 显示结果 -import fitz -from PIL import Image -import cv2 -import numpy as np -imgs = [] -with fitz.open(pdf_path) as pdf: - for pg in range(0, PAGE_NUM): - page = pdf[pg] - mat = fitz.Matrix(2, 2) - pm = page.get_pixmap(matrix=mat, alpha=False) - # if width or height > 2000 pixels, don't enlarge the image - if pm.width > 2000 or pm.height > 2000: - pm = page.get_pixmap(matrix=fitz.Matrix(1, 1), alpha=False) - img = Image.frombytes("RGB", [pm.width, pm.height], pm.samples) - img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR) - imgs.append(img) -for idx in range(len(result)): - res = result[idx] - if res == None: - continue - image = imgs[idx] - boxes = [line[0] for line in res] - txts = [line[1][0] for line in res] - scores = [line[1][1] for line in res] - im_show = draw_ocr(image, boxes, txts, scores, font_path='doc/fonts/simfang.ttf') - im_show = Image.fromarray(im_show) - im_show.save('result_page_{}.jpg'.format(idx)) -``` - -* 使用滑动窗口进行检测和识别 - -要使用滑动窗口进行光学字符识别(OCR),可以使用以下代码片段: - -```Python -from paddleocr import PaddleOCR -from PIL import Image, ImageDraw, ImageFont - -# 初始化OCR引擎 -ocr = PaddleOCR(use_angle_cls=True, lang="en") - -img_path = "./very_large_image.jpg" -slice = {'horizontal_stride': 300, 'vertical_stride': 500, 'merge_x_thres': 50, 'merge_y_thres': 35} -results = ocr.ocr(img_path, cls=True, slice=slice) - -# 加载图像 -image = Image.open(img_path).convert("RGB") -draw = ImageDraw.Draw(image) -font = ImageFont.truetype("./doc/fonts/simfang.ttf", size=20) # 根据需要调整大小 - -# 处理并绘制结果 -for res in results: - for line in res: - box = [tuple(point) for point in line[0]] - # 找出边界框 - box = [(min(point[0] for point in box), min(point[1] for point in box)), - (max(point[0] for point in box), max(point[1] for point in box))] - txt = line[1][0] - draw.rectangle(box, outline="red", width=2) # 绘制矩形 - draw.text((box[0][0], box[0][1] - 25), txt, fill="blue", font=font) # 在矩形上方绘制文本 - -# 保存结果 -image.save("result.jpg") - -``` - -此示例初始化了启用角度分类的PaddleOCR实例,并将语言设置为英语。然后调用`ocr`方法,并使用多个参数来自定义检测和识别过程,包括处理图像切片的`slice`参数。 - -要更全面地了解切片操作,请参考[切片操作文档](./slice.md)。 - -## 3. 小结 - -通过本节内容,相信您已经熟练掌握PaddleOCR whl包的使用方法并获得了初步效果。 - -飞桨AI套件(PaddleX)提供了飞桨生态优质模型,是训压推一站式全流程高效率开发平台,其使命是助力AI技术快速落地,愿景是使人人成为AI Developer!目前PP-OCRv4已上线PaddleX,您可以进入[通用OCR](https://aistudio.baidu.com/aistudio/modelsdetail?modelId=286)体验模型训练、压缩和推理部署全流程。 diff --git a/doc/doc_ch/rec_algo_compare.png b/doc/doc_ch/rec_algo_compare.png deleted file mode 100644 index 2dde496c75..0000000000 Binary files a/doc/doc_ch/rec_algo_compare.png and /dev/null differ diff --git a/doc/doc_ch/recognition.md b/doc/doc_ch/recognition.md deleted file mode 100644 index 99374abf02..0000000000 --- a/doc/doc_ch/recognition.md +++ /dev/null @@ -1,587 +0,0 @@ -# 文字识别 - -本文提供了PaddleOCR文本识别任务的全流程指南,包括数据准备、模型训练、调优、评估、预测,各个阶段的详细说明: - -- [1. 数据准备](#1-数据准备) - - [1.1. 准备数据集](#11-准备数据集) - - [1.2. 自定义数据集](#12-自定义数据集) - - [1.3. 数据下载](#13-数据下载) - - [1.4. 字典](#14-字典) - - [1.5. 添加空格类别](#15-添加空格类别) - - [1.6. 数据增强](#16-数据增强) -- [2. 开始训练](#2-开始训练) - - [2.1. 启动训练](#21-启动训练) - - [2.2. 断点训练](#22-断点训练) - - [2.3. 更换Backbone 训练](#23-更换backbone-训练) - - [2.4. 混合精度训练](#24-混合精度训练) - - [2.5. 分布式训练](#25-分布式训练) - - [2.6. 知识蒸馏训练](#26-知识蒸馏训练) - - [2.7. 多语言模型训练](#27-多语言模型训练) - - [2.8. 其他训练环境](#28-其他训练环境) - - [2.9. 模型微调](#29-模型微调) -- [3. 模型评估与预测](#3-模型评估与预测) - - [3.1. 指标评估](#31-指标评估) - - [3.2. 测试识别效果](#32-测试识别效果) -- [4. 模型导出与预测](#4-模型导出与预测) -- [5. FAQ](#5-faq) - -# 1. 数据准备 - -## 1.1. 准备数据集 - -PaddleOCR 支持两种数据格式: - - `lmdb` 用于训练以lmdb格式存储的数据集(LMDBDataSet); - - `通用数据` 用于训练以文本文件存储的数据集(SimpleDataSet); - -训练数据的默认存储路径是 `PaddleOCR/train_data`,如果您的磁盘上已有数据集,只需创建软链接至数据集目录: - -``` -# linux and mac os -ln -sf /train_data/dataset -# windows -mklink /d /train_data/dataset -``` - -## 1.2. 自定义数据集 - -下面以通用数据集为例, 介绍如何准备数据集: - -* 训练集 - -建议将训练图片放入同一个文件夹,并用一个txt文件(rec_gt_train.txt)记录图片路径和标签,txt文件里的内容如下: - -**注意:** txt文件中默认请将图片路径和图片标签用 \t 分割,如用其他方式分割将造成训练报错。 - -``` -" 图像文件名 图像标注信息 " - -train_data/rec/train/word_001.jpg 简单可依赖 -train_data/rec/train/word_002.jpg 用科技让复杂的世界更简单 -... -``` - -最终训练集应有如下文件结构: -``` -|-train_data - |-rec - |- rec_gt_train.txt - |- train - |- word_001.png - |- word_002.jpg - |- word_003.jpg - | ... -``` - -除上述单张图像为一行格式之外,PaddleOCR也支持对离线增广后的数据进行训练,为了防止相同样本在同一个batch中被多次采样,我们可以将相同标签对应的图片路径写在一行中,以列表的形式给出,在训练中,PaddleOCR会随机选择列表中的一张图片进行训练。对应地,标注文件的格式如下。 - -``` -["11.jpg", "12.jpg"] 简单可依赖 -["21.jpg", "22.jpg", "23.jpg"] 用科技让复杂的世界更简单 -3.jpg ocr -``` - -上述示例标注文件中,"11.jpg"和"12.jpg"的标签相同,都是`简单可依赖`,在训练的时候,对于该行标注,会随机选择其中的一张图片进行训练。 - - -- 验证集 - -同训练集类似,验证集也需要提供一个包含所有图片的文件夹(test)和一个rec_gt_test.txt,验证集的结构如下所示: - -``` -|-train_data - |-rec - |- rec_gt_test.txt - |- test - |- word_001.jpg - |- word_002.jpg - |- word_003.jpg - | ... -``` - -## 1.3. 数据下载 - -- ICDAR2015 - -若您本地没有数据集,可以在官网下载 [ICDAR2015](http://rrc.cvc.uab.es/?ch=4&com=downloads) 数据,用于快速验证。也可以参考[DTRB](https://github.com/clovaai/deep-text-recognition-benchmark#download-lmdb-dataset-for-traininig-and-evaluation-from-here) ,下载 benchmark 所需的lmdb格式数据集。 - -如果你使用的是icdar2015的公开数据集,PaddleOCR 提供了一份用于训练 ICDAR2015 数据集的标签文件,通过以下方式下载: - -``` -# 训练集标签 -wget -P ./train_data/ic15_data https://paddleocr.bj.bcebos.com/dataset/rec_gt_train.txt -# 测试集标签 -wget -P ./train_data/ic15_data https://paddleocr.bj.bcebos.com/dataset/rec_gt_test.txt -``` - -PaddleOCR 也提供了数据格式转换脚本,可以将ICDAR官网 label 转换为PaddleOCR支持的数据格式。 数据转换工具在 `ppocr/utils/gen_label.py`, 这里以训练集为例: - -``` -# 将官网下载的标签文件转换为 rec_gt_label.txt -python gen_label.py --mode="rec" --input_path="{path/of/origin/label}" --output_label="rec_gt_label.txt" -``` - -数据样式格式如下,(a)为原始图片,(b)为每张图片对应的 Ground Truth 文本文件: -![](../datasets/icdar_rec.png) - -- 多语言数据集 - -多语言模型的训练数据集均为100w的合成数据,使用了开源合成工具 [text_renderer](https://github.com/Sanster/text_renderer) ,少量的字体可以通过下面两种方式下载。 -* [百度网盘](https://pan.baidu.com/s/1bS_u207Rm7YbY33wOECKDA) 提取码:frgi -* [google drive](https://drive.google.com/file/d/18cSWX7wXSy4G0tbKJ0d9PuIaiwRLHpjA/view) - -## 1.4. 字典 - -最后需要提供一个字典({word_dict_name}.txt),使模型在训练时,可以将所有出现的字符映射为字典的索引。 - -因此字典需要包含所有希望被正确识别的字符,{word_dict_name}.txt需要写成如下格式,并以 `utf-8` 编码格式保存: - -``` -l -d -a -d -r -n -``` - -word_dict.txt 每行有一个单字,将字符与数字索引映射在一起,“and” 将被映射成 [2 5 1] - -* 内置字典 - -PaddleOCR内置了一部分字典,可以按需使用。 - -`ppocr/utils/ppocr_keys_v1.txt` 是一个包含6623个字符的中文字典 - -`ppocr/utils/ic15_dict.txt` 是一个包含36个字符的英文字典 - -`ppocr/utils/dict/french_dict.txt` 是一个包含118个字符的法文字典 - -`ppocr/utils/dict/japan_dict.txt` 是一个包含4399个字符的日文字典 - -`ppocr/utils/dict/korean_dict.txt` 是一个包含3636个字符的韩文字典 - -`ppocr/utils/dict/german_dict.txt` 是一个包含131个字符的德文字典 - -`ppocr/utils/en_dict.txt` 是一个包含96个字符的英文字典 - - -目前的多语言模型仍处在demo阶段,会持续优化模型并补充语种,**非常欢迎您为我们提供其他语言的字典和字体**, -如您愿意可将字典文件提交至 [dict](../../ppocr/utils/dict),我们会在Repo中感谢您。 - -- 自定义字典 - -如需自定义dic文件,请在 `configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml` 中添加 `character_dict_path` 字段, 指向您的字典路径。 - -## 1.5. 添加空格类别 - -如果希望支持识别"空格"类别, 请将yml文件中的 `use_space_char` 字段设置为 `True`。 - -## 1.6. 数据增强 - -PaddleOCR提供了多种数据增强方式,默认配置文件中已经添加了数据增广。 - -默认的扰动方式有:颜色空间转换(cvtColor)、模糊(blur)、抖动(jitter)、噪声(Gasuss noise)、随机切割(random crop)、透视(perspective)、颜色反转(reverse)、TIA数据增广。 - -训练过程中每种扰动方式以40%的概率被选择,具体代码实现请参考:[rec_img_aug.py](../../ppocr/data/imaug/rec_img_aug.py) - -*由于OpenCV的兼容性问题,扰动操作暂时只支持Linux* - -# 2. 开始训练 - -PaddleOCR提供了训练脚本、评估脚本和预测脚本,本节将以 PP-OCRv4 英文识别模型为例: - -## 2.1. 启动训练 - -首先下载pretrain model,您可以下载训练好的模型在 icdar2015 数据上进行finetune - -``` -cd PaddleOCR/ -# 下载英文PP-OCRv4的预训练模型 -wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/PP-OCRv4/english/en_PP-OCRv4_rec_train.tar -# 解压模型参数 -cd pretrain_models -tar -xf en_PP-OCRv4_rec_train.tar && rm -rf en_PP-OCRv4_rec_train.tar -``` - -开始训练: - -*如果您安装的是cpu版本,请将配置文件中的 `use_gpu` 字段修改为false* - -``` -# GPU训练 支持单卡,多卡训练 -# 训练icdar15英文数据 训练日志会自动保存为 "{save_model_dir}" 下的train.log - -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml -o Global.pretrained_model=./pretrain_models/en_PP-OCRv4_rec_train/best_accuracy - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml -o Global.pretrained_model=./pretrain_models/en_PP-OCRv4_rec_train/best_accuracy -``` - -正常启动训练后,会看到以下log输出: - -``` -[2024/06/16 11:28:26] ppocr INFO: epoch: [1/50], global_step: 10, lr: 0.000007, acc: 0.343750, norm_edit_dis: 0.752802, CTCLoss: 13.178495, NRTRLoss: 1.398275, loss: 14.568232, avg_reader_cost: 0.28627 s, avg_batch_cost: 1.04250 s, avg_samples: 67.2, ips: 64.46042 samples/s, eta: 0:57:09, max_mem_reserved: 12078 MB, max_mem_allocated: 11935 MB -[2024/06/16 11:28:30] ppocr INFO: epoch: [1/50], global_step: 20, lr: 0.000014, acc: 0.361979, norm_edit_dis: 0.764480, CTCLoss: 12.389563, NRTRLoss: 1.389737, loss: 13.795437, avg_reader_cost: 0.00035 s, avg_batch_cost: 0.47960 s, avg_samples: 65.6, ips: 136.78172 samples/s, eta: 0:41:36, max_mem_reserved: 12078 MB, max_mem_allocated: 11949 MB -[2024/06/16 11:28:35] ppocr INFO: epoch: [1/50], global_step: 30, lr: 0.000030, acc: 0.390625, norm_edit_dis: 0.788205, CTCLoss: 10.617269, NRTRLoss: 1.334532, loss: 11.975240, avg_reader_cost: 0.00035 s, avg_batch_cost: 0.48364 s, avg_samples: 78.4, ips: 162.10323 samples/s, eta: 0:36:26, max_mem_reserved: 12078 MB, max_mem_allocated: 11976 MB -[2024/06/16 11:28:40] ppocr INFO: epoch: [1/50], global_step: 40, lr: 0.000045, acc: 0.393229, norm_edit_dis: 0.785400, CTCLoss: 10.627735, NRTRLoss: 1.330406, loss: 11.949156, avg_reader_cost: 0.00036 s, avg_batch_cost: 0.48152 s, avg_samples: 73.6, ips: 152.84850 samples/s, eta: 0:33:47, max_mem_reserved: 12078 MB, max_mem_allocated: 11976 MB -[2024/06/16 11:28:45] ppocr INFO: epoch: [1/50], global_step: 50, lr: 0.000060, acc: 0.401042, norm_edit_dis: 0.804457, CTCLoss: 8.343242, NRTRLoss: 1.220365, loss: 9.561325, avg_reader_cost: 0.00035 s, avg_batch_cost: 0.48413 s, avg_samples: 67.2, ips: 138.80508 samples/s, eta: 0:32:11, max_mem_reserved: 12078 MB, max_mem_allocated: 11976 MB -[2024/06/16 11:28:50] ppocr INFO: epoch: [1/50], global_step: 60, lr: 0.000075, acc: 0.468750, norm_edit_dis: 0.833311, CTCLoss: 7.356572, NRTRLoss: 1.191381, loss: 8.570213, avg_reader_cost: 0.00028 s, avg_batch_cost: 0.47786 s, avg_samples: 68.8, ips: 143.97404 samples/s, eta: 0:31:02, max_mem_reserved: 12078 MB, max_mem_allocated: 11990 MB -[2024/06/16 11:28:53] ppocr INFO: epoch: [1/50], global_step: 66, lr: 0.000084, acc: 0.489583, norm_edit_dis: 0.841231, CTCLoss: 7.246758, NRTRLoss: 1.181412, loss: 8.424673, avg_reader_cost: 0.00018 s, avg_batch_cost: 0.28659 s, avg_samples: 36.8, ips: 128.40797 samples/s, eta: 0:30:30, max_mem_reserved: 12078 MB, max_mem_allocated: 11990 MB -``` - -log 中自动打印如下信息: - -| 字段 | 含义 | -| :----: | :------: | -| epoch | 当前迭代轮次 | -| iter | 当前迭代次数 | -| lr | 当前学习率 | -| loss | 当前损失函数 | -| acc | 当前batch的准确率 | -| norm_edit_dis | 当前 batch 的编辑距离 | -| reader_cost | 当前 batch 数据处理耗时 | -| batch_cost | 当前 batch 总耗时 | -| samples | 当前 batch 内的样本数 | -| ips | 每秒处理图片的数量 | - - -PaddleOCR支持训练和评估交替进行, 可以在 `configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml` 中修改 `eval_batch_step` 设置评估频率,默认每500个iter评估一次。评估过程中默认将最佳acc模型,保存为 `output/en_PP-OCRv4_rec/best_accuracy` 。 - -如果验证集很大,测试将会比较耗时,建议减少评估次数,或训练完再进行评估。 - -**提示:** 可通过 -c 参数选择 `configs/rec/` 路径下的多种模型配置进行训练,PaddleOCR支持的识别算法可以参考[前沿算法列表](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/doc/doc_ch/algorithm_overview.md#12-%E6%96%87%E6%9C%AC%E8%AF%86%E5%88%AB%E7%AE%97%E6%B3%95): - - -训练中文数据,推荐使用[ch_PP-OCRv4_rec_distillation.yml](../../configs/rec/PP-OCRv4/ch_PP-OCRv4_rec_distillation.yml),如您希望尝试其他算法在中文数据集上的效果,请参考下列说明修改配置文件: - -以 `ch_PP-OCRv4_rec_distillation.yml` 为例: -``` -Global: - ... - # 添加自定义字典,如修改字典请将路径指向新字典 - character_dict_path: ppocr/utils/ppocr_keys_v1.txt - ... - # 识别空格 - use_space_char: True - - -Optimizer: - ... - # 添加学习率衰减策略 - lr: - name: Cosine - learning_rate: 0.0005 - ... - -... - -Train: - dataset: - # 数据集格式,支持LMDBDataSet以及SimpleDataSet - name: SimpleDataSet - # 数据集路径 - data_dir: ./train_data/ - # 训练集标签文件 - label_file_list: ["./train_data/train_list.txt"] - transforms: - ... - - RecResizeImg: - # 修改 image_shape 以适应长文本 - image_shape: [3, 48, 320] - ... - loader: - ... - # 单卡训练的batch_size - batch_size_per_card: 256 - ... - -Eval: - dataset: - # 数据集格式,支持LMDBDataSet以及SimpleDataSet - name: SimpleDataSet - # 数据集路径 - data_dir: ./train_data - # 验证集标签文件 - label_file_list: ["./train_data/val_list.txt"] - transforms: - ... - - RecResizeImg: - # 修改 image_shape 以适应长文本 - image_shape: [3, 48, 320] - ... - loader: - # 单卡验证的batch_size - batch_size_per_card: 256 - ... -``` -**注意,预测/评估时的配置文件请务必与训练一致。** - -## 2.2. 断点训练 - -如果训练程序中断,如果希望加载训练中断的模型从而恢复训练,可以通过指定Global.checkpoints指定要加载的模型路径: -```shell -python3 tools/train.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml -o Global.checkpoints=./your/trained/model -``` - -**注意**:`Global.checkpoints`的优先级高于`Global.pretrained_model`的优先级,即同时指定两个参数时,优先加载`Global.checkpoints`指定的模型,如果`Global.checkpoints`指定的模型路径有误,会加载`Global.pretrained_model`指定的模型。 - -## 2.3. 更换Backbone 训练 - -PaddleOCR将网络划分为四部分,分别在[ppocr/modeling](../../ppocr/modeling)下。 进入网络的数据将按照顺序(transforms->backbones->necks->heads)依次通过这四个部分。 - -```bash -├── architectures # 网络的组网代码 -├── transforms # 网络的图像变换模块 -├── backbones # 网络的特征提取模块 -├── necks # 网络的特征增强模块 -└── heads # 网络的输出模块 -``` -如果要更换的Backbone 在PaddleOCR中有对应实现,直接修改配置yml文件中`Backbone`部分的参数即可。 - -如果要使用新的Backbone,更换backbones的例子如下: - -1. 在 [ppocr/modeling/backbones](../../ppocr/modeling/backbones) 文件夹下新建文件,如my_backbone.py。 -2. 在 my_backbone.py 文件内添加相关代码,示例代码如下: - -```python -import paddle -import paddle.nn as nn -import paddle.nn.functional as F - - -class MyBackbone(nn.Layer): - def __init__(self, *args, **kwargs): - super(MyBackbone, self).__init__() - # your init code - self.conv = nn.xxxx - - def forward(self, inputs): - # your network forward - y = self.conv(inputs) - return y -``` - -3. 在 [ppocr/modeling/backbones/\__init\__.py](../../ppocr/modeling/backbones/__init__.py)文件内导入添加的`MyBackbone`模块,然后修改配置文件中Backbone进行配置即可使用,格式如下: - -```yaml -Backbone: -name: MyBackbone -args1: args1 -``` - -**注意**:如果要更换网络的其他模块,可以参考[文档](./add_new_algorithm.md)。 - -## 2.4. 混合精度训练 - -如果您想进一步加快训练速度,可以使用[自动混合精度训练](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/01_paddle2.0_introduction/basic_concept/amp_cn.html), 以单机单卡为例,命令如下: - -```shell -python3 tools/train.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml \ - -o Global.pretrained_model=./pretrain_models/en_PP-OCRv4_rec_train/best_accuracy \ - Global.use_amp=True Global.scale_loss=1024.0 Global.use_dynamic_loss_scaling=True -``` - -## 2.5. 分布式训练 - -多机多卡训练时,通过 `--ips` 参数设置使用的机器IP地址,通过 `--gpus` 参数设置使用的GPU ID: - -```bash -python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml \ - -o Global.pretrained_model=./pretrain_models/en_PP-OCRv4_rec_train/best_accuracy -``` - -**注意:** (1)采用多机多卡训练时,需要替换上面命令中的ips值为您机器的地址,机器之间需要能够相互ping通;(2)训练时需要在多个机器上分别启动命令。查看机器ip地址的命令为`ifconfig`;(3)更多关于分布式训练的性能优势等信息,请参考:[分布式训练教程](./distributed_training.md)。 - -## 2.6. 知识蒸馏训练 - -PaddleOCR支持了基于知识蒸馏的文本识别模型训练过程,更多内容可以参考[知识蒸馏说明文档](./knowledge_distillation.md)。 - -## 2.7. 多语言模型训练 - -PaddleOCR目前已支持80种(除中文外)语种识别,`configs/rec/multi_languages` 路径下提供了一个多语言的配置文件模版: [rec_multi_language_lite_train.yml](../../configs/rec/multi_language/rec_multi_language_lite_train.yml)。 - -按语系划分,目前PaddleOCR支持的语种有: - -| 配置文件 | 算法名称 | backbone | trans | seq | pred | language | -| :--------: | :-------: | :-------: | :-------: | :-----: | :-----: | :-----: | -| rec_chinese_cht_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 中文繁体 | -| rec_en_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 英语(区分大小写) | -| rec_french_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 法语 | -| rec_ger_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 德语 | -| rec_japan_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 日语 | -| rec_korean_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 韩语 | -| rec_latin_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 拉丁字母 | -| rec_arabic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 阿拉伯字母 | -| rec_cyrillic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 斯拉夫字母 | -| rec_devanagari_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 梵文字母 | - -更多支持语种请参考: [多语言模型](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/doc/doc_ch/multi_languages.md#%E8%AF%AD%E7%A7%8D%E7%BC%A9%E5%86%99) - -如您希望在现有模型效果的基础上调优,请参考下列说明修改配置文件: - -以 `rec_french_lite_train` 为例: -``` -Global: - ... - # 添加自定义字典,如修改字典请将路径指向新字典 - character_dict_path: ./ppocr/utils/dict/french_dict.txt - ... - # 识别空格 - use_space_char: True - -... - -Train: - dataset: - # 数据集格式,支持LMDBDataSet以及SimpleDataSet - name: SimpleDataSet - # 数据集路径 - data_dir: ./train_data/ - # 训练集标签文件 - label_file_list: ["./train_data/french_train.txt"] - ... - -Eval: - dataset: - # 数据集格式,支持LMDBDataSet以及SimpleDataSet - name: SimpleDataSet - # 数据集路径 - data_dir: ./train_data - # 验证集标签文件 - label_file_list: ["./train_data/french_val.txt"] - ... -``` - -## 2.8. 其他训练环境 - -- Windows GPU/CPU -在Windows平台上与Linux平台略有不同: -Windows平台只支持`单卡`的训练与预测,指定GPU进行训练`set CUDA_VISIBLE_DEVICES=0` -在Windows平台,DataLoader只支持单进程模式,因此需要设置 `num_workers` 为0; - -- macOS -不支持GPU模式,需要在配置文件中设置`use_gpu`为False,其余训练评估预测命令与Linux GPU完全相同。 - -- Linux DCU -DCU设备上运行需要设置环境变量 `export HIP_VISIBLE_DEVICES=0,1,2,3`,其余训练评估预测命令与Linux GPU完全相同。 - -## 2.9 模型微调 - -实际使用过程中,建议加载官方提供的预训练模型,在自己的数据集中进行微调,关于识别模型的微调方法,请参考:[模型微调教程](./finetune.md)。 - - -# 3. 模型评估与预测 - -## 3.1. 指标评估 - -训练中模型参数默认保存在`Global.save_model_dir`目录下。在评估指标时,需要设置`Global.checkpoints`指向保存的参数文件。评估数据集可以通过 `configs/rec/PP-OCRv3/en_PP-OCRv3_rec.yml` 修改Eval中的 `label_file_path` 设置。 - - -``` -# GPU 评估, Global.checkpoints 为待测权重 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml -o Global.checkpoints={path/to/weights}/best_accuracy -``` - -## 3.2. 测试识别效果 - -使用 PaddleOCR 训练好的模型,可以通过以下脚本进行快速预测。 - -默认预测图片存储在 `infer_img` 里,通过 `-o Global.checkpoints` 加载训练好的参数文件: - -根据配置文件中设置的 `save_model_dir` 和 `save_epoch_step` 字段,会有以下几种参数被保存下来: - -``` -output/rec/ -├── best_accuracy.pdopt -├── best_accuracy.pdparams -├── best_accuracy.states -├── config.yml -├── iter_epoch_3.pdopt -├── iter_epoch_3.pdparams -├── iter_epoch_3.states -├── latest.pdopt -├── latest.pdparams -├── latest.states -└── train.log -``` -其中 best_accuracy.* 是评估集上的最优模型;iter_epoch_x.* 是以 `save_epoch_step` 为间隔保存下来的模型;latest.* 是最后一个epoch的模型。 - -``` -# 预测英文结果 -python3 tools/infer_rec.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - -预测图片: - -![](../imgs_words/en/word_1.png) - -得到输入图像的预测结果: - -``` -infer_img: doc/imgs_words/en/word_1.png - result: ('joint', 0.9998967) -``` - -预测使用的配置文件必须与训练一致,如您通过 `python3 tools/train.py -c configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml` 完成了中文模型的训练, -您可以使用如下命令进行中文模型预测。 - -``` -# 预测中文结果 -python3 tools/infer_rec.py -c configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/ch/word_1.jpg -``` - -预测图片: - -![](../imgs_words/ch/word_1.jpg) - -得到输入图像的预测结果: - -``` -infer_img: doc/imgs_words/ch/word_1.jpg - result: ('韩国小馆', 0.997218) -``` - -# 4. 模型导出与预测 - -inference 模型(`paddle.jit.save`保存的模型) -一般是模型训练,把模型结构和模型参数保存在文件中的固化模型,多用于预测部署场景。 -训练过程中保存的模型是checkpoints模型,保存的只有模型的参数,多用于恢复训练等。 -与checkpoints模型相比,inference 模型会额外保存模型的结构信息,在预测部署、加速推理上性能优越,灵活方便,适合于实际系统集成。 - -识别模型转inference模型与检测的方式相同,如下: - -``` -# -c 后面设置训练算法的yml配置文件 -# -o 配置可选参数 -# Global.pretrained_model 参数设置待转换的训练模型地址,不用添加文件后缀 .pdmodel,.pdopt或.pdparams。 -# Global.save_inference_dir参数设置转换的模型将保存的地址。 - -python3 tools/export_model.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml -o Global.pretrained_model=./pretrain_models/en_PP-OCRv4_rec_train/best_accuracy Global.save_inference_dir=./inference/en_PP-OCRv4_rec/ -``` - -**注意:**如果您是在自己的数据集上训练的模型,并且调整了中文字符的字典文件,请注意修改配置文件中的`character_dict_path`为自定义字典文件。 - -转换成功后,在目录下有三个文件: - -``` -inference/en_PP-OCRv4_rec/ - ├── inference.pdiparams # 识别inference模型的参数文件 - ├── inference.pdiparams.info # 识别inference模型的参数信息,可忽略 - └── inference.pdmodel # 识别inference模型的program文件 -``` - -- 自定义模型推理 - - 如果训练时修改了文本的字典,在使用inference模型预测时,需要通过`--rec_char_dict_path`指定使用的字典路径,更多关于推理超参数的配置与解释,请参考:[模型推理超参数解释教程](./inference_args.md)。 - - ``` - python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./your inference model" --rec_image_shape="3, 48, 320" --rec_char_dict_path="your text dict path" - ``` - - -# 5. FAQ - -Q1: 训练模型转inference 模型之后预测效果不一致? - -**A**:此类问题出现较多,问题多是trained model预测时候的预处理、后处理参数和inference model预测的时候的预处理、后处理参数不一致导致的。可以对比训练使用的配置文件中的预处理、后处理和预测时是否存在差异。 diff --git a/doc/doc_ch/reference.md b/doc/doc_ch/reference.md deleted file mode 100644 index 3347447741..0000000000 --- a/doc/doc_ch/reference.md +++ /dev/null @@ -1,125 +0,0 @@ -# 参考文献 - -``` -1. EAST: -@inproceedings{zhou2017east, - title={EAST: an efficient and accurate scene text detector}, - author={Zhou, Xinyu and Yao, Cong and Wen, He and Wang, Yuzhi and Zhou, Shuchang and He, Weiran and Liang, Jiajun}, - booktitle={Proceedings of the IEEE conference on Computer Vision and Pattern Recognition}, - pages={5551--5560}, - year={2017} -} - -2. DB: -@inproceedings{liao2020real, - title={Real-Time Scene Text Detection with Differentiable Binarization.}, - author={Liao, Minghui and Wan, Zhaoyi and Yao, Cong and Chen, Kai and Bai, Xiang}, - booktitle={AAAI}, - pages={11474--11481}, - year={2020} -} - -3. DTRB: -@inproceedings{baek2019wrong, - title={What is wrong with scene text recognition model comparisons? dataset and model analysis}, - author={Baek, Jeonghun and Kim, Geewook and Lee, Junyeop and Park, Sungrae and Han, Dongyoon and Yun, Sangdoo and Oh, Seong Joon and Lee, Hwalsuk}, - booktitle={Proceedings of the IEEE International Conference on Computer Vision}, - pages={4715--4723}, - year={2019} -} - -4. SAST: -@inproceedings{wang2019single, - title={A Single-Shot Arbitrarily-Shaped Text Detector based on Context Attended Multi-Task Learning}, - author={Wang, Pengfei and Zhang, Chengquan and Qi, Fei and Huang, Zuming and En, Mengyi and Han, Junyu and Liu, Jingtuo and Ding, Errui and Shi, Guangming}, - booktitle={Proceedings of the 27th ACM International Conference on Multimedia}, - pages={1277--1285}, - year={2019} -} - -5. SRN: -@inproceedings{yu2020towards, - title={Towards accurate scene text recognition with semantic reasoning networks}, - author={Yu, Deli and Li, Xuan and Zhang, Chengquan and Liu, Tao and Han, Junyu and Liu, Jingtuo and Ding, Errui}, - booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, - pages={12113--12122}, - year={2020} -} - -6. end2end-psl: -@inproceedings{sun2019chinese, - title={Chinese Street View Text: Large-scale Chinese Text Reading with Partially Supervised Learning}, - author={Sun, Yipeng and Liu, Jiaming and Liu, Wei and Han, Junyu and Ding, Errui and Liu, Jingtuo}, - booktitle={Proceedings of the IEEE International Conference on Computer Vision}, - pages={9086--9095}, - year={2019} -} - -7. CRNN: -@article{shi2016end, - title={An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition}, - author={Shi, Baoguang and Bai, Xiang and Yao, Cong}, - journal={IEEE transactions on pattern analysis and machine intelligence}, - volume={39}, - number={11}, - pages={2298--2304}, - year={2016}, - publisher={IEEE} -} - -8. FPGM: -@inproceedings{he2019filter, - title={Filter pruning via geometric median for deep convolutional neural networks acceleration}, - author={He, Yang and Liu, Ping and Wang, Ziwei and Hu, Zhilan and Yang, Yi}, - booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, - pages={4340--4349}, - year={2019} -} - -9. PACT: -@article{choi2018pact, - title={Pact: Parameterized clipping activation for quantized neural networks}, - author={Choi, Jungwook and Wang, Zhuo and Venkataramani, Swagath and Chuang, Pierce I-Jen and Srinivasan, Vijayalakshmi and Gopalakrishnan, Kailash}, - journal={arXiv preprint arXiv:1805.06085}, - year={2018} -} - -10.Rosetta -@inproceedings{borisyuk2018rosetta, - title={Rosetta: Large scale system for text detection and recognition in images}, - author={Borisyuk, Fedor and Gordo, Albert and Sivakumar, Viswanath}, - booktitle={Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining}, - pages={71--79}, - year={2018} -} - -11.STAR-Net -@inproceedings{liu2016star, - title={STAR-Net: A SpaTial Attention Residue Network for Scene Text Recognition.}, - author={Liu, Wei and Chen, Chaofeng and Wong, Kwan-Yee K and Su, Zhizhong and Han, Junyu}, - booktitle={BMVC}, - volume={2}, - pages={7}, - year={2016} -} - -12.RARE -@inproceedings{shi2016robust, - title={Robust scene text recognition with automatic rectification}, - author={Shi, Baoguang and Wang, Xinggang and Lyu, Pengyuan and Yao, Cong and Bai, Xiang}, - booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition}, - pages={4168--4176}, - year={2016} -} - -13.NRTR -@misc{sheng2019nrtr, - title={NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition}, - author={Fenfen Sheng and Zhineng Chen and Bo Xu}, - year={2019}, - eprint={1806.00926}, - archivePrefix={arXiv}, - primaryClass={cs.CV} -} - -``` diff --git a/doc/doc_ch/slice.md b/doc/doc_ch/slice.md deleted file mode 100644 index 4c1b7fab06..0000000000 --- a/doc/doc_ch/slice.md +++ /dev/null @@ -1,21 +0,0 @@ -# 切片操作 - -如果希望运行 PaddleOCR 处理一张非常大的图像或文档,对其进行检测和识别,可以使用切片操作,如下所示: - -```python -ocr_inst = PaddleOCR(**ocr_settings) -results = ocr_inst.ocr(img, det=True, rec=True, slice=slice, cls=False, bin=False, inv=False, alpha_color=False) -``` - -其中, -`slice = {'horizontal_stride': h_stride, 'vertical_stride': v_stride, 'merge_x_thres': x_thres, 'merge_y_thres': y_thres}` - -这里的 `h_stride`、`v_stride`、`x_thres` 和 `y_thres` 是用户可配置的参数,需要手动设置。切片操作符的工作原理是,在大图像上运行一个滑动窗口,创建图像的切片,并在这些切片上运行 OCR 算法。 - -然后将这些切片级别的零散结果合并,生成图像级别的检测和识别结果。水平和垂直步幅不能低于一定限度,因为过低的值会产生太多切片,导致计算结果非常耗时。例如,对于尺寸为 6616x14886 的图像,推荐使用以下参数: - -```python -slice = {'horizontal_stride': 300, 'vertical_stride': 500, 'merge_x_thres': 50, 'merge_y_thres': 35} -``` - -所有边界框接近 `merge_x_thres` 和 `merge_y_thres` 的切片级检测结果将被合并在一起。 diff --git a/doc/doc_ch/table_recognition.md b/doc/doc_ch/table_recognition.md deleted file mode 100644 index 88260466d4..0000000000 --- a/doc/doc_ch/table_recognition.md +++ /dev/null @@ -1,383 +0,0 @@ -# 表格识别 - -本文提供了PaddleOCR表格识别模型的全流程指南,包括数据准备、模型训练、调优、评估、预测,各个阶段的详细说明: - -- [1. 数据准备](#1-数据准备) - - [1.1. 数据集格式](#11-数据集格式) - - [1.2. 数据下载](#12-数据下载) - - [1.3. 数据集生成](#13-数据集生成) - - [1.4 数据标注](#14-数据标注) -- [2. 开始训练](#2-开始训练) - - [2.1. 启动训练](#21-启动训练) - - [2.2. 断点训练](#22-断点训练) - - [2.3. 更换Backbone 训练](#23-更换backbone-训练) - - [2.4. 混合精度训练](#24-混合精度训练) - - [2.5. 分布式训练](#25-分布式训练) - - [2.6. 其他训练环境](#26-其他训练环境) - - [2.7. 模型微调](#27-模型微调) - - [2.7.1 数据选择](#271-数据选择) - - [2.7.2 模型选择](#272-模型选择) - - [2.7.3 训练超参选择](#273-训练超参选择) -- [3. 模型评估与预测](#3-模型评估与预测) - - [3.1. 指标评估](#31-指标评估) - - [3.2. 测试表格结构识别效果](#32-测试表格结构识别效果) -- [4. 模型导出与预测](#4-模型导出与预测) - - [4.1 模型导出](#41-模型导出) - - [4.2 模型预测](#42-模型预测) -- [5. FAQ](#5-faq) - -# 1. 数据准备 - -## 1.1. 数据集格式 - -PaddleOCR 表格识别模型数据集格式如下: -```txt -img_label # 每张图片标注经过json.dumps()之后的字符串 -... -img_label -``` - -每一行的json格式为: -```txt -{ - 'filename': PMC5755158_010_01.png, # 图像名 - 'split': ’train‘, # 图像属于训练集还是验证集 - 'imgid': 0, # 图像的index - 'html': { - 'structure': {'tokens': ['', '', '', ...]}, # 表格的HTML字符串 - 'cells': [ - { - 'tokens': ['P', 'a', 'd', 'd', 'l', 'e', 'P', 'a', 'd', 'd', 'l', 'e'], # 表格中的单个文本 - 'bbox': [x0, y0, x1, y1] # 表格中的单个文本的坐标 - } - ] - } -} -``` - -训练数据的默认存储路径是 `PaddleOCR/train_data`,如果您的磁盘上已有数据集,只需创建软链接至数据集目录: - -``` -# linux and mac os -ln -sf /train_data/dataset -# windows -mklink /d /train_data/dataset -``` - -## 1.2. 数据下载 - -公开数据集下载可参考 [table_datasets](dataset/table_datasets.md)。 - -## 1.3. 数据集生成 - -使用[TableGeneration](https://github.com/WenmuZhou/TableGeneration)可进行扫描表格图像的生成。 - -TableGeneration是一个开源表格数据集生成工具,其通过浏览器渲染的方式对html字符串进行渲染后获得表格图像。部分样张如下: - -|类型|样例| -|---|---| -|简单表格|![](https://raw.githubusercontent.com/WenmuZhou/TableGeneration/main/imgs/simple.jpg)| -|彩色表格|![](https://raw.githubusercontent.com/WenmuZhou/TableGeneration/main/imgs/color.jpg)| - -## 1.4 数据标注 - -数据标注可参考[PPOCRLabel](https://github.com/PFCCLab/PPOCRLabel/blob/main/README_ch.md) - -# 2. 开始训练 - -PaddleOCR提供了训练脚本、评估脚本和预测脚本,本节将以 [SLANet](../../configs/table/SLANet.yml) 模型训练PubTabNet英文数据集为例: - -## 2.1. 启动训练 - -*如果您安装的是cpu版本,请将配置文件中的 `use_gpu` 字段修改为false* - -``` -# GPU训练 支持单卡,多卡训练 -# 训练日志会自动保存为 "{save_model_dir}" 下的train.log - -#单卡训练(训练周期长,不建议) -python3 tools/train.py -c configs/table/SLANet.yml - -#多卡训练,通过--gpus参数指定卡号 -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/table/SLANet.yml -``` - -正常启动训练后,会看到以下log输出: - -``` -[2022/08/16 03:07:33] ppocr INFO: epoch: [1/400], global_step: 20, lr: 0.000100, acc: 0.000000, loss: 3.915012, structure_loss: 3.229450, loc_loss: 0.670590, avg_reader_cost: 2.63382 s, avg_batch_cost: 6.32390 s, avg_samples: 48.0, ips: 7.59025 samples/s, eta: 9 days, 2:29:27 -[2022/08/16 03:08:41] ppocr INFO: epoch: [1/400], global_step: 40, lr: 0.000100, acc: 0.000000, loss: 1.750859, structure_loss: 1.082116, loc_loss: 0.652822, avg_reader_cost: 0.02533 s, avg_batch_cost: 3.37251 s, avg_samples: 48.0, ips: 14.23271 samples/s, eta: 6 days, 23:28:43 -[2022/08/16 03:09:46] ppocr INFO: epoch: [1/400], global_step: 60, lr: 0.000100, acc: 0.000000, loss: 1.395154, structure_loss: 0.776803, loc_loss: 0.625030, avg_reader_cost: 0.02550 s, avg_batch_cost: 3.26261 s, avg_samples: 48.0, ips: 14.71214 samples/s, eta: 6 days, 5:11:48 -``` - -log 中自动打印如下信息: - -| 字段 | 含义 | -| :----: | :------: | -| epoch | 当前迭代轮次 | -| global_step | 当前迭代次数 | -| lr | 当前学习率 | -| acc | 当前batch的准确率 | -| loss | 当前损失函数 | -| structure_loss | 表格结构损失值 | -| loc_loss | 单元格坐标损失值 | -| avg_reader_cost | 当前 batch 数据处理耗时 | -| avg_batch_cost | 当前 batch 总耗时 | -| avg_samples | 当前 batch 内的样本数 | -| ips | 每秒处理图片的数量 | - - -PaddleOCR支持训练和评估交替进行, 可以在 `configs/table/SLANet.yml` 中修改 `eval_batch_step` 设置评估频率,默认每1000个iter评估一次。评估过程中默认将最佳acc模型,保存为 `output/SLANet/best_accuracy` 。 - -如果验证集很大,测试将会比较耗时,建议减少评估次数,或训练完再进行评估。 - -**提示:** 可通过 -c 参数选择 `configs/table/` 路径下的多种模型配置进行训练,PaddleOCR支持的表格识别算法可以参考[前沿算法列表](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/doc/doc_ch/algorithm_overview.md#3-%E8%A1%A8%E6%A0%BC%E8%AF%86%E5%88%AB%E7%AE%97%E6%B3%95): - -**注意,预测/评估时的配置文件请务必与训练一致。** - -## 2.2. 断点训练 - -如果训练程序中断,如果希望加载训练中断的模型从而恢复训练,可以通过指定Global.checkpoints指定要加载的模型路径: -```shell -python3 tools/train.py -c configs/table/SLANet.yml -o Global.checkpoints=./your/trained/model -``` - -**注意**:`Global.checkpoints`的优先级高于`Global.pretrained_model`的优先级,即同时指定两个参数时,优先加载`Global.checkpoints`指定的模型,如果`Global.checkpoints`指定的模型路径有误,会加载`Global.pretrained_model`指定的模型。 - -## 2.3. 更换Backbone 训练 - -PaddleOCR将网络划分为四部分,分别在[ppocr/modeling](../../ppocr/modeling)下。 进入网络的数据将按照顺序(transforms->backbones->necks->heads)依次通过这四个部分。 - -```bash -├── architectures # 网络的组网代码 -├── transforms # 网络的图像变换模块 -├── backbones # 网络的特征提取模块 -├── necks # 网络的特征增强模块 -└── heads # 网络的输出模块 -``` -如果要更换的Backbone 在PaddleOCR中有对应实现,直接修改配置yml文件中`Backbone`部分的参数即可。 - -如果要使用新的Backbone,更换backbones的例子如下: - -1. 在 [ppocr/modeling/backbones](../../ppocr/modeling/backbones) 文件夹下新建文件,如my_backbone.py。 -2. 在 my_backbone.py 文件内添加相关代码,示例代码如下: - -```python -import paddle -import paddle.nn as nn -import paddle.nn.functional as F - - -class MyBackbone(nn.Layer): - def __init__(self, *args, **kwargs): - super(MyBackbone, self).__init__() - # your init code - self.conv = nn.xxxx - - def forward(self, inputs): - # your network forward - y = self.conv(inputs) - return y -``` - -3. 在 [ppocr/modeling/backbones/\__init\__.py](../../ppocr/modeling/backbones/__init__.py)文件内导入添加的`MyBackbone`模块,然后修改配置文件中Backbone进行配置即可使用,格式如下: - -```yaml -Backbone: -name: MyBackbone -args1: args1 -``` - -**注意**:如果要更换网络的其他模块,可以参考[文档](./add_new_algorithm.md)。 - -## 2.4. 混合精度训练 - -如果您想进一步加快训练速度,可以使用[自动混合精度训练](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/01_paddle2.0_introduction/basic_concept/amp_cn.html), 以单机单卡为例,命令如下: - -```shell -python3 tools/train.py -c configs/table/SLANet.yml \ - -o Global.pretrained_model=./pretrain_models/SLANet/best_accuracy \ - Global.use_amp=True Global.scale_loss=1024.0 Global.use_dynamic_loss_scaling=True - ``` - -## 2.5. 分布式训练 - -多机多卡训练时,通过 `--ips` 参数设置使用的机器IP地址,通过 `--gpus` 参数设置使用的GPU ID: - -```bash -python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/table/SLANet.yml \ - -o Global.pretrained_model=./pretrain_models/SLANet/best_accuracy -``` - -**注意:** (1)采用多机多卡训练时,需要替换上面命令中的ips值为您机器的地址,机器之间需要能够相互ping通;(2)训练时需要在多个机器上分别启动命令。查看机器ip地址的命令为`ifconfig`;(3)更多关于分布式训练的性能优势等信息,请参考:[分布式训练教程](./distributed_training.md)。 - - -## 2.6. 其他训练环境 - -- Windows GPU/CPU -在Windows平台上与Linux平台略有不同: -Windows平台只支持`单卡`的训练与预测,指定GPU进行训练`set CUDA_VISIBLE_DEVICES=0` -在Windows平台,DataLoader只支持单进程模式,因此需要设置 `num_workers` 为0; - -- macOS -不支持GPU模式,需要在配置文件中设置`use_gpu`为False,其余训练评估预测命令与Linux GPU完全相同。 - -- Linux DCU -DCU设备上运行需要设置环境变量 `export HIP_VISIBLE_DEVICES=0,1,2,3`,其余训练评估预测命令与Linux GPU完全相同。 - -## 2.7. 模型微调 - -### 2.7.1 数据选择 - -数据量:建议至少准备2000张的表格识别数据集用于模型微调。 - -### 2.7.2 模型选择 - -建议选择SLANet模型(配置文件:[SLANet_ch.yml](../../configs/table/SLANet_ch.yml),预训练模型:[ch_ppstructure_mobile_v2.0_SLANet_train.tar](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_train.tar))进行微调,其精度与泛化性能是目前提供的最优中文表格预训练模型。 - -更多表格识别模型,请参考[PP-Structure 系列模型库](../../ppstructure/docs/models_list.md)。 - -### 2.7.3 训练超参选择 - -在模型微调的时候,最重要的超参就是预训练模型路径`pretrained_model`, 学习率`learning_rate`,部分配置文件如下所示。 - -```yaml -Global: - pretrained_model: ./ch_ppstructure_mobile_v2.0_SLANet_train/best_accuracy.pdparams # 预训练模型路径 -Optimizer: - lr: - name: Cosine - learning_rate: 0.001 # - warmup_epoch: 0 - regularizer: - name: 'L2' - factor: 0 -``` - -上述配置文件中,首先需要将`pretrained_model`字段指定为`best_accuracy.pdparams`文件路径。 - -PaddleOCR提供的配置文件是在4卡训练(相当于总的batch size是`4*48=192`)、且没有加载预训练模型情况下的配置文件,因此您的场景中,学习率与总的batch size需要对应线性调整,例如 - -* 如果您的场景中是单卡训练,单卡batch_size=48,则总的batch_size=48,建议将学习率调整为`0.00025`左右。 -* 如果您的场景中是单卡训练,由于显存限制,只能设置单卡batch_size=32,则总的batch_size=32,建议将学习率调整为`0.00017`左右。 - - -# 3. 模型评估与预测 - -## 3.1. 指标评估 - -训练中模型参数默认保存在`Global.save_model_dir`目录下。在评估指标时,需要设置`Global.checkpoints`指向保存的参数文件。评估数据集可以通过 `configs/table/SLANet.yml` 修改Eval中的 `label_file_list` 设置。 - - -``` -# GPU 评估, Global.checkpoints 为待测权重 -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/table/SLANet.yml -o Global.checkpoints={path/to/weights}/best_accuracy -``` - -运行完成后,会输出模型的acc指标,如对英文表格识别模型进行评估,会见到如下输出。 -```bash -[2022/08/16 07:59:55] ppocr INFO: acc:0.7622245132160782 -[2022/08/16 07:59:55] ppocr INFO: fps:30.991640622573044 -``` - -## 3.2. 测试表格结构识别效果 - -使用 PaddleOCR 训练好的模型,可以通过以下脚本进行快速预测。 - -默认预测图片存储在 `infer_img` 里,通过 `-o Global.checkpoints` 加载训练好的参数文件: - -根据配置文件中设置的 `save_model_dir` 和 `save_epoch_step` 字段,会有以下几种参数被保存下来: - -``` -output/SLANet/ -├── best_accuracy.pdopt -├── best_accuracy.pdparams -├── best_accuracy.states -├── config.yml -├── latest.pdopt -├── latest.pdparams -├── latest.states -└── train.log -``` -其中 best_accuracy.* 是评估集上的最优模型;latest.* 是最后一个epoch的模型。 - -``` -# 预测表格图像 -python3 tools/infer_table.py -c configs/table/SLANet.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=ppstructure/docs/table/table.jpg -``` - -预测图片: - -![](../../ppstructure/docs/table/table.jpg) - -得到输入图像的预测结果: - -``` -['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '
', '', ''],[[320.0562438964844, 197.83375549316406, 350.0928955078125, 214.4309539794922], ... , [318.959228515625, 271.0166931152344, 353.7394104003906, 286.4538269042969]] -``` - -单元格坐标可视化结果为 - -![](../../ppstructure/docs/imgs/slanet_result.jpg) - -# 4. 模型导出与预测 - -## 4.1 模型导出 - -inference 模型(`paddle.jit.save`保存的模型) -一般是模型训练,把模型结构和模型参数保存在文件中的固化模型,多用于预测部署场景。 -训练过程中保存的模型是checkpoints模型,保存的只有模型的参数,多用于恢复训练等。 -与checkpoints模型相比,inference 模型会额外保存模型的结构信息,在预测部署、加速推理上性能优越,灵活方便,适合于实际系统集成。 - -表格识别模型转inference模型与文字检测识别的方式相同,如下: - -``` -# -c 后面设置训练算法的yml配置文件 -# -o 配置可选参数 -# Global.pretrained_model 参数设置待转换的训练模型地址,不用添加文件后缀 .pdmodel,.pdopt或.pdparams。 -# Global.save_inference_dir参数设置转换的模型将保存的地址。 - -python3 tools/export_model.py -c configs/table/SLANet.yml -o Global.pretrained_model=./pretrain_models/SLANet/best_accuracy Global.save_inference_dir=./inference/SLANet/ -``` - -转换成功后,在目录下有三个文件: - -``` -inference/SLANet/ - ├── inference.pdiparams # inference模型的参数文件 - ├── inference.pdiparams.info # inference模型的参数信息,可忽略 - └── inference.pdmodel # inference模型的program文件 -``` - -## 4.2 模型预测 - -模型导出后,使用如下命令即可完成inference模型的预测 - -```python -python3.7 table/predict_structure.py \ - --table_model_dir={path/to/inference model} \ - --table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \ - --image_dir=docs/table/table.jpg \ - --output=../output/table -``` - -预测图片: - -![](../../ppstructure/docs/table/table.jpg) - -得到输入图像的预测结果: - -``` -['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '
', '', ''],[[320.0562438964844, 197.83375549316406, 350.0928955078125, 214.4309539794922], ... , [318.959228515625, 271.0166931152344, 353.7394104003906, 286.4538269042969]] -``` - -单元格坐标可视化结果为 - -![](../../ppstructure/docs/imgs/slanet_result.jpg) - - -# 5. FAQ - -Q1: 训练模型转inference 模型之后预测效果不一致? - -**A**:此类问题出现较多,问题多是trained model预测时候的预处理、后处理参数和inference model预测的时候的预处理、后处理参数不一致导致的。可以对比训练使用的配置文件中的预处理、后处理和预测时是否存在差异。 diff --git a/doc/doc_ch/thirdparty.md b/doc/doc_ch/thirdparty.md deleted file mode 100644 index ff9059cdf6..0000000000 --- a/doc/doc_ch/thirdparty.md +++ /dev/null @@ -1,143 +0,0 @@ -# 社区贡献 - -感谢大家长久以来对PaddleOCR的支持和关注,与广大开发者共同构建一个专业、和谐、相互帮助的开源社区是PaddleOCR的目标。本文档展示了已有的社区贡献、对于各类贡献说明、新的机会与流程,希望贡献流程更加高效、路径更加清晰。 - -PaddleOCR希望可以通过AI的力量助力任何一位有梦想的开发者实现自己的想法,享受创造价值带来的愉悦。 - ---- - - - - - -> 上图为PaddleOCR目前的Contributor,定期更新 - -## 1. 社区贡献 - -### 1.1 基于PaddleOCR的社区项目 - -| 类别 | 项目 | 描述 | 开发者 | -| -------- | ------------------------------------------------------------ | -------------------------- | ------------------------------------------------------------ | -| 通用工具 | [FastOCRLabel](https://gitee.com/BaoJianQiang/FastOCRLabel) | 完整的C#版本标注GUI | [包建强](https://gitee.com/BaoJianQiang) | -| 通用工具 | [DangoOCR离线版](https://github.com/PantsuDango/DangoOCR) | 通用型桌面级即时翻译GUI | [PantsuDango](https://github.com/PantsuDango) | -| 通用工具 | [scr2txt](https://github.com/lstwzd/scr2txt) | 截屏转文字GUI | [lstwzd](https://github.com/lstwzd) | -| 通用工具 | [ocr_sdk](https://github.com/mymagicpower/AIAS/blob/main/1_image_sdks/text_recognition/ocr_sdk) | OCR java SDK工具箱 | [Calvin](https://github.com/mymagicpower) | -| 通用工具 | [iocr](https://github.com/mymagicpower/AIAS/blob/main/8_suite_hub/iocr) | IOCR 自定义模板识别(支持表格识别) | [Calvin](https://github.com/mymagicpower) | -| 通用工具 | [Lmdb Dataset Format Conversion Tool](https://github.com/OneYearIsEnough/PaddleOCR-Recog-LmdbDataset-Conversion) | 文本识别任务中lmdb数据格式转换工具 | [OneYearIsEnough](https://github.com/OneYearIsEnough) | -| 通用工具 | [用paddleocr打造一款“盗幕笔记”](https://github.com/kjf4096/paddleocr_dmbj) | 用PaddleOCR记笔记 | [kjf4096](https://github.com/kjf4096) | -| 垂类工具 | [AI Studio项目](https://aistudio.baidu.com/aistudio/projectdetail/1054614?channelType=0&channel=0) | 英文视频自动生成字幕 | [叶月水狐](https://aistudio.baidu.com/aistudio/personalcenter/thirdview/322052) | -| 垂类工具 | [id_card_ocr](https://github.com/baseli/id_card_ocr) | 身份证复印件识别 | [baseli](https://github.com/baseli) | -| 垂类工具 | [Paddle_Table_Image_Reader](https://github.com/thunder95/Paddle_Table_Image_Reader) | 能看懂表格图片的数据助手 | [thunder95](https://github.com/thunder95]) | -| 垂类工具 | [AI Studio项目](https://aistudio.baidu.com/aistudio/projectdetail/3382897) | OCR流程中对手写体进行过滤 | [daassh](https://github.com/daassh) | -| 垂类场景调优 | [AI Studio项目](https://aistudio.baidu.com/aistudio/projectdetail/2803693) | 电表读数和编号识别 | [深渊上的坑](https://github.com/edencfc) | -| 垂类场景调优 | [AI Studio项目](https://aistudio.baidu.com/aistudio/projectdetail/3284199) | LCD液晶字符检测 | [Dream拒杰](https://github.com/zhangyingying520) | -| 前后处理 | [paddleOCRCorrectOutputs](https://github.com/yuranusduke/paddleOCRCorrectOutputs) | 获取OCR识别结果的key-value | [yuranusduke](https://github.com/yuranusduke) | -|前处理| [optlab](https://github.com/GreatV/optlab) |OCR前处理工具箱,基于Qt和Leptonica。|[GreatV](https://github.com/GreatV)| -|应用部署| [PaddleOCRSharp](https://github.com/raoyutian/PaddleOCRSharp) |PaddleOCR的.NET封装与应用部署。|[raoyutian](https://github.com/raoyutian/PaddleOCRSharp)| -|应用部署| [PaddleSharp](https://github.com/sdcb/PaddleSharp) |PaddleOCR的.NET封装与应用部署,支持跨平台、GPU|[sdcb](https://github.com/sdcb)| -| 应用部署 | [PaddleOCR-Streamlit-Demo](https://github.com/Lovely-Pig/PaddleOCR-Streamlit-Demo) | 使用Streamlit部署PaddleOCR | [Lovely-Pig](https://github.com/Lovely-Pig) | -| 应用部署 | [PaddleOCR-PyWebIO-Demo](https://github.com/Lovely-Pig/PaddleOCR-PyWebIO-Demo) | 使用PyWebIO部署PaddleOCR | [Lovely-Pig](https://github.com/Lovely-Pig) | -| 应用部署 | [PaddleOCR-Paddlejs-Vue-Demo](https://github.com/Lovely-Pig/PaddleOCR-Paddlejs-Vue-Demo) | 使用Paddle.js和Vue部署PaddleOCR | [Lovely-Pig](https://github.com/Lovely-Pig) | -| 应用部署 | [PaddleOCR-Paddlejs-React-Demo](https://github.com/Lovely-Pig/PaddleOCR-Paddlejs-React-Demo) | 使用Paddle.js和React部署PaddleOCR | [Lovely-Pig](https://github.com/Lovely-Pig) | -| 学术前沿模型训练与推理 | [AI Studio项目](https://aistudio.baidu.com/aistudio/projectdetail/3397137) | StarNet-MobileNetV3算法–中文训练 | [xiaoyangyang2](https://github.com/xiaoyangyang2) | -| 学术前沿模型训练与推理 | [ABINet-paddle](https://github.com/Huntersdeng/abinet-paddle) | ABINet算法前向运算的paddle实现以及模型各部分的实现细节分析 | [Huntersdeng](https://github.com/Huntersdeng) | - -### 1.2 为PaddleOCR新增功能 - -- 非常感谢 [authorfu](https://github.com/authorfu) 贡献Android([#340](https://github.com/PaddlePaddle/PaddleOCR/pull/340))和[xiadeye](https://github.com/xiadeye) 贡献IOS的demo代码([#325](https://github.com/PaddlePaddle/PaddleOCR/pull/325)) -- 非常感谢 [tangmq](https://gitee.com/tangmq) 给PaddleOCR增加Docker化部署服务,支持快速发布可调用的Restful API服务([#507](https://github.com/PaddlePaddle/PaddleOCR/pull/507))。 -- 非常感谢 [lijinhan](https://github.com/lijinhan) 给PaddleOCR增加java SpringBoot 调用OCR Hubserving接口完成对OCR服务化部署的使用([#1027](https://github.com/PaddlePaddle/PaddleOCR/pull/1027))。 -- 非常感谢 [Evezerest](https://github.com/Evezerest), [ninetailskim](https://github.com/ninetailskim), [edencfc](https://github.com/edencfc), [BeyondYourself](https://github.com/BeyondYourself), [1084667371](https://github.com/1084667371) 贡献了[PPOCRLabel](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/PPOCRLabel/README_ch.md) 的完整代码。 -- 非常感谢 [bupt906](https://github.com/bupt906) 贡献MicroNet结构代码([#5251](https://github.com/PaddlePaddle/PaddleOCR/pull/5251))和贡献OneCycle学习率策略代码([#5252](https://github.com/PaddlePaddle/PaddleOCR/pull/5252)) - -### 1.3 代码修复 - -- 非常感谢 [zhangxin](https://github.com/ZhangXinNan)([Blog](https://blog.csdn.net/sdlypyzq)) 贡献新的可视化方式、添加.gitgnore、处理手动设置PYTHONPATH环境变量的问题([#210](https://github.com/PaddlePaddle/PaddleOCR/pull/210))。 -- 非常感谢 [lyl120117](https://github.com/lyl120117) 贡献打印网络结构的代码([#304](https://github.com/PaddlePaddle/PaddleOCR/pull/304))。 -- 非常感谢 [BeyondYourself](https://github.com/BeyondYourself) 给PaddleOCR提了很多非常棒的建议,并简化了PaddleOCR的部分代码风格([so many commits)](https://github.com/PaddlePaddle/PaddleOCR/commits?author=BeyondYourself)。 - -### 1.4 文档优化与翻译 - -- 非常感谢 **[RangeKing](https://github.com/RangeKing),[HustBestCat](https://github.com/HustBestCat),[v3fc](https://github.com/v3fc),[1084667371](https://github.com/1084667371)** 贡献翻译《动手学OCR》notebook[电子书英文版](https://github.com/PaddlePaddle/PaddleOCR/tree/dygraph/notebook/notebook_en)。 -- 非常感谢 [thunderstudying](https://github.com/thunderstudying),[RangeKing](https://github.com/RangeKing),[livingbody](https://github.com/livingbody), [WZMIAOMIAO](https://github.com/WZMIAOMIAO),[haigang1975](https://github.com/haigang1975) 补充多个英文markdown文档。 -- 非常感谢 **[fanruinet](https://github.com/fanruinet)** 润色和修复35篇英文文档([#5205](https://github.com/PaddlePaddle/PaddleOCR/pull/5205))。 -- 非常感谢 [Khanh Tran](https://github.com/xxxpsyduck) 和 [Karl Horky](https://github.com/karlhorky) 贡献修改英文文档。 - -### 1.5 多语言语料 - -- 非常感谢 [xiangyubo](https://github.com/xiangyubo) 贡献手写中文OCR数据集([#321](https://github.com/PaddlePaddle/PaddleOCR/pull/321))。 -- 非常感谢 [Mejans](https://github.com/Mejans) 给PaddleOCR增加新语言奥克西坦语Occitan的字典和语料([#954](https://github.com/PaddlePaddle/PaddleOCR/pull/954))。 - -## 2. 贡献说明 - -### 2.1 新增功能类 - -PaddleOCR非常欢迎社区贡献以PaddleOCR为核心的各种服务、部署实例与软件应用,经过认证的社区贡献会被添加在上述社区贡献表中,为广大开发者增加曝光,也是PaddleOCR的荣耀,其中: - -- 项目形式:官方社区认证的项目代码应有良好的规范和结构,同时,还应配备一个详细的README.md,说明项目的使用方法。通过在requirements.txt文件中增加一行 `paddleocr` 可以自动收录到PaddleOCR的usedby中。 - -- 合入方式:如果是对PaddleOCR现有工具的更新升级,则会合入主repo。如果为PaddleOCR拓展了新功能,请先与官方人员联系,确认项目是否合入主repo,*即使新功能未合入主repo,我们同样也会以社区贡献的方式为您的个人项目增加曝光。* - - -### 2.2 代码优化 - -如果您在使用PaddleOCR时遇到了代码bug、功能不符合预期等问题,可以为PaddleOCR贡献您的修改,其中: - -- Python代码规范可参考[附录1:Python代码规范](./code_and_doc.md/#附录1)。 - -- 提交代码前请再三确认不会引入新的bug,并在PR中描述优化点。如果该PR解决了某个issue,请在PR中连接到该issue。所有的PR都应该遵守附录3中的[3.2.10 提交代码的一些约定。](./code_and_doc.md/#提交代码的一些约定) - -- 请在提交之前参考下方的[附录3:Pull Request说明](./code_and_doc.md#附录3)。如果您对git的提交流程不熟悉,同样可以参考附录3的3.2节。 - -**最后请在PR的题目中加上标签`【third-party】` , 在说明中@Evezerest,拥有此标签的PR将会被高优处理**。 - -### 2.3 文档优化 - -如果您在使用PaddleOCR时遇到了文档表述不清楚、描述缺失、链接失效等问题,可以为PaddleOCR贡献您的修改。文档书写规范请参考[附录2:文档规范](./code_and_doc.md/#附录2)。**最后请在PR的题目中加上标签`【third-party】` , 在说明中@Evezerest,拥有此标签的PR将会被高优处理。** - -## 3. 更多贡献机会 - -我们非常鼓励开发者使用PaddleOCR实现自己的想法,同时我们也列出一些经过分析后认为有价值的拓展方向,整体收集在社区项目常规赛中。 - -## 4. 联系我们 - -我们非常欢迎广大开发者在有意向为PaddleOCR贡献代码、文档、语料等内容前与我们联系,这样可以大大降低PR过程中的沟通成本。同时,如果您觉得某些想法个人难以实现,我们也可以通过SIG的形式定向为项目招募志同道合的开发者一起共建。通过SIG渠道贡献的项目将会获得深层次的研发支持与运营资源(如公众号宣传、直播课等)。 - -我们推荐的贡献流程是: - -- 通过在github issue的题目中增加 `【third-party】` 标记,说明遇到的问题(以及解决的思路)或想拓展的功能,等待值班人员回复。例如 `【third-party】为PaddleOCR贡献IOS示例` -- 与我们沟通确认技术方案或bug、优化点准确无误后进行功能新增或相应的修改,代码与文档遵循相关规范。 -- PR链接到上述issue,等待review。 - -## 5. 致谢与后续 - - - 合入代码之后会在本文档第一节中更新信息,默认链接为github名字及主页,如果有需要更换主页,也可以联系我们。 - - 新增重要功能类,会在用户群广而告之,享受开源社区荣誉时刻。 - - **如果您有基于PaddleOCR的项目,但未出现在上述列表中,请按照 `4. 联系我们` 的步骤与我们联系。** - - - -## 附录:社区常规赛积分榜 - -| 开发者 | 总积分 | 开发者 | 总积分 | -| ------------------------------------------------------- | ------ | ----------------------------------------------------- | ------ | -| [RangeKing](https://github.com/RangeKing) | 220 | [WZMIAOMIAO](https://github.com/WZMIAOMIAO) | 36 | -| [hao6699](https://github.com/hao6699) | 145 | [v3fc](https://github.com/v3fc) | 35 | -| [mymagicpower](https://github.com/mymagicpower) | 140 | [imiyu](https://github.com/imiyu) | 30 | -| [raoyutian](https://github.com/raoyutian) | 90 | [haigang1975](https://github.com/haigang1975) | 29 | -| [sdcb](https://github.com/sdcb) | 80 | [daassh](https://github.com/daassh) | 23 | -| [zhiminzhang0830](https://github.com/zhiminzhang0830) | 70 | [xiaoyangyang2](https://github.com/xiaoyangyang2) | 20 | -| [Lovely-Pig](https://github.com/Lovely-Pig) | 70 | [prettyocean85](https://github.com/prettyocean85) | 20 | -| [livingbody](https://github.com/livingbody) | 70 | [nmusik](https://github.com/nmusik) | 20 | -| [fanruinet](https://github.com/fanruinet) | 70 | [kjf4096](https://github.com/kjf4096) | 20 | -| [bupt906](https://github.com/bupt906) | 60 | [chccc1994](https://github.com/chccc1994) | 20 | -| [edencfc](https://github.com/edencfc) | 57 | [BeyondYourself ](https://github.com/BeyondYourself) | 20 | -| [zhangyingying520](https://github.com/zhangyingying520) | 57 | chenguoqi08161 | 18 | -| [ITerydh](https://github.com/ITerydh) | 55 | [weiwenlan](https://github.com/weiwenlan) | 10 | -| [telppa](https://github.com/telppa) | 40 | [shaoshenchen thinc](https://github.com/shaoshenchen) | 10 | -| sosojust1984 | 40 | [jordan2013](https://github.com/jordan2013) | 10 | -| [redearly123](https://github.com/redearly123) | 40 | [JimEverest](https://github.com/JimEverest) | 10 | -| [OneYearIsEnough](https://github.com/OneYearIsEnough) | 40 | [HustBestCat](https://github.com/HustBestCat) | 10 | -| [Huntersdeng](https://github.com/Huntersdeng) | 40 | | | -| [GreatV](https://github.com/GreatV) | 40 | | | -| CLXK294 | 40 | | | diff --git a/doc/doc_ch/training.md b/doc/doc_ch/training.md deleted file mode 100644 index 37c2ec26ed..0000000000 --- a/doc/doc_ch/training.md +++ /dev/null @@ -1,151 +0,0 @@ -# PP-OCR模型训练 - -本文将介绍模型训练时需掌握的基本概念,和训练时的调优方法。 - -同时会简单介绍PaddleOCR模型训练数据的组成部分,以及如何在垂类场景中准备数据finetune模型。 - -- [1. 配置文件说明](#1-配置文件说明) -- [2. 基本概念](#2-基本概念) - - [2.1 学习率](#21-学习率) - - [2.2 正则化](#22-正则化) - - [2.3 评估指标](#23-评估指标) -- [3. 数据与垂类场景](#3-数据与垂类场景) - - [3.1 训练数据](#31-训练数据) - - [3.2 垂类场景](#32-垂类场景) - - [3.3 自己构建数据集](#33-自己构建数据集) -- [4. 常见问题](#4-常见问题) - - - -## 1. 配置文件说明 - -PaddleOCR模型使用配置文件管理网络训练、评估的参数。在配置文件中,可以设置组建模型、优化器、损失函数、模型前后处理的参数,PaddleOCR从配置文件中读取到这些参数,进而组建出完整的训练流程,完成模型训练,在需要对模型进行优化的时,可以通过修改配置文件中的参数完成配置,使用简单且方便修改。 - -完整的配置文件说明可以参考[配置文件](./config.md) - - - -## 2. 基本概念 - -模型训练过程中需要手动调整一些超参数,帮助模型以最小的代价获得最优指标。不同的数据量可能需要不同的超参,当您希望在自己的数据上finetune或对模型效果调优时,有以下几个参数调整策略可供参考: - - -### 2.1 学习率 - -学习率是训练神经网络的重要超参数之一,它代表在每一次迭代中梯度向损失函数最优解移动的步长。 -在PaddleOCR中提供了多种学习率更新策略,可以通过配置文件修改,例如: - -``` -Optimizer: - ... - lr: - name: Piecewise - decay_epochs : [700, 800] - values : [0.001, 0.0001] - warmup_epoch: 5 -``` - -Piecewise 代表分段常数衰减,在不同的学习阶段指定不同的学习率,在每段内学习率相同。 -warmup_epoch 代表在前5个epoch中,学习率将逐渐从0增加到base_lr。全部策略可以参考代码[learning_rate.py](../../ppocr/optimizer/learning_rate.py) 。 - - -### 2.2 正则化 - -正则化可以有效的避免算法过拟合,PaddleOCR中提供了L1、L2正则方法,L1 和 L2 正则化是最常用的正则化方法。L1 正则化向目标函数添加正则化项,以减少参数的绝对值总和;而 L2 正则化中,添加正则化项的目的在于减少参数平方的总和。配置方法如下: - -``` -Optimizer: - ... - regularizer: - name: L2 - factor: 2.0e-05 -``` - - -### 2.3 评估指标 - -(1)检测阶段:先按照检测框和标注框的IOU评估,IOU大于某个阈值判断为检测准确。这里检测框和标注框不同于一般的通用目标检测框,是采用多边形进行表示。检测准确率:正确的检测框个数在全部检测框的占比,主要是判断检测指标。检测召回率:正确的检测框个数在全部标注框的占比,主要是判断漏检的指标。 - -(2)识别阶段: 字符识别准确率,即正确识别的文本行占标注的文本行数量的比例,只有整行文本识别对才算正确识别。 - -(3)端到端统计: 端对端召回率:准确检测并正确识别文本行在全部标注文本行的占比; 端到端准确率:准确检测并正确识别文本行在 检测到的文本行数量 的占比; 准确检测的标准是检测框与标注框的IOU大于某个阈值,正确识别的检测框中的文本与标注的文本相同。 - - - -## 3. 数据与垂类场景 - - -### 3.1 训练数据 -目前开源的模型,数据集和量级如下: - - - 检测: - - 英文数据集,ICDAR2015 - - 中文数据集,LSVT街景数据集训练数据3w张图片 - - - 识别: - - 英文数据集,MJSynth和SynthText合成数据,数据量上千万。 - - 中文数据集,LSVT街景数据集根据真值将图crop出来,并进行位置校准,总共30w张图像。此外基于LSVT的语料,合成数据500w。 - - 小语种数据集,使用不同语料和字体,分别生成了100w合成数据集,并使用ICDAR-MLT作为验证集。 - -其中,公开数据集都是开源的,用户可自行搜索下载,也可参考[中文数据集](dataset/datasets.md),合成数据暂不开源,用户可使用开源合成工具自行合成,可参考的合成工具包括[text_renderer](https://github.com/Sanster/text_renderer) 、[SynthText](https://github.com/ankush-me/SynthText) 、[TextRecognitionDataGenerator](https://github.com/Belval/TextRecognitionDataGenerator) 等。 - - -### 3.2 垂类场景 - -PaddleOCR主要聚焦通用OCR,如果有垂类需求,您可以用PaddleOCR+垂类数据自己训练; -如果缺少带标注的数据,或者不想投入研发成本,建议直接调用开放的API,开放的API覆盖了目前比较常见的一些垂类。 - - - -### 3.3 自己构建数据集 - -在构建数据集时有几个经验可供参考: - -(1) 训练集的数据量: - - a. 检测需要的数据相对较少,在PaddleOCR模型的基础上进行Fine-tune,一般需要500张可达到不错的效果。 - b. 识别分英文和中文,一般英文场景需要几十万数据可达到不错的效果,中文则需要几百万甚至更多。 - - -(2)当训练数据量少时,可以尝试以下三种方式获取更多的数据: - - a. 人工采集更多的训练数据,最直接也是最有效的方式。 - b. 基于PIL和opencv基本图像处理或者变换。例如PIL中ImageFont, Image, ImageDraw三个模块将文字写到背景中,opencv的旋转仿射变换,高斯滤波等。 - c. 利用数据生成算法合成数据,例如pix2pix或[StyleText](https://github.com/PFCCLab/StyleText)等算法。 - - - -## 4. 常见问题 - -**Q**:训练CRNN识别时,如何选择合适的网络输入shape? - - A:一般高度采用32,最长宽度的选择,有两种方法: - - (1)统计训练样本图像的宽高比分布。最大宽高比的选取考虑满足80%的训练样本。 - - (2)统计训练样本文字数目。最长字符数目的选取考虑满足80%的训练样本。然后中文字符长宽比近似认为是1,英文认为3:1,预估一个最长宽度。 - -**Q**:识别训练时,训练集精度已经到达90了,但验证集精度一直在70,涨不上去怎么办? - - A:训练集精度90,测试集70多的话,应该是过拟合了,有两个可尝试的方法: - - (1)加入更多的增广方式或者调大增广prob的[概率](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppocr/data/imaug/rec_img_aug.py#L341),默认为0.4。 - - (2)调大系统的[l2 dcay值](https://github.com/PaddlePaddle/PaddleOCR/blob/a501603d54ff5513fc4fc760319472e59da25424/configs/rec/ch_ppocr_v1.1/rec_chinese_lite_train_v1.1.yml#L47) - -**Q**: 识别模型训练时,loss能正常下降,但acc一直为0 - - A:识别模型训练初期acc为0是正常的,多训一段时间指标就上来了。 - - - -*** - -具体的训练教程可点击下方链接跳转: - -- [文本检测模型训练](./detection.md) - -- [文本识别模型训练](./recognition.md) - -- [文本方向分类器训练](./angle_class.md) -- [知识蒸馏](./knowledge_distillation.md) diff --git a/doc/doc_ch/tree.md b/doc/doc_ch/tree.md deleted file mode 100644 index cc3530b3df..0000000000 --- a/doc/doc_ch/tree.md +++ /dev/null @@ -1,216 +0,0 @@ -# 整体目录结构 - -PaddleOCR 的整体目录结构介绍如下: - -``` -PaddleOCR -├── configs // 配置文件,可通过 yml 文件选择模型结构并修改超参 -│ ├── cls // 方向分类器相关配置文件 -│ │ ├── cls_mv3.yml // 训练配置相关,包括骨干网络、head、loss、优化器和数据 -│ ├── det // 检测相关配置文件 -│ │ ├── det_mv3_db.yml // 训练配置 -│ │ ... -│ └── rec // 识别相关配置文件 -│ ├── rec_mv3_none_bilstm_ctc.yml // crnn 训练配置 -│ ... -├── deploy // 部署相关 -│ ├── android_demo // android_demo -│ │ ... -│ ├── cpp_infer // C++ infer -│ │ ├── CMakeLists.txt // Cmake 文件 -│ │ ├── docs // 说明文档 -│ │ │ └── windows_vs2019_build.md -│ │ ├── include // 头文件 -│ │ │ ├── clipper.h // clipper 库 -│ │ │ ├── config.h // 预测配置 -│ │ │ ├── ocr_cls.h // 方向分类器 -│ │ │ ├── ocr_det.h // 文字检测 -│ │ │ ├── ocr_rec.h // 文字识别 -│ │ │ ├── postprocess_op.h // 检测后处理 -│ │ │ ├── preprocess_op.h // 检测预处理 -│ │ │ └── utility.h // 工具 -│ │ ├── readme.md // 说明文档 -│ │ ├── ... -│ │ ├── src // 源文件 -│ │ │ ├── clipper.cpp -│ │ │ ├── config.cpp -│ │ │ ├── main.cpp -│ │ │ ├── ocr_cls.cpp -│ │ │ ├── ocr_det.cpp -│ │ │ ├── ocr_rec.cpp -│ │ │ ├── postprocess_op.cpp -│ │ │ ├── preprocess_op.cpp -│ │ │ └── utility.cpp -│ │ └── tools // 编译、执行脚本 -│ │ ├── build.sh // 编译脚本 -│ │ ├── config.txt // 配置文件 -│ │ └── run.sh // 测试启动脚本 -│ ├── docker -│ │ └── hubserving -│ │ ├── cpu -│ │ │ └── Dockerfile -│ │ ├── gpu -│ │ │ └── Dockerfile -│ │ ├── README_cn.md -│ │ ├── README.md -│ │ └── sample_request.txt -│ ├── hubserving // hubserving -│ │ ├── ocr_cls // 方向分类器 -│ │ │ ├── config.json // serving 配置 -│ │ │ ├── __init__.py -│ │ │ ├── module.py // 预测模型 -│ │ │ └── params.py // 预测参数 -│ │ ├── ocr_det // 文字检测 -│ │ │ ├── config.json // serving 配置 -│ │ │ ├── __init__.py -│ │ │ ├── module.py // 预测模型 -│ │ │ └── params.py // 预测参数 -│ │ ├── ocr_rec // 文字识别 -│ │ │ ├── config.json -│ │ │ ├── __init__.py -│ │ │ ├── module.py -│ │ │ └── params.py -│ │ └── ocr_system // 系统预测 -│ │ ├── config.json -│ │ ├── __init__.py -│ │ ├── module.py -│ │ └── params.py -│ ├── imgs // 预测图片 -│ │ ├── cpp_infer_pred_12.png -│ │ └── demo.png -│ ├── ios_demo // ios demo -│ │ ... -│ ├── lite // lite 部署 -│ │ ├── cls_process.cc // 方向分类器数据处理 -│ │ ├── cls_process.h -│ │ ├── config.txt // 检测配置参数 -│ │ ├── crnn_process.cc // crnn 数据处理 -│ │ ├── crnn_process.h -│ │ ├── db_post_process.cc // db 数据处理 -│ │ ├── db_post_process.h -│ │ ├── Makefile // 编译文件 -│ │ ├── ocr_db_crnn.cc // 串联预测 -│ │ ├── prepare.sh // 数据准备 -│ │ ├── readme.md // 说明文档 -│ │ ... -│ ├── pdserving // pdserving 部署 -│ │ ├── det_local_server.py // 检测 快速版,部署方便预测速度快 -│ │ ├── det_web_server.py // 检测 完整版,稳定性高分布式部署 -│ │ ├── ocr_local_server.py // 检测+识别 快速版 -│ │ ├── ocr_web_client.py // 客户端 -│ │ ├── ocr_web_server.py // 检测+识别 完整版 -│ │ ├── readme.md // 说明文档 -│ │ ├── rec_local_server.py // 识别 快速版 -│ │ └── rec_web_server.py // 识别 完整版 -│ └── slim -│ └── quantization // 量化相关 -│ ├── export_model.py // 导出模型 -│ ├── quant.py // 量化 -│ └── README.md // 说明文档 -├── doc // 文档教程 -│ ... -├── ppocr // 网络核心代码 -│ ├── data // 数据处理 -│ │ ├── imaug // 图片和 label 处理代码 -│ │ │ ├── text_image_aug // 文本识别的 tia 数据扩充 -│ │ │ │ ├── __init__.py -│ │ │ │ ├── augment.py // tia_distort,tia_stretch 和 tia_perspective 的代码 -│ │ │ │ ├── warp_mls.py -│ │ │ ├── __init__.py -│ │ │ ├── east_process.py // EAST 算法的数据处理步骤 -│ │ │ ├── make_border_map.py // 生成边界图 -│ │ │ ├── make_shrink_map.py // 生成收缩图 -│ │ │ ├── operators.py // 图像基本操作,如读取和归一化 -│ │ │ ├── randaugment.py // 随机数据增广操作 -│ │ │ ├── random_crop_data.py // 随机裁剪 -│ │ │ ├── rec_img_aug.py // 文本识别的数据扩充 -│ │ │ └── sast_process.py // SAST 算法的数据处理步骤 -│ │ ├── __init__.py // 构造 dataloader 相关代码 -│ │ ├── lmdb_dataset.py // 读取lmdb数据集的 dataset -│ │ ├── simple_dataset.py // 读取文本格式存储数据集的 dataset -│ ├── losses // 损失函数 -│ │ ├── __init__.py // 构造 loss 相关代码 -│ │ ├── cls_loss.py // 方向分类器 loss -│ │ ├── det_basic_loss.py // 检测基础 loss -│ │ ├── det_db_loss.py // DB loss -│ │ ├── det_east_loss.py // EAST loss -│ │ ├── det_sast_loss.py // SAST loss -│ │ ├── rec_ctc_loss.py // CTC loss -│ │ ├── rec_att_loss.py // Attention loss -│ ├── metrics // 评估指标 -│ │ ├── __init__.py // 构造 metric 相关代码 -│ │ ├── cls_metric.py // 方向分类器 metric -│ │ ├── det_metric.py // 检测 metric - │ ├── eval_det_iou.py // 检测 iou 相关 -│ │ ├── rec_metric.py // 识别 metric -│ ├── modeling // 组网相关 -│ │ ├── architectures // 网络 -│ │ │ ├── __init__.py // 构造 model 相关代码 -│ │ │ ├── base_model.py // 组网代码 -│ │ ├── backbones // 骨干网络 -│ │ │ ├── __init__.py // 构造 backbone 相关代码 -│ │ │ ├── det_mobilenet_v3.py // 检测 mobilenet_v3 -│ │ │ ├── det_resnet_vd.py // 检测 resnet -│ │ │ ├── det_resnet_vd_sast.py // 检测 SAST算法的resnet backbone -│ │ │ ├── rec_mobilenet_v3.py // 识别 mobilenet_v3 -│ │ │ └── rec_resnet_vd.py // 识别 resnet -│ │ ├── necks // 颈函数 -│ │ │ ├── __init__.py // 构造 neck 相关代码 -│ │ │ ├── db_fpn.py // 标准 fpn 网络 -│ │ │ ├── east_fpn.py // EAST 算法的 fpn 网络 -│ │ │ ├── sast_fpn.py // SAST 算法的 fpn 网络 -│ │ │ ├── rnn.py // 识别 序列编码 -│ │ ├── heads // 头函数 -│ │ │ ├── __init__.py // 构造 head 相关代码 -│ │ │ ├── cls_head.py // 方向分类器 分类头 -│ │ │ ├── det_db_head.py // DB 检测头 -│ │ │ ├── det_east_head.py // EAST 检测头 -│ │ │ ├── det_sast_head.py // SAST 检测头 -│ │ │ ├── rec_ctc_head.py // 识别 ctc -│ │ │ ├── rec_att_head.py // 识别 attention -│ │ ├── transforms // 图像变换 -│ │ │ ├── __init__.py // 构造 transform 相关代码 -│ │ │ └── tps.py // TPS 变换 -│ ├── optimizer // 优化器 -│ │ ├── __init__.py // 构造 optimizer 相关代码 -│ │ └── learning_rate.py // 学习率衰减 -│ │ └── optimizer.py // 优化器 -│ │ └── regularizer.py // 网络正则化 -│ ├── postprocess // 后处理 -│ │ ├── cls_postprocess.py // 方向分类器 后处理 -│ │ ├── db_postprocess.py // DB 后处理 -│ │ ├── east_postprocess.py // EAST 后处理 -│ │ ├── locality_aware_nms.py // NMS -│ │ ├── rec_postprocess.py // 识别网络 后处理 -│ │ └── sast_postprocess.py // SAST 后处理 -│ └── utils // 工具 -│ ├── dict // 小语种字典 -│ .... -│ ├── ic15_dict.txt // 英文数字字典,区分大小写 -│ ├── ppocr_keys_v1.txt // 中文字典,用于训练中文模型 -│ ├── logging.py // logger -│ ├── save_load.py // 模型保存和加载函数 -│ ├── stats.py // 统计 -│ └── utility.py // 工具函数 -├── tools -│ ├── eval.py // 评估函数 -│ ├── export_model.py // 导出 inference 模型 -│ ├── infer // 基于预测引擎预测 -│ │ ├── predict_cls.py -│ │ ├── predict_det.py -│ │ ├── predict_rec.py -│ │ ├── predict_system.py -│ │ └── utility.py -│ ├── infer_cls.py // 基于训练引擎 预测分类 -│ ├── infer_det.py // 基于训练引擎 预测检测 -│ ├── infer_rec.py // 基于训练引擎 预测识别 -│ ├── program.py // 整体流程 -│ ├── test_hubserving.py -│ └── train.py // 启动训练 -├── paddleocr.py -├── README_ch.md // 中文说明文档 -├── README_en.md // 英文说明文档 -├── README.md // 主页说明文档 -├── requirements.txt // 安装依赖 -├── setup.py // whl包打包脚本 -├── train.sh // 启动训练脚本 diff --git a/doc/doc_ch/update.md b/doc/doc_ch/update.md deleted file mode 100644 index a14f5905d7..0000000000 --- a/doc/doc_ch/update.md +++ /dev/null @@ -1,39 +0,0 @@ -# 更新 -- 2022.5.9 发布PaddleOCR v2.5。发布内容包括: - - [PP-OCRv3](./ppocr_introduction.md#pp-ocrv3),速度可比情况下,中文场景效果相比于PP-OCRv2再提升5%,英文场景提升11%,80语种多语言模型平均识别准确率提升5%以上; - - 半自动标注工具[PPOCRLabelv2](https://github.com/PFCCLab/PPOCRLabel):新增表格文字图像、图像关键信息抽取任务和不规则文字图像的标注功能; - - OCR产业落地工具集:打通22种训练部署软硬件环境与方式,覆盖企业90%的训练部署环境需求 - - 交互式OCR开源电子书[《动手学OCR》](./ocr_book.md),覆盖OCR全栈技术的前沿理论与代码实践,并配套教学视频。 -- 2022.5.7 添加对[Weights & Biases](https://docs.wandb.ai/)训练日志记录工具的支持。 -- 2021.12.21 《OCR十讲》课程开讲,12月21日起每晚八点半线上授课! 【免费】报名地址:https://aistudio.baidu.com/aistudio/course/introduce/25207 -- 2021.12.21 发布PaddleOCR v2.4。OCR算法新增1种文本检测算法(PSENet),3种文本识别算法(NRTR、SEED、SAR);文档结构化算法新增1种关键信息提取算法(SDMGR),3种DocVQA算法(LayoutLM、LayoutLMv2,LayoutXLM)。 -- 2021.9.7 发布PaddleOCR v2.3,发布[PP-OCRv2](#PP-OCRv2),CPU推理速度相比于PP-OCR server提升220%;效果相比于PP-OCR mobile 提升7%。 -- 2021.8.3 发布PaddleOCR v2.2,新增文档结构分析[PP-Structure](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.2/ppstructure/README_ch.md)工具包,支持版面分析与表格识别(含Excel导出)。 -- 2021.6.29 [FAQ](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.2/doc/doc_ch/FAQ.md)新增5个高频问题,总数248个,每周一都会更新,欢迎大家持续关注。 -- 2021.4.8 release 2.1版本,新增AAAI 2021论文[端到端识别算法PGNet](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.2/doc/doc_ch/pgnet.md)开源,[多语言模型](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.2/doc/doc_ch/multi_languages.md)支持种类增加到80+。 -- 2020.12.15 更新数据合成工具[Style-Text](https://github.com/PFCCLab/StyleText/blob/main/README_ch.md),可以批量合成大量与目标场景类似的图像,在多个场景验证,效果明显提升。 -- 2020.12.07 [FAQ](../../doc/doc_ch/FAQ.md)新增5个高频问题,总数124个,并且计划以后每周一都会更新,欢迎大家持续关注。 -- 2020.11.25 更新半自动标注工具[PPOCRLabel](https://github.com/PFCCLab/PPOCRLabel/blob/main/README_ch.md),辅助开发者高效完成标注任务,输出格式与PP-OCR训练任务完美衔接。 -- 2020.9.22 更新PP-OCR技术文章,https://arxiv.org/abs/2009.09941 -- 2020.9.19 更新超轻量压缩ppocr_mobile_slim系列模型,整体模型3.5M(详见PP-OCR Pipeline),适合在移动端部署使用。 -- 2020.9.17 更新超轻量ppocr_mobile系列和通用ppocr_server系列中英文ocr模型,媲美商业效果。 -- 2020.9.17 更新[英文识别模型](./models_list.md#english-recognition-model)和[多语种识别模型](./models_list.md#english-recognition-model),已支持`德语、法语、日语、韩语`,更多语种识别模型将持续更新。 -- 2020.8.26 更新OCR相关的84个常见问题及解答,具体参考[FAQ](./FAQ.md) -- 2020.8.24 支持通过whl包安装使用PaddleOCR,具体参考[Paddleocr Package使用说明](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/doc/doc_ch/whl.md) -- 2020.8.21 更新8月18日B站直播课回放和PPT,课节2,易学易用的OCR工具大礼包,[获取地址](https://aistudio.baidu.com/aistudio/education/group/info/1519) -- 2020.8.16 开源文本检测算法[SAST](https://arxiv.org/abs/1908.05498)和文本识别算法[SRN](https://arxiv.org/abs/2003.12294) -- 2020.7.23 发布7月21日B站直播课回放和PPT,课节1,PaddleOCR开源大礼包全面解读,[获取地址](https://aistudio.baidu.com/aistudio/course/introduce/1519) -- 2020.7.15 添加基于EasyEdge和Paddle-Lite的移动端DEMO,支持iOS和Android系统 -- 2020.7.15 完善预测部署,添加基于C++预测引擎推理、服务化部署和端侧部署方案,以及超轻量级中文OCR模型预测耗时Benchmark -- 2020.7.15 整理OCR相关数据集、常用数据标注以及合成工具 -- 2020.7.9 添加支持空格的识别模型,识别效果,预测及训练方式请参考快速开始和文本识别训练相关文档 -- 2020.7.9 添加数据增强、学习率衰减策略,具体参考[配置文件](./config.md) -- 2020.6.8 添加[数据集](dataset/datasets.md),并保持持续更新 -- 2020.6.5 支持 `attetnion` 模型导出 `inference_model` -- 2020.6.5 支持单独预测识别时,输出结果得分 -- 2020.5.30 提供超轻量级中文OCR在线体验 -- 2020.5.30 模型预测、训练支持Windows系统 -- 2020.5.30 开源通用中文OCR模型 -- 2020.5.14 发布[PaddleOCR公开课](https://www.bilibili.com/video/BV1nf4y1U7RX?p=4) -- 2020.5.14 发布[PaddleOCR实战练习](https://aistudio.baidu.com/aistudio/projectdetail/467229) -- 2020.5.14 开源8.6M超轻量级中文OCR模型 diff --git a/doc/doc_ch/visualization.md b/doc/doc_ch/visualization.md deleted file mode 100644 index 2546347532..0000000000 --- a/doc/doc_ch/visualization.md +++ /dev/null @@ -1,59 +0,0 @@ -# 效果展示 - - - -## 超轻量PP-OCRv3效果展示 - -### PP-OCRv3中文模型 -
- - - -
- -### PP-OCRv3英文数字模型 - -
- - - -
- -### PP-OCRv3多语言模型 - -
- - -
- - - -## 超轻量PP-OCRv2效果展示 - - - - - - -## 通用PP-OCR server 效果展示 - -
- - - -
- - - -## 英文识别模型效果展示 -
- -
- - - -## 多语言识别模型效果展示 -
- - -
diff --git a/doc/doc_ch/whl.md b/doc/doc_ch/whl.md deleted file mode 100644 index 2f9976f6d8..0000000000 --- a/doc/doc_ch/whl.md +++ /dev/null @@ -1,492 +0,0 @@ -# paddleocr package使用说明 - -## 1 快速上手 - -### 1.1 安装whl包 - -pip安装 - -```bash -pip install paddleocr -``` - -本地构建并安装 - -```bash -python3 -m build -pip3 install dist/paddleocr-x.x.x-py3-none-any.whl # x.x.x是paddleocr的版本号 -``` - -## 2 使用 - -### 2.1 代码使用 - -paddleocr whl包会自动下载ppocr轻量级模型作为默认模型,可以根据第3节**自定义模型**进行自定义更换。 - -* 检测+方向分类器+识别全流程 - -```python -from paddleocr import PaddleOCR, draw_ocr - -# Paddleocr目前支持中英文、英文、法语、德语、韩语、日语,可以通过修改lang参数进行切换 -# 参数依次为`ch`, `en`, `french`, `german`, `korean`, `japan`。 -ocr = PaddleOCR(use_angle_cls=True, lang="ch") # need to run only once to download and load model into memory -img_path = 'PaddleOCR/doc/imgs/11.jpg' -result = ocr.ocr(img_path, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# 显示结果 -from PIL import Image -result = result[0] -image = Image.open(img_path).convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` - -结果是一个list,每个item包含了文本框,文字和识别置信度 - -```bash -[[[24.0, 36.0], [304.0, 34.0], [304.0, 72.0], [24.0, 74.0]], ['纯臻营养护发素', 0.964739]] -[[[24.0, 80.0], [172.0, 80.0], [172.0, 104.0], [24.0, 104.0]], ['产品信息/参数', 0.98069626]] -[[[24.0, 109.0], [333.0, 109.0], [333.0, 136.0], [24.0, 136.0]], ['(45元/每公斤,100公斤起订)', 0.9676722]] -...... -``` - -结果可视化 - -
- -
- -* 检测+识别 - -```python -from paddleocr import PaddleOCR, draw_ocr - -ocr = PaddleOCR() # need to run only once to download and load model into memory -img_path = 'PaddleOCR/doc/imgs/11.jpg' -result = ocr.ocr(img_path, cls=False) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# 显示结果 -from PIL import Image -result = result[0] -image = Image.open(img_path).convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` - -结果是一个list,每个item包含了文本框,文字和识别置信度 - -```bash -[[[24.0, 36.0], [304.0, 34.0], [304.0, 72.0], [24.0, 74.0]], ['纯臻营养护发素', 0.964739]] -[[[24.0, 80.0], [172.0, 80.0], [172.0, 104.0], [24.0, 104.0]], ['产品信息/参数', 0.98069626]] -[[[24.0, 109.0], [333.0, 109.0], [333.0, 136.0], [24.0, 136.0]], ['(45元/每公斤,100公斤起订)', 0.9676722]] -...... -``` - -结果可视化 - -
- -
- -* 方向分类器+识别 - -```python -from paddleocr import PaddleOCR - -ocr = PaddleOCR(use_angle_cls=True) # need to run only once to download and load model into memory -img_path = 'PaddleOCR/doc/imgs_words/ch/word_1.jpg' -result = ocr.ocr(img_path, det=False, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) -``` - -结果是一个list,每个item只包含识别结果和识别置信度 - -```bash -['韩国小馆', 0.9907421] -``` - -* 单独执行检测 - -```python -from paddleocr import PaddleOCR, draw_ocr - -ocr = PaddleOCR() # need to run only once to download and load model into memory -img_path = 'PaddleOCR/doc/imgs/11.jpg' -result = ocr.ocr(img_path, rec=False) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# 显示结果 -from PIL import Image -result = result[0] -image = Image.open(img_path).convert('RGB') -im_show = draw_ocr(image, result, txts=None, scores=None, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` - -结果是一个list,每个item只包含文本框 - -```bash -[[26.0, 457.0], [137.0, 457.0], [137.0, 477.0], [26.0, 477.0]] -[[25.0, 425.0], [372.0, 425.0], [372.0, 448.0], [25.0, 448.0]] -[[128.0, 397.0], [273.0, 397.0], [273.0, 414.0], [128.0, 414.0]] -...... -``` - -结果可视化 - - -
- -
- -* 单独执行识别 - -```python -from paddleocr import PaddleOCR - -ocr = PaddleOCR() # need to run only once to download and load model into memory -img_path = 'PaddleOCR/doc/imgs_words/ch/word_1.jpg' -result = ocr.ocr(img_path, det=False) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) -``` - -结果是一个list,每个item只包含识别结果和识别置信度 - -```bash -['韩国小馆', 0.9907421] -``` - -* 单独执行方向分类器 - -```python -from paddleocr import PaddleOCR - -ocr = PaddleOCR(use_angle_cls=True) # need to run only once to download and load model into memory -img_path = 'PaddleOCR/doc/imgs_words/ch/word_1.jpg' -result = ocr.ocr(img_path, det=False, rec=False, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) -``` - -结果是一个list,每个item只包含分类结果和分类置信度 - -```bash -['0', 0.9999924] -``` - -### 2.2 通过命令行使用 - -查看帮助信息 - -```bash -paddleocr -h -``` - -* 检测+方向分类器+识别全流程 - -```bash -paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg --use_angle_cls true -``` - -结果是一个list,每个item包含了文本框,文字和识别置信度 - -```bash -[[[28.0, 37.0], [302.0, 39.0], [302.0, 72.0], [27.0, 70.0]], ('纯臻营养护发素', 0.9658738374710083)] -...... -``` - -此外,paddleocr也支持输入pdf文件,并且可以通过指定参数`page_num`来控制推理前面几页,默认为0,表示推理所有页。 -```bash -paddleocr --image_dir ./xxx.pdf --use_angle_cls true --use_gpu false --page_num 2 -``` - -* 检测+识别 - -```bash -paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg -``` - -结果是一个list,每个item包含了文本框,文字和识别置信度 - -```bash -[[[28.0, 37.0], [302.0, 39.0], [302.0, 72.0], [27.0, 70.0]], ('纯臻营养护发素', 0.9658738374710083)] -...... -``` - -* 方向分类器+识别 - -```bash -paddleocr --image_dir PaddleOCR/doc/imgs_words/ch/word_1.jpg --use_angle_cls true --det false -``` - -结果是一个list,每个item只包含识别结果和识别置信度 - -```bash -['韩国小馆', 0.994467] -``` - -* 单独执行检测 - -```bash -paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg --rec false -``` - -结果是一个list,每个item只包含文本框 - -```bash -[[27.0, 459.0], [136.0, 459.0], [136.0, 479.0], [27.0, 479.0]] -[[28.0, 429.0], [372.0, 429.0], [372.0, 445.0], [28.0, 445.0]] -...... -``` - -* 单独执行识别 - -```bash -paddleocr --image_dir PaddleOCR/doc/imgs_words/ch/word_1.jpg --det false -``` - -结果是一个list,每个item只包含识别结果和识别置信度 - -```bash -['韩国小馆', 0.994467] -``` - -* 单独执行方向分类器 - -```bash -paddleocr --image_dir PaddleOCR/doc/imgs_words/ch/word_1.jpg --use_angle_cls true --det false --rec false -``` - -结果是一个list,每个item只包含分类结果和分类置信度 - -```bash -['0', 0.9999924] -``` - -## 3 自定义模型 - -当内置模型无法满足需求时,需要使用到自己训练的模型。 首先,参照[模型导出](./detection.md#4-模型导出与预测)将检测、分类和识别模型转换为inference模型,然后按照如下方式使用 - -### 3.1 代码使用 - -```python -from paddleocr import PaddleOCR, draw_ocr - -# 模型路径下必须含有model和params文件 -ocr = PaddleOCR(det_model_dir='{your_det_model_dir}', rec_model_dir='{your_rec_model_dir}', - rec_char_dict_path='{your_rec_char_dict_path}', cls_model_dir='{your_cls_model_dir}', - use_angle_cls=True) -img_path = 'PaddleOCR/doc/imgs/11.jpg' -result = ocr.ocr(img_path, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# 显示结果 -from PIL import Image -result = result[0] -image = Image.open(img_path).convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` - -### 3.2 通过命令行使用 - -```bash -paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg --det_model_dir {your_det_model_dir} --rec_model_dir {your_rec_model_dir} --rec_char_dict_path {your_rec_char_dict_path} --cls_model_dir {your_cls_model_dir} --use_angle_cls true -``` - -## 4 使用网络图片或者numpy数组作为输入 - -### 4.1 网络图片 - -- 代码使用 - -```python -from paddleocr import PaddleOCR, draw_ocr, download_with_progressbar - -# Paddleocr目前支持中英文、英文、法语、德语、韩语、日语,可以通过修改lang参数进行切换 -# 参数依次为`ch`, `en`, `french`, `german`, `korean`, `japan`。 -ocr = PaddleOCR(use_angle_cls=True, lang="ch") # need to run only once to download and load model into memory -img_path = 'http://n.sinaimg.cn/ent/transform/w630h933/20171222/o111-fypvuqf1838418.jpg' -result = ocr.ocr(img_path, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# 显示结果 -from PIL import Image -result = result[0] -download_with_progressbar(img_path, 'tmp.jpg') -image = Image.open('tmp.jpg').convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` - -- 命令行模式 - -```bash -paddleocr --image_dir http://n.sinaimg.cn/ent/transform/w630h933/20171222/o111-fypvuqf1838418.jpg --use_angle_cls=true -``` - -### 4.2 numpy数组 - -仅通过代码使用时支持numpy数组作为输入 - -```python -import cv2 -from paddleocr import PaddleOCR, draw_ocr - -# Paddleocr目前支持中英文、英文、法语、德语、韩语、日语,可以通过修改lang参数进行切换 -# 参数依次为`ch`, `en`, `french`, `german`, `korean`, `japan`。 -ocr = PaddleOCR(use_angle_cls=True, lang="ch") # need to run only once to download and load model into memory -img_path = 'PaddleOCR/doc/imgs/11.jpg' -img = cv2.imread(img_path) -# img = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY), 如果你自己训练的模型支持灰度图,可以将这句话的注释取消 -result = ocr.ocr(img, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# 显示结果 -from PIL import Image -result = result[0] -image = Image.open(img_path).convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` -## 5 PDF文件作为输入 -- 命令行模式 - -可以通过指定参数`page_num`来控制推理前面几页,默认为0,表示推理所有页。 -```bash -paddleocr --image_dir ./xxx.pdf --use_angle_cls true --use_gpu false --page_num 2 -``` -- 代码使用 - -```python -from paddleocr import PaddleOCR, draw_ocr - -# Paddleocr目前支持的多语言语种可以通过修改lang参数进行切换 -# 例如`ch`, `en`, `fr`, `german`, `korean`, `japan` -ocr = PaddleOCR(use_angle_cls=True, lang="ch", page_num=2) # need to run only once to download and load model into memory -img_path = './xxx.pdf' -result = ocr.ocr(img_path, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# 显示结果 -import fitz -from PIL import Image -import cv2 -import numpy as np -imgs = [] -with fitz.open(img_path) as pdf: - for pg in range(0, pdf.pageCount): - page = pdf[pg] - mat = fitz.Matrix(2, 2) - pm = page.getPixmap(matrix=mat, alpha=False) - # if width or height > 2000 pixels, don't enlarge the image - if pm.width > 2000 or pm.height > 2000: - pm = page.getPixmap(matrix=fitz.Matrix(1, 1), alpha=False) - - img = Image.frombytes("RGB", [pm.width, pm.height], pm.samples) - img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR) - imgs.append(img) -for idx in range(len(result)): - res = result[idx] - image = imgs[idx] - boxes = [line[0] for line in res] - txts = [line[1][0] for line in res] - scores = [line[1][1] for line in res] - im_show = draw_ocr(image, boxes, txts, scores, font_path='doc/fonts/simfang.ttf') - im_show = Image.fromarray(im_show) - im_show.save('result_page_{}.jpg'.format(idx)) -``` - -## 6 参数说明 - -| 字段 | 说明 | 默认值 | -|-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------| -| use_gpu | 是否使用GPU | TRUE | -| gpu_mem | 初始化占用的GPU内存大小 | 8000M | -| image_dir | 通过命令行调用时执行预测的图片或文件夹路径 | -| page_num | 当输入类型为pdf文件时有效,指定预测前面page_num页,默认预测所有页 | 0 | -| det_algorithm | 使用的检测算法类型 | DB | -| det_model_dir | 检测模型所在文件夹。传参方式有两种,1. None: 自动下载内置模型到 `~/.paddleocr/det`;2.自己转换好的inference模型路径,模型路径下必须包含model和params文件 | None | -| det_max_side_len | 检测算法前向时图片长边的最大尺寸,当长边超出这个值时会将长边resize到这个大小,短边等比例缩放 | 960 | -| det_db_thresh | DB模型输出预测图的二值化阈值 | 0.3 | -| det_db_box_thresh | DB模型输出框的阈值,低于此值的预测框会被丢弃 | 0.5 | -| det_db_unclip_ratio | DB模型输出框扩大的比例 | 2 | -| det_db_score_mode | 计算检测框score的方式,有'fast'和'slow',如果要检测的文字有弯曲,建议用'slow','slow'模式计算的box的score偏大,box不容易被过滤掉 | 'fast' | -| det_east_score_thresh | EAST模型输出预测图的二值化阈值 | 0.8 | -| det_east_cover_thresh | EAST模型输出框的阈值,低于此值的预测框会被丢弃 | 0.1 | -| det_east_nms_thresh | EAST模型输出框NMS的阈值 | 0.2 | -| rec_algorithm | 使用的识别算法类型 | CRNN | -| rec_model_dir | 识别模型所在文件夹。传参方式有两种,1. None: 自动下载内置模型到 `~/.paddleocr/rec`;2.自己转换好的inference模型路径,模型路径下必须包含model和params文件 | None | -| rec_image_shape | 识别算法的输入图片尺寸 | "3,32,320" | -| rec_batch_num | 进行识别时,同时前向的图片数 | 30 | -| max_text_length | 识别算法能识别的最大文字长度 | 25 | -| rec_char_dict_path | 识别模型字典路径,当rec_model_dir使用方式2传参时需要修改为自己的字典路径 | ./ppocr/utils/ppocr_keys_v1.txt | -| use_space_char | 是否识别空格 | TRUE | -| drop_score | 对输出按照分数(来自于识别模型)进行过滤,低于此分数的不返回 | 0.5 | -| use_angle_cls | 是否加载分类模型 | FALSE | -| cls_model_dir | 分类模型所在文件夹。传参方式有两种,1. None: 自动下载内置模型到 `~/.paddleocr/cls`;2.自己转换好的inference模型路径,模型路径下必须包含model和params文件 | None | -| cls_image_shape | 分类算法的输入图片尺寸 | "3, 48, 192" | -| label_list | 分类算法的标签列表 | ['0', '180'] | -| cls_batch_num | 进行分类时,同时前向的图片数 |30 | -| enable_mkldnn | 是否启用mkldnn | FALSE | -| use_zero_copy_run | 是否通过zero_copy_run的方式进行前向 | FALSE | -| lang | 模型语言类型,目前支持 目前支持中英文(ch)、英文(en)、法语(french)、德语(german)、韩语(korean)、日语(japan) | ch | -| det | 前向时使用启动检测 | TRUE | -| rec | 前向时是否启动识别 | TRUE | -| cls | 前向时是否启动分类 (命令行模式下使用use_angle_cls控制前向是否启动分类) | FALSE | -| show_log | 是否打印logger信息 | FALSE | -| type | 执行ocr或者表格结构化, 值可选['ocr','structure'] | ocr | -| ocr_version | OCR模型版本,可选PP-OCRv3, PP-OCRv2, PP-OCR。PP-OCRv3 支持中、英文的检测、识别、多语种识别,方向分类器等模型;PP-OCRv2 目前仅支持中文的检测和识别模型;PP-OCR支持中文的检测,识别,多语种识别,方向分类器等模型 | PP-OCRv3 | diff --git a/doc/doc_en/FAQ_en.md b/doc/doc_en/FAQ_en.md deleted file mode 100644 index 1074c73719..0000000000 --- a/doc/doc_en/FAQ_en.md +++ /dev/null @@ -1,58 +0,0 @@ -## FAQ - -1. **Prediction error: got an unexpected keyword argument 'gradient_clip'** -The installed version of paddle is incorrect. Currently, this project only supports Paddle 1.7, which will be adapted to 1.8 in the near future. - -2. **Error when converting attention recognition model: KeyError: 'predict'** -Solved. Please update to the latest version of the code. - -3. **About inference speed** -When there are many words in the picture, the prediction time will increase. You can use `--rec_batch_num` to set a smaller prediction batch num. The default value is 30, which can be changed to 10 or other values. - -4. **Service deployment and mobile deployment** -It is expected that the service deployment based on Serving and the mobile deployment based on Paddle Lite will be released successively in mid-to-late June. Stay tuned for more updates. - -5. **Release time of self-developed algorithm** -Baidu Self-developed algorithms such as SAST, SRN and end2end PSL will be released in June or July. Please be patient. - -6. **How to run on Windows or Mac?** -PaddleOCR has completed the adaptation to Windows and MAC systems. Two points should be noted during operation: - 1. In [Quick installation](./installation_en.md), if you do not want to install docker, you can skip the first step and start with the second step. - 2. When downloading the inference model, if wget is not installed, you can directly click the model link or copy the link address to the browser to download, then extract and place it in the corresponding directory. - -7. **The difference between ultra-lightweight model and General OCR model** -At present, PaddleOCR has opensourced two Chinese models, namely 8.6M ultra-lightweight Chinese model and general Chinese OCR model. The comparison information between the two is as follows: - - Similarities: Both use the same **algorithm** and **training data**; - - Differences: The difference lies in **backbone network** and **channel parameters**, the ultra-lightweight model uses MobileNetV3 as the backbone network, the general model uses Resnet50_vd as the detection model backbone, and Resnet34_vd as the recognition model backbone. You can compare the two model training configuration files to see the differences in parameters. - -|Model|Backbone|Detection configuration file|Recognition configuration file| -|-|-|-|-| -|8.6M ultra-lightweight Chinese OCR model|MobileNetV3+MobileNetV3|det_mv3_db.yml|rec_chinese_lite_train.yml| -|General Chinese OCR model|Resnet50_vd+Resnet34_vd|det_r50_vd_db.yml|rec_chinese_common_train.yml| - -8. **Is there a plan to opensource a model that only recognizes numbers or only English + numbers?** -It is not planned to opensource numbers only, numbers + English only, or other vertical text models. PaddleOCR has opensourced a variety of detection and recognition algorithms for customized training. The two Chinese models are also based on the training output of the open-source algorithm library. You can prepare the data according to the tutorial, choose the appropriate configuration file, train yourselves, and we believe that you can get good result. If you have any questions during the training, you are welcome to open issues or ask in the communication group. We will answer them in time. - -9. **What is the training data used by the open-source model? Can it be opensourced?** -At present, the open source model, dataset and magnitude are as follows: - - Detection: - English dataset: ICDAR2015 - Chinese dataset: LSVT street view dataset with 3w pictures - - Recognition: - English dataset: MJSynth and SynthText synthetic dataset, the amount of data is tens of millions. - Chinese dataset: LSVT street view dataset with cropped text area, a total of 30w images. In addition, the synthesized data based on LSVT corpus is 500w. - - Among them, the public datasets are opensourced, users can search and download by themselves, or refer to [Chinese data set](dataset/datasets_en.md), synthetic data is not opensourced, users can use open-source synthesis tools to synthesize data themselves. Current available synthesis tools include [text_renderer](https://github.com/Sanster/text_renderer), [SynthText](https://github.com/ankush-me/SynthText), [TextRecognitionDataGenerator](https://github.com/Belval/TextRecognitionDataGenerator), etc. - -10. **Error in using the model with TPS module for prediction** -Error message: Input(X) dims[3] and Input(Grid) dims[2] should be equal, but received X dimension[3]\(108) != Grid dimension[2]\(100) -Solution: TPS does not support variable shape. Please set --rec_image_shape='3,32,100' and --rec_char_type='en' - -11. **Custom dictionary used during training, the recognition results show that words do not appear in the dictionary** -The used custom dictionary path is not set when making prediction. The solution is setting parameter `rec_char_dict_path` to the corresponding dictionary file. - -12. **Results of cpp_infer and python_inference are very different** -Versions of exported inference model and inference library should be same. For example, on Windows platform, version of the inference library that PaddlePaddle provides is 1.8, but version of the inference model that PaddleOCR provides is 1.7, you should export model yourself(`tools/export_model.py`) on PaddlePaddle 1.8 and then use the exported model for inference. - -13. **How to identify artistic fonts in signs or advertising images** -Recognizing artistic fonts in signs or advertising images is a very challenging task because the variation in individual characters is much greater compared to standard fonts. If the artistic font to be identified is within a dictionary list, each word in the dictionary can be treated as a template for recognition using a general image retrieval system. You can try using PaddleClas image recognition system. diff --git a/doc/doc_en/PP-OCRv3_introduction_en.md b/doc/doc_en/PP-OCRv3_introduction_en.md deleted file mode 100644 index 8d5a36edf7..0000000000 --- a/doc/doc_en/PP-OCRv3_introduction_en.md +++ /dev/null @@ -1,239 +0,0 @@ -English | [简体中文](../doc_ch/PP-OCRv3_introduction.md) - -# PP-OCRv3 - -- [1. Introduction](#1) -- [2. Optimization for Text Detection Model](#2) -- [3. Optimization for Text Recognition Model](#3) -- [4. End-to-end Evaluation](#4) - - - -## 1. Introduction - -PP-OCRv3 is further upgraded on the basis of PP-OCRv2. The overall framework of PP-OCRv3 is same as that of PP-OCRv2. The text detection model and text recognition model are further optimized, respectively. Specifically, the detection network is still optimized based on DBNet, and base model of recognition network is replaced from CRNN to [SVTR](https://arxiv.org/abs/2205.00159), which is recorded in IJCAI 2022. The block diagram of the PP-OCRv3 system is as follows (strategies in the pink box are newly introduced in PP-OCRv3): - -
- -
- -There are 9 optimization strategies for text detection and recognition models in PP-OCRv3, which are as follows. - -- Text detection: - - LK-PAN: A PAN structure with large receptive field; - - DML: Deep Mutual Learning strategy for teacher model; - - RSE-FPN: A FPN structure with residual attention mechanism; - -- Text recognition: - - SVTR_LCNet: A Light-weight text recognition network; - - GTC: Guided training of CTC by Attention; - - TextConAug: A data augmentation strategy for mining textual context information; - - TextRotNet: Self-supervised strategy for a better pretrained model; - - UDML: Unified deep mutual learning strategy; - - UIM: Unlabeled data mining strategy. - -In terms of effect, when the speed is comparable, the accuracy of various scenes is greatly improved: - -- In Chinese scenarios, PP-OCRv3 outperforms PP-OCRv2 by more than 5%. -- In English scenarios, PP-OCRv3 outperforms PP-OCRv2 by more than 11%. -- In multi-language scenarios, models for more than 80 languages are optimized, the average accuracy is increased by more than 5%. - - - - -## 2. Optimization for Text Detection Model - -The PP-OCRv3 detection model upgrades the [CML](https://arxiv.org/pdf/2109.03144.pdf) (Collaborative Mutual Learning) distillation strategy proposed in PP-OCRv2. As shown in the figure below, the main idea of CML combines ① the traditional distillation strategy of Teacher guiding Student and ② the DML strategy, which allows the Students network to learn from each other. PP-OCRv3 further optimizes the effect of teacher model and student model respectively. For the Teacher model, a pan module with large receptive field named LK-PAN is proposed and the DML distillation strategy is adopted; for the student model, a FPN module with residual attention mechanism named RSE-FPN is proposed. - - -
- -
- - -The ablation experiments are as follows: - -|ID|Strategy|Model Size|Hmean|The Inference Time(cpu + mkldnn)| -|-|-|-|-|-| -|baseline teacher|PP-OCR server|49.0M|83.20%|171ms| -|teacher1|DB-R50-LK-PAN|124.0M|85.00%|396ms| -|teacher2|DB-R50-LK-PAN-DML|124.0M|86.00%|396ms| -|baseline student|PP-OCRv2|3.0M|83.20%|117ms| -|student0|DB-MV3-RSE-FPN|3.6M|84.50%|124ms| -|student1|DB-MV3-CML(teacher2)|3.0M|84.30%|117ms| -|student2|DB-MV3-RSE-FPN-CML(teacher2)|3.6M|85.40%|124ms| - -Testing environment: Intel Gold 6148 CPU, with MKLDNN acceleration enabled during inference. - - -**(1) LK-PAN: A PAN structure with large receptive field** - -LK-PAN (Large Kernel PAN) is a lightweight [PAN](https://arxiv.org/pdf/1803.01534.pdf) structure with larger receptive field. The main idea is to change the convolution kernel size in the path augmentation of the PAN structure from `3*3` to `9*9`. By increasing the convolution kernel size, the receptive field of each position of the feature map is improved, making it easier to detect text in large fonts and text with extreme aspect ratios. Using LK-PAN, the hmean of the teacher model can be improved from 83.2% to 85.0%. - -
- -
- - -**(2) DML: Deep Mutual Learning Strategy for Teacher Model** - -[DML](https://arxiv.org/abs/1706.00384)(Deep Mutual Learning), as shown in the figure below, can effectively improve the accuracy of the text detection model by learning from each other with two models with the same structure. The DML strategy is adopted in the teacher model training, and the hmean is increased from 85% to 86%. By updating the teacher model of CML in PP-OCRv2 to the above-mentioned higher-precision one, the hmean of the student model can be further improved from 83.2% to 84.3%. - - -
- -
- - -**(3) RSE-FPN: A FPN structure with residual attention mechanism** - -RSE-FPN (Residual Squeeze-and-Excitation FPN) is shown in the figure below. RSE-FPN introduces residual attention mechanism by replacing the convolutional layer in the FPN with RSEConv, to improve the representation ability of the feature map. - -Considering that the features of some channels will be suppressed if the convolution layer in FPN is directly replaced with SEblock, as the number of FPN channels in the detection model of PP-OCRv2 is 96, which is very small. The introduction of residual structure in RSEConv can alleviate the above problems and improve the text detection effect. By updating the FPN structure of the student model of CML to RSE-FPN, the hmean of the student model can be further improved from 84.3% to 85.4%. - -
- -
- - - -## 3. Optimization for Text Recognition Model - -The recognition module of PP-OCRv3 is optimized based on the text recognition algorithm [SVTR](https://arxiv.org/abs/2205.00159). RNN is abandoned in SVTR, and the context information of the text line image is more effectively mined by introducing the Transformers structure, thereby improving the text recognition ability. - -The recognition accuracy of SVTR_tiny outperforms PP-OCRv2 recognition model by 5.3%, while the prediction speed nearly 11 times slower. It takes nearly 100ms to predict a text line on CPU. Therefore, as shown in the figure below, PP-OCRv3 adopts the following six optimization strategies to accelerate the recognition model. - -
- -
- -Based on the above strategy, compared with PP-OCRv2, the PP-OCRv3 recognition model further improves the accuracy by 4.6% with comparable speed. The ablation experiments are as follows: - -| ID | strategy | Model size | accuracy | prediction speed(CPU + MKLDNN)| -|-----|-----|--------|----| --- | -| 01 | PP-OCRv2 | 8.0M | 74.80% | 8.54ms | -| 02 | SVTR_Tiny | 21.0M | 80.10% | 97.00ms | -| 03 | SVTR_LCNet(h32) | 12.0M | 71.90% | 6.60ms | -| 04 | SVTR_LCNet(h48) | 12.0M | 73.98% | 7.60ms | -| 05 | + GTC | 12.0M | 75.80% | 7.60ms | -| 06 | + TextConAug | 12.0M | 76.30% | 7.60ms | -| 07 | + TextRotNet | 12.0M | 76.90% | 7.60ms | -| 08 | + UDML | 12.0M | 78.40% | 7.60ms | -| 09 | + UIM | 12.0M | 79.40% | 7.60ms | - -Note: When testing the speed, the input image shape of Experiment 01-03 is (3, 32, 320), and the input image shape of 04-08 is (3, 48, 320). In the actual prediction, the image is a variable-length input, and the speed will vary. Testing environment: Intel Gold 6148 CPU, with MKLDNN acceleration enabled during prediction. - -**(1)SVTR_LCNet:Lightweight Text Recognition Network** - -SVTR_LCNet is a lightweight text recognition network fused by Transformer-based network [SVTR](https://arxiv.org/abs/2205.00159) and lightweight CNN-based network [PP-LCNet](https://arxiv.org/abs/2109.15099). The prediction speed of SVTR_LCNet is 20% faster than that of PP-OCRv2 recognizer while the effect is slightly worse because the distillation strategy is not adopted. In addition, the height of the input image is further increased from 32 to 48, which makes the prediction speed slightly slower, but the model effect greatly improved. The recognition accuracy reaches 73.98% (+2.08%), which is close to the accuracy of PP-OCRv2 recognizer trained with the distillation strategy. - -SVTR_Tiny network structure is as follows: - -
- -
- -Due to the limited model structure supported by the MKLDNN acceleration library, SVTR is 10 times slower than PP-OCRv2 on CPU+MKLDNN. PP-OCRv3 expects to improve the accuracy of the model without bringing additional inference time. Through analysis, it is found that the main time-consuming module of SVTR_Tiny structure is Mixing Block, so we have carried out a series of optimizations to the structure of SVTR_Tiny (for detailed speed data, please refer to the ablation experiment table below): - - -1. Replace the first half of the SVTR network with the first three stages of PP-LCNet, retain 4 Global Mixing Blocks, the accuracy is 76%, and the speedup is 69%. The network structure is as follows: - -
- -
- -2. Reduce the number of Global Mixing Blocks from 4 to 2, the accuracy is 72.9%, and the speedup is 69%. The network structure is as follows: - -
- -
- -3. The experiment found that the prediction speed of the Global Mixing Block is related to the shape of the input features. Therefore, after moving the position of the Global Mixing Block to the back of pooling layer, the accuracy dropped to 71.9%, and the speed surpassed the PP-OCRv2-baseline based on the CNN structure by 22%. The network structure is as follows: - -
- -
- -The ablation experiments are as follows: - -| ID | strategy | Model size | accuracy | prediction speed(CPU + MKLDNN)| -|-----|-----|--------|----| --- | -| 01 | PP-OCRv2-baseline | 8.0M | 69.30% | 8.54ms | -| 02 | SVTR_Tiny | 21.0M | 80.10% | 97.00ms | -| 03 | SVTR_LCNet(G4) | 9.2M | 76.00% | 30.00ms | -| 04 | SVTR_LCNet(G2) | 13.0M | 72.98% | 9.37ms | -| 05 | SVTR_LCNet(h32) | 12.0M | 71.90% | 6.60ms | -| 06 | SVTR_LCNet(h48) | 12.0M | 73.98% | 7.60ms | - -Note: When testing the speed, the input image shape of 01-05 are all (3, 32, 320); PP-OCRv2-baseline represents the model trained without distillation method - -**(2)GTC:Attention guides CTC training strategy** - -[GTC](https://arxiv.org/pdf/2002.01276.pdf) (Guided Training of CTC), using the Attention module to guide the training of CTC to fuse multiple features is an effective strategy to improve text recognition accuracy. No more time-consuming is added in the inference process as the Attention module is completely removed during prediction. The accuracy of the recognition model is further improved to 75.8% (+1.82%). The training process is as follows: - -
- -
- -**(3)TextConAug:Data Augmentation Strategy for Mining Text Context Information** - -TextConAug is a data augmentation strategy for mining textual context information. The main idea comes from the paper [ConCLR](https://www.cse.cuhk.edu.hk/~byu/papers/C139-AAAI2022-ConCLR.pdf), in which the author proposes data augmentation strategy ConAug to concat 2 different images in a batch to form new images and perform self-supervised comparative learning. PP-OCRv3 applies this method to supervised learning tasks, and designs the TextConAug data augmentation method, which can enrich the context information of training data and improve the diversity of training data. Using this strategy, the accuracy of the recognition model is further improved to 76.3% (+0.5%). The schematic diagram of TextConAug is as follows: - -
- -
- - -**(4)TextRotNet:Self-Supervised Pre-trained Model** - -TextRotNet is a pre-trained model trained with a large amount of unlabeled text line data in a self-supervised manner, refered to the paper [STR-Fewer-Labels](https://github.com/ku21fan/STR-Fewer-Labels). This model can initialize the weights of SVTR_LCNet, which helps the text recognition model to converge to a better position. Using this strategy, the accuracy of the recognition model is further improved to 76.9% (+0.6%). The TextRotNet training process is shown in the following figure: - -
- -
- - -**(5)UDML:Unified-Deep Mutual Learning** - -UDML (Unified-Deep Mutual Learning) is a strategy proposed in PP-OCRv2 which is very effective to improve the model accuracy. In PP-OCRv3, for two different structures SVTR_LCNet and Attention, the feature map of PP-LCNet, the output of the SVTR module and the output of the Attention module between them are simultaneously supervised and trained. Using this strategy, the accuracy of the recognition model is further improved to 78.4% (+1.5%). - - -**(6)UIM:Unlabeled Images Mining** - -UIM (Unlabeled Images Mining) is a very simple unlabeled data mining strategy. The main idea is to use a high-precision text recognition model to predict unlabeled images to obtain pseudo-labels, and select samples with high prediction confidence as training data for training lightweight models. Using this strategy, the accuracy of the recognition model is further improved to 79.4% (+1%). In practice, we use the full data set to train the high-precision SVTR_Tiny model (acc=82.5%) for data mining. [SVTR_Tiny model download and tutorial](../../applications/高精度中文识别模型.md). - -
- -
- - - -## 4. End-to-end Evaluation - -With the optimization strategies mentioned above, PP-OCRv3 outperforms PP-OCRv2 by 5.00% in terms of end-to-end Hmean for Chinese scenarios with comparable speed. The specific metrics are shown as follows. - -| Model | Hmean | Model Size (M) | Time Cost (CPU, ms) | Time Cost (T4 GPU, ms) | -|-----|-----|--------|----| --- | -| PP-OCR mobile | 50.30% | 8.1 | 356.00 | 116.00 | -| PP-OCR server | 57.00% | 155.1 | 1056.00 | 200.00 | -| PP-OCRv2 | 57.60% | 11.6 | 330.00 | 111.00 | -| PP-OCRv3 | 62.90% | 15.6 | 331.00 | 86.64 | - - -Testing environment: -- CPU: Intel Gold 6148, and MKLDNN acceleration is enabled during CPU inference. - - -In addition to Chinese scenarios, the recognition model for English is also optimized with an increasement of 11.00% for end-to-end Hmean, which is shown as follows. - -| Model | Recall | Precision | Hmean | -|-----|-----|--------|----| -| PP-OCR_en | 38.99% | 45.91% | 42.17% | -| PP-OCRv3_en | 50.95% | 55.53% | 53.14% | - -At the same time, recognition models for more than 80 language are also upgraded. The accuracy of the four language families with evaluation sets is increased by more than 5% on average, which is shown as follows. - -| Model | Latin | Arabic | Japanese | Korean | -|-----|-----|--------|----| --- | -| PP-OCR_mul | 69.60% | 40.50% | 38.50% | 55.40% | -| PP-OCRv3_mul | 75.20% | 45.37% | 45.80% | 60.10% | diff --git a/doc/doc_en/add_new_algorithm_en.md b/doc/doc_en/add_new_algorithm_en.md deleted file mode 100644 index a8903b0a20..0000000000 --- a/doc/doc_en/add_new_algorithm_en.md +++ /dev/null @@ -1,304 +0,0 @@ -# Add New Algorithm - -PaddleOCR decomposes an algorithm into the following parts, and modularizes each part to make it more convenient to develop new algorithms. - -* Data loading and processing -* Network -* Post-processing -* Loss -* Metric -* Optimizer - -The following will introduce each part separately, and introduce how to add the modules required for the new algorithm. - - -## Data loading and processing - -Data loading and processing are composed of different modules, which complete the image reading, data augment and label production. This part is under [ppocr/data](../../ppocr/data). The explanation of each file and folder are as follows: - -```bash -ppocr/data/ -├── imaug # Scripts for image reading, data augment and label production -│ ├── label_ops.py # Modules that transform the label -│ ├── operators.py # Modules that transform the image -│ ├──..... -├── __init__.py -├── lmdb_dataset.py # The dataset that reads the lmdb -└── simple_dataset.py # Read the dataset saved in the form of `image_path\tgt` -``` - -PaddleOCR has a large number of built-in image operation related modules. For modules that are not built-in, you can add them through the following steps: - -1. Create a new file under the [ppocr/data/imaug](../../ppocr/data/imaug) folder, such as my_module.py. -2. Add code in the my_module.py file, the sample code is as follows: - -```python -class MyModule: - def __init__(self, *args, **kwargs): - # your init code - pass - - def __call__(self, data): - img = data['image'] - label = data['label'] - # your process code - - data['image'] = img - data['label'] = label - return data -``` - -3. Import the added module in the [ppocr/data/imaug/\__init\__.py](../../ppocr/data/imaug/__init__.py) file. - -All different modules of data processing are executed by sequence, combined and executed in the form of a list in the config file. Such as: - -```yaml -# angle class data process -transforms: - - DecodeImage: # load image - img_mode: BGR - channel_first: False - - MyModule: - args1: args1 - args2: args2 - - KeepKeys: - keep_keys: [ 'image', 'label' ] # dataloader will return list in this order -``` - -## Network - -The network part completes the construction of the network, and PaddleOCR divides the network into four parts, which are under [ppocr/modeling](../../ppocr/modeling). The data entering the network will pass through these four parts in sequence(transforms->backbones-> -necks->heads). - -```bash -├── architectures # Code for building network -├── transforms # Image Transformation Module -├── backbones # Feature extraction module -├── necks # Feature enhancement module -└── heads # Output module -``` - -PaddleOCR has built-in commonly used modules related to algorithms such as DB, EAST, SAST, CRNN and Attention. For modules that do not have built-in, you can add them through the following steps, the four parts are added in the same steps, take backbones as an example: - -1. Create a new file under the [ppocr/modeling/backbones](../../ppocr/modeling/backbones) folder, such as my_backbone.py. -2. Add code in the my_backbone.py file, the sample code is as follows: - -```python -import paddle -import paddle.nn as nn -import paddle.nn.functional as F - - -class MyBackbone(nn.Layer): - def __init__(self, *args, **kwargs): - super(MyBackbone, self).__init__() - # your init code - self.conv = nn.xxxx - - def forward(self, inputs): - # your network forward - y = self.conv(inputs) - return y -``` - -3. Import the added module in the [ppocr/modeling/backbones/\__init\__.py](../../ppocr/modeling/backbones/__init__.py) file. - -After adding the four-part modules of the network, you only need to configure them in the configuration file to use, such as: - -```yaml -Architecture: - model_type: rec - algorithm: CRNN - Transform: - name: MyTransform - args1: args1 - args2: args2 - Backbone: - name: MyBackbone - args1: args1 - Neck: - name: MyNeck - args1: args1 - Head: - name: MyHead - args1: args1 -``` - -## Post-processing - -Post-processing realizes decoding network output to obtain text box or recognized text. This part is under [ppocr/postprocess](../../ppocr/postprocess). -PaddleOCR has built-in post-processing modules related to algorithms such as DB, EAST, SAST, CRNN and Attention. For components that are not built-in, they can be added through the following steps: - -1. Create a new file under the [ppocr/postprocess](../../ppocr/postprocess) folder, such as my_postprocess.py. -2. Add code in the my_postprocess.py file, the sample code is as follows: - -```python -import paddle - - -class MyPostProcess: - def __init__(self, *args, **kwargs): - # your init code - pass - - def __call__(self, preds, label=None, *args, **kwargs): - if isinstance(preds, paddle.Tensor): - preds = preds.numpy() - # you preds decode code - preds = self.decode_preds(preds) - if label is None: - return preds - # you label decode code - label = self.decode_label(label) - return preds, label - - def decode_preds(self, preds): - # you preds decode code - pass - - def decode_label(self, preds): - # you label decode code - pass -``` - -3. Import the added module in the [ppocr/postprocess/\__init\__.py](../../ppocr/postprocess/__init__.py) file. - -After the post-processing module is added, you only need to configure it in the configuration file to use, such as: - -```yaml -PostProcess: - name: MyPostProcess - args1: args1 - args2: args2 -``` - -## Loss - -The loss function is used to calculate the distance between the network output and the label. This part is under [ppocr/losses](../../ppocr/losses). -PaddleOCR has built-in loss function modules related to algorithms such as DB, EAST, SAST, CRNN and Attention. For modules that do not have built-in modules, you can add them through the following steps: - -1. Create a new file in the [ppocr/losses](../../ppocr/losses) folder, such as my_loss.py. -2. Add code in the my_loss.py file, the sample code is as follows: - -```python -import paddle -from paddle import nn - - -class MyLoss(nn.Layer): - def __init__(self, **kwargs): - super(MyLoss, self).__init__() - # you init code - pass - - def __call__(self, predicts, batch): - label = batch[1] - # your loss code - loss = self.loss(input=predicts, label=label) - return {'loss': loss} -``` - -3. Import the added module in the [ppocr/losses/\__init\__.py](../../ppocr/losses/__init__.py) file. - -After the loss function module is added, you only need to configure it in the configuration file to use it, such as: - -```yaml -Loss: - name: MyLoss - args1: args1 - args2: args2 -``` - -## Metric - -Metric is used to calculate the performance of the network on the current batch. This part is under [ppocr/metrics](../../ppocr/metrics). PaddleOCR has built-in evaluation modules related to algorithms such as detection, classification and recognition. For modules that do not have built-in modules, you can add them through the following steps: - -1. Create a new file under the [ppocr/metrics](../../ppocr/metrics) folder, such as my_metric.py. -2. Add code in the my_metric.py file, the sample code is as follows: - -```python - -class MyMetric(object): - def __init__(self, main_indicator='acc', **kwargs): - # main_indicator is used for select best model - self.main_indicator = main_indicator - self.reset() - - def __call__(self, preds, batch, *args, **kwargs): - # preds is out of postprocess - # batch is out of dataloader - labels = batch[1] - cur_correct_num = 0 - cur_all_num = 0 - # you metric code - self.correct_num += cur_correct_num - self.all_num += cur_all_num - return {'acc': cur_correct_num / cur_all_num, } - - def get_metric(self): - """ - return metrics { - 'acc': 0, - 'norm_edit_dis': 0, - } - """ - acc = self.correct_num / self.all_num - self.reset() - return {'acc': acc} - - def reset(self): - # reset metric - self.correct_num = 0 - self.all_num = 0 - -``` - -3. Import the added module in the [ppocr/metrics/\__init\__.py](../../ppocr/metrics/__init__.py) file. - -After the metric module is added, you only need to configure it in the configuration file to use it, such as: - -```yaml -Metric: - name: MyMetric - main_indicator: acc -``` - -## Optimizer - -The optimizer is used to train the network. The optimizer also contains network regularization and learning rate decay modules. This part is under [ppocr/optimizer](../../ppocr/optimizer). PaddleOCR has built-in -Commonly used optimizer modules such as `Momentum`, `Adam` and `RMSProp`, common regularization modules such as `Linear`, `Cosine`, `Step` and `Piecewise`, and common learning rate decay modules such as `L1Decay` and `L2Decay`. -Modules without built-in can be added through the following steps, take `optimizer` as an example: - -1. Create your own optimizer in the [ppocr/optimizer/optimizer.py](../../ppocr/optimizer/optimizer.py) file, the sample code is as follows: - -```python -from paddle import optimizer as optim - - -class MyOptim(object): - def __init__(self, learning_rate=0.001, *args, **kwargs): - self.learning_rate = learning_rate - - def __call__(self, parameters): - # It is recommended to wrap the built-in optimizer of paddle - opt = optim.XXX( - learning_rate=self.learning_rate, - parameters=parameters) - return opt - -``` - -After the optimizer module is added, you only need to configure it in the configuration file to use, such as: - -```yaml -Optimizer: - name: MyOptim - args1: args1 - args2: args2 - lr: - name: Cosine - learning_rate: 0.001 - regularizer: - name: 'L2' - factor: 0 -``` diff --git a/doc/doc_en/algorithm_det_ct_en.md b/doc/doc_en/algorithm_det_ct_en.md deleted file mode 100644 index d56b3fc6b3..0000000000 --- a/doc/doc_en/algorithm_det_ct_en.md +++ /dev/null @@ -1,96 +0,0 @@ -# CT - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [CentripetalText: An Efficient Text Instance Representation for Scene Text Detection](https://arxiv.org/abs/2107.05945) -> Tao Sheng, Jie Chen, Zhouhui Lian -> NeurIPS, 2021 - - -On the Total-Text dataset, the text detection result is as follows: - -|Model|Backbone|Configuration|Precision|Recall|Hmean|Download| -| --- | --- | --- | --- | --- | --- | --- | -|CT|ResNet18_vd|[configs/det/det_r18_vd_ct.yml](../../configs/det/det_r18_vd_ct.yml)|88.68%|81.70%|85.05%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r18_ct_train.tar)| - - - -## 2. Environment -Please prepare your environment referring to [prepare the environment](./environment_en.md) and [clone the repo](./clone_en.md). - - - -## 3. Model Training / Evaluation / Prediction - - -The above CT model is trained using the Total-Text text detection public dataset. For the download of the dataset, please refer to [Total-Text-Dataset](https://github.com/cs-chan/Total-Text-Dataset/tree/master/Dataset). PaddleOCR format annotation download link [train.txt](https://paddleocr.bj.bcebos.com/dataset/ct_tipc/train.txt), [test.txt](https://paddleocr.bj.bcebos.com/dataset/ct_tipc/test.txt). - - -Please refer to [text detection training tutorial](./detection_en.md). PaddleOCR has modularized the code structure, so that you only need to **replace the configuration file** to train different detection models. - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, convert the model saved in the CT text detection training process into an inference model. Taking the model based on the Resnet18_vd backbone network and trained on the Total Text English dataset as example ([model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r18_ct_train.tar)), you can use the following command to convert: - -```shell -python3 tools/export_model.py -c configs/det/det_r18_vd_ct.yml -o Global.pretrained_model=./det_r18_ct_train/best_accuracy Global.save_inference_dir=./inference/det_ct -``` - -CT text detection model inference, you can execute the following command: - -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img623.jpg" --det_model_dir="./inference/det_ct/" --det_algorithm="CT" -``` - -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_img623_ct.jpg) - - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@inproceedings{sheng2021centripetaltext, - title={CentripetalText: An Efficient Text Instance Representation for Scene Text Detection}, - author={Tao Sheng and Jie Chen and Zhouhui Lian}, - booktitle={Thirty-Fifth Conference on Neural Information Processing Systems}, - year={2021} -} -``` diff --git a/doc/doc_en/algorithm_det_db_en.md b/doc/doc_en/algorithm_det_db_en.md deleted file mode 100644 index fde344c357..0000000000 --- a/doc/doc_en/algorithm_det_db_en.md +++ /dev/null @@ -1,117 +0,0 @@ -# DB && DB++ - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [Real-time Scene Text Detection with Differentiable Binarization](https://arxiv.org/abs/1911.08947) -> Liao, Minghui and Wan, Zhaoyi and Yao, Cong and Chen, Kai and Bai, Xiang -> AAAI, 2020 - -> [Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion](https://arxiv.org/abs/2202.10304) -> Liao, Minghui and Zou, Zhisheng and Wan, Zhaoyi and Yao, Cong and Bai, Xiang -> TPAMI, 2022 - -On the ICDAR2015 dataset, the text detection result is as follows: - -|Model|Backbone|Configuration|Precision|Recall|Hmean|Download| -| --- | --- | --- | --- | --- | --- | --- | -|DB|ResNet50_vd|[configs/det/det_r50_vd_db.yml](../../configs/det/det_r50_vd_db.yml)|86.41%|78.72%|82.38%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_db_v2.0_train.tar)| -|DB|MobileNetV3|[configs/det/det_mv3_db.yml](../../configs/det/det_mv3_db.yml)|77.29%|73.08%|75.12%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_db_v2.0_train.tar)| -|DB++|ResNet50|[configs/det/det_r50_db++_ic15.yml](../../configs/det/det_r50_db++_ic15.yml)|90.89%|82.66%|86.58%|[pretrained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/ResNet50_dcn_asf_synthtext_pretrained.pdparams)/[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_r50_db%2B%2B_icdar15_train.tar)| - -On the TD_TR dataset, the text detection result is as follows: - -|Model|Backbone|Configuration|Precision|Recall|Hmean|Download| -| --- | --- | --- | --- | --- | --- | --- | -|DB++|ResNet50|[configs/det/det_r50_db++_td_tr.yml](../../configs/det/det_r50_db++_td_tr.yml)|92.92%|86.48%|89.58%|[pretrained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/ResNet50_dcn_asf_synthtext_pretrained.pdparams)/[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_r50_db%2B%2B_td_tr_train.tar)| - - -## 2. Environment -Please prepare your environment referring to [prepare the environment](./environment_en.md) and [clone the repo](./clone_en.md). - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [text detection training tutorial](./detection_en.md). PaddleOCR has modularized the code structure, so that you only need to **replace the configuration file** to train different detection models. - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, convert the model saved in the DB text detection training process into an inference model. Taking the model based on the Resnet50_vd backbone network and trained on the ICDAR2015 English dataset as example ([model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_db_v2.0_train.tar)), you can use the following command to convert: - -```shell -python3 tools/export_model.py -c configs/det/det_r50_vd_db.yml -o Global.pretrained_model=./det_r50_vd_db_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_db -``` - -DB text detection model inference, you can execute the following command: - -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_db/" -``` - -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_img_10_db.jpg) - -**Note**: Since the ICDAR2015 dataset has only 1,000 training images, mainly for English scenes, the above model has very poor detection result on Chinese text images. - - - -### 4.2 C++ Inference - -With the inference model prepared, refer to the [cpp infer](../../deploy/cpp_infer/) tutorial for C++ inference. - - -### 4.3 Serving - -With the inference model prepared, refer to the [pdserving](../../deploy/pdserving/) tutorial for service deployment by Paddle Serving. - - -### 4.4 More - -More deployment schemes supported for DB: - -- Paddle2ONNX: with the inference model prepared, please refer to the [paddle2onnx](../../deploy/paddle2onnx/) tutorial. - - -## 5. FAQ - - -## Citation - -```bibtex -@inproceedings{liao2020real, - title={Real-time scene text detection with differentiable binarization}, - author={Liao, Minghui and Wan, Zhaoyi and Yao, Cong and Chen, Kai and Bai, Xiang}, - booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, - volume={34}, - number={07}, - pages={11474--11481}, - year={2020} -} - -@article{liao2022real, - title={Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion}, - author={Liao, Minghui and Zou, Zhisheng and Wan, Zhaoyi and Yao, Cong and Bai, Xiang}, - journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, - year={2022}, - publisher={IEEE} -} -``` diff --git a/doc/doc_en/algorithm_det_drrg_en.md b/doc/doc_en/algorithm_det_drrg_en.md deleted file mode 100644 index 8d6538a02d..0000000000 --- a/doc/doc_en/algorithm_det_drrg_en.md +++ /dev/null @@ -1,79 +0,0 @@ -# DRRG - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [Deep Relational Reasoning Graph Network for Arbitrary Shape Text Detection](https://arxiv.org/abs/2003.07493) -> Zhang, Shi-Xue and Zhu, Xiaobin and Hou, Jie-Bo and Liu, Chang and Yang, Chun and Wang, Hongfa and Yin, Xu-Cheng -> CVPR, 2020 - -On the CTW1500 dataset, the text detection result is as follows: - -|Model|Backbone|Configuration|Precision|Recall|Hmean|Download| -| --- | --- | --- | --- | --- | --- | --- | -| DRRG | ResNet50_vd | [configs/det/det_r50_drrg_ctw.yml](../../configs/det/det_r50_drrg_ctw.yml)| 89.92%|80.91%|85.18%|[trained model](https://paddleocr.bj.bcebos.com/contribution/det_r50_drrg_ctw_train.tar)| - - -## 2. Environment -Please prepare your environment referring to [prepare the environment](./environment_en.md) and [clone the repo](./clone_en.md). - - - -## 3. Model Training / Evaluation / Prediction - -The above DRRG model is trained using the CTW1500 text detection public dataset. For the download of the dataset, please refer to [ocr_datasets](./dataset/ocr_datasets_en.md). - -After the data download is complete, please refer to [Text Detection Training Tutorial](./detection_en.md) for training. PaddleOCR has modularized the code structure, so that you only need to **replace the configuration file** to train different detection models. - - -## 4. Inference and Deployment - - -### 4.1 Python Inference - -Since the model needs to be converted to Numpy data for many times in the forward, DRRG dynamic graph to static graph is not supported. - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@inproceedings{zhang2020deep, - title={Deep relational reasoning graph network for arbitrary shape text detection}, - author={Zhang, Shi-Xue and Zhu, Xiaobin and Hou, Jie-Bo and Liu, Chang and Yang, Chun and Wang, Hongfa and Yin, Xu-Cheng}, - booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, - pages={9699--9708}, - year={2020} -} -``` diff --git a/doc/doc_en/algorithm_det_east_en.md b/doc/doc_en/algorithm_det_east_en.md deleted file mode 100644 index 85440debfa..0000000000 --- a/doc/doc_en/algorithm_det_east_en.md +++ /dev/null @@ -1,100 +0,0 @@ -# EAST - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [EAST: An Efficient and Accurate Scene Text Detector](https://arxiv.org/abs/1704.03155) -> Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, Jiajun Liang -> CVPR, 2017 - - -On the ICDAR2015 dataset, the text detection result is as follows: - -|Model|Backbone|Configuration|Precision|Recall|Hmean|Download| -| --- | --- | --- | --- | --- | --- | --- | -|EAST|ResNet50_vd| [det_r50_vd_east.yml](../../configs/det/det_r50_vd_east.yml)|88.71%| 81.36%| 84.88%| [model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_east_v2.0_train.tar)| -|EAST|MobileNetV3|[det_mv3_east.yml](../../configs/det/det_mv3_east.yml) | 78.20%| 79.10%| 78.65%| [model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_east_v2.0_train.tar)| - - - - -## 2. Environment -Please prepare your environment referring to [prepare the environment](./environment_en.md) and [clone the repo](./clone_en.md). - - - -## 3. Model Training / Evaluation / Prediction - -The above EAST model is trained using the ICDAR2015 text detection public dataset. For the download of the dataset, please refer to [ocr_datasets](./dataset/ocr_datasets_en.md). - -After the data download is complete, please refer to [Text Detection Training Tutorial](./detection_en.md) for training. PaddleOCR has modularized the code structure, so that you only need to **replace the configuration file** to train different detection models. - - - -## 4. Inference and Deployment - - - -### 4.1 Python Inference - -First, convert the model saved in the EAST text detection training process into an inference model. Taking the model based on the Resnet50_vd backbone network and trained on the ICDAR2015 English dataset as example ([model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_east_v2.0_train.tar)), you can use the following command to convert: - -```shell -python3 tools/export_model.py -c configs/det/det_r50_vd_east.yml -o Global.pretrained_model=./det_r50_vd_east_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_r50_east/ -``` - -For EAST text detection model inference, you need to set the parameter --det_algorithm="EAST", run the following command: -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_r50_east/" --det_algorithm="EAST" -``` - - -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'det_res'. - -![](../imgs_results/det_res_img_10_east.jpg) - - - -### 4.2 C++ Inference - -Since the post-processing is not written in CPP, the EAST text detection model does not support CPP inference. - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@inproceedings{zhou2017east, - title={East: an efficient and accurate scene text detector}, - author={Zhou, Xinyu and Yao, Cong and Wen, He and Wang, Yuzhi and Zhou, Shuchang and He, Weiran and Liang, Jiajun}, - booktitle={Proceedings of the IEEE conference on Computer Vision and Pattern Recognition}, - pages={5551--5560}, - year={2017} -} -``` diff --git a/doc/doc_en/algorithm_det_fcenet_en.md b/doc/doc_en/algorithm_det_fcenet_en.md deleted file mode 100644 index f3c51a91a4..0000000000 --- a/doc/doc_en/algorithm_det_fcenet_en.md +++ /dev/null @@ -1,104 +0,0 @@ -# FCENet - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [Fourier Contour Embedding for Arbitrary-Shaped Text Detection](https://arxiv.org/abs/2104.10442) -> Yiqin Zhu and Jianyong Chen and Lingyu Liang and Zhanghui Kuang and Lianwen Jin and Wayne Zhang -> CVPR, 2021 - -On the CTW1500 dataset, the text detection result is as follows: - -|Model|Backbone|Configuration|Precision|Recall|Hmean|Download| -| --- | --- | --- | --- | --- | --- | --- | -| FCE | ResNet50_dcn | [configs/det/det_r50_vd_dcn_fce_ctw.yml](../../configs/det/det_r50_vd_dcn_fce_ctw.yml)| 88.39%|82.18%|85.27%|[trained model](https://paddleocr.bj.bcebos.com/contribution/det_r50_dcn_fce_ctw_v2.0_train.tar)| - - -## 2. Environment -Please prepare your environment referring to [prepare the environment](./environment_en.md) and [clone the repo](./clone_en.md). - - - -## 3. Model Training / Evaluation / Prediction - -The above FCE model is trained using the CTW1500 text detection public dataset. For the download of the dataset, please refer to [ocr_datasets](./dataset/ocr_datasets_en.md). - -After the data download is complete, please refer to [Text Detection Training Tutorial](./detection_en.md) for training. PaddleOCR has modularized the code structure, so that you only need to **replace the configuration file** to train different detection models. - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, convert the model saved in the FCE text detection training process into an inference model. Taking the model based on the Resnet50_vd_dcn backbone network and trained on the CTW1500 English dataset as example ([model download link](https://paddleocr.bj.bcebos.com/contribution/det_r50_dcn_fce_ctw_v2.0_train.tar)), you can use the following command to convert: - -```shell -python3 tools/export_model.py -c configs/det/det_r50_vd_dcn_fce_ctw.yml -o Global.pretrained_model=./det_r50_dcn_fce_ctw_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_fce -``` - -FCE text detection model inference, to perform non-curved text detection, you can run the following commands: - -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_fce/" --det_algorithm="FCE" --det_fce_box_type=quad -``` - -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_img_10_fce.jpg) - -If you want to perform curved text detection, you can execute the following command: - -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img623.jpg" --det_model_dir="./inference/det_fce/" --det_algorithm="FCE" --det_fce_box_type=poly -``` - -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_img623_fce.jpg) - -**Note**: Since the CTW1500 dataset has only 1,000 training images, mainly for English scenes, the above model has very poor detection result on Chinese or curved text images. - - - -### 4.2 C++ Inference - -Since the post-processing is not written in CPP, the FCE text detection model does not support CPP inference. - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@InProceedings{zhu2021fourier, - title={Fourier Contour Embedding for Arbitrary-Shaped Text Detection}, - author={Yiqin Zhu and Jianyong Chen and Lingyu Liang and Zhanghui Kuang and Lianwen Jin and Wayne Zhang}, - year={2021}, - booktitle = {CVPR} -} -``` diff --git a/doc/doc_en/algorithm_det_psenet_en.md b/doc/doc_en/algorithm_det_psenet_en.md deleted file mode 100644 index 3977a156ac..0000000000 --- a/doc/doc_en/algorithm_det_psenet_en.md +++ /dev/null @@ -1,107 +0,0 @@ -# PSENet - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [Shape robust text detection with progressive scale expansion network](https://arxiv.org/abs/1903.12473) -> Wang, Wenhai and Xie, Enze and Li, Xiang and Hou, Wenbo and Lu, Tong and Yu, Gang and Shao, Shuai -> CVPR, 2019 - -On the ICDAR2015 dataset, the text detection result is as follows: - -|Model|Backbone|Configuration|Precision|Recall|Hmean|Download| -| --- | --- | --- | --- | --- | --- | --- | -|PSE| ResNet50_vd | [configs/det/det_r50_vd_pse.yml](../../configs/det/det_r50_vd_pse.yml)| 85.81% |79.53%|82.55%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_r50_vd_pse_v2.0_train.tar)| -|PSE| MobileNetV3| [configs/det/det_mv3_pse.yml](../../configs/det/det_mv3_pse.yml) | 82.20% |70.48%|75.89%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_mv3_pse_v2.0_train.tar)| - - - -## 2. Environment -Please prepare your environment referring to [prepare the environment](./environment_en.md) and [clone the repo](./clone_en.md). - - - -## 3. Model Training / Evaluation / Prediction - -The above PSE model is trained using the ICDAR2015 text detection public dataset. For the download of the dataset, please refer to [ocr_datasets](./dataset/ocr_datasets_en.md). - -After the data download is complete, please refer to [Text Detection Training Tutorial](./detection_en.md) for training. PaddleOCR has modularized the code structure, so that you only need to **replace the configuration file** to train different detection models. - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, convert the model saved in the PSE text detection training process into an inference model. Taking the model based on the Resnet50_vd backbone network and trained on the ICDAR2015 English dataset as example ([model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_r50_vd_pse_v2.0_train.tar)), you can use the following command to convert: - -```shell -python3 tools/export_model.py -c configs/det/det_r50_vd_pse.yml -o Global.pretrained_model=./det_r50_vd_pse_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_pse -``` - -PSE text detection model inference, to perform non-curved text detection, you can run the following commands: - -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_pse/" --det_algorithm="PSE" --det_pse_box_type=quad -``` - -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_img_10_pse.jpg) - -If you want to perform curved text detection, you can execute the following command: - -```shell -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_pse/" --det_algorithm="PSE" --det_pse_box_type=poly -``` - -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_img_10_pse_poly.jpg) - -**Note**: Since the ICDAR2015 dataset has only 1,000 training images, mainly for English scenes, the above model has very poor detection result on Chinese or curved text images. - - - -### 4.2 C++ Inference - -Since the post-processing is not written in CPP, the PSE text detection model does not support CPP inference. - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@inproceedings{wang2019shape, - title={Shape robust text detection with progressive scale expansion network}, - author={Wang, Wenhai and Xie, Enze and Li, Xiang and Hou, Wenbo and Lu, Tong and Yu, Gang and Shao, Shuai}, - booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, - pages={9336--9345}, - year={2019} -} -``` diff --git a/doc/doc_en/algorithm_det_sast_en.md b/doc/doc_en/algorithm_det_sast_en.md deleted file mode 100644 index b39990d5be..0000000000 --- a/doc/doc_en/algorithm_det_sast_en.md +++ /dev/null @@ -1,118 +0,0 @@ -# SAST - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [A Single-Shot Arbitrarily-Shaped Text Detector based on Context Attended Multi-Task Learning](https://arxiv.org/abs/1908.05498) -> Wang, Pengfei and Zhang, Chengquan and Qi, Fei and Huang, Zuming and En, Mengyi and Han, Junyu and Liu, Jingtuo and Ding, Errui and Shi, Guangming -> ACM MM, 2019 - -On the ICDAR2015 dataset, the text detection result is as follows: - -|Model|Backbone|Configuration|Precision|Recall|Hmean|Download| -| --- | --- | --- | --- | --- | --- | --- | -|SAST|ResNet50_vd|[configs/det/det_r50_vd_sast_icdar15.yml](../../configs/det/det_r50_vd_sast_icdar15.yml)|91.39%|83.77%|87.42%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_icdar15_v2.0_train.tar)| - - -On the Total-text dataset, the text detection result is as follows: - -|Model|Backbone|Configuration|Precision|Recall|Hmean|Download| -| --- | --- | --- | --- | --- | --- | --- | -|SAST|ResNet50_vd|[configs/det/det_r50_vd_sast_totaltext.yml](../../configs/det/det_r50_vd_sast_totaltext.yml)|89.63%|78.44%|83.66%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_totaltext_v2.0_train.tar)| - - - -## 2. Environment -Please prepare your environment referring to [prepare the environment](./environment_en.md) and [clone the repo](./clone_en.md). - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [text detection training tutorial](./detection_en.md). PaddleOCR has modularized the code structure, so that you only need to **replace the configuration file** to train different detection models. - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -#### (1). Quadrangle text detection model (ICDAR2015) -First, convert the model saved in the SAST text detection training process into an inference model. Taking the model based on the Resnet50_vd backbone network and trained on the ICDAR2015 English dataset as an example ([model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_icdar15_v2.0_train.tar)), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_sast_icdar15.yml -o Global.pretrained_model=./det_r50_vd_sast_icdar15_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_sast_ic15 -``` - -**For SAST quadrangle text detection model inference, you need to set the parameter `--det_algorithm="SAST"`**, run the following command: - -``` -python3 tools/infer/predict_det.py --det_algorithm="SAST" --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_sast_ic15/" -``` - -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_img_10_sast.jpg) - -#### (2). Curved text detection model (Total-Text) -First, convert the model saved in the SAST text detection training process into an inference model. Taking the model based on the Resnet50_vd backbone network and trained on the Total-Text English dataset as an example ([model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_totaltext_v2.0_train.tar)), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_sast_totaltext.yml -o Global.pretrained_model=./det_r50_vd_sast_totaltext_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_sast_tt -``` - -For SAST curved text detection model inference, you need to set the parameter `--det_algorithm="SAST"` and `--det_box_type=poly`, run the following command: - -``` -python3 tools/infer/predict_det.py --det_algorithm="SAST" --image_dir="./doc/imgs_en/img623.jpg" --det_model_dir="./inference/det_sast_tt/" --det_box_type='poly' -``` - -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_img623_sast.jpg) - -**Note**: SAST post-processing locality aware NMS has two versions: Python and C++. The speed of C++ version is obviously faster than that of Python version. Due to the compilation version problem of NMS of C++ version, C++ version NMS will be called only in Python 3.5 environment, and python version NMS will be called in other cases. - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@inproceedings{wang2019single, - title={A Single-Shot Arbitrarily-Shaped Text Detector based on Context Attended Multi-Task Learning}, - author={Wang, Pengfei and Zhang, Chengquan and Qi, Fei and Huang, Zuming and En, Mengyi and Han, Junyu and Liu, Jingtuo and Ding, Errui and Shi, Guangming}, - booktitle={Proceedings of the 27th ACM International Conference on Multimedia}, - pages={1277--1285}, - year={2019} -} -``` diff --git a/doc/doc_en/algorithm_e2e_pgnet_en.md b/doc/doc_en/algorithm_e2e_pgnet_en.md deleted file mode 100644 index d03731e393..0000000000 --- a/doc/doc_en/algorithm_e2e_pgnet_en.md +++ /dev/null @@ -1,185 +0,0 @@ -# End-to-end OCR Algorithm-PGNet -- [1. Brief Introduction](#Brief_Introduction) -- [2. Environment Configuration](#Environment_Configuration) -- [3. Quick Use](#Quick_Use) -- [4. Model Training,Evaluation And Inference](#Model_Training_Evaluation_And_Inference) - - -## 1. Brief Introduction -OCR algorithms can be divided into two categories: two-stage algorithm and end-to-end algorithm. The two-stage OCR algorithm is generally divided into two parts, text detection and text recognition algorithm. The text detection algorithm locates the box of the text line from the image, and then the recognition algorithm identifies the content of the text box. The end-to-end OCR algorithm combines text detection and recognition in one algorithm. Its basic idea is to design a model with both detection unit and recognition module, share the CNN features of both and train them together. Because one algorithm can complete character recognition, the end-to-end model is smaller and faster. -### Introduction Of PGNet Algorithm -During the recent years, the end-to-end OCR algorithm has been well developed, including MaskTextSpotter series, TextSnake, TextDragon, PGNet series and so on. Among these algorithms, PGNet algorithm has some advantages over the other algorithms. -- PGNet loss is designed to guide training, and no character-level annotations is needed. -- NMS and ROI related operations are not needed. It can accelerate the prediction -- The reading order prediction module is proposed -- A graph based modification module (GRM) is proposed to further improve the performance of model recognition -- Higher accuracy and faster prediction speed - -For details of PGNet algorithm, please refer to [paper](https://www.aaai.org/AAAI21Papers/AAAI-2885.WangP.pdf). The schematic diagram of the algorithm is as follows: -![](../pgnet_framework.png) -After feature extraction, the input image is sent to four branches: TBO module for text edge offset prediction, TCL module for text center-line prediction, TDO module for text direction offset prediction, and TCC module for text character classification graph prediction. -The output of TBO and TCL can get text detection results after post-processing, and TCL, TDO and TCC are responsible for text recognition. - -The results of detection and recognition are as follows: -![](../imgs_results/e2e_res_img293_pgnet.png) -![](../imgs_results/e2e_res_img295_pgnet.png) -### Performance -#### Test set: Total Text - -#### Test environment: NVIDIA Tesla V100-SXM2-16GB -|PGNetA|det_precision|det_recall|det_f_score|e2e_precision|e2e_recall|e2e_f_score|FPS|download| -| --- | --- | --- | --- | --- | --- | --- | --- | --- | -|Paper|85.30|86.80|86.10|-|-|61.70|38.20 (size=640)|-| -|Ours|87.03|82.48|84.69|61.71|58.43|60.03|48.73 (size=768)|[download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/en_server_pgnetA.tar)| - -*note:PGNet in PaddleOCR optimizes the prediction speed, and can significantly improve the end-to-end prediction speed within the acceptable range of accuracy reduction* - - -## 2. Environment Configuration -Please refer to [Operation Environment Preparation](./environment_en.md) to configure PaddleOCR operating environment first, refer to [Project Clone](./clone_en.md) to clone the project - - -## 3. Quick Use -### Inference model download -This section takes the trained end-to-end model as an example to quickly use the model prediction. First, download the trained end-to-end inference model [download address](https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/e2e_server_pgnetA_infer.tar) -``` -mkdir inference && cd inference -# Download the English end-to-end model and unzip it -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/e2e_server_pgnetA_infer.tar && tar xf e2e_server_pgnetA_infer.tar -``` -* In Windows environment, if 'wget' is not installed, the link can be copied to the browser when downloading the model, and decompressed and placed in the corresponding directory - -After decompression, there should be the following file structure: -``` -├── e2e_server_pgnetA_infer -│ ├── inference.pdiparams -│ ├── inference.pdiparams.info -│ └── inference.pdmodel -``` -### Single image or image set prediction -```bash -# Prediction single image specified by image_dir -python3 tools/infer/predict_e2e.py --e2e_algorithm="PGNet" --image_dir="./doc/imgs_en/img623.jpg" --e2e_model_dir="./inference/e2e_server_pgnetA_infer/" --e2e_pgnet_valid_set="totaltext" - -# Prediction the collection of images specified by image_dir -python3 tools/infer/predict_e2e.py --e2e_algorithm="PGNet" --image_dir="./doc/imgs_en/" --e2e_model_dir="./inference/e2e_server_pgnetA_infer/" --e2e_pgnet_valid_set="totaltext" - -# If you want to use CPU for prediction, you need to set use_gpu parameter is false -python3 tools/infer/predict_e2e.py --e2e_algorithm="PGNet" --image_dir="./doc/imgs_en/img623.jpg" --e2e_model_dir="./inference/e2e_server_pgnetA_infer/" --use_gpu=False --e2e_pgnet_valid_set="totaltext" -``` -### Visualization results -The visualized end-to-end results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'e2e_res'. Examples of results are as follows: -![](../imgs_results/e2e_res_img623_pgnet.jpg) - - -## 4. Model Training,Evaluation And Inference -This section takes the totaltext dataset as an example to introduce the training, evaluation and testing of the end-to-end model in PaddleOCR. - -### Data Preparation -Download and unzip [totaltext](https://paddleocr.bj.bcebos.com/dataset/total_text.tar) dataset to PaddleOCR/train_data/, dataset organization structure is as follow: -``` -/PaddleOCR/train_data/total_text/train/ - |- rgb/ # total_text training data of dataset - |- img11.png - | ... - |- train.txt # total_text training annotation of dataset -``` - -total_text.txt: the format of dimension file is as follows,the file name and annotation information are separated by "\t": -``` -" Image file name Image annotation information encoded by json.dumps" -rgb/img11.jpg [{"transcription": "ASRAMA", "points": [[214.0, 325.0], [235.0, 308.0], [259.0, 296.0], [286.0, 291.0], [313.0, 295.0], [338.0, 305.0], [362.0, 320.0], [349.0, 347.0], [330.0, 337.0], [310.0, 329.0], [290.0, 324.0], [269.0, 328.0], [249.0, 336.0], [231.0, 346.0]]}, {...}] -``` -The image annotation after **json.dumps()** encoding is a list containing multiple dictionaries. - -The `points` in the dictionary represent the coordinates (x, y) of the four points of the text box, arranged clockwise from the point at the upper left corner. - -`transcription` represents the text of the current text box. **When its content is "###" it means that the text box is invalid and will be skipped during training.** - -If you want to train PaddleOCR on other datasets, please build the annotation file according to the above format. - - -### Start Training - -PGNet training is divided into two steps: Step 1: training on the synthetic data to get the pretrain_model, and the accuracy of the model is still low; step 2: loading the pretrain_model and training on the totaltext data set; for fast training, we directly provide the pre training model of step 1[download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/train_step1.tar). -```shell -cd PaddleOCR/ -download step1 pretrain_models -wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/train_step1.tar -You can get the following file format -./pretrain_models/train_step1/ - └─ best_accuracy.pdopt - └─ best_accuracy.states - └─ best_accuracy.pdparams -``` -*If CPU version installed, please set the parameter `use_gpu` to `false` in the configuration.* - -```shell -# single GPU training -python3 tools/train.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.pretrained_model=./pretrain_models/train_step1/best_accuracy Global.load_static_weights=False -# multi-GPU training -# Set the GPU ID used by the '--gpus' parameter. -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.pretrained_model=./pretrain_models/train_step1/best_accuracy Global.load_static_weights=False -``` - -In the above instruction, use `-c` to select the training to use the `configs/e2e/e2e_r50_vd_pg.yml` configuration file. -For a detailed explanation of the configuration file, please refer to [config](./config_en.md). - -You can also use `-o` to change the training parameters without modifying the yml file. For example, adjust the training learning rate to 0.0001 -```shell -python3 tools/train.py -c configs/e2e/e2e_r50_vd_pg.yml -o Optimizer.base_lr=0.0001 -``` - -#### Load trained model and continue training -If you would like to load trained model and continue the training again, you can specify the parameter `Global.checkpoints` as the model path to be loaded. -```shell -python3 tools/train.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.checkpoints=./your/trained/model -``` - -**Note**: The priority of `Global.checkpoints` is higher than that of `Global.pretrain_weights`, that is, when two parameters are specified at the same time, the model specified by `Global.checkpoints` will be loaded first. If the model path specified by `Global.checkpoints` is wrong, the one specified by `Global.pretrain_weights` will be loaded. - -PaddleOCR calculates three indicators for evaluating performance of OCR end-to-end task: Precision, Recall, and Hmean. - - -Run the following code to calculate the evaluation indicators. The result will be saved in the test result file specified by `save_res_path` in the configuration file `e2e_r50_vd_pg.yml` -When evaluating, set post-processing parameters `max_side_len=768`. If you use different datasets, different models for training. -The model parameters during training are saved in the `Global.save_model_dir` directory by default. When evaluating indicators, you need to set `Global.checkpoints` to point to the saved parameter file. -```shell -python3 tools/eval.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.checkpoints="{path/to/weights}/best_accuracy" -``` - -### Model Test -Test the end-to-end result on a single image: -```shell -python3 tools/infer_e2e.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.infer_img="./doc/imgs_en/img_10.jpg" Global.pretrained_model="./output/e2e_pgnet/best_accuracy" Global.load_static_weights=false -``` - -Test the end-to-end result on all images in the folder: -```shell -python3 tools/infer_e2e.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.infer_img="./doc/imgs_en/" Global.pretrained_model="./output/e2e_pgnet/best_accuracy" Global.load_static_weights=false -``` - -### Model inference -#### (1).Quadrangle text detection model (ICDAR2015) -First, convert the model saved in the PGNet end-to-end training process into an inference model. In the first stage of training based on composite dataset, the model of English data set training is taken as an example[model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/en_server_pgnetA.tar), you can use the following command to convert: -``` -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/en_server_pgnetA.tar && tar xf en_server_pgnetA.tar -python3 tools/export_model.py -c configs/e2e/e2e_r50_vd_pg.yml -o Global.pretrained_model=./en_server_pgnetA/best_accuracy Global.load_static_weights=False Global.save_inference_dir=./inference/e2e -``` -**For PGNet quadrangle end-to-end model inference, you need to set the parameter `--e2e_algorithm="PGNet"` and `--e2e_pgnet_valid_set="partvgg"`**, run the following command: -``` -python3 tools/infer/predict_e2e.py --e2e_algorithm="PGNet" --image_dir="./doc/imgs_en/img_10.jpg" --e2e_model_dir="./inference/e2e/" --e2e_pgnet_valid_set="partvgg" -``` -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'e2e_res'. Examples of results are as follows: - -![](../imgs_results/e2e_res_img_10_pgnet.jpg) - -#### (2). Curved text detection model (Total-Text) -For the curved text example, we use the same model as the quadrilateral -**For PGNet end-to-end curved text detection model inference, you need to set the parameter `--e2e_algorithm="PGNet"` and `--e2e_pgnet_valid_set="totaltext"`**, run the following command: -``` -python3 tools/infer/predict_e2e.py --e2e_algorithm="PGNet" --image_dir="./doc/imgs_en/img623.jpg" --e2e_model_dir="./inference/e2e/" --e2e_pgnet_valid_set="totaltext" -``` -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'e2e_res'. Examples of results are as follows: - -![](../imgs_results/e2e_res_img623_pgnet.jpg) diff --git a/doc/doc_en/algorithm_kie_layoutxlm_en.md b/doc/doc_en/algorithm_kie_layoutxlm_en.md deleted file mode 100644 index 0c82b0423b..0000000000 --- a/doc/doc_en/algorithm_kie_layoutxlm_en.md +++ /dev/null @@ -1,194 +0,0 @@ -# KIE Algorithm - LayoutXLM - - -- [1. Introduction](#1-introduction) -- [2. Environment](#2-environment) -- [3. Model Training / Evaluation / Prediction](#3-model-training--evaluation--prediction) -- [4. Inference and Deployment](#4-inference-and-deployment) - - [4.1 Python Inference](#41-python-inference) - - [4.2 C++ Inference](#42-c-inference) - - [4.3 Serving](#43-serving) - - [4.4 More](#44-more) -- [5. FAQ](#5-faq) -- [Citation](#Citation) - - -## 1. Introduction - -Paper: - -> [LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding](https://arxiv.org/abs/2104.08836) -> -> Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei -> -> 2021 - -On XFUND_zh dataset, the algorithm reproduction Hmean is as follows. - -|Model|Backbone|Task |Cnnfig|Hmean|Download link| -| --- | --- |--|--- | --- | --- | -|LayoutXLM|LayoutXLM-base|SER |[ser_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml)|90.38%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)/[inference model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh_infer.tar)| -|LayoutXLM|LayoutXLM-base|RE | [re_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml)|74.83%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar)/[inference model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh_infer.tar)| - - -## 2. Environment - -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [KIE tutorial](./kie_en.md)。PaddleOCR has modularized the code structure, so that you only need to **replace the configuration file** to train different models. - - - -## 4. Inference and Deployment - -### 4.1 Python Inference - -- SER - -First, we need to export the trained model into inference model. Take LayoutXLM model trained on XFUND_zh as an example ([trained model download link](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)). Use the following command to export. - - -``` bash -wget https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar -tar -xf ser_LayoutXLM_xfun_zh.tar -python3 tools/export_model.py -c configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./ser_LayoutXLM_xfun_zh Global.save_inference_dir=./inference/ser_layoutxlm_infer -``` - -Use the following command to infer using LayoutXLM SER model. - -```bash -cd ppstructure -python3 kie/predict_kie_token_ser.py \ - --kie_algorithm=LayoutXLM \ - --ser_model_dir=../inference/ser_layoutxlm_infer \ - --image_dir=./docs/kie/input/zh_val_42.jpg \ - --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \ - --vis_font_path=../doc/fonts/simfang.ttf -``` - -The SER visualization results are saved in the `./output` directory by default. The results are as follows. - - -
- -
- - -- RE - -First, we need to export the trained model into inference model. Take LayoutXLM model trained on XFUND_zh as an example ([trained model download link](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar)). Use the following command to export. - - -``` bash -wget https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar -tar -xf re_LayoutXLM_xfun_zh.tar -python3 tools/export_model.py -c configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./re_LayoutXLM_xfun_zh Global.save_inference_dir=./inference/re_layoutxlm_infer -``` - -Use the following command to infer using LayoutXLM RE model. - - -```bash -cd ppstructure -python3 kie/predict_kie_token_ser_re.py \ - --kie_algorithm=LayoutXLM \ - --re_model_dir=../inference/re_layoutxlm_infer \ - --ser_model_dir=../inference/ser_layoutxlm_infer \ - --image_dir=./docs/kie/input/zh_val_42.jpg \ - --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \ - --vis_font_path=../doc/fonts/simfang.ttf -``` -The RE visualization results are saved in the `./output` directory by default. The results are as follows. - - -
- -
- - -### 4.2 C++ Inference - -Not supported - -### 4.3 Serving - -Not supported - -### 4.4 More - -Not supported - -## 5. FAQ - -## Citation - -```bibtex -@article{DBLP:journals/corr/abs-2104-08836, - author = {Yiheng Xu and - Tengchao Lv and - Lei Cui and - Guoxin Wang and - Yijuan Lu and - Dinei Flor{\^{e}}ncio and - Cha Zhang and - Furu Wei}, - title = {LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich - Document Understanding}, - journal = {CoRR}, - volume = {abs/2104.08836}, - year = {2021}, - url = {https://arxiv.org/abs/2104.08836}, - eprinttype = {arXiv}, - eprint = {2104.08836}, - timestamp = {Thu, 14 Oct 2021 09:17:23 +0200}, - biburl = {https://dblp.org/rec/journals/corr/abs-2104-08836.bib}, - bibsource = {dblp computer science bibliography, https://dblp.org} -} - -@article{DBLP:journals/corr/abs-1912-13318, - author = {Yiheng Xu and - Minghao Li and - Lei Cui and - Shaohan Huang and - Furu Wei and - Ming Zhou}, - title = {LayoutLM: Pre-training of Text and Layout for Document Image Understanding}, - journal = {CoRR}, - volume = {abs/1912.13318}, - year = {2019}, - url = {http://arxiv.org/abs/1912.13318}, - eprinttype = {arXiv}, - eprint = {1912.13318}, - timestamp = {Mon, 01 Jun 2020 16:20:46 +0200}, - biburl = {https://dblp.org/rec/journals/corr/abs-1912-13318.bib}, - bibsource = {dblp computer science bibliography, https://dblp.org} -} - -@article{DBLP:journals/corr/abs-2012-14740, - author = {Yang Xu and - Yiheng Xu and - Tengchao Lv and - Lei Cui and - Furu Wei and - Guoxin Wang and - Yijuan Lu and - Dinei A. F. Flor{\^{e}}ncio and - Cha Zhang and - Wanxiang Che and - Min Zhang and - Lidong Zhou}, - title = {LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding}, - journal = {CoRR}, - volume = {abs/2012.14740}, - year = {2020}, - url = {https://arxiv.org/abs/2012.14740}, - eprinttype = {arXiv}, - eprint = {2012.14740}, - timestamp = {Tue, 27 Jul 2021 09:53:52 +0200}, - biburl = {https://dblp.org/rec/journals/corr/abs-2012-14740.bib}, - bibsource = {dblp computer science bibliography, https://dblp.org} -} -``` diff --git a/doc/doc_en/algorithm_kie_sdmgr_en.md b/doc/doc_en/algorithm_kie_sdmgr_en.md deleted file mode 100644 index ce52ef135e..0000000000 --- a/doc/doc_en/algorithm_kie_sdmgr_en.md +++ /dev/null @@ -1,130 +0,0 @@ - -# KIE Algorithm - SDMGR - -- [1. Introduction](#1-introduction) -- [2. Environment](#2-environment) -- [3. Model Training / Evaluation / Prediction](#3-model-training--evaluation--prediction) -- [4. Inference and Deployment](#4-inference-and-deployment) - - [4.1 Python Inference](#41-python-inference) - - [4.2 C++ Inference](#42-c-inference) - - [4.3 Serving](#43-serving) - - [4.4 More](#44-more) -- [5. FAQ](#5-faq) -- [Citation](#Citation) - -## 1. Introduction - -Paper: - -> [Spatial Dual-Modality Graph Reasoning for Key Information Extraction](https://arxiv.org/abs/2103.14470) -> -> Hongbin Sun and Zhanghui Kuang and Xiaoyu Yue and Chenhao Lin and Wayne Zhang -> -> 2021 - -On wildreceipt dataset, the algorithm reproduction Hmean is as follows. - -|Model|Backbone |Cnnfig|Hmean|Download link| -| --- | --- | --- | --- | --- | -|SDMGR|VGG6|[configs/kie/sdmgr/kie_unet_sdmgr.yml](../../configs/kie/sdmgr/kie_unet_sdmgr.yml)|86.70%|[trained model]( https://paddleocr.bj.bcebos.com/dygraph_v2.1/kie/kie_vgg16.tar)/[inference model(coming soon)]()| - - - -## 2. 环境配置 - -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -SDMGR is a key information extraction algorithm that classifies each detected textline into predefined categories, such as order ID, invoice number, amount, etc. - -The training and test data are collected in the wildreceipt dataset, use following command to downloaded the dataset. - - -```bash -wget https://paddleocr.bj.bcebos.com/ppstructure/dataset/wildreceipt.tar && tar xf wildreceipt.tar -``` - -Create dataset soft link to `PaddleOCR/train_data` directory. - -```bash -cd PaddleOCR/ && mkdir train_data && cd train_data -ln -s ../../wildreceipt ./ -``` - - -### 3.1 Model training - -The config file is `configs/kie/sdmgr/kie_unet_sdmgr.yml`, the default dataset path is `train_data/wildreceipt`. - -Use the following command to train the model. - -```bash -python3 tools/train.py -c configs/kie/sdmgr/kie_unet_sdmgr.yml -o Global.save_model_dir=./output/kie/ -``` - -### 3.2 Model evaluation - -Use the following command to evaluate the model. - -```bash -python3 tools/eval.py -c configs/kie/sdmgr/kie_unet_sdmgr.yml -o Global.checkpoints=./output/kie/best_accuracy -``` - -An example of output information is shown below. - -```py -[2022/08/10 05:22:23] ppocr INFO: metric eval *************** -[2022/08/10 05:22:23] ppocr INFO: hmean:0.8670120239257812 -[2022/08/10 05:22:23] ppocr INFO: fps:10.18816520530961 -``` - -### 3.3 Model prediction - -Use the following command to load the model and predict. During the prediction, the text file storing the image path and OCR information needs to be loaded in advance. Use `Global.infer_img` to assign. - -```bash -python3 tools/infer_kie.py -c configs/kie/kie_unet_sdmgr.yml -o Global.checkpoints=kie_vgg16/best_accuracy Global.infer_img=./train_data/wildreceipt/1.txt -``` - -The visualization results and texts are saved in the `./output/sdmgr_kie/` directory by default. The results are as follows. - - -
- -
- -## 4. Inference and Deployment - -### 4.1 Python Inference - -Not supported - -### 4.2 C++ Inference - -Not supported - -### 4.3 Serving - -Not supported - -### 4.4 More - -Not supported - -## 5. FAQ - -## Citation - -```bibtex -@misc{sun2021spatial, - title={Spatial Dual-Modality Graph Reasoning for Key Information Extraction}, - author={Hongbin Sun and Zhanghui Kuang and Xiaoyu Yue and Chenhao Lin and Wayne Zhang}, - year={2021}, - eprint={2103.14470}, - archivePrefix={arXiv}, - primaryClass={cs.CV} -} -``` diff --git a/doc/doc_en/algorithm_kie_vi_layoutxlm_en.md b/doc/doc_en/algorithm_kie_vi_layoutxlm_en.md deleted file mode 100644 index 4bf55cb3dd..0000000000 --- a/doc/doc_en/algorithm_kie_vi_layoutxlm_en.md +++ /dev/null @@ -1,193 +0,0 @@ -# KIE Algorithm - VI-LayoutXLM - - -- [1. Introduction](#1-introduction) -- [2. Environment](#2-environment) -- [3. Model Training / Evaluation / Prediction](#3-model-training--evaluation--prediction) -- [4. Inference and Deployment](#4-inference-and-deployment) - - [4.1 Python Inference](#41-python-inference) - - [4.2 C++ Inference](#42-c-inference) - - [4.3 Serving](#43-serving) - - [4.4 More](#44-more) -- [5. FAQ](#5-faq) -- [Citation](#Citation) - - -## 1. Introduction - -VI-LayoutXLM is improved based on LayoutXLM. In the process of downstream finetuning, the visual backbone network module is removed, and the model infernce speed is further improved on the basis of almost lossless accuracy. - -On XFUND_zh dataset, the algorithm reproduction Hmean is as follows. - -|Model|Backbone|Task |Config|Hmean|Download link| -| --- | --- |---| --- | --- | --- | -|VI-LayoutXLM |VI-LayoutXLM-base | SER |[ser_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml)|93.19%|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)/[inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar)| -|VI-LayoutXLM |VI-LayoutXLM-base |RE | [re_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml)|83.92%|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar)/[inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_infer.tar)| - - -## 2. Environment - -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [KIE tutorial](./kie_en.md). PaddleOCR has modularized the code structure, so that you only need to **replace the configuration file** to train different models. - - -## 4. Inference and Deployment - -### 4.1 Python Inference - -- SER - -First, we need to export the trained model into inference model. Take VI-LayoutXLM model trained on XFUND_zh as an example ([trained model download link](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)). Use the following command to export. - - -``` bash -wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar -tar -xf ser_vi_layoutxlm_xfund_pretrained.tar -python3 tools/export_model.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./ser_vi_layoutxlm_xfund_pretrained/best_accuracy Global.save_inference_dir=./inference/ser_vi_layoutxlm_infer -``` - -Use the following command to infer using VI-LayoutXLM SER model. - - -```bash -cd ppstructure -python3 kie/predict_kie_token_ser.py \ - --kie_algorithm=LayoutXLM \ - --ser_model_dir=../inference/ser_vi_layoutxlm_infer \ - --image_dir=./docs/kie/input/zh_val_42.jpg \ - --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \ - --vis_font_path=../doc/fonts/simfang.ttf \ - --ocr_order_method="tb-yx" -``` - -The SER visualization results are saved in the `./output` folder by default. The results are as follows. - - -
- -
- - -- RE - -First, we need to export the trained model into inference model. Take VI-LayoutXLM model trained on XFUND_zh as an example ([trained model download link](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar)). Use the following command to export. - - -``` bash -wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar -tar -xf re_vi_layoutxlm_xfund_pretrained.tar -python3 tools/export_model.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./re_vi_layoutxlm_xfund_pretrained/best_accuracy Global.save_inference_dir=./inference/re_vi_layoutxlm_infer -``` - -Use the following command to infer using VI-LayoutXLM RE model. - - -```bash -cd ppstructure -python3 kie/predict_kie_token_ser_re.py \ - --kie_algorithm=LayoutXLM \ - --re_model_dir=../inference/re_vi_layoutxlm_infer \ - --ser_model_dir=../inference/ser_vi_layoutxlm_infer \ - --use_visual_backbone=False \ - --image_dir=./docs/kie/input/zh_val_42.jpg \ - --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \ - --vis_font_path=../doc/fonts/simfang.ttf \ - --ocr_order_method="tb-yx" -``` - -The RE visualization results are saved in the `./output` folder by default. The results are as follows. - - -
- -
- - -### 4.2 C++ Inference - -Not supported - -### 4.3 Serving - -Not supported - -### 4.4 More - -Not supported - -## 5. FAQ - -## Citation - - -```bibtex -@article{DBLP:journals/corr/abs-2104-08836, - author = {Yiheng Xu and - Tengchao Lv and - Lei Cui and - Guoxin Wang and - Yijuan Lu and - Dinei Flor{\^{e}}ncio and - Cha Zhang and - Furu Wei}, - title = {LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich - Document Understanding}, - journal = {CoRR}, - volume = {abs/2104.08836}, - year = {2021}, - url = {https://arxiv.org/abs/2104.08836}, - eprinttype = {arXiv}, - eprint = {2104.08836}, - timestamp = {Thu, 14 Oct 2021 09:17:23 +0200}, - biburl = {https://dblp.org/rec/journals/corr/abs-2104-08836.bib}, - bibsource = {dblp computer science bibliography, https://dblp.org} -} - -@article{DBLP:journals/corr/abs-1912-13318, - author = {Yiheng Xu and - Minghao Li and - Lei Cui and - Shaohan Huang and - Furu Wei and - Ming Zhou}, - title = {LayoutLM: Pre-training of Text and Layout for Document Image Understanding}, - journal = {CoRR}, - volume = {abs/1912.13318}, - year = {2019}, - url = {http://arxiv.org/abs/1912.13318}, - eprinttype = {arXiv}, - eprint = {1912.13318}, - timestamp = {Mon, 01 Jun 2020 16:20:46 +0200}, - biburl = {https://dblp.org/rec/journals/corr/abs-1912-13318.bib}, - bibsource = {dblp computer science bibliography, https://dblp.org} -} - -@article{DBLP:journals/corr/abs-2012-14740, - author = {Yang Xu and - Yiheng Xu and - Tengchao Lv and - Lei Cui and - Furu Wei and - Guoxin Wang and - Yijuan Lu and - Dinei A. F. Flor{\^{e}}ncio and - Cha Zhang and - Wanxiang Che and - Min Zhang and - Lidong Zhou}, - title = {LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding}, - journal = {CoRR}, - volume = {abs/2012.14740}, - year = {2020}, - url = {https://arxiv.org/abs/2012.14740}, - eprinttype = {arXiv}, - eprint = {2012.14740}, - timestamp = {Tue, 27 Jul 2021 09:53:52 +0200}, - biburl = {https://dblp.org/rec/journals/corr/abs-2012-14740.bib}, - bibsource = {dblp computer science bibliography, https://dblp.org} -} -``` diff --git a/doc/doc_en/algorithm_overview_en.md b/doc/doc_en/algorithm_overview_en.md deleted file mode 100755 index 4c893ddcf4..0000000000 --- a/doc/doc_en/algorithm_overview_en.md +++ /dev/null @@ -1,206 +0,0 @@ -# Algorithms - -- [1. Two-stage OCR Algorithms](#1) - - [1.1 Text Detection Algorithms](#11) - - [1.2 Text Recognition Algorithms](#12) - - [1.3 Text Super-Resolution Algorithms](#13) - - [1.4 Formula Recognition Algorithm](#14) -- [2. End-to-end OCR Algorithms](#2) -- [3. Table Recognition Algorithms](#3) -- [4. Key Information Extraction Algorithms](#4) - -This tutorial lists the OCR algorithms supported by PaddleOCR, as well as the models and metrics of each algorithm on **English public datasets**. It is mainly used for algorithm introduction and algorithm performance comparison. For more models on other datasets including Chinese, please refer to [PP-OCRv3 models list](./models_list_en.md). - ->> -Developers are welcome to contribute more algorithms! Please refer to [add new algorithm](./add_new_algorithm_en.md) guideline. - - - - -## 1. Two-stage OCR Algorithms - - - -### 1.1 Text Detection Algorithms - -Supported text detection algorithms (Click the link to get the tutorial): -- [x] [DB && DB++](./algorithm_det_db_en.md) -- [x] [EAST](./algorithm_det_east_en.md) -- [x] [SAST](./algorithm_det_sast_en.md) -- [x] [PSENet](./algorithm_det_psenet_en.md) -- [x] [FCENet](./algorithm_det_fcenet_en.md) -- [x] [DRRG](./algorithm_det_drrg_en.md) -- [x] [CT](./algorithm_det_ct_en.md) - -On the ICDAR2015 dataset, the text detection result is as follows: - -|Model|Backbone|Precision|Recall|Hmean|Download link| -| --- | --- | --- | --- | --- | --- | -|EAST|ResNet50_vd|88.71%|81.36%|84.88%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_east_v2.0_train.tar)| -|EAST|MobileNetV3|78.20%|79.10%|78.65%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_east_v2.0_train.tar)| -|DB|ResNet50_vd|86.41%|78.72%|82.38%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_db_v2.0_train.tar)| -|DB|MobileNetV3|77.29%|73.08%|75.12%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_mv3_db_v2.0_train.tar)| -|SAST|ResNet50_vd|91.39%|83.77%|87.42%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_icdar15_v2.0_train.tar)| -|PSE|ResNet50_vd|85.81%|79.53%|82.55%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_r50_vd_pse_v2.0_train.tar)| -|PSE|MobileNetV3|82.20%|70.48%|75.89%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_mv3_pse_v2.0_train.tar)| -|DB++|ResNet50|90.89%|82.66%|86.58%|[pretrained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/ResNet50_dcn_asf_synthtext_pretrained.pdparams)/[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/en_det/det_r50_db%2B%2B_icdar15_train.tar)| - -On Total-Text dataset, the text detection result is as follows: - -|Model|Backbone|Precision|Recall|Hmean|Download link| -| --- | --- | --- | --- | --- | --- | -|SAST|ResNet50_vd|89.63%|78.44%|83.66%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_totaltext_v2.0_train.tar)| -|CT|ResNet18_vd|88.68%|81.70%|85.05%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r18_ct_train.tar)| - -On CTW1500 dataset, the text detection result is as follows: - -|Model|Backbone|Precision|Recall|Hmean| Download link| -| --- | --- | --- | --- | --- |---| -|FCE|ResNet50_dcn|88.39%|82.18%|85.27%| [trained model](https://paddleocr.bj.bcebos.com/contribution/det_r50_dcn_fce_ctw_v2.0_train.tar) | -|DRRG|ResNet50_vd|89.92%|80.91%|85.18%|[trained model](https://paddleocr.bj.bcebos.com/contribution/det_r50_drrg_ctw_train.tar)| - -**Note:** Additional data, like icdar2013, icdar2017, COCO-Text, ArT, was added to the model training of SAST. Download English public dataset in organized format used by PaddleOCR from: -* [Baidu Drive](https://pan.baidu.com/s/12cPnZcVuV1zn5DOd4mqjVw) (download code: 2bpi). -* [Google Drive](https://drive.google.com/drive/folders/1ll2-XEVyCQLpJjawLDiRlvo_i4BqHCJe?usp=sharing) - - - -### 1.2 Text Recognition Algorithms - -Supported text recognition algorithms (Click the link to get the tutorial): -- [x] [CRNN](./algorithm_rec_crnn_en.md) -- [x] [Rosetta](./algorithm_rec_rosetta_en.md) -- [x] [STAR-Net](./algorithm_rec_starnet_en.md) -- [x] [RARE](./algorithm_rec_rare_en.md) -- [x] [SRN](./algorithm_rec_srn_en.md) -- [x] [NRTR](./algorithm_rec_nrtr_en.md) -- [x] [SAR](./algorithm_rec_sar_en.md) -- [x] [SEED](./algorithm_rec_seed_en.md) -- [x] [SVTR](./algorithm_rec_svtr_en.md) -- [x] [ViTSTR](./algorithm_rec_vitstr_en.md) -- [x] [ABINet](./algorithm_rec_abinet_en.md) -- [x] [VisionLAN](./algorithm_rec_visionlan_en.md) -- [x] [SPIN](./algorithm_rec_spin_en.md) -- [x] [RobustScanner](./algorithm_rec_robustscanner_en.md) -- [x] [RFL](./algorithm_rec_rfl_en.md) -- [x] [ParseQ](./algorithm_rec_parseq.md) -- [x] [CPPD](./algorithm_rec_cppd_en.md) -- [x] [SATRN](./algorithm_rec_satrn_en.md) - -Refer to [DTRB](https://arxiv.org/abs/1904.01906), the training and evaluation result of these above text recognition (using MJSynth and SynthText for training, evaluate on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE) is as follow: - -|Model|Backbone|Avg Accuracy|Module combination|Download link| -|---|---|---|---|---| -|Rosetta|Resnet34_vd|79.11%|rec_r34_vd_none_none_ctc|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_none_ctc_v2.0_train.tar)| -|Rosetta|MobileNetV3|75.80%|rec_mv3_none_none_ctc|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_none_none_ctc_v2.0_train.tar)| -|CRNN|Resnet34_vd|81.04%|rec_r34_vd_none_bilstm_ctc|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar)| -|CRNN|MobileNetV3|77.95%|rec_mv3_none_bilstm_ctc|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_none_bilstm_ctc_v2.0_train.tar)| -|StarNet|Resnet34_vd|82.85%|rec_r34_vd_tps_bilstm_ctc|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_ctc_v2.0_train.tar)| -|StarNet|MobileNetV3|79.28%|rec_mv3_tps_bilstm_ctc|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_tps_bilstm_ctc_v2.0_train.tar)| -|RARE|Resnet34_vd|83.98%|rec_r34_vd_tps_bilstm_att |[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_att_v2.0_train.tar)| -|RARE|MobileNetV3|81.76%|rec_mv3_tps_bilstm_att |[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_tps_bilstm_att_v2.0_train.tar)| -|SRN|Resnet50_vd_fpn| 86.31% | rec_r50fpn_vd_none_srn |[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r50_vd_srn_train.tar)| -|NRTR|NRTR_MTB| 84.21% | rec_mtb_nrtr | [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mtb_nrtr_train.tar) | -|SAR|Resnet31| 87.20% | rec_r31_sar | [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_r31_sar_train.tar) | -|SEED|Aster_Resnet| 85.35% | rec_resnet_stn_bilstm_att | [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_resnet_stn_bilstm_att.tar) | -|SVTR|SVTR-Tiny| 89.25% | rec_svtr_tiny_none_ctc_en | [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar) | -|ViTSTR|ViTSTR| 79.82% | rec_vitstr_none_ce | [trained model](https://paddleocr.bj.bcebos.com/rec_vitstr_none_none_train.tar) | -|ABINet|Resnet45| 90.75% | rec_r45_abinet | [trained model](https://paddleocr.bj.bcebos.com/rec_r45_abinet_train.tar) | -|VisionLAN|Resnet45| 90.30% | rec_r45_visionlan | [trained model](https://paddleocr.bj.bcebos.com/VisionLAN/rec_r45_visionlan_train.tar) | -|SPIN|ResNet32| 90.00% | rec_r32_gaspin_bilstm_att | [trained model](https://paddleocr.bj.bcebos.com/contribution/rec_r32_gaspin_bilstm_att.tar) | -|RobustScanner|ResNet31| 87.77% | rec_r31_robustscanner | [trained model](https://paddleocr.bj.bcebos.com/contribution/rec_r31_robustscanner.tar)| -|RFL|ResNetRFL| 88.63% | rec_resnet_rfl_att | [trained model](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl_att_train.tar) | -|ParseQ|VIT| 91.24% | rec_vit_parseq_synth | [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_synth.tgz) | -|CPPD|SVTR-Base| 93.8% | rec_svtrnet_cppd_base_en | [trained model](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar) | -|SATRN|ShallowCNN| 88.05% | rec_satrn | [trained model](https://pan.baidu.com/s/10J-Bsd881bimKaclKszlaQ?pwd=lk8a) | - - - - -### 1.3 Text Super-Resolution Algorithms - -Supported text super-resolution algorithms (Click the link to get the tutorial): -- [x] [Text Gestalt](./algorithm_sr_gestalt_en.md) -- [x] [Text Telescope](./algorithm_sr_telescope_en.md) - -On the TextZoom public dataset, the effect of the algorithm is as follows: - -|Model|Backbone|PSNR_Avg|SSIM_Avg|Config|Download link| -|---|---|---|---|---|---| -|Text Gestalt|tsrn|19.28|0.6560| [configs/sr/sr_tsrn_transformer_strock.yml](../../configs/sr/sr_tsrn_transformer_strock.yml)|[trained model](https://paddleocr.bj.bcebos.com/sr_tsrn_transformer_strock_train.tar)| -|Text Telescope|tbsrn|21.56|0.7411| [configs/sr/sr_telescope.yml](../../configs/sr/sr_telescope.yml)|[trained model](https://paddleocr.bj.bcebos.com/contribution/sr_telescope_train.tar)| - - - -### 1.4 Formula Recognition Algorithm - -Supported formula recognition algorithms (Click the link to get the tutorial): - -- [x] [CAN](./algorithm_rec_can_en.md) -- [x] [LaTeX-OCR](./algorithm_rec_latex_ocr_en.md) - - -On the CROHME handwritten formula dataset, the effect of the algorithm is as follows: - -|Model |Backbone|Config|ExpRate|Download link| -| ----- | ----- | ----- | ----- | ----- | -|CAN|DenseNet|[rec_d28_can.yml](../../configs/rec/rec_d28_can.yml)|51.72%|[trained model](https://paddleocr.bj.bcebos.com/contribution/rec_d28_can_train.tar)| - - -On the LaTeX-OCR printed formula dataset, the effect of the algorithm is as follows: - -| Model | Backbone |config| BLEU score | normed edit distance | ExpRate |Download link| -|-----------|----------| ---- |:-----------:|:---------------------:|:---------:| ----- | -| LaTeX-OCR | Hybrid ViT |[rec_latex_ocr.yml](../../configs/rec/rec_latex_ocr.yml)| 0.8821 | 0.0823 | 40.01% |[trained model](https://paddleocr.bj.bcebos.com/contribution/rec_latex_ocr_train.tar)| - - - - -## 2. End-to-end OCR Algorithms - -Supported end-to-end algorithms (Click the link to get the tutorial): -- [x] [PGNet](./algorithm_e2e_pgnet_en.md) - - - -## 3. Table Recognition Algorithms - -Supported table recognition algorithms (Click the link to get the tutorial): -- [x] [TableMaster](./algorithm_table_master_en.md) - -On the PubTabNet dataset, the algorithm result is as follows: - -|Model|Backbone|Config|Acc|Download link| -|---|---|---|---|---| -|TableMaster|TableResNetExtra|[configs/table/table_master.yml](../../configs/table/table_master.yml)|77.47%|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/tablemaster/table_structure_tablemaster_train.tar) / [inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/tablemaster/table_structure_tablemaster_infer.tar)| - - - - -## 4. Key Information Extraction Algorithms - -Supported KIE algorithms (Click the link to get the tutorial): - -- [x] [VI-LayoutXLM](./algorithm_kie_vi_layoutxlm_en.md) -- [x] [LayoutLM](./algorithm_kie_layoutxlm_en.md) -- [x] [LayoutLMv2](./algorithm_kie_layoutxlm_en.md) -- [x] [LayoutXLM](./algorithm_kie_layoutxlm_en.md) -- [x] [SDMGR](./algorithm_kie_sdmgr_en.md) - -On wildreceipt dataset, the algorithm result is as follows: - -|Model|Backbone|Config|Hmean|Download link| -| --- | --- | --- | --- | --- | -|SDMGR|VGG6|[configs/kie/sdmgr/kie_unet_sdmgr.yml](../../configs/kie/sdmgr/kie_unet_sdmgr.yml)|86.70%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/kie/kie_vgg16.tar)| - -On XFUND_zh dataset, the algorithm result is as follows: - -|Model|Backbone|Task|Config|Hmean|Download link| -| --- | --- | --- | --- | --- | --- | -|VI-LayoutXLM| VI-LayoutXLM-base | SER | [ser_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml)|**93.19%**|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)| -|LayoutXLM| LayoutXLM-base | SER | [ser_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml)|90.38%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)| -|LayoutLM| LayoutLM-base | SER | [ser_layoutlm_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutlm_xfund_zh.yml)|77.31%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLM_xfun_zh.tar)| -|LayoutLMv2| LayoutLMv2-base | SER | [ser_layoutlmv2_xfund_zh.yml](../../configs/kie/layoutlm_series/ser_layoutlmv2_xfund_zh.yml)|85.44%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLMv2_xfun_zh.tar)| -|VI-LayoutXLM| VI-LayoutXLM-base | RE | [re_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml)|**83.92%**|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar)| -|LayoutXLM| LayoutXLM-base | RE | [re_layoutxlm_xfund_zh.yml](../../configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml)|74.83%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar)| -|LayoutLMv2| LayoutLMv2-base | RE | [re_layoutlmv2_xfund_zh.yml](../../configs/kie/layoutlm_series/re_layoutlmv2_xfund_zh.yml)|67.77%|[trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutLMv2_xfun_zh.tar)| diff --git a/doc/doc_en/algorithm_rec_abinet_en.md b/doc/doc_en/algorithm_rec_abinet_en.md deleted file mode 100644 index 767ca65f64..0000000000 --- a/doc/doc_en/algorithm_rec_abinet_en.md +++ /dev/null @@ -1,136 +0,0 @@ -# ABINet - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [ABINet: Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://openaccess.thecvf.com/content/CVPR2021/papers/Fang_Read_Like_Humans_Autonomous_Bidirectional_and_Iterative_Language_Modeling_for_CVPR_2021_paper.pdf) -> Shancheng Fang and Hongtao Xie and Yuxin Wang and Zhendong Mao and Yongdong Zhang -> CVPR, 2021 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Model|Backbone|config|Acc|Download link| -| --- | --- | --- | --- | --- | -|ABINet|ResNet45|[rec_r45_abinet.yml](../../configs/rec/rec_r45_abinet.yml)|90.75%|[pretrained & trained model](https://paddleocr.bj.bcebos.com/rec_r45_abinet_train.tar)| - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_r45_abinet.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r45_abinet.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r45_abinet.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_r45_abinet.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_r45_abinet_train/best_accuracy -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the ABINet text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/rec_r45_abinet_train.tar)) ), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_r45_abinet.yml -o Global.pretrained_model=./rec_r45_abinet_train/best_accuracy Global.save_inference_dir=./inference/rec_r45_abinet -``` - -**Note:** -- If you are training the model on your own dataset and have modified the dictionary file, please pay attention to modify the `character_dict_path` in the configuration file to the modified dictionary file. -- If you modified the input size during training, please modify the `infer_shape` corresponding to ABINet in the `tools/export_model.py` file. - -After the conversion is successful, there are three files in the directory: -``` -/inference/rec_r45_abinet/ - ├── inference.pdiparams - ├── inference.pdiparams.info - └── inference.pdmodel -``` - - -For ABINet text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_r45_abinet/' --rec_algorithm='ABINet' --rec_image_shape='3,32,128' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' -``` - -![](../imgs_words_en/word_10.png) - -After executing the command, the prediction result (recognized text and score) of the image above is printed to the screen, an example is as follows: -The result is as follows: -```shell -Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9999995231628418) -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - -1. Note that the MJSynth and SynthText datasets come from [ABINet repo](https://github.com/FangShancheng/ABINet). -2. We use the pre-trained model provided by the ABINet authors for finetune training. - -## Citation - -```bibtex -@article{Fang2021ABINet, - title = {ABINet: Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition}, - author = {Shancheng Fang and Hongtao Xie and Yuxin Wang and Zhendong Mao and Yongdong Zhang}, - booktitle = {CVPR}, - year = {2021}, - url = {https://arxiv.org/abs/2103.06495}, - pages = {7098-7107} -} -``` diff --git a/doc/doc_en/algorithm_rec_aster_en.md b/doc/doc_en/algorithm_rec_aster_en.md deleted file mode 100644 index b949cb5b37..0000000000 --- a/doc/doc_en/algorithm_rec_aster_en.md +++ /dev/null @@ -1,122 +0,0 @@ -# STAR-Net - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [STAR-Net: a spatial attention residue network for scene text recognition.](http://www.bmva.org/bmvc/2016/papers/paper043/paper043.pdf) - -> Wei Liu, Chaofeng Chen, Kwan-Yee K. Wong, Zhizhong Su and Junyu Han. - -> BMVC, pages 43.1-43.13, 2016 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Model|Backbone|ACC|config|Download link| -| --- | --- | --- | --- | --- | -|---|---|---|---|---| -|StarNet|Resnet34_vd|84.44%|[configs/rec/rec_r34_vd_tps_bilstm_ctc.yml](../../configs/rec/rec_r34_vd_tps_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_ctc_v2.0_train.tar)| -|StarNet|MobileNetV3|81.42%|[configs/rec/rec_mv3_tps_bilstm_ctc.yml](../../configs/rec/rec_mv3_tps_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_tps_bilstm_ctc_v2.0_train.tar)| - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c rec_r34_vd_tps_bilstm_ctc.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the STAR-Net text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_r31_STAR-Net_train.tar) ), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model=./rec_r34_vd_tps_bilstm_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_starnet -``` - -For STAR-Net text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./inference/rec_starnet/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" -``` - - -### 4.2 C++ Inference - -With the inference model prepared, refer to the [cpp infer](../../deploy/cpp_infer/) tutorial for C++ inference. - - - -### 4.3 Serving - -With the inference model prepared, refer to the [pdserving](../../deploy/pdserving/) tutorial for service deployment by Paddle Serving. - - - -### 4.4 More - -More deployment schemes supported for STAR-Net: - -- Paddle2ONNX: with the inference model prepared, please refer to the [paddle2onnx](../../deploy/paddle2onnx/) tutorial. - - - -## 5. FAQ - - -## Citation - -```bibtex -@inproceedings{liu2016star, - title={STAR-Net: a spatial attention residue network for scene text recognition.}, - author={Liu, Wei and Chen, Chaofeng and Wong, Kwan-Yee K and Su, Zhizhong and Han, Junyu}, - booktitle={BMVC}, - volume={2}, - pages={7}, - year={2016} -} -``` diff --git a/doc/doc_en/algorithm_rec_can_en.md b/doc/doc_en/algorithm_rec_can_en.md deleted file mode 100644 index ef114990a8..0000000000 --- a/doc/doc_en/algorithm_rec_can_en.md +++ /dev/null @@ -1,119 +0,0 @@ -# CAN - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition](https://arxiv.org/abs/2207.11463) -> Bohan Li, Ye Yuan, Dingkang Liang, Xiao Liu, Zhilong Ji, Jinfeng Bai, Wenyu Liu, Xiang Bai -> ECCV, 2022 - -Using CROHME handwrittem mathematical expression recognition datasets for training, and evaluating on its test sets, the algorithm reproduction effect is as follows: - -|Model|Backbone|config|exprate|Download link| -| --- | --- | --- | --- | --- | -|CAN|DenseNet|[rec_d28_can.yml](../../configs/rec/rec_d28_can.yml)|51.72%|[trained model](https://paddleocr.bj.bcebos.com/contribution/rec_d28_can_train.tar)| - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_d28_can.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_d28_can.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_d28_can.yml -o Global.pretrained_model=./rec_d28_can_train/best_accuracy.pdparams -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_d28_can.yml -o Architecture.Head.attdecoder.is_train=False Global.infer_img='./doc/crohme_demo/hme_00.jpg' Global.pretrained_model=./rec_d28_can_train/best_accuracy.pdparams -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the CAN handwritten mathematical expression recognition training process is converted into an inference model. you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_d28_can.yml -o Global.pretrained_model=./rec_d28_can_train/best_accuracy.pdparams Global.save_inference_dir=./inference/rec_d28_can/ Architecture.Head.attdecoder.is_train=False - -# The default output max length of the model is 36. If you need to predict a longer sequence, please specify its output sequence as an appropriate value when exporting the model, as: Architecture.Head.max_ text_ length=72 -``` - -For CAN handwritten mathematical expression recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/datasets/crohme_demo/hme_00.jpg" --rec_algorithm="CAN" --rec_batch_num=1 --rec_model_dir="./inference/rec_d28_can/" --rec_char_dict_path="./ppocr/utils/dict/latex_symbol_dict.txt" - -# If you need to predict on a picture with black characters on a white background, please set: -- rec_ image_ inverse=False -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@misc{https://doi.org/10.48550/arxiv.2207.11463, - doi = {10.48550/ARXIV.2207.11463}, - url = {https://arxiv.org/abs/2207.11463}, - author = {Li, Bohan and Yuan, Ye and Liang, Dingkang and Liu, Xiao and Ji, Zhilong and Bai, Jinfeng and Liu, Wenyu and Bai, Xiang}, - keywords = {Computer Vision and Pattern Recognition (cs.CV), Artificial Intelligence (cs.AI), FOS: Computer and information sciences, FOS: Computer and information sciences}, - title = {When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition}, - publisher = {arXiv}, - year = {2022}, - copyright = {arXiv.org perpetual, non-exclusive license} -} -``` diff --git a/doc/doc_en/algorithm_rec_cppd_en.md b/doc/doc_en/algorithm_rec_cppd_en.md deleted file mode 100644 index 3bc3861988..0000000000 --- a/doc/doc_en/algorithm_rec_cppd_en.md +++ /dev/null @@ -1,171 +0,0 @@ -# CPPD - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) - - -## 1. Introduction - -Paper: -> [Context Perception Parallel Decoder for Scene Text Recognition](https://arxiv.org/abs/2307.12270) -> Yongkun Du and Zhineng Chen and Caiyan Jia and Xiaoting Yin and Chenxia Li and Yuning Du and Yu-Gang Jiang - - - -Scene text recognition models based on deep learning typically follow an Encoder-Decoder structure, where the decoder can be categorized into two types: (1) CTC and (2) Attention-based. Currently, most state-of-the-art (SOTA) models use an Attention-based decoder, which can be further divided into AR and PD types. In general, AR decoders achieve higher recognition accuracy than PD, while PD decoders are faster than AR. CPPD, with carefully designed CO and CC modules, achieves a balance between the accuracy of AR and the speed of PD. - - - -The accuracy (%) and model files of CPPD on the public dataset of scene text recognition are as follows:: - -* English dataset from [PARSeq](https://github.com/baudm/parseq). - - -| Model |IC13
857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg | Download | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:| -| CPPD Tiny | 97.1 | 94.4 | 96.6 | 86.6 | 88.5 | 90.3 | 92.25 | [en](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_tiny_en_train.tar) | -| CPPD Base | 98.2 | 95.5 | 97.6 | 87.9 | 90.0 | 92.7 | 93.80 | [en](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar)| -| CPPD Base 48*160 | 97.5 | 95.5 | 97.7 | 87.7 | 92.4 | 93.7 | 94.10 | [en](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_48_160_en_train.tar) | - -* Trained on Synth dataset(MJ+ST), Test on Union14M-L benchmark from [U14m](https://github.com/Mountchicken/Union14M/). - -| Model |Curve | Multi-
Oriented |Artistic |Contextless| Salient | Multi-
word | General | Avg | Download | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:| -| CPPD Tiny | 52.4 | 12.3 | 48.2 | 54.4 | 61.5 | 53.4 | 61.4 | 49.10 | Same as the table above. | -| CPPD Base | 65.5 | 18.6 | 56.0 | 61.9 | 71.0 | 57.5 | 65.8 | 56.63 | Same as the table above. | -| CPPD Base 48*160 | 71.9 | 22.1 | 60.5 | 67.9 | 78.3 | 63.9 | 67.1 | 61.69 | Same as the table above. | - -* Trained on Union14M-L training dataset. - -| Model |IC13
857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg | Download | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:| -| CPPD Base 32*128 | 98.7 | 98.5 | 99.4 | 91.7 | 96.7 | 99.7 | 97.44 | [en](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_u14m_train.tar) | - -| Model |Curve | Multi-
Oriented |Artistic |Contextless| Salient | Multi-
word | General | Avg | Download | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:| -| CPPD Base 32*128 | 87.5 | 70.7 | 78.2 | 82.9 | 85.5 | 85.4 | 84.3 | 82.08 | Same as the table above. | - -* Chinese dataset from [Chinese Benckmark](https://github.com/FudanVI/benchmarking-chinese-text-recognition). - -| Model | Scene | Web | Document | Handwriting | Avg | Download | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:| -| CPPD Base | 74.4 | 76.1 | 98.6 | 55.3 | 76.10 | [ch](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_ch_train.tar) | -| CPPD Base + STN | 78.4 | 79.3 | 98.9 | 57.6 | 78.55 | [ch](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_stn_ch_train.tar) | - - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - -#### Dataset Preparation - -[English dataset download](https://github.com/baudm/parseq) -[Union14M-Benchmark download](https://github.com/Mountchicken/Union14M) -[Chinese dataset download](https://github.com/fudanvi/benchmarking-chinese-text-recognition#download) - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_svtrnet_cppd_base_en.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_svtrnet_cppd_base_en.yml -``` - -Evaluation: - -You can download the model files and configuration files provided by `CPPD`: [download link](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar), take `CPPD-B` as an example, using the following command to evaluate: - -``` -# Download the tar archive containing the model files and configuration files of CPPD-B and extract it -wget https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar && tar xf rec_svtr_cppd_base_en_train.tar -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c ./rec_svtr_cppd_base_en_train/rec_svtrnet_cppd_base_en.yml -o Global.pretrained_model=./rec_svtr_cppd_base_en_train/best_model -``` - -Prediction: - -``` -python3 tools/infer_rec.py -c ./rec_svtr_cppd_base_en_train/rec_svtrnet_cppd_base_en.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_svtr_cppd_base_en_train/best_model -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the CPPD text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar) ), you can use the following command to convert: - -``` -# export model -# en -python3 tools/export_model.py -c configs/rec/rec_svtrnet_cppd_base_en.yml -o Global.pretrained_model=./rec_svtr_cppd_base_en_train/best_model.pdparams Global.save_inference_dir=./rec_svtr_cppd_base_en_infer -# ch -python3 tools/export_model.py -c configs/rec/rec_svtrnet_cppd_base_ch.yml -o Global.pretrained_model=./rec_svtr_cppd_base_ch_train/best_model.pdparams Global.save_inference_dir=./rec_svtr_cppd_base_ch_infer - -# speed test -# docker image https://hub.docker.com/r/paddlepaddle/paddle/tags/: sudo docker pull paddlepaddle/paddle:2.4.2-gpu-cuda11.2-cudnn8.2-trt8.0 -# install auto_log: pip install https://paddleocr.bj.bcebos.com/libs/auto_log-1.2.0-py3-none-any.whl -# en -python3 tools/infer/predict_rec.py --image_dir='../iiik' --rec_model_dir='./rec_svtr_cppd_base_en_infer/' --rec_algorithm='CPPD' --rec_image_shape='3,32,100' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' --warmup=True --benchmark=True --rec_batch_num=1 --use_tensorrt=True -# ch -python3 tools/infer/predict_rec.py --image_dir='../iiik' --rec_model_dir='./rec_svtr_cppd_base_ch_infer/' --rec_algorithm='CPPDPadding' --rec_image_shape='3,32,256' --warmup=True --benchmark=True --rec_batch_num=1 --use_tensorrt=True -# stn_ch -python3 tools/infer/predict_rec.py --image_dir='../iiik' --rec_model_dir='./rec_svtr_cppd_base_stn_ch_infer/' --rec_algorithm='CPPD' --rec_image_shape='3,64,256' --warmup=True --benchmark=True --rec_batch_num=1 --use_tensorrt=True -``` - -**Note:** -- If you are training the model on your own dataset and have modified the dictionary file, please pay attention to modify the `character_dict_path` in the configuration file to the modified dictionary file. - -After the conversion is successful, there are three files in the directory: -``` -/inference/rec_svtr_cppd_base_en_infer/ - ├── inference.pdiparams - ├── inference.pdiparams.info - └── inference.pdmodel -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## Citation - -```bibtex -@article{Du2023CPPD, - title = {Context Perception Parallel Decoder for Scene Text Recognition}, - author = {Du, Yongkun and Chen, Zhineng and Jia, Caiyan and Yin, Xiaoting and Li, Chenxia and Du, Yuning and Jiang, Yu-Gang}, - booktitle = {Arxiv}, - year = {2023}, - url = {https://arxiv.org/abs/2307.12270} -} -``` diff --git a/doc/doc_en/algorithm_rec_crnn_en.md b/doc/doc_en/algorithm_rec_crnn_en.md deleted file mode 100644 index 8548c2fa62..0000000000 --- a/doc/doc_en/algorithm_rec_crnn_en.md +++ /dev/null @@ -1,123 +0,0 @@ -# CRNN - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://arxiv.org/abs/1507.05717) - -> Baoguang Shi, Xiang Bai, Cong Yao - -> IEEE, 2015 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Model|Backbone|ACC|config|Download link| -| --- | --- | --- | --- | --- | -|---|---|---|---|---| -|CRNN|Resnet34_vd|81.04%|[configs/rec/rec_r34_vd_none_bilstm_ctc.yml](../../configs/rec/rec_r34_vd_none_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar)| -|CRNN|MobileNetV3|77.95%|[configs/rec/rec_mv3_none_bilstm_ctc.yml](../../configs/rec/rec_mv3_none_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_none_bilstm_ctc_v2.0_train.tar)| - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the CRNN text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_r31_CRNN_train.tar) ), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model=./rec_r34_vd_none_bilstm_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_crnn -``` - -For CRNN text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./inference/rec_crnn/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" -``` - - -### 4.2 C++ Inference - -With the inference model prepared, refer to the [cpp infer](../../deploy/cpp_infer/) tutorial for C++ inference. - - - -### 4.3 Serving - -With the inference model prepared, refer to the [pdserving](../../deploy/pdserving/) tutorial for service deployment by Paddle Serving. - - - -### 4.4 More - -More deployment schemes supported for CRNN: - -- Paddle2ONNX: with the inference model prepared, please refer to the [paddle2onnx](../../deploy/paddle2onnx/) tutorial. - - - -## 5. FAQ - - -## Citation - -```bibtex -@ARTICLE{7801919, - author={Shi, Baoguang and Bai, Xiang and Yao, Cong}, - journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, - title={An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition}, - year={2017}, - volume={39}, - number={11}, - pages={2298-2304}, - doi={10.1109/TPAMI.2016.2646371}} -``` diff --git a/doc/doc_en/algorithm_rec_nrtr_en.md b/doc/doc_en/algorithm_rec_nrtr_en.md deleted file mode 100644 index 309d7ab123..0000000000 --- a/doc/doc_en/algorithm_rec_nrtr_en.md +++ /dev/null @@ -1,270 +0,0 @@ -# NRTR - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) -- [6. Release Note](#6) - - -## 1. Introduction - -Paper: -> [NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition](https://arxiv.org/abs/1806.00926) -> Fenfen Sheng and Zhineng Chen and Bo Xu -> ICDAR, 2019 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Model|Backbone|config|Acc|Download link| -| --- | --- | --- | --- | --- | -|NRTR|MTB|[rec_mtb_nrtr.yml](../../configs/rec/rec_mtb_nrtr.yml)|84.21%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mtb_nrtr_train.tar)| - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_mtb_nrtr.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_mtb_nrtr.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_mtb_nrtr.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_mtb_nrtr.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_mtb_nrtr_train/best_accuracy -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the NRTR text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mtb_nrtr_train.tar)) ), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_mtb_nrtr.yml -o Global.pretrained_model=./rec_mtb_nrtr_train/best_accuracy Global.save_inference_dir=./inference/rec_mtb_nrtr -``` - -**Note:** -- If you are training the model on your own dataset and have modified the dictionary file, please pay attention to modify the `character_dict_path` in the configuration file to the modified dictionary file. -- If you modified the input size during training, please modify the `infer_shape` corresponding to NRTR in the `tools/export_model.py` file. - -After the conversion is successful, there are three files in the directory: -``` -/inference/rec_mtb_nrtr/ - ├── inference.pdiparams - ├── inference.pdiparams.info - └── inference.pdmodel -``` - - -For NRTR text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_mtb_nrtr/' --rec_algorithm='NRTR' --rec_image_shape='1,32,100' --rec_char_dict_path='./ppocr/utils/EN_symbol_dict.txt' -``` - -![](../imgs_words_en/word_10.png) - -After executing the command, the prediction result (recognized text and score) of the image above is printed to the screen, an example is as follows: -The result is as follows: -```shell -Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9465042352676392) -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - -1. In the `NRTR` paper, Beam search is used to decode characters, but the speed is slow. Beam search is not used by default here, and greedy search is used to decode characters. - - -## 6. Release Note - -1. The release/2.6 version updates the NRTR code structure. The new version of NRTR can load the model parameters of the old version (release/2.5 and before), and you may use the following code to convert the old version model parameters to the new version model parameters: - -```python - - params = paddle.load('path/' + '.pdparams') # the old version parameters - state_dict = model.state_dict() # the new version model parameters - new_state_dict = {} - - for k1, v1 in state_dict.items(): - - k = k1 - if 'encoder' in k and 'self_attn' in k and 'qkv' in k and 'weight' in k: - - k_para = k[:13] + 'layers.' + k[13:] - q = params[k_para.replace('qkv', 'conv1')].transpose((1, 0, 2, 3)) - k = params[k_para.replace('qkv', 'conv2')].transpose((1, 0, 2, 3)) - v = params[k_para.replace('qkv', 'conv3')].transpose((1, 0, 2, 3)) - - new_state_dict[k1] = np.concatenate([q[:, :, 0, 0], k[:, :, 0, 0], v[:, :, 0, 0]], -1) - - elif 'encoder' in k and 'self_attn' in k and 'qkv' in k and 'bias' in k: - - k_para = k[:13] + 'layers.' + k[13:] - q = params[k_para.replace('qkv', 'conv1')] - k = params[k_para.replace('qkv', 'conv2')] - v = params[k_para.replace('qkv', 'conv3')] - - new_state_dict[k1] = np.concatenate([q, k, v], -1) - - elif 'encoder' in k and 'self_attn' in k and 'out_proj' in k: - - k_para = k[:13] + 'layers.' + k[13:] - new_state_dict[k1] = params[k_para] - - elif 'encoder' in k and 'norm3' in k: - k_para = k[:13] + 'layers.' + k[13:] - new_state_dict[k1] = params[k_para.replace('norm3', 'norm2')] - - elif 'encoder' in k and 'norm1' in k: - k_para = k[:13] + 'layers.' + k[13:] - new_state_dict[k1] = params[k_para] - - - elif 'decoder' in k and 'self_attn' in k and 'qkv' in k and 'weight' in k: - k_para = k[:13] + 'layers.' + k[13:] - q = params[k_para.replace('qkv', 'conv1')].transpose((1, 0, 2, 3)) - k = params[k_para.replace('qkv', 'conv2')].transpose((1, 0, 2, 3)) - v = params[k_para.replace('qkv', 'conv3')].transpose((1, 0, 2, 3)) - new_state_dict[k1] = np.concatenate([q[:, :, 0, 0], k[:, :, 0, 0], v[:, :, 0, 0]], -1) - - elif 'decoder' in k and 'self_attn' in k and 'qkv' in k and 'bias' in k: - k_para = k[:13] + 'layers.' + k[13:] - q = params[k_para.replace('qkv', 'conv1')] - k = params[k_para.replace('qkv', 'conv2')] - v = params[k_para.replace('qkv', 'conv3')] - new_state_dict[k1] = np.concatenate([q, k, v], -1) - - elif 'decoder' in k and 'self_attn' in k and 'out_proj' in k: - - k_para = k[:13] + 'layers.' + k[13:] - new_state_dict[k1] = params[k_para] - - elif 'decoder' in k and 'cross_attn' in k and 'q' in k and 'weight' in k: - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('cross_attn', 'multihead_attn') - q = params[k_para.replace('q', 'conv1')].transpose((1, 0, 2, 3)) - new_state_dict[k1] = q[:, :, 0, 0] - - elif 'decoder' in k and 'cross_attn' in k and 'q' in k and 'bias' in k: - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('cross_attn', 'multihead_attn') - q = params[k_para.replace('q', 'conv1')] - new_state_dict[k1] = q - - elif 'decoder' in k and 'cross_attn' in k and 'kv' in k and 'weight' in k: - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('cross_attn', 'multihead_attn') - k = params[k_para.replace('kv', 'conv2')].transpose((1, 0, 2, 3)) - v = params[k_para.replace('kv', 'conv3')].transpose((1, 0, 2, 3)) - new_state_dict[k1] = np.concatenate([k[:, :, 0, 0], v[:, :, 0, 0]], -1) - - elif 'decoder' in k and 'cross_attn' in k and 'kv' in k and 'bias' in k: - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('cross_attn', 'multihead_attn') - k = params[k_para.replace('kv', 'conv2')] - v = params[k_para.replace('kv', 'conv3')] - new_state_dict[k1] = np.concatenate([k, v], -1) - - elif 'decoder' in k and 'cross_attn' in k and 'out_proj' in k: - - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('cross_attn', 'multihead_attn') - new_state_dict[k1] = params[k_para] - elif 'decoder' in k and 'norm' in k: - k_para = k[:13] + 'layers.' + k[13:] - new_state_dict[k1] = params[k_para] - elif 'mlp' in k and 'weight' in k: - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('fc', 'conv') - k_para = k_para.replace('mlp.', '') - w = params[k_para].transpose((1, 0, 2, 3)) - new_state_dict[k1] = w[:, :, 0, 0] - elif 'mlp' in k and 'bias' in k: - k_para = k[:13] + 'layers.' + k[13:] - k_para = k_para.replace('fc', 'conv') - k_para = k_para.replace('mlp.', '') - w = params[k_para] - new_state_dict[k1] = w - - else: - new_state_dict[k1] = params[k1] - - if list(new_state_dict[k1].shape) != list(v1.shape): - print(k1) - - - for k, v1 in state_dict.items(): - if k not in new_state_dict.keys(): - print(1, k) - elif list(new_state_dict[k].shape) != list(v1.shape): - print(2, k) - - - - model.set_state_dict(new_state_dict) - paddle.save(model.state_dict(), 'nrtrnew_from_old_params.pdparams') - -``` - -2. The new version has a clean code structure and improved inference speed compared with the old version. - -## Citation - -```bibtex -@article{Sheng2019NRTR, - title = {NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition}, - author = {Fenfen Sheng and Zhineng Chen and Bo Xu}, - booktitle = {ICDAR}, - year = {2019}, - url = {http://arxiv.org/abs/1806.00926}, - pages = {781-786} -} -``` diff --git a/doc/doc_en/algorithm_rec_parseq_en.md b/doc/doc_en/algorithm_rec_parseq_en.md deleted file mode 100644 index 618095ec75..0000000000 --- a/doc/doc_en/algorithm_rec_parseq_en.md +++ /dev/null @@ -1,123 +0,0 @@ -# PasreQ - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://arxiv.org/abs/2207.06966) -> Darwin Bautista, Rowel Atienza -> ECCV, 2021 - -Using real datasets (real) and synthetic datsets (synth) for training respectively,and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets. -- The real datasets include COCO-Text, RCTW17, Uber-Text, ArT, LSVT, MLT19, ReCTS, TextOCR and OpenVINO datasets. -- The synthesis datasets include MJSynth and SynthText datasets. - -the algorithm reproduction effect is as follows: - -|Training Dataset|Model|Backbone|config|Acc|Download link| -| --- | --- | --- | --- | --- | --- | -|Synth|ParseQ|VIT|[rec_vit_parseq.yml](../../configs/rec/rec_vit_parseq.yml)|91.24%|[train model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_synth.tgz)| -|Real|ParseQ|VIT|[rec_vit_parseq.yml](../../configs/rec/rec_vit_parseq.yml)|94.74%|[train model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_real.tgz)| -||||||| - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_vit_parseq.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_vit_parseq.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_vit_parseq.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_vit_parseq.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the SAR text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_real.tgz) ), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_vit_parseq.yml -o Global.pretrained_model=./rec_vit_parseq_real/best_accuracy Global.save_inference_dir=./inference/rec_parseq -``` - -For SAR text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_parseq/" --rec_image_shape="3, 32, 128" --rec_algorithm="ParseQ" --rec_char_dict_path="ppocr/utils/dict/parseq_dict.txt" --max_text_length=25 --use_space_char=False -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@InProceedings{bautista2022parseq, - title={Scene Text Recognition with Permuted Autoregressive Sequence Models}, - author={Bautista, Darwin and Atienza, Rowel}, - booktitle={European Conference on Computer Vision}, - pages={178--196}, - month={10}, - year={2022}, - publisher={Springer Nature Switzerland}, - address={Cham}, - doi={10.1007/978-3-031-19815-1_11}, - url={https://doi.org/10.1007/978-3-031-19815-1_11} -} -``` diff --git a/doc/doc_en/algorithm_rec_rare_en.md b/doc/doc_en/algorithm_rec_rare_en.md deleted file mode 100644 index a756ac75b3..0000000000 --- a/doc/doc_en/algorithm_rec_rare_en.md +++ /dev/null @@ -1,119 +0,0 @@ -# RARE - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper information: -> [Robust Scene Text Recognition with Automatic Rectification](https://arxiv.org/abs/1603.03915v2) -> Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, Xiang Bai∗ -> CVPR, 2016 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Models|Backbone Networks|Configuration Files|Avg Accuracy|Download Links| -| --- | --- | --- | --- | --- | -|RARE|Resnet34_vd|[configs/rec/rec_r34_vd_tps_bilstm_att.yml](../../configs/rec/rec_r34_vd_tps_bilstm_att.yml)|83.60%|[training model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_att_v2.0_train.tar)| -|RARE|MobileNetV3|[configs/rec/rec_mv3_tps_bilstm_att.yml](../../configs/rec/rec_mv3_tps_bilstm_att.yml)|82.50%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_tps_bilstm_att_v2.0_train.tar)| - - - -## 2. Environment -Please refer to [Operating Environment Preparation](./environment_en.md) to configure the PaddleOCR operating environment, and refer to [Project Clone](./clone_en.md) to clone the project code. - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Training Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. Take the backbone network based on Resnet34_vd as an example: - - -### 3.1 Training - -```` -#Single card training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -#Multi-card training, specify the card number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -```` - - -### 3.2 Evaluation - -```` -# GPU evaluation, Global.pretrained_model is the model to be evaluated -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -```` - - -### 3.3 Prediction - -```` -python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -```` - - -## 4. Inference - - -### 4.1 Python Inference -First, convert the model saved during the RARE text recognition training process into an inference model. Take the model trained on the MJSynth and SynthText text recognition datasets based on the Resnet34_vd backbone network as an example ([Model download address](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_att_v2.0_train.tar) ), which can be converted using the following command: - -```shell -python3 tools/export_model.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -o Global.pretrained_model=./rec_r34_vd_tps_bilstm_att_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_rare -```` - -RARE text recognition model inference, you can execute the following commands: - -```shell -python3 tools/infer/predict_rec.py --image_dir="doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_rare/" --rec_image_shape="3, 32, 100" --rec_char_dict_path= "./ppocr/utils/ic15_dict.txt" -```` -The inference results are as follows: - -![](../../doc/imgs_words/en/word_1.png) - -```` -Predicts of doc/imgs_words/en/word_1.png:('joint ', 0.9999969601631165) -```` - - -### 4.2 C++ Inference - -Not currently supported - - -### 4.3 Serving - -Not currently supported - - -### 4.4 More - -The RARE model also supports the following inference deployment methods: - -- Paddle2ONNX Inference: After preparing the inference model, refer to the [paddle2onnx](../../deploy/paddle2onnx/) tutorial. - - -## 5. FAQ - -## Quote - -````bibtex -@inproceedings{2016Robust, - title={Robust Scene Text Recognition with Automatic Rectification}, - author={ Shi, B. and Wang, X. and Lyu, P. and Cong, Y. and Xiang, B. }, - booktitle={2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, - year={2016}, -} -```` diff --git a/doc/doc_en/algorithm_rec_rfl_en.md b/doc/doc_en/algorithm_rec_rfl_en.md deleted file mode 100644 index 36a93cb59e..0000000000 --- a/doc/doc_en/algorithm_rec_rfl_en.md +++ /dev/null @@ -1,143 +0,0 @@ -# RFL - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [Reciprocal Feature Learning via Explicit and Implicit Tasks in Scene Text Recognition](https://arxiv.org/abs/2105.06229.pdf) -> Hui Jiang, Yunlu Xu, Zhanzhan Cheng, Shiliang Pu, Yi Niu, Wenqi Ren, Fei Wu, and Wenming Tan -> ICDAR, 2021 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Model|Backbone|config|Acc|Download link| -| --- | --- | --- | --- | --- | -|RFL-CNT|ResNetRFL|[rec_resnet_rfl_visual.yml](../../configs/rec/rec_resnet_rfl_visual.yml)|93.40%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl_visual_train.tar)| -|RFL-Att|ResNetRFL|[rec_resnet_rfl_att.yml](../../configs/rec/rec_resnet_rfl_att.yml)|88.63%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl_att_train.tar)| - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#step1:train the CNT branch -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_resnet_rfl_visual.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_resnet_rfl_visual.yml - -#step2:joint training of CNT and Att branches -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy - - -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model={path/to/weights}/best_accuracy -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the RFL text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl.tar)) ), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/rec_resnet_rfl_att -``` - -**Note:** -- If you are training the model on your own dataset and have modified the dictionary file, please pay attention to modify the `character_dict_path` in the configuration file to the modified dictionary file. -- If you modified the input size during training, please modify the `infer_shape` corresponding to NRTR in the `tools/export_model.py` file. - -After the conversion is successful, there are three files in the directory: -``` -/inference/rec_resnet_rfl_att/ - ├── inference.pdiparams - ├── inference.pdiparams.info - └── inference.pdmodel -``` - - -For RFL text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_resnet_rfl_att/' --rec_algorithm='RFL' --rec_image_shape='1,32,100' -``` - -![](../imgs_words_en/word_10.png) - -After executing the command, the prediction result (recognized text and score) of the image above is printed to the screen, an example is as follows: -The result is as follows: -```shell -Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9999927282333374) -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - -## Citation - -```bibtex -@article{2021Reciprocal, - title = {Reciprocal Feature Learning via Explicit and Implicit Tasks in Scene Text Recognition}, - author = {Jiang, H. and Xu, Y. and Cheng, Z. and Pu, S. and Niu, Y. and Ren, W. and Wu, F. and Tan, W. }, - booktitle = {ICDAR}, - year = {2021}, - url = {https://arxiv.org/abs/2105.06229} -} -``` diff --git a/doc/doc_en/algorithm_rec_robustscanner_en.md b/doc/doc_en/algorithm_rec_robustscanner_en.md deleted file mode 100644 index 99372d5130..0000000000 --- a/doc/doc_en/algorithm_rec_robustscanner_en.md +++ /dev/null @@ -1,114 +0,0 @@ -# RobustScanner - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition](https://arxiv.org/pdf/2007.07542.pdf) -> Xiaoyu Yue, Zhanghui Kuang, Chenhao Lin, Hongbin Sun, Wayne -Zhang -> ECCV, 2020 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Model|Backbone|config|Acc|Download link| -| --- | --- | --- | --- | --- | -|RobustScanner|ResNet31|[rec_r31_robustscanner.yml](../../configs/rec/rec_r31_robustscanner.yml)|87.77%|[trained model](https://paddleocr.bj.bcebos.com/contribution/rec_r31_robustscanner.tar)| - -Note:In addition to using the two text recognition datasets MJSynth and SynthText, [SynthAdd](https://pan.baidu.com/share/init?surl=uV0LtoNmcxbO-0YA7Ch4dg) data (extraction code: 627x), and some real data are used in training, the specific data details can refer to the paper. - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_r31_robustscanner.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r31_robustscanner.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r31_robustscanner.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_r31_robustscanner.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the RobustScanner text recognition training process is converted into an inference model. you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_r31_robustscanner.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/rec_r31_robustscanner -``` - -For RobustScanner text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_r31_robustscanner/" --rec_image_shape="3, 48, 48, 160" --rec_algorithm="RobustScanner" --rec_char_dict_path="ppocr/utils/dict90.txt" --use_space_char=False -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@article{2020RobustScanner, - title={RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition}, - author={Xiaoyu Yue and Zhanghui Kuang and Chenhao Lin and Hongbin Sun and Wayne Zhang}, - journal={ECCV2020}, - year={2020}, -} -``` diff --git a/doc/doc_en/algorithm_rec_rosetta_en.md b/doc/doc_en/algorithm_rec_rosetta_en.md deleted file mode 100644 index 2a1d7b3127..0000000000 --- a/doc/doc_en/algorithm_rec_rosetta_en.md +++ /dev/null @@ -1,121 +0,0 @@ -# Rosetta - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper information: -> [Rosetta: Large Scale System for Text Detection and Recognition in Images](https://arxiv.org/abs/1910.05085) -> Borisyuk F , Gordo A , V Sivakumar -> KDD, 2018 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Models|Backbone Networks|Configuration Files|Avg Accuracy|Download Links| -| --- | --- | --- | --- | --- | -|Rosetta|Resnet34_vd|[configs/rec/rec_r34_vd_none_none_ctc.yml](../../configs/rec/rec_r34_vd_none_none_ctc.yml)|79.11%|[training model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_none_ctc_v2.0_train.tar)| -|Rosetta|MobileNetV3|[configs/rec/rec_mv3_none_none_ctc.yml](../../configs/rec/rec_mv3_none_none_ctc.yml)|75.80%|[training model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_none_none_ctc_v2.0_train.tar)| - - - -## 2. Environment -Please refer to [Operating Environment Preparation](./environment_en.md) to configure the PaddleOCR operating environment, and refer to [Project Clone](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Training Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. Take the backbone network based on Resnet34_vd as an example: - - -### 3.1 Training - -```` -#Single card training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -#Multi-card training, specify the card number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -```` - - -### 3.2 Evaluation - -```` -# GPU evaluation, Global.pretrained_model is the model to be evaluated -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -```` - - -### 3.3 Prediction - -```` -python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -```` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, convert the model saved during the Rosetta text recognition training process into an inference model. Take the model trained on the MJSynth and SynthText text recognition datasets based on the Resnet34_vd backbone network as an example ( [Model download address](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_none_ctc_v2.0_train.tar) ), which can be converted using the following command: - -```shell -python3 tools/export_model.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -o Global.pretrained_model=./rec_r34_vd_none_none_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_rosetta -```` - -Rosetta text recognition model inference, you can execute the following commands: - -```shell -python3 tools/infer/predict_rec.py --image_dir="doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_rosetta/" --rec_image_shape="3, 32, 100" --rec_char_dict_path= "./ppocr/utils/ic15_dict.txt" -```` - -The inference results are as follows: - -![](../../doc/imgs_words/en/word_1.png) - -```` -Predicts of doc/imgs_words/en/word_1.png:('joint', 0.9999982714653015) -```` - - -### 4.2 C++ Inference - -Not currently supported - - -### 4.3 Serving - -Not currently supported - - -### 4.4 More - -The Rosetta model also supports the following inference deployment methods: - -- Paddle2ONNX Inference: After preparing the inference model, refer to the [paddle2onnx](../../deploy/paddle2onnx/) tutorial. - - -## 5. FAQ - -## Quote - -````bibtex -@inproceedings{2018Rosetta, - title={Rosetta: Large Scale System for Text Detection and Recognition in Images}, - author={ Borisyuk, Fedor and Gordo, Albert and Sivakumar, Viswanath }, - booktitle={the 24th ACM SIGKDD International Conference}, - year={2018}, -} -```` diff --git a/doc/doc_en/algorithm_rec_sar_en.md b/doc/doc_en/algorithm_rec_sar_en.md deleted file mode 100644 index 5c8319da3b..0000000000 --- a/doc/doc_en/algorithm_rec_sar_en.md +++ /dev/null @@ -1,114 +0,0 @@ -# SAR - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition](https://arxiv.org/abs/1811.00751) -> Hui Li, Peng Wang, Chunhua Shen, Guyu Zhang -> AAAI, 2019 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Model|Backbone|config|Acc|Download link| -| --- | --- | --- | --- | --- | -|SAR|ResNet31|[rec_r31_sar.yml](../../configs/rec/rec_r31_sar.yml)|87.20%|[train model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_r31_sar_train.tar)| - -Note:In addition to using the two text recognition datasets MJSynth and SynthText, [SynthAdd](https://pan.baidu.com/share/init?surl=uV0LtoNmcxbO-0YA7Ch4dg) data (extraction code: 627x), and some real data are used in training, the specific data details can refer to the paper. - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_r31_sar.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r31_sar.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r31_sar.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_r31_sar.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the SAR text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_r31_sar_train.tar) ), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_r31_sar.yml -o Global.pretrained_model=./rec_r31_sar_train/best_accuracy Global.save_inference_dir=./inference/rec_sar -``` - -For SAR text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_sar/" --rec_image_shape="3, 48, 48, 160" --rec_algorithm="SAR" --rec_char_dict_path="ppocr/utils/dict90.txt" --max_text_length=30 --use_space_char=False -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@article{Li2019ShowAA, - title={Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition}, - author={Hui Li and Peng Wang and Chunhua Shen and Guyu Zhang}, - journal={ArXiv}, - year={2019}, - volume={abs/1811.00751} -} -``` diff --git a/doc/doc_en/algorithm_rec_satrn_en.md b/doc/doc_en/algorithm_rec_satrn_en.md deleted file mode 100644 index acc95ac035..0000000000 --- a/doc/doc_en/algorithm_rec_satrn_en.md +++ /dev/null @@ -1,111 +0,0 @@ -# SATRN - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -论文信息: -> [On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention](https://arxiv.org/abs/1910.04396) -> Junyeop Lee, Sungrae Park, Jeonghun Baek, Seong Joon Oh, Seonghyeon Kim, Hwalsuk Lee -> CVPR, 2020 -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Model|Backbone|config|Acc|Download link| -| --- | --- | --- | --- | --- | -|SATRN|ShallowCNN|88.05%|[configs/rec/rec_satrn.yml](../../configs/rec/rec_satrn.yml)|[训练模型](https://pan.baidu.com/s/10J-Bsd881bimKaclKszlaQ?pwd=lk8a)| - - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_satrn.yml -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_satrn.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_satrn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_satrn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the SATRN text recognition training process is converted into an inference model. ( [Model download link](https://pan.baidu.com/s/10J-Bsd881bimKaclKszlaQ?pwd=lk8a) ), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_satrn.yml -o Global.pretrained_model=./rec_satrn_train/best_accuracy Global.save_inference_dir=./inference/rec_satrn -``` - -For SATRN text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_satrn/" --rec_image_shape="3, 48, 48, 160" --rec_algorithm="SATRN" --rec_char_dict_path="ppocr/utils/dict90.txt" --max_text_length=30 --use_space_char=False -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - -## 引用 - -```bibtex -@article{lee2019recognizing, - title={On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention}, - author={Junyeop Lee and Sungrae Park and Jeonghun Baek and Seong Joon Oh and Seonghyeon Kim and Hwalsuk Lee}, - year={2019}, - eprint={1910.04396}, - archivePrefix={arXiv}, - primaryClass={cs.CV} -} -``` diff --git a/doc/doc_en/algorithm_rec_seed_en.md b/doc/doc_en/algorithm_rec_seed_en.md deleted file mode 100644 index 83cadfceac..0000000000 --- a/doc/doc_en/algorithm_rec_seed_en.md +++ /dev/null @@ -1,111 +0,0 @@ -# SEED - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://arxiv.org/pdf/2005.10977.pdf) - -> Qiao, Zhi and Zhou, Yu and Yang, Dongbao and Zhou, Yucan and Wang, Weiping - -> CVPR, 2020 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Model|Backbone|ACC|config|Download link| -| --- | --- | --- | --- | --- | -|SEED|Aster_Resnet| 85.20% | [configs/rec/rec_resnet_stn_bilstm_att.yml](../../configs/rec/rec_resnet_stn_bilstm_att.yml) | [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_resnet_stn_bilstm_att.tar) | - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -The SEED model needs to additionally load the [language model](https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.bin.gz) trained by FastText, and install the fasttext dependencies: - -``` -python3 -m pip install fasttext==0.9.1 -``` - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_resnet_stn_bilstm_att.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c rec_resnet_stn_bilstm_att.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_resnet_stn_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_resnet_stn_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference - -Not support - - -### 4.2 C++ Inference - -Not support - - -### 4.3 Serving - -Not support - - -### 4.4 More - -Not support - - -## 5. FAQ - - -## Citation - -```bibtex -@inproceedings{qiao2020seed, - title={Seed: Semantics enhanced encoder-decoder framework for scene text recognition}, - author={Qiao, Zhi and Zhou, Yu and Yang, Dongbao and Zhou, Yucan and Wang, Weiping}, - booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, - pages={13528--13537}, - year={2020} -} -``` diff --git a/doc/doc_en/algorithm_rec_spin_en.md b/doc/doc_en/algorithm_rec_spin_en.md deleted file mode 100644 index 3aea580976..0000000000 --- a/doc/doc_en/algorithm_rec_spin_en.md +++ /dev/null @@ -1,112 +0,0 @@ -# SPIN: Structure-Preserving Inner Offset Network for Scene Text Recognition - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [SPIN: Structure-Preserving Inner Offset Network for Scene Text Recognition](https://arxiv.org/abs/2005.13117) -> Chengwei Zhang, Yunlu Xu, Zhanzhan Cheng, Shiliang Pu, Yi Niu, Fei Wu, Futai Zou -> AAAI, 2020 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets. The algorithm reproduction effect is as follows: - -|Model|Backbone|config|Acc|Download link| -| --- | --- | --- | --- | --- | -|SPIN|ResNet32|[rec_r32_gaspin_bilstm_att.yml](../../configs/rec/rec_r32_gaspin_bilstm_att.yml)|90.00%|[trained model](https://paddleocr.bj.bcebos.com/contribution/rec_r32_gaspin_bilstm_att.tar) | - - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the SPIN text recognition training process is converted into an inference model. you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/rec_r32_gaspin_bilstm_att -``` - -For SPIN text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_r32_gaspin_bilstm_att/" --rec_image_shape="3, 32, 100" --rec_algorithm="SPIN" --rec_char_dict_path="/ppocr/utils/dict/spin_dict.txt" --use_space_char=False -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@article{2020SPIN, - title={SPIN: Structure-Preserving Inner Offset Network for Scene Text Recognition}, - author={Chengwei Zhang and Yunlu Xu and Zhanzhan Cheng and Shiliang Pu and Yi Niu and Fei Wu and Futai Zou}, - journal={AAAI2020}, - year={2020}, -} -``` diff --git a/doc/doc_en/algorithm_rec_srn_en.md b/doc/doc_en/algorithm_rec_srn_en.md deleted file mode 100644 index 1d7fc07dc2..0000000000 --- a/doc/doc_en/algorithm_rec_srn_en.md +++ /dev/null @@ -1,113 +0,0 @@ -# SRN - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [Towards Accurate Scene Text Recognition with Semantic Reasoning Networks](https://arxiv.org/abs/2003.12294#) -> Deli Yu, Xuan Li, Chengquan Zhang, Junyu Han, Jingtuo Liu, Errui Ding -> CVPR,2020 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Model|Backbone|config|Acc|Download link| -| --- | --- | --- | --- | --- | -|SRN|Resnet50_vd_fpn|[rec_r50_fpn_srn.yml](../../configs/rec/rec_r50_fpn_srn.yml)|86.31%|[train model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r50_vd_srn_train.tar)| - - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_r50_fpn_srn.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r50_fpn_srn.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r50_fpn_srn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_r50_fpn_srn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the SRN text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r50_vd_srn_train.tar) ), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_r50_fpn_srn.yml -o Global.pretrained_model=./rec_r50_vd_srn_train/best_accuracy Global.save_inference_dir=./inference/rec_srn -``` - -For SRN text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_srn/" --rec_image_shape="1,64,256" --rec_char_type="ch" --rec_algorithm="SRN" --rec_char_dict_path="ppocr/utils/ic15_dict.txt" --use_space_char=False -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@article{Yu2020TowardsAS, - title={Towards Accurate Scene Text Recognition With Semantic Reasoning Networks}, - author={Deli Yu and Xuan Li and Chengquan Zhang and Junyu Han and Jingtuo Liu and Errui Ding}, - journal={2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, - year={2020}, - pages={12110-12119} -} -``` diff --git a/doc/doc_en/algorithm_rec_starnet.md b/doc/doc_en/algorithm_rec_starnet.md deleted file mode 100644 index 45802c8939..0000000000 --- a/doc/doc_en/algorithm_rec_starnet.md +++ /dev/null @@ -1,137 +0,0 @@ -# STAR-Net - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper information: -> [STAR-Net: a spatial attention residue network for scene text recognition.](http://www.bmva.org/bmvc/2016/papers/paper043/paper043.pdf) -> Wei Liu, Chaofeng Chen, Kwan-Yee K. Wong, Zhizhong Su and Junyu Han. -> BMVC, pages 43.1-43.13, 2016 - -Refer to [DTRB](https://arxiv.org/abs/1904.01906) text Recognition Training and Evaluation Process . Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Models|Backbone Networks|Avg Accuracy|Configuration Files|Download Links| -| --- | --- | --- | --- | --- | -|StarNet|Resnet34_vd|84.44%|[configs/rec/rec_r34_vd_tps_bilstm_ctc.yml](../../configs/rec/rec_r34_vd_tps_bilstm_ctc.yml)|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_ctc_v2.0_train.tar)| -|StarNet|MobileNetV3|81.42%|[configs/rec/rec_mv3_tps_bilstm_ctc.yml](../../configs/rec/rec_mv3_tps_bilstm_ctc.yml)|[ trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_tps_bilstm_ctc_v2.0_train.tar)| - - - -## 2. Environment -Please refer to [Operating Environment Preparation](./environment_en.md) to configure the PaddleOCR operating environment, and refer to [Project Clone](./clone_en.md) to clone the project code. - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Training Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. Take the backbone network based on Resnet34_vd as an example: - - -### 3.1 Training -After the data preparation is complete, the training can be started. The training command is as follows: - -```` -#Single card training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml #Multi-card training, specify the card number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c rec_r34_vd_tps_bilstm_ctc.yml - ```` - - -### 3.2 Evaluation - -```` -# GPU evaluation, Global.pretrained_model is the model to be evaluated -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy - ```` - - -### 3.3 Prediction - -```` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png - ```` - - -## 4. Inference - - -### 4.1 Python Inference -First, convert the model saved during the STAR-Net text recognition training process into an inference model. Take the model trained on the MJSynth and SynthText text recognition datasets based on the Resnet34_vd backbone network as an example [Model download address]( https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar) , which can be converted using the following command: - -```shell -python3 tools/export_model.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model=./rec_r34_vd_tps_bilstm_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_starnet - ```` - -STAR-Net text recognition model inference, you can execute the following commands: - -```shell -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./inference/rec_starnet/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" - ```` - -![](../imgs_words_en/word_336.png) - -The inference results are as follows: - - -```bash -Predicts of ./doc/imgs_words_en/word_336.png:('super', 0.9999073) -``` - -**Attention** Since the above model refers to the [DTRB](https://arxiv.org/abs/1904.01906) text recognition training and evaluation process, it is different from the ultra-lightweight Chinese recognition model training in two aspects: - -- The image resolutions used during training are different. The image resolutions used for training the above models are [3, 32, 100], while for Chinese model training, in order to ensure the recognition effect of long texts, the image resolutions used during training are [ 3, 32, 320]. The default shape parameter of the predictive inference program is the image resolution used for training Chinese, i.e. [3, 32, 320]. Therefore, when inferring the above English model here, it is necessary to set the shape of the recognized image through the parameter rec_image_shape. - -- Character list, the experiment in the DTRB paper is only for 26 lowercase English letters and 10 numbers, a total of 36 characters. All uppercase and lowercase characters are converted to lowercase characters, and characters not listed above are ignored and considered spaces. Therefore, there is no input character dictionary here, but a dictionary is generated by the following command. Therefore, the parameter rec_char_dict_path needs to be set during inference, which is specified as an English dictionary "./ppocr/utils/ic15_dict.txt". - -``` -self.character_str = "0123456789abcdefghijklmnopqrstuvwxyz" -dict_character = list(self.character_str) - - - ``` - - -### 4.2 C++ Inference - -After preparing the inference model, refer to the [cpp infer](../../deploy/cpp_infer/) tutorial to operate. - - -### 4.3 Serving - -After preparing the inference model, refer to the [pdserving](../../deploy/pdserving/) tutorial for Serving deployment, including two modes: Python Serving and C++ Serving. - - -### 4.4 More - -The STAR-Net model also supports the following inference deployment methods: - -- Paddle2ONNX Inference: After preparing the inference model, refer to the [paddle2onnx](../../deploy/paddle2onnx/) tutorial. - - -## 5. FAQ - -## Quote - -```bibtex -@inproceedings{liu2016star, - title={STAR-Net: a spatial attention residue network for scene text recognition.}, - author={Liu, Wei and Chen, Chaofeng and Wong, Kwan-Yee K and Su, Zhizhong and Han, Junyu}, - booktitle={BMVC}, - volume={2}, - pages={7}, - year={2016} -} -``` diff --git a/doc/doc_en/algorithm_rec_svtr_en.md b/doc/doc_en/algorithm_rec_svtr_en.md deleted file mode 100644 index d22fe73e6f..0000000000 --- a/doc/doc_en/algorithm_rec_svtr_en.md +++ /dev/null @@ -1,161 +0,0 @@ -# SVTR - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [SVTR: Scene Text Recognition with a Single Visual Model](https://arxiv.org/abs/2205.00159) -> Yongkun Du and Zhineng Chen and Caiyan Jia Xiaoting Yin and Tianlun Zheng and Chenxia Li and Yuning Du and Yu-Gang Jiang -> IJCAI, 2022 - - -The accuracy (%) and model files of SVTR on the public dataset of scene text recognition are as follows: -* Chinese dataset from [Chinese Benckmark](https://arxiv.org/abs/2112.15093) , and the Chinese training evaluation strategy of SVTR follows the paper. - -| Model |IC13
857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg_6 |IC15
2077 |IC13
1015 |IC03
867|IC03
860|Avg_10 | Chinese
scene_test| Download link | -|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:|:-----:|:-----:|:---------------------------------------------:|:-----:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| -| SVTR Tiny | 96.85 | 91.34 | 94.53 | 83.99 | 85.43 | 89.24 | 90.87 | 80.55 | 95.37 | 95.27 | 95.70 | 90.13 | 67.90 | [English](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar) / [Chinese](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_ch_train.tar) | -| SVTR Small | 95.92 | 93.04 | 95.03 | 84.70 | 87.91 | 92.01 | 91.63 | 82.72 | 94.88 | 96.08 | 96.28 | 91.02 | 69.00 | [English](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_small_none_ctc_en_train.tar) / [Chinese](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_small_none_ctc_ch_train.tar) | -| SVTR Base | 97.08 | 91.50 | 96.03 | 85.20 | 89.92 | 91.67 | 92.33 | 83.73 | 95.66 | 95.62 | 95.81 | 91.61 | 71.40 | [English](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_base_none_ctc_en_train.tar) / - | -| SVTR Large | 97.20 | 91.65 | 96.30 | 86.58 | 88.37 | 95.14 | 92.82 | 84.54 | 96.35 | 96.54 | 96.74 | 92.24 | 72.10 | [English](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_large_none_ctc_en_train.tar) / [Chinese](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_large_none_ctc_ch_train.tar) | - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - -#### Dataset Preparation - -[English dataset download](https://github.com/clovaai/deep-text-recognition-benchmark#download-lmdb-dataset-for-traininig-and-evaluation-from-here) -[Chinese dataset download](https://github.com/fudanvi/benchmarking-chinese-text-recognition#download) - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_svtrnet.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_svtrnet.yml -``` - -Evaluation: - -You can download the model files and configuration files provided by `SVTR`: [download link](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar), take `SVTR-T` as an example, using the following command to evaluate: - -``` -# Download the tar archive containing the model files and configuration files of SVTR-T and extract it -wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar && tar xf rec_svtr_tiny_none_ctc_en_train.tar -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c ./rec_svtr_tiny_none_ctc_en_train/rec_svtr_tiny_6local_6global_stn_en.yml -o Global.pretrained_model=./rec_svtr_tiny_none_ctc_en_train/best_accuracy -``` - -Prediction: - -``` -python3 tools/infer_rec.py -c ./rec_svtr_tiny_none_ctc_en_train/rec_svtr_tiny_6local_6global_stn_en.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_svtr_tiny_none_ctc_en_train/best_accuracy -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the SVTR text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar) ), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_svtrnet.yml -o Global.pretrained_model=./rec_svtr_tiny_none_ctc_en_train/best_accuracy Global.save_inference_dir=./inference/rec_svtr_tiny_stn_en -``` - -**Note:** -- If you are training the model on your own dataset and have modified the dictionary file, please pay attention to modify the `character_dict_path` in the configuration file to the modified dictionary file. - -After the conversion is successful, there are three files in the directory: -``` -/inference/rec_svtr_tiny_stn_en/ - ├── inference.pdiparams - ├── inference.pdiparams.info - └── inference.pdmodel -``` - - -For SVTR text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_svtr_tiny_stn_en/' --rec_algorithm='SVTR' --rec_image_shape='3,64,256' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' -``` - -![](../imgs_words_en/word_10.png) - -After executing the command, the prediction result (recognized text and score) of the image above is printed to the screen, an example is as follows: -The result is as follows: -```shell -Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9999998807907104) -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - -- 1. Speed situation on CPU and GPU - - Since most of the operators used by `SVTR` are matrix multiplication, in the GPU environment, the speed has an advantage, but in the environment where mkldnn is enabled on the CPU, `SVTR` has no advantage over the optimized convolutional network. -- 2. SVTR model convert to ONNX failed - - Ensure `paddle2onnx` and `onnxruntime` versions are up to date, refer to [SVTR model to onnx step-by-step example](https://github.com/PaddlePaddle/PaddleOCR/issues/7821#issuecomment-) for the convert onnx command. 1271214273). -- 3. SVTR model convert to ONNX is successful but the inference result is incorrect - - The possible reason is that the model parameter `out_char_num` is not set correctly, it should be set to W//4, W//8 or W//12, please refer to [Section 3.3.3 of SVTR, a high-precision Chinese scene text recognition model](https://aistudio.baidu.com/aistudio/) projectdetail/5073182?contributionType=1). -- 4. Optimization of long text recognition - - Refer to [Section 3.3 of SVTR, a high-precision Chinese scene text recognition model](https://aistudio.baidu.com/aistudio/projectdetail/5073182?contributionType=1). -- 5. Notes on the reproduction of the paper results - - Dataset using provided by [ABINet](https://github.com/FangShancheng/ABINet). - - By default, 4 cards of GPUs are used for training, the default Batchsize of a single card is 512, and the total Batchsize is 2048, corresponding to a learning rate of 0.0005. When modifying the Batchsize or changing the number of GPU cards, the learning rate should be modified in equal proportion. -- 6. Exploration Directions for further optimization - - Learning rate adjustment: adjusting to twice the default to keep Batchsize unchanged; or reducing Batchsize to 1/2 the default to keep the learning rate unchanged. - - Data augmentation strategies: optionally `RecConAug` and `RecAug`. - - If STN is not used, `Local` of `mixer` can be replaced by `Conv` and `local_mixer` can all be modified to `[5, 5]`. - - Grid search for optimal `embed_dim`, `depth`, `num_heads` configurations. - - Use the `Post-Normalization strategy`, which is to modify the model configuration `prenorm` to `True`. - -## Citation - -```bibtex -@article{Du2022SVTR, - title = {SVTR: Scene Text Recognition with a Single Visual Model}, - author = {Du, Yongkun and Chen, Zhineng and Jia, Caiyan and Yin, Xiaoting and Zheng, Tianlun and Li, Chenxia and Du, Yuning and Jiang, Yu-Gang}, - booktitle = {IJCAI}, - year = {2022}, - url = {https://arxiv.org/abs/2205.00159} -} -``` diff --git a/doc/doc_en/algorithm_rec_visionlan_en.md b/doc/doc_en/algorithm_rec_visionlan_en.md deleted file mode 100644 index 585e853913..0000000000 --- a/doc/doc_en/algorithm_rec_visionlan_en.md +++ /dev/null @@ -1,135 +0,0 @@ -# VisionLAN - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network](https://arxiv.org/abs/2108.09661) -> Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang, Shenggao Zhu, Yongdong Zhang -> ICCV, 2021 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Model|Backbone|config|Acc|Download link| -| --- | --- | --- | --- | --- | -|VisionLAN|ResNet45|[rec_r45_visionlan.yml](../../configs/rec/rec_r45_visionlan.yml)|90.30%|[预训练、训练模型](https://paddleocr.bj.bcebos.com/VisionLAN/rec_r45_visionlan_train.tar)| - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_r45_visionlan.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r45_visionlan.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 tools/eval.py -c configs/rec/rec_r45_visionlan.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_r45_visionlan.yml -o Global.infer_img='./doc/imgs_words/en/word_2.png' Global.pretrained_model=./rec_r45_visionlan_train/best_accuracy -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the VisionLAN text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/VisionLAN/rec_r45_visionlan_train.tar)) ), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_r45_visionlan.yml -o Global.pretrained_model=./rec_r45_visionlan_train/best_accuracy Global.save_inference_dir=./inference/rec_r45_visionlan/ -``` - -**Note:** -- If you are training the model on your own dataset and have modified the dictionary file, please pay attention to modify the `character_dict_path` in the configuration file to the modified dictionary file. -- If you modified the input size during training, please modify the `infer_shape` corresponding to VisionLAN in the `tools/export_model.py` file. - -After the conversion is successful, there are three files in the directory: -``` -./inference/rec_r45_visionlan/ - ├── inference.pdiparams - ├── inference.pdiparams.info - └── inference.pdmodel -``` - - -For VisionLAN text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words/en/word_2.png' --rec_model_dir='./inference/rec_r45_visionlan/' --rec_algorithm='VisionLAN' --rec_image_shape='3,64,256' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' --use_space_char=False -``` - -![](../imgs_words/en/word_2.png) - -After executing the command, the prediction result (recognized text and score) of the image above is printed to the screen, an example is as follows: -The result is as follows: -```shell -Predicts of ./doc/imgs_words/en/word_2.png:('yourself', 0.9999493) -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - -1. Note that the MJSynth and SynthText datasets come from [VisionLAN repo](https://github.com/wangyuxin87/VisionLAN). -2. We use the pre-trained model provided by the VisionLAN authors for finetune training. The dictionary for the pre-trained model is 'ppocr/utils/ic15_dict.txt'. - -## Citation - -```bibtex -@inproceedings{wang2021two, - title={From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network}, - author={Wang, Yuxin and Xie, Hongtao and Fang, Shancheng and Wang, Jing and Zhu, Shenggao and Zhang, Yongdong}, - booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision}, - pages={14194--14203}, - year={2021} -} -``` diff --git a/doc/doc_en/algorithm_rec_vitstr_en.md b/doc/doc_en/algorithm_rec_vitstr_en.md deleted file mode 100644 index 7d5597f69c..0000000000 --- a/doc/doc_en/algorithm_rec_vitstr_en.md +++ /dev/null @@ -1,134 +0,0 @@ -# ViTSTR - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - -## 1. Introduction - -Paper: -> [Vision Transformer for Fast and Efficient Scene Text Recognition](https://arxiv.org/abs/2105.08582) -> Rowel Atienza -> ICDAR, 2021 - -Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: - -|Model|Backbone|config|Acc|Download link| -| --- | --- | --- | --- | --- | -|ViTSTR|ViTSTR|[rec_vitstr_none_ce.yml](../../configs/rec/rec_vitstr_none_ce.yml)|79.82%|[trained model](https://paddleocr.bj.bcebos.com/rec_vitstr_none_ce_train.tar)| - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) -python3 tools/train.py -c configs/rec/rec_vitstr_none_ce.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_vitstr_none_ce.yml -``` - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_vitstr_none_ce.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training -python3 tools/infer_rec.py -c configs/rec/rec_vitstr_none_ce.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_vitstr_none_ce_train/best_accuracy -``` - - -## 4. Inference and Deployment - - -### 4.1 Python Inference -First, the model saved during the ViTSTR text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/rec_vitstr_none_none_train.tar)) ), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/rec/rec_vitstr_none_ce.yml -o Global.pretrained_model=./rec_vitstr_none_ce_train/best_accuracy Global.save_inference_dir=./inference/rec_vitstr -``` - -**Note:** -- If you are training the model on your own dataset and have modified the dictionary file, please pay attention to modify the `character_dict_path` in the configuration file to the modified dictionary file. -- If you modified the input size during training, please modify the `infer_shape` corresponding to ViTSTR in the `tools/export_model.py` file. - -After the conversion is successful, there are three files in the directory: -``` -/inference/rec_vitstr/ - ├── inference.pdiparams - ├── inference.pdiparams.info - └── inference.pdmodel -``` - - -For ViTSTR text recognition model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_vitstr/' --rec_algorithm='ViTSTR' --rec_image_shape='1,224,224' --rec_char_dict_path='./ppocr/utils/EN_symbol_dict.txt' -``` - -![](../imgs_words_en/word_10.png) - -After executing the command, the prediction result (recognized text and score) of the image above is printed to the screen, an example is as follows: -The result is as follows: -```shell -Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9998350143432617) -``` - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - -1. In the `ViTSTR` paper, using pre-trained weights on ImageNet1k for initial training, we did not use pre-trained weights in training, and the final accuracy did not change or even improved. - -## Citation - -```bibtex -@article{Atienza2021ViTSTR, - title = {Vision Transformer for Fast and Efficient Scene Text Recognition}, - author = {Rowel Atienza}, - booktitle = {ICDAR}, - year = {2021}, - url = {https://arxiv.org/abs/2105.08582} -} -``` diff --git a/doc/doc_en/algorithm_sdmgr_en.md b/doc/doc_en/algorithm_sdmgr_en.md deleted file mode 100644 index 7b3752223d..0000000000 --- a/doc/doc_en/algorithm_sdmgr_en.md +++ /dev/null @@ -1,72 +0,0 @@ -- [Key Information Extraction(KIE)](#key-information-extractionkie) - - [1. Quick Use](#1-quick-use) - - [2. Model Training](#2-model-training) - - [3. Model Evaluation](#3-model-evaluation) - - [4. Reference](#4-reference) - -# Key Information Extraction(KIE) - -This section provides a tutorial example on how to quickly use, train, and evaluate a key information extraction(KIE) model, [SDMGR](https://arxiv.org/abs/2103.14470), in PaddleOCR. - -[SDMGR(Spatial Dual-Modality Graph Reasoning)](https://arxiv.org/abs/2103.14470) is a KIE algorithm that classifies each detected text region into predefined categories, such as order ID, invoice number, amount, and etc. - -## 1. Quick Use - -[Wildreceipt dataset](https://paperswithcode.com/dataset/wildreceipt) is used for this tutorial. It contains 1765 photos, with 25 classes, and 50000 text boxes, which can be downloaded by wget: - -```shell -wget https://paddleocr.bj.bcebos.com/ppstructure/dataset/wildreceipt.tar && tar xf wildreceipt.tar -``` - -Download the pretrained model and predict the result: - -```shell -cd PaddleOCR/ -wget https://paddleocr.bj.bcebos.com/dygraph_v2.1/kie/kie_vgg16.tar && tar xf kie_vgg16.tar -python3.7 tools/infer_kie.py -c configs/kie/kie_unet_sdmgr.yml -o Global.checkpoints=kie_vgg16/best_accuracy Global.infer_img=../wildreceipt/1.txt -``` - -The prediction result is saved as `./output/sdmgr_kie/predicts_kie.txt`, and the visualization results are saved in the folder`/output/sdmgr_kie/kie_results/`. - -The visualization results are shown in the figure below: - -
- -
- -## 2. Model Training - -Create a softlink to the folder, `PaddleOCR/train_data`: -```shell -cd PaddleOCR/ && mkdir train_data && cd train_data - -ln -s ../../wildreceipt ./ -``` - -The configuration file used for training is `configs/kie/kie_unet_sdmgr.yml`. The default training data path in the configuration file is `train_data/wildreceipt`. After preparing the data, you can execute the model training with the following command: -```shell -python3.7 tools/train.py -c configs/kie/kie_unet_sdmgr.yml -o Global.save_model_dir=./output/kie/ -``` - -## 3. Model Evaluation - -After training, you can execute the model evaluation with the following command: - -```shell -python3.7 tools/eval.py -c configs/kie/kie_unet_sdmgr.yml -o Global.checkpoints=./output/kie/best_accuracy -``` - -## 4. Reference - - - -```bibtex -@misc{sun2021spatial, - title={Spatial Dual-Modality Graph Reasoning for Key Information Extraction}, - author={Hongbin Sun and Zhanghui Kuang and Xiaoyu Yue and Chenhao Lin and Wayne Zhang}, - year={2021}, - eprint={2103.14470}, - archivePrefix={arXiv}, - primaryClass={cs.CV} -} -``` diff --git a/doc/doc_en/algorithm_sr_gestalt_en.md b/doc/doc_en/algorithm_sr_gestalt_en.md deleted file mode 100644 index 516b90cb30..0000000000 --- a/doc/doc_en/algorithm_sr_gestalt_en.md +++ /dev/null @@ -1,136 +0,0 @@ -# Text Gestalt - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - - -## 1. Introduction - -Paper: -> [Text Gestalt: Stroke-Aware Scene Text Image Super-Resolution](https://arxiv.org/pdf/2112.08171.pdf) - -> Chen, Jingye and Yu, Haiyang and Ma, Jianqi and Li, Bin and Xue, Xiangyang - -> AAAI, 2022 - -Referring to the [FudanOCR](https://github.com/FudanVI/FudanOCR/tree/main/text-gestalt) data download instructions, the effect of the super-score algorithm on the TextZoom test set is as follows: - -|Model|Backbone|config|Acc|Download link| -|---|---|---|---|---|---| -|Text Gestalt|tsrn|19.28|0.6560| [configs/sr/sr_tsrn_transformer_strock.yml](../../configs/sr/sr_tsrn_transformer_strock.yml)|[train model](https://paddleocr.bj.bcebos.com/sr_tsrn_transformer_strock_train.tar)| - - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) - -python3 tools/train.py -c configs/sr/sr_tsrn_transformer_strock.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter - -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/sr/sr_tsrn_transformer_strock.yml - -``` - - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/sr/sr_tsrn_transformer_strock.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training - -python3 tools/infer_sr.py -c configs/sr/sr_tsrn_transformer_strock.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words_en/word_52.png -``` - -![](../imgs_words_en/word_52.png) - -After executing the command, the super-resolution result of the above image is as follows: - -![](../imgs_results/sr_word_52.png) - - -## 4. Inference and Deployment - - -### 4.1 Python Inference - -First, the model saved during the training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/sr_tsrn_transformer_strock_train.tar) ), you can use the following command to convert: - -```shell -python3 tools/export_model.py -c configs/sr/sr_tsrn_transformer_strock.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/sr_out -``` - -For Text-Gestalt super-resolution model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_sr.py --sr_model_dir=./inference/sr_out --image_dir=doc/imgs_words_en/word_52.png --sr_image_shape=3,32,128 - -``` - -After executing the command, the super-resolution result of the above image is as follows: - -![](../imgs_results/sr_word_52.png) - - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@inproceedings{chen2022text, - title={Text gestalt: Stroke-aware scene text image super-resolution}, - author={Chen, Jingye and Yu, Haiyang and Ma, Jianqi and Li, Bin and Xue, Xiangyang}, - booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, - volume={36}, - number={1}, - pages={285--293}, - year={2022} -} -``` diff --git a/doc/doc_en/algorithm_sr_telescope_en.md b/doc/doc_en/algorithm_sr_telescope_en.md deleted file mode 100644 index 334b58b6e8..0000000000 --- a/doc/doc_en/algorithm_sr_telescope_en.md +++ /dev/null @@ -1,137 +0,0 @@ -# Text Gestalt - -- [1. Introduction](#1) -- [2. Environment](#2) -- [3. Model Training / Evaluation / Prediction](#3) - - [3.1 Training](#3-1) - - [3.2 Evaluation](#3-2) - - [3.3 Prediction](#3-3) -- [4. Inference and Deployment](#4) - - [4.1 Python Inference](#4-1) - - [4.2 C++ Inference](#4-2) - - [4.3 Serving](#4-3) - - [4.4 More](#4-4) -- [5. FAQ](#5) - - - -## 1. Introduction - -Paper: -> [Scene Text Telescope: Text-Focused Scene Image Super-Resolution](https://openaccess.thecvf.com/content/CVPR2021/papers/Chen_Scene_Text_Telescope_Text-Focused_Scene_Image_Super-Resolution_CVPR_2021_paper.pdf) - -> Chen, Jingye, Bin Li, and Xiangyang Xue - -> CVPR, 2021 - -Referring to the [FudanOCR](https://github.com/FudanVI/FudanOCR/tree/main/scene-text-telescope) data download instructions, the effect of the super-score algorithm on the TextZoom test set is as follows: - -|Model|Backbone|config|Acc|Download link| -|---|---|---|---|---| -|Text Gestalt|tsrn|21.56|0.7411| [configs/sr/sr_telescope.yml](../../configs/sr/sr_telescope.yml)|[train model](https://paddleocr.bj.bcebos.com/contribution/sr_telescope_train.tar)| - -The [TextZoom dataset](https://paddleocr.bj.bcebos.com/dataset/TextZoom.tar) comes from two superfraction data sets, RealSR and SR-RAW, both of which contain LR-HR pairs. TextZoom has 17367 pairs of training data and 4373 pairs of test data. - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -Please refer to [Text Recognition Tutorial](./recognition_en.md). PaddleOCR modularizes the code, and training different models only requires **changing the configuration file**. - -Training: - -Specifically, after the data preparation is completed, the training can be started. The training command is as follows: - -``` -#Single GPU training (long training period, not recommended) - -python3 tools/train.py -c configs/sr/sr_telescope.yml - -#Multi GPU training, specify the gpu number through the --gpus parameter - -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/sr/sr_telescope.yml - -``` - - -Evaluation: - -``` -# GPU evaluation -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/sr/sr_telescope.yml -o Global.pretrained_model={path/to/weights}/best_accuracy -``` - -Prediction: - -``` -# The configuration file used for prediction must match the training - -python3 tools/infer_sr.py -c configs/sr/sr_telescope.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words_en/word_52.png -``` - -![](../imgs_words_en/word_52.png) - -After executing the command, the super-resolution result of the above image is as follows: - -![](../imgs_results/sr_word_52.png) - - -## 4. Inference and Deployment - - -### 4.1 Python Inference - -First, the model saved during the training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/contribution/Telescope_train.tar.gz) ), you can use the following command to convert: - -```shell -python3 tools/export_model.py -c configs/sr/sr_telescope.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/sr_out -``` - -For Text-Telescope super-resolution model inference, the following commands can be executed: - -``` -python3 tools/infer/predict_sr.py --sr_model_dir=./inference/sr_out --image_dir=doc/imgs_words_en/word_52.png --sr_image_shape=3,32,128 - -``` - -After executing the command, the super-resolution result of the above image is as follows: - -![](../imgs_results/sr_word_52.png) - - - -### 4.2 C++ Inference - -Not supported - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - - -## Citation - -```bibtex -@INPROCEEDINGS{9578891, - author={Chen, Jingye and Li, Bin and Xue, Xiangyang}, - booktitle={2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, - title={Scene Text Telescope: Text-Focused Scene Image Super-Resolution}, - year={2021}, - volume={}, - number={}, - pages={12021-12030}, - doi={10.1109/CVPR46437.2021.01185}} -``` diff --git a/doc/doc_en/algorithm_table_master_en.md b/doc/doc_en/algorithm_table_master_en.md deleted file mode 100644 index e9249a2a05..0000000000 --- a/doc/doc_en/algorithm_table_master_en.md +++ /dev/null @@ -1,112 +0,0 @@ -# Table Recognition Algorithm-TableMASTER - -- [1. Introduction](#1-introduction) -- [2. Environment](#2-environment) -- [3. Model Training / Evaluation / Prediction](#3-model-training--evaluation--prediction) -- [4. Inference and Deployment](#4-inference-and-deployment) - - [4.1 Python Inference](#41-python-inference) - - [4.2 C++ Inference](#42-c-inference) - - [4.3 Serving](#43-serving) - - [4.4 More](#44-more) -- [5. FAQ](#5-faq) -- [Citation](#citation) - - -## 1. Introduction - -Paper: -> [TableMaster: PINGAN-VCGROUP’S SOLUTION FOR ICDAR 2021 COMPETITION ON SCIENTIFIC LITERATURE PARSING TASK B: TABLE RECOGNITION TO HTML](https://arxiv.org/pdf/2105.01848.pdf) -> Ye, Jiaquan and Qi, Xianbiao and He, Yelin and Chen, Yihao and Gu, Dengyi and Gao, Peng and Xiao, Rong -> 2021 - - -On the PubTabNet table recognition public data set, the algorithm reproduction acc is as follows: - -|Model|Backbone|Cnnfig|Acc|Download link| -| --- | --- | --- | --- | --- | -|TableMaster|TableResNetExtra|[configs/table/table_master.yml](../../configs/table/table_master.yml)|77.47%|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/tablemaster/table_structure_tablemaster_train.tar)/[inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/tablemaster/table_structure_tablemaster_infer.tar)| - - - -## 2. Environment -Please refer to ["Environment Preparation"](./environment_en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](./clone_en.md) to clone the project code. - - - -## 3. Model Training / Evaluation / Prediction - -The above TableMaster model is trained using the PubTabNet table recognition public dataset. For the download of the dataset, please refer to [table_datasets](./dataset/table_datasets_en.md). - -After the data download is complete, please refer to [Text Recognition Training Tutorial](./recognition_en.md) for training. PaddleOCR has modularized the code structure, so that you only need to **replace the configuration file** to train different models. - - - -## 4. Inference and Deployment - - -### 4.1 Python Inference - -First, convert the model saved in the TableMaster table recognition training process into an inference model. Taking the model based on the TableResNetExtra backbone network and trained on the PubTabNet dataset as example ([model download link](https://paddleocr.bj.bcebos.com/contribution/table_master.tar)), you can use the following command to convert: - - -```shell -python3 tools/export_model.py -c configs/table/table_master.yml -o Global.pretrained_model=output/table_master/best_accuracy Global.save_inference_dir=./inference/table_master -``` - -**Note: ** -- If you trained the model on your own dataset and adjusted the dictionary file, please pay attention to whether the `character_dict_path` in the modified configuration file is the correct dictionary file - - -Execute the following command for model inference: - -```shell -cd ppstructure/ -# When predicting all images in a folder, you can modify image_dir to a folder, such as --image_dir='docs/table'. -python3.7 table/predict_structure.py --table_model_dir=../output/table_master/table_structure_tablemaster_infer/ --table_algorithm=TableMaster --table_char_dict_path=../ppocr/utils/dict/table_master_structure_dict.txt --table_max_len=480 --image_dir=docs/table/table.jpg - -``` - -After executing the command, the prediction results of the above image (structural information and the coordinates of each cell in the table) are printed to the screen, and the visualization of the cell coordinates is also saved. An example is as follows: - -result: -```shell -[2022/06/16 13:06:54] ppocr INFO: result: ['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '
', '', ''], [[72.17591094970703, 10.759100914001465, 60.29658508300781, 16.6805362701416], [161.85562133789062, 10.884308815002441, 14.9495210647583, 16.727018356323242], [277.79876708984375, 29.54340362548828, 31.490320205688477, 18.143272399902344], -... -[336.11724853515625, 280.3601989746094, 39.456939697265625, 18.121286392211914]] -[2022/06/16 13:06:54] ppocr INFO: save vis result to ./output/table.jpg -[2022/06/16 13:06:54] ppocr INFO: Predict time of docs/table/table.jpg: 17.36806297302246 -``` - -**Note**: - -- TableMaster is relatively slow during inference, and it is recommended to use GPU for use. - - -### 4.2 C++ Inference - -Since the post-processing is not written in CPP, the TableMaster does not support CPP inference. - - - -### 4.3 Serving - -Not supported - - -### 4.4 More - -Not supported - - -## 5. FAQ - -## Citation - -```bibtex -@article{ye2021pingan, - title={PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Literature Parsing Task B: Table Recognition to HTML}, - author={Ye, Jiaquan and Qi, Xianbiao and He, Yelin and Chen, Yihao and Gu, Dengyi and Gao, Peng and Xiao, Rong}, - journal={arXiv preprint arXiv:2105.01848}, - year={2021} -} -``` diff --git a/doc/doc_en/angle_class_en.md b/doc/doc_en/angle_class_en.md deleted file mode 100644 index 00888fb04b..0000000000 --- a/doc/doc_en/angle_class_en.md +++ /dev/null @@ -1,153 +0,0 @@ -# Text Direction Classification - -- [1. Method Introduction](#method-introduction) -- [2. Data Preparation](#data-preparation) -- [3. Training](#training) -- [4. Evaluation](#evaluation) -- [5. Prediction](#prediction) - - - -## 1. Method Introduction -The angle classification is used in the scene where the image is not 0 degrees. In this scene, it is necessary to perform a correction operation on the text line detected in the picture. In the PaddleOCR system, -The text line image obtained after text detection is sent to the recognition model after affine transformation. At this time, only a 0 and 180 degree angle classification of the text is required, so the built-in PaddleOCR text angle classifier **only supports 0 and 180 degree classification**. If you want to support more angles, you can modify the algorithm yourself to support. - -Example of 0 and 180 degree data samples: - -![](../imgs_results/angle_class_example.jpg) - - -## 2. Data Preparation - -Please organize the dataset as follows: - -The default storage path for training data is `PaddleOCR/train_data/cls`, if you already have a dataset on your disk, just create a soft link to the dataset directory: - -``` -ln -sf /train_data/cls/dataset -``` - -please refer to the following to organize your data. - -- Training set - -First put the training images in the same folder (train_images), and use a txt file (cls_gt_train.txt) to store the image path and label. - -* Note: by default, the image path and image label are split with `\t`, if you use other methods to split, it will cause training error - -0 and 180 indicate that the angle of the image is 0 degrees and 180 degrees, respectively. - -``` -" Image file name Image annotation " - -train/word_001.jpg 0 -train/word_002.jpg 180 -``` - -The final training set should have the following file structure: - -``` -|-train_data - |-cls - |- cls_gt_train.txt - |- train - |- word_001.png - |- word_002.jpg - |- word_003.jpg - | ... -``` - -- Test set - -Similar to the training set, the test set also needs to be provided a folder -containing all images (test) and a cls_gt_test.txt. The structure of the test set is as follows: - -``` -|-train_data - |-cls - |- cls_gt_test.txt - |- test - |- word_001.jpg - |- word_002.jpg - |- word_003.jpg - | ... -``` - -## 3. Training -Write the prepared txt file and image folder path into the configuration file under the `Train/Eval.dataset.label_file_list` and `Train/Eval.dataset.data_dir` fields, the absolute path of the image consists of the `Train/Eval.dataset.data_dir` field and the image name recorded in the txt file. - -PaddleOCR provides training scripts, evaluation scripts, and prediction scripts. - -Start training: - -``` -# Set PYTHONPATH path -export PYTHONPATH=$PYTHONPATH:. -# GPU training Support single card and multi-card training, specify the card number through --gpus. -# Start training, the following command has been written into the train.sh file, just modify the configuration file path in the file -python3 -m paddle.distributed.launch --gpus '0,1,2,3,4,5,6,7' tools/train.py -c configs/cls/cls_mv3.yml -``` - -- Data Augmentation - -PaddleOCR provides a variety of data augmentation methods. If you want to add disturbance during training, Please uncomment the `RecAug` and `RandAugment` fields under `Train.dataset.transforms` in the configuration file. - -The default perturbation methods are: cvtColor, blur, jitter, Gauss noise, random crop, perspective, color reverse, RandAugment. - -Except for RandAugment, each disturbance method is selected with a 50% probability during the training process. For specific code implementation, please refer to: -[rec_img_aug.py](../../ppocr/data/imaug/rec_img_aug.py) -[randaugment.py](../../ppocr/data/imaug/randaugment.py) - - -- Training - -PaddleOCR supports alternating training and evaluation. You can modify `eval_batch_step` in `configs/cls/cls_mv3.yml` to set the evaluation frequency. By default, it is evaluated every 1000 iter. The following content will be saved during training: -```bash -├── best_accuracy.pdopt # Optimizer parameters for the best model -├── best_accuracy.pdparams # Parameters of the best model -├── best_accuracy.states # Metric info and epochs of the best model -├── config.yml # Configuration file for this experiment -├── latest.pdopt # Optimizer parameters for the latest model -├── latest.pdparams # Parameters of the latest model -├── latest.states # Metric info and epochs of the latest model -└── train.log # Training log -``` - -If the evaluation set is large, the test will be time-consuming. It is recommended to reduce the number of evaluations, or evaluate after training. - -**Note that the configuration file for prediction/evaluation must be consistent with the training.** - - -## 4. Evaluation - -The evaluation dataset can be set by modifying the `Eval.dataset.label_file_list` field in the `configs/cls/cls_mv3.yml` file. - -``` -export CUDA_VISIBLE_DEVICES=0 -# GPU evaluation, Global.checkpoints is the weight to be tested -python3 tools/eval.py -c configs/cls/cls_mv3.yml -o Global.checkpoints={path/to/weights}/best_accuracy -``` - -## 5. Prediction - -* Training engine prediction - -Using the model trained by paddleocr, you can quickly get prediction through the following script. - -Use `Global.infer_img` to specify the path of the predicted picture or folder, and use `Global.checkpoints` to specify the weight: - -``` -# Predict English results -python3 tools/infer_cls.py -c configs/cls/cls_mv3.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.load_static_weights=false Global.infer_img=doc/imgs_words_en/word_10.png -``` - -Input image: - -![](../imgs_words_en/word_10.png) - -Get the prediction result of the input image: - -``` -infer_img: doc/imgs_words_en/word_10.png - result: ('0', 0.9999995) -``` diff --git a/doc/doc_en/benchmark_en.md b/doc/doc_en/benchmark_en.md deleted file mode 100755 index 62ac0258a9..0000000000 --- a/doc/doc_en/benchmark_en.md +++ /dev/null @@ -1,41 +0,0 @@ -# Benchmark - -This document gives the performance of the series models for Chinese and English recognition. - -## Test Data - -We collected 300 images for different real application scenarios to evaluate the overall OCR system, including contract samples, license plates, nameplates, train tickets, test sheets, forms, certificates, street view images, business cards, digital meter, etc. The following figure shows some images of the test set. - -
- -
- -## Measurement - -Explanation: - -- The long size of the input for the text detector is 960. - -- The evaluation time-consuming stage is the complete stage from image input to result output, including image pre-processing and post-processing. - -- ```Intel Xeon 6148``` is the server-side CPU model. Intel MKL-DNN is used in the test to accelerate the CPU prediction speed. - -- ```Snapdragon 855``` is a mobile processing platform model. - -Compares the model size and F-score: - -| Model Name | Model Size
of the
Whole System\(M\) | Model Size
of the Text
Detector\(M\) | Model Size
of the Direction
Classifier\(M\) | Model Size
of the Text
Recognizer \(M\) | F\-score | -|:-:|:-:|:-:|:-:|:-:|:-:| -| PP-OCRv2 | 11\.6 | 3\.0 | 0\.9 | 8\.6 | 0\.5224 | -| PP-OCR mobile | 8\.1 | 2\.6 | 0\.9 | 4\.6 | 0\.503 | -| PP-OCR server | 155\.1 | 47\.2 | 0\.9 | 107 | 0\.570 | - -Compares the time-consuming on CPU and T4 GPU (ms): - -| Model Name | CPU | T4 GPU | -|:-:|:-:|:-:| -| PP-OCRv2 | 330 | 111 | -| PP-OCR mobile | 356 | 116| -| PP-OCR server | 1056 | 200 | - -More indicators of PP-OCR series models can be referred to [PP-OCR Benchmark](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.2/doc/doc_en/benchmark_en.md) diff --git a/doc/doc_en/clone_en.md b/doc/doc_en/clone_en.md deleted file mode 100644 index f9af283ce7..0000000000 --- a/doc/doc_en/clone_en.md +++ /dev/null @@ -1,21 +0,0 @@ -# Project Clone - -## 1. Clone PaddleOCR - -```bash -# Recommend -git clone https://github.com/PaddlePaddle/PaddleOCR - -# If you cannot pull successfully due to network problems, you can switch to the mirror hosted on Gitee: - -git clone https://gitee.com/paddlepaddle/PaddleOCR - -# Note: The mirror on Gitee may not keep in synchronization with the latest project on GitHub. There might be a delay of 3-5 days. Please try GitHub at first. -``` - -## 2. Install third-party libraries - -```bash -cd PaddleOCR -pip3 install -r requirements.txt -``` diff --git a/doc/doc_en/code_and_doc.md b/doc/doc_en/code_and_doc.md deleted file mode 100644 index d4a7cf4835..0000000000 --- a/doc/doc_en/code_and_doc.md +++ /dev/null @@ -1,349 +0,0 @@ - - Appendix - - This appendix contains python, document specifications and Pull Request process. Please follow the relevant contents - - - [Appendix 1:Python Code Specification](#Appendix1) - - - [Appendix 2:Document Specification](#Appendix2) - - - [Appendix 3:Pull Request Description](#Appendix3) - - - - ## Appendix 1:Python Code Specification - - The Python code of PaddleOCR follows [PEP8 Specification]( https://www.python.org/dev/peps/pep-0008/ ), some of the key concerns include the following - - - Space - - - Spaces should be added after commas, semicolons, colons, not before them - - ```python - # true: - print(x, y) - - # false: - print(x , y) - ``` - - - When specifying a keyword parameter or default parameter value in a function, do not use spaces on both sides of it - - ```python - # true: - def complex(real, imag=0.0) - # false: - def complex(real, imag = 0.0) - ``` - - - comment - - - Inline comments: inline comments are indicated by the` # `sign. Two spaces should be left between code and` # `, and one space should be left between` # `and comments, for example - - ```python - x = x + 1 # Compensate for border - ``` - - - Functions and methods: The definition of each function should include the following: - - - Function description: Utility, input and output of function - - - Args: Name and description of each parameter - - Returns: The meaning and type of the return value - - ```python - def fetch_bigtable_rows(big_table, keys, other_silly_variable=None): - """Fetches rows from a Bigtable. - - Retrieves rows pertaining to the given keys from the Table instance - represented by big_table. Silly things may happen if - other_silly_variable is not None. - - Args: - big_table: An open Bigtable Table instance. - keys: A sequence of strings representing the key of each table row - to fetch. - other_silly_variable: Another optional variable, that has a much - longer name than the other args, and which does nothing. - - Returns: - A dict mapping keys to the corresponding table row data - fetched. Each row is represented as a tuple of strings. For - example: - - {'Serak': ('Rigel VII', 'Preparer'), - 'Zim': ('Irk', 'Invader'), - 'Lrrr': ('Omicron Persei 8', 'Emperor')} - - If a key from the keys argument is missing from the dictionary, - then that row was not found in the table. - """ - pass - ``` - - - - ## Appendix 2: Document Specification - - ### 2.1 Overall Description - - - Document Location: If you add new features to your original Markdown file, please **Do not re-create** a new file. If you don't know where to add it, you can first PR the code and then ask the official in commit. - - - New Markdown Document Name: Describe the content of the document in English, typically a combination of lowercase letters and underscores, such as `add_New_Algorithm.md` - - - New Markdown Document Format: Catalog - Body - FAQ - - > The directory generation method can use [this site](https://ecotrust-canada.github.io/markdown-toc/ ) Automatically extract directories after copying MD contents, and then add ` before each heading of the MD file - - - English and Chinese: Any changes or additions to the document need to be made in both Chinese and English documents. - - ### 2.2 Format Specification - - - Title format: The document title format follows the format of: Arabic decimal point combination-space-title (for example, `2.1 XXXX`, `2.XXXX`) - - - Code block: Displays code in code block format that needs to be run, describing the meaning of command parameters before the code block. for example: - - > Pipeline of detection + direction Classify + recognition: Vertical text can be recognized after set direction classifier parameters`--use_angle_cls true`. - > - > ``` - > paddleocr --image_dir ./imgs/11.jpg --use_angle_cls true - > ``` - - - Variable Rrferences: If code variables or command parameters are referenced in line, they need to be represented in line code, for example, above `--use_angle_cls true` with one space in front and one space in back - - - Uniform naming: e.g. PP-OCRv2, PP-OCR mobile, `paddleocr` whl package, PPOCRLabel, Paddle Lite, etc. - - - Supplementary notes: Supplementary notes by reference format `>`. - - - Picture: If a picture is added to the description document, specify the naming of the picture (describing its content) and add the picture under `doc/`. - - - Title: Capitalize the first letter of each word in the title. - - - - ## Appendix 3: Pull Request Description - - ### 3.1 PaddleOCR Branch Description - - PaddleOCR will maintain two branches in the future, one for each: - - - release/x.x family branch: stable release version branch, also the default branch. PaddleOCR releases a new release branch based on feature updates and adapts to the release version of Paddle. As versions iterate, more and more release/x.x family branches are maintained by default with the latest version of the release branch. - - dygraph branch: For the development branch, adapts the dygraph version of the Paddle dynamic graph to primarily develop new functionality. If you need to redevelop, choose the dygraph branch. To ensure that the dygraph branch pulls out the release/x.x branch when needed, the code for the dygraph branch can only use the valid API in the latest release branch of Paddle. That is, if a new API has been developed in the Paddle dygraph branch but has not yet appeared in the release branch code, do not use it in Paddle OCR. In addition, performance optimization, parameter tuning, policy updates that do not involve API can be developed normally. - - The historical branch of PaddleOCR will no longer be maintained in the future. These branches will continue to be maintained, considering that some of you may still be using them: - - - Develop branch: This branch was used for the development and testing of static diagrams and is currently compatible with version >=1.7. If you have special needs, you can also use this branch to accommodate older versions of Paddle, but you won't update your code until you fix the bug. - - PaddleOCR welcomes you to actively contribute code to repo. Here are some basic processes for contributing code. - - ### 3.2 PaddleOCR Code Submission Process And Specification - - > If you are familiar with Git use, you can jump directly to [Some Conventions For Submitting Code in 3.2.10](#Some_conventions_for_submitting_code) - - #### 3.2.1 Create Your `Remote Repo` - - - In PaddleOCR [GitHub Home]( https://github.com/PaddlePaddle/PaddleOCR ) Click the `Fork` button in the upper left corner to create a `remote repo`in your personal directory, such as ` https://github.com/ {your_name}/PaddleOCR`. - - ![banner](../banner.png) - - - Clone `Remote repo` - - ``` - # pull code of develop branch - git clone https://github.com/{your_name}/PaddleOCR.git -b dygraph - cd PaddleOCR - ``` - - > Clone failures are mostly due to network reasons, try again later or configure the proxy - - #### 3.2.2 Login And Connect Using Token - - Start by viewing the information for the current `remote repo`. - - ``` - git remote -v - # origin https://github.com/{your_name}/PaddleOCR.git (fetch) - # origin https://github.com/{your_name}/PaddleOCR.git (push) - ``` - - Only the information of the clone `remote repo`, i.e. the PaddleOCR under your username, is available. Due to the change in Github's login method, you need to reconfigure the `remote repo` address by means of a Token. The token is generated as follows: - - 1. Find Personal Access Tokens: Click on your avatar in the upper right corner of the Github page and choose Settings --> Developer settings --> Personal access tokens, - - 2. Click Generate new token: Fill in the token name in Note, such as 'paddle'. In Select scopes, select repo (required), admin:repo_hook, delete_repo, etc. You can check them according to your needs. Then click Generate token to generate the token, and finally copy the generated token. - - Delete the original origin configuration - - ``` - git remote rm origin - ``` - - Change the remote branch to `https://oauth2:{token}@github.com/{your_name}/PaddleOCR.git`. For example, if the token value is 12345 and your user name is PPOCR, run the following command - - ``` - git remote add origin https://oauth2:12345@github.com/PPOCR/PaddleOCR.git - ``` - - This establishes a connection to our own `remote repo`. Next we create a remote host of the original PaddleOCR repo, named upstream. - - ``` - git remote add upstream https://github.com/PaddlePaddle/PaddleOCR.git - ``` - - Use `git remote -v` to view current `remote warehouse` information, output as follows, found to include two origin and two upstream of `remote repo` . - - ``` - origin https://github.com/{your_name}/PaddleOCR.git (fetch) - origin https://github.com/{your_name}/PaddleOCR.git (push) - upstream https://github.com/PaddlePaddle/PaddleOCR.git (fetch) - upstream https://github.com/PaddlePaddle/PaddleOCR.git (push) - ``` - - This is mainly to keep the local repository up to date when subsequent pull request (PR) submissions are made. - - #### 3.2.3 Create Local Branch - - First get the latest code of upstream, then create a new_branch branch based on the dygraph of the upstream repo (upstream). - - ``` - git fetch upstream - git checkout -b new_branch upstream/dygraph - ``` - - > If for a newly forked PaddleOCR project, the user's remote repo (origin) has the same branch updates as the upstream repository (upstream), you can also create a new local branch based on the default branch of the origin repo or a specified branch with the following command - > - > ``` - > # Create new_branch branch on user remote repo (origin) based on develop branch - > git checkout -b new_branch origin/develop - > # Create new_branch branch based on upstream remote repo develop branch - > # If you need to create a new branch from upstream, - > # you need to first use git fetch upstream to get upstream code - > git checkout -b new_branch upstream/develop - > ``` - - The final switch to the new branch is displayed with the following output information. - - ``` - Branch new_branch set up to track remote branch develop from upstream. - Switched to a new branch 'new_branch' - ``` - - After switching branches, file changes can be made on this branch - - #### 3.2.4 Use Pre-Commit Hook - - Paddle developers use the pre-commit tool to manage Git pre-submit hooks. It helps us format the source code (C++, Python) and automatically check for basic things (such as having only one EOL per file, not adding large files to Git) before committing it. - - The pre-commit test is part of the unit test in Travis-CI. PR that does not satisfy the hook cannot be submitted to PaddleOCR. Install it first and run it in the current directory: - - ``` - pip install pre-commit - pre-commit install - ``` - - > 1. Paddle uses clang-format to adjust the C/C++ source code format. Make sure the `clang-format` version is above 3.8. - > - > 2. Yapf installed through pip install pre-commit is slightly different from conda install-c conda-forge pre-commit, and PaddleOCR developers use `pip install pre-commit`. - - #### 3.2.5 Modify And Submit Code - - If you make some changes on `README.Md ` on PaddleOCR, you can view the changed file through `git status`, and then add the changed file using `git add`。 - - ``` - git status # View change files - git add README.md - pre-commit - ``` - - Repeat these steps until the pre-comit format check does not error. As shown below. - - ![img](../precommit_pass.png) - - Use the following command to complete the submission. - - ``` - git commit -m "your commit info" - ``` - - #### 3.2.6 Keep Local Repo Up To Date - - Get the latest code for upstream and update the current branch. Here the upstream comes from section 2.2, `Connecting to a remote repo`. - - ``` - git fetch upstream - # If you want to commit to another branch, you need to pull code from another branch of upstream, here is develop - git pull upstream develop - ``` - - #### 3.2.7 Push To Remote Repo - - ``` - git push origin new_branch - ``` - - #### 3.2.7 Submit Pull Request - - Click the new pull request to select the local branch and the target branch, as shown in the following figure. In the description of PR, fill in the functions completed by the PR. Next, wait for review, and if you need to modify something, update the corresponding branch in origin with the steps above. - - ![banner](../pr.png) - - #### 3.2.8 Sign CLA Agreement And Pass Unit Tests - - - Signing the CLA When submitting a Pull Request to PaddlePaddle for the first time, you need to sign a CLA (Contributor License Agreement) agreement to ensure that your code can be incorporated as follows: - - 1. Please check the Check section in PR, find the license/cla, and click on the right detail to enter the CLA website - - 2. Click Sign in with GitHub to agree on the CLA website and when clicked, it will jump back to your Pull Request page - - #### 3.2.9 Delete Branch - - - Remove remote branch - - After PR is merged into the main repo, we can delete the branch of the remote repofrom the PR page. - You can also use `git push origin:branch name` to delete remote branches, such as: - - ``` - git push origin :new_branch - ``` - -- Delete local branch - - ``` - # Switch to the development branch, otherwise the current branch cannot be deleted - git checkout develop - - # Delete new_ Branch Branch - git branch -D new_branch - ``` - - - - #### 3.2.10 Some Conventions For Submitting Code - - In order for official maintainers to better focus on the code itself when reviewing it, please follow the following conventions each time you submit your code: - - 1)Please ensure that the unit tests in Travis-CI pass smoothly. If not, indicate that there is a problem with the submitted code, and the official maintainer generally does not review it. - - 2)Before submitting a Pull Request. - - - Note the number of commits. - - Reason: If you only modify one file and submit more than a dozen commits, each commit will only make a few modifications, which can be very confusing to the reviewer. The reviewer needs to look at each commit individually to see what changes have been made, and does not exclude the fact that changes between commits overlap each other. - - Suggestion: Keep as few commits as possible each time you submit, and supplement your last commit with git commit --amend. For multiple commits that have been Push to a remote warehouse, you can refer to [squash commits after push](https://stackoverflow.com/questions/5667884/how-to-squash-commits-in-git-after-they-have-been-pushed ). - - - Note the name of each commit: it should reflect the content of the current commit, not be too arbitrary. - - - 3) If you have solved a problem, add in the first comment box of the Pull Request:fix #issue_number,This will automatically close the corresponding Issue when the Pull Request is merged. Key words include:close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved,please choose the right vocabulary. Detailed reference [Closing issues via commit messages](https://help.github.com/articles/closing-issues-via-commit-messages). - - In addition, in response to the reviewer's comments, you are requested to abide by the following conventions: - - 1) Each review comment from an official maintainer would like a response, which would better enhance the contribution of the open source community. - - - If you agree to the review opinion and modify it accordingly, give a simple Done. - - If you disagree with the review, please give your own reasons for refuting. - - 2)If there are many reviews: - - - Please give an overview of the changes. - - Please reply with `start a review', not directly. The reason is that each reply sends an e-mail message, which can cause a mail disaster. diff --git a/doc/doc_en/community_contribution_en.md b/doc/doc_en/community_contribution_en.md deleted file mode 100644 index e2d24f6144..0000000000 --- a/doc/doc_en/community_contribution_en.md +++ /dev/null @@ -1,100 +0,0 @@ -# COMMUNITY CONTRIBUTION - -Thank you for your support and interest in PaddleOCR. The goal of PaddleOCR is to build a professional, harmonious and supportive open source community with developers. This document presents existing community contributions, explanations for various contributions, and new opportunities and processes to make the contribution process more efficient and clear. - -PaddleOCR wants to help any developer with a dream realize their vision and enjoy the joy of creating value through the power of AI. - ---- - - - - - -> The picture above shows PaddleOCR's current Contributor, updated regularly - -## 1. COMMUNITY CONTRIBUTION - -### 1.1 PaddleOCR BASED COMMUNITY PROJECT - -- 【The lastest】 [FastOCRLabel](https://gitee.com/BaoJianQiang/FastOCRLabel): Complete C# version annotation tool (@ [包建强](https://gitee.com/BaoJianQiang) ) - -#### 1.1.1 UNIVERSAL TOOL - -- [DangoOCR offline version](https://github.com/PantsuDango/DangoOCR):Universal desktop instant translation tool (@ [PantsuDango](https://github.com/PantsuDango)) -- [scr2txt](https://github.com/lstwzd/scr2txt):Screenshot to Text tool (@ [lstwzd](https://github.com/lstwzd)) -- [AI Studio project](https://aistudio.baidu.com/aistudio/projectdetail/1054614?channelType=0&channel=0):English video automatically generates subtitles( @ [叶月水狐](https://aistudio.baidu.com/aistudio/personalcenter/thirdview/322052)) - -#### 1.1.2 VERTICAL SCENE TOOLS - -- [id_card_ocr](https://github.com/baseli/id_card_ocr):Identification of copy of ID card(@ [baseli](https://github.com/baseli)) -- [Paddle_Table_Image_Reader](https://github.com/thunder95/Paddle_Table_Image_Reader): A data assistant that can read tables and pictures(@ [thunder95](https://github.com/thunder95])) - -#### 1.1.3 PRE AND POST PROCESSING - -- [paddleOCRCorrectOutputs](https://github.com/yuranusduke/paddleOCRCorrectOutputs):Get the key-value of OCR recognition result (@ [yuranusduke](https://github.com/yuranusduke)) - -### 1.2 NEW FEATURES FOR PaddleOCR - -- Thanks [authorfu](https://github.com/authorfu) for contributing Android([#340](https://github.com/PaddlePaddle/PaddleOCR/pull/340)) and [xiadeye](https://github.com/xiadeye) for contributing IOS demo code([#325](https://github.com/PaddlePaddle/PaddleOCR/pull/325)). -- Thanks [tangmq](https://gitee.com/tangmq) for adding docker deployment service to PaddleOCR to support quick release of callable restful API services([#507](https://github.com/PaddlePaddle/PaddleOCR/pull/507)). -- Thanks [lijinhan](https://github.com/lijinhan) for adding Java springboot to PaddleOCR and call OCR hubserving interface to complete the use of OCR service deployment([#1027](https://github.com/PaddlePaddle/PaddleOCR/pull/1027)). -- Thanks [Evezerest](https://github.com/Evezerest), [ninetailskim](https://github.com/ninetailskim), [edencfc](https://github.com/edencfc), [BeyondYourself](https://github.com/BeyondYourself), [1084667371](https://github.com/1084667371) for contributing complete code of [PPOCRLabel](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/PPOCRLabel/README_ch.md). - -### 1.3 CODE AND DOCUMENT OPTIMIZATION - -- Thanks [zhangxin](https://github.com/ZhangXinNan)([Blog](https://blog.csdn.net/sdlypyzq)) for contributing new visualization methods and adding .gitgnore, handling the problem of manually setting the PYTHONPATH environment variable([#210](https://github.com/PaddlePaddle/PaddleOCR/pull/210)). -- Thanks [lyl120117](https://github.com/lyl120117) for contributing code to print network structure([#304](https://github.com/PaddlePaddle/PaddleOCR/pull/304)). -- Thanks [BeyondYourself](https://github.com/BeyondYourself) for making a lot of great suggestions for PaddleOCR and simplifying some code styles of paddleocr([so many commits)](https://github.com/PaddlePaddle/PaddleOCR/commits?author=BeyondYourself). -- Thanks [Khanh Tran](https://github.com/xxxpsyduck) and [Karl Horky](https://github.com/karlhorky) for contributing modifing English documents. - -### 1.4 MULTILINGUAL CORPUS - -- Thanks [xiangyubo](https://github.com/xiangyubo) for contributing handwritting Chinese OCR dataset([#321](https://github.com/PaddlePaddle/PaddleOCR/pull/321)). -- Thanks [Mejans](https://github.com/Mejans) for contributing dictionary and corpus of the new language Occitan to PaddleOCR([#954](https://github.com/PaddlePaddle/PaddleOCR/pull/954)). - -## 2. CONTRIBUTION ILLUSTRATING - -### 2.1 NEW FUNCTION CLASS - -PaddleOCR welcomes community contributions to various services, deployment examples and software applications with paddleOCR as the core. Certified community contributions will be added to the above community contribution table to increase exposure for the majority of developers, which is also the glory of PaddleOCR, including: - -- Project form: the project code certified by the official community shall have good specifications and structure, and shall be equipped with a detailed README.md, which describes how to use the project. Through add a line 'paddleocr' to the requirements.txt, which can be automatically included in the usedby of paddleocr. - -- Integration method: if it is an update to the existing PaddleOCR tool, it will be integrated into the main repo. If a new function is expanded for paddleocr, please contact the official personnel first to confirm whether the project is integrated into the master repo, *even if the new function is not integrated into the master repo, we will also increase the exposure of your personal project in the way of community contribution.* - - -### 2.2 CODE OPTIMIZATION - -If you encounter code bugs and unexpected functions when using PaddleOCR, you can contribute your modifications to PaddleOCR, including: - -- Python code specifications are available for reference [Appendix 1:Python code specifications](./code_and_doc.md/#Appendix1). - -- Before submitting the code, please confirm again and again that no new bugs will be introduced, and describe the optimization points in the PR. If the PR solves an issue, please connect to the issue in the PR. All PR shall comply with the requirements in Appendix [3.2.10 Some conventions for submitting code.](./code_and_doc.md/#Some conventions for submitting code) - -- Please refer to the below before submitting. If you are not familiar with the git submission process, you can also refer to Section 3.2 of [Appendix 3: description of Pull Request](./code_and_doc.md/#Appendix3).If you are not familiar with the git submission process, you can also refer to Section 3.2 of Appendix 3. - -**Finally, please add the label Third Party in the title of PR and @ Everest in the description , PR with this label will be treated with high priority`[third-part]`.** - -### 2.3 DOCUMENT OPTIMIZATION - -If you encounter problems such as unclear document description, missing description and invalid link when using PaddleOCR, you can contribute your modifications to PaddleOCR. For document writing specifications, please refer to [Appendix 2: document specifications](./code_and_doc.md/#Appendix2). **Finally, please add the label Third Party in the title of PR and @ Everest in the description , PR with this label will be treated with high priority`[third-party].** - -## 3. MORE CONTRIBUTION OPPORTUNITIES - -We encourage developers to use PaddleOCR to realize their ideas. At the same time, we also list some valuable development directions after analysis, which are collected in the regular season of community projects as a whole. - -## 4. CONTACT US - -We very much welcome developers to contact us before they intend to contribute code, documents, corpus and other contents to PaddleOCR, which can greatly reduce the communication cost in the PR process. At the same time, if you find some ideas difficult to realize personally, we can also recruit like-minded developers for the project in the form of SIG. Projects funded through SIG channels will receive deep R & D support and operational resources (such as official account publicity, live broadcast lessons, etc.). - -Our recommended contribution process is: - -- By adding the `[Third Party]` mark in the topic of GitHub issue, explain the problems encountered (and the ideas to solve) or the functions to be expanded, and wait for the reply of the person on duty. For example, ` [Third Party] contributes IOS examples to PaddleOCR`. -- After communicating with us and confirming that the technical scheme or bugs and optimization points are correct, add functions or modify them accordingly, and the codes and documents shall comply with relevant specifications. -- PR links to the above issue and waits for review. - -## 5. THANKS AND FOLLOW-UP - - - After the code is combined, the information will be updated in the first section of this document. The default link is GitHub name and home page. If you need to change the home page, you can also contact us. - - New important function classes will be advertised in the user group and enjoy the honor of the open source community. - - **If you have a PaddleOCR based project that does not appear in the above list, follow `4. CONTACT US` .** diff --git a/doc/doc_en/config_en.md b/doc/doc_en/config_en.md deleted file mode 100644 index b7c828f071..0000000000 --- a/doc/doc_en/config_en.md +++ /dev/null @@ -1,248 +0,0 @@ -# Configuration - -- [1. Optional Parameter List](#1-optional-parameter-list) -- [2. Introduction to Global Parameters of Configuration File](#2-introduction-to-global-parameters-of-configuration-file) -- [3. Multilingual Config File Generation](#3-multilingual-config-file-generation) - - - -## 1. Optional Parameter List - -The following list can be viewed through `--help` - -| FLAG | Supported script | Use | Defaults | Note | -| :----------------------: | :------------: | :---------------: | :--------------: | :-----------------: | -| -c | ALL | Specify configuration file to use | None | **Please refer to the parameter introduction for configuration file usage** | -| -o | ALL | set configuration options | None | Configuration using -o has higher priority than the configuration file selected with -c. E.g: -o Global.use_gpu=false | - - - -## 2. Introduction to Global Parameters of Configuration File - -Take rec_chinese_lite_train_v2.0.yml as an example -### Global - -| Parameter | Use | Defaults | Note | -| :----------------------: | :---------------------: | :--------------: | :--------------------: | -| use_gpu | Set using GPU or not | true | \ | -| epoch_num | Maximum training epoch number | 500 | \ | -| log_smooth_window | Log queue length, the median value in the queue each time will be printed | 20 | \ | -| print_batch_step | Set print log interval | 10 | \ | -| save_model_dir | Set model save path | output/{算法名称} | \ | -| save_epoch_step | Set model save interval | 3 | \ | -| eval_batch_step | Set the model evaluation interval | 2000 or [1000, 2000] | running evaluation every 2000 iters or evaluation is run every 2000 iterations after the 1000th iteration | -| cal_metric_during_train | Set whether to evaluate the metric during the training process. At this time, the metric of the model under the current batch is evaluated | true | \ | -| load_static_weights | Set whether the pre-training model is saved in static graph mode (currently only required by the detection algorithm) | true | \ | -| pretrained_model | Set the path of the pre-trained model | ./pretrain_models/CRNN/best_accuracy | \ | -| checkpoints | set model parameter path | None | Used to load parameters after interruption to continue training| -| use_visualdl | Set whether to enable visualdl for visual log display | False | [Tutorial](https://www.paddlepaddle.org.cn/paddle/visualdl) | -| use_wandb | Set whether to enable W&B for visual log display | False | [Documentation](https://docs.wandb.ai/) -| infer_img | Set inference image path or folder path | ./infer_img | \|| -| character_dict_path | Set dictionary path | ./ppocr/utils/ppocr_keys_v1.txt | If the character_dict_path is None, model can only recognize number and lower letters | -| max_text_length | Set the maximum length of text | 25 | \ | -| use_space_char | Set whether to recognize spaces | True | \| | -| label_list | Set the angle supported by the direction classifier | ['0','180'] | Only valid in angle classifier model | -| save_res_path | Set the save address of the test model results | ./output/det_db/predicts_db.txt | Only valid in the text detection model | - -### Optimizer ([ppocr/optimizer](../../ppocr/optimizer)) - -| Parameter | Use | Defaults | Note | -| :---------------------: | :---------------------: | :--------------: | :--------------------: | -| name | Optimizer class name | Adam | Currently supports`Momentum`,`Adam`,`RMSProp`, see [ppocr/optimizer/optimizer.py](../../ppocr/optimizer/optimizer.py) | -| beta1 | Set the exponential decay rate for the 1st moment estimates | 0.9 | \ | -| beta2 | Set the exponential decay rate for the 2nd moment estimates | 0.999 | \ | -| clip_norm | The maximum norm value | - | \ | -| **lr** | Set the learning rate decay method | - | \ | -| name | Learning rate decay class name | Cosine | Currently supports`Linear`,`Cosine`,`Step`,`Piecewise`, see[ppocr/optimizer/learning_rate.py](../../ppocr/optimizer/learning_rate.py) | -| learning_rate | Set the base learning rate | 0.001 | \ | -| **regularizer** | Set network regularization method | - | \ | -| name | Regularizer class name | L2 | Currently support`L1`,`L2`, see[ppocr/optimizer/regularizer.py](../../ppocr/optimizer/regularizer.py) | -| factor | Regularizer coefficient | 0.00001 | \ | - - -### Architecture ([ppocr/modeling](../../ppocr/modeling)) -In PaddleOCR, the network is divided into four stages: Transform, Backbone, Neck and Head - -| Parameter | Use | Defaults | Note | -| :---------------------: | :---------------------: | :--------------: | :--------------------: | -| model_type | Network Type | rec | Currently support`rec`,`det`,`cls` | -| algorithm | Model name | CRNN | See [algorithm_overview](./algorithm_overview_en.md) for the support list | -| **Transform** | Set the transformation method | - | Currently only recognition algorithms are supported, see [ppocr/modeling/transform](../../ppocr/modeling/transforms) for details | -| name | Transformation class name | TPS | Currently supports `TPS` | -| num_fiducial | Number of TPS control points | 20 | Ten on the top and bottom | -| loc_lr | Localization network learning rate | 0.1 | \ | -| model_name | Localization network size | small | Currently support`small`,`large` | -| **Backbone** | Set the network backbone class name | - | see [ppocr/modeling/backbones](../../ppocr/modeling/backbones) | -| name | backbone class name | ResNet | Currently support`MobileNetV3`,`ResNet` | -| layers | resnet layers | 34 | Currently support18,34,50,101,152,200 | -| model_name | MobileNetV3 network size | small | Currently support`small`,`large` | -| **Neck** | Set network neck | - | see[ppocr/modeling/necks](../../ppocr/modeling/necks) | -| name | neck class name | SequenceEncoder | Currently support`SequenceEncoder`,`DBFPN` | -| encoder_type | SequenceEncoder encoder type | rnn | Currently support`reshape`,`fc`,`rnn` | -| hidden_size | rnn number of internal units | 48 | \ | -| out_channels | Number of DBFPN output channels | 256 | \ | -| **Head** | Set the network head | - | see[ppocr/modeling/heads](../../ppocr/modeling/heads) | -| name | head class name | CTCHead | Currently support`CTCHead`,`DBHead`,`ClsHead` | -| fc_decay | CTCHead regularization coefficient | 0.0004 | \ | -| k | DBHead binarization coefficient | 50 | \ | -| class_dim | ClsHead output category number | 2 | \ | - - -### Loss ([ppocr/losses](../../ppocr/losses)) - -| Parameter | Use | Defaults | Note | -| :---------------------: | :---------------------: | :--------------: | :--------------------: | -| name | loss class name | CTCLoss | Currently support`CTCLoss`,`DBLoss`,`ClsLoss` | -| balance_loss | Whether to balance the number of positive and negative samples in DBLossloss (using OHEM) | True | \ | -| ohem_ratio | The negative and positive sample ratio of OHEM in DBLossloss | 3 | \ | -| main_loss_type | The loss used by shrink_map in DBLossloss | DiceLoss | Currently support`DiceLoss`,`BCELoss` | -| alpha | The coefficient of shrink_map_loss in DBLossloss | 5 | \ | -| beta | The coefficient of threshold_map_loss in DBLossloss | 10 | \ | - -### PostProcess ([ppocr/postprocess](../../ppocr/postprocess)) - -| Parameter | Use | Defaults | Note | -| :---------------------: | :---------------------: | :--------------: | :--------------------: | -| name | Post-processing class name | CTCLabelDecode | Currently support`CTCLoss`,`AttnLabelDecode`,`DBPostProcess`,`ClsPostProcess` | -| thresh | The threshold for binarization of the segmentation map in DBPostProcess | 0.3 | \ | -| box_thresh | The threshold for filtering output boxes in DBPostProcess. Boxes below this threshold will not be output | 0.7 | \ | -| max_candidates | The maximum number of text boxes output in DBPostProcess | 1000 | | -| unclip_ratio | The unclip ratio of the text box in DBPostProcess | 2.0 | \ | - -### Metric ([ppocr/metrics](../../ppocr/metrics)) - -| Parameter | Use | Defaults | Note | -| :---------------------: | :---------------------: | :--------------: | :--------------------: | -| name | Metric method name | CTCLabelDecode | Currently support`DetMetric`,`RecMetric`,`ClsMetric` | -| main_indicator | Main indicators, used to select the best model | acc | For the detection method is hmean, the recognition and classification method is acc | - -### Dataset ([ppocr/data](../../ppocr/data)) -| Parameter | Use | Defaults | Note | -| :---------------------: | :---------------------: | :--------------: | :--------------------: | -| **dataset** | Return one sample per iteration | - | - | -| name | dataset class name | SimpleDataSet | Currently support`SimpleDataSet`,`LMDBDataSet` | -| data_dir | Image folder path | ./train_data | \ | -| label_file_list | Groundtruth file path | ["./train_data/train_list.txt"] | This parameter is not required when dataset is LMDBDataSet | -| ratio_list | Ratio of data set | [1.0] | If there are two train_lists in label_file_list and ratio_list is [0.4,0.6], 40% will be sampled from train_list1, and 60% will be sampled from train_list2 to combine the entire dataset | -| transforms | List of methods to transform images and labels | [DecodeImage,CTCLabelEncode,RecResizeImg,KeepKeys] | see[ppocr/data/imaug](../../ppocr/data/imaug) | -| **loader** | dataloader related | - | | -| shuffle | Does each epoch disrupt the order of the data set | True | \ | -| batch_size_per_card | Single card batch size during training | 256 | \ | -| drop_last | Whether to discard the last incomplete mini-batch because the number of samples in the data set cannot be divisible by batch_size | True | \ | -| num_workers | The number of sub-processes used to load data, if it is 0, the sub-process is not started, and the data is loaded in the main process | 8 | \ | - -### Weights & Biases ([W&B](../../ppocr/utils/loggers/wandb_logger.py)) -| Parameter | Use | Defaults | Note | -| :---------------------: | :---------------------: | :--------------: | :--------------------: | -| project | Project to which the run is to be logged | uncategorized | \ -| name | Alias/Name of the run | Randomly generated by wandb | \ -| id | ID of the run | Randomly generated by wandb | \ -| entity | User or team to which the run is being logged | The logged in user | \ -| save_dir | local directory in which all the models and other data is saved | wandb | \ -| config | model configuration | None | \ - - - - -## 3. Multilingual Config File Generation - -PaddleOCR currently supports recognition for 80 languages (besides Chinese). A multi-language configuration file template is -provided under the path `configs/rec/multi_languages`: [rec_multi_language_lite_train.yml](../../configs/rec/multi_language/rec_multi_language_lite_train.yml)。 - -There are two ways to create the required configuration file: - -1. Automatically generated by script - -Script [generate_multi_language_configs.py](../../configs/rec/multi_language/generate_multi_language_configs.py) can help you generate configuration files for multi-language models. - -- Take Italian as an example, if your data is prepared in the following format: - ``` - |-train_data - |- it_train.txt # train_set label - |- it_val.txt # val_set label - |- data - |- word_001.jpg - |- word_002.jpg - |- word_003.jpg - | ... - ``` - - You can use the default parameters to generate a configuration file: - - ```bash - # The code needs to be run in the specified directory - cd PaddleOCR/configs/rec/multi_language/ - # Set the configuration file of the language to be generated through the -l or --language parameter. - # This command will write the default parameters into the configuration file - python3 generate_multi_language_configs.py -l it - ``` - -- If your data is placed in another location, or you want to use your own dictionary, you can generate the configuration file by specifying the relevant parameters: - - ```bash - # -l or --language field is required - # --train to modify the training set - # --val to modify the validation set - # --data_dir to modify the data set directory - # --dict to modify the dict path - # -o to modify the corresponding default parameters - cd PaddleOCR/configs/rec/multi_language/ - python3 generate_multi_language_configs.py -l it \ # language - --train {path/of/train_label.txt} \ # path of train_label - --val {path/of/val_label.txt} \ # path of val_label - --data_dir {train_data/path} \ # root directory of training data - --dict {path/of/dict} \ # path of dict - -o Global.use_gpu=False # whether to use gpu - ... - - ``` -Italian is made up of Latin letters, so after executing the command, you will get the rec_latin_lite_train.yml. - -2. Manually modify the configuration file - - You can also manually modify the following fields in the template: - - ``` - Global: - use_gpu: True - epoch_num: 500 - ... - character_dict_path: {path/of/dict} # path of dict - - Train: - dataset: - name: SimpleDataSet - data_dir: train_data/ # root directory of training data - label_file_list: ["./train_data/train_list.txt"] # train label path - ... - - Eval: - dataset: - name: SimpleDataSet - data_dir: train_data/ # root directory of val data - label_file_list: ["./train_data/val_list.txt"] # val label path - ... - - ``` - - -Currently, the multi-language algorithms supported by PaddleOCR are: - -| Configuration file | Algorithm name | backbone | trans | seq | pred | language | -| :--------: | :-------: | :-------: | :-------: | :-----: | :-----: | :-----: | -| rec_chinese_cht_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | chinese traditional | -| rec_en_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | English(Case sensitive) | -| rec_french_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | French | -| rec_ger_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | German | -| rec_japan_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | Japanese | -| rec_korean_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | Korean | -| rec_latin_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | Latin | -| rec_arabic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | arabic | -| rec_cyrillic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | cyrillic | -| rec_devanagari_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | devanagari | - -For more supported languages, please refer to : [Multi-language model](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/doc/doc_en/multi_languages_en.md#4-support-languages-and-abbreviations) - -The multi-language model training method is the same as the Chinese model. The training data set is 100w synthetic data. A small amount of fonts and test data can be downloaded using the following two methods. -* [Baidu Netdisk](https://pan.baidu.com/s/1bS_u207Rm7YbY33wOECKDA),Extraction code:frgi. -* [Google drive](https://drive.google.com/file/d/18cSWX7wXSy4G0tbKJ0d9PuIaiwRLHpjA/view) diff --git a/doc/doc_en/customize_en.md b/doc/doc_en/customize_en.md deleted file mode 100644 index 80b8302238..0000000000 --- a/doc/doc_en/customize_en.md +++ /dev/null @@ -1,30 +0,0 @@ -# HOW TO MAKE YOUR OWN LIGHTWEIGHT OCR MODEL? - -The process of making a customized ultra-lightweight OCR models can be divided into three steps: training text detection model, training text recognition model, and concatenate the predictions from previous steps. - -## STEP1: TRAIN TEXT DETECTION MODEL - -PaddleOCR provides two text detection algorithms: EAST and DB. Both support MobileNetV3 and ResNet50_vd backbone networks, select the corresponding configuration file as needed and start training. For example, to train with MobileNetV3 as the backbone network for DB detection model : -``` -python3 tools/train.py -c configs/det/det_mv3_db.yml 2>&1 | tee det_db.log -``` -For more details about data preparation and training tutorials, refer to the documentation [Text detection model training/evaluation/prediction](./detection_en.md) - -## STEP2: TRAIN TEXT RECOGNITION MODEL - -PaddleOCR provides four text recognition algorithms: CRNN, Rosetta, STAR-Net, and RARE. They all support two backbone networks: MobileNetV3 and ResNet34_vd, select the corresponding configuration files as needed to start training. For example, to train a CRNN recognition model that uses MobileNetV3 as the backbone network: -``` -python3 tools/train.py -c configs/rec/rec_chinese_lite_train.yml 2>&1 | tee rec_ch_lite.log -``` -For more details about data preparation and training tutorials, refer to the documentation [Text recognition model training/evaluation/prediction](./recognition_en.md) - -## STEP3: CONCATENATE PREDICTIONS - -PaddleOCR provides a concatenation tool for detection and recognition models, which can connect any trained detection model and any recognition model into a two-stage text recognition system. The input image goes through four main stages: text detection, text rectification, text recognition, and score filtering to output the text position and recognition results, and at the same time, you can choose to visualize the results. - -When performing prediction, you need to specify the path of a single image or a image folder through the parameter `image_dir`, the parameter `det_model_dir` specifies the path of detection model, and the parameter `rec_model_dir` specifies the path of recognition model. The visualized results are saved to the `./inference_results` folder by default. - -``` -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/11.jpg" --det_model_dir="./inference/det/" --rec_model_dir="./inference/rec/" -``` -For more details about text detection and recognition concatenation, please refer to the document [Inference](./inference_en.md) diff --git a/doc/doc_en/data_annotation_en.md b/doc/doc_en/data_annotation_en.md deleted file mode 100644 index 71c8f4f61c..0000000000 --- a/doc/doc_en/data_annotation_en.md +++ /dev/null @@ -1,20 +0,0 @@ -# DATA ANNOTATION TOOLS -There are the commonly used data annotation tools, which will be continuously updated. Welcome to contribute tools~ - -### 1. labelImg -- Tool description: Rectangular label -- Tool address: https://github.com/tzutalin/labelImg -- Sketch diagram: -![labelimg](../datasets/labelimg.jpg) - -### 2. roLabelImg -- Tool description: Label tool rewritten based on labelImg, supporting rotating rectangular label -- Tool address: https://github.com/cgvict/roLabelImg -- Sketch diagram: -![roLabelImg](../datasets/roLabelImg.png) - -### 3. labelme -- Tool description: Support four points, polygons, circles and other labels -- Tool address: https://github.com/wkentaro/labelme -- Sketch diagram: -![labelme](../datasets/labelme.jpg) diff --git a/doc/doc_en/data_synthesis_en.md b/doc/doc_en/data_synthesis_en.md deleted file mode 100644 index f05a793cf5..0000000000 --- a/doc/doc_en/data_synthesis_en.md +++ /dev/null @@ -1,12 +0,0 @@ -# DATA SYNTHESIS TOOLS - -In addition to open source data, users can also use synthesis tools to synthesize data. -There are the commonly used data synthesis tools, which will be continuously updated. Welcome to contribute tools~ - -* [Text_renderer](https://github.com/Sanster/text_renderer) -* [SynthText](https://github.com/ankush-me/SynthText) -* [SynthText_Chinese_version](https://github.com/JarveeLee/SynthText_Chinese_version) -* [TextRecognitionDataGenerator](https://github.com/Belval/TextRecognitionDataGenerator) -* [SynthText3D](https://github.com/MhLiao/SynthText3D) -* [UnrealText](https://github.com/Jyouhou/UnrealText/) -* [SynthTIGER](https://github.com/clovaai/synthtiger) diff --git a/doc/doc_en/dataset/datasets_en.md b/doc/doc_en/dataset/datasets_en.md deleted file mode 100644 index 2a7c9c5df2..0000000000 --- a/doc/doc_en/dataset/datasets_en.md +++ /dev/null @@ -1,61 +0,0 @@ -## DATASET -This is a collection of commonly used Chinese datasets, which is being updated continuously. You are welcome to contribute to this list~ -- [ICDAR2019-LSVT](#ICDAR2019-LSVT) -- [ICDAR2017-RCTW-17](#ICDAR2017-RCTW-17) -- [Chinese Street View Text Recognition](#中文街景文字识别) -- [Chinese Document Text Recognition](#中文文档文字识别) -- [ICDAR2019-ArT](#ICDAR2019-ArT) - -In addition to opensource data, users can also use synthesis tools to synthesize data themselves. Current available synthesis tools include [text_renderer](https://github.com/Sanster/text_renderer), [SynthText](https://github.com/ankush-me/SynthText), [TextRecognitionDataGenerator](https://github.com/Belval/TextRecognitionDataGenerator), etc. - - -#### 1. ICDAR2019-LSVT -- **Data sources**:https://ai.baidu.com/broad/introduction?dataset=lsvt -- **Introduction**: A total of 45w Chinese street view images, including 5w (2w test + 3w training) fully labeled data (text coordinates + text content), 40w weakly labeled data (text content only), as shown in the following figure: - ![](../../datasets/LSVT_1.jpg) - - (a) Fully labeled data - - ![](../../datasets/LSVT_2.jpg) - - (b) Weakly labeled data -- **Download link**:https://ai.baidu.com/broad/download?dataset=lsvt - - -#### 2. ICDAR2017-RCTW-17 -- **Data sources**:https://rctw.vlrlab.net/ -- **Introduction**:It contains 12000 + images, most of them are collected in the wild through mobile camera. Some are screenshots. These images show a variety of scenes, including street views, posters, menus, indoor scenes and screenshots of mobile applications. - ![](../../datasets/rctw.jpg) -- **Download link**:https://rctw.vlrlab.net/dataset/ - - -#### 3. Chinese Street View Text Recognition -- **Data sources**:https://aistudio.baidu.com/aistudio/competition/detail/8 -- **Introduction**:A total of 290000 pictures are included, of which 210000 are used as training sets (with labels) and 80000 are used as test sets (without labels). The dataset is collected from the Chinese street view, and is formed by by cutting out the text line area (such as shop signs, landmarks, etc.) in the street view picture. All the images are preprocessed: by using affine transform, the text area is proportionally mapped to a picture with a height of 48 pixels, as shown in the figure: - - ![](../../datasets/ch_street_rec_1.png) - (a) Label: 魅派集成吊顶 - ![](../../datasets/ch_street_rec_2.png) - (b) Label: 母婴用品连锁 -- **Download link** -https://aistudio.baidu.com/aistudio/datasetdetail/8429 - - -#### 4. Chinese Document Text Recognition -- **Data sources**:https://github.com/YCG09/chinese_ocr -- **Introduction**: - - A total of 3.64 million pictures are divided into training set and validation set according to 99:1. - - Using Chinese corpus (news + classical Chinese), the data is randomly generated through changes in font, size, grayscale, blur, perspective, stretching, etc. - - 5990 characters including Chinese characters, English letters, numbers and punctuation(Characters set: https://github.com/YCG09/chinese_ocr/blob/master/train/char_std_5990.txt ) - - Each sample is fixed with 10 characters, and the characters are randomly intercepted from the sentences in the corpus - - Image resolution is 280x32 - ![](../../datasets/ch_doc1.jpg) - ![](../../datasets/ch_doc3.jpg) -- **Download link**:https://pan.baidu.com/s/1QkI7kjah8SPHwOQ40rS1Pw (Password: lu7m) - - -#### 5、ICDAR2019-ArT -- **Data source**:https://ai.baidu.com/broad/introduction?dataset=art -- **Introduction**:It includes 10166 images, 5603 in training sets and 4563 in test sets. It is composed of three parts: total text, scut-ctw1500 and Baidu curved scene text, including text with various shapes such as horizontal, multi-directional and curved. - ![](../../datasets/ArT.jpg) -- **Download link**:https://ai.baidu.com/broad/download?dataset=art diff --git a/doc/doc_en/dataset/handwritten_datasets_en.md b/doc/doc_en/dataset/handwritten_datasets_en.md deleted file mode 100644 index 2059549601..0000000000 --- a/doc/doc_en/dataset/handwritten_datasets_en.md +++ /dev/null @@ -1,28 +0,0 @@ -# Handwritten OCR dataset -Here we have sorted out the commonly used handwritten OCR dataset datasets, which are being updated continuously. We welcome you to contribute datasets ~ -- [Institute of automation, Chinese Academy of Sciences - handwritten Chinese dataset](#Institute of automation, Chinese Academy of Sciences - handwritten Chinese dataset) -- [NIST handwritten single character dataset - English](#NIST handwritten single character dataset - English) - - -## Institute of automation, Chinese Academy of Sciences - handwritten Chinese dataset -- **Data source**:http://www.nlpr.ia.ac.cn/databases/handwriting/Download.html -- **Data introduction**: - * It includes online and offline handwritten data,`HWDB1.0~1.2` has totally 3895135 handwritten single character samples, which belong to 7356 categories (7185 Chinese characters and 171 English letters, numbers and symbols);`HWDB2.0~2.2` has totally 5091 pages of images, which are divided into 52230 text lines and 1349414 words. All text and text samples are stored as grayscale images. Some sample words are shown below. - - ![](../../datasets/CASIA_0.jpg) - -- **Download address**:http://www.nlpr.ia.ac.cn/databases/handwriting/Download.html -- **使用建议**:Data for single character, white background, can form a large number of text lines for training. White background can be processed into transparent state, which is convenient to add various backgrounds. For the case of semantic needs, it is suggested to extract single character from real corpus to form text lines. - - - -## NIST handwritten single character dataset - English(NIST Handprinted Forms and Characters Database) - -- **Data source**: [https://www.nist.gov/srd/nist-special-database-19](https://www.nist.gov/srd/nist-special-database-19) - -- **Data introduction**: NIST19 dataset is suitable for handwritten document and character recognition model training. It is extracted from the handwritten sample form of 3600 authors and contains 810000 character images in total. Nine of them are shown below. - - ![](../../datasets/nist_demo.png) - - -- **Download address**: [https://www.nist.gov/srd/nist-special-database-19](https://www.nist.gov/srd/nist-special-database-19) diff --git a/doc/doc_en/dataset/kie_datasets_en.md b/doc/doc_en/dataset/kie_datasets_en.md deleted file mode 100644 index ff839097c5..0000000000 --- a/doc/doc_en/dataset/kie_datasets_en.md +++ /dev/null @@ -1,49 +0,0 @@ -## Key Information Extraction dataset - -Here are the common datasets key information extraction, which are being updated continuously. Welcome to contribute datasets. - -- [FUNSD dataset](#funsd) -- [XFUND dataset](#xfund) -- [wildreceipt dataset](#wildreceipt-dataset) - - -#### 1. FUNSD dataset -- **Data source**: https://guillaumejaume.github.io/FUNSD/ -- **Data Introduction**: The FUNSD dataset is a dataset for form comprehension. It contains 199 real, fully annotated scanned images, including market reports, advertisements, and academic reports, etc., and is divided into 149 training set and 50 test set. The FUNSD dataset is suitable for many types of DocVQA tasks, such as field-level entity classification, field-level entity connection, etc. Part of the image and the annotation box visualization are shown below: -
- - -
- In the figure, the orange area represents `header`, the light blue area represents `question`, the green area represents `answer`, and the pink area represents `other`. - -- **Download address**: https://guillaumejaume.github.io/FUNSD/download/ - - -#### 2. XFUND dataset -- **Data source**: https://github.com/doc-analysis/XFUND -- **Data introduction**: XFUND is a multilingual form comprehension dataset, which contains form data in 7 different languages, and all are manually annotated in the form of key-value pairs. The data for each language contains 199 form data, which are divided into 149 training sets and 50 test sets. Part of the image and the annotation box visualization are shown below. - -
- - -
- -- **Download address**: https://github.com/doc-analysis/XFUND/releases/tag/v1.0 - - - -## 3. wildreceipt dataset - -- **Data source**: https://arxiv.org/abs/2103.14470 -- **Data introduction**: wildreceipt is an English receipt dataset, which contains 26 different categories. There are 1267 training images and 472 evaluation images, in which 50,000 textlines and boxes are annotated. Part of the image and the annotation box visualization are shown below. - -
- - -
- -**Note:** Boxes with category `Ignore` or `Others` are not visualized here. - -- **Download address**: - - Offical dataset: [link](https://download.openmmlab.com/mmocr/data/wildreceipt.tar) - - Dataset converted for PaddleOCR training process: [link](https://paddleocr.bj.bcebos.com/ppstructure/dataset/wildreceipt.tar) diff --git a/doc/doc_en/dataset/layout_datasets_en.md b/doc/doc_en/dataset/layout_datasets_en.md deleted file mode 100644 index 54c88609d0..0000000000 --- a/doc/doc_en/dataset/layout_datasets_en.md +++ /dev/null @@ -1,55 +0,0 @@ -## Layout Analysis Dataset - -Here are the common datasets of layout anlysis, which are being updated continuously. Welcome to contribute datasets. - -- [PubLayNet dataset](#publaynet) -- [CDLA dataset](#CDLA) -- [TableBank dataset](#TableBank) - - -Most of the layout analysis datasets are object detection datasets. In addition to open source datasets, you can also label or synthesize datasets using tools such as [labelme](https://github.com/wkentaro/labelme) and so on. - - - - -#### 1. PubLayNet dataset - -- **Data source**: https://github.com/ibm-aur-nlp/PubLayNet -- **Data introduction**: The PubLayNet dataset contains 350000 training images and 11000 validation images. There are 5 categories in total, namely: `text, title, list, table, figure`. Some images and their annotations as shown below. - -
- - -
- -- **Download address**: https://developer.ibm.com/exchanges/data/all/publaynet/ -- **Note**: When using this dataset, you need to follow [CDLA-Permissive](https://cdla.io/permissive-1-0/) license. - - - - -#### 2、CDLA数据集 -- **Data source**: https://github.com/buptlihang/CDLA -- **Data introduction**: CDLA dataset contains 5000 training images and 1000 validation images with 10 categories, which are `Text, Title, Figure, Figure caption, Table, Table caption, Header, Footer, Reference, Equation`. Some images and their annotations as shown below. - -
- - -
- -- **Download address**: https://github.com/buptlihang/CDLA -- **Note**: When you train detection model on CDLA dataset using [PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection/tree/develop), you need to remove the label `__ignore__` and `_background_`. - - - -#### 3、TableBank dataet -- **Data source**: https://doc-analysis.github.io/tablebank-page/index.html -- **Data introduction**: TableBank dataset contains 2 types of document: Latex (187199 training images, 7265 validation images and 5719 testing images) and Word (73383 training images 2735 validation images and 2281 testing images). Some images and their annotations as shown below. - -
- - -
- -- **Data source**: https://doc-analysis.github.io/tablebank-page/index.html -- **Note**: When using this dataset, you need to follow [Apache-2.0](https://github.com/doc-analysis/TableBank/blob/master/LICENSE) license. diff --git a/doc/doc_en/dataset/ocr_datasets_en.md b/doc/doc_en/dataset/ocr_datasets_en.md deleted file mode 100644 index 0b9abd529d..0000000000 --- a/doc/doc_en/dataset/ocr_datasets_en.md +++ /dev/null @@ -1,157 +0,0 @@ -# OCR datasets - -- [1. Text detection](#1-text-detection) - - [1.1 PaddleOCR text detection format annotation](#11-paddleocr-text-detection-format-annotation) - - [1.2 Public dataset](#12-public-dataset) - - [1.2.1 ICDAR 2015](#121-icdar-2015) -- [2. Text recognition](#2-text-recognition) - - [2.1 PaddleOCR text recognition format annotation](#21-paddleocr-text-recognition-format-annotation) - - [2.2 Public dataset](#22-public-dataset) - - [2.1 ICDAR 2015](#21-icdar-2015) -- [3. Data storage path](#3-data-storage-path) - -Here is a list of public datasets commonly used in OCR, which are being continuously updated. Welcome to contribute datasets~ - -## 1. Text detection - -### 1.1 PaddleOCR text detection format annotation - -The annotation file formats supported by the PaddleOCR text detection algorithm are as follows, separated by "\t": -``` -" Image file name Image annotation information encoded by json.dumps" -ch4_test_images/img_61.jpg [{"transcription": "MASA", "points": [[310, 104], [416, 141], [418, 216], [312, 179]]}, {...}] -``` -The image annotation after **json.dumps()** encoding is a list containing multiple dictionaries. - -The `points` in the dictionary represent the coordinates (x, y) of the four points of the text box, arranged clockwise from the point at the upper left corner. - -`transcription` represents the text of the current text box. **When its content is "###" it means that the text box is invalid and will be skipped during training.** - -If you want to train PaddleOCR on other datasets, please build the annotation file according to the above format. - -### 1.2 Public dataset -| dataset | Image download link | PaddleOCR format annotation download link | -|---|---|---| -| ICDAR 2015 | https://rrc.cvc.uab.es/?ch=4&com=downloads | [train](https://paddleocr.bj.bcebos.com/dataset/train_icdar2015_label.txt) / [test](https://paddleocr.bj.bcebos.com/dataset/test_icdar2015_label.txt) | -| ctw1500 | https://paddleocr.bj.bcebos.com/dataset/ctw1500.zip | Included in the downloaded image zip | -| total text | https://paddleocr.bj.bcebos.com/dataset/total_text.tar | Included in the downloaded image zip | - -#### 1.2.1 ICDAR 2015 - -The icdar2015 dataset contains train set which has 1000 images obtained with wearable cameras and test set which has 500 images obtained with wearable cameras. The icdar2015 dataset can be downloaded from the link in the table above. Registration is required for downloading. - - -After registering and logging in, download the part marked in the red box in the figure below. And, the content downloaded by `Training Set Images` should be saved as the folder `icdar_c4_train_imgs`, and the content downloaded by `Test Set Images` is saved as the folder `ch4_test_images` - -

- -

- -Decompress the downloaded dataset to the working directory, assuming it is decompressed under PaddleOCR/train_data/. Then download the PaddleOCR format annotation file from the table above. - -PaddleOCR also provides a data format conversion script, which can convert the official website label to the PaddleOCR format. The data conversion tool is in `ppocr/utils/gen_label.py`, here is the training set as an example: -``` -# Convert the label file downloaded from the official website to train_icdar2015_label.txt -python gen_label.py --mode="det" --root_path="/path/to/icdar_c4_train_imgs/" \ - --input_path="/path/to/ch4_training_localization_transcription_gt" \ - --output_label="/path/to/train_icdar2015_label.txt" -``` - -After decompressing the data set and downloading the annotation file, PaddleOCR/train_data/ has two folders and two files, which are: -``` -/PaddleOCR/train_data/icdar2015/text_localization/ - └─ icdar_c4_train_imgs/ Training data of icdar dataset - └─ ch4_test_images/ Testing data of icdar dataset - └─ train_icdar2015_label.txt Training annotation of icdar dataset - └─ test_icdar2015_label.txt Test annotation of icdar dataset -``` - - -## 2. Text recognition - -### 2.1 PaddleOCR text recognition format annotation - -The text recognition algorithm in PaddleOCR supports two data formats: - - `lmdb` is used to train data sets stored in lmdb format, use [lmdb_dataset.py](../../../ppocr/data/lmdb_dataset.py) to load; - - `common dataset` is used to train data sets stored in text files, use [simple_dataset.py](../../../ppocr/data/simple_dataset.py) to load. - - -If you want to use your own data for training, please refer to the following to organize your data. - -- Training set - -It is recommended to put the training images in the same folder, and use a txt file (rec_gt_train.txt) to store the image path and label. The contents of the txt file are as follows: - -* Note: by default, the image path and image label are split with \t, if you use other methods to split, it will cause training error - -``` -" Image file name Image annotation " - -train_data/rec/train/word_001.jpg 简单可依赖 -train_data/rec/train/word_002.jpg 用科技让复杂的世界更简单 -... -``` - -The final training set should have the following file structure: - -``` -|-train_data - |-rec - |- rec_gt_train.txt - |- train - |- word_001.png - |- word_002.jpg - |- word_003.jpg - | ... -``` - -- Test set - -Similar to the training set, the test set also needs to be provided a folder containing all images (test) and a rec_gt_test.txt. The structure of the test set is as follows: - -``` -|-train_data - |-rec - |-ic15_data - |- rec_gt_test.txt - |- test - |- word_001.jpg - |- word_002.jpg - |- word_003.jpg - | ... -``` - -### 2.2 Public dataset -| dataset | Image download link | PaddleOCR format annotation download link | -|---|---|---| -| en benchmark(MJ, SJ, IIIT, SVT, IC03, IC13, IC15, SVTP, and CUTE.) | [DTRB](https://github.com/clovaai/deep-text-recognition-benchmark#download-lmdb-dataset-for-traininig-and-evaluation-from-here) | LMDB format, which can be loaded directly with [lmdb_dataset.py](../../../ppocr/data/lmdb_dataset.py) | -|ICDAR 2015| http://rrc.cvc.uab.es/?ch=4&com=downloads | [train](https://paddleocr.bj.bcebos.com/dataset/rec_gt_train.txt)/ [test](https://paddleocr.bj.bcebos.com/dataset/rec_gt_test.txt) | -| Multilingual datasets |[Baidu network disk](https://pan.baidu.com/s/1bS_u207Rm7YbY33wOECKDA) Extraction code: frgi
[google drive](https://drive.google.com/file/d/18cSWX7wXSy4G0tbKJ0d9PuIaiwRLHpjA/view) | Included in the downloaded image zip | - -#### 2.1 ICDAR 2015 - -The ICDAR 2015 dataset can be downloaded from the link in the table above for quick validation. The lmdb format dataset required by en benchmark can also be downloaded from the table above. - -Then download the PaddleOCR format annotation file from the table above. - -PaddleOCR also provides a data format conversion script, which can convert the ICDAR official website label to the data format supported by PaddleOCR. The data conversion tool is in `ppocr/utils/gen_label.py`, here is the training set as an example: - -``` -# Convert the label file downloaded from the official website to rec_gt_label.txt -python gen_label.py --mode="rec" --input_path="{path/of/origin/label}" --output_label="rec_gt_label.txt" -``` - -The data format is as follows, (a) is the original picture, (b) is the Ground Truth text file corresponding to each picture: - -![](../../datasets/icdar_rec.png) - -## 3. Data storage path - -The default storage path for PaddleOCR training data is `PaddleOCR/train_data`, if you already have a dataset on your disk, just create a soft link to the dataset directory: - -``` -# linux and mac os -ln -sf /train_data/dataset -# windows -mklink /d /train_data/dataset -``` diff --git a/doc/doc_en/dataset/table_datasets_en.md b/doc/doc_en/dataset/table_datasets_en.md deleted file mode 100644 index 70ca830979..0000000000 --- a/doc/doc_en/dataset/table_datasets_en.md +++ /dev/null @@ -1,42 +0,0 @@ -# Table Recognition Datasets - -- [Dataset Summary](#dataset-summary) -- [1. PubTabNet](#1-pubtabnet) -- [2. TAL Table Recognition Competition Dataset](#2-tal-table-recognition-competition-dataset) -- [3. WTW Chinese scene table dataset](#3-wtw-chinese-scene-table-dataset) - -Here are the commonly used table recognition datasets, which are being updated continuously. Welcome to contribute datasets~ - -## Dataset Summary - -| dataset | Image download link | PPOCR format annotation download link | -|---|---|---| -| PubTabNet |https://github.com/ibm-aur-nlp/PubTabNet| jsonl format, which can be loaded directly with [pubtab_dataset.py](../../../ppocr/data/pubtab_dataset.py) | -| TAL Table Recognition Competition Dataset |https://ai.100tal.com/dataset| jsonl format, which can be loaded directly with [pubtab_dataset.py](../../../ppocr/data/pubtab_dataset.py) | -| WTW Chinese scene table dataset |https://github.com/wangwen-whu/WTW-Dataset| Conversion is required to load with [pubtab_dataset.py](../../../ppocr/data/pubtab_dataset.py)| - -## 1. PubTabNet -- **Data Introduction**:The training set of the PubTabNet dataset contains 500,000 images and the validation set contains 9000 images. Part of the image visualization is shown below. - -

- - -
- -- **illustrate**:When using this dataset, the [CDLA-Permissive](https://cdla.io/permissive-1-0/) protocol is required. - -## 2. TAL Table Recognition Competition Dataset -- **Data Introduction**:The training set of the TAL table recognition competition dataset contains 16,000 images. The validation set does not give trainable annotations. - -
- - -
- -## 3. WTW Chinese scene table dataset -- **Data Introduction**:The WTW Chinese scene table dataset consists of two parts: table detection and table data. The dataset contains images of two scenes, scanned and photographed. -https://github.com/wangwen-whu/WTW-Dataset/blob/main/demo/20210816_210413.gif - -
- -
diff --git a/doc/doc_en/dataset/vertical_and_multilingual_datasets_en.md b/doc/doc_en/dataset/vertical_and_multilingual_datasets_en.md deleted file mode 100644 index ea366ef202..0000000000 --- a/doc/doc_en/dataset/vertical_and_multilingual_datasets_en.md +++ /dev/null @@ -1,79 +0,0 @@ -# Vertical multi-language OCR dataset -Here we have sorted out the commonly used vertical multi-language OCR dataset datasets, which are being updated continuously. We welcome you to contribute datasets ~ -- [Chinese urban license plate dataset](#Chinese urban license plate dataset) -- [Bank credit card dataset](#Bank credit card dataset) -- [Captcha dataset-Captcha](#Captcha dataset-Captcha) -- [multi-language dataset](#multi-language dataset) - - - -## Chinese urban license plate dataset - -- **Data source**:[https://github.com/detectRecog/CCPD](https://github.com/detectRecog/CCPD) - -- **Data introduction**: It contains more than 250000 vehicle license plate images and vehicle license plate detection and recognition information labeling. It contains the following license plate image information in different scenes. - * CCPD-Base: General license plate picture - * CCPD-DB: The brightness of license plate area is bright, dark or uneven - * CCPD-FN: The license plate is farther or closer to the camera location - * CCPD-Rotate: License plate includes rotation (horizontal 20\~50 degrees, vertical-10\~10 degrees) - * CCPD-Tilt: License plate includes rotation (horizontal 15\~45 degrees, vertical 15\~45 degrees) - * CCPD-Blur: The license plate contains blurring due to camera lens jitter - * CCPD-Weather: The license plate is photographed on rainy, snowy or foggy days - * CCPD-Challenge: So far, some of the most challenging images in license plate detection and recognition tasks - * CCPD-NP: Pictures of new cars without license plates. - - ![](../../datasets/ccpd_demo.png) - - -- **Download address** - * Baidu cloud download address (extracted code is hm0U): [https://pan.baidu.com/s/1i5AOjAbtkwb17Zy-NQGqkw](https://pan.baidu.com/s/1i5AOjAbtkwb17Zy-NQGqkw) - * Google drive download address:[https://drive.google.com/file/d/1rdEsCUcIUaYOVRkx5IMTRNA7PcGMmSgc/view](https://drive.google.com/file/d/1rdEsCUcIUaYOVRkx5IMTRNA7PcGMmSgc/view) - - - -## Bank credit card dataset - -- **Data source**: [https://www.kesci.com/home/dataset/5954cf1372ead054a5e25870](https://www.kesci.com/home/dataset/5954cf1372ead054a5e25870) - -- **Data introduction**: There are three types of training data - * 1.Sample card data of China Merchants Bank: including card image data and annotation data, a total of 618 pictures - * 2.Single character data: including pictures and annotation data, 37 pictures in total. - * 3.There are only other bank cards, no more detailed information, a total of 50 pictures. - - * The demo image is shown as follows. The annotation information is stored in excel, and the demo image below is marked as - * Top 8 card number: 62257583 - * Card type: card of our bank - * End of validity: 07/41 - * Chinese phonetic alphabet of card users: MICHAEL - - ![](../../datasets/cmb_demo.jpg) - -- **Download address**: [https://cdn.kesci.com/cmb2017-2.zip](https://cdn.kesci.com/cmb2017-2.zip) - - - - -## Captcha dataset-Captcha - -- **Data source**: [https://github.com/lepture/captcha](https://github.com/lepture/captcha) - -- **Data introduction**: This is a toolkit for data synthesis. You can output captcha images according to the input text. Use the toolkit to generate several demo images as follows. - - ![](../../datasets/captcha_demo.png) - -- **Download address**: The dataset is generated and has no download address. - - - - -## multi-language dataset(Multi-lingual scene text detection and recognition) - -- **Data source**: [https://rrc.cvc.uab.es/?ch=15&com=downloads](https://rrc.cvc.uab.es/?ch=15&com=downloads) - -- **Data introduction**: Multi language detection dataset MLT contains both language recognition and detection tasks. - * In the detection task, the training set contains 10000 images in 10 languages, and each language contains 1000 training images. The test set contains 10000 images. - * In the recognition task, the training set contains 111998 samples. - - -- **Download address**: The training set is large and can be downloaded in two parts. It can only be downloaded after registering on the website: -[https://rrc.cvc.uab.es/?ch=15&com=downloads](https://rrc.cvc.uab.es/?ch=15&com=downloads) diff --git a/doc/doc_en/detection_en.md b/doc/doc_en/detection_en.md deleted file mode 100644 index ab2e868c54..0000000000 --- a/doc/doc_en/detection_en.md +++ /dev/null @@ -1,250 +0,0 @@ -# Text Detection - -This section uses the icdar2015 dataset as an example to introduce the training, evaluation, and testing of the detection model in PaddleOCR. - -- [1. Data and Weights Preparation](#1-data-and-weights-preparation) - - [1.1 Data Preparation](#11-data-preparation) - - [1.2 Download Pre-trained Model](#12-download-pre-trained-model) -- [2. Training](#2-training) - * [2.1 Start Training](#21-start-training) - * [2.2 Load Trained Model and Continue Training](#22-load-trained-model-and-continue-training) - * [2.3 Training with New Backbone](#23-training-with-new-backbone) - * [2.4 Mixed Precision Training](#24-amp-training) - * [2.5 Distributed Training](#25-distributed-training) - * [2.6 Training with knowledge distillation](#26) - * [2.7 Training on other platform(Windows/macOS/Linux DCU)](#27) - * [2.8 Fine-tuning](#28) -- [3. Evaluation and Test](#3-evaluation-and-test) - - [3.1 Evaluation](#31-evaluation) - - [3.2 Test](#32-test) -- [4. Inference](#4-inference) -- [5. FAQ](#5-faq) - -## 1. Data and Weights Preparation - -### 1.1 Data Preparation - -To prepare datasets, refer to [ocr_datasets](./dataset/ocr_datasets_en.md) . - -### 1.2 Download Pre-trained Model - -First download the pre-trained model. The detection model of PaddleOCR currently supports 3 backbones, namely MobileNetV3, ResNet18_vd and ResNet50_vd. You can use the model in [PaddleClas](https://github.com/PaddlePaddle/PaddleClas/tree/release/2.0/ppcls/modeling/architectures) to replace backbone according to your needs. -And the responding download link of backbone pre-trained weights can be found in (https://github.com/PaddlePaddle/PaddleClas/blob/release%2F2.0/README_cn.md#resnet%E5%8F%8A%E5%85%B6vd%E7%B3%BB%E5%88%97). - -```shell -cd PaddleOCR/ -# Download the pre-trained model of MobileNetV3 -wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/MobileNetV3_large_x0_5_pretrained.pdparams -# or, download the pre-trained model of ResNet18_vd -wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet18_vd_pretrained.pdparams -# or, download the pre-trained model of ResNet50_vd -wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet50_vd_ssld_pretrained.pdparams - -``` - -## 2. Training - -### 2.1 Start Training - -*If CPU version installed, please set the parameter `use_gpu` to `false` in the configuration.* -```shell -python3 tools/train.py -c configs/det/det_mv3_db.yml \ - -o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained -``` - -In the above instruction, use `-c` to select the training to use the `configs/det/det_mv3_db.yml` configuration file. -For a detailed explanation of the configuration file, please refer to [config](./config_en.md). - -You can also use `-o` to change the training parameters without modifying the yml file. For example, adjust the training learning rate to 0.0001 -```shell -# single GPU training -python3 tools/train.py -c configs/det/det_mv3_db.yml -o \ - Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained \ - Optimizer.base_lr=0.0001 - -# multi-GPU training -# Set the GPU ID used by the '--gpus' parameter. -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/det_mv3_db.yml -o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained - -# multi-Node, multi-GPU training -# Set the IPs of your nodes used by the '--ips' parameter. Set the GPU ID used by the '--gpus' parameter. -python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/det/det_mv3_db.yml \ - -o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained -``` -**Note:** For multi-Node multi-GPU training, you need to replace the `ips` value in the preceding command with the address of your machine, and the machines must be able to ping each other. In addition, it requires activating commands separately on multiple machines when we start the training. The command for viewing the IP address of the machine is `ifconfig`. - -If you want to further speed up the training, you can use [automatic mixed precision training](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/01_paddle2.0_introduction/basic_concept/amp_en.html). for single card training, the command is as follows: -``` -python3 tools/train.py -c configs/det/det_mv3_db.yml \ - -o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained \ - Global.use_amp=True Global.scale_loss=1024.0 Global.use_dynamic_loss_scaling=True -``` - -### 2.2 Load Trained Model and Continue Training -If you expect to load trained model and continue the training again, you can specify the parameter `Global.checkpoints` as the model path to be loaded. - -For example: -```shell -python3 tools/train.py -c configs/det/det_mv3_db.yml -o Global.checkpoints=./your/trained/model -``` - -**Note**: The priority of `Global.checkpoints` is higher than that of `Global.pretrained_model`, that is, when two parameters are specified at the same time, the model specified by `Global.checkpoints` will be loaded first. If the model path specified by `Global.checkpoints` is wrong, the one specified by `Global.pretrained_model` will be loaded. - - -### 2.3 Training with New Backbone - -The network part completes the construction of the network, and PaddleOCR divides the network into four parts, which are under [ppocr/modeling](../../ppocr/modeling). The data entering the network will pass through these four parts in sequence(transforms->backbones-> -necks->heads). - -```bash -├── architectures # Code for building network -├── transforms # Image Transformation Module -├── backbones # Feature extraction module -├── necks # Feature enhancement module -└── heads # Output module -``` - -If the Backbone to be replaced has a corresponding implementation in PaddleOCR, you can directly modify the parameters in the `Backbone` part of the configuration yml file. - -However, if you want to use a new Backbone, an example of replacing the backbones is as follows: - -1. Create a new file under the [ppocr/modeling/backbones](../../ppocr/modeling/backbones) folder, such as my_backbone.py. -2. Add code in the my_backbone.py file, the sample code is as follows: - -```python -import paddle -import paddle.nn as nn -import paddle.nn.functional as F - - -class MyBackbone(nn.Layer): - def __init__(self, *args, **kwargs): - super(MyBackbone, self).__init__() - # your init code - self.conv = nn.xxxx - - def forward(self, inputs): - # your network forward - y = self.conv(inputs) - return y -``` - -3. Import the added module in the [ppocr/modeling/backbones/\__init\__.py](../../ppocr/modeling/backbones/__init__.py) file. - -After adding the four-part modules of the network, you only need to configure them in the configuration file to use, such as: - -```yaml - Backbone: - name: MyBackbone - args1: args1 -``` - -**NOTE**: More details about replace Backbone and other mudule can be found in [doc](add_new_algorithm_en.md). - -### 2.4 Mixed Precision Training - -If you want to speed up your training further, you can use [Auto Mixed Precision Training](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/01_paddle2.0_introduction/basic_concept/amp_cn.html), taking a single machine and a single gpu as an example, the commands are as follows: - -```shell -python3 tools/train.py -c configs/det/det_mv3_db.yml \ - -o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained \ - Global.use_amp=True Global.scale_loss=1024.0 Global.use_dynamic_loss_scaling=True - ``` - -### 2.5 Distributed Training - -During multi-machine multi-gpu training, use the `--ips` parameter to set the used machine IP address, and the `--gpus` parameter to set the used GPU ID: - -```bash -python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/det/det_mv3_db.yml \ - -o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained -``` - -**Note:** (1) When using multi-machine and multi-gpu training, you need to replace the ips value in the above command with the address of your machine, and the machines need to be able to ping each other. (2) Training needs to be launched separately on multiple machines. The command to view the ip address of the machine is `ifconfig`. (3) For more details about the distributed training speedup ratio, please refer to [Distributed Training Tutorial](./distributed_training_en.md). - -### 2.6 Training with knowledge distillation - -Knowledge distillation is supported in PaddleOCR for text detection training process. For more details, please refer to [doc](./knowledge_distillation_en.md). - -### 2.7 Training on other platform(Windows/macOS/Linux DCU) - -- Windows GPU/CPU -The Windows platform is slightly different from the Linux platform: -Windows platform only supports `single gpu` training and inference, specify GPU for training `set CUDA_VISIBLE_DEVICES=0` -On the Windows platform, DataLoader only supports single-process mode, so you need to set `num_workers` to 0; - -- macOS -GPU mode is not supported, you need to set `use_gpu` to False in the configuration file, and the rest of the training evaluation prediction commands are exactly the same as Linux GPU. - -- Linux DCU -Running on a DCU device requires setting the environment variable `export HIP_VISIBLE_DEVICES=0,1,2,3`, and the rest of the training and evaluation prediction commands are exactly the same as the Linux GPU. - -### 2.8 Fine-tuning - -In actual use, it is recommended to load the official pre-trained model and fine-tune it in your own data set. For the fine-tuning method of the detection model, please refer to: [Model Fine-tuning Tutorial](./finetune_en.md). - -## 3. Evaluation and Test - -### 3.1 Evaluation - -PaddleOCR calculates three indicators for evaluating performance of OCR detection task: Precision, Recall, and Hmean(F-Score). - -Run the following code to calculate the evaluation indicators. The result will be saved in the test result file specified by `save_res_path` in the configuration file `det_db_mv3.yml` - -When evaluating, set post-processing parameters `box_thresh=0.6`, `unclip_ratio=1.5`. If you use different datasets, different models for training, these two parameters should be adjusted for better result. - -The model parameters during training are saved in the `Global.save_model_dir` directory by default. When evaluating indicators, you need to set `Global.checkpoints` to point to the saved parameter file. -```shell -python3 tools/eval.py -c configs/det/det_mv3_db.yml -o Global.checkpoints="{path/to/weights}/best_accuracy" PostProcess.box_thresh=0.6 PostProcess.unclip_ratio=1.5 -``` - -* Note: `box_thresh` and `unclip_ratio` are parameters required for DB post-processing, and not need to be set when evaluating the EAST and SAST model. - -### 3.2 Test - -Test the detection result on a single image: -```shell -python3 tools/infer_det.py -c configs/det/det_mv3_db.yml -o Global.infer_img="./doc/imgs_en/img_10.jpg" Global.pretrained_model="./output/det_db/best_accuracy" -``` - -When testing the DB model, adjust the post-processing threshold: -```shell -python3 tools/infer_det.py -c configs/det/det_mv3_db.yml -o Global.infer_img="./doc/imgs_en/img_10.jpg" Global.pretrained_model="./output/det_db/best_accuracy" PostProcess.box_thresh=0.6 PostProcess.unclip_ratio=2.0 -``` - - -Test the detection result on all images in the folder: -```shell -python3 tools/infer_det.py -c configs/det/det_mv3_db.yml -o Global.infer_img="./doc/imgs_en/" Global.pretrained_model="./output/det_db/best_accuracy" -``` - -## 4. Inference - -The inference model (the model saved by `paddle.jit.save`) is generally a solidified model saved after the model training is completed, and is mostly used to give prediction in deployment. - -The model saved during the training process is the checkpoints model, which saves the parameters of the model and is mostly used to resume training. - -Compared with the checkpoints model, the inference model will additionally save the structural information of the model. Therefore, it is easier to deploy because the model structure and model parameters are already solidified in the inference model file, and is suitable for integration with actual systems. - -Firstly, we can convert DB trained model to inference model: -```shell -python3 tools/export_model.py -c configs/det/det_mv3_db.yml -o Global.pretrained_model="./output/det_db/best_accuracy" Global.save_inference_dir="./output/det_db_inference/" -``` - -The detection inference model prediction: -```shell -python3 tools/infer/predict_det.py --det_algorithm="DB" --det_model_dir="./output/det_db_inference/" --image_dir="./doc/imgs/" --use_gpu=True -``` - -If it is other detection algorithms, such as the EAST, the det_algorithm parameter needs to be modified to EAST, and the default is the DB algorithm: -```shell -python3 tools/infer/predict_det.py --det_algorithm="EAST" --det_model_dir="./output/det_db_inference/" --image_dir="./doc/imgs/" --use_gpu=True -``` - -## 5. FAQ - -Q1: The prediction results of trained model and inference model are inconsistent? - -**A**: Most of the problems are caused by the inconsistency of the pre-processing and post-processing parameters during the prediction of the trained model and the pre-processing and post-processing parameters during the prediction of the inference model. Taking the model trained by the det_mv3_db.yml configuration file as an example, the solution to the problem of inconsistent prediction results between the training model and the inference model is as follows: -- Check whether the [trained model preprocessing](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/configs/det/det_mv3_db.yml#L116) is consistent with the prediction [preprocessing function of the inference model](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/tools/infer/predict_det.py#L42). When the algorithm is evaluated, the input image size will affect the accuracy. In order to be consistent with the paper, the image is resized to [736, 1280] in the training icdar15 configuration file, but there is only a set of default parameters when the inference model predicts, which will be considered To predict the speed problem, the longest side of the image is limited to 960 for resize by default. The preprocessing function of the training model preprocessing and the inference model is located in [ppocr/data/imaug/operators.py](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/ppocr/data/imaug/operators.py#L147) -- Check whether the [post-processing of the trained model](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/configs/det/det_mv3_db.yml#L51) is consistent with the [post-processing parameters of the inference](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/tools/infer/utility.py#L50). diff --git a/doc/doc_en/distributed_training_en.md b/doc/doc_en/distributed_training_en.md deleted file mode 100644 index 947fb139bc..0000000000 --- a/doc/doc_en/distributed_training_en.md +++ /dev/null @@ -1,68 +0,0 @@ -# Distributed training - -## Introduction - -The high performance of distributed training is one of the core advantages of PaddlePaddle. In the classification task, distributed training can achieve almost linear speedup ratio. Generally, OCR training task need massive training data. Such as recognition, PP-OCR v2.0 model is trained based on 1800W dataset, which is very time-consuming if using single machine. Therefore, the distributed training is used in PaddleOCR to speedup the training task. For more information about distributed training, please refer to [distributed training quick start tutorial](https://fleet-x.readthedocs.io/en/latest/paddle_fleet_rst/parameter_server/ps_quick_start.html). - -## Quick Start - -### Training with single machine - -Take recognition as an example. After the data is prepared locally, start the training task with the interface of `paddle.distributed.launch`. The start command as follows: - -```shell -python3 -m paddle.distributed.launch \ - --log_dir=./log/ \ - --gpus "0,1,2,3,4,5,6,7" \ - tools/train.py \ - -c configs/rec/rec_mv3_none_bilstm_ctc.yml -``` - -### Training with multi machine - -Compared with single machine, training with multi machine only needs to add the parameter `--ips` to start command, which represents the IP list of machines used for distributed training, and the IP of different machines are separated by commas. The start command as follows: - -```shell -ip_list="192.168.0.1,192.168.0.2" -python3 -m paddle.distributed.launch \ - --log_dir=./log/ \ - --ips="${ip_list}" \ - --gpus="0,1,2,3,4,5,6,7" \ - tools/train.py \ - -c configs/rec/rec_mv3_none_bilstm_ctc.yml -``` - -**Notice:** -* The IP addresses of different machines need to be separated by commas, which can be queried through `ifconfig` or `ipconfig`. -* Different machines need to be set to be secret free and can `ping` success with others directly, otherwise communication cannot establish between them. -* The code, data and start command between different machines must be completely consistent and then all machines need to run start command. The first machine in the `ip_list` is set to `trainer0`, and so on. - - -## Performance comparison - -* We conducted model training on 2x8 P40 GPUs. Accuracy, training time, and multi machine acceleration ratio of different models are shown below. - -| Model | Configuration | Configuration | 8 GPU training time / Accuracy | 3x8 GPU training time / Accuracy | Acceleration ratio | - - -| Model | Configuration | Configuration | 8 GPU training time / Accuracy | 3x8 GPU training time / Accuracy | Acceleration ratio | -|:------:|:-----:|:--------:|:--------:|:--------:|:-----:| -| CRNN | [rec_chinese_lite_train_v2.0.yml](../../configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml) | 260k Chinese dataset | 2.50d/66.70% | 1.67d/67.00% | **1.5** | - - -* We conducted model training on 3x8 V100 GPUs. Accuracy, training time, and multi machine acceleration ratio of different models are shown below. - -| Model | Configuration | Configuration | 8 GPU training time / Accuracy | 3x8 GPU training time / Accuracy | Acceleration ratio | -|:------:|:-----:|:--------:|:--------:|:--------:|:-----:| -| SLANet | [SLANet.yml](../../configs/table/SLANet.yml) | PubTabNet | 49.80h/76.20% | 19.75h/74.77% | **2.52** | - - - > Note: when training with 3x8 GPUs, the single card batch size is unchanged compared with the 1x8 GPUs' training process, and the learning rate is multiplied by 2 (if it is multiplied by 3 by default, the accuracy is only 73.42%). - - -* We conducted model training on 4x8 V100 GPUs. Accuracy, training time, and multi machine acceleration ratio of different models are shown below. - - -| Model | Configuration | Configuration | 8 GPU training time / Accuracy | 4x8 GPU training time / Accuracy | Acceleration ratio | -|:------:|:-----:|:--------:|:--------:|:--------:|:-----:| -| SVTR | [ch_PP-OCRv3_rec_distillation.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml) | PP-OCRv3_rec data | 10d/- | 2.84d/74.00% | **3.5** | diff --git a/doc/doc_en/enhanced_ctc_loss_en.md b/doc/doc_en/enhanced_ctc_loss_en.md deleted file mode 100644 index 72cada714e..0000000000 --- a/doc/doc_en/enhanced_ctc_loss_en.md +++ /dev/null @@ -1,110 +0,0 @@ -# Enhanced CTC Loss - -In OCR recognition, CRNN is a text recognition algorithm widely applied in the industry. In the training phase, it uses CTCLoss to calculate the network loss. In the inference phase, it uses CTCDecode to obtain the decoding result. Although the CRNN algorithm has been proven to achieve reliable recognition results in actual business, users have endless requirements for recognition accuracy. So how to improve the accuracy of text recognition? Taking CTCLoss as the starting point, this paper explores the improved fusion scheme of CTCLoss from three different perspectives: Hard Example Mining, Multi-task Learning, and Metric Learning. Based on the exploration, we propose EnhancedCTCLoss, which includes the following 3 components: Focal-CTC Loss, A-CTC Loss, C-CTC Loss. - -## 1. Focal-CTC Loss - -Focal Loss was proposed by the paper, "[Focal Loss for Dense Object Detection](https://arxiv.org/abs/1708.02002)". When the loss was first proposed, it was mainly to solve the problem of a serious imbalance in the ratio of positive and negative samples in one-stage target detection. This loss function reduces the weight of a large number of simple negative samples in training and also can be understood as a kind of difficult sample mining. -The form of the loss function is as follows: - -
- -
- -Among them, y' is the output of the activation function, and the value is between 0-1. It adds a modulation factor (1-y’)^γ and a balance factor α on the basis of the original cross-entropy loss. When α = 1, y = 1, the comparison between the loss function and the cross-entropy loss is shown in the following figure: - -
- -
- - - -As can be seen from the above figure, when γ > 0, the adjustment coefficient (1-y’)^γ gives smaller weight to the easy-to-classify sample loss, making the network pay more attention to the difficult and misclassified samples. The adjustment factor γ is used to adjust the rate at which the weight of simple samples decreases. When γ = 0, it is the cross-entropy loss function. When γ increases, the influence of the adjustment factor will also increase. Experiments revealed that 2 is the optimal value of γ. The balance factor α is used to balance the uneven proportions of the positive and negative samples. In the text, α is taken as 0.25. - -For the classic CTC algorithm, suppose a certain feature sequence (f1, f2, ......ft), after CTC decoding, the probability that the result is equal to label is y', then the probability that the CTC decoding result is not equal to label is (1-y'); it is not difficult to find that the CTCLoss value and y' have the following relationship: - -
- -
- - - -Combining the idea of Focal Loss, assigning larger weights to difficult samples and smaller weights to simple samples can make the network focus more on the mining of difficult samples and further improve the accuracy of recognition. Therefore, we propose Focal-CTC Loss. Its definition is as follows: - -
- -
- - - -In the experiment, the value of γ is 2, α = 1, see this for specific implementation: [rec_ctc_loss.py](../../ppocr/losses/rec_ctc_loss.py) - - - -## 2. A-CTC Loss - -A-CTC Loss is short for CTC Loss + ACE Loss. Among them, ACE Loss was proposed by the paper, “[Aggregation Cross-Entropy for Sequence Recognition](https://arxiv.org/abs/1904.08364)”. Compared with CTCLoss, ACE Loss has the following two advantages: -+ ACE Loss can solve the recognition problem of 2-D text, while CTCLoss can only process 1-D text -+ ACE Loss is better than CTC loss in time complexity and space complexity - -The advantages and disadvantages of the OCR recognition algorithm summarized by the predecessors are shown in the following figure: - -
- -
- - -Although ACELoss does handle 2D predictions, as shown in the figure above, and has advantages in memory usage and inference speed, in practice, we found that using ACELoss alone, the recognition effect is not as good as CTCLoss. Consequently, we tried to combine CTCLoss and ACELoss, and CTCLoss is the mainstay while ACELoss acts as an auxiliary supervision loss. This attempt has achieved better results. On our internal experimental data set, compared to using CTCLoss alone, the recognition accuracy can be improved by about 1%. -A_CTC Loss is defined as follows: - -
- -
- - - -In the experiment, λ = 0.1. See the ACE loss implementation code: [ace_loss.py](../../ppocr/losses/ace_loss.py) - - - -## 3. C-CTC Loss - -C-CTC Loss is short for CTC Loss + Center Loss. Among them, Center Loss was proposed by the paper, “[A Discriminative Feature Learning Approach for Deep Face Recognition](https://link.springer.com/chapter/10.1007/978-3-319-46478-7_31)“. It was first used in face recognition tasks to increase the distance between classes and reduce the distance within classes. It is an earlier and also widely used algorithm. - -In the task of Chinese OCR recognition, through the analysis of bad cases, we found that a major difficulty in Chinese recognition is that there are many similar characters, which are easy to misunderstand. From this, we thought about whether we can learn from the idea of n to increase the class spacing of similar characters, to improve recognition accuracy. However, Metric Learning is mainly used in the field of image recognition, and the label of the training data is a fixed value; for OCR recognition, it is a sequence recognition task essentially, and there is no explicit alignment between features and labels. Therefore, how to combine the two is still a direction worth exploring. - -By trying Arcmargin, Cosmargin and other methods, we finally found that Centerloss can help further improve the accuracy of recognition. C_CTC Loss is defined as follows: - -
- -
- -In the experiment, we set λ=0.25. See the center_loss implementation code: [center_loss.py](../../ppocr/losses/center_loss.py) - -It is worth mentioning that in C-CTC Loss, choosing to initialize the Center randomly does not bring significant improvement. Our Center initialization method is as follows: -+ Based on the original CTCLoss, a network N is obtained by training -+ Select the training set, identify the completely correct part, and form the set G -+ Send each sample in G to the network, perform forward calculation, and extract the correspondence between the input of the last FC layer (ie feature) and the result of argmax calculation (ie index) -+ Aggregate features with the same index, calculate the average, and get the initial center of each character. - -Taking the configuration file `configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml` as an example, the center extraction command is as follows: - -``` -python tools/export_center.py -c configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml -o Global.pretrained_model="./output/rec_mobile_pp-OCRv2/best_accuracy" -``` - -After running, `train_center.pkl` will be generated in the main directory of PaddleOCR. - - - -## 4. Experiment - -For the above three solutions, we conducted training and evaluation based on Baidu's internal data set. The experimental conditions are shown in the following table: - -| algorithm | Focal_CTC | A_CTC | C-CTC | -| :-------- | :-------- | ----: | :---: | -| gain | +0.3% | +0.7% | +1.7% | - -Based on the above experimental conclusions, we adopted the C-CTC strategy in PP-OCRv2. It is worth mentioning that, because PP-OCRv2 deals with the recognition task of 6625 Chinese characters, the character set is relatively large and there are many similar characters, so the C-CTC solution brings a significant improvement on this task. But if you switch to other OCR recognition tasks, the conclusion may be different. You can try Focal-CTC, A-CTC, C-CTC, and the combined solution EnhancedCTC. We believe it will bring different degrees of improvement. - -The unified combined plan is shown in the following file: [rec_enhanced_ctc_loss.py](../../ppocr/losses/rec_enhanced_ctc_loss.py) diff --git a/doc/doc_en/environment_en.md b/doc/doc_en/environment_en.md deleted file mode 100644 index 453287385f..0000000000 --- a/doc/doc_en/environment_en.md +++ /dev/null @@ -1,333 +0,0 @@ -# Environment Preparation - -Windows and Mac users are recommended to use Anaconda to build a Python environment, and Linux users are recommended to use docker to build a Python environment. - -Recommended working environment: -- PaddlePaddle >= 2.1.2 -- Python 3.7 -- CUDA 10.1 / CUDA 10.2 -- cuDNN 7.6 - -> If you already have a Python environment installed, you can skip to [PaddleOCR Quick Start](./quickstart_en.md). - -* [1. Python Environment Setup](#1) - + [1.1 Windows](#1.1) - + [1.2 Mac](#1.2) - + [1.3 Linux](#1.3) - - - - -## 1. Python Environment Setup - - - -### 1.1 Windows - -#### 1.1.1 Install Anaconda - -- Note: To use PaddlePaddle you need to install python environment first, here we choose python integrated environment Anaconda toolkit - - - Anaconda is a common python package manager - - After installing Anaconda, you can install the python environment, as well as numpy and other required toolkit environment. - -- Anaconda download. - - - Address: https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/?C=M&O=D - - - Most Win10 computers are 64-bit operating systems, choose x86_64 version; if the computer is a 32-bit operating system, choose x86.exe - - anaconda download - - - After the download is complete, double-click the installer to enter the graphical interface - - - The default installation location is C drive, it is recommended to change the installation location to D drive. - - install config - - - Check Conda to add environment variables and ignore the warning that - - add conda to path - - -#### 1.1.2 Opening the terminal and creating the Conda environment - -- Open Anaconda Prompt terminal: bottom left Windows Start Menu -> Anaconda3 -> Anaconda Prompt start console - - anaconda download - - -- Create a new Conda environment - - ```shell - # Enter the following command at the command line to create an environment named paddle_env - # Here to speed up the download, use the Tsinghua source - conda create --name paddle_env python=3.8 --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/ # This is a one line command - ``` - - This command will create an executable environment named paddle_env with python version 3.8, which will take a while depending on the network status - - The command line will then output a prompt, type y and enter to continue the installation - - conda create - -- To activate the Conda environment you just created, enter the following command at the command line. - - ```shell - # Activate the paddle_env environment - conda activate paddle_env - # View the current location of python - where python - ``` - - create environment - -The above anaconda environment and python environment are installed - - - - - -### 1.2 Mac - -#### 1.2.1 Installing Anaconda - -- Note: To use PaddlePaddle you need to install the python environment first, here we choose the python integrated environment Anaconda toolkit - - - Anaconda is a common python package manager - - After installing Anaconda, you can install the python environment, as well as numpy and other required toolkit environment - -- Anaconda download:. - - - Address: https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/?C=M&O=D - - anaconda download - - - Select `Anaconda3-2021.05-MacOSX-x86_64.pkg` at the bottom to download - -- After downloading, double click on the .pkg file to enter the graphical interface - - - Just follow the default settings, it will take a while to install - -- It is recommended to install a code editor such as VSCode or PyCharm - -#### 1.2.2 Open a terminal and create a Conda environment - -- Open the terminal - - - Press command and spacebar at the same time, type "terminal" in the focus search, double click to enter terminal - -- **Add Conda to the environment variables** - - - Environment variables are added so that the system can recognize the Conda command - - - Open `~/.bash_profile` in the terminal by typing the following command. - - ```shell - vim ~/.bash_profile - ``` - - - Add Conda as an environment variable in `~/.bash_profile`. - - ```shell - # Press i first to enter edit mode - # In the first line type. - export PATH="~/opt/anaconda3/bin:$PATH" - # If you customized the installation location during installation, change ~/opt/anaconda3/bin to the bin folder in the customized installation directory - ``` - - ```shell - # The modified ~/.bash_profile file should look like this (where xxx is the username) - export PATH="~/opt/anaconda3/bin:$PATH" - # >>> conda initialize >>> - # !!! Contents within this block are managed by 'conda init' !!! - __conda_setup="$('/Users/xxx/opt/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)" - if [ $? -eq 0 ]; then - eval "$__conda_setup" - else - if [ -f "/Users/xxx/opt/anaconda3/etc/profile.d/conda.sh" ]; then - . "/Users/xxx/opt/anaconda3/etc/profile.d/conda.sh" - else - export PATH="/Users/xxx/opt/anaconda3/bin:$PATH" - fi - fi - unset __conda_setup - # <<< conda initialize <<< - ``` - - - When you are done, press `esc` to exit edit mode, then type `:wq!` and enter to save and exit - - - Verify that the Conda command is recognized. - - - Enter `source ~/.bash_profile` in the terminal to update the environment variables - - Enter `conda info --envs` in the terminal again, if it shows that there is a base environment, then Conda has been added to the environment variables - -- Create a new Conda environment - - ```shell - # Enter the following command at the command line to create an environment called paddle_env - # Here to speed up the download, use Tsinghua source - conda create --name paddle_env python=3.8 --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/ - ``` - - - This command will create an executable environment named paddle_env with python version 3.8, which will take a while depending on the network status - - - The command line will then output a prompt, type y and enter to continue the installation - - - conda_create - -- To activate the Conda environment you just created, enter the following command at the command line. - - ```shell - # Activate the paddle_env environment - conda activate paddle_env - # View the current location of python - where python - ``` - - conda_actviate - -The above anaconda environment and python environment are installed - - - - - -### 1.3 Linux - -Linux users can choose to run either Anaconda or Docker. If you are familiar with Docker and need to train the PaddleOCR model, it is recommended to use the Docker environment, where the development process of PaddleOCR is run. If you are not familiar with Docker, you can also use Anaconda to run the project. - -#### 1.3.1 Anaconda environment configuration - -- Note: To use PaddlePaddle you need to install the python environment first, here we choose the python integrated environment Anaconda toolkit - - - Anaconda is a common python package manager - - After installing Anaconda, you can install the python environment, as well as numpy and other required toolkit environment - -- **Download Anaconda**. - - - Download at: https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/?C=M&O=D - - - - - - - Select the appropriate version for your operating system - - Type `uname -m` in the terminal to check the command set used by your system - - - Download method 1: Download locally, then transfer the installation package to the Linux server - - - Download method 2: Directly use Linux command line to download - - ```shell - # First install wget - sudo apt-get install wget # Ubuntu - sudo yum install wget # CentOS - ``` - ```bash - # Then use wget to download from Tsinghua source - # If you want to download Anaconda3-2021.05-Linux-x86_64.sh, the download command is as follows - wget https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/Anaconda3-2021.05-Linux-x86_64.sh - # If you want to download another version, you need to change the file name after the last 1 / to the version you want to download - ``` - -- To install Anaconda. - - - Type `sh Anaconda3-2021.05-Linux-x86_64.sh` at the command line - - If you downloaded a different version, replace the file name of the command with the name of the file you downloaded - - Just follow the installation instructions - - You can exit by typing q when viewing the license - -- **Add conda to the environment variables** - - - If you have already added conda to the environment variable path during the installation, you can skip this step - - - Open `~/.bashrc` in a terminal. - - ```shell - # Enter the following command in the terminal. - vim ~/.bashrc - ``` - - - Add conda as an environment variable in `~/.bashrc`. - - ```shell - # Press i first to enter edit mode # In the first line enter. - export PATH="~/anaconda3/bin:$PATH" - # If you customized the installation location during installation, change ~/anaconda3/bin to the bin folder in the customized installation directory - ``` - - ```shell - # The modified ~/.bash_profile file should look like this (where xxx is the username) - export PATH="~/opt/anaconda3/bin:$PATH" - # >>> conda initialize >>> - # !!! Contents within this block are managed by 'conda init' !!! - __conda_setup="$('/Users/xxx/opt/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)" - if [ $? -eq 0 ]; then - eval "$__conda_setup" - else - if [ -f "/Users/xxx/opt/anaconda3/etc/profile.d/conda.sh" ]; then - . "/Users/xxx/opt/anaconda3/etc/profile.d/conda.sh" - else - export PATH="/Users/xxx/opt/anaconda3/bin:$PATH" - fi - fi - unset __conda_setup - # <<< conda initialize <<< - ``` - - - When you are done, press `esc` to exit edit mode, then type `:wq!` and enter to save and exit - - - Verify that the Conda command is recognized. - - - Enter `source ~/.bash_profile` in the terminal to update the environment variables - - Enter `conda info --envs` in the terminal again, if it shows that there is a base environment, then Conda has been added to the environment variables - -- Create a new Conda environment - - ```shell - # Enter the following command at the command line to create an environment called paddle_env - # Here to speed up the download, use Tsinghua source - conda create --name paddle_env python=3.8 --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/ - ``` - - - This command will create an executable environment named paddle_env with python version 3.8, which will take a while depending on the network status - - - The command line will then output a prompt, type y and enter to continue the installation - - conda_create - -- To activate the Conda environment you just created, enter the following command at the command line. - - ```shell - # Activate the paddle_env environment - conda activate paddle_env - ``` - -The above anaconda environment and python environment are installed - - -#### 1.3.2 Docker environment preparation - -**The first time you use this docker image, it will be downloaded automatically. Please be patient.** - -```bash -# Switch to the working directory -cd /home/Projects -# You need to create a docker container for the first run, and do not need to run the current command when you run it again -# Create a docker container named ppocr and map the current directory to the /paddle directory of the container - -# If using CPU, use docker instead of nvidia-docker to create docker -sudo docker run --name ppocr -v $PWD:/paddle --network=host -it registry.baidubce.com/paddlepaddle/paddle:2.1.3-gpu-cuda10.2-cudnn7 /bin/bash - -# If using GPU, use nvidia-docker to create docker -# docker image registry.baidubce.com/paddlepaddle/paddle:2.1.3-gpu-cuda11.2-cudnn8 is recommended for CUDA11.2 + CUDNN8. -sudo nvidia-docker run --name ppocr -v $PWD:/paddle --shm-size=64G --network=host -it registry.baidubce.com/paddlepaddle/paddle:2.1.3-gpu-cuda10.2-cudnn7 /bin/bash - -``` -You can also visit [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to get the image that fits your machine. - -``` -# ctrl+P+Q to exit docker, to re-enter docker using the following command: -sudo docker container exec -it ppocr /bin/bash -``` diff --git a/doc/doc_en/finetune_en.md b/doc/doc_en/finetune_en.md deleted file mode 100644 index e76eb1e26a..0000000000 --- a/doc/doc_en/finetune_en.md +++ /dev/null @@ -1,229 +0,0 @@ -# Fine-tune - -## 1. background and meaning - -The PP-OCR series models provided by PaddleOCR have excellent performance in general scenarios and can solve detection and recognition problems in most cases. In vertical scenarios, if you want to obtain better model, you can further improve the accuracy of the PP-OCR series detection and recognition models through fine-tune. - -This article mainly introduces some precautions when fine-tuning the text detection and recognition model. Finally, you can obtain a text detection and recognition model with higher accuracy through model fine-tuning in your own scenarios. - -The core points of this article are as follows: - -1. The pre-trained model provided by PP-OCR has better generalization ability -2. Adding a small amount of real data (detection:>=500, recognition:>=5000) will greatly improve the detection and recognition effect of vertical scenes -3. When fine-tuning the model, adding real general scene data can further improve the model accuracy and generalization performance -4. In the text detection task, increasing the prediction shape of the image can further improve the detection effect of the smaller text area -5. When fine-tuning the model, it is necessary to properly adjust the hyperparameters (learning rate, batch size are the most important) to obtain a better fine-tuning effect. - -For more details, please refer to Chapter 2 and Chapter 3。 - -## 2. Text detection model fine-tuning - -### 2.1 Dataset - -* Dataset: It is recommended to prepare at least 500 text detection datasets for model fine-tuning. - -* Dataset annotation: single-line text annotation format, it is recommended that the labeled detection frame be consistent with the actual semantic content. For example, in the train ticket scene, the surname and first name may be far apart, but they belong to the same detection field semantically. Here, the entire name also needs to be marked as a detection frame. - -### 2.2 Model - -It is recommended to choose the PP-OCRv3 model (configuration file: [ch_PP-OCRv3_det_student.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_student.yml),pre-trained model: [ch_PP-OCRv3_det_distill_train.tar](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar), its accuracy and generalization performance is the best pre-training model currently available. - -For more PP-OCR series models, please refer to [PP-OCR Series Model Library](./models_list_en.md)。 - -Note: When using the above pre-trained model, you need to use the `student.pdparams` file in the folder as the pre-trained model, that is, only use the student model. - - -### 2.3 Training hyperparameter - -When fine-tuning the model, the most important hyperparameter is the pre-training model path `pretrained_model`, `learning_rate`与`batch_size`,some hyperparameters are as follows: - -```yaml -Global: - pretrained_model: ./ch_PP-OCRv3_det_distill_train/student.pdparams # pre-training model path -Optimizer: - lr: - name: Cosine - learning_rate: 0.001 # learning_rate - warmup_epoch: 2 - regularizer: - name: 'L2' - factor: 0 - -Train: - loader: - shuffle: True - drop_last: False - batch_size_per_card: 8 # single gpu batch size - num_workers: 4 -``` - -In the above configuration file, you need to specify the `pretrained_model` field as the `student.pdparams` file path. - -The configuration file provided by PaddleOCR is for 8-gpu training (equivalent to a total batch size of `8*8=64`) and no pre-trained model is loaded. Therefore, in your scenario, the learning rate is the same as the total The batch size needs to be adjusted linearly, for example - -* If your scenario is single-gpu training, single gpu batch_size=8, then the total batch_size=8, it is recommended to adjust the learning rate to about `1e-4`. -* If your scenario is for single-gpu training, due to memory limitations, you can only set batch_size=4 for a single gpu, and the total batch_size=4. It is recommended to adjust the learning rate to about `5e-5`. - -### 2.4 Prediction hyperparameter - -When exporting and inferring the trained model, you can further adjust the predicted image scale to improve the detection effect of small-area text. The following are some hyperparameters during DBNet inference, which can be adjusted appropriately to improve the effect. - -| hyperparameter | type | default | meaning | -| :--: | :--: | :--: | :--: | -| det_db_thresh | float | 0.3 | In the probability map output by DB, pixels with a score greater than the threshold will be considered as text pixels | -| det_db_box_thresh | float | 0.6 | When the average score of all pixels within the frame of the detection result is greater than the threshold, the result will be considered as a text area | -| det_db_unclip_ratio | float | 1.5 | The expansion coefficient of `Vatti clipping`, using this method to expand the text area | -| max_batch_size | int | 10 | batch size | -| use_dilation | bool | False | Whether to expand the segmentation results to obtain better detection results | -| det_db_score_mode | str | "fast" | DB's detection result score calculation method supports `fast` and `slow`. `fast` calculates the average score based on all pixels in the polygon’s circumscribed rectangle border, and `slow` calculates the average score based on all pixels in the original polygon. The calculation speed is relatively slower, but more accurate. | - - -For more information on inference methods, please refer to[Paddle Inference doc](././inference_ppocr_en.md)。 - - -## 3. Text recognition model fine-tuning - - -### 3.1 Dataset - -* Dataset:If the dictionary is not changed, it is recommended to prepare at least 5,000 text recognition datasets for model fine-tuning; if the dictionary is changed (not recommended), more quantities are required. - -* Data distribution: It is recommended that the distribution be as consistent as possible with the actual measurement scenario. If the actual scene contains a lot of short text, it is recommended to include more short text in the training data. If the actual scene has high requirements for the recognition effect of spaces, it is recommended to include more text content with spaces in the training data. - -* Data synthesis: In the case of some character recognition errors, it is recommended to obtain a batch of specific character dataset, add it to the original dataset and use a small learning rate for fine-tuning. The ratio of original dataset to new dataset can be 10:1 to 5:1 to avoid overfitting of the model caused by too much data in a single scene. At the same time, try to balance the word frequency of the corpus to ensure that the frequency of common words will not be too low. - - Specific characters can be generated using the TextRenderer tool, for synthesis examples, please refer to [data synthesis](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/applications/%E5%85%89%E5%8A%9F%E7%8E%87%E8%AE%A1%E6%95%B0%E7%A0%81%E7%AE%A1%E5%AD%97%E7%AC%A6%E8%AF%86%E5%88%AB/%E5%85%89%E5%8A%9F%E7%8E%87%E8%AE%A1%E6%95%B0%E7%A0%81%E7%AE%A1%E5%AD%97%E7%AC%A6%E8%AF%86%E5%88%AB.md#31-%E6%95%B0%E6%8D%AE%E5%87%86%E5%A4%87) - . The synthetic data corpus should come from real usage scenarios as much as possible, and keep the richness of fonts and backgrounds on the basis of being close to the real scene, which will help improve the model effect. - -* Common Chinese and English data: During training, common real data can be added to the training set (for example, in the fine-tuning scenario without changing the dictionary, it is recommended to add real data such as LSVT, RCTW, MTWI) to further improve the generalization performance of the model. - -### 3.2 Model - -It is recommended to choose the PP-OCRv3 model (configuration file: [ch_PP-OCRv3_rec_distillation.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml),pre-trained model: [ch_PP-OCRv3_rec_train.tar](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_train.tar),its accuracy and generalization performance is the best pre-training model currently available. - -For more PP-OCR series models, please refer to [PP-OCR Series Model Library](./models_list_en.md)。 - -The PP-OCRv3 model uses the GTC strategy. The SAR branch has a large number of parameters. When the training data is a simple scene, the model is easy to overfit, resulting in poor fine-tuning effect. It is recommended to remove the GTC strategy. The configuration file of the model structure is modified as follows: - -```yaml -Architecture: - model_type: rec - algorithm: SVTR - Transform: - Backbone: - name: MobileNetV1Enhance - scale: 0.5 - last_conv_stride: [1, 2] - last_pool_type: avg - Neck: - name: SequenceEncoder - encoder_type: svtr - dims: 64 - depth: 2 - hidden_dims: 120 - use_guide: False - Head: - name: CTCHead - fc_decay: 0.00001 -Loss: - name: CTCLoss - -Train: - dataset: - ...... - transforms: - # remove RecConAug - # - RecConAug: - # prob: 0.5 - # ext_data_num: 2 - # image_shape: [48, 320, 3] - # max_text_length: *max_text_length - - RecAug: - # modify Encode - - CTCLabelEncode: - - KeepKeys: - keep_keys: - - image - - label - - length -... - -Eval: - dataset: - ... - transforms: - ... - - CTCLabelEncode: - - KeepKeys: - keep_keys: - - image - - label - - length -... - - -``` - -### 3.3 Training hyperparameter - -Similar to text detection task fine-tuning, when fine-tuning the recognition model, the most important hyperparameters are the pre-trained model path `pretrained_model`, `learning_rate` and `batch_size`, some default configuration files are shown below. - -```yaml -Global: - pretrained_model: # pre-training model path -Optimizer: - lr: - name: Piecewise - decay_epochs : [700, 800] - values : [0.001, 0.0001] # learning_rate - warmup_epoch: 5 - regularizer: - name: 'L2' - factor: 0 - -Train: - dataset: - name: SimpleDataSet - data_dir: ./train_data/ - label_file_list: - - ./train_data/train_list.txt - ratio_list: [1.0] # Sampling ratio, the default value is [1.0] - loader: - shuffle: True - drop_last: False - batch_size_per_card: 128 # single gpu batch size - num_workers: 8 - -``` - - -In the above configuration file, you first need to specify the `pretrained_model` field as the `ch_PP-OCRv3_rec_train/best_accuracy.pdparams` file path decompressed in Chapter 3.2. - -The configuration file provided by PaddleOCR is for 8-gpu training (equivalent to a total batch size of `8*128=1024`) and no pre-trained model is loaded. Therefore, in your scenario, the learning rate is the same as the total The batch size needs to be adjusted linearly, for example: - -* If your scenario is single-gpu training, single gpu batch_size=128, then the total batch_size=128, in the case of loading the pre-trained model, it is recommended to adjust the learning rate to about `[1e-4, 2e-5]` (For the piecewise learning rate strategy, two values need to be set, the same below). -* If your scenario is for single-gpu training, due to memory limitations, you can only set batch_size=64 for a single gpu, and the total batch_size=64. When loading the pre-trained model, it is recommended to adjust the learning rate to `[5e-5 , 1e-5]`about. - - -If there is general real scene data added, it is recommended that in each epoch, the amount of vertical scene data and real scene data should be kept at about 1:1. - -For example: your own vertical scene recognition data volume is 1W, the data label file is `vertical.txt`, the collected general scene recognition data volume is 10W, and the data label file is `general.txt`. - -Then, the `label_file_list` and `ratio_list` parameters can be set as shown below. In each epoch, `vertical.txt` will be fully sampled (sampling ratio is 1.0), including 1W pieces of data; `general.txt` will be sampled according to a sampling ratio of 0.1, including `10W*0.1=1W` pieces of data, the final ratio of the two is `1:1`. - -```yaml -Train: - dataset: - name: SimpleDataSet - data_dir: ./train_data/ - label_file_list: - - vertical.txt - - general.txt - ratio_list: [1.0, 0.1] -``` - -### 3.4 training optimization - -The training process does not happen overnight. After completing a stage of training evaluation, it is recommended to collect and analyze the badcase of the current model in the real scene, adjust the proportion of training data in a targeted manner, or further add synthetic data. Through multiple iterations of training, the model effect is continuously optimized. - -If you modify the custom dictionary during training, since the parameters of the last layer of FC cannot be loaded, it is normal for acc=0 at the beginning of the iteration. Don't worry, loading the pre-trained model can still speed up the model convergence. diff --git a/doc/doc_en/inference_args_en.md b/doc/doc_en/inference_args_en.md deleted file mode 100644 index ee2faedf40..0000000000 --- a/doc/doc_en/inference_args_en.md +++ /dev/null @@ -1,130 +0,0 @@ -# PaddleOCR Model Inference Parameter Explanation - -When using PaddleOCR for model inference, you can customize the modification parameters to modify the model, data, preprocessing, postprocessing, etc. (parameter file: [utility.py](../../tools/infer/utility.py)),The detailed parameter explanation is as follows: - -* Global parameters - -| parameters | type | default | implication | -| :--: | :--: | :--: | :--: | -| image_dir | str | None, must be specified explicitly | Image or folder path | -| page_num | int | 0 | Valid when the input type is pdf file, specify to predict the previous page_num pages, all pages are predicted by default | -| vis_font_path | str | "./doc/fonts/simfang.ttf" | font path for visualization | -| drop_score | float | 0.5 | Results with a recognition score less than this value will be discarded and will not be returned as results | -| use_pdserving | bool | False | Whether to use Paddle Serving for prediction | -| warmup | bool | False | Whether to enable warmup, this method can be used when statistical prediction time | -| draw_img_save_dir | str | "./inference_results" | The saving folder of the system's tandem prediction OCR results | -| save_crop_res | bool | False | Whether to save the recognized text image for OCR | -| crop_res_save_dir | str | "./output" | Save the text image path recognized by OCR | -| use_mp | bool | False | Whether to enable multi-process prediction | -| total_process_num | int | 6 | The number of processes, which takes effect when `use_mp` is `True` | -| process_id | int | 0 | The id number of the current process, no need to modify it yourself | -| benchmark | bool | False | Whether to enable benchmark, and make statistics on prediction speed, memory usage, etc. | -| save_log_path | str | "./log_output/" | Folder where log results are saved when `benchmark` is enabled | -| show_log | bool | True | Whether to show the log information in the inference | -| use_onnx | bool | False | Whether to enable onnx prediction | - - -* Prediction engine related parameters - -| parameters | type | default | implication | -| :--: | :--: | :--: | :--: | -| use_gpu | bool | True | Whether to use GPU for prediction | -| ir_optim | bool | True | Whether to analyze and optimize the calculation graph. The prediction process can be accelerated when `ir_optim` is enabled | -| use_tensorrt | bool | False | Whether to enable tensorrt | -| min_subgraph_size | int | 15 | The minimum subgraph size in tensorrt. When the size of the subgraph is greater than this value, it will try to use the trt engine to calculate the subgraph. | -| precision | str | fp32 | The precision of prediction, supports `fp32`, `fp16`, `int8` | -| enable_mkldnn | bool | True | Whether to enable mkldnn | -| cpu_threads | int | 10 | When mkldnn is enabled, the number of threads predicted by the cpu | - -* Text detection model related parameters - -| parameters | type | default | implication | -| :--: | :--: | :--: | :--: | -| det_algorithm | str | "DB" | Text detection algorithm name, currently supports `DB`, `EAST`, `SAST`, `PSE`, `DB++`, `FCE` | -| det_model_dir | str | xx | Detection inference model paths | -| det_limit_side_len | int | 960 | image side length limit | -| det_limit_type | str | "max" | The side length limit type, currently supports `min`and `max`. `min` means to ensure that the shortest side of the image is not less than `det_limit_side_len`, `max` means to ensure that the longest side of the image is not greater than `det_limit_side_len` | - -The relevant parameters of the DB algorithm are as follows - -| parameters | type | default | implication | -| :--: | :--: | :--: | :--: | -| det_db_thresh | float | 0.3 | In the probability map output by DB, only pixels with a score greater than this threshold will be considered as text pixels | -| det_db_box_thresh | float | 0.6 | Within the detection box, when the average score of all pixels is greater than the threshold, the result will be considered as a text area | -| det_db_unclip_ratio | float | 1.5 | The expansion factor of the `Vatti clipping` algorithm, which is used to expand the text area | -| max_batch_size | int | 10 | max batch size | -| use_dilation | bool | False | Whether to inflate the segmentation results to obtain better detection results | -| det_db_score_mode | str | "fast" | DB detection result score calculation method, supports `fast` and `slow`, `fast` calculates the average score according to all pixels within the bounding rectangle of the polygon, `slow` calculates the average score according to all pixels within the original polygon, The calculation speed is relatively slower, but more accurate. | - -The relevant parameters of the EAST algorithm are as follows - -| parameters | type | default | implication | -| :--: | :--: | :--: | :--: | -| det_east_score_thresh | float | 0.8 | Threshold for score map in EAST postprocess | -| det_east_cover_thresh | float | 0.1 | Average score threshold for text boxes in EAST postprocess | -| det_east_nms_thresh | float | 0.2 | Threshold of nms in EAST postprocess | - -The relevant parameters of the SAST algorithm are as follows - -| parameters | type | default | implication | -| :--: | :--: | :--: | :--: | -| det_sast_score_thresh | float | 0.5 | Score thresholds in SAST postprocess | -| det_sast_nms_thresh | float | 0.5 | Thresholding of nms in SAST postprocess | -| det_box_type | str | 'quad' | Whether polygon detection, curved text scene (such as Total-Text) is set to 'poly' | - -The relevant parameters of the PSE algorithm are as follows - -| parameters | type | default | implication | -| :--: | :--: | :--: | :--: | -| det_pse_thresh | float | 0.0 | Threshold for binarizing the output image | -| det_pse_box_thresh | float | 0.85 | Threshold for filtering boxes, below this threshold is discarded | -| det_pse_min_area | float | 16 | The minimum area of the box, below this threshold is discarded | -| det_box_type | str | "quad" | The type of the returned box, quad: four point coordinates, poly: all point coordinates of the curved text | -| det_pse_scale | int | 1 | The ratio of the input image relative to the post-processed image, such as an image of `640*640`, the network output is `160*160`, and when the scale is 2, the shape of the post-processed image is `320*320`. Increasing this value can speed up the post-processing speed, but it will bring about a decrease in accuracy | - -* Text recognition model related parameters - -| parameters | type | default | implication | -| :--: | :--: | :--: | :--: | -| rec_algorithm | str | "CRNN" | Text recognition algorithm name, currently supports `CRNN`, `SRN`, `RARE`, `NETR`, `SAR`, `ViTSTR`, `ABINet`, `VisionLAN`, `SPIN`, `RobustScanner`, `SVTR`, `SVTR_LCNet` | -| rec_model_dir | str | None, it is required if using the recognition model | recognition inference model paths | -| rec_image_shape | str | "3,48,320" ] | Image size at the time of recognition | -| rec_batch_num | int | 6 | batch size | -| max_text_length | int | 25 | The maximum length of the recognition result, valid in `SRN` | -| rec_char_dict_path | str | "./ppocr/utils/ppocr_keys_v1.txt" | character dictionary file | -| use_space_char | bool | True | Whether to include spaces, if `True`, the `space` character will be added at the end of the character dictionary | - - -* End-to-end text detection and recognition model related parameters - -| parameters | type | default | implication | -| :--: | :--: | :--: | :--: | -| e2e_algorithm | str | "PGNet" | End-to-end algorithm name, currently supports `PGNet` | -| e2e_model_dir | str | None, it is required if using the end-to-end model | end-to-end model inference model path | -| e2e_limit_side_len | int | 768 | End-to-end input image side length limit | -| e2e_limit_type | str | "max" | End-to-end side length limit type, currently supports `min` and `max`. `min` means to ensure that the shortest side of the image is not less than `e2e_limit_side_len`, `max` means to ensure that the longest side of the image is not greater than `e2e_limit_side_len` | -| e2e_pgnet_score_thresh | float | 0.5 | End-to-end score threshold, results below this threshold are discarded | -| e2e_char_dict_path | str | "./ppocr/utils/ic15_dict.txt" | Recognition dictionary file path | -| e2e_pgnet_valid_set | str | "totaltext" | The name of the validation set, currently supports `totaltext`, `partvgg`, the post-processing methods corresponding to different data sets are different, and it can be consistent with the training process | -| e2e_pgnet_mode | str | "fast" | PGNet's detection result score calculation method, supports `fast` and `slow`, `fast` calculates the average score according to all pixels within the bounding rectangle of the polygon, `slow` calculates the average score according to all pixels within the original polygon, The calculation speed is relatively slower, but more accurate. | - - -* Angle classifier model related parameters - -| parameters | type | default | implication | -| :--: | :--: | :--: | :--: | -| use_angle_cls | bool | False | whether to use an angle classifier | -| cls_model_dir | str | None, if you need to use, you must specify the path explicitly | angle classifier inference model path | -| cls_image_shape | str | "3,48,192" | prediction shape | -| label_list | list | ['0', '180'] | The angle value corresponding to the class id | -| cls_batch_num | int | 6 | batch size | -| cls_thresh | float | 0.9 | Prediction threshold, when the model prediction result is 180 degrees, and the score is greater than the threshold, the final prediction result is considered to be 180 degrees and needs to be flipped | - - -* OCR image preprocessing parameters - -| parameters | type | default | implication | -| :--: | :--: | :--: | :--: | -| invert | bool | False | whether to invert image before processing | -| binarize | bool | False | whether to threshold binarize image before processing | -| alphacolor | tuple | "255,255,255" | Replacement color for the alpha channel, if the latter is present; R,G,B integers | diff --git a/doc/doc_en/inference_en.md b/doc/doc_en/inference_en.md deleted file mode 100755 index ec52ed9d48..0000000000 --- a/doc/doc_en/inference_en.md +++ /dev/null @@ -1,405 +0,0 @@ - -# Inference Based on Python Prediction Engine - -The inference model (the model saved by `paddle.jit.save`) is generally a solidified model saved after the model training is completed, and is mostly used to give prediction in deployment. - -The model saved during the training process is the checkpoints model, which saves the parameters of the model and is mostly used to resume training. - -Compared with the checkpoints model, the inference model will additionally save the structural information of the model. Therefore, it is easier to deploy because the model structure and model parameters are already solidified in the inference model file, and is suitable for integration with actual systems. -For more details, please refer to the document [Classification Framework](https://github.com/PaddlePaddle/PaddleClas/blob/release%2F2.0/docs/zh_CN/extension/paddle_mobile_inference.md). - -Next, we first introduce how to convert a trained model into an inference model, and then we will introduce text detection, text recognition, angle class, and the concatenation of them based on inference model. - -- [Inference Based on Python Prediction Engine](#inference-based-on-python-prediction-engine) - - [1. Convert Training Model to Inference Model](#1-convert-training-model-to-inference-model) - - [1.1 Convert Detection Model to Inference Model](#11-convert-detection-model-to-inference-model) - - [1.2 Convert Recognition Model to Inference Model](#12-convert-recognition-model-to-inference-model) - - [1.3 Convert Angle Classification Model to Inference Model](#13-convert-angle-classification-model-to-inference-model) - - [2. Text Detection Model Inference](#2-text-detection-model-inference) - - [2.1 Lightweight Chinese Detection Model Inference](#21-lightweight-chinese-detection-model-inference) - - [2.2 DB Text Detection Model Inference](#22-db-text-detection-model-inference) - - [2.3 EAST TEXT DETECTION MODEL INFERENCE](#23-east-text-detection-model-inference) - - [2.4 Sast Text Detection Model Inference](#24-sast-text-detection-model-inference) - - [(1). Quadrangle text detection model (ICDAR2015)](#1-quadrangle-text-detection-model-icdar2015) - - [(2). Curved text detection model (Total-Text)](#2-curved-text-detection-model-total-text) - - [3. Text Recognition Model Inference](#3-text-recognition-model-inference) - - [3.1 Lightweight Chinese Text Recognition Model Reference](#31-lightweight-chinese-text-recognition-model-reference) - - [3.2 CTC-Based Text Recognition Model Inference](#32-ctc-based-text-recognition-model-inference) - - [3.3 SRN-Based Text Recognition Model Inference](#33-srn-based-text-recognition-model-inference) - - [3.4 Text Recognition Model Inference Using Custom Characters Dictionary](#34-text-recognition-model-inference-using-custom-characters-dictionary) - - [3.5 Multilingual Model Inference](#35-multilingual-model-inference) - - [4. Angle Classification Model Inference](#4-angle-classification-model-inference) - - [5. Text Detection Angle Classification and Recognition Inference Concatenation](#5-text-detection-angle-classification-and-recognition-inference-concatenation) - - [5.1 Lightweight Chinese Model](#51-lightweight-chinese-model) - - [5.2 Other Models](#52-other-models) - - -## 1. Convert Training Model to Inference Model - - -### 1.1 Convert Detection Model to Inference Model - -Download the lightweight Chinese detection model: -``` -wget -P ./ch_lite/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_train.tar && tar xf ./ch_lite/ch_ppocr_mobile_v2.0_det_train.tar -C ./ch_lite/ -``` - -The above model is a DB algorithm trained with MobileNetV3 as the backbone. To convert the trained model into an inference model, just run the following command: -``` -# -c Set the training algorithm yml configuration file -# -o Set optional parameters -# Global.pretrained_model parameter Set the training model address to be converted without adding the file suffix .pdmodel, .pdopt or .pdparams. -# Global.save_inference_dir Set the address where the converted model will be saved. - -python3 tools/export_model.py -c configs/det/ch_ppocr_v2.0/ch_det_mv3_db_v2.0.yml -o Global.pretrained_model=./ch_lite/ch_ppocr_mobile_v2.0_det_train/best_accuracy Global.save_inference_dir=./inference/det_db/ -``` - -When converting to an inference model, the configuration file used is the same as the configuration file used during training. In addition, you also need to set the `Global.pretrained_model` parameter in the configuration file. -After the conversion is successful, there are three files in the model save directory: -``` -inference/det_db/ - ├── inference.pdiparams # The parameter file of detection inference model - ├── inference.pdiparams.info # The parameter information of detection inference model, which can be ignored - └── inference.pdmodel # The program file of detection inference model -``` - - -### 1.2 Convert Recognition Model to Inference Model - -Download the lightweight Chinese recognition model: -``` -wget -P ./ch_lite/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_train.tar && tar xf ./ch_lite/ch_ppocr_mobile_v2.0_rec_train.tar -C ./ch_lite/ -``` - -The recognition model is converted to the inference model in the same way as the detection, as follows: -``` -# -c Set the training algorithm yml configuration file -# -o Set optional parameters -# Global.pretrained_model parameter Set the training model address to be converted without adding the file suffix .pdmodel, .pdopt or .pdparams. -# Global.save_inference_dir Set the address where the converted model will be saved. - -python3 tools/export_model.py -c configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml -o Global.pretrained_model=./ch_lite/ch_ppocr_mobile_v2.0_rec_train/best_accuracy Global.save_inference_dir=./inference/rec_crnn/ -``` - -If you have a model trained on your own dataset with a different dictionary file, please make sure that you modify the `character_dict_path` in the configuration file to your dictionary file path. - -After the conversion is successful, there are three files in the model save directory: -``` -inference/det_db/ - ├── inference.pdiparams # The parameter file of recognition inference model - ├── inference.pdiparams.info # The parameter information of recognition inference model, which can be ignored - └── inference.pdmodel # The program file of recognition model -``` - - -### 1.3 Convert Angle Classification Model to Inference Model - -Download the angle classification model: -``` -wget -P ./ch_lite/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar && tar xf ./ch_lite/ch_ppocr_mobile_v2.0_cls_train.tar -C ./ch_lite/ -``` - -The angle classification model is converted to the inference model in the same way as the detection, as follows: -``` -# -c Set the training algorithm yml configuration file -# -o Set optional parameters -# Global.pretrained_model parameter Set the training model address to be converted without adding the file suffix .pdmodel, .pdopt or .pdparams. -# Global.save_inference_dir Set the address where the converted model will be saved. - -python3 tools/export_model.py -c configs/cls/cls_mv3.yml -o Global.pretrained_model=./ch_lite/ch_ppocr_mobile_v2.0_cls_train/best_accuracy Global.save_inference_dir=./inference/cls/ -``` - -After the conversion is successful, there are two files in the directory: -``` -inference/det_db/ - ├── inference.pdiparams # The parameter file of angle class inference model - ├── inference.pdiparams.info # The parameter information of angle class inference model, which can be ignored - └── inference.pdmodel # The program file of angle class model -``` - - - -## 2. Text Detection Model Inference - -The following will introduce the lightweight Chinese detection model inference, DB text detection model inference and EAST text detection model inference. The default configuration is based on the inference setting of the DB text detection model. -Because EAST and DB algorithms are very different, when inference, it is necessary to **adapt the EAST text detection algorithm by passing in corresponding parameters**. - - -### 2.1 Lightweight Chinese Detection Model Inference - -For lightweight Chinese detection model inference, you can execute the following commands: - -``` -# download DB text detection inference model -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_infer.tar -tar xf ch_ppocr_mobile_v2.0_det_infer.tar -# predict -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./inference/det_db/" -``` - -The visual text detection results are saved to the ./inference_results folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_00018069.jpg) - -You can use the parameters `limit_type` and `det_limit_side_len` to limit the size of the input image, -The optional parameters of `limit_type` are [`max`, `min`], and -`det_limit_size_len` is a positive integer, generally set to a multiple of 32, such as 960. - -The default setting of the parameters is `limit_type='max', det_limit_side_len=960`. Indicates that the longest side of the network input image cannot exceed 960, -If this value is exceeded, the image will be resized with the same width ratio to ensure that the longest side is `det_limit_side_len`. -Set as `limit_type='min', det_limit_side_len=960`, it means that the shortest side of the image is limited to 960. - -If the resolution of the input picture is relatively large and you want to use a larger resolution prediction, you can set det_limit_side_len to the desired value, such as 1216: -``` -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./inference/det_db/" --det_limit_type=max --det_limit_side_len=1216 -``` - -If you want to use the CPU for prediction, execute the command as follows -``` -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./inference/det_db/" --use_gpu=False -``` - - -### 2.2 DB Text Detection Model Inference - -First, convert the model saved in the DB text detection training process into an inference model. Taking the model based on the Resnet50_vd backbone network and trained on the ICDAR2015 English dataset as an example ([model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_db_v2.0_train.tar)), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_db.yml -o Global.pretrained_model=./det_r50_vd_db_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_db -``` - -DB text detection model inference, you can execute the following command: - -``` -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_db/" -``` - -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_img_10_db.jpg) - -**Note**: Since the ICDAR2015 dataset has only 1,000 training images, mainly for English scenes, the above model has very poor detection result on Chinese text images. - - -### 2.3 EAST TEXT DETECTION MODEL INFERENCE - -First, convert the model saved in the EAST text detection training process into an inference model. Taking the model based on the Resnet50_vd backbone network and trained on the ICDAR2015 English dataset as an example ([model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_east_v2.0_train.tar)), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_east.yml -o Global.pretrained_model=./det_r50_vd_east_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_east -``` -**For EAST text detection model inference, you need to set the parameter ``--det_algorithm="EAST"``**, run the following command: - -``` -python3 tools/infer/predict_det.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_east/" --det_algorithm="EAST" -``` - -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_img_10_east.jpg) - -**Note**: EAST post-processing locality aware NMS has two versions: Python and C++. The speed of C++ version is obviously faster than that of Python version. Due to the compilation version problem of NMS of C++ version, C++ version NMS will be called only in Python 3.5 environment, and python version NMS will be called in other cases. - - - -### 2.4 Sast Text Detection Model Inference -#### (1). Quadrangle text detection model (ICDAR2015) -First, convert the model saved in the SAST text detection training process into an inference model. Taking the model based on the Resnet50_vd backbone network and trained on the ICDAR2015 English dataset as an example ([model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_icdar15_v2.0_train.tar)), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_sast_icdar15.yml -o Global.pretrained_model=./det_r50_vd_sast_icdar15_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_sast_ic15 -``` - -**For SAST quadrangle text detection model inference, you need to set the parameter `--det_algorithm="SAST"`**, run the following command: - -``` -python3 tools/infer/predict_det.py --det_algorithm="SAST" --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_sast_ic15/" -``` - -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_img_10_sast.jpg) - -#### (2). Curved text detection model (Total-Text) -First, convert the model saved in the SAST text detection training process into an inference model. Taking the model based on the Resnet50_vd backbone network and trained on the Total-Text English dataset as an example ([model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/det_r50_vd_sast_totaltext_v2.0_train.tar)), you can use the following command to convert: - -``` -python3 tools/export_model.py -c configs/det/det_r50_vd_sast_totaltext.yml -o Global.pretrained_model=./det_r50_vd_sast_totaltext_v2.0_train/best_accuracy Global.save_inference_dir=./inference/det_sast_tt -``` - -For SAST curved text detection model inference, you need to set the parameter `--det_algorithm="SAST"` and `--det_sast_polygon=True`, run the following command: - -``` -python3 tools/infer/predict_det.py --det_algorithm="SAST" --image_dir="./doc/imgs_en/img623.jpg" --det_model_dir="./inference/det_sast_tt/" --det_sast_polygon=True -``` - -The visualized text detection results are saved to the `./inference_results` folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_img623_sast.jpg) - -**Note**: SAST post-processing locality aware NMS has two versions: Python and C++. The speed of C++ version is obviously faster than that of Python version. Due to the compilation version problem of NMS of C++ version, C++ version NMS will be called only in Python 3.5 environment, and python version NMS will be called in other cases. - - -## 3. Text Recognition Model Inference - -The following will introduce the lightweight Chinese recognition model inference, other CTC-based and Attention-based text recognition models inference. For Chinese text recognition, it is recommended to choose the recognition model based on CTC loss. In practice, it is also found that the result of the model based on Attention loss is not as good as the one based on CTC loss. In addition, if the characters dictionary is modified during training, make sure that you use the same characters set during inference. Please check below for details. - - - -### 3.1 Lightweight Chinese Text Recognition Model Reference - -For lightweight Chinese recognition model inference, you can execute the following commands: - -``` -# download CRNN text recognition inference model -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_infer.tar -tar xf ch_ppocr_mobile_v2.0_rec_infer.tar -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_10.png" --rec_model_dir="ch_ppocr_mobile_v2.0_rec_infer" -``` - -![](../imgs_words_en/word_10.png) - -After executing the command, the prediction results (recognized text and score) of the above image will be printed on the screen. - -```bash -Predicts of ./doc/imgs_words_en/word_10.png:('PAIN', 0.9897658) -``` - - -### 3.2 CTC-Based Text Recognition Model Inference - -Taking CRNN as an example, we introduce the recognition model inference based on CTC loss. Rosetta and Star-Net are used in a similar way, No need to set the recognition algorithm parameter rec_algorithm. - -First, convert the model saved in the CRNN text recognition training process into an inference model. Taking the model based on Resnet34_vd backbone network, using MJSynth and SynthText (two English text recognition synthetic datasets) for training, as an example ([model download address](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar)). It can be converted as follow: - -``` -python3 tools/export_model.py -c configs/det/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model=./rec_r34_vd_none_bilstm_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_crnn -``` - -For CRNN text recognition model inference, execute the following commands: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./inference/starnet/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" -``` - -![](../imgs_words_en/word_336.png) - -After executing the command, the recognition result of the above image is as follows: - -```bash -Predicts of ./doc/imgs_words_en/word_336.png:('super', 0.9999073) -``` - -**Note**:Since the above model refers to [DTRB](https://arxiv.org/abs/1904.01906) text recognition training and evaluation process, it is different from the training of lightweight Chinese recognition model in two aspects: - -- The image resolution used in training is different: the image resolution used in training the above model is [3,32,100], while during our Chinese model training, in order to ensure the recognition effect of long text, the image resolution used in training is [3, 32, 320]. The default shape parameter of the inference stage is the image resolution used in training phase, that is [3, 32, 320]. Therefore, when running inference of the above English model here, you need to set the shape of the recognition image through the parameter `rec_image_shape`. - -- Character list: the experiment in the DTRB paper is only for 26 lowercase English characters and 10 numbers, a total of 36 characters. All upper and lower case characters are converted to lower case characters, and characters not in the above list are ignored and considered as spaces. Therefore, no characters dictionary file is used here, but a dictionary is generated by the below command. - -``` -self.character_str = "0123456789abcdefghijklmnopqrstuvwxyz" -dict_character = list(self.character_str) -``` - - -### 3.3 SRN-Based Text Recognition Model Inference - -The recognition model based on SRN requires additional setting of the recognition algorithm parameter ---rec_algorithm="SRN". At the same time, it is necessary to ensure that the predicted shape is consistent -with the training, such as: --rec_image_shape="1, 64, 256" - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" \ - --rec_model_dir="./inference/srn/" \ - --rec_image_shape="1, 64, 256" \ - --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" \ - --rec_algorithm="SRN" -``` - - -### 3.4 Text Recognition Model Inference Using Custom Characters Dictionary -If the text dictionary is modified during training, when using the inference model to predict, you need to specify the dictionary path used by `--rec_char_dict_path` - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./your inference model" --rec_image_shape="3, 32, 100" --rec_char_dict_path="your text dict path" -``` - - - -### 3.5 Multilingual Model Inference -If you need to predict other language models, when using inference model prediction, you need to specify the dictionary path used by `--rec_char_dict_path`. At the same time, in order to get the correct visualization results, -You need to specify the visual font path through `--vis_font_path`. There are small language fonts provided by default under the `doc/fonts` path, such as Korean recognition: - -``` -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/korean/1.jpg" --rec_model_dir="./your inference model" --rec_char_dict_path="ppocr/utils/dict/korean_dict.txt" --vis_font_path="doc/fonts/korean.ttf" -``` -![](../imgs_words/korean/1.jpg) - -After executing the command, the prediction result of the above figure is: - -``` text -Predicts of ./doc/imgs_words/korean/1.jpg:('바탕으로', 0.9948904) -``` - - -## 4. Angle Classification Model Inference - -For angle classification model inference, you can execute the following commands: - -``` -python3 tools/infer/predict_cls.py --image_dir="./doc/imgs_words_en/word_10.png" --cls_model_dir="./inference/cls/" -``` -``` -# download text angle class inference model: -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar -tar xf ch_ppocr_mobile_v2.0_cls_infer.tar -python3 tools/infer/predict_cls.py --image_dir="./doc/imgs_words_en/word_10.png" --cls_model_dir="ch_ppocr_mobile_v2.0_cls_infer" -``` -![](../imgs_words_en/word_10.png) - -After executing the command, the prediction results (classification angle and score) of the above image will be printed on the screen. - -``` - Predicts of ./doc/imgs_words_en/word_10.png:['0', 0.9999995] -``` - - -## 5. Text Detection Angle Classification and Recognition Inference Concatenation - - -### 5.1 Lightweight Chinese Model - -When performing prediction, you need to specify the path of a single image or a folder of images through the parameter `image_dir`, the parameter `det_model_dir` specifies the path to detect the inference model, the parameter `cls_model_dir` specifies the path to angle classification inference model and the parameter `rec_model_dir` specifies the path to identify the inference model. The parameter `use_angle_cls` is used to control whether to enable the angle classification model. The parameter `use_mp` specifies whether to use multi-process to infer `total_process_num` specifies process number when using multi-process. The parameter(Paddle Inference is not thread-safe, it is recommended to use multi-process) . The visualized recognition results are saved to the `./inference_results` folder by default. - -```shell -# use direction classifier -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./inference/det_db/" --cls_model_dir="./inference/cls/" --rec_model_dir="./inference/rec_crnn/" --use_angle_cls=true - -# not use use direction classifier -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./inference/det_db/" --rec_model_dir="./inference/rec_crnn/" - -# use multi-process -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./inference/det_db/" --rec_model_dir="./inference/rec_crnn/" --use_angle_cls=false --use_mp=True --total_process_num=6 -``` - - -After executing the command, the recognition result image is as follows: - -![](../imgs_results/system_res_00018069.jpg) - - -### 5.2 Other Models - -If you want to try other detection algorithms or recognition algorithms, please refer to the above text detection model inference and text recognition model inference, update the corresponding configuration and model. - -**Note: due to the limitation of rotation logic of detected box, SAST curved text detection model (using the parameter `det_sast_polygon=True`) is not supported for model combination yet.** - -The following command uses the combination of the EAST text detection and STAR-Net text recognition: - -``` -python3 tools/infer/predict_system.py --image_dir="./doc/imgs_en/img_10.jpg" --det_model_dir="./inference/det_east/" --det_algorithm="EAST" --rec_model_dir="./inference/starnet/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" -``` - -After executing the command, the recognition result image is as follows: - -![](../imgs_results/img_10_east_starnet.jpg) diff --git a/doc/doc_en/inference_ppocr_en.md b/doc/doc_en/inference_ppocr_en.md deleted file mode 100755 index ba1c23babf..0000000000 --- a/doc/doc_en/inference_ppocr_en.md +++ /dev/null @@ -1,197 +0,0 @@ - -# Python Inference for PP-OCR Model Zoo - -This article introduces the use of the Python inference engine for the PP-OCR model library. The content is in order of text detection, text recognition, direction classifier and the prediction method of the three in series on the CPU and GPU. - - -- [Python Inference for PP-OCR Model Zoo](#python-inference-for-pp-ocr-model-zoo) - - [Text Detection Model Inference](#text-detection-model-inference) - - [Text Recognition Model Inference](#text-recognition-model-inference) - - [1. Lightweight Chinese Recognition Model Inference](#1-lightweight-chinese-recognition-model-inference) - - [2. English Recognition Model Inference](#2-english-recognition-model-inference) - - [3. Multilingual Model Inference](#3-multilingual-model-inference) - - [Angle Classification Model Inference](#angle-classification-model-inference) - - [Text Detection Angle Classification and Recognition Inference Concatenation](#text-detection-angle-classification-and-recognition-inference-concatenation) - - [TensorRT Inference](TensorRT-Inference) - - - -## Text Detection Model Inference - -The default configuration is based on the inference setting of the DB text detection model. For lightweight Chinese detection model inference, you can execute the following commands: - -``` -# download DB text detection inference model -wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar -tar xf ch_PP-OCRv3_det_infer.tar -# run inference -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" -``` - -The visual text detection results are saved to the ./inference_results folder by default, and the name of the result file is prefixed with 'det_res'. Examples of results are as follows: - -![](../imgs_results/det_res_00018069.jpg) - -You can use the parameters `limit_type` and `det_limit_side_len` to limit the size of the input image, -The optional parameters of `limit_type` are [`max`, `min`], and -`det_limit_size_len` is a positive integer, generally set to a multiple of 32, such as 960. - -The default setting of the parameters is `limit_type='max', det_limit_side_len=960`. Indicates that the longest side of the network input image cannot exceed 960, -If this value is exceeded, the image will be resized with the same width ratio to ensure that the longest side is `det_limit_side_len`. -Set as `limit_type='min', det_limit_side_len=960`, it means that the shortest side of the image is limited to 960. - -If the resolution of the input picture is relatively large and you want to use a larger resolution prediction, you can set det_limit_side_len to the desired value, such as 1216: -``` -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --det_limit_type=max --det_limit_side_len=1216 -``` - -If you want to use the CPU for prediction, execute the command as follows -``` -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --use_gpu=False -``` - - - -## Text Recognition Model Inference - - - -### 1. Lightweight Chinese Recognition Model Inference - -**Note**: The input shape used by the recognition model of `PP-OCRv3` is `3, 48, 320`. If you use other recognition models, you need to set the parameter `--rec_image_shape` according to the model. In addition, the `rec_algorithm` used by the recognition model of `PP-OCRv3` is `SVTR_LCNet` by default. Note the difference from the original `SVTR`. - - -For lightweight Chinese recognition model inference, you can execute the following commands: - -``` -# download CRNN text recognition inference model -wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar -tar xf ch_PP-OCRv3_rec_infer.tar -# run inference -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_10.png" --rec_model_dir="./ch_PP-OCRv3_rec_infer/" --rec_image_shape=3,48,320 -``` - -![](../imgs_words_en/word_10.png) - -After executing the command, the prediction results (recognized text and score) of the above image will be printed on the screen. - -```bash -Predicts of ./doc/imgs_words_en/word_10.png:('PAIN', 0.988671) -``` - -### 2. English Recognition Model Inference - -For English recognition model inference, you can execute the following commands,you need to specify the dictionary path used by `--rec_char_dict_path`: - -``` -# download en model: -wget https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_infer.tar -tar xf en_PP-OCRv3_rec_infer.tar -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./en_PP-OCRv3_rec_infer/" --rec_char_dict_path="ppocr/utils/en_dict.txt" -``` - -![](../imgs_words/en/word_1.png) - - -After executing the command, the prediction result of the above figure is: - -``` -Predicts of ./doc/imgs_words/en/word_1.png: ('JOINT', 0.998160719871521) -``` - - - - -### 3. Multilingual Model Inference -If you need to predict [other language models](./models_list_en.md#Multilingual), when using inference model prediction, you need to specify the dictionary path used by `--rec_char_dict_path`. At the same time, in order to get the correct visualization results, -You need to specify the visual font path through `--vis_font_path`. There are small language fonts provided by default under the `doc/fonts` path, such as Korean recognition: - -``` -wget wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/multilingual/korean_mobile_v2.0_rec_infer.tar - -python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/korean/1.jpg" --rec_model_dir="./your inference model" --rec_char_dict_path="ppocr/utils/dict/korean_dict.txt" --vis_font_path="doc/fonts/korean.ttf" -``` -![](../imgs_words/korean/1.jpg) - -After executing the command, the prediction result of the above figure is: - -``` text -Predicts of ./doc/imgs_words/korean/1.jpg:('바탕으로', 0.9948904) -``` - - - -## Angle Classification Model Inference - -For angle classification model inference, you can execute the following commands: - - -``` -# download text angle class inference model: -wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar -tar xf ch_ppocr_mobile_v2.0_cls_infer.tar -python3 tools/infer/predict_cls.py --image_dir="./doc/imgs_words_en/word_10.png" --cls_model_dir="ch_ppocr_mobile_v2.0_cls_infer" -``` -![](../imgs_words_en/word_10.png) - -After executing the command, the prediction results (classification angle and score) of the above image will be printed on the screen. - -``` - Predicts of ./doc/imgs_words_en/word_10.png:['0', 0.9999995] -``` - - -## Text Detection Angle Classification and Recognition Inference Concatenation - -**Note**: The input shape used by the recognition model of `PP-OCRv3` is `3, 48, 320`. If you use other recognition models, you need to set the parameter `--rec_image_shape` according to the model. In addition, the `rec_algorithm` used by the recognition model of `PP-OCRv3` is `SVTR_LCNet` by default. Note the difference from the original `SVTR`. - -When performing prediction, you need to specify the path of a single image or a folder of images through the parameter `image_dir`, pdf file is also supported, the parameter `det_model_dir` specifies the path to detect the inference model, the parameter `cls_model_dir` specifies the path to angle classification inference model and the parameter `rec_model_dir` specifies the path to identify the inference model. The parameter `use_angle_cls` is used to control whether to enable the angle classification model. The parameter `use_mp` specifies whether to use multi-process to infer `total_process_num` specifies process number when using multi-process. The parameter . The visualized recognition results are saved to the `./inference_results` folder by default. - -```shell -# use direction classifier -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --cls_model_dir="./cls/" --rec_model_dir="./ch_PP-OCRv3_rec_infer/" --use_angle_cls=true -# not use use direction classifier -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --rec_model_dir="./ch_PP-OCRv3_rec_infer/" --use_angle_cls=false -# use multi-process -python3 tools/infer/predict_system.py --image_dir="./doc/imgs/00018069.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --rec_model_dir="./ch_PP-OCRv3_rec_infer/" --use_angle_cls=false --use_mp=True --total_process_num=6 -# use PDF files, you can infer the first few pages by using the `page_num` parameter, the default is 0, which means infer all pages -python3 tools/infer/predict_system.py --image_dir="./xxx.pdf" --det_model_dir="./ch_PP-OCRv3_det_infer/" --cls_model_dir="./cls/" --rec_model_dir="./ch_PP-OCRv3_rec_infer/" --use_angle_cls=true --page_num=2 -``` - - -After executing the command, the recognition result image is as follows: - -![](../imgs_results/system_res_00018069_v3.jpg) - -For more configuration and explanation of inference parameters, please refer to:[Model Inference Parameters Explained Tutorial](./inference_args_en.md)。 - - -## TensorRT Inference - -Paddle Inference ensembles TensorRT using subgraph mode. For GPU deployment scenarios, TensorRT can optimize some subgraphs, including horizontal and vertical integration of OPs, filter redundant OPs, and automatically select the optimal OP kernels for to speed up inference. - -You need to do the following 2 steps for inference using TRT. - -* (1) Collect the dynamic shape information of the model about a specific dataset and store it in a file. -* (2) Load the dynamic shape information file for TRT inference. - - -Taking the text detection model as an example. Firstly, you can use the following command to generate a dynamic shape file, which will eventually be named as `det_trt_dynamic_shape.txt` and stored in the `ch_PP-OCRv3_det_infer` folder. - -```bash -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --use_tensorrt=True -``` - -The above command is only used to collect dynamic shape information, and TRT is not used during inference. - -Then, you can use the following command to perform TRT inference. - - -```bash -python3 tools/infer/predict_det.py --image_dir="./doc/imgs/1.jpg" --det_model_dir="./ch_PP-OCRv3_det_infer/" --use_tensorrt=True -``` - -**Note:** - -* In the first step, if the dynamic shape information file already exists, it does not need to be collected again. If you want to regenerate the dynamic shape information file, you need to delete the dynamic shape information file in the model folder firstly, and then regenerate it. -* In general, dynamic shape information file only needs to be generated once. In the actual deployment process, it is recommended that the dynamic shape information file can be generated on offline validation set or test set, and then the file can be directly loaded for online TRT inference. diff --git a/doc/doc_en/installation_en.md b/doc/doc_en/installation_en.md deleted file mode 100644 index 5d1ffb007e..0000000000 --- a/doc/doc_en/installation_en.md +++ /dev/null @@ -1,65 +0,0 @@ -## QUICK INSTALLATION - -After testing, PaddleOCR can run on glibc 2.23. You can also test other glibc versions or install glibc 2.23 for the best compatibility. - -PaddleOCR working environment: -- PaddlePaddle 2.0.0 -- Python 3.7 -- glibc 2.23 - -It is recommended to use the docker provided by us to run PaddleOCR. Please refer to the docker tutorial [link](https://www.runoob.com/docker/docker-tutorial.html/). - -*If you want to directly run the prediction code on Mac or Windows, you can start from step 2.* - -**1. (Recommended) Prepare a docker environment. For the first time you use this docker image, it will be downloaded automatically. Please be patient.** -``` -# Switch to the working directory -cd /home/Projects -# You need to create a docker container for the first run, and do not need to run the current command when you run it again -# Create a docker container named ppocr and map the current directory to the /paddle directory of the container - -#If using CPU, use docker instead of nvidia-docker to create docker -sudo docker run --name ppocr -v $PWD:/paddle --network=host -it paddlepaddle/paddle:latest-dev-cuda10.1-cudnn7-gcc82 /bin/bash -``` - -With CUDA10, please run the following command to create a container. -It is recommended to set a shared memory greater than or equal to 32G through the --shm-size parameter: -``` -sudo nvidia-docker run --name ppocr -v $PWD:/paddle --shm-size=64G --network=host -it paddlepaddle/paddle:latest-dev-cuda10.1-cudnn7-gcc82 /bin/bash -``` -You can also visit [DockerHub](https://hub.docker.com/r/paddlepaddle/paddle/tags/) to get the image that fits your machine. -``` -# ctrl+P+Q to exit docker, to re-enter docker using the following command: -sudo docker container exec -it ppocr /bin/bash -``` - -**2. Install PaddlePaddle 2.0** -``` -pip3 install --upgrade pip - -# If you have cuda9 or cuda10 installed on your machine, please run the following command to install -python3 -m pip install paddlepaddle-gpu==2.0.0 -i https://mirror.baidu.com/pypi/simple - -# If you only have cpu on your machine, please run the following command to install -python3 -m pip install paddlepaddle==2.0.0 -i https://mirror.baidu.com/pypi/simple -``` -For more software version requirements, please refer to the instructions in [Installation Document](https://www.paddlepaddle.org.cn/install/quick) for operation. - - -**3. Clone PaddleOCR repo** -``` -# Recommend -git clone https://github.com/PaddlePaddle/PaddleOCR - -# If you cannot pull successfully due to network problems, you can switch to the mirror hosted on Gitee: - -git clone https://gitee.com/paddlepaddle/PaddleOCR - -# Note: The mirror on Gitee may not keep in synchronization with the latest update with the project on GitHub. There might be a delay of 3-5 days. Please try GitHub at first. -``` - -**4. Install third-party libraries** -``` -cd PaddleOCR -pip3 install -r requirements.txt -``` diff --git a/doc/doc_en/kie_en.md b/doc/doc_en/kie_en.md deleted file mode 100644 index b041b39107..0000000000 --- a/doc/doc_en/kie_en.md +++ /dev/null @@ -1,528 +0,0 @@ -# Key Information Extraction - -This tutorial provides a guide to the whole process of key information extraction using PaddleOCR, including data preparation, model training, optimization, evaluation, prediction of semantic entity recognition (SER) and relationship extraction (RE) tasks. - - -- [1. Data Preparation](#Data-Preparation) - - [1.1. Prepare for dataset](#11-Prepare-for-dataset) - - [1.2. Custom Dataset](#12-Custom-Dataset) - - [1.3. Download data](#13-Download-data) -- [2. Training](#2-Training) - - [2.1. Start Training](#21-start-training) - - [2.2. Resume Training](#22-Resume-Training) - - [2.3. Mixed Precision Training](#23-Mixed-Precision-Training) - - [2.4. Distributed Training](#24-Distributed-Training) - - [2.5. Train using knowledge distillation](#25-Train-using-knowledge-distillation) - - [2.6. Training on other platform](#26-Training-on-other-platform) -- [3. Evaluation and Test](#3-Evaluation-and-Test) - - [3.1. Evaluation](#31-指标评估) - - [3.2. Test](#32-Test) -- [4. Model inference](#4-Model-inference) -- [5. FAQ](#5-faq) - - -# 1. Data Preparation - -## 1.1. Prepare for dataset - -PaddleOCR supports the following data format when training KIE models. - -- `general data` is used to train a dataset whose annotation is stored in a text file (SimpleDataset). - - -The default storage path of training data is `PaddleOCR/train_data`. If you already have datasets on your disk, you only need to create a soft link to the dataset directory. - -``` -# linux and mac os -ln -sf /train_data/dataset -# windows -mklink /d /train_data/dataset -``` - -## 1.2. Custom Dataset - -The training process generally includes the training set and the evaluation set. The data formats of the two sets are same. - -**(1) Training set** - -It is recommended to put the training images into the same folder, record the path and annotation of images in a text file. The contents of the text file are as follows: - - -```py -" image path annotation information " -zh_train_0.jpg [{"transcription": "汇丰晋信", "label": "other", "points": [[104, 114], [530, 114], [530, 175], [104, 175]], "id": 1, "linking": []}, {"transcription": "受理时间:", "label": "question", "points": [[126, 267], [266, 267], [266, 305], [126, 305]], "id": 7, "linking": [[7, 13]]}, {"transcription": "2020.6.15", "label": "answer", "points": [[321, 239], [537, 239], [537, 285], [321, 285]], "id": 13, "linking": [[7, 13]]}] -zh_train_1.jpg [{"transcription": "中国人体器官捐献", "label": "other", "points": [[544, 459], [954, 459], [954, 517], [544, 517]], "id": 1, "linking": []}, {"transcription": ">编号:MC545715483585", "label": "other", "points": [[1462, 470], [2054, 470], [2054, 543], [1462, 543]], "id": 10, "linking": []}, {"transcription": "CHINAORGANDONATION", "label": "other", "points": [[543, 516], [958, 516], [958, 551], [543, 551]], "id": 14, "linking": []}, {"transcription": "中国人体器官捐献志愿登记表", "label": "header", "points": [[635, 793], [1892, 793], [1892, 904], [635, 904]], "id": 18, "linking": []}] -... -``` - -**Note:** In the text file, please split the image path and annotation with `\t`. Otherwise, error will happen when training. - -The annotation can be parsed by `json` into a list of sub-annotations. Each element in the list is a dict, which stores the required information of each text line. The required fields are as follows. - -- transcription: stores the text content of the text line -- label: the category of the text line content -- points: stores the four point position information of the text line -- id: stores the ID information of the text line for RE model training -- linking: stores the connection information between text lines for RE model training - -**(2) Evaluation set** - -The evaluation set is constructed in the same way as the training set. - -**(3) Dictionary file** - -The textlines in the training set and the evaluation set contain label information. The list of all labels is stored in the dictionary file (such as `class_list.txt`). Each line in the dictionary file is represented as a label name. - -For example, FUND_zh data contains four categories. The contents of the dictionary file are as follows. - -``` -OTHER -QUESTION -ANSWER -HEADER -``` - -In the annotation file, the annotation information of the `label` field of the text line content of each annotation needs to belong to the dictionary content. - - -The final dataset shall have the following file structure. - -``` -|-train_data - |-data_name - |- train.json - |- train - |- zh_train_0.png - |- zh_train_1.jpg - | ... - |- val.json - |- val - |- zh_val_0.png - |- zh_val_1.jpg - | ... -``` - -**Note:** - --The category information in the annotation file is not case sensitive. For example, 'HEADER' and 'header' will be seen as the same category ID. -- In the dictionary file, it is recommended to put the `other` category (other textlines that need not be paid attention to can be labeled as `other`) on the first line. When parsing, the category ID of the 'other' category will be resolved to 0, and the textlines predicted as `other` will not be visualized later. - -## 1.3. Download data - -If you do not have local dataset, you can donwload the source files of [XFUND](https://github.com/doc-analysis/XFUND) or [FUNSD](https://guillaumejaume.github.io/FUNSD) and use the scripts of [XFUND](../../ppstructure/kie/tools/trans_xfun_data.py) or [FUNSD](../../ppstructure/kie/tools/trans_funsd_label.py) for tranform them into PaddleOCR format. Then you can use the public dataset to quick experience KIE. - -For more information about public KIE datasets, please refer to [KIE dataset tutorial](./dataset/kie_datasets_en.md). - -PaddleOCR also supports the annotation of KIE models. Please refer to [PPOCRLabel tutorial](https://github.com/PFCCLab/PPOCRLabel/blob/main/README.md). - -# 2. Training - -PaddleOCR provides training scripts, evaluation scripts and inference scripts. We will introduce based on VI-LayoutXLM model in this section. -This section will take the VI layoutxlm multimodal pre training model as an example to explain. - -> If you want to use the SDMGR based KIE algorithm, please refer to: [SDMGR tutorial](./algorithm_kie_sdmgr_en.md). - - -## 2.1. Start Training - -If you do not use a custom dataset, you can use XFUND_zh that has been processed in PaddleOCR dataset for quick experience. - - -```bash -mkdir train_data -cd train_data -wget https://paddleocr.bj.bcebos.com/ppstructure/dataset/XFUND.tar && tar -xf XFUND.tar -cd .. -``` - -If you don't want to train, and want to directly experience the process of model evaluation, prediction, and inference, you can download the training model provided in PaddleOCR and skip section 2.1. - - -Use the following command to download the trained model. - -```bash -mkdir pretrained_model -cd pretrained_model -# download and uncompress SER model -wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar & tar -xf ser_vi_layoutxlm_xfund_pretrained.tar - -# download and uncompress RE model -wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar & tar -xf re_vi_layoutxlm_xfund_pretrained.tar -``` - -Start training: - -- If your paddlepaddle version is `CPU`, you need to set `Global.use_gpu=False` in your config file. -- During training, PaddleOCR will download the VI-LayoutXLM pretraining model by default. There is no need to download it in advance. - -```bash -# GPU training, support single card and multi-cards -# The training log will be save in "{Global.save_model_dir}/train.log" - -# train SER model using single card -python3 tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml - -# train SER model using multi-cards, you can use --gpus to assign the GPU ids. -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml - -# train RE model using single card -python3 tools/train.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml -``` - -Take the SER model training as an example. After the training is started, you will see the following log output. - -``` -[2022/08/08 16:28:28] ppocr INFO: epoch: [1/200], global_step: 10, lr: 0.000006, loss: 1.871535, avg_reader_cost: 0.28200 s, avg_batch_cost: 0.82318 s, avg_samples: 8.0, ips: 9.71838 samples/s, eta: 0:51:59 -[2022/08/08 16:28:33] ppocr INFO: epoch: [1/200], global_step: 19, lr: 0.000018, loss: 1.461939, avg_reader_cost: 0.00042 s, avg_batch_cost: 0.32037 s, avg_samples: 6.9, ips: 21.53773 samples/s, eta: 0:37:55 -[2022/08/08 16:28:39] ppocr INFO: cur metric, precision: 0.11526348939743859, recall: 0.19776657060518732, hmean: 0.14564265817747712, fps: 34.008392345050055 -[2022/08/08 16:28:45] ppocr INFO: save best model is to ./output/ser_vi_layoutxlm_xfund_zh/best_accuracy -[2022/08/08 16:28:45] ppocr INFO: best metric, hmean: 0.14564265817747712, precision: 0.11526348939743859, recall: 0.19776657060518732, fps: 34.008392345050055, best_epoch: 1 -[2022/08/08 16:28:51] ppocr INFO: save model in ./output/ser_vi_layoutxlm_xfund_zh/latest -``` - -The following information will be automatically printed. - - -|Field | meaning| -| :----: | :------: | -|epoch | current iteration round| -|iter | current iteration times| -|lr | current learning rate| -|loss | current loss function| -| reader_cost | current batch data processing time| -| batch_ Cost | total current batch time| -|samples | number of samples in the current batch| -|ips | number of samples processed per second| - - -PaddleOCR supports evaluation during training. you can modify `eval_batch_step` in the config file `configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` (default as 19 iters). Trained model with best hmean will be saved as `output/ser_vi_layoutxlm_xfund_zh/best_accuracy/`. - -If the evaluation dataset is very large, it's recommended to enlarge the eval interval or evaluate the model after training. - -**Note:** for more KIE models training and configuration files, you can go into `configs/kie/` or refer to [Frontier KIE algorithms](./algorithm_overview_en.md). - - -If you want to train model on your own dataset, you need to modify the data path, dictionary file and category number in the configuration file. - - -Take `configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` as an example, contents we need to fix is as follows. - -```yaml -Architecture: - # ... - Backbone: - name: LayoutXLMForSer - pretrained: True - mode: vi - # Assuming that n categroies are included in the dictionary file (other is included), the the num_classes is set as 2n-1 - num_classes: &num_classes 7 - -PostProcess: - name: kieSerTokenLayoutLMPostProcess - # Modify the dictionary file path for your custom dataset - class_path: &class_path train_data/XFUND/class_list_xfun.txt - -Train: - dataset: - name: SimpleDataSet - # Modify the data path for your training dataset - data_dir: train_data/XFUND/zh_train/image - # Modify the data annotation path for your training dataset - label_file_list: - - train_data/XFUND/zh_train/train.json - ... - loader: - # batch size for single card when training - batch_size_per_card: 8 - ... - -Eval: - dataset: - name: SimpleDataSet - # Modify the data path for your evaluation dataset - data_dir: train_data/XFUND/zh_val/image - # Modify the data annotation path for your evaluation dataset - label_file_list: - - train_data/XFUND/zh_val/val.json - ... - loader: - # batch size for single card when evaluation - batch_size_per_card: 8 -``` - -**Note that the configuration file for prediction/evaluation must be consistent with the training file.** - - -## 2.2. Resume Training - -If the training process is interrupted and you want to load the saved model to resume training, you can specify the path of the model to be loaded by specifying `Architecture.Backbone.checkpoints`. - - -```bash -python3 tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy -``` - -**Note:** - -- Priority of `Architecture.Backbone.checkpoints` is higher than` Architecture.Backbone.pretrained`. You need to set `Architecture.Backbone.checkpoints` for model finetuning, resume and evalution. If you want to train with the NLP pretrained model, you need to set `Architecture.Backbone.pretrained` as `True` and set `Architecture.Backbone.checkpoints` as null (`null`). -- PaddleNLP pretrained models are used here for LayoutXLM series models, the model loading and saving logic is same as those in PaddleNLP. Therefore we do not need to set `Global.pretrained_model` or `Global.checkpoints` here. -- If you use knowledge distillation to train the LayoutXLM series models, resuming training is not supported now. - -## 2.3. Mixed Precision Training - -coming soon! - -## 2.4. Distributed Training - -During multi-machine multi-gpu training, use the `--ips` parameter to set the used machine IP address, and the `--gpus` parameter to set the used GPU ID: - -```bash -python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -``` - -**Note:** (1) When using multi-machine and multi-gpu training, you need to replace the ips value in the above command with the address of your machine, and the machines need to be able to ping each other. (2) Training needs to be launched separately on multiple machines. The command to view the ip address of the machine is `ifconfig`. (3) For more details about the distributed training speedup ratio, please refer to [Distributed Training Tutorial](./distributed_training_en.md). - - -## 2.5. Train with Knowledge Distillation - -Knowledge distillation is supported in PaddleOCR for KIE model training process. The configuration file is [ser_vi_layoutxlm_xfund_zh_udml.yml](../../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml). For more information, please refer to [doc](./knowledge_distillation_en.md). - -**Note:** The saving and loading logic of the LayoutXLM series KIE models in PaddleOCR is consistent with PaddleNLP, so only the parameters of the student model are saved in the distillation process. If you want to use the saved model for evaluation, you need to use the configuration of the student model (the student model corresponding to the distillation file above is [ser_vi_layoutxlm_xfund_zh.yml](../../configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml). - - - -## 2.6. Training on other platform - -- Windows GPU/CPU -The Windows platform is slightly different from the Linux platform: -Windows platform only supports `single gpu` training and inference, specify GPU for training `set CUDA_VISIBLE_DEVICES=0` -On the Windows platform, DataLoader only supports single-process mode, so you need to set `num_workers` to 0; - -- macOS -GPU mode is not supported, you need to set `use_gpu` to False in the configuration file, and the rest of the training evaluation prediction commands are exactly the same as Linux GPU. - -- Linux DCU -Running on a DCU device requires setting the environment variable `export HIP_VISIBLE_DEVICES=0,1,2,3`, and the rest of the training and evaluation prediction commands are exactly the same as the Linux GPU. - -# 3. Evaluation and Test - -## 3.1. Evaluation - -The trained model will be saved in `Global.save_model_dir`. When evaluation, you need to set `Architecture.Backbone.checkpoints` as your model directroy. The evaluation dataset can be set by modifying the `Eval.dataset.label_file_list` field in the `configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` file. - - -```bash -# GPU evaluation, Global.checkpoints is the weight to be tested -python3 tools/eval.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy -``` - -The following information will be printed such as precision, recall, hmean and so on. - -```py -[2022/08/09 07:59:28] ppocr INFO: metric eval *************** -[2022/08/09 07:59:28] ppocr INFO: precision:0.697476609016161 -[2022/08/09 07:59:28] ppocr INFO: recall:0.8861671469740634 -[2022/08/09 07:59:28] ppocr INFO: hmean:0.7805806758686339 -[2022/08/09 07:59:28] ppocr INFO: fps:17.367364606899105 -``` - - -## 3.2. Test - -Using the model trained by PaddleOCR, we can quickly get prediction through the following script. - -The default prediction image is stored in `Global.infer_img`, and the trained model weight is specified via `-o Global.checkpoints`. - -According to the `Global.save_model_dir` and `save_epoch_step` fields set in the configuration file, the following parameters will be saved. - - -``` -output/ser_vi_layoutxlm_xfund_zh/ -├── best_accuracy - ├── metric.states - ├── model_config.json - ├── model_state.pdparams -├── best_accuracy.pdopt -├── config.yml -├── train.log -├── latest - ├── metric.states - ├── model_config.json - ├── model_state.pdparams -├── latest.pdopt -``` - -Among them, best_accuracy.* is the best model on the evaluation set; latest.* is the model of the last epoch. - -The configuration file for prediction must be consistent with the training file. If you finish the training process using `python3 tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml`. You can use the following command for prediction. - - -```bash -python3 tools/infer_kie_token_ser.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.infer_img=./ppstructure/docs/kie/input/zh_val_42.jpg -``` - -The output image is as follows, which is also saved in `Global.save_res_path`. - - -
- -
- -During the prediction process, the detection and recognition model of PP-OCRv3 will be loaded by default for information extraction of OCR. If you want to load the OCR results obtained in advance, you can use the following method to predict, and specify `Global.infer_img` as the annotation file, which contains the image path and OCR information, and specifies `Global.infer_mode` as False, indicating that the OCR inference engine is not used at this time. - -```bash -python3 tools/infer_kie_token_ser.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.infer_img=./train_data/XFUND/zh_val/val.json Global.infer_mode=False -``` - -For the above image, if information extraction is performed using the labeled OCR results, the prediction results are as follows. - -
- -
- -It can be seen that part of the detection information is more accurate, but the overall information extraction results are basically the same. - -In RE model prediction, the SER model result needs to be given first, so the configuration file and model weight of SER need to be loaded at the same time, as shown in the following example. - -```bash -python3 ./tools/infer_kie_token_ser_re.py \ - -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \ - -o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_udml_xfund_zh/best_accuracy/ \ - Global.infer_img=./train_data/XFUND/zh_val/image/ \ - -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \ - -o_ser Architecture.Backbone.checkpoints=pretrain_models/ \ - ser_vi_layoutxlm_udml_xfund_zh/best_accuracy/ -``` - -The result is as follows. - -
- -
- - -If you want to load the OCR results obtained in advance, you can use the following method to predict, and specify `Global.infer_img` as the annotation file, which contains the image path and OCR information, and specifies `Global.infer_mode` as False, indicating that the OCR inference engine is not used at this time. - -```bash -python3 ./tools/infer_kie_token_ser_re.py \ - -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \ - -o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_udml_xfund_zh/best_accuracy/ \ - Global.infer_img=./train_data/XFUND/zh_val/val.json \ - Global.infer_mode=False \ - -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \ - -o_ser Architecture.Backbone.checkpoints=pretrain_models/ser_vi_layoutxlm_udml_xfund_zh/best_accuracy/ -``` - -`c_ser` denotes SER configurations file, `o_ser` denotes the SER model configurations that will override corresponding content in the file. - - -The result is as follows. - -
- -
- - -It can be seen that the re prediction results directly using the annotated OCR results are more accurate. - - -# 4. Model inference - - -## 4.1 Export the model - -The inference model (the model saved by `paddle.jit.save`) is generally a solidified model saved after the model training is completed, and is mostly used to give prediction in deployment. - -The model saved during the training process is the checkpoints model, which saves the parameters of the model and is mostly used to resume training. - -Compared with the checkpoints model, the inference model will additionally save the structural information of the model. Therefore, it is easier to deploy because the model structure and model parameters are already solidified in the inference model file, and is suitable for integration with actual systems. - -The SER model can be converted to the inference model using the following command. - - -```bash -# -c Set the training algorithm yml configuration file. -# -o Set optional parameters. -# Architecture.Backbone.checkpoints Set the training model address. -# Global.save_inference_dir Set the address where the converted model will be saved. -python3 tools/export_model.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.save_inference_dir=./inference/ser_vi_layoutxlm -``` - -After the conversion is successful, there are three files in the model save directory: - -``` -inference/ser_vi_layoutxlm/ - ├── inference.pdiparams # The parameter file of recognition inference model - ├── inference.pdiparams.info # The parameter information of recognition inference model, which can be ignored - └── inference.pdmodel # The program file of recognition -``` - -The RE model can be converted to the inference model using the following command. - - -```bash -# -c Set the training algorithm yml configuration file. -# -o Set optional parameters. -# Architecture.Backbone.checkpoints Set the training model address. -# Global.save_inference_dir Set the address where the converted model will be saved. -python3 tools/export_model.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/re_vi_layoutxlm_xfund_zh/best_accuracy Global.save_inference_dir=./inference/re_vi_layoutxlm -``` - -After the conversion is successful, there are three files in the model save directory: - -``` -inference/re_vi_layoutxlm/ - ├── inference.pdiparams # The parameter file of recognition inference model - ├── inference.pdiparams.info # The parameter information of recognition inference model, which can be ignored - └── inference.pdmodel # The program file of recognition -``` -## 4.2 Model inference - -The VI layoutxlm model performs reasoning based on the ser task, and can execute the following commands: - - -Using the following command to infer the VI-LayoutXLM SER model. - -```bash -cd ppstructure -python3 kie/predict_kie_token_ser.py \ - --kie_algorithm=LayoutXLM \ - --ser_model_dir=../inference/ser_vi_layoutxlm \ - --image_dir=./docs/kie/input/zh_val_42.jpg \ - --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \ - --vis_font_path=../doc/fonts/simfang.ttf \ - --ocr_order_method="tb-yx" -``` - -The visualized result will be saved in `./output`, which is shown as follows. - -
- -
- -Using the following command to infer the VI-LayoutXLM RE model. - -```bash -cd ppstructure -python3 kie/predict_kie_token_ser_re.py \ - --kie_algorithm=LayoutXLM \ - --re_model_dir=../inference/re_vi_layoutxlm \ - --ser_model_dir=../inference/ser_vi_layoutxlm \ - --use_visual_backbone=False \ - --image_dir=./docs/kie/input/zh_val_42.jpg \ - --ser_dict_path=../train_data/XFUND/class_list_xfun.txt \ - --vis_font_path=../doc/fonts/simfang.ttf \ - --ocr_order_method="tb-yx" -``` - -The visualized result will be saved in `./output`, which is shown as follows. - -
- -
- -# 5. FAQ - -Q1: After the training model is transferred to the inference model, the prediction effect is inconsistent? - -**A**:The problems are mostly caused by inconsistent preprocessing and postprocessing parameters when the trained model predicts and the preprocessing and postprocessing parameters when the inference model predicts. You can compare whether there are differences in preprocessing, postprocessing, and prediction in the configuration files used for training. diff --git a/doc/doc_en/knowledge_distillation_en.md b/doc/doc_en/knowledge_distillation_en.md deleted file mode 100755 index 334a8c4ba7..0000000000 --- a/doc/doc_en/knowledge_distillation_en.md +++ /dev/null @@ -1,637 +0,0 @@ - -# Knowledge Distillation - -+ [Knowledge Distillation](#0) - + [1. Introduction](#1) - - [1.1 Introduction to Knowledge Distillation](#11) - - [1.2 Introduction to PaddleOCR Knowledge Distillation](#12) - + [2. Configuration File Analysis](#2) - + [2.1 Recognition Model Configuration File Analysis](#21) - - [2.1.1 Model Structure](#211) - - [2.1.2 Loss Function ](#212) - - [2.1.3 Post-processing](#213) - - [2.1.4 Metric Calculation](#214) - - [2.1.5 Fine-tuning Distillation Model](#215) - + [2.2 Detection Model Configuration File Analysis](#22) - - [2.2.1 Model Structure](#221) - - [2.2.2 Loss Function](#222) - - [2.2.3 Post-processing](#223) - - [2.2.4 Metric Calculation](#224) - - [2.2.5 Fine-tuning Distillation Model](#225) - - - - -## 1. Introduction - -### 1.1 Introduction to Knowledge Distillation - -In recent years, deep neural networks have been proved to be an extremely effective method for solving problems in the fields of computer vision and natural language processing. -By constructing a suitable neural network and training it, the performance metrics of the final network model will basically exceed the traditional algorithm. -When the amount of data is large enough, increasing the amount of parameters by constructing a reasonable network model can significantly improve the performance of the model, -but this brings about the problem of a sharp increase in the complexity of the model. Large models are more expensive to use in actual scenarios. -Deep neural networks generally have more parameter redundancy. At present, there are several main methods to compress the model and reduce the amount of its parameters. -Such as pruning, quantification, knowledge distillation, etc., where knowledge distillation refers to the use of teacher models to guide student models to learn specific tasks, -to ensure that the small model obtains a relatively large performance improvement under the condition of unchanged parameters. -In addition, in the knowledge distillation task, a mutual learning model training method was also derived. -The paper [Deep Mutual Learning](https://arxiv.org/abs/1706.00384) pointed out that using two identical models to supervise each other during the training process can achieve better results than a single model training. - - -### 1.2 Introduction to PaddleOCR Knowledge Distillation - -Whether it is a large model distilling a small model, or a small model learning from each other and updating parameters, -they are essentially the output between different models or mutual supervision between feature maps. -The only difference is (1) whether the model requires fixed parameters. (2) Whether the model needs to be loaded with a pre-trained model. -For the case where a large model distills a small model, the large model generally needs to load the pre-trained model and fix the parameters. -For the situation where small models distill each other, the small models generally do not load the pre-trained model, and the parameters are also in a learnable state. - -In the task of knowledge distillation, it is not only the distillation between two models, but also the situation where multiple models learn from each other. -Therefore, in the knowledge distillation code framework, it is also necessary to support this type of distillation method. - -The algorithm of knowledge distillation is integrated in PaddleOCR. Specifically, it has the following main features: -- It supports mutual learning of any network, and does not require the sub-network structure to be completely consistent or to have a pre-trained model. At the same time, there is no limit to the number of sub-networks, just add it in the configuration file. -- Support arbitrarily configuring the loss function through the configuration file, not only can use a certain loss, but also a combination of multiple losses. -- Support all model-related environments such as knowledge distillation training, prediction, evaluation, and export, which is convenient for use and deployment. - -Through knowledge distillation, in the common Chinese and English text recognition task, without adding any time-consuming prediction, -the accuracy of the model can be improved by more than 3%. Combining the learning rate adjustment strategy and the model structure fine-tuning strategy, -the final improvement is more than 5%. - - -## 2. Configuration File Analysis - -In the process of knowledge distillation training, there is no change in data preprocessing, optimizer, learning rate, and some global attributes. -The configuration files of the model structure, loss function, post-processing, metric calculation and other modules need to be fine-tuned. - -The following takes the knowledge distillation configuration file for recognition and detection as an example to analyze the training and configuration of knowledge distillation. - - -### 2.1 Recognition Model Configuration File Analysis - -The configuration file is in [ch_PP-OCRv2_rec_distillation.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec_distillation.yml). - - -#### 2.1.1 Model Structure - -In the knowledge distillation task, the model structure configuration is as follows. - -```yaml -Architecture: - model_type: &model_type "rec" # Model category, recognition, detection, etc. - name: DistillationModel # Structure name, in the distillation task, it is DistillationModel - algorithm: Distillation # Algorithm name - Models: # Model, including the configuration information of the subnet - Teacher: # The name of the subnet, it must include at least the `pretrained` and `freeze_params` parameters, and the other parameters are the construction parameters of the subnet - pretrained: # Does this sub-network need to load pre-training weights - freeze_params: false # Do you need fixed parameters - return_all_feats: true # Do you need to return all features, if it is False, only the final output is returned - model_type: *model_type # Model category - algorithm: SVTR # The algorithm name of the sub-network. The remaining parameters of the sub-network are consistent with the general model training configuration - Transform: - Backbone: - name: MobileNetV1Enhance - scale: 0.5 - last_conv_stride: [1, 2] - last_pool_type: avg - Head: - name: MultiHead - head_list: - - CTCHead: - Neck: - name: svtr - dims: 64 - depth: 2 - hidden_dims: 120 - use_guide: True - Head: - fc_decay: 0.00001 - - SARHead: - enc_dim: 512 - max_text_length: *max_text_length - Student: # Another sub-network, here is a distillation example of DML, the two sub-networks have the same structure, and both need to learn parameters - pretrained: # The following parameters are the same as above - freeze_params: false - return_all_feats: true - model_type: *model_type - algorithm: SVTR - Transform: - Backbone: - name: MobileNetV1Enhance - scale: 0.5 - last_conv_stride: [1, 2] - last_pool_type: avg - Head: - name: MultiHead - head_list: - - CTCHead: - Neck: - name: svtr - dims: 64 - depth: 2 - hidden_dims: 120 - use_guide: True - Head: - fc_decay: 0.00001 - - SARHead: - enc_dim: 512 - max_text_length: *max_text_length -``` - -If you want to add more sub-networks for training, you can also add the corresponding fields in the configuration file according to the way of adding `Student` and `Teacher`. -For example, if you want 3 models to supervise each other and train together, then `Architecture` can be written in the following format. - -```yaml -Architecture: - model_type: &model_type "rec" - name: DistillationModel - algorithm: Distillation - Models: - Teacher: - pretrained: - freeze_params: false - return_all_feats: true - model_type: *model_type - algorithm: SVTR - Transform: - Backbone: - name: MobileNetV1Enhance - scale: 0.5 - last_conv_stride: [1, 2] - last_pool_type: avg - Head: - name: MultiHead - head_list: - - CTCHead: - Neck: - name: svtr - dims: 64 - depth: 2 - hidden_dims: 120 - use_guide: True - Head: - fc_decay: 0.00001 - - SARHead: - enc_dim: 512 - max_text_length: *max_text_length - Student: - pretrained: - freeze_params: false - return_all_feats: true - model_type: *model_type - algorithm: SVTR - Transform: - Backbone: - name: MobileNetV1Enhance - scale: 0.5 - last_conv_stride: [1, 2] - last_pool_type: avg - Head: - name: MultiHead - head_list: - - CTCHead: - Neck: - name: svtr - dims: 64 - depth: 2 - hidden_dims: 120 - use_guide: True - Head: - fc_decay: 0.00001 - - SARHead: - enc_dim: 512 - max_text_length: *max_text_length - Student2: - pretrained: - freeze_params: false - return_all_feats: true - model_type: *model_type - algorithm: SVTR - Transform: - Backbone: - name: MobileNetV1Enhance - scale: 0.5 - last_conv_stride: [1, 2] - last_pool_type: avg - Head: - name: MultiHead - head_list: - - CTCHead: - Neck: - name: svtr - dims: 64 - depth: 2 - hidden_dims: 120 - use_guide: True - Head: - fc_decay: 0.00001 - - SARHead: - enc_dim: 512 - max_text_length: *max_text_length -``` -``` - -When the model is finally trained, it contains 3 sub-networks: `Teacher`, `Student`, `Student2`. - -The specific implementation code of the `DistillationModel` class can refer to [distillation_model.py](../../ppocr/modeling/architectures/distillation_model.py). -The final model output is a dictionary, the key is the name of all the sub-networks, for example, here are `Student` and `Teacher`, and the value is the output of the corresponding sub-network, -which can be `Tensor` (only the last layer of the network is returned) and `dict` (also returns the characteristic information in the middle). -In the recognition task, in order to add more loss functions and ensure the scalability of the distillation method, the output of each sub-network is saved as a `dict`, which contains the sub-module output. -Take the recognition model as an example. The output result of each sub-network is `dict`, the key contains `backbone_out`, `neck_out`, `head_out`, and `value` is the tensor of the corresponding module. Finally, for the above configuration file, `DistillationModel` The output format is as follows. - -```json -{ - "Teacher": { - "backbone_out": tensor, - "neck_out": tensor, - "head_out": tensor, - }, - "Student": { - "backbone_out": tensor, - "neck_out": tensor, - "head_out": tensor, - } -} -``` - - -#### 2.1.2 Loss Function - -In the knowledge distillation task, the loss function configuration is as follows. - -```yaml -Loss: - name: CombinedLoss # Loss function name - loss_config_list: # List of loss function configuration files, mandatory functions for CombinedLoss - - DistillationCTCLoss: # CTC loss function based on distillation, inherited from standard CTC loss - weight: 1.0 # The weight of the loss function. In loss_config_list, each loss function must include this field - model_name_list: ["Student", "Teacher"] # For the prediction results of the distillation model, extract the output of these two sub-networks and calculate the CTC loss with gt - key: head_out # In the sub-network output dict, take the corresponding tensor - - DistillationDMLLoss: # DML loss function, inherited from the standard DMLLoss - weight: 1.0 - act: "softmax" # Activation function, use it to process the input, can be softmax, sigmoid or None, the default is None - model_name_pairs: # The subnet name pair used to calculate DML loss. If you want to calculate the DML loss of other subnets, you can continue to add it below the list - - ["Student", "Teacher"] - key: head_out - multi_head: True # whether to use mult_head - dis_head: ctc # assign the head name to calculate loss - name: dml_ctc # prefix name of the loss - - DistillationDMLLoss: # DML loss function, inherited from the standard DMLLoss - weight: 0.5 - act: "softmax" # Activation function, use it to process the input, can be softmax, sigmoid or None, the default is None - model_name_pairs: # The subnet name pair used to calculate DML loss. If you want to calculate the DML loss of other subnets, you can continue to add it below the list - - ["Student", "Teacher"] - key: head_out - multi_head: True # whether to use mult_head - dis_head: sar # assign the head name to calculate loss - name: dml_sar # prefix name of the loss - - DistillationDistanceLoss: # Distilled distance loss function - weight: 1.0 - mode: "l2" # Support l1, l2 or smooth_l1 - model_name_pairs: # Calculate the distance loss of the subnet name pair - - ["Student", "Teacher"] - key: backbone_out - - DistillationSARLoss: # SAR loss function based on distillation, inherited from standard SAR loss - weight: 1.0 # The weight of the loss function. In loss_config_list, each loss function must include this field - model_name_list: ["Student", "Teacher"] # For the prediction results of the distillation model, extract the output of these two sub-networks and calculate the SAR loss with gt - key: head_out # In the sub-network output dict, take the corresponding tensor - multi_head: True # whether it is multi-head or not, if true, SAR branch is used to calculate the loss -``` - -Among the above loss functions, all distillation loss functions are inherited from the standard loss function class. -The main functions are: Analyze the output of the distillation model, find the intermediate node (tensor) used to calculate the loss, -and then use the standard loss function class to calculate. - -Taking the above configuration as an example, the final distillation training loss function contains the following five parts. - -- CTC branch of the final output `head_out` for `Student` and `Teacher` calculates the CTC loss with gt (loss weight equals 1.0). Here, because both sub-networks need to update the parameters, both of them need to calculate the loss with gt. -- SAR branch of the final output `head_out` for `Student` and `Teacher` calculates the SAR loss with gt (loss weight equals 1.0). Here, because both sub-networks need to update the parameters, both of them need to calculate the loss with gt. -- DML loss between CTC branch of `Student` and `Teacher`'s final output `head_out` (loss weight equals 1.0). -- DML loss between SAR branch of `Student` and `Teacher`'s final output `head_out` (loss weight equals 0.5). -- L2 loss between `Student` and `Teacher`'s backbone network output `backbone_out` (loss weight equals 1.0). - -For more specific implementation of `CombinedLoss`, please refer to: [combined_loss.py](../../ppocr/losses/combined_loss.py#L23). -For more specific implementations of distillation loss functions such as `DistillationCTCLoss`, please refer to [distillation_loss.py](../../ppocr/losses/distillation_loss.py) - - - -#### 2.1.3 Post-processing - -In the knowledge distillation task, the post-processing configuration is as follows. - -```yaml -PostProcess: - name: DistillationCTCLabelDecode # CTC decoding post-processing of distillation tasks, inherited from the standard CTCLabelDecode class - model_name: ["Student", "Teacher"] # For the prediction results of the distillation model, extract the outputs of these two sub-networks and decode them - key: head_out # Take the corresponding tensor in the subnet output dict - multi_head: True # whether it is multi-head or not, if true, CTC branch is used to calculate the loss -``` - -Taking the above configuration as an example, the CTC decoding output of the two sub-networks `Student` and `Teahcer` will be calculated at the same time. -Among them, `key` is the name of the subnet, and `value` is the list of subnets. - -For more specific implementation of `DistillationCTCLabelDecode`, please refer to: [rec_postprocess.py](../../ppocr/postprocess/rec_postprocess.py#L128) - - - -#### 2.1.4 Metric Calculation - -In the knowledge distillation task, the metric calculation configuration is as follows. - -```yaml -Metric: - name: DistillationMetric # CTC decoding post-processing of distillation tasks, inherited from the standard CTCLabelDecode class - base_metric_name: RecMetric # The base class of indicator calculation. For the output of the model, the indicator will be calculated based on this class - main_indicator: acc # The name of the indicator - key: "Student" # Select the main_indicator of this subnet as the criterion for saving the best model - ignore_space: False # whether to ignore space during evaulation -``` - -Taking the above configuration as an example, the accuracy metric of the `Student` subnet will be used as the judgment metric for saving the best model. -At the same time, the accuracy metric of all subnets will be printed out in the log. - -For more specific implementation of `DistillationMetric`, please refer to: [distillation_metric.py](../../ppocr/metrics/distillation_metric.py#L24). - - - -#### 2.1.5 Fine-tuning Distillation Model - -There are two ways to fine-tune the recognition distillation task. - -1. Fine-tuning based on knowledge distillation: this situation is relatively simple, download the pre-trained model. Then configure the pre-training model path and your own data path in [ch_PP-OCRv2_rec_distillation.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml) to perform fine-tuning training of the model. -2. Do not use knowledge distillation in fine-tuning: In this case, you need to first extract the student model parameters from the pre-training model. The specific steps are as follows. - -- First download the pre-trained model and unzip it. -```shell -wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_train.tar -tar -xf ch_PP-OCRv3_rec_train.tar -``` - -- Then use python to extract the student model parameters - -```python -import paddle -# Load the pre-trained model -all_params = paddle.load("ch_PP-OCRv3_rec_train/best_accuracy.pdparams") -# View the keys of the weight parameter -print(all_params.keys()) -# Weight extraction of student model -s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key} -# View the keys of the weight parameters of the student model -print(s_params.keys()) -# Save weight parameters -paddle.save(s_params, "ch_PP-OCRv3_rec_train/student.pdparams") -``` - -After the extraction is complete, use [ch_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec.yml) to modify the path of the pre-trained model (the path of the exported `student.pdparams` model) and your own data path to fine-tune the model. - - -### 2.2 Detection Model Configuration File Analysis - -The configuration file of the detection model distillation is in the ```PaddleOCR/configs/det/ch_PP-OCRv3/``` directory, which contains three distillation configuration files: - -- ```ch_PP-OCRv3_det_cml.yml```, Use one large model to distill two small models, and the two small models learn from each other -- ```ch_PP-OCRv3_det_dml.yml```, Method of mutual distillation of two student models - - -#### 2.2.1 Model Structure - -In the knowledge distillation task, the model structure configuration is as follows: -``` -Architecture: - name: DistillationModel # Structure name, in the distillation task, it is DistillationModel - algorithm: Distillation # Algorithm name - Models: # Model, including the configuration information of the subnet - Student: # The name of the subnet, it must include at least the `pretrained` and `freeze_params` parameters, and the other parameters are the construction parameters of the subnet - pretrained: ./pretrain_models/MobileNetV3_large_x0_5_pretrained # Does this sub-network need to load pre-training weights - freeze_params: false # Do you need fixed parameters - return_all_feats: false # Do you need to return all features, if it is False, only the final output is returned - model_type: det - algorithm: DB - Backbone: - name: ResNet - in_channels: 3 - layers: 50 - Neck: - name: LKPAN - out_channels: 256 - Head: - name: DBHead - kernel_list: [7,2,2] - k: 50 - Teacher: # Another sub-network, here is a distillation example of a large model distill a small model - pretrained: ./pretrain_models/ch_ppocr_server_v2.0_det_train/best_accuracy - return_all_feats: false - model_type: det - algorithm: DB - Transform: - Backbone: - name: ResNet - in_channels: 3 - layers: 50 - Neck: - name: LKPAN - out_channels: 256 - Head: - name: DBHead - kernel_list: [7,2,2] - k: 50 - -``` -If DML is used, that is, the method of two small models learning from each other, the Teacher network structure in the above configuration file needs to be set to the same configuration as the Student model. -Refer to the configuration file for details. [ch_PP-OCRv3_det_dml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_dml.yml) - - -The following describes the configuration file parameters [ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml): - -``` -Architecture: - name: DistillationModel - algorithm: Distillation - model_type: det - Models: - Teacher: # Teacher model configuration of CML distillation - pretrained: ./pretrain_models/ch_ppocr_server_v2.0_det_train/best_accuracy - freeze_params: true # Teacher does not train - return_all_feats: false - model_type: det - algorithm: DB - Transform: - Backbone: - name: ResNet - in_channels: 3 - layers: 50 - Neck: - name: LKPAN - out_channels: 256 - Head: - name: DBHead - kernel_list: [7,2,2] - k: 50 - Student: # Student model configuration for CML distillation - pretrained: ./pretrain_models/MobileNetV3_large_x0_5_pretrained - freeze_params: false - return_all_feats: false - model_type: det - algorithm: DB - Backbone: - name: MobileNetV3 - scale: 0.5 - model_name: large - disable_se: true - Neck: - name: RSEFPN - out_channels: 96 - shortcut: True - Head: - name: DBHead - k: 50 - Student2: # Student2 model configuration for CML distillation - pretrained: ./pretrain_models/MobileNetV3_large_x0_5_pretrained - freeze_params: false - return_all_feats: false - model_type: det - algorithm: DB - Transform: - Backbone: - name: MobileNetV3 - scale: 0.5 - model_name: large - disable_se: true - Neck: - name: RSEFPN - out_channels: 96 - shortcut: True - Head: - name: DBHead - k: 50 - -``` - -The specific implementation code of the distillation model `DistillationModel` class can refer to [distillation_model.py](../../ppocr/modeling/architectures/distillation_model.py). - -The final model output is a dictionary, the key is the name of all the sub-networks, for example, here are `Student` and `Teacher`, and the value is the output of the corresponding sub-network, -which can be `Tensor` (only the last layer of the network is returned) and `dict` (also returns the characteristic information in the middle). - -In the distillation task, in order to facilitate the addition of the distillation loss function, the output of each network is saved as a `dict`, which contains the sub-module output. -The key contains `backbone_out`, `neck_out`, `head_out`, and `value` is the tensor of the corresponding module. Finally, for the above configuration file, the output format of `DistillationModel` is as follows. - -```json -{ - "Teacher": { - "backbone_out": tensor, - "neck_out": tensor, - "head_out": tensor, - }, - "Student": { - "backbone_out": tensor, - "neck_out": tensor, - "head_out": tensor, - } -} -``` - - -#### 2.2.2 Loss Function -The distillation loss function configuration(`ch_PP-OCRv3_det_cml.yml`) is shown below. -```yaml -Loss: - name: CombinedLoss - loss_config_list: - - DistillationDilaDBLoss: - weight: 1.0 - model_name_pairs: - - ["Student", "Teacher"] - - ["Student2", "Teacher"] # 1. Calculate the loss of two Student and Teacher - key: maps - balance_loss: true - main_loss_type: DiceLoss - alpha: 5 - beta: 10 - ohem_ratio: 3 - - DistillationDMLLoss: # 2. Add to calculate the loss between two students - model_name_pairs: - - ["Student", "Student2"] - maps_name: "thrink_maps" - weight: 1.0 - # act: None - key: maps - - DistillationDBLoss: - weight: 1.0 - model_name_list: ["Student", "Student2"] # 3. Calculate the loss between two students and GT - balance_loss: true - main_loss_type: DiceLoss - alpha: 5 - beta: 10 - ohem_ratio: 3 -``` - -For more specific implementation of `DistillationDilaDBLoss`, please refer to: [distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.4/ppocr/losses/distillation_loss.py#L185). -For more specific implementations of distillation loss functions such as `DistillationDBLoss`, please refer to: [distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/04c44974b13163450dfb6bd2c327863f8a194b3c/ppocr/losses/distillation_loss.py?_pjax=%23js-repo-pjax-container%2C%20div%5Bitemtype%3D%22http%3A%2F%2Fschema.org%2FSoftwareSourceCode%22%5D%20main%2C%20%5Bdata-pjax-container%5D#L148) - - -#### 2.2.3 Post-processing - -In the task of detecting knowledge distillation, the post-processing configuration of detecting distillation is as follows. - -```yaml -PostProcess: - name: DistillationDBPostProcess # The post-processing of the DB detection distillation task, inherited from the standard DBPostProcess class - model_name: ["Student", "Student2", "Teacher"] # Extract the output of multiple sub-networks and decode them. The network that does not require post-processing is not set in model_name - thresh: 0.3 - box_thresh: 0.6 - max_candidates: 1000 - unclip_ratio: 1.5 -``` - -Taking the above configuration as an example, the output of the three subnets `Student`, `Student2` and `Teacher` will be calculated at the same time for post-processing calculations. -Since there are multiple inputs, there are also multiple outputs returned by post-processing. -For a more specific implementation of `DistillationDBPostProcess`, please refer to: [db_postprocess.py](../../ppocr/postprocess/db_postprocess.py#L195) - - -#### 2.2.4 Metric Calculation -In the knowledge distillation task, the metric calculation configuration is as follows. -```yaml -Metric: - name: DistillationMetric - base_metric_name: DetMetric - main_indicator: hmean - key: "Student" -``` - -Since distillation needs to include multiple networks, only one network metrics needs to be calculated when calculating the metrics. -The `key` field is set to `Student`, it means that only the metrics of the `Student` network is calculated. -Model Structure - - -#### 2.2.5 Fine-tuning Distillation Model - -There are three ways to fine-tune the detection distillation task: -- `ch_PP-OCRv3_det_distill.yml`, The teacher model is set to the model provided by PaddleOCR or the large model you have trained. -- `ch_PP-OCRv3_det_cml.yml`, Use cml distillation. Similarly, the Teacher model is set to the model provided by PaddleOCR or the large model you have trained. -- `ch_PP-OCRv3_det_dml.yml`, Distillation using DML. The method of mutual distillation of the two Student models has an accuracy improvement of about 1.7% on the data set used by PaddleOCR. - -In fine-tune, you need to set the pre-trained model to be loaded in the `pretrained` parameter of the network structure. - -In terms of accuracy improvement, `cml` > `dml` > `distill`. When the amount of data is insufficient or the accuracy of the teacher model is similar to that of the student, this conclusion may change. - -In addition, since the distillation pre-training model provided by PaddleOCR contains multiple model parameters, if you want to extract the parameters of the student model, you can refer to the following code: -```sh -# Download the parameters of the distillation training model -wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar -``` - -```python -import paddle -# Load the pre-trained model -all_params = paddle.load("ch_PP-OCRv3_det_distill_train/best_accuracy.pdparams") -# View the keys of the weight parameter -print(all_params.keys()) -# Extract the weights of the student model -s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key} -# View the keys of the weight parameters of the student model -print(s_params.keys()) -# Save -paddle.save(s_params, "ch_PP-OCRv3_det_distill_train/student.pdparams") -``` - -Finally, the parameters of the student model will be saved in `ch_PP-OCRv3_det_distill_train/student.pdparams` for the fine-tune of the model. diff --git a/doc/doc_en/logging_en.md b/doc/doc_en/logging_en.md deleted file mode 100644 index 10c400c084..0000000000 --- a/doc/doc_en/logging_en.md +++ /dev/null @@ -1,61 +0,0 @@ -## Logging metrics and models - -PaddleOCR comes with two metric logging tools integrated directly into the training API: [VisualDL](https://readthedocs.org/projects/visualdl/) and [Weights & Biases](https://docs.wandb.ai/). - -### VisualDL -VisualDL is a visualization analysis tool of PaddlePaddle. The integration allows all training metrics to be logged to a VisualDL dashboard. To use it, add the following line to the `Global` section of the config yaml file - - -``` -Global: - use_visualdl: True -``` - -To see the visualizations run the following command in your terminal - -```shell -visualdl --logdir -``` - -Now open `localhost:8040` in your browser of choice! - -### Weights & Biases -W&B is a MLOps tool that can be used for experiment tracking, dataset/model versioning, visualizing results and collaborating with colleagues. A W&B logger is integrated directly into PaddleOCR and to use it, first you need to install the `wandb` sdk and login to your wandb account. - -```shell -pip install wandb -wandb login -``` - -If you do not have a wandb account, you can make one [here](https://wandb.ai/site). - -To visualize and track your model training add the following flag to your config yaml file under the `Global` section - - -``` -Global: - use_wandb: True -``` - -To add more arguments to the `WandbLogger` listed [here](./config_en.md) add the header `wandb` to the yaml file and add the arguments under it - - -``` -wandb: - project: my_project - entity: my_team -``` - -These config variables from the yaml file are used to instantiate the `WandbLogger` object with the project name, entity name (the logged in user by default), directory to store metadata (`./wandb` by default) and more. During the training process, the `log_metrics` function is called to log training and evaluation metrics at the training and evaluation steps respectively from the rank 0 process only. - -At every model saving step, the WandbLogger, logs the model using the `log_model` function along with relavant metadata and tags showing the epoch in which the model is saved, the model is best or not and so on. - -All the logging mentioned above is integrated into the `program.train` function and will generate dashboards like this - - -![W&B Dashboard](../imgs_en/wandb_metrics.png) - -![W&B Models](../imgs_en/wandb_models.png) - -For more advanced usage to log images, audios, videos or any other form of data, you can use `WandbLogger().run.log`. More examples on how to log different kinds of data are available [here](https://docs.wandb.ai/examples). - -To view the dashboard, the link to the dashboard is printed to the console at the beginning and end of every training job and you can also access it by logging into your W&B account on your browser. - -### Using Multiple Loggers -Both VisualDL and W&B can also be used simultaneously by just setting both the aforementioned flags to True. diff --git a/doc/doc_en/models_en.md b/doc/doc_en/models_en.md deleted file mode 100644 index d60ca2a332..0000000000 --- a/doc/doc_en/models_en.md +++ /dev/null @@ -1,46 +0,0 @@ -# PP-OCR Model Zoo -The PP-OCR model zoo section explains some basic concepts of the OCR model and how to quickly use the models in the PP-OCR model library. - -This section contains two parts. Firstly, [PP-OCR Model Download](./models_list_en.md) explains the concept of PP-OCR model types and provides links to download all models. The next [Python Inference for PP-OCR Model Zoo](./inference_ppocr_en.md) is an introduction to the use of the PP-OCR model library, which can quickly utilize the rich model library models to obtain test results through the Python inference engine. - ------- - -Let's first understand some basic concepts. - -- [Introduction to OCR](#introduction-to-ocr) - * [Basic Concepts of OCR Detection Model](#basic-concepts-of-ocr-detection-model) - * [Basic Concepts of OCR Recognition Model](#basic-concepts-of-ocr-recognition-model) - * [PP-OCR Model](#pp-ocr-model) - - -## 1. Introduction to OCR - -This section briefly introduces the basic concepts of OCR detection model and recognition model, and introduces PaddleOCR's PP-OCR model. - -OCR (Optical Character Recognition, Optical Character Recognition) is currently the general term for text recognition. It is not limited to document or book text recognition, but also includes recognizing text in natural scenes. It can also be called STR (Scene Text Recognition). - -OCR text recognition generally includes two parts, text detection and text recognition. The text detection module first uses detection algorithms to detect text lines in the image. And then the recognition algorithm to identify the specific text in the text line. - - -### 1.1 Basic Concepts of OCR Detection Model - -Text detection can locate the text area in the image, and then usually mark the word or text line in the form of a bounding box. Traditional text detection algorithms mostly extract features manually, which are characterized by fast speed and good effect in simple scenes, but the effect will be greatly reduced when faced with natural scenes. Currently, deep learning methods are mostly used. - -Text detection algorithms based on deep learning can be roughly divided into the following categories: -1. Method based on target detection. Generally, after the text box is predicted, the final text box is filtered through NMS, which is mostly four-point text box, which is not ideal for curved text scenes. Typical algorithms are methods such as EAST and Text Box. -2. Method based on text segmentation. The text line is regarded as the segmentation target, and then the external text box is constructed through the segmentation result, which can handle curved text, and the effect is not ideal for the text cross scene problem. Typical algorithms are DB, PSENet and other methods. -3. Hybrid target detection and segmentation method. - - -### 1.2 Basic Concepts of OCR Recognition Model - -The input of the OCR recognition algorithm is generally text lines images which has less background information, and the text information occupies the main part. The recognition algorithm can be divided into two types of algorithms: -1. CTC-based method. The text prediction module of the recognition algorithm is based on CTC, and the commonly used algorithm combination is CNN+RNN+CTC. There are also some algorithms that try to add transformer modules to the network and so on. -2. Attention-based method. The text prediction module of the recognition algorithm is based on Attention, and the commonly used algorithm combination is CNN+RNN+Attention. - - -### 1.3 PP-OCR Model - -PaddleOCR integrates many OCR algorithms, text detection algorithms include DB, EAST, SAST, etc., text recognition algorithms include CRNN, RARE, StarNet, Rosetta, SRN and other algorithms. - -Among them, PaddleOCR has released the PP-OCR series model for the general OCR in Chinese and English natural scenes. The PP-OCR model is composed of the DB+CRNN algorithm. It uses massive Chinese data training and model tuning methods to have high text detection and recognition capabilities in Chinese scenes. And PaddleOCR has launched a high-precision and ultra-lightweight PP-OCRv2 model. The detection model is only 3M, and the recognition model is only 8.5M. Using [PaddleSlim](https://github.com/PaddlePaddle/PaddleSlim)'s model quantification method, the detection model can be compressed to 0.8M without reducing the accuracy. The recognition is compressed to 3M, which is more suitable for mobile deployment scenarios. diff --git a/doc/doc_en/models_list_en.md b/doc/doc_en/models_list_en.md deleted file mode 100644 index a56f72d8d6..0000000000 --- a/doc/doc_en/models_list_en.md +++ /dev/null @@ -1,141 +0,0 @@ -# OCR Model List(V3, updated on 2022.4.28) -> **Note** -> 1. Compared with model v2, the 3rd version of the detection model has an improvement in accuracy, and the 2.1 version of the recognition model has optimizations in accuracy and speed with CPU. -> 2. Compared with [models 1.1](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/doc/doc_en/models_list_en.md), which are trained with static graph programming paradigm, models 2.0 or higher are the dynamic graph trained version and achieve close performance. -> 3. All models in this tutorial are from the PaddleOCR series, for more introduction to algorithms and models based on the public dataset, you can refer to [algorithm overview tutorial](./algorithm_overview_en.md). - -- [OCR Model List(V3, updated on 2022.4.28)]() - - [1. Text Detection Model](#1-text-detection-model) - - [1.1 Chinese Detection Model](#1.1) - - [2.2 English Detection Model](#1.2) - - [1.3 Multilingual Detection Model](#1.3) - - [2. Text Recognition Model](#2-text-recognition-model) - - [2.1 Chinese Recognition Model](#21-chinese-recognition-model) - - [2.2 English Recognition Model](#22-english-recognition-model) - - [2.3 Multilingual Recognition Model(Updating...)](#23-multilingual-recognition-modelupdating) - - [3. Text Angle Classification Model](#3-text-angle-classification-model) - - [4. Paddle-Lite Model](#4-paddle-lite-model) - -The downloadable models provided by PaddleOCR include the `inference model`, `trained model`, `pre-trained model` and `nb model`. The differences between the models are as follows: - -|model type|model format|description| -|--- | --- | --- | -|inference model|inference.pdmodel、inference.pdiparams|Used for inference based on Paddle inference engine,[detail](./inference_ppocr_en.md)| -|trained model, pre-trained model|\*.pdparams、\*.pdopt、\*.states |The checkpoints model saved in the training process, which stores the parameters of the model, is mostly used for model evaluation and continuous training.| -|nb model|\*.nb| Model optimized by Paddle-Lite, which is suitable for mobile-side deployment scenarios (Paddle-Lite is needed for nb model deployment). | - -The relationship of the above models is as follows. - -![](../imgs_en/model_prod_flow_en.png) - - -## 1. Text Detection Model - - - -### 1. Chinese Detection Model - -|model name|description|config|model size|download| -| --- | --- | --- | --- | --- | -|ch_PP-OCRv3_det_slim| [New] slim quantization with distillation lightweight model, supporting Chinese, English, multilingual text detection |[ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml)| 1.1M |[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_slim_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_slim_distill_train.tar) / [nb model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_slim_infer.nb)| -|ch_PP-OCRv3_det| [New] Original lightweight model, supporting Chinese, English, multilingual text detection |[ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml)| 3.8M |[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar)| -|ch_PP-OCRv2_det_slim| [New] slim quantization with distillation lightweight model, supporting Chinese, English, multilingual text detection|[ch_PP-OCRv2_det_cml.yml](../../configs/det/ch_PP-OCRv2/ch_PP-OCRv2_det_cml.yml)| 3.0M |[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_slim_quant_infer.tar)| -|ch_PP-OCRv2_det| [New] Original lightweight model, supporting Chinese, English, multilingual text detection|[ch_PP-OCRv2_det_cml.yml](../../configs/det/ch_PP-OCRv2/ch_PP-OCRv2_det_cml.yml)|3.0M|[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_distill_train.tar)| -|ch_ppocr_mobile_slim_v2.0_det|Slim pruned lightweight model, supporting Chinese, English, multilingual text detection|[ch_det_mv3_db_v2.0.yml](../../configs/det/ch_ppocr_v2.0/ch_det_mv3_db_v2.0.yml)|2.6M |[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/slim/ch_ppocr_mobile_v2.0_det_prune_infer.tar)| -|ch_ppocr_mobile_v2.0_det|Original lightweight model, supporting Chinese, English, multilingual text detection|[ch_det_mv3_db_v2.0.yml](../../configs/det/ch_ppocr_v2.0/ch_det_mv3_db_v2.0.yml)|3.0M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_train.tar)| -|ch_ppocr_server_v2.0_det|General model, which is larger than the lightweight model, but achieved better performance|[ch_det_res18_db_v2.0.yml](../../configs/det/ch_ppocr_v2.0/ch_det_res18_db_v2.0.yml)|47.0M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_det_train.tar)| - - - -### 1.2 English Detection Model - -|model name|description|config|model size|download| -| --- | --- | --- | --- | --- | -|en_PP-OCRv3_det_slim | [New] Slim quantization with distillation lightweight detection model, supporting English | [ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml) | 1.1M |[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_slim_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_slim_distill_train.tar) / [nb model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_slim_infer.nb) | -|en_PP-OCRv3_det | [New] Original lightweight detection model, supporting English |[ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml)| 3.8M | [inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_distill_train.tar) | - -* Note: English configuration file is the same as Chinese except for training data, here we only provide one configuration file. - - - -### 1.3 Multilingual Detection Model - -|model name|description|config|model size|download| -| --- | --- | --- | --- | --- | -| ml_PP-OCRv3_det_slim | [New] Slim quantization with distillation lightweight detection model, supporting English | [ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml) | 1.1M | [inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/Multilingual_PP-OCRv3_det_slim_infer.tar) / [trained model ](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/Multilingual_PP-OCRv3_det_slim_distill_train.tar) / [nb model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/Multilingual_PP-OCRv3_det_slim_infer.nb) | -| ml_PP-OCRv3_det |[New] Original lightweight detection model, supporting English | [ch_PP-OCRv3_det_cml.yml](../../configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml)| 3.8M | [inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/Multilingual_PP-OCRv3_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/Multilingual_PP-OCRv3_det_distill_train.tar) | - -* Note: English configuration file is the same as Chinese except for training data, here we only provide one configuration file. - - -## 2. Text Recognition Model - - -### 2.1 Chinese Recognition Model - -|model name|description|config|model size|download| -| --- | --- | --- | --- | --- | -|ch_PP-OCRv3_rec_slim | [New] Slim quantization with distillation lightweight model, supporting Chinese, English text recognition |[ch_PP-OCRv3_rec_distillation.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml)| 4.9M |[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_slim_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_slim_train.tar) / [nb model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_slim_infer.nb) | -|ch_PP-OCRv3_rec| [New] Original lightweight model, supporting Chinese, English, multilingual text recognition |[ch_PP-OCRv3_rec_distillation.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml)| 12.4M |[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_train.tar) | -|ch_PP-OCRv2_rec_slim| Slim quantization with distillation lightweight model, supporting Chinese, English text recognition|[ch_PP-OCRv2_rec.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml)| 9.0M |[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_slim_quant_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_slim_quant_train.tar) | -|ch_PP-OCRv2_rec| Original lightweight model, supporting Chinese, English, and multilingual text recognition |[ch_PP-OCRv2_rec_distillation.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec_distillation.yml)|8.5M|[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_train.tar) | -|ch_ppocr_mobile_slim_v2.0_rec|Slim pruned and quantized lightweight model, supporting Chinese, English and number recognition|[rec_chinese_lite_train_v2.0.yml](../../configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml)| 6.0M | [inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_slim_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_slim_train.tar) | -|ch_ppocr_mobile_v2.0_rec|Original lightweight model, supporting Chinese, English and number recognition|[rec_chinese_lite_train_v2.0.yml](../../configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml)|5.2M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_train.tar) / [pre-trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_pre.tar) | -|ch_ppocr_server_v2.0_rec|General model, supporting Chinese, English and number recognition|[rec_chinese_common_train_v2.0.yml](../../configs/rec/ch_ppocr_v2.0/rec_chinese_common_train_v2.0.yml)|94.8M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_rec_train.tar) / [pre-trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_rec_pre.tar) | - - -**Note:** The `trained model` is fine-tuned on the `pre-trained model` with real data and synthesized vertical text data, which achieved better performance in the real scene. The `pre-trained model` is directly trained on the full amount of real data and synthesized data, which is more suitable for fine-tuning your dataset. - - -### 2.2 English Recognition Model - -|model name|description|config|model size|download| -| --- | --- | --- | --- | --- | -|en_PP-OCRv3_rec_slim | [New] Slim quantization with distillation lightweight model, supporting English, English text recognition |[en_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/en_PP-OCRv3_rec.yml)| 3.2M |[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_slim_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_slim_train.tar) / [nb model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_slim_infer.nb) | -|en_PP-OCRv3_rec| [New] Original lightweight model, supporting English, English, multilingual text recognition |[en_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/en_PP-OCRv3_rec.yml)| 9.6M |[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_train.tar) | -|en_number_mobile_slim_v2.0_rec|Slim pruned and quantized lightweight model, supporting English and number recognition|[rec_en_number_lite_train.yml](../../configs/rec/multi_language/rec_en_number_lite_train.yml)| 2.7M | [inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/en_number_mobile_v2.0_rec_slim_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/en_number_mobile_v2.0_rec_slim_train.tar) | -|en_number_mobile_v2.0_rec|Original lightweight model, supporting English and number recognition|[rec_en_number_lite_train.yml](../../configs/rec/multi_language/rec_en_number_lite_train.yml)|2.6M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/multilingual/en_number_mobile_v2.0_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/multilingual/en_number_mobile_v2.0_rec_train.tar) | - -**Note:** Dictionary file of all English recognition models is `ppocr/utils/en_dict.txt`. - - -### 2.3 Multilingual Recognition Model(Updating...) - -|model name| dict file | description|config|model size|download| -| --- | --- | --- |--- | --- | --- | -| korean_PP-OCRv3_rec | ppocr/utils/dict/korean_dict.txt |Lightweight model for Korean recognition|[korean_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/korean_PP-OCRv3_rec.yml)|11.0M|[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/korean_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/korean_PP-OCRv3_rec_train.tar) | -| japan_PP-OCRv3_rec | ppocr/utils/dict/japan_dict.txt |Lightweight model for Japanese recognition|[japan_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/japan_PP-OCRv3_rec.yml)|11.0M|[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/japan_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/japan_PP-OCRv3_rec_train.tar) | -| chinese_cht_PP-OCRv3_rec | ppocr/utils/dict/chinese_cht_dict.txt | Lightweight model for chinese cht|[chinese_cht_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/chinese_cht_PP-OCRv3_rec.yml)|12.0M|[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/chinese_cht_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/chinese_cht_PP-OCRv3_rec_train.tar) | -| te_PP-OCRv3_rec | ppocr/utils/dict/te_dict.txt | Lightweight model for Telugu recognition |[te_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/te_PP-OCRv3_rec.yml)|9.6M|[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/te_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/te_PP-OCRv3_rec_train.tar) | -| ka_PP-OCRv3_rec | ppocr/utils/dict/ka_dict.txt | Lightweight model for Kannada recognition |[ka_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/ka_PP-OCRv3_rec.yml)|9.9M|[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/ka_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/ka_PP-OCRv3_rec_train.tar) | -| ta_PP-OCRv3_rec | ppocr/utils/dict/ta_dict.txt |Lightweight model for Tamil recognition|[ta_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/ta_PP-OCRv3_rec.yml)|9.6M|[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/ta_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/ta_PP-OCRv3_rec_train.tar) | -| latin_PP-OCRv3_rec | ppocr/utils/dict/latin_dict.txt | Lightweight model for latin recognition | [latin_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/latin_PP-OCRv3_rec.yml) |9.7M|[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/latin_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/latin_PP-OCRv3_rec_train.tar) | -| arabic_PP-OCRv3_rec | ppocr/utils/dict/arabic_dict.txt | Lightweight model for arabic recognition | [arabic_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/arabic_PP-OCRv3_rec.yml) |9.6M|[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/arabic_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/arabic_PP-OCRv3_rec_train.tar) | -| cyrillic_PP-OCRv3_rec | ppocr/utils/dict/cyrillic_dict.txt | Lightweight model for cyrillic recognition | [cyrillic_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/cyrillic_PP-OCRv3_rec.yml) |9.6M|[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/cyrillic_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/cyrillic_PP-OCRv3_rec_train.tar) | -| devanagari_PP-OCRv3_rec | ppocr/utils/dict/devanagari_dict.txt | Lightweight model for devanagari recognition | [devanagari_PP-OCRv3_rec.yml](../../configs/rec/PP-OCRv3/multi_language/devanagari_PP-OCRv3_rec.yml) |9.9M|[inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/devanagari_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/multilingual/devanagari_PP-OCRv3_rec_train.tar) | - -For a complete list of languages ​​and tutorials, please refer to [Multi-language model](./multi_languages_en.md) - - -## 3. Text Angle Classification Model - -|model name|description|config|model size|download| -| --- | --- | --- | --- | --- | -|ch_ppocr_mobile_slim_v2.0_cls|Slim quantized model for text angle classification|[cls_mv3.yml](../../configs/cls/cls_mv3.yml)| 2.1M | [inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_slim_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_slim_train.tar) / [nb model](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_ppocr_mobile_v2.0_cls_infer_opt.nb) | -|ch_ppocr_mobile_v2.0_cls|Original model for text angle classification|[cls_mv3.yml](../../configs/cls/cls_mv3.yml)|1.38M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | - - -## 4. Paddle-Lite Model - -Paddle Lite is an updated version of Paddle-Mobile, an open-open source deep learning framework designed to make it easy to perform inference on mobile, embedded, and IoT devices. It can further optimize the inference model and generate the `nb model` used for edge devices. It's suggested to optimize the quantization model using Paddle-Lite because the `INT8` format is used for the model storage and inference. - -This chapter lists OCR nb models with PP-OCRv2 or earlier versions. You can access the latest nb models from the above tables. - -|Version|Introduction|Model size|Detection model|Text Direction model|Recognition model|Paddle-Lite branch| -|---|---|---|---|---|---|---| -|PP-OCRv2|extra-lightweight chinese OCR optimized model|11.0M|[download link](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_PP-OCRv2_det_infer_opt.nb)|[download link](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_ppocr_mobile_v2.0_cls_infer_opt.nb)|[download link](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_PP-OCRv2_rec_infer_opt.nb)|v2.10| -|PP-OCRv2(slim)|extra-lightweight chinese OCR optimized model|4.6M|[download link](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_PP-OCRv2_det_slim_opt.nb)|[download link](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_ppocr_mobile_v2.0_cls_slim_opt.nb)|[download link](https://paddleocr.bj.bcebos.com/PP-OCRv2/lite/ch_PP-OCRv2_rec_slim_opt.nb)|v2.10| -|PP-OCRv2|extra-lightweight chinese OCR optimized model|11.0M|[download link](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_infer_opt.nb)|[download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_cls_opt.nb)|[download link](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_infer_opt.nb)|v2.9| -|PP-OCRv2(slim)|extra-lightweight chinese OCR optimized model|4.9M|[download link](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_slim_opt.nb)|[download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_cls_slim_opt.nb)|[download link](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_slim_opt.nb)|v2.9| -|V2.0|ppocr_v2.0 extra-lightweight chinese OCR optimized model|7.8M|[download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_det_opt.nb)|[download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_cls_opt.nb)|[download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_rec_opt.nb)|v2.9| -|V2.0(slim)|ppovr_v2.0 extra-lightweight chinese OCR optimized model|3.3M|[download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_det_slim_opt.nb)|[download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_cls_slim_opt.nb)|[download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/lite/ch_ppocr_mobile_v2.0_rec_slim_opt.nb)|v2.9| diff --git a/doc/doc_en/multi_languages_en.md b/doc/doc_en/multi_languages_en.md deleted file mode 100644 index 4d2e5fe513..0000000000 --- a/doc/doc_en/multi_languages_en.md +++ /dev/null @@ -1,239 +0,0 @@ -# Multi-language model - -**Recent Update** - -- 2022.5.8 update the `PP-OCRv3` version of the multi-language detection and recognition model, and the average recognition accuracy has increased by more than 5%. -- 2021.4.9 supports the detection and recognition of 80 languages -- 2021.4.9 supports **lightweight high-precision** English model detection and recognition - -PaddleOCR aims to create a rich, leading, and practical OCR tool library, which not only provides -Chinese and English models in general scenarios, but also provides models specifically trained -in English scenarios. And multilingual models covering [80 languages](#language_abbreviations). - -Among them, the English model supports the detection and recognition of uppercase and lowercase -letters and common punctuation, and the recognition of space characters is optimized: - -
- -
- -The multilingual models cover Latin, Arabic, Traditional Chinese, Korean, Japanese, etc.: - -
- - - - -
- -This document will briefly introduce how to use the multilingual model. - -- [1 Installation](#Install) - - [1.1 Paddle installation](#paddleinstallation) - - [1.2 PaddleOCR package installation](#paddleocr_package_install) - -- [2 Quick Use](#Quick_Use) - - [2.1 Command line operation](#Command_line_operation) - - [2.2 Run with Python script](#python_Script_running) -- [3 Custom Training](#Custom_Training) -- [4 Inference and Deployment](#inference) -- [4 Supported languages and abbreviations](#language_abbreviations) - - -## 1 Installation - - -### 1.1 Paddle installation -``` -# cpu -pip install paddlepaddle - -# gpu -pip install paddlepaddle-gpu -``` - - -### 1.2 PaddleOCR package installation - - -pip install -``` -pip install paddleocr -``` -Build and install locally -``` -python3 -m build -pip3 install dist/paddleocr-x.x.x-py3-none-any.whl # x.x.x is the version number of paddleocr -``` - - -## 2 Quick use - - -### 2.1 Command line operation - -View help information - -``` -paddleocr -h -``` - -* Whole image prediction (detection + recognition) - -PaddleOCR currently supports 80 languages, which can be specified by the --lang parameter. -The supported languages are listed in the [table](#language_abbreviations). - -``` bash -paddleocr --image_dir doc/imgs_en/254.jpg --lang=en -``` -
- - -
- -The result is a list. Each item contains a text box, text and recognition confidence -```text -[('PHO CAPITAL', 0.95723116), [[66.0, 50.0], [327.0, 44.0], [327.0, 76.0], [67.0, 82.0]]] -[('107 State Street', 0.96311164), [[72.0, 90.0], [451.0, 84.0], [452.0, 116.0], [73.0, 121.0]]] -[('Montpelier Vermont', 0.97389287), [[69.0, 132.0], [501.0, 126.0], [501.0, 158.0], [70.0, 164.0]]] -[('8022256183', 0.99810505), [[71.0, 175.0], [363.0, 170.0], [364.0, 202.0], [72.0, 207.0]]] -[('REG 07-24-201706:59 PM', 0.93537045), [[73.0, 299.0], [653.0, 281.0], [654.0, 318.0], [74.0, 336.0]]] -[('045555', 0.99346405), [[509.0, 331.0], [651.0, 325.0], [652.0, 356.0], [511.0, 362.0]]] -[('CT1', 0.9988654), [[535.0, 367.0], [654.0, 367.0], [654.0, 406.0], [535.0, 406.0]]] -...... -``` - -* Recognition - -```bash -paddleocr --image_dir doc/imgs_words_en/word_308.png --det false --lang=en -``` - -![](https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/release/2.1/doc/imgs_words_en/word_308.png) - -The result is a 2-tuple, which contains the recognition result and recognition confidence - -```text -(0.99879867, 'LITTLE') -``` - -* Detection - -``` -paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg --rec false -``` - -The result is a list. Each item represents the coordinates of a text box. - -``` -[[26.0, 457.0], [137.0, 457.0], [137.0, 477.0], [26.0, 477.0]] -[[25.0, 425.0], [372.0, 425.0], [372.0, 448.0], [25.0, 448.0]] -[[128.0, 397.0], [273.0, 397.0], [273.0, 414.0], [128.0, 414.0]] -...... -``` - - -### 2.2 Run with Python script - -PPOCR is able to run with Python scripts for easy integration with your own code: - -* Whole image prediction (detection + recognition) - -``` -from paddleocr import PaddleOCR, draw_ocr - -# Also switch the language by modifying the lang parameter -ocr = PaddleOCR(lang="korean") # The model file will be downloaded automatically when executed for the first time -img_path ='doc/imgs/korean_1.jpg' -result = ocr.ocr(img_path) -# Recognition and detection can be performed separately through parameter control -# result = ocr.ocr(img_path, det=False) Only perform recognition -# result = ocr.ocr(img_path, rec=False) Only perform detection -# Print detection frame and recognition result -for line in result: - print(line) - -# Visualization -from PIL import Image -image = Image.open(img_path).convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/korean.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` - -Visualization of results: -![](https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/release/2.1/doc/imgs_results/korean.jpg) - - -PPOCR also supports direction classification. For more detailed usage, please refer to: [whl package instructions](whl_en.md). - - -## 3 Custom training - -PPOCR supports using your own data for custom training or fine-tune, where the recognition model can refer to [French configuration file](../../configs/rec/multi_language/rec_french_lite_train.yml) -Modify the training data path, dictionary and other parameters. - -For specific data preparation and training process, please refer to: [Text Detection](../doc_en/detection_en.md), [Text Recognition](../doc_en/recognition_en.md), more functions such as predictive deployment, -For functions such as data annotation, you can read the complete [Document Tutorial](../../README.md). - - - -## 4 Inference and Deployment - -In addition to installing the whl package for quick forecasting, -PPOCR also provides a variety of forecasting deployment methods. -If necessary, you can read related documents: - -- [Python Inference](./inference_ppocr_en.md) -- [C++ Inference](../../deploy/cpp_infer/readme.md) -- [Serving](../../deploy/hubserving/readme_en.md) -- [Mobile](../../deploy/lite/readme.md) -- [Benchmark](./benchmark_en.md) - - - -## 5 Support languages and abbreviations - -| Language | Abbreviation | | Language | Abbreviation | -| --- | --- | --- | --- | --- | -|Chinese & English|ch| |Arabic|ar| -|English|en| |Hindi|hi| -|French|fr| |Uyghur|ug| -|German|german| |Persian|fa| -|Japan|japan| |Urdu|ur| -|Korean|korean| | Serbian(latin) |rs_latin| -|Chinese Traditional |chinese_cht| |Occitan |oc| -| Italian |it| |Marathi|mr| -|Spanish |es| |Nepali|ne| -| Portuguese|pt| |Serbian(cyrillic)|rs_cyrillic| -|Russia|ru||Bulgarian |bg| -|Ukranian|uk| |Estonian |et| -|Belarusian|be| |Irish |ga| -|Telugu |te| |Croatian |hr| -|Saudi Arabia|sa| |Hungarian |hu| -|Tamil |ta| |Indonesian|id| -|Afrikaans |af| |Icelandic|is| -|Azerbaijani |az||Kurdish|ku| -|Bosnian|bs| |Lithuanian |lt| -|Czech|cs| |Latvian |lv| -|Welsh |cy| |Maori|mi| -|Danish|da| |Malay|ms| -|Maltese |mt| |Adyghe |ady| -|Dutch |nl| |Kabardian |kbd| -|Norwegian |no| |Avar |ava| -|Polish |pl| |Dargwa |dar| -|Romanian |ro| |Ingush |inh| -|Slovak |sk| |Lak |lbe| -|Slovenian |sl| |Lezghian |lez| -|Albanian |sq| |Tabassaran |tab| -|Swedish |sv| |Bihari |bh| -|Swahili |sw| |Maithili |mai| -|Tagalog |tl| |Angika |ang| -|Turkish |tr| |Bhojpuri |bho| -|Uzbek |uz| |Magahi |mah| -|Vietnamese |vi| |Nagpur |sck| -|Mongolian |mn| |Newari |new| -|Abaza |abq| |Goan Konkani|gom| diff --git a/doc/doc_en/ocr_book_en.md b/doc/doc_en/ocr_book_en.md deleted file mode 100644 index ec2b65529e..0000000000 --- a/doc/doc_en/ocr_book_en.md +++ /dev/null @@ -1,29 +0,0 @@ -# E-book: *Dive Into OCR* - -"Dive Into OCR" is a textbook that combines OCR theory and practice, written by the PaddleOCR community. The main features are as follows: - -- OCR full-stack technology covering text detection, recognition and document analysis -- Closely integrate theory and practice, cross the code implementation gap, and supporting instructional videos -- Jupyter Notebook textbook, flexibly modifying code for instant results - -## Structure - -
- -
- -- The first part is the preliminary knowledge of the book, including the knowledge index and resource links needed in the process of positioning and using the book content of the book - -- The second part is chapters 4-8 of the book, which introduce the concepts, applications, and industry practices related to the detection and identification capabilities of the OCR engine. In the "Introduction to OCR Technology", the application scenarios and challenges of OCR, the basic concepts of technology, and the pain points in industrial applications are comprehensively explained. Then, in the two chapters of "Text Detection" and "Text Recognition", the two basic tasks of OCR are introduced. In each chapter, an algorithm is accompanied by a detailed explanation of the code and practical exercises. Chapters 6 and 7 are a detailed introduction to the PP-OCR series model, PP-OCR is a set of OCR systems for industrial applications, on the basis of the basic detection and identification model, after a series of optimization strategies to achieve the general field of industrial SOTA model, while opening up a variety of predictive deployment solutions, enabling enterprises to quickly land OCR applications. - -- The third part is chapter 9-12 of the book, which introduces applications other than the two-stage OCR engine, including data synthesis, preprocessing algorithm, and end-to-end model, focusing on OCR's layout analysis, table recognition, visual document question and answer capabilities in the document scene, and also through the combination of algorithm and code, so that readers can deeply understand and apply. - - -## Address -- [E-book: *Dive Into OCR* (PDF)](https://paddleocr.bj.bcebos.com/ebook/Dive_into_OCR.pdf) -- [Notebook (.ipynb)](https://github.com/PaddleOCR-Community/Dive-into-OCR) -- [Videos (Chinese only)](https://aistudio.baidu.com/aistudio/education/group/info/25207) - - -trackgit-views - diff --git a/doc/doc_en/ppocr_introduction_en.md b/doc/doc_en/ppocr_introduction_en.md deleted file mode 100644 index 16a078dc02..0000000000 --- a/doc/doc_en/ppocr_introduction_en.md +++ /dev/null @@ -1,128 +0,0 @@ -English | [简体中文](../doc_ch/ppocr_introduction.md) - -# PP-OCR - -- [1. Introduction](#1) -- [2. Features](#2) -- [3. Benchmark](#3) -- [4. Visualization](#4) -- [5. Tutorial](#5) - - [5.1 Quick start](#51) - - [5.2 Model training / compression / deployment](#52) -- [6. Model zoo](#6) - - - -## 1. Introduction - -PP-OCR is a self-developed practical ultra-lightweight OCR system, which is slimed and optimized based on the reimplemented [academic algorithms](algorithm_en.md), considering the balance between **accuracy** and **speed**. - -#### PP-OCR -PP-OCR is a two-stage OCR system, in which the text detection algorithm is [DB](algorithm_det_db_en.md), and the text recognition algorithm is [CRNN](algorithm_rec_crnn_en.md). Besides, a [text direction classifier](angle_class_en.md) is added between the detection and recognition modules to deal with text in different directions. - -PP-OCR pipeline is as follows: - -
- -
- - -PP-OCR system is in continuous optimization. At present, PP-OCR and PP-OCRv2 have been released: - -PP-OCR adopts 19 effective strategies from 8 aspects including backbone network selection and adjustment, prediction head design, data augmentation, learning rate transformation strategy, regularization parameter selection, pre-training model use, and automatic model tailoring and quantization to optimize and slim down the models of each module (as shown in the green box above). The final results are an ultra-lightweight Chinese and English OCR model with an overall size of 3.5M and a 2.8M English digital OCR model. For more details, please refer to [PP-OCR technical report](https://arxiv.org/abs/2009.09941). - -#### PP-OCRv2 -On the basis of PP-OCR, PP-OCRv2 is further optimized in five aspects. The detection model adopts CML(Collaborative Mutual Learning) knowledge distillation strategy and CopyPaste data expansion strategy. The recognition model adopts LCNet lightweight backbone network, U-DML knowledge distillation strategy and enhanced CTC loss function improvement (as shown in the red box above), which further improves the inference speed and prediction effect. For more details, please refer to [PP-OCRv2 technical report](https://arxiv.org/abs/2109.03144). - -#### PP-OCRv3 - -PP-OCRv3 upgraded the detection model and recognition model in 9 aspects based on PP-OCRv2: -- PP-OCRv3 detector upgrades the CML(Collaborative Mutual Learning) text detection strategy proposed in PP-OCRv2, and further optimizes the effect of teacher model and student model respectively. In the optimization of teacher model, a pan module with large receptive field named LK-PAN is proposed and the DML distillation strategy is adopted; In the optimization of student model, a FPN module with residual attention mechanism named RSE-FPN is proposed. -- PP-OCRv3 recognizer is optimized based on text recognition algorithm [SVTR](https://arxiv.org/abs/2205.00159). SVTR no longer adopts RNN by introducing transformers structure, which can mine the context information of text line image more effectively, so as to improve the ability of text recognition. PP-OCRv3 adopts lightweight text recognition network SVTR_LCNet, guided training of CTC by attention, data augmentation strategy TextConAug, better pre-trained model by self-supervised TextRotNet, UDML(Unified Deep Mutual Learning), and UIM (Unlabeled Images Mining) to accelerate the model and improve the effect. - -PP-OCRv3 pipeline is as follows: - -
- -
- -For more details, please refer to [PP-OCRv3 technical report](https://arxiv.org/abs/2206.03001v2). - - -## 2. Features - -- Ultra lightweight PP-OCRv4 series models: detection (4.7M) + direction classifier (1.4M) + recognition 10M) = 16.1M -- Ultra lightweight PP-OCRv3 series models: detection (3.6M) + direction classifier (1.4M) + recognition 12M) = 17.0M -- Ultra lightweight PP-OCRv2 series models: detection (3.1M) + direction classifier (1.4M) + recognition 8.5M) = 13.0M -- Ultra lightweight PP-OCR mobile series models: detection (3.0M) + direction classifier (1.4M) + recognition (5.0M) = 9.4M -- General PP-OCR server series models: detection (47.1M) + direction classifier (1.4M) + recognition (94.9M) = 143.4M -- Support Chinese, English, and digit recognition, vertical text recognition, and long text recognition -- Support multi-lingual recognition: about 80 languages like Korean, Japanese, German, French, etc - - -## 3. benchmark - -For the performance comparison between PP-OCR series models, please check the [benchmark](./benchmark_en.md) documentation. - - -## 4. Visualization [more](./visualization.md) - -
-PP-OCRv3 Chinese model -
- - - -
-
- -
-PP-OCRv3 English model -
- - -
-
- -
-PP-OCRv3 Multilingual model -
- - -
-
- - - -## 5. Tutorial - - -### 5.1 Quick start - -- You can also quickly experience the ultra-lightweight OCR : [Online Experience](https://www.paddlepaddle.org.cn/hub/scene/ocr) -- Mobile DEMO experience (based on EasyEdge and Paddle-Lite, supports iOS and Android systems): [Sign in to the website to obtain the QR code for installing the App](https://ai.baidu.com/easyedge/app/openSource?from=paddlelite) -- One line of code quick use: [Quick Start](./quickstart_en.md) - - -### 5.2 Model training / compression / deployment - -For more tutorials, including model training, model compression, deployment, etc., please refer to [tutorials](../../README.md#Tutorials)。 - - -## 6. Model zoo - -## PP-OCR Series Model List(Update on 2022.04.28) - -| Model introduction | Model name | Recommended scene | Detection model | Direction classifier | Recognition model | -| ------------------------------------------------------------ | ---------------------------- | ----------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | -| Chinese and English ultra-lightweight PP-OCRv4 model(15.8M) | ch_PP-OCRv4_xx | Mobile & Server | [inference model](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_det_distill_train.tar) | [inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [inference model](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv4/chinese/ch_PP-OCRv4_rec_train.tar) | -| Chinese and English ultra-lightweight PP-OCRv3 model(16.2M) | ch_PP-OCRv3_xx | Mobile & Server | [inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar) | [inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_train.tar) | -| English ultra-lightweight PP-OCRv3 model(13.4M) | en_PP-OCRv3_xx | Mobile & Server | [inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_distill_train.tar) | [inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [inference model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_train.tar) | -| Chinese and English ultra-lightweight PP-OCRv2 model(11.6M) | ch_PP-OCRv2_xx | Mobile & Server | [inference model](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_distill_train.tar) | [inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [inference model](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_train.tar) | -| Chinese and English ultra-lightweight PP-OCR model (9.4M) | ch_ppocr_mobile_v2.0_xx | Mobile & server | [inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_train.tar) | [inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_train.tar) | -| Chinese and English general PP-OCR model (143.4M) | ch_ppocr_server_v2.0_xx | Server | [inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_det_train.tar) | [inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) |[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_rec_train.tar) | - - -For more model downloads (including multiple languages), please refer to [PP-OCR series model downloads](./models_list_en.md). - -For a new language request, please refer to [Guideline for new language_requests](../../README.md#language_requests). diff --git a/doc/doc_en/quickstart_en.md b/doc/doc_en/quickstart_en.md deleted file mode 100644 index dc2becd167..0000000000 --- a/doc/doc_en/quickstart_en.md +++ /dev/null @@ -1,311 +0,0 @@ -# PaddleOCR Quick Start - -**Note:** This tutorial mainly introduces the usage of PP-OCR series models, please refer to [PP-Structure Quick Start](../../ppstructure/docs/quickstart_en.md) for the quick use of document analysis related functions. - -- [1. Installation](#1-installation) - - [1.1 Install PaddlePaddle](#11-install-paddlepaddle) - - [1.2 Install PaddleOCR Whl Package](#12-install-paddleocr-whl-package) -- [2. Easy-to-Use](#2-easy-to-use) - - [2.1 Use by Command Line](#21-use-by-command-line) - - [2.1.1 Chinese and English Model](#211-chinese-and-english-model) - - [2.1.2 Multi-language Model](#212-multi-language-model) - - [2.2 Use by Code](#22-use-by-code) - - [2.2.1 Chinese & English Model and Multilingual Model](#221-chinese--english-model-and-multilingual-model) -- [3. Summary](#3-summary) - - - - - -## 1. Installation - - - -### 1.1 Install PaddlePaddle - -> If you do not have a Python environment, please refer to [Environment Preparation](./environment_en.md). - -- If you have CUDA 11 installed on your machine, please run the following command to install - - ```bash - pip install paddlepaddle-gpu - ``` - -- If you have no available GPU on your machine, please run the following command to install the CPU version - - ```bash - python -m pip install paddlepaddle - ``` - -For more software version requirements, please refer to the instructions in [Installation Document](https://www.paddlepaddle.org.cn/en/install/quick) for operation. - - - -### 1.2 Install PaddleOCR Whl Package - -```bash -pip install "paddleocr>=2.0.1" # Recommend to use version 2.0.1+ -``` - - - -## 2. Easy-to-Use - - - -### 2.1 Use by Command Line - -PaddleOCR provides a series of test images, click [here](https://paddleocr.bj.bcebos.com/dygraph_v2.1/ppocr_img.zip) to download, and then switch to the corresponding directory in the terminal - -```bash -cd /path/to/ppocr_img -``` - -If you do not use the provided test image, you can replace the following `--image_dir` parameter with the corresponding test image path - - - -#### 2.1.1 Chinese and English Model - -* Detection, direction classification and recognition: set the parameter`--use_gpu false` to disable the gpu device - - ```bash - paddleocr --image_dir ./imgs_en/img_12.jpg --use_angle_cls true --lang en --use_gpu false - ``` - - Output will be a list, each item contains bounding box, text and recognition confidence - - ```bash - [[[441.0, 174.0], [1166.0, 176.0], [1165.0, 222.0], [441.0, 221.0]], ('ACKNOWLEDGEMENTS', 0.9971134662628174)] - [[[403.0, 346.0], [1204.0, 348.0], [1204.0, 384.0], [402.0, 383.0]], ('We would like to thank all the designers and', 0.9761400818824768)] - [[[403.0, 396.0], [1204.0, 398.0], [1204.0, 434.0], [402.0, 433.0]], ('contributors who have been involved in the', 0.9791957139968872)] - ...... - ``` - - pdf file is also supported, you can infer the first few pages by using the `page_num` parameter, the default is 0, which means infer all pages - - ```bash - paddleocr --image_dir ./xxx.pdf --use_angle_cls true --use_gpu false --page_num 2 - ``` - -* Only detection: set `--rec` to `false` - - ```bash - paddleocr --image_dir ./imgs_en/img_12.jpg --rec false - ``` - - Output will be a list, each item only contains bounding box - - ```bash - [[397.0, 802.0], [1092.0, 802.0], [1092.0, 841.0], [397.0, 841.0]] - [[397.0, 750.0], [1211.0, 750.0], [1211.0, 789.0], [397.0, 789.0]] - [[397.0, 702.0], [1209.0, 698.0], [1209.0, 734.0], [397.0, 738.0]] - ...... - ``` - -* Only recognition: set `--det` to `false` - - ```bash - paddleocr --image_dir ./imgs_words_en/word_10.png --det false --lang en - ``` - - Output will be a list, each item contains text and recognition confidence - - ```bash - ['PAIN', 0.9934559464454651] - ``` - -**Version** -paddleocr uses the PP-OCRv4 model by default(`--ocr_version PP-OCRv4`). If you want to use other versions, you can set the parameter `--ocr_version`, the specific version description is as follows: -| version name | description | -| --- | --- | -| PP-OCRv4 | support Chinese and English detection and recognition, direction classifier, support multilingual recognition | -| PP-OCRv3 | support Chinese and English detection and recognition, direction classifier, support multilingual recognition | -| PP-OCRv2 | only supports Chinese and English detection and recognition, direction classifier, multilingual model is not updated | -| PP-OCR | support Chinese and English detection and recognition, direction classifier, support multilingual recognition | - -If you want to add your own trained model, you can add model links and keys in [paddleocr](../../paddleocr.py) and recompile. - -More whl package usage can be found in [whl package](./whl_en.md) - - - -#### 2.1.2 Multi-language Model - -PaddleOCR currently supports 80 languages, which can be switched by modifying the `--lang` parameter. - -``` bash -paddleocr --image_dir ./doc/imgs_en/254.jpg --lang=en -``` - -
- - -
-The result is a list, each item contains a text box, text and recognition confidence - -```text -[[[67.0, 51.0], [327.0, 46.0], [327.0, 74.0], [68.0, 80.0]], ('PHOCAPITAL', 0.9944712519645691)] -[[[72.0, 92.0], [453.0, 84.0], [454.0, 114.0], [73.0, 122.0]], ('107 State Street', 0.9744491577148438)] -[[[69.0, 135.0], [501.0, 125.0], [501.0, 156.0], [70.0, 165.0]], ('Montpelier Vermont', 0.9357033967971802)] -...... -``` - -Commonly used multilingual abbreviations include - -| Language | Abbreviation | | Language | Abbreviation | | Language | Abbreviation | -| ------------------- | ------------ | ---- | -------- | ------------ | ---- | -------- | ------------ | -| Chinese & English | ch | | French | fr | | Japanese | japan | -| English | en | | German | german | | Korean | korean | -| Chinese Traditional | chinese_cht | | Italian | it | | Russian | ru | - -A list of all languages and their corresponding abbreviations can be found in [Multi-Language Model Tutorial](./multi_languages_en.md) - - - - -### 2.2 Use by Code - - -#### 2.2.1 Chinese & English Model and Multilingual Model - -* detection, angle classification and recognition: - -```python -from paddleocr import PaddleOCR,draw_ocr -# Paddleocr supports Chinese, English, French, German, Korean and Japanese. -# You can set the parameter `lang` as `ch`, `en`, `fr`, `german`, `korean`, `japan` -# to switch the language model in order. -ocr = PaddleOCR(use_angle_cls=True, lang='en') # need to run only once to download and load model into memory -img_path = './imgs_en/img_12.jpg' -result = ocr.ocr(img_path, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - - -# draw result -from PIL import Image -result = result[0] -image = Image.open(img_path).convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='./fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` - -Output will be a list, each item contains bounding box, text and recognition confidence - -```bash -[[[441.0, 174.0], [1166.0, 176.0], [1165.0, 222.0], [441.0, 221.0]], ('ACKNOWLEDGEMENTS', 0.9971134662628174)] - [[[403.0, 346.0], [1204.0, 348.0], [1204.0, 384.0], [402.0, 383.0]], ('We would like to thank all the designers and', 0.9761400818824768)] - [[[403.0, 396.0], [1204.0, 398.0], [1204.0, 434.0], [402.0, 433.0]], ('contributors who have been involved in the', 0.9791957139968872)] - ...... -``` - -Visualization of results - -
- -
- -If the input is a PDF file, you can refer to the following code for visualization - -```python -from paddleocr import PaddleOCR, draw_ocr - -# Paddleocr supports Chinese, English, French, German, Korean and Japanese. -# You can set the parameter `lang` as `ch`, `en`, `fr`, `german`, `korean`, `japan` -# to switch the language model in order. -PAGE_NUM = 10 # Set the recognition page number -pdf_path = 'default.pdf' -ocr = PaddleOCR(use_angle_cls=True, lang="ch", page_num=PAGE_NUM) # need to run only once to download and load model into memory -# ocr = PaddleOCR(use_angle_cls=True, lang="ch", page_num=PAGE_NUM,use_gpu=0) # To Use GPU,uncomment this line and comment the above one. -result = ocr.ocr(pdf_path, cls=True) -for idx in range(len(result)): - res = result[idx] - if res == None: # Skip when empty result detected to avoid TypeError:NoneType - print(f"[DEBUG] Empty page {idx+1} detected, skip it.") - continue - for line in res: - print(line) -# draw the result -import fitz -from PIL import Image -import cv2 -import numpy as np -imgs = [] -with fitz.open(pdf_path) as pdf: - for pg in range(0, PAGE_NUM): - page = pdf[pg] - mat = fitz.Matrix(2, 2) - pm = page.get_pixmap(matrix=mat, alpha=False) - # if width or height > 2000 pixels, don't enlarge the image - if pm.width > 2000 or pm.height > 2000: - pm = page.get_pixmap(matrix=fitz.Matrix(1, 1), alpha=False) - img = Image.frombytes("RGB", [pm.width, pm.height], pm.samples) - img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR) - imgs.append(img) -for idx in range(len(result)): - res = result[idx] - if res == None: - continue - image = imgs[idx] - boxes = [line[0] for line in res] - txts = [line[1][0] for line in res] - scores = [line[1][1] for line in res] - im_show = draw_ocr(image, boxes, txts, scores, font_path='doc/fonts/simfang.ttf') - im_show = Image.fromarray(im_show) - im_show.save('result_page_{}.jpg'.format(idx)) -``` - -* Detection and Recognition Using Sliding Windows - -To perform OCR using sliding windows, the following code snippet can be employed: - -```Python -from paddleocr import PaddleOCR -from PIL import Image, ImageDraw, ImageFont - -# Initialize OCR engine -ocr = PaddleOCR(use_angle_cls=True, lang="en") - -img_path = "./very_large_image.jpg" -slice = {'horizontal_stride': 300, 'vertical_stride': 500, 'merge_x_thres': 50, 'merge_y_thres': 35} -results = ocr.ocr(img_path, cls=True, slice=slice) - -# Load image -image = Image.open(img_path).convert("RGB") -draw = ImageDraw.Draw(image) -font = ImageFont.truetype("./doc/fonts/simfang.ttf", size=20) # Adjust size as needed - -# Process and draw results -for res in results: - for line in res: - box = [tuple(point) for point in line[0]] - # Finding the bounding box - box = [(min(point[0] for point in box), min(point[1] for point in box)), - (max(point[0] for point in box), max(point[1] for point in box))] - txt = line[1][0] - draw.rectangle(box, outline="red", width=2) # Draw rectangle - draw.text((box[0][0], box[0][1] - 25), txt, fill="blue", font=font) # Draw text above the box - -# Save result -image.save("result.jpg") - -``` - -This example initializes the PaddleOCR instance with angle classification enabled and sets the language to English. The `ocr` method is then called with several parameters to customize the detection and recognition process, including the `slice` parameter for handling image slices. - -For a more comprehensive understanding of the slicing operation, please refer to the [slice operation documentation](./slice_en.md). - - - -## 3. Summary - -In this section, you have mastered the use of PaddleOCR whl package. - -PaddleX provides a high-quality ecological model of the paddle. It is a one-stop full-process high-efficiency development platform for training, pressing and pushing. Its mission is to help AI technology to be implemented quickly. The vision is to make everyone an AI Developer! Currently PP-OCRv4 has been launched on PaddleX, you can enter [General OCR](https://aistudio.baidu.com/aistudio/modelsdetail?modelId=286) to experience the whole process of model training, compression and inference deployment. diff --git a/doc/doc_en/recognition_en.md b/doc/doc_en/recognition_en.md deleted file mode 100644 index 13b689ea9a..0000000000 --- a/doc/doc_en/recognition_en.md +++ /dev/null @@ -1,515 +0,0 @@ -# Text Recognition - -- [1. Data Preparation](#DATA_PREPARATION) - * [1.1 Custom Dataset](#Custom_Dataset) - * [1.2 Dataset Download](#Dataset_download) - * [1.3 Dictionary](#Dictionary) - * [1.4 Add Space Category](#Add_space_category) - * [1.5 Data Augmentation](#Data_Augmentation) -- [2. Training](#TRAINING) - * [2.1 Start Training](#21-start-training) - * [2.2 Load Trained Model and Continue Training](#22-load-trained-model-and-continue-training) - * [2.3 Training with New Backbone](#23-training-with-new-backbone) - * [2.4 Mixed Precision Training](#24-amp-training) - * [2.5 Distributed Training](#25-distributed-training) - * [2.6 Training with knowledge distillation](#kd) - * [2.7 Multi-language Training](#Multi_language) - * [2.8 Training on other platform(Windows/macOS/Linux DCU)](#28) - * [2.9 Fine-tuning](#29) -- [3. Evaluation and Test](#3-evaluation-and-test) - * [3.1 Evaluation](#31-evaluation) - * [3.2 Test](#32-test) -- [4. Inference](#4-inference) -- [5. FAQ](#5-faq) - - -## 1. Data Preparation - -### 1.1 DataSet Preparation - -To prepare datasets, refer to [ocr_datasets](./dataset/datasets_en.md) . - -PaddleOCR provides label files for training the icdar2015 dataset, which can be downloaded in the following ways: - -``` -# Training set label -wget -P ./train_data/ic15_data https://paddleocr.bj.bcebos.com/dataset/rec_gt_train.txt -# Test Set Label -wget -P ./train_data/ic15_data https://paddleocr.bj.bcebos.com/dataset/rec_gt_test.txt -``` - -PaddleOCR also provides a data format conversion script, which can convert ICDAR official website label to a data format -supported by PaddleOCR. The data conversion tool is in `ppocr/utils/gen_label.py`, here is the training set as an example: - -``` -# convert the official gt to rec_gt_label.txt -python gen_label.py --mode="rec" --input_path="{path/of/origin/label}" --output_label="rec_gt_label.txt" -``` - -The data format is as follows, (a) is the original picture, (b) is the Ground Truth text file corresponding to each picture: - -![](../datasets/icdar_rec.png) - - -- Multilingual dataset - -The multi-language model training method is the same as the Chinese model. The training data set is 100w synthetic data. A small amount of fonts and test data can be downloaded using the following two methods. -* [Baidu Netdisk](https://pan.baidu.com/s/1bS_u207Rm7YbY33wOECKDA) ,Extraction code:frgi. -* [Google drive](https://drive.google.com/file/d/18cSWX7wXSy4G0tbKJ0d9PuIaiwRLHpjA/view) - - - -### 1.2 Dictionary - -Finally, a dictionary ({word_dict_name}.txt) needs to be provided so that when the model is trained, all the characters that appear can be mapped to the dictionary index. - -Therefore, the dictionary needs to contain all the characters that you want to be recognized correctly. {word_dict_name}.txt needs to be written in the following format and saved in the `utf-8` encoding format: - -``` -l -d -a -d -r -n -``` - -In `word_dict.txt`, there is a single word in each line, which maps characters and numeric indexes together, e.g "and" will be mapped to [2 5 1] - -PaddleOCR has built-in dictionaries, which can be used on demand. - -`ppocr/utils/ppocr_keys_v1.txt` is a Chinese dictionary with 6623 characters. - -`ppocr/utils/ic15_dict.txt` is an English dictionary with 36 characters - -`ppocr/utils/dict/french_dict.txt` is a French dictionary with 118 characters - -`ppocr/utils/dict/japan_dict.txt` is a Japanese dictionary with 4399 characters - -`ppocr/utils/dict/korean_dict.txt` is a Korean dictionary with 3636 characters - -`ppocr/utils/dict/german_dict.txt` is a German dictionary with 131 characters - -`ppocr/utils/en_dict.txt` is a English dictionary with 96 characters - - -The current multi-language model is still in the demo stage and will continue to optimize the model and add languages. **You are very welcome to provide us with dictionaries and fonts in other languages**, -If you like, you can submit the dictionary file to [dict](../../ppocr/utils/dict) and we will thank you in the Repo. - - -To customize the dict file, please modify the `character_dict_path` field in `configs/rec/rec_icdar15_train.yml` . - -- Custom dictionary - -If you need to customize dic file, please add character_dict_path field in configs/rec/rec_icdar15_train.yml to point to your dictionary path. And set character_type to ch. - - -### 1.4 Add Space Category - -If you want to support the recognition of the `space` category, please set the `use_space_char` field in the yml file to `True`. - - -### 1.5 Data Augmentation - -PaddleOCR provides a variety of data augmentation methods. All the augmentation methods are enabled by default. - -The default perturbation methods are: cvtColor, blur, jitter, Gasuss noise, random crop, perspective, color reverse, TIA augmentation. - -Each disturbance method is selected with a 40% probability during the training process. For specific code implementation, please refer to: [rec_img_aug.py](../../ppocr/data/imaug/rec_img_aug.py) - - -## 2.Training - -PaddleOCR provides training scripts, evaluation scripts, and prediction scripts. In this section, the CRNN recognition model will be used as an example: - - -### 2.1 Start Training - -First download the pretrain model, you can download the trained model to finetune on the icdar2015 data: - -``` -cd PaddleOCR/ -# Download the pre-trained model of en_PP-OCRv4 -wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/PP-OCRv4/english/en_PP-OCRv4_rec_train.tar -# Decompress model parameters -cd pretrain_models -tar -xf en_PP-OCRv4_rec_train.tar && rm -rf en_PP-OCRv4_rec_train.tar -``` - -Start training: - -``` -# GPU training Support single card and multi-card training -# Training icdar15 English data and The training log will be automatically saved as train.log under "{save_model_dir}" - -#specify the single card training(Long training time, not recommended) -python3 tools/train.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml -o Global.pretrained_model=en_PP-OCRv4_rec_train/best_accuracy - -#specify the card number through --gpus -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml -o Global.pretrained_model=en_PP-OCRv4_rec_train/best_accuracy -``` - - -PaddleOCR supports alternating training and evaluation. You can modify `eval_batch_step` in `configs/rec/rec_icdar15_train.yml` to set the evaluation frequency. By default, it is evaluated every 500 iter and the best acc model is saved under `output/rec_CRNN/best_accuracy` during the evaluation process. - -If the evaluation set is large, the test will be time-consuming. It is recommended to reduce the number of evaluations, or evaluate after training. - -* Tip: You can use the `-c` parameter to select multiple model configurations under the `configs/rec/` path for training. The recognition algorithms supported at [rec_algorithm](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/doc/doc_en/algorithm_overview_en.md): - - -For training Chinese data, it is recommended to use -[ch_PP-OCRv4_rec_distillation.yml](../../configs/rec/PP-OCRv4/ch_PP-OCRv4_rec_distillation.yml). If you want to try the result of other algorithms on the Chinese data set, please refer to the following instructions to modify the configuration file: - -Take `ch_PP-OCRv4_rec_distillation.yml` as an example: -``` -Global: - ... - # Add a custom dictionary, such as modify the dictionary, please point the path to the new dictionary - character_dict_path: ppocr/utils/ppocr_keys_v1.txt - # Modify character type - ... - # Whether to recognize spaces - use_space_char: True - - -Optimizer: - ... - # Add learning rate decay strategy - lr: - name: Cosine - learning_rate: 0.001 - ... - -... - -Train: - dataset: - # Type of dataset,we support LMDBDataSet and SimpleDataSet - name: SimpleDataSet - # Path of dataset - data_dir: ./train_data/ - # Path of train list - label_file_list: ["./train_data/train_list.txt"] - transforms: - ... - - RecResizeImg: - # Modify image_shape to fit long text - image_shape: [3, 48, 320] - ... - loader: - ... - # Train batch_size for Single card - batch_size_per_card: 256 - ... - -Eval: - dataset: - # Type of dataset,we support LMDBDataSet and SimpleDataSet - name: SimpleDataSet - # Path of dataset - data_dir: ./train_data - # Path of eval list - label_file_list: ["./train_data/val_list.txt"] - transforms: - ... - - RecResizeImg: - # Modify image_shape to fit long text - image_shape: [3, 48, 320] - ... - loader: - # Eval batch_size for Single card - batch_size_per_card: 256 - ... -``` -**Note that the configuration file for prediction/evaluation must be consistent with the training.** - - -### 2.2 Load Trained Model and Continue Training - -If you expect to load trained model and continue the training again, you can specify the parameter `Global.checkpoints` as the model path to be loaded. - -For example: -```shell -python3 tools/train.py -c configs/rec/rec_icdar15_train.yml -o Global.checkpoints=./your/trained/model -``` - -**Note**: The priority of `Global.checkpoints` is higher than that of `Global.pretrained_model`, that is, when two parameters are specified at the same time, the model specified by `Global.checkpoints` will be loaded first. If the model path specified by `Global.checkpoints` is wrong, the one specified by `Global.pretrained_model` will be loaded. - - -### 2.3 Training with New Backbone - -The network part completes the construction of the network, and PaddleOCR divides the network into four parts, which are under [ppocr/modeling](../../ppocr/modeling). The data entering the network will pass through these four parts in sequence(transforms->backbones-> -necks->heads). - -```bash -├── architectures # Code for building network -├── transforms # Image Transformation Module -├── backbones # Feature extraction module -├── necks # Feature enhancement module -└── heads # Output module -``` - -If the Backbone to be replaced has a corresponding implementation in PaddleOCR, you can directly modify the parameters in the `Backbone` part of the configuration yml file. - -However, if you want to use a new Backbone, an example of replacing the backbones is as follows: - -1. Create a new file under the [ppocr/modeling/backbones](../../ppocr/modeling/backbones) folder, such as my_backbone.py. -2. Add code in the my_backbone.py file, the sample code is as follows: - -```python -import paddle -import paddle.nn as nn -import paddle.nn.functional as F - - -class MyBackbone(nn.Layer): - def __init__(self, *args, **kwargs): - super(MyBackbone, self).__init__() - # your init code - self.conv = nn.xxxx - - def forward(self, inputs): - # your network forward - y = self.conv(inputs) - return y -``` - -3. Import the added module in the [ppocr/modeling/backbones/\__init\__.py](../../ppocr/modeling/backbones/__init__.py) file. - -After adding the four-part modules of the network, you only need to configure them in the configuration file to use, such as: - -```yaml - Backbone: - name: MyBackbone - args1: args1 -``` - -**NOTE**: More details about replace Backbone and other mudule can be found in [doc](add_new_algorithm_en.md). - - -### 2.4 Mixed Precision Training - -If you want to speed up your training further, you can use [Auto Mixed Precision Training](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/01_paddle2.0_introduction/basic_concept/amp_cn.html), taking a single machine and a single gpu as an example, the commands are as follows: - -```shell -python3 tools/train.py -c configs/rec/rec_icdar15_train.yml \ - -o Global.pretrained_model=./pretrain_models/en_PP-OCRv3_rec_train/best_accuracy \ - Global.use_amp=True Global.scale_loss=1024.0 Global.use_dynamic_loss_scaling=True -``` - - -### 2.5 Distributed Training - -During multi-machine multi-gpu training, use the `--ips` parameter to set the used machine IP address, and the `--gpus` parameter to set the used GPU ID: - -```bash -python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_icdar15_train.yml \ - -o Global.pretrained_model=./pretrain_models/en_PP-OCRv3_rec_train/best_accuracy -``` - -**Note:** (1) When using multi-machine and multi-gpu training, you need to replace the ips value in the above command with the address of your machine, and the machines need to be able to ping each other. (2) Training needs to be launched separately on multiple machines. The command to view the ip address of the machine is `ifconfig`. (3) For more details about the distributed training speedup ratio, please refer to [Distributed Training Tutorial](./distributed_training_en.md). - - -### 2.6 Training with Knowledge Distillation - -Knowledge distillation is supported in PaddleOCR for text recognition training process. For more details, please refer to [doc](./knowledge_distillation_en.md). - - -### 2.7 Multi-language Training - -Currently, the multi-language algorithms supported by PaddleOCR are: - -| Configuration file | Algorithm name | backbone | trans | seq | pred | language | -| :--------: | :-------: | :-------: | :-------: | :-----: | :-----: | :-----: | -| rec_chinese_cht_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | chinese traditional | -| rec_en_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | English(Case sensitive) | -| rec_french_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | French | -| rec_ger_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | German | -| rec_japan_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | Japanese | -| rec_korean_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | Korean | -| rec_latin_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | Latin | -| rec_arabic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | arabic | -| rec_cyrillic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | cyrillic | -| rec_devanagari_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | devanagari | - -For more supported languages, please refer to : [Multi-language model](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/doc/doc_en/multi_languages_en.md#4-support-languages-and-abbreviations) - - -If you want to finetune on the basis of the existing model effect, please refer to the following instructions to modify the configuration file: - -Take `rec_french_lite_train` as an example: - -``` -Global: - ... - # Add a custom dictionary, such as modify the dictionary, please point the path to the new dictionary - character_dict_path: ./ppocr/utils/dict/french_dict.txt - ... - # Whether to recognize spaces - use_space_char: True - -... - -Train: - dataset: - # Type of dataset,we support LMDBDataSet and SimpleDataSet - name: SimpleDataSet - # Path of dataset - data_dir: ./train_data/ - # Path of train list - label_file_list: ["./train_data/french_train.txt"] - ... - -Eval: - dataset: - # Type of dataset,we support LMDBDataSet and SimpleDataSet - name: SimpleDataSet - # Path of dataset - data_dir: ./train_data - # Path of eval list - label_file_list: ["./train_data/french_val.txt"] - ... -``` - - -### 2.8 Training on other platform(Windows/macOS/Linux DCU) - -- Windows GPU/CPU -The Windows platform is slightly different from the Linux platform: -Windows platform only supports `single gpu` training and inference, specify GPU for training `set CUDA_VISIBLE_DEVICES=0` -On the Windows platform, DataLoader only supports single-process mode, so you need to set `num_workers` to 0; - -- macOS -GPU mode is not supported, you need to set `use_gpu` to False in the configuration file, and the rest of the training evaluation prediction commands are exactly the same as Linux GPU. - -- Linux DCU -Running on a DCU device requires setting the environment variable `export HIP_VISIBLE_DEVICES=0,1,2,3`, and the rest of the training and evaluation prediction commands are exactly the same as the Linux GPU. - - -## 2.9 Fine-tuning - -In actual use, it is recommended to load the official pre-trained model and fine-tune it in your own data set. For the fine-tuning method of the recognition model, please refer to: [Model Fine-tuning Tutorial](./finetune_en.md). - - -## 3. Evaluation and Test - - -### 3.1 Evaluation - -The model parameters during training are saved in the `Global.save_model_dir` directory by default. When evaluating indicators, you need to set `Global.checkpoints` to point to the saved parameter file. The evaluation dataset can be set by modifying the `Eval.dataset.label_file_list` field in the `configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml` file. - - -``` -# GPU evaluation, Global.checkpoints is the weight to be tested -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml -o Global.checkpoints={path/to/weights}/best_accuracy -``` - - -### 3.2 Test - - -Using the model trained by paddleocr, you can quickly get prediction through the following script. - -The default prediction picture is stored in `infer_img`, and the trained weight is specified via `-o Global.checkpoints`: - - -According to the `save_model_dir` and `save_epoch_step` fields set in the configuration file, the following parameters will be saved: - -``` -output/rec/ -├── best_accuracy.pdopt -├── best_accuracy.pdparams -├── best_accuracy.states -├── config.yml -├── iter_epoch_3.pdopt -├── iter_epoch_3.pdparams -├── iter_epoch_3.states -├── latest.pdopt -├── latest.pdparams -├── latest.states -└── train.log -``` - -Among them, best_accuracy.* is the best model on the evaluation set; iter_epoch_x.* is the model saved at intervals of `save_epoch_step`; latest.* is the model of the last epoch. - -``` -# Predict English results -python3 tools/infer_rec.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png -``` - - -Input image: - -![](../imgs_words/en/word_1.png) - -Get the prediction result of the input image: - -``` -infer_img: doc/imgs_words/en/word_1.png - result: ('joint', 0.9998967) -``` - -The configuration file used for prediction must be consistent with the training. For example, you completed the training of the Chinese model with `python3 tools/train.py -c configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml`, you can use the following command to predict the Chinese model: - -``` -# Predict Chinese results -python3 tools/infer_rec.py -c configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/ch/word_1.jpg -``` - -Input image: - -![](../imgs_words/ch/word_1.jpg) - -Get the prediction result of the input image: - -``` -infer_img: doc/imgs_words/ch/word_1.jpg - result: ('韩国小馆', 0.997218) -``` - - -## 4. Inference - -The inference model (the model saved by `paddle.jit.save`) is generally a solidified model saved after the model training is completed, and is mostly used to give prediction in deployment. - -The model saved during the training process is the checkpoints model, which saves the parameters of the model and is mostly used to resume training. - -Compared with the checkpoints model, the inference model will additionally save the structural information of the model. Therefore, it is easier to deploy because the model structure and model parameters are already solidified in the inference model file, and is suitable for integration with actual systems. - -The recognition model is converted to the inference model in the same way as the detection, as follows: - -``` -# -c Set the training algorithm yml configuration file -# -o Set optional parameters -# Global.pretrained_model parameter Set the training model address to be converted without adding the file suffix .pdmodel, .pdopt or .pdparams. -# Global.save_inference_dir Set the address where the converted model will be saved. - -python3 tools/export_model.py -c configs/rec/PP-OCRv4/en_PP-OCRv4_rec.yml -o Global.pretrained_model=en_PP-OCRv4_rec_train/best_accuracy Global.save_inference_dir=./inference/en_PP-OCRv4_rec/ -``` - -If you have a model trained on your own dataset with a different dictionary file, please make sure that you modify the `character_dict_path` in the configuration file to your dictionary file path. - -After the conversion is successful, there are three files in the model save directory: - -``` - -inference/en_PP-OCRv4_rec/ - ├── inference.pdiparams # The parameter file of recognition inference model - ├── inference.pdiparams.info # The parameter information of recognition inference model, which can be ignored - └── inference.pdmodel # The program file of recognition model -``` - -- Text recognition model Inference using custom characters dictionary - - If the text dictionary is modified during training, when using the inference model to predict, you need to specify the dictionary path used by `--rec_char_dict_path` - - ``` - python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./your inference model" --rec_image_shape="3, 32, 100" --rec_char_dict_path="your text dict path" - ``` - - -## 5. FAQ - -Q1: After the training model is transferred to the inference model, the prediction effect is inconsistent? - -**A**: There are many such problems, and the problems are mostly caused by inconsistent preprocessing and postprocessing parameters when the trained model predicts and the preprocessing and postprocessing parameters when the inference model predicts. You can compare whether there are differences in preprocessing, postprocessing, and prediction in the configuration files used for training. diff --git a/doc/doc_en/reference_en.md b/doc/doc_en/reference_en.md deleted file mode 100644 index 066a86a16c..0000000000 --- a/doc/doc_en/reference_en.md +++ /dev/null @@ -1,55 +0,0 @@ -# REFERENCE - -``` -1. EAST: -@inproceedings{zhou2017east, - title={EAST: an efficient and accurate scene text detector}, - author={Zhou, Xinyu and Yao, Cong and Wen, He and Wang, Yuzhi and Zhou, Shuchang and He, Weiran and Liang, Jiajun}, - booktitle={Proceedings of the IEEE conference on Computer Vision and Pattern Recognition}, - pages={5551--5560}, - year={2017} -} - -2. DB: -@article{liao2019real, - title={Real-time Scene Text Detection with Differentiable Binarization}, - author={Liao, Minghui and Wan, Zhaoyi and Yao, Cong and Chen, Kai and Bai, Xiang}, - journal={arXiv preprint arXiv:1911.08947}, - year={2019} -} - -3. DTRB: -@inproceedings{baek2019wrong, - title={What is wrong with scene text recognition model comparisons? dataset and model analysis}, - author={Baek, Jeonghun and Kim, Geewook and Lee, Junyeop and Park, Sungrae and Han, Dongyoon and Yun, Sangdoo and Oh, Seong Joon and Lee, Hwalsuk}, - booktitle={Proceedings of the IEEE International Conference on Computer Vision}, - pages={4715--4723}, - year={2019} -} - -4. SAST: -@inproceedings{wang2019single, - title={A Single-Shot Arbitrarily-Shaped Text Detector based on Context Attended Multi-Task Learning}, - author={Wang, Pengfei and Zhang, Chengquan and Qi, Fei and Huang, Zuming and En, Mengyi and Han, Junyu and Liu, Jingtuo and Ding, Errui and Shi, Guangming}, - booktitle={Proceedings of the 27th ACM International Conference on Multimedia}, - pages={1277--1285}, - year={2019} -} - -5. SRN: -@article{yu2020towards, - title={Towards Accurate Scene Text Recognition with Semantic Reasoning Networks}, - author={Yu, Deli and Li, Xuan and Zhang, Chengquan and Han, Junyu and Liu, Jingtuo and Ding, Errui}, - journal={arXiv preprint arXiv:2003.12294}, - year={2020} -} - -6. end2end-psl: -@inproceedings{sun2019chinese, - title={Chinese Street View Text: Large-scale Chinese Text Reading with Partially Supervised Learning}, - author={Sun, Yipeng and Liu, Jiaming and Liu, Wei and Han, Junyu and Ding, Errui and Liu, Jingtuo}, - booktitle={Proceedings of the IEEE International Conference on Computer Vision}, - pages={9086--9095}, - year={2019} -} -``` diff --git a/doc/doc_en/slice_en.md b/doc/doc_en/slice_en.md deleted file mode 100644 index e48f7e836c..0000000000 --- a/doc/doc_en/slice_en.md +++ /dev/null @@ -1,16 +0,0 @@ -# Slice Operator -If you have a very large image/document that you would like to run PaddleOCR (detection and recognition) on, you can use the slice operation as follows: - -`ocr_inst = PaddleOCR(**ocr_settings)` -`results = ocr_inst.ocr(img, det=True,rec=True, slice=slice, cls=False,bin=False,inv=False,alpha_color=False)` - -where -`slice = {'horizontal_stride': h_stride, 'vertical_stride':v_stride, 'merge_x_thres':x_thres, 'merge_y_thres': y_thres}` - -Here, `h_stride`, `v_stride`, `x_thres`, and `y_thres` are user-configurable values and need to be set manually. The way the `slice` operator works is that it runs a sliding window across the large input image, creating slices of it and runs the OCR algorithms on it. - -The fragmented slice-level results are then merged together to output image-level detection and recognition results. The horizontal and vertical strides cannot be lower than a certain limit (as too low values would create so many slices it would be very computationally expensive to get results for each of them). However, as an example the recommended values for an image with dimensions 6616x14886 would be as follows. - -`slice = {'horizontal_stride': 300, 'vertical_stride':500, 'merge_x_thres':50, 'merge_y_thres': 35}` - -All slice-level detections with bounding boxes as close as `merge_x_thres` and `merge_y_thres` will be merged together. diff --git a/doc/doc_en/table_recognition_en.md b/doc/doc_en/table_recognition_en.md deleted file mode 100644 index c53ec1c0df..0000000000 --- a/doc/doc_en/table_recognition_en.md +++ /dev/null @@ -1,394 +0,0 @@ -# Table Recognition - -This article provides a full-process guide for the PaddleOCR table recognition model, including data preparation, model training, tuning, evaluation, prediction, and detailed descriptions of each stage: - -- [1. Data Preparation](#1-data-preparation) - - [1.1. DataSet Format](#11-dataset-format) - - [1.2. Data Download](#12-data-download) - - [1.3. Dataset Generation](#13-dataset-generation) - - [1.4 Data annotation](#14-data-annotation) -- [2. Training](#2-training) - - [2.1. Start Training](#21-start-training) - - [2.2. Resume Training](#22-resume-training) - - [2.3. Training with New Backbone](#23-training-with-new-backbone) - - [2.4. Mixed Precision Training](#24-mixed-precision-training) - - [2.5. Distributed Training](#25-distributed-training) - - [2.6. Training on other platform(Windows/macOS/Linux DCU)](#26-training-on-other-platformwindowsmacoslinux-dcu) - - [2.7. Fine-tuning](#27-fine-tuning) - - [2.7.1 Dataset](#271-dataset) - - [2.7.2 model selection](#272-model-selection) - - [2.7.3 Training hyperparameter selection](#273-training-hyperparameter-selection) -- [3. Evaluation and Test](#3-evaluation-and-test) - - [3.1. Evaluation](#31-evaluation) - - [3.2. Test table structure recognition effect](#32-test-table-structure-recognition-effect) -- [4. Model export and prediction](#4-model-export-and-prediction) - - [4.1 Model export](#41-model-export) - - [4.2 Prediction](#42-prediction) -- [5. FAQ](#5-faq) - -# 1. Data Preparation - -## 1.1. DataSet Format - -The format of the PaddleOCR table recognition model dataset is as follows: -```txt -img_label # Each image is marked with a string after json.dumps() -... -img_label -``` - -The json format of each line is: -```txt -{ - 'filename': PMC5755158_010_01.png,# image name - 'split': ’train‘, # whether the image belongs to the training set or the validation set - 'imgid': 0,# index of image - 'html': { - 'structure': {'tokens': ['', '', '', ...]}, # HTML string of the table - 'cells': [ - { - 'tokens': ['P', 'a', 'd', 'd', 'l', 'e', 'P', 'a', 'd', 'd', 'l', 'e'], # text in cell - 'bbox': [x0, y0, x1, y1] # bbox of cell - } - ] - } -} -``` - -The default storage path for training data is `PaddleOCR/train_data`, if you already have a dataset on disk, just create a soft link to the dataset directory: - -``` -# linux and mac os -ln -sf /train_data/dataset -# windows -mklink /d /train_data/dataset -``` - -## 1.2. Data Download - -Download the public dataset reference [table_datasets](dataset/table_datasets_en.md)。 - -## 1.3. Dataset Generation - -Use [TableGeneration](https://github.com/WenmuZhou/TableGeneration) to generate scanned table images. - -TableGeneration is an open source table dataset generation tool, which renders html strings through browser rendering to obtain table images. - -Some samples are as follows: - -|Type|Sample| -|---|---| -|Simple Table|![](https://raw.githubusercontent.com/WenmuZhou/TableGeneration/main/imgs/simple.jpg)| -|Simple Color Table|![](https://raw.githubusercontent.com/WenmuZhou/TableGeneration/main/imgs/color.jpg)| - -## 1.4 Data annotation - -Data annotation can refer to[PPOCRLabel](https://github.com/PFCCLab/PPOCRLabel/blob/main/README.md) - -# 2. Training - -PaddleOCR provides training scripts, evaluation scripts, and prediction scripts. In this section, the [SLANet](../../configs/table/SLANet.yml) model will be used as an example: - -## 2.1. Start Training - -*If you are installing the cpu version, please modify the `use_gpu` field in the configuration file to false* - -``` -# GPU training Support single card and multi-card training -# The training log will be automatically saved as train.log under "{save_model_dir}" - -# specify the single card training(Long training time, not recommended) -python3 tools/train.py -c configs/table/SLANet.yml - -# specify the card number through --gpus -python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/table/SLANet.yml -``` - -After starting training normally, you will see the following log output: - -``` -[2022/08/16 03:07:33] ppocr INFO: epoch: [1/400], global_step: 20, lr: 0.000100, acc: 0.000000, loss: 3.915012, structure_loss: 3.229450, loc_loss: 0.670590, avg_reader_cost: 2.63382 s, avg_batch_cost: 6.32390 s, avg_samples: 48.0, ips: 7.59025 samples/s, eta: 9 days, 2:29:27 -[2022/08/16 03:08:41] ppocr INFO: epoch: [1/400], global_step: 40, lr: 0.000100, acc: 0.000000, loss: 1.750859, structure_loss: 1.082116, loc_loss: 0.652822, avg_reader_cost: 0.02533 s, avg_batch_cost: 3.37251 s, avg_samples: 48.0, ips: 14.23271 samples/s, eta: 6 days, 23:28:43 -[2022/08/16 03:09:46] ppocr INFO: epoch: [1/400], global_step: 60, lr: 0.000100, acc: 0.000000, loss: 1.395154, structure_loss: 0.776803, loc_loss: 0.625030, avg_reader_cost: 0.02550 s, avg_batch_cost: 3.26261 s, avg_samples: 48.0, ips: 14.71214 samples/s, eta: 6 days, 5:11:48 -``` - -The following information is automatically printed in the log: - -| Field | Meaning | -| :----: | :------: | -| epoch | current iteration round | -| global_step | current iteration count | -| lr | current learning rate | -| acc | The accuracy of the current batch | -| loss | current loss function | -| structure_loss | Table Structure Loss Values | -| loc_loss | Cell Coordinate Loss Value | -| avg_reader_cost | Current batch data processing time | -| avg_batch_cost | The total time spent in the current batch | -| avg_samples | The number of samples in the current batch | -| ips | Number of images processed per second | - - -PaddleOCR supports alternating training and evaluation. You can modify `eval_batch_step` in `configs/table/SLANet.yml` to set the evaluation frequency. By default, it is evaluated once every 1000 iters. During the evaluation process, the best acc model is saved as `output/SLANet/best_accuracy` by default. - -If the validation set is large, the test will be time-consuming. It is recommended to reduce the number of evaluations, or perform evaluation after training. - -**Tips:** You can use the -c parameter to select various model configurations under the `configs/table/` path for training. For the table recognition algorithms supported by PaddleOCR, please refer to [Table Algorithms List](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/doc/doc_en/algorithm_overview_en.md#3): - -**Note that the configuration file for prediction/evaluation must be the same as training. ** - -## 2.2. Resume Training - -If the training program is interrupted, if you want to load the interrupted model to resume training, you can specify the path of the model to be loaded by specifying Global.checkpoints: - -```shell -python3 tools/train.py -c configs/table/SLANet.yml -o Global.checkpoints=./your/trained/model -``` -**Note**: The priority of `Global.checkpoints` is higher than that of `Global.pretrained_model`, that is, when two parameters are specified at the same time, the model specified by `Global.checkpoints` will be loaded first. If `Global.checkpoints` The specified model path is incorrect, and the model specified by `Global.pretrained_model` will be loaded. - -## 2.3. Training with New Backbone - -The network part completes the construction of the network, and PaddleOCR divides the network into four parts, which are under [ppocr/modeling](../../ppocr/modeling). The data entering the network will pass through these four parts in sequence(transforms->backbones-> -necks->heads). - -```bash -├── architectures # Code for building network -├── transforms # Image Transformation Module -├── backbones # Feature extraction module -├── necks # Feature enhancement module -└── heads # Output module -``` - -If the Backbone to be replaced has a corresponding implementation in PaddleOCR, you can directly modify the parameters in the `Backbone` part of the configuration yml file. - -However, if you want to use a new Backbone, an example of replacing the backbones is as follows: - -1. Create a new file under the [ppocr/modeling/backbones](../../ppocr/modeling/backbones) folder, such as my_backbone.py. -2. Add code in the my_backbone.py file, the sample code is as follows: - -```python -import paddle -import paddle.nn as nn -import paddle.nn.functional as F - - -class MyBackbone(nn.Layer): - def __init__(self, *args, **kwargs): - super(MyBackbone, self).__init__() - # your init code - self.conv = nn.xxxx - - def forward(self, inputs): - # your network forward - y = self.conv(inputs) - return y -``` - -3. Import the added module in the [ppocr/modeling/backbones/\__init\__.py](../../ppocr/modeling/backbones/__init__.py) file. - -After adding the four-part modules of the network, you only need to configure them in the configuration file to use, such as: - -```yaml - Backbone: - name: MyBackbone - args1: args1 -``` - -**NOTE**: More details about replace Backbone and other mudule can be found in [doc](add_new_algorithm_en.md). - -## 2.4. Mixed Precision Training - -If you want to speed up your training further, you can use [Auto Mixed Precision Training](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/01_paddle2.0_introduction/basic_concept/amp_cn.html), taking a single machine and a single gpu as an example, the commands are as follows: - -```shell -python3 tools/train.py -c configs/table/SLANet.yml \ - -o Global.pretrained_model=./pretrain_models/SLANet/best_accuracy \ - Global.use_amp=True Global.scale_loss=1024.0 Global.use_dynamic_loss_scaling=True - ``` - -## 2.5. Distributed Training - -During multi-machine multi-gpu training, use the `--ips` parameter to set the used machine IP address, and the `--gpus` parameter to set the used GPU ID: - -```bash -python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/table/SLANet.yml \ - -o Global.pretrained_model=./pretrain_models/SLANet/best_accuracy -``` - - -**Note:** (1) When using multi-machine and multi-gpu training, you need to replace the ips value in the above command with the address of your machine, and the machines need to be able to ping each other. (2) Training needs to be launched separately on multiple machines. The command to view the ip address of the machine is `ifconfig`. (3) For more details about the distributed training speedup ratio, please refer to [Distributed Training Tutorial](./distributed_training_en.md). - -## 2.6. Training on other platform(Windows/macOS/Linux DCU) - -- Windows GPU/CPU -The Windows platform is slightly different from the Linux platform: -Windows platform only supports `single gpu` training and inference, specify GPU for training `set CUDA_VISIBLE_DEVICES=0` -On the Windows platform, DataLoader only supports single-process mode, so you need to set `num_workers` to 0; - -- macOS -GPU mode is not supported, you need to set `use_gpu` to False in the configuration file, and the rest of the training evaluation prediction commands are exactly the same as Linux GPU. - -- Linux DCU -Running on a DCU device requires setting the environment variable `export HIP_VISIBLE_DEVICES=0,1,2,3`, and the rest of the training and evaluation prediction commands are exactly the same as the Linux GPU. - - -## 2.7. Fine-tuning - - -### 2.7.1 Dataset - -Data number: It is recommended to prepare at least 2000 table recognition datasets for model fine-tuning. - -### 2.7.2 model selection - -It is recommended to choose the SLANet model (configuration file: [SLANet_ch.yml](../../configs/table/SLANet_ch.yml), pre-training model: [ch_ppstructure_mobile_v2.0_SLANet_train.tar](https://paddleocr.bj.bcebos .com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_train.tar)) for fine-tuning, its accuracy and generalization performance is the best Chinese table pre-training model currently available. - -For more table recognition models, please refer to [PP-Structure Series Model Library](../../ppstructure/docs/models_list.md). - -### 2.7.3 Training hyperparameter selection - -When fine-tuning the model, the most important hyperparameters are the pretrained model path `pretrained_model`, the learning rate `learning_rate`, and some configuration files are shown below. - -```yaml -Global: - pretrained_model: ./ch_ppstructure_mobile_v2.0_SLANet_train/best_accuracy.pdparams # Pre-trained model path -Optimizer: - lr: - name: Cosine - learning_rate: 0.001 # - warmup_epoch: 0 - regularizer: - name: 'L2' - factor: 0 -``` - -In the above configuration file, you first need to specify the `pretrained_model` field as the `best_accuracy.pdparams` file path. - -The configuration file provided by PaddleOCR is for 4-card training (equivalent to a total batch size of `4*48=192`) and no pre-trained model is loaded. Therefore, in your scenario, the learning rate is the same as the total The batch size needs to be adjusted linearly, for example - -* If your scenario is single card training, single card batch_size=48, then the total batch_size=48, it is recommended to adjust the learning rate to about `0.00025`. -* If your scenario is for single-card training, due to memory limitations, you can only set batch_size=32 for a single card, then the total batch_size=32, it is recommended to adjust the learning rate to about `0.00017`. - -# 3. Evaluation and Test - -## 3.1. Evaluation - -The model parameters during training are saved in the `Global.save_model_dir` directory by default. When evaluating metrics, you need to set `Global.checkpoints` to point to the saved parameter file. Evaluation datasets can be modified via the `label_file_list` setting in Eval via `configs/table/SLANet.yml`. - -``` -# GPU evaluation, Global.checkpoints is the weight to be tested -python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/table/SLANet.yml -o Global.checkpoints={path/to/weights}/best_accuracy -``` - -After the operation is completed, the acc indicator of the model will be output. If you evaluate the English table recognition model, you will see the following output. - -```bash -[2022/08/16 07:59:55] ppocr INFO: acc:0.7622245132160782 -[2022/08/16 07:59:55] ppocr INFO: fps:30.991640622573044 -``` - -## 3.2. Test table structure recognition effect - -Using the model trained by PaddleOCR, you can quickly get prediction through the following script. - -The default prediction picture is stored in `infer_img`, and the trained weight is specified via `-o Global.checkpoints`: - - -According to the `save_model_dir` and `save_epoch_step` fields set in the configuration file, the following parameters will be saved: - - -``` -output/SLANet/ -├── best_accuracy.pdopt -├── best_accuracy.pdparams -├── best_accuracy.states -├── config.yml -├── latest.pdopt -├── latest.pdparams -├── latest.states -└── train.log -``` -Among them, best_accuracy.* is the best model on the evaluation set; latest.* is the model of the last epoch. - -``` -# Predict table image -python3 tools/infer_table.py -c configs/table/SLANet.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=ppstructure/docs/table/table.jpg -``` - -Input image: - -![](../../ppstructure/docs/table/table.jpg) - -Get the prediction result of the input image: - -``` -['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '
', '', ''],[[320.0562438964844, 197.83375549316406, 350.0928955078125, 214.4309539794922], ... , [318.959228515625, 271.0166931152344, 353.7394104003906, 286.4538269042969]] -``` - -The cell coordinates are visualized as - -![](../../ppstructure/docs/imgs/slanet_result.jpg) - -# 4. Model export and prediction - -## 4.1 Model export - -inference model (model saved by `paddle.jit.save`) -Generally, it is model training, a solidified model that saves the model structure and model parameters in a file, and is mostly used to predict deployment scenarios. -The model saved during the training process is the checkpoints model, and only the parameters of the model are saved, which are mostly used to resume training. -Compared with the checkpoints model, the inference model will additionally save the structural information of the model. It has superior performance in predicting deployment and accelerating reasoning, and is flexible and convenient, and is suitable for actual system integration. - -The way to convert the form recognition model to the inference model is the same as the text detection and recognition, as follows: - -``` -# -c Set the training algorithm yml configuration file -# -o Set optional parameters -# Global.pretrained_model parameter Set the training model address to be converted without adding the file suffix .pdmodel, .pdopt or .pdparams. -# Global.save_inference_dir Set the address where the converted model will be saved. - -python3 tools/export_model.py -c configs/table/SLANet.yml -o Global.pretrained_model=./pretrain_models/SLANet/best_accuracy Global.save_inference_dir=./inference/SLANet/ -``` - -After the conversion is successful, there are three files in the model save directory: - - -``` -inference/SLANet/ - ├── inference.pdiparams # The parameter file of inference model - ├── inference.pdiparams.info # The parameter information of inference model, which can be ignored - └── inference.pdmodel # The program file of model -``` - -## 4.2 Prediction - -After the model is exported, use the following command to complete the prediction of the inference model - -```python -python3.7 table/predict_structure.py \ - --table_model_dir={path/to/inference model} \ - --table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \ - --image_dir=docs/table/table.jpg \ - --output=../output/table -``` - -Input image: - -![](../../ppstructure/docs/table/table.jpg) - -Get the prediction result of the input image: - -``` -['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '
', '', ''],[[320.0562438964844, 197.83375549316406, 350.0928955078125, 214.4309539794922], ... , [318.959228515625, 271.0166931152344, 353.7394104003906, 286.4538269042969]] -``` - -The cell coordinates are visualized as - -![](../../ppstructure/docs/imgs/slanet_result.jpg) - - - -# 5. FAQ - -Q1: After the training model is transferred to the inference model, the prediction effect is inconsistent? - -**A**: There are many such problems, and the problems are mostly caused by inconsistent preprocessing and postprocessing parameters when the trained model predicts and the preprocessing and postprocessing parameters when the inference model predicts. You can compare whether there are differences in preprocessing, postprocessing, and prediction in the configuration files used for training. diff --git a/doc/doc_en/training_en.md b/doc/doc_en/training_en.md deleted file mode 100644 index 86c4deb386..0000000000 --- a/doc/doc_en/training_en.md +++ /dev/null @@ -1,154 +0,0 @@ -# Model Training - -- [1.Yml Configuration ](#1-Yml-Configuration) -- [2. Basic Concepts](#1-basic-concepts) - * [2.1 Learning Rate](#11-learning-rate) - * [2.2 Regularization](#12-regularization) - * [2.3 Evaluation Indicators](#13-evaluation-indicators-) -- [3. Data and Vertical Scenes](#2-data-and-vertical-scenes) - * [3.1 Training Data](#21-training-data) - * [3.2 Vertical Scene](#22-vertical-scene) - * [3.3 Build Your Own Dataset](#23-build-your-own-data-set) -* [4. FAQ](#3-faq) - - -This article will introduce the basic concepts that is necessary for model training and tuning. - -At the same time, it will briefly introduce the structure of the training data and how to prepare the data to fine-tune model in vertical scenes. - - - -## 1. Yml Configuration - -The PaddleOCR uses configuration files to control network training and evaluation parameters. In the configuration file, you can set the model, optimizer, loss function, and pre- and post-processing parameters of the model. PaddleOCR reads these parameters from the configuration file, and then builds a complete training process to train the model. Fine-tuning can also be completed by modifying the parameters in the configuration file, which is simple and convenient. - -For the complete configuration file description, please refer to [Configuration File](./config_en.md) - - - -## 2. Basic Concepts - -During the model training process, some hyper-parameters can be manually specified to obtain the optimal result at the least cost. Different data volumes may require different hyper-parameters. When you want to fine-tune the model based on your own data, there are several parameter adjustment strategies for reference: - - -### 2.1 Learning Rate - -The learning rate is one of the most important hyper-parameters for training neural networks. It represents the step length of the gradient moving towards the optimal solution of the loss function in each iteration. -A variety of learning rate update strategies are provided by PaddleOCR, which can be specified in configuration files. For example, - -``` -Optimizer: - ... - lr: - name: Piecewise - decay_epochs : [700, 800] - values : [0.001, 0.0001] - warmup_epoch: 5 -``` - -`Piecewise` stands for piece-wise constant attenuation. Different learning rates are specified in different learning stages, and the learning rate stay the same in each stage. - -`warmup_epoch` means that in the first 5 epochs, the learning rate will be increased gradually from 0 to base_lr. For all strategies, please refer to the code [learning_rate.py](../../ppocr/optimizer/learning_rate.py). - - -### 2.2 Regularization - -Regularization can effectively avoid algorithm over-fitting. PaddleOCR provides L1 and L2 regularization methods. -L1 and L2 regularization are the most widely used regularization methods. -L1 regularization adds a regularization term to the objective function to reduce the sum of absolute values of the parameters; -while in L2 regularization, the purpose of adding a regularization term is to reduce the sum of squared parameters. -The configuration method is as follows: - -``` -Optimizer: - ... - regularizer: - name: L2 - factor: 2.0e-05 -``` - -### 2.3 Evaluation Indicators - -(1) Detection stage: First, evaluate according to the IOU of the detection frame and the labeled frame. If the IOU is greater than a certain threshold, it is judged that the detection is accurate. Here, the detection frame and the label frame are different from the general general target detection frame, and they are represented by polygons. Detection accuracy: the percentage of the correct detection frame number in all detection frames is mainly used to judge the detection index. Detection recall rate: the percentage of correct detection frames in all marked frames, which is mainly an indicator of missed detection. - -(2) Recognition stage: Character recognition accuracy, that is, the ratio of correctly recognized text lines to the number of marked text lines. Only the entire line of text recognition pairs can be regarded as correct recognition. - -(3) End-to-end statistics: End-to-end recall rate: accurately detect and correctly identify the proportion of text lines in all labeled text lines; End-to-end accuracy rate: accurately detect and correctly identify the number of text lines in the detected text lines The standard for accurate detection is that the IOU of the detection box and the labeled box is greater than a certain threshold, and the text in the correctly identified detection box is the same as the labeled text. - - - -## 3. Data and Vertical Scenes - - - -### 3.1 Training Data - -The current open source models, data sets and magnitudes are as follows: - -- Detection: - - English data set, ICDAR2015 - - Chinese data set, LSVT street view data set training data 3w pictures - -- Identification: - - English data set, MJSynth and SynthText synthetic data, the data volume is tens of millions. - - Chinese data set, LSVT street view data set crops the image according to the truth value, and performs position calibration, a total of 30w images. In addition, based on the LSVT corpus, 500w of synthesized data. - - Small language data set, using different corpora and fonts, respectively generated 100w synthetic data set, and using ICDAR-MLT as the verification set. - -Among them, the public data sets are all open source, users can search and download by themselves, or refer to [Chinese data set](dataset/datasets_en.md), synthetic data is not open source, users can use open source synthesis tools to synthesize by themselves. Synthesis tools include [text_renderer](https://github.com/Sanster/text_renderer), [SynthText](https://github.com/ankush-me/SynthText), [TextRecognitionDataGenerator](https://github.com/Belval/TextRecognitionDataGenerator) etc. - - - -### 3.2 Vertical Scene - -PaddleOCR mainly focuses on general OCR. If you have vertical requirements, you can use PaddleOCR + vertical data to train yourself; -If there is a lack of labeled data, or if you do not want to invest in research and development costs, it is recommended to directly call the open API, which covers some of the more common vertical categories. - - - -### 3.3 Build Your Own Dataset - -There are several experiences for reference when constructing the data set: - -(1) The amount of data in the training set: - - a. The data required for detection is relatively small. For Fine-tune based on the PaddleOCR model, 500 sheets are generally required to achieve good results. - b. Recognition is divided into English and Chinese. Generally, English scenarios require hundreds of thousands of data to achieve good results, while Chinese requires several million or more. - - -(2) When the amount of training data is small, you can try the following three ways to get more data: - - a. Manually collect more training data, the most direct and effective way. - b. Basic image processing or transformation based on PIL and opencv. For example, the three modules of ImageFont, Image, ImageDraw in PIL write text into the background, opencv's rotating affine transformation, Gaussian filtering and so on. - c. Use data generation algorithms to synthesize data, such as algorithms such as pix2pix. - - - -## 4. FAQ - -**Q**: How to choose a suitable network input shape when training CRNN recognition? - - A: The general height is 32, the longest width is selected, there are two methods: - - (1) Calculate the aspect ratio distribution of training sample images. The selection of the maximum aspect ratio considers 80% of the training samples. - - (2) Count the number of texts in training samples. The selection of the longest number of characters considers the training sample that satisfies 80%. Then the aspect ratio of Chinese characters is approximately considered to be 1, and that of English is 3:1, and the longest width is estimated. - -**Q**: During the recognition training, the accuracy of the training set has reached 90, but the accuracy of the verification set has been kept at 70, what should I do? - - A: If the accuracy of the training set is 90 and the test set is more than 70, it should be over-fitting. There are two methods to try: - - (1) Add more augmentation methods or increase the [probability] of augmented prob (https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppocr/data/imaug/rec_img_aug.py#L341), The default is 0.4. - - (2) Increase the [l2 dcay value] of the system (https://github.com/PaddlePaddle/PaddleOCR/blob/a501603d54ff5513fc4fc760319472e59da25424/configs/rec/ch_ppocr_v1.1/rec_chinese_lite_train_v1.1.yml#L47) - -**Q**: When the recognition model is trained, loss can drop normally, but acc is always 0 - - A: It is normal for the acc to be 0 at the beginning of the recognition model training, and the indicator will come up after a longer training period. - -*** - -Click the following links for detailed training tutorial: - -- [text detection model training](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/doc/doc_ch/detection.md) -- [text recognition model training](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/doc/doc_ch/recognition.md) -- [text direction classification model training](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.3/doc/doc_ch/angle_class.md) diff --git a/doc/doc_en/tree_en.md b/doc/doc_en/tree_en.md deleted file mode 100644 index cb188bff50..0000000000 --- a/doc/doc_en/tree_en.md +++ /dev/null @@ -1,219 +0,0 @@ -# Overall directory structure - -The overall directory structure of PaddleOCR is introduced as follows: - - -``` -PaddleOCR -├── configs // Configuration file, you can config the model structure and modify the hyperparameters through the yml file -│ ├── cls // Angle classifier config files -│ │ ├── cls_mv3.yml // Training config, including backbone network, head, loss, optimizer and data -│ ├── det // Text detection config files -│ │ ├── det_mv3_db.yml // Training config -│ │ ... -│ └── rec // Text recognition config files -│ ├── rec_mv3_none_bilstm_ctc.yml // CRNN config -│ ... -├── deploy // Depoly -│ ├── android_demo // Android demo -│ │ ... -│ ├── cpp_infer // C++ infer -│ │ ├── CMakeLists.txt // Cmake file -│ │ ├── docs // Docs -│ │ │ └── windows_vs2019_build.md -│ │ ├── include // Head Files -│ │ │ ├── clipper.h // clipper -│ │ │ ├── config.h // Inference config -│ │ │ ├── ocr_cls.h // Angle class -│ │ │ ├── ocr_det.h // Text detection -│ │ │ ├── ocr_rec.h // Text recognition -│ │ │ ├── postprocess_op.h // Post-processing -│ │ │ ├── preprocess_op.h // Pre-processing -│ │ │ └── utility.h // tools -│ │ ├── readme.md // Documentation -│ │ ├── ... -│ │ ├── src // Source code files -│ │ │ ├── clipper.cpp -│ │ │ ├── config.cpp -│ │ │ ├── main.cpp -│ │ │ ├── ocr_cls.cpp -│ │ │ ├── ocr_det.cpp -│ │ │ ├── ocr_rec.cpp -│ │ │ ├── postprocess_op.cpp -│ │ │ ├── preprocess_op.cpp -│ │ │ └── utility.cpp -│ │ └── tools // Compile and execute script -│ │ ├── build.sh // Compile script -│ │ ├── config.txt // Config file -│ │ └── run.sh // Execute script -│ ├── docker -│ │ └── hubserving -│ │ ├── cpu -│ │ │ └── Dockerfile -│ │ ├── gpu -│ │ │ └── Dockerfile -│ │ ├── README_cn.md -│ │ ├── README.md -│ │ └── sample_request.txt -│ ├── hubserving // hubserving -│ │ ├── ocr_cls // Angle class -│ │ │ ├── config.json // Serving config -│ │ │ ├── __init__.py -│ │ │ ├── module.py // Model -│ │ │ └── params.py // Parameters -│ │ ├── ocr_det // Text detection -│ │ │ ├── config.json // serving config -│ │ │ ├── __init__.py -│ │ │ ├── module.py // Model -│ │ │ └── params.py // Parameters -│ │ ├── ocr_rec // Text recognition -│ │ │ ├── config.json -│ │ │ ├── __init__.py -│ │ │ ├── module.py -│ │ │ └── params.py -│ │ └── ocr_system // Inference System -│ │ ├── config.json -│ │ ├── __init__.py -│ │ ├── module.py -│ │ └── params.py -│ ├── imgs // Inference images -│ │ ├── cpp_infer_pred_12.png -│ │ └── demo.png -│ ├── ios_demo // IOS demo -│ │ ... -│ ├── lite // Lite depoly -│ │ ├── cls_process.cc // Pre-process for angle class -│ │ ├── cls_process.h -│ │ ├── config.txt // Config file -│ │ ├── crnn_process.cc // Pre-process for CRNN -│ │ ├── crnn_process.h -│ │ ├── db_post_process.cc // Pre-process for DB -│ │ ├── db_post_process.h -│ │ ├── Makefile // Compile file -│ │ ├── ocr_db_crnn.cc // Inference system -│ │ ├── prepare.sh // Prepare bash script -│ │ ├── readme.md // Documentation -│ │ ... -│ ├── pdserving // Pdserving depoly -│ │ ├── det_local_server.py // Text detection fast version, easy to deploy and fast to predict -│ │ ├── det_web_server.py // Text detection full version, high stability distributed deployment -│ │ ├── ocr_local_server.py // Text detection + recognition fast version -│ │ ├── ocr_web_client.py // client -│ │ ├── ocr_web_server.py // Text detection + recognition full version -│ │ ├── readme.md // Documentation -│ │ ├── rec_local_server.py // Text recognition fast version -│ │ └── rec_web_server.py // Text recognition full version -│ └── slim -│ └── quantization // Quantization -│ ├── export_model.py // Export model -│ ├── quant.py // Quantization script -│ └── README.md // Documentation -├── doc // Documentation and Tutorials -│ ... -├── ppocr // Core code -│ ├── data // Data processing -│ │ ├── imaug // Image and label processing code -│ │ │ ├── text_image_aug // Tia data augment for text recognition -│ │ │ │ ├── __init__.py -│ │ │ │ ├── augment.py // Tia_distort,tia_stretch and tia_perspective -│ │ │ │ ├── warp_mls.py -│ │ │ ├── __init__.py -│ │ │ ├── east_process.py // Data processing steps of EAST algorithm -│ │ │ ├── iaa_augment.py // Data augmentation operations -│ │ │ ├── label_ops.py // label encode operations -│ │ │ ├── make_border_map.py // Generate boundary map -│ │ │ ├── make_shrink_map.py // Generate shrink graph -│ │ │ ├── operators.py // Basic image operations, such as reading and normalization -│ │ │ ├── randaugment.py // Random data augmentation operation -│ │ │ ├── random_crop_data.py // Random crop -│ │ │ ├── rec_img_aug.py // Data augmentation for text recognition -│ │ │ └── sast_process.py // Data processing steps of SAST algorithm -│ │ ├── __init__.py // Construct dataloader code -│ │ ├── lmdb_dataset.py // Read lmdb dataset -│ │ ├── simple_dataset.py // Read the dataset stored in text format -│ ├── losses // Loss function -│ │ ├── __init__.py // Construct loss code -│ │ ├── cls_loss.py // Angle class loss -│ │ ├── det_basic_loss.py // Text detection basic loss -│ │ ├── det_db_loss.py // DB loss -│ │ ├── det_east_loss.py // EAST loss -│ │ ├── det_sast_loss.py // SAST loss -│ │ ├── rec_ctc_loss.py // CTC loss -│ │ ├── rec_att_loss.py // Attention loss -│ ├── metrics // Metrics -│ │ ├── __init__.py // Construct metric code -│ │ ├── cls_metric.py // Angle class metric -│ │ ├── det_metric.py // Text detection metric - │ ├── eval_det_iou.py // Text detection iou code -│ │ ├── rec_metric.py // Text recognition metric -│ ├── modeling // Network -│ │ ├── architectures // Architecture -│ │ │ ├── __init__.py // Construct model code -│ │ │ ├── base_model.py // Base model -│ │ ├── backbones // backbones -│ │ │ ├── __init__.py // Construct backbone code -│ │ │ ├── det_mobilenet_v3.py // Text detection mobilenet_v3 -│ │ │ ├── det_resnet_vd.py // Text detection resnet -│ │ │ ├── det_resnet_vd_sast.py // Text detection resnet backbone of the SAST algorithm -│ │ │ ├── rec_mobilenet_v3.py // Text recognition mobilenet_v3 -│ │ │ └── rec_resnet_vd.py // Text recognition resnet -│ │ ├── necks // Necks -│ │ │ ├── __init__.py // Construct neck code -│ │ │ ├── db_fpn.py // Standard fpn -│ │ │ ├── east_fpn.py // EAST algorithm fpn network -│ │ │ ├── sast_fpn.py // SAST algorithm fpn network -│ │ │ ├── rnn.py // Character recognition sequence encoding -│ │ ├── heads // Heads -│ │ │ ├── __init__.py // Construct head code -│ │ │ ├── cls_head.py // Angle class head -│ │ │ ├── det_db_head.py // DB head -│ │ │ ├── det_east_head.py // EAST head -│ │ │ ├── det_sast_head.py // SAST head -│ │ │ ├── rec_ctc_head.py // CTC head -│ │ │ ├── rec_att_head.py // Attention head -│ │ ├── transforms // Transforms -│ │ │ ├── __init__.py // Construct transform code -│ │ │ └── tps.py // TPS transform -│ ├── optimizer // Optimizer -│ │ ├── __init__.py // Construct optimizer code -│ │ └── learning_rate.py // Learning rate decay -│ │ └── optimizer.py // Optimizer -│ │ └── regularizer.py // Network regularization -│ ├── postprocess // Post-processing -│ │ ├── cls_postprocess.py // Angle class post-processing -│ │ ├── db_postprocess.py // DB post-processing -│ │ ├── east_postprocess.py // EAST post-processing -│ │ ├── locality_aware_nms.py // NMS -│ │ ├── rec_postprocess.py // Text recognition post-processing -│ │ └── sast_postprocess.py // SAST post-processing -│ └── utils // utils -│ ├── dict // Minor language dictionary -│ .... -│ ├── ic15_dict.txt // English number dictionary, case sensitive -│ ├── ppocr_keys_v1.txt // Chinese dictionary for training Chinese models -│ ├── logging.py // logger -│ ├── save_load.py // Model saving and loading functions -│ ├── stats.py // Training status statistics -│ └── utility.py // Utility function -├── tools -│ ├── eval.py // Evaluation function -│ ├── export_model.py // Export inference model -│ ├── infer // Inference based on Inference engine -│ │ ├── predict_cls.py -│ │ ├── predict_det.py -│ │ ├── predict_rec.py -│ │ ├── predict_system.py -│ │ └── utility.py -│ ├── infer_cls.py // Angle classification inference based on training engine -│ ├── infer_det.py // Text detection inference based on training engine -│ ├── infer_rec.py // Text recognition inference based on training engine -│ ├── program.py // Inference system -│ ├── test_hubserving.py -│ └── train.py // Start training script -├── paddleocr.py -├── README_ch.md // Chinese documentation -├── README_en.md // English documentation -├── README.md // Home page documentation -├── requirements.txt // Requirements -├── setup.py // Whl package packaging script -├── train.sh // Start training bash script diff --git a/doc/doc_en/tricks_en.md b/doc/doc_en/tricks_en.md deleted file mode 100644 index 92edce2666..0000000000 --- a/doc/doc_en/tricks_en.md +++ /dev/null @@ -1,68 +0,0 @@ -## Tricks -Here we have sorted out some Chinese OCR training and prediction tricks, which are being updated continuously. You are welcome to contribute more OCR tricks ~ - -- [Replace Backbone Network](#ReplaceBackboneNetwork) -- [Long Chinese Text Recognition](#LongChineseTextRecognition) -- [Space Recognition](#SpaceRecognition) - - -#### 1、Replace Backbone Network -- **Problem Description** - - At present, ResNet_vd series and MobileNetV3 series are the backbone networks used in PaddleOCR, whether replacing the other backbone networks will help to improve the accuracy? What should be paid attention to when replacing? - -- **Tips** - - Whether text detection or text recognition, the choice of backbone network is a trade-off between prediction effect and prediction efficiency. Generally, a larger backbone network is selected, e.g. ResNet101_vd, then the performance of the detection or recognition is more accurate, but the time cost will increase accordingly. And a smaller backbone network is selected, e.g. MobileNetV3_small_x0_35, the prediction speed is faster, but the accuracy of detection or recognition will be reduced. Fortunately, the detection or recognition effect of different backbone networks is positively correlated with the performance of ImageNet 1000 classification task. [**PaddleClas**](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.3/docs/en/models/models_intro_en.md) have sorted out the 23 series of classification network structures, such as ResNet_vd、Res2Net、HRNet、MobileNetV3、GhostNet. It provides the top1 accuracy of classification, the time cost of GPU(V100 and T4) and CPU(SD 855), and the 117 pretrained models [**download addresses**](https://paddleclas-en.readthedocs.io/en/latest/models/models_intro_en.html). - - - Similar as the 4 stages of ResNet, the replacement of text detection backbone network is to determine those four stages to facilitate the integration of FPN like the object detection heads. In addition, for the text detection problem, the pre trained model in ImageNet1000 can accelerate the convergence and improve the accuracy. - - - In order to replace the backbone network of text recognition, we need to pay attention to the descending position of network width and height stride. Since the ratio between width and height is large in chinese text recognition, the frequency of height decrease is less and the frequency of width decrease is more. You can refer the [modifies of MobileNetV3](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/ppocr/modeling/backbones/rec_mobilenet_v3.py) in PaddleOCR. - - -#### 2、Long Chinese Text Recognition -- **Problem Description** - The maximum resolution of Chinese recognition model during training is [3,32,320], if the text image to be recognized is too long, as shown in the figure below, how to adapt? - -
- -
- -- **Tips** - - During the training, the training samples are not directly resized to [3,32,320]. At first, the height of samples are resized to 32 and keep the ratio between the width and the height. When the width is less than 320, the excess parts are padding 0. Besides, when the ratio between the width and the height of the samples is larger than 10, these samples will be ignored. When the prediction for one image, do as above, but do not limit the max ratio between the width and the height. When the prediction for an images batch, do as training, but the resized target width is the longest width of the images in the batch. [Code as following](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/tools/infer/predict_rec.py): - - ``` - def resize_norm_img(self, img, max_wh_ratio): - imgC, imgH, imgW = self.rec_image_shape - assert imgC == img.shape[2] - if self.character_type == "ch": - imgW = int((32 * max_wh_ratio)) - h, w = img.shape[:2] - ratio = w / float(h) - if math.ceil(imgH * ratio) > imgW: - resized_w = imgW - else: - resized_w = int(math.ceil(imgH * ratio)) - resized_image = cv2.resize(img, (resized_w, imgH)) - resized_image = resized_image.astype('float32') - resized_image = resized_image.transpose((2, 0, 1)) / 255 - resized_image -= 0.5 - resized_image /= 0.5 - padding_im = np.zeros((imgC, imgH, imgW), dtype=np.float32) - padding_im[:, :, 0:resized_w] = resized_image - return padding_im - ``` - - -#### 3、Space Recognition -- **Problem Description** - - As shown in the figure below, for Chinese and English mixed scenes, in order to facilitate reading and using the recognition results, it is often necessary to recognize the spaces between words. How can this situation be adapted? - -
- -
- -- **Tips** - - There are two possible methods for space recognition. (1) Optimize the text detection. For spliting the text at the space in detection results, it needs to divide the text line with space into many segments when label the data for detection. (2) Optimize the text recognition. The space character is introduced into the recognition dictionary. Label the blank line in the training data for text recognition. In addition, we can also concat multiple word lines to synthesize the training data with spaces. PaddleOCR currently uses the second method. diff --git a/doc/doc_en/update_en.md b/doc/doc_en/update_en.md deleted file mode 100644 index ebce9f0289..0000000000 --- a/doc/doc_en/update_en.md +++ /dev/null @@ -1,35 +0,0 @@ -# RECENT UPDATES -- 2022.5.9 release PaddleOCR v2.5, including: - - [PP-OCRv3](./ppocr_introduction_en.md#pp-ocrv3): With comparable speed, the effect of Chinese scene is further improved by 5% compared with PP-OCRv2, the effect of English scene is improved by 11%, and the average recognition accuracy of 80 language multilingual models is improved by more than 5%. - - [PPOCRLabelv2](https://github.com/PFCCLab/PPOCRLabel/blob/main/README.md): Add the annotation function for table recognition task, key information extraction task and irregular text image. - - Interactive e-book [*"Dive into OCR"*](./ocr_book_en.md), covers the cutting-edge theory and code practice of OCR full stack technology. -- 2022.5.7 Add support for metric and model logging during training to [Weights & Biases](https://docs.wandb.ai/). -- 2021.12.21 OCR open source online course starts. The lesson starts at 8:30 every night and lasts for ten days. Free registration: https://aistudio.baidu.com/aistudio/course/introduce/25207 -- 2021.12.21 release PaddleOCR v2.4, release 1 text detection algorithm (PSENet), 3 text recognition algorithms (NRTR、SEED、SAR), 1 key information extraction algorithm (SDMGR) and 3 DocVQA algorithms (LayoutLM、LayoutLMv2,LayoutXLM). -- 2021.9.7 release PaddleOCR v2.3, [PP-OCRv2](#PP-OCRv2) is proposed. The CPU inference speed of PP-OCRv2 is 220% higher than that of PP-OCR server. The F-score of PP-OCRv2 is 7% higher than that of PP-OCR mobile. -- 2021.8.3 released PaddleOCR v2.2, add a new structured documents analysis toolkit, i.e., [PP-Structure](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.2/ppstructure/README.md), support layout analysis and table recognition (One-key to export chart images to Excel files). -- 2021.4.8 release end-to-end text recognition algorithm [PGNet](https://www.aaai.org/AAAI21Papers/AAAI-2885.WangP.pdf) which is published in AAAI 2021. Find tutorial [here](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/doc/doc_en/pgnet_en.md);release multi language recognition [models](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/doc/doc_en/multi_languages_en.md), support more than 80 languages recognition; especically, the performance of [English recognition model](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.1/doc/doc_en/models_list_en.md#English) is Optimized. - -- 2021.1.21 update more than 25+ multilingual recognition models [models list](./models_list_en.md), including:English, Chinese, German, French, Japanese,Spanish,Portuguese Russia Arabic and so on. Models for more languages will continue to be updated [Develop Plan](https://github.com/PaddlePaddle/PaddleOCR/issues/1048). -- 2020.12.15 update Data synthesis tool, i.e., [Style-Text](https://github.com/PFCCLab/StyleText/blob/main/README.md),easy to synthesize a large number of images which are similar to the target scene image. -- 2020.11.25 Update a new data annotation tool, i.e., [PPOCRLabel](https://github.com/PFCCLab/PPOCRLabel/blob/main/README.md), which is helpful to improve the labeling efficiency. Moreover, the labeling results can be used in training of the PP-OCR system directly. -- 2020.9.22 Update the PP-OCR technical article, https://arxiv.org/abs/2009.09941 -- 2020.9.19 Update the ultra lightweight compressed ppocr_mobile_slim series models, the overall model size is 3.5M, suitable for mobile deployment. -- 2020.9.17 update English recognition model and Multilingual recognition model, `English`, `Chinese`, `German`, `French`, `Japanese` and `Korean` have been supported. Models for more languages will continue to be updated. -- 2020.8.24 Support the use of PaddleOCR through whl package installation,pelease refer [PaddleOCR Package](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/doc/doc_en/whl_en.md) -- 2020.8.16 Release text detection algorithm [SAST](https://arxiv.org/abs/1908.05498) and text recognition algorithm [SRN](https://arxiv.org/abs/2003.12294) -- 2020.7.23, Release the playback and PPT of live class on BiliBili station, PaddleOCR Introduction, [address](https://aistudio.baidu.com/aistudio/course/introduce/1519) -- 2020.7.15, Add mobile App demo , support both iOS and Android ( based on easyedge and Paddle Lite) -- 2020.7.15, Improve the deployment ability, add the C + + inference , serving deployment. In addtion, the benchmarks of the ultra-lightweight Chinese OCR model are provided. -- 2020.7.15, Add several related datasets, data annotation and synthesis tools. -- 2020.7.9 Add a new model to support recognize the character "space". -- 2020.7.9 Add the data augument and learning rate decay strategies during training. -- 2020.6.8 Add [datasets](dataset/datasets_en.md) and keep updating -- 2020.6.5 Support exporting `attention` model to `inference_model` -- 2020.6.5 Support separate prediction and recognition, output result score -- 2020.5.30 Provide Lightweight Chinese OCR online experience -- 2020.5.30 Model prediction and training support on Windows system -- 2020.5.30 Open source general Chinese OCR model -- 2020.5.14 Release [PaddleOCR Open Class](https://www.bilibili.com/video/BV1nf4y1U7RX?p=4) -- 2020.5.14 Release [PaddleOCR Practice Notebook](https://aistudio.baidu.com/aistudio/projectdetail/467229) -- 2020.5.14 Open source 8.6M lightweight Chinese OCR model diff --git a/doc/doc_en/visualization_en.md b/doc/doc_en/visualization_en.md deleted file mode 100644 index 8ea64925ea..0000000000 --- a/doc/doc_en/visualization_en.md +++ /dev/null @@ -1,57 +0,0 @@ -# Visualization - - -## PP-OCRv3 - -### PP-OCRv3 Chinese model -
- - - -
- -### PP-OCRv3 English model - -
- - - -
- -### PP-OCRv3 Multilingual model - -
- - -
- - -## PP-OCRv2 - - - - - -## ch_ppocr_server_2.0 - -
- - - -
- - - - -## en_ppocr_mobile_2.0 -
- -
- - - -## (multilingual)_ppocr_mobile_2.0 -
- - -
diff --git a/doc/doc_en/whl_en.md b/doc/doc_en/whl_en.md deleted file mode 100644 index 42cc8f10b5..0000000000 --- a/doc/doc_en/whl_en.md +++ /dev/null @@ -1,447 +0,0 @@ -# Paddleocr Package - -## 1 Get started quickly -### 1.1 install package -install by pypi -```bash -pip install "paddleocr>=2.0.1" # Recommend to use version 2.0.1+ -``` - -build own whl package and install -```bash -python3 -m build -pip3 install dist/paddleocr-x.x.x-py3-none-any.whl # x.x.x is the version of paddleocr -``` -## 2 Use -### 2.1 Use by code -The paddleocr whl package will automatically download the ppocr lightweight model as the default model, which can be customized and replaced according to the section 3 **Custom Model**. - -* detection angle classification and recognition -```python -from paddleocr import PaddleOCR,draw_ocr -# Paddleocr supports Chinese, English, French, German, Korean and Japanese. -# You can set the parameter `lang` as `ch`, `en`, `french`, `german`, `korean`, `japan` -# to switch the language model in order. -ocr = PaddleOCR(use_angle_cls=True, lang='en') # need to run only once to download and load model into memory -img_path = 'PaddleOCR/doc/imgs_en/img_12.jpg' -result = ocr.ocr(img_path, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# draw result -from PIL import Image -result = result[0] -image = Image.open(img_path).convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` - -Output will be a list, each item contains bounding box, text and recognition confidence -```bash -[[[442.0, 173.0], [1169.0, 173.0], [1169.0, 225.0], [442.0, 225.0]], ['ACKNOWLEDGEMENTS', 0.99283075]] -[[[393.0, 340.0], [1207.0, 342.0], [1207.0, 389.0], [393.0, 387.0]], ['We would like to thank all the designers and', 0.9357758]] -[[[399.0, 398.0], [1204.0, 398.0], [1204.0, 433.0], [399.0, 433.0]], ['contributors whohave been involved in the', 0.9592447]] -...... -``` - -Visualization of results - -
- -
- -* detection and recognition -```python -from paddleocr import PaddleOCR,draw_ocr -ocr = PaddleOCR(lang='en') # need to run only once to download and load model into memory -img_path = 'PaddleOCR/doc/imgs_en/img_12.jpg' -result = ocr.ocr(img_path, cls=False) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# draw result -from PIL import Image -result = result[0] -image = Image.open(img_path).convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` - -Output will be a list, each item contains bounding box, text and recognition confidence -```bash -[[[442.0, 173.0], [1169.0, 173.0], [1169.0, 225.0], [442.0, 225.0]], ['ACKNOWLEDGEMENTS', 0.99283075]] -[[[393.0, 340.0], [1207.0, 342.0], [1207.0, 389.0], [393.0, 387.0]], ['We would like to thank all the designers and', 0.9357758]] -[[[399.0, 398.0], [1204.0, 398.0], [1204.0, 433.0], [399.0, 433.0]], ['contributors whohave been involved in the', 0.9592447]] -...... -``` - -Visualization of results - -
- -
- -* classification and recognition -```python -from paddleocr import PaddleOCR -ocr = PaddleOCR(use_angle_cls=True, lang='en') # need to run only once to load model into memory -img_path = 'PaddleOCR/doc/imgs_words_en/word_10.png' -result = ocr.ocr(img_path, det=False, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) -``` - -Output will be a list, each item contains recognition text and confidence -```bash -['PAIN', 0.990372] -``` - -* only detection -```python -from paddleocr import PaddleOCR,draw_ocr -ocr = PaddleOCR() # need to run only once to download and load model into memory -img_path = 'PaddleOCR/doc/imgs_en/img_12.jpg' -result = ocr.ocr(img_path,rec=False) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# draw result -from PIL import Image -result = result[0] -image = Image.open(img_path).convert('RGB') -im_show = draw_ocr(image, result, txts=None, scores=None, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` - -Output will be a list, each item only contains bounding box -```bash -[[756.0, 812.0], [805.0, 812.0], [805.0, 830.0], [756.0, 830.0]] -[[820.0, 803.0], [1085.0, 801.0], [1085.0, 836.0], [820.0, 838.0]] -[[393.0, 801.0], [715.0, 805.0], [715.0, 839.0], [393.0, 836.0]] -...... -``` - -Visualization of results - -
- -
- -* only recognition -```python -from paddleocr import PaddleOCR -ocr = PaddleOCR(lang='en') # need to run only once to load model into memory -img_path = 'PaddleOCR/doc/imgs_words_en/word_10.png' -result = ocr.ocr(img_path, det=False, cls=False) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) -``` - -Output will be a list, each item contains recognition text and confidence -```bash -['PAIN', 0.990372] -``` - -* only classification -```python -from paddleocr import PaddleOCR -ocr = PaddleOCR(use_angle_cls=True) # need to run only once to load model into memory -img_path = 'PaddleOCR/doc/imgs_words_en/word_10.png' -result = ocr.ocr(img_path, det=False, rec=False, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) -``` - -Output will be a list, each item contains classification result and confidence -```bash -['0', 0.99999964] -``` - -### 2.2 Use by command line - -show help information -```bash -paddleocr -h -``` - -* detection classification and recognition -```bash -paddleocr --image_dir PaddleOCR/doc/imgs_en/img_12.jpg --use_angle_cls true --lang en -``` - -Output will be a list, each item contains bounding box, text and recognition confidence -```bash -[[[441.0, 174.0], [1166.0, 176.0], [1165.0, 222.0], [441.0, 221.0]], ('ACKNOWLEDGEMENTS', 0.9971134662628174)] -[[[403.0, 346.0], [1204.0, 348.0], [1204.0, 384.0], [402.0, 383.0]], ('We would like to thank all the designers and', 0.9761400818824768)] -[[[403.0, 396.0], [1204.0, 398.0], [1204.0, 434.0], [402.0, 433.0]], ('contributors who have been involved in the', 0.9791957139968872)] -...... -``` - -pdf file is also supported, you can infer the first few pages by using the `page_num` parameter, the default is 0, which means infer all pages -```bash -paddleocr --image_dir ./xxx.pdf --use_angle_cls true --use_gpu false --page_num 2 -``` - -* detection and recognition -```bash -paddleocr --image_dir PaddleOCR/doc/imgs_en/img_12.jpg --lang en -``` - -Output will be a list, each item contains bounding box, text and recognition confidence -```bash -[[[441.0, 174.0], [1166.0, 176.0], [1165.0, 222.0], [441.0, 221.0]], ('ACKNOWLEDGEMENTS', 0.9971134662628174)] -[[[403.0, 346.0], [1204.0, 348.0], [1204.0, 384.0], [402.0, 383.0]], ('We would like to thank all the designers and', 0.9761400818824768)] -[[[403.0, 396.0], [1204.0, 398.0], [1204.0, 434.0], [402.0, 433.0]], ('contributors who have been involved in the', 0.9791957139968872)] -...... -``` - -* classification and recognition -```bash -paddleocr --image_dir PaddleOCR/doc/imgs_words_en/word_10.png --use_angle_cls true --det false --lang en -``` - -Output will be a list, each item contains text and recognition confidence -```bash -['PAIN', 0.9934559464454651] -``` - -* only detection -```bash -paddleocr --image_dir PaddleOCR/doc/imgs_en/img_12.jpg --rec false -``` - -Output will be a list, each item only contains bounding box -```bash -[[397.0, 802.0], [1092.0, 802.0], [1092.0, 841.0], [397.0, 841.0]] -[[397.0, 750.0], [1211.0, 750.0], [1211.0, 789.0], [397.0, 789.0]] -[[397.0, 702.0], [1209.0, 698.0], [1209.0, 734.0], [397.0, 738.0]] -...... -``` - -* only recognition -```bash -paddleocr --image_dir PaddleOCR/doc/imgs_words_en/word_10.png --det false --lang en -``` - -Output will be a list, each item contains text and recognition confidence -```bash -['PAIN', 0.9934559464454651] -``` - -* only classification -```bash -paddleocr --image_dir PaddleOCR/doc/imgs_words_en/word_10.png --use_angle_cls true --det false --rec false -``` - -Output will be a list, each item contains classification result and confidence -```bash -['0', 0.99999964] -``` - -## 3 Use custom model -When the built-in model cannot meet the needs, you need to use your own trained model. -First, refer to [export](./detection_en.md#4-inference) doc to convert your det and rec model to inference model, and then use it as follows - -### 3.1 Use by code - -```python -from paddleocr import PaddleOCR,draw_ocr -# The path of detection and recognition model must contain model and params files -ocr = PaddleOCR(det_model_dir='{your_det_model_dir}', rec_model_dir='{your_rec_model_dir}', rec_char_dict_path='{your_rec_char_dict_path}', cls_model_dir='{your_cls_model_dir}', use_angle_cls=True) -img_path = 'PaddleOCR/doc/imgs_en/img_12.jpg' -result = ocr.ocr(img_path, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# draw result -from PIL import Image -result = result[0] -image = Image.open(img_path).convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` - -### 3.2 Use by command line - -```bash -paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg --det_model_dir {your_det_model_dir} --rec_model_dir {your_rec_model_dir} --rec_char_dict_path {your_rec_char_dict_path} --cls_model_dir {your_cls_model_dir} --use_angle_cls true -``` - -## 4 Use web images or numpy array as input - -### 4.1 Web image - -- Use by code -```python -from paddleocr import PaddleOCR, draw_ocr -ocr = PaddleOCR(use_angle_cls=True, lang="ch") # need to run only once to download and load model into memory -img_path = 'http://n.sinaimg.cn/ent/transform/w630h933/20171222/o111-fypvuqf1838418.jpg' -result = ocr.ocr(img_path, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# show result -from PIL import Image -result = result[0] -image = Image.open(img_path).convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` -- Use by command line -```bash -paddleocr --image_dir http://n.sinaimg.cn/ent/transform/w630h933/20171222/o111-fypvuqf1838418.jpg --use_angle_cls=true -``` - -### 4.2 Numpy array -Support numpy array as input only when used by code - -```python -import cv2 -from paddleocr import PaddleOCR, draw_ocr, download_with_progressbar -ocr = PaddleOCR(use_angle_cls=True, lang="ch") # need to run only once to download and load model into memory -img_path = 'PaddleOCR/doc/imgs/11.jpg' -img = cv2.imread(img_path) -# img = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY), If your own training model supports grayscale images, you can uncomment this line -result = ocr.ocr(img, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# show result -from PIL import Image -result = result[0] -download_with_progressbar(img_path, 'tmp.jpg') -image = Image.open('tmp.jpg').convert('RGB') -boxes = [line[0] for line in result] -txts = [line[1][0] for line in result] -scores = [line[1][1] for line in result] -im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf') -im_show = Image.fromarray(im_show) -im_show.save('result.jpg') -``` -## 5 PDF file -- Use by command line - -you can infer the first few pages by using the `page_num` parameter, the default is 0, which means infer all pages -```bash -paddleocr --image_dir ./xxx.pdf --use_angle_cls true --use_gpu false --page_num 2 -``` -- Use by code - -```python -from paddleocr import PaddleOCR, draw_ocr - -# Paddleocr supports Chinese, English, French, German, Korean and Japanese. -# You can set the parameter `lang` as `ch`, `en`, `fr`, `german`, `korean`, `japan` -# to switch the language model in order. -ocr = PaddleOCR(use_angle_cls=True, lang="ch", page_num=2) # need to run only once to download and load model into memory -img_path = './xxx.pdf' -result = ocr.ocr(img_path, cls=True) -for idx in range(len(result)): - res = result[idx] - for line in res: - print(line) - -# draw result -import fitz -from PIL import Image -import cv2 -import numpy as np -imgs = [] -with fitz.open(img_path) as pdf: - for pg in range(0, pdf.pageCount): - page = pdf[pg] - mat = fitz.Matrix(2, 2) - pm = page.getPixmap(matrix=mat, alpha=False) - # if width or height > 2000 pixels, don't enlarge the image - if pm.width > 2000 or pm.height > 2000: - pm = page.getPixmap(matrix=fitz.Matrix(1, 1), alpha=False) - - img = Image.frombytes("RGB", [pm.width, pm.height], pm.samples) - img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR) - imgs.append(img) -for idx in range(len(result)): - res = result[idx] - image = imgs[idx] - boxes = [line[0] for line in res] - txts = [line[1][0] for line in res] - scores = [line[1][1] for line in res] - im_show = draw_ocr(image, boxes, txts, scores, font_path='doc/fonts/simfang.ttf') - im_show = Image.fromarray(im_show) - im_show.save('result_page_{}.jpg'.format(idx)) -``` - -## 6 Parameter Description - -| Parameter | Description | Default value | -|-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------| -| use_gpu | use GPU or not | TRUE | -| gpu_mem | GPU memory size used for initialization | 8000M | -| image_dir | The images path or folder path for predicting when used by the command line | | -| page_num | Valid when the input type is pdf file, specify to predict the previous page_num pages, all pages are predicted by default | 0 | -| det_algorithm | Type of detection algorithm selected | DB | -| det_model_dir | the text detection inference model folder. There are two ways to transfer parameters, 1. None: Automatically download the built-in model to `~/.paddleocr/det`; 2. The path of the inference model converted by yourself, the model and params files must be included in the model path | None | -| det_max_side_len | The maximum size of the long side of the image. When the long side exceeds this value, the long side will be resized to this size, and the short side will be scaled proportionally | 960 | -| det_db_thresh | Binarization threshold value of DB output map | 0.3 | -| det_db_box_thresh | The threshold value of the DB output box. Boxes score lower than this value will be discarded | 0.5 | -| det_db_unclip_ratio | The expanded ratio of DB output box | 2 | -| det_db_score_mode | The parameter that control how the score of the detection frame is calculated. There are 'fast' and 'slow' options. If the text to be detected is curved, it is recommended to use 'slow' | 'fast' | -| det_east_score_thresh | Binarization threshold value of EAST output map | 0.8 | -| det_east_cover_thresh | The threshold value of the EAST output box. Boxes score lower than this value will be discarded | 0.1 | -| det_east_nms_thresh | The NMS threshold value of EAST model output box | 0.2 | -| rec_algorithm | Type of recognition algorithm selected | CRNN | -| rec_model_dir | the text recognition inference model folder. There are two ways to transfer parameters, 1. None: Automatically download the built-in model to `~/.paddleocr/rec`; 2. The path of the inference model converted by yourself, the model and params files must be included in the model path | None | -| rec_image_shape | image shape of recognition algorithm | "3,32,320" | -| rec_batch_num | When performing recognition, the batchsize of forward images | 30 | -| max_text_length | The maximum text length that the recognition algorithm can recognize | 25 | -| rec_char_dict_path | the alphabet path which needs to be modified to your own path when `rec_model_Name` use mode 2 | ./ppocr/utils/ppocr_keys_v1.txt | -| use_space_char | Whether to recognize spaces | TRUE | -| drop_score | Filter the output by score (from the recognition model), and those below this score will not be returned | 0.5 | -| use_angle_cls | Whether to load classification model | FALSE | -| cls_model_dir | the classification inference model folder. There are two ways to transfer parameters, 1. None: Automatically download the built-in model to `~/.paddleocr/cls`; 2. The path of the inference model converted by yourself, the model and params files must be included in the model path | None | -| cls_image_shape | image shape of classification algorithm | "3,48,192" | -| label_list | label list of classification algorithm | ['0','180'] | -| cls_batch_num | When performing classification, the batchsize of forward images | 30 | -| enable_mkldnn | Whether to enable mkldnn | FALSE | -| use_zero_copy_run | Whether to forward by zero_copy_run | FALSE | -| lang | The support language, now only Chinese(ch)、English(en)、French(french)、German(german)、Korean(korean)、Japanese(japan) are supported | ch | -| det | Enable detction when `ppocr.ocr` func exec | TRUE | -| rec | Enable recognition when `ppocr.ocr` func exec | TRUE | -| cls | Enable classification when `ppocr.ocr` func exec((Use use_angle_cls in command line mode to control whether to start classification in the forward direction) | FALSE | -| show_log | Whether to print log| FALSE | -| type | Perform ocr or table structuring, the value is selected in ['ocr','structure'] | ocr | -| ocr_version | OCR Model version number, the current model support list is as follows: PP-OCRv3 supports Chinese and English detection, recognition, multilingual recognition, direction classifier models, PP-OCRv2 support Chinese detection and recognition model, PP-OCR support Chinese detection, recognition and direction classifier, multilingual recognition model | PP-OCRv3 | diff --git a/doc/doc_i18n/README_Ру́сский_язы́к.md b/doc/doc_i18n/README_Ру́сский_язы́к.md index 79389845cf..5193895902 100644 --- a/doc/doc_i18n/README_Ру́сский_язы́к.md +++ b/doc/doc_i18n/README_Ру́сский_язы́к.md @@ -1,7 +1,7 @@ [English](../../README_en.md) | [简体中文](../../README.md) | [हिन्दी](./README_हिन्द.md) | [日本語](./README_日本語.md) | [한국인](./README_한국어.md) | Pу́сский язы́к

- +

diff --git a/doc/doc_i18n/README_हिन्द.md b/doc/doc_i18n/README_हिन्द.md index 0598647c8b..f8ab49ec52 100644 --- a/doc/doc_i18n/README_हिन्द.md +++ b/doc/doc_i18n/README_हिन्द.md @@ -1,7 +1,7 @@ [English](../../README_en.md) | [简体中文](../../README.md) | हिन्दी | [日本語](./README_日本語.md) | [한국인](./README_한국어.md) | [Pу́сский язы́к](./README_Ру́сский_язы́к.md)

- +

diff --git a/doc/doc_i18n/README_日本語.md b/doc/doc_i18n/README_日本語.md index 7d3121f3e2..9a1f97736e 100644 --- a/doc/doc_i18n/README_日本語.md +++ b/doc/doc_i18n/README_日本語.md @@ -1,7 +1,7 @@ [English](../../README_en.md) | [简体中文](../../README.md) | [हिन्दी](./README_हिन्द.md) | 日本語 | [한국인](./README_한국어.md) | [Pу́сский язы́к](./README_Ру́сский_язы́к.md)

- +

diff --git a/doc/imgs/00006737.jpg b/doc/imgs/00006737.jpg deleted file mode 100755 index 5c3329a8a0..0000000000 Binary files a/doc/imgs/00006737.jpg and /dev/null differ diff --git a/doc/imgs/00009282.jpg b/doc/imgs/00009282.jpg deleted file mode 100755 index 448d0f1120..0000000000 Binary files a/doc/imgs/00009282.jpg and /dev/null differ diff --git a/doc/imgs/00015504.jpg b/doc/imgs/00015504.jpg deleted file mode 100755 index 9d6aaee181..0000000000 Binary files a/doc/imgs/00015504.jpg and /dev/null differ diff --git a/doc/imgs/00018069.jpg b/doc/imgs/00018069.jpg deleted file mode 100755 index e768d8adfc..0000000000 Binary files a/doc/imgs/00018069.jpg and /dev/null differ diff --git a/doc/imgs/00056221.jpg b/doc/imgs/00056221.jpg deleted file mode 100755 index 698e0dfc53..0000000000 Binary files a/doc/imgs/00056221.jpg and /dev/null differ diff --git a/doc/imgs/00057937.jpg b/doc/imgs/00057937.jpg deleted file mode 100755 index 82a45c47bb..0000000000 Binary files a/doc/imgs/00057937.jpg and /dev/null differ diff --git a/doc/imgs/00059985.jpg b/doc/imgs/00059985.jpg deleted file mode 100755 index 0b5b656a16..0000000000 Binary files a/doc/imgs/00059985.jpg and /dev/null differ diff --git a/doc/imgs/00077949.jpg b/doc/imgs/00077949.jpg deleted file mode 100755 index 1f832d7903..0000000000 Binary files a/doc/imgs/00077949.jpg and /dev/null differ diff --git a/doc/imgs/00111002.jpg b/doc/imgs/00111002.jpg deleted file mode 100755 index 2aae5f7cb8..0000000000 Binary files a/doc/imgs/00111002.jpg and /dev/null differ diff --git a/doc/imgs/00207393.jpg b/doc/imgs/00207393.jpg deleted file mode 100755 index e278adfb59..0000000000 Binary files a/doc/imgs/00207393.jpg and /dev/null differ diff --git a/doc/imgs/1.jpg b/doc/imgs/1.jpg deleted file mode 100644 index 08010177fe..0000000000 Binary files a/doc/imgs/1.jpg and /dev/null differ diff --git a/doc/imgs/11.jpg b/doc/imgs/11.jpg deleted file mode 100755 index ed91b8c5ca..0000000000 Binary files a/doc/imgs/11.jpg and /dev/null differ diff --git a/doc/imgs/12.jpg b/doc/imgs/12.jpg deleted file mode 100644 index 3cd2e79182..0000000000 Binary files a/doc/imgs/12.jpg and /dev/null differ diff --git a/doc/imgs/french_0.jpg b/doc/imgs/french_0.jpg deleted file mode 100644 index 0c3cc4de54..0000000000 Binary files a/doc/imgs/french_0.jpg and /dev/null differ diff --git a/doc/imgs/ger_1.jpg b/doc/imgs/ger_1.jpg deleted file mode 100644 index 050bf7c7b7..0000000000 Binary files a/doc/imgs/ger_1.jpg and /dev/null differ diff --git a/doc/imgs/ger_2.jpg b/doc/imgs/ger_2.jpg deleted file mode 100644 index b922702bfc..0000000000 Binary files a/doc/imgs/ger_2.jpg and /dev/null differ diff --git a/doc/imgs/japan_1.jpg b/doc/imgs/japan_1.jpg deleted file mode 100644 index ffcb2e4a80..0000000000 Binary files a/doc/imgs/japan_1.jpg and /dev/null differ diff --git a/doc/imgs/japan_2.jpg b/doc/imgs/japan_2.jpg deleted file mode 100644 index 0353874860..0000000000 Binary files a/doc/imgs/japan_2.jpg and /dev/null differ diff --git a/doc/imgs/korean_1.jpg b/doc/imgs/korean_1.jpg deleted file mode 100644 index 4259c046c2..0000000000 Binary files a/doc/imgs/korean_1.jpg and /dev/null differ diff --git a/doc/imgs/model_prod_flow_ch.png b/doc/imgs/model_prod_flow_ch.png deleted file mode 100644 index 4906b2716e..0000000000 Binary files a/doc/imgs/model_prod_flow_ch.png and /dev/null differ diff --git a/doc/imgs_en/img623.jpg b/doc/imgs_en/img623.jpg deleted file mode 100755 index 2fae1b5b1c..0000000000 Binary files a/doc/imgs_en/img623.jpg and /dev/null differ diff --git a/doc/imgs_en/img_11.jpg b/doc/imgs_en/img_11.jpg deleted file mode 100644 index 34397beb6b..0000000000 Binary files a/doc/imgs_en/img_11.jpg and /dev/null differ diff --git a/doc/imgs_en/img_12.jpg b/doc/imgs_en/img_12.jpg deleted file mode 100644 index b0d289538a..0000000000 Binary files a/doc/imgs_en/img_12.jpg and /dev/null differ diff --git a/doc/imgs_en/img_195.jpg b/doc/imgs_en/img_195.jpg deleted file mode 100644 index 5d5546cadd..0000000000 Binary files a/doc/imgs_en/img_195.jpg and /dev/null differ diff --git a/doc/imgs_en/model_prod_flow_en.png b/doc/imgs_en/model_prod_flow_en.png deleted file mode 100644 index eb5cb9cc9f..0000000000 Binary files a/doc/imgs_en/model_prod_flow_en.png and /dev/null differ diff --git a/doc/imgs_en/wandb_metrics.png b/doc/imgs_en/wandb_metrics.png deleted file mode 100644 index 45f0041ae4..0000000000 Binary files a/doc/imgs_en/wandb_metrics.png and /dev/null differ diff --git a/doc/imgs_en/wandb_models.png b/doc/imgs_en/wandb_models.png deleted file mode 100644 index f9a7042bd5..0000000000 Binary files a/doc/imgs_en/wandb_models.png and /dev/null differ diff --git a/doc/imgs_results/PP-OCRv2/PP-OCRv2-pic001.jpg b/doc/imgs_results/PP-OCRv2/PP-OCRv2-pic001.jpg deleted file mode 100644 index 45ffdb53aa..0000000000 Binary files a/doc/imgs_results/PP-OCRv2/PP-OCRv2-pic001.jpg and /dev/null differ diff --git a/doc/imgs_results/PP-OCRv2/PP-OCRv2-pic002.jpg b/doc/imgs_results/PP-OCRv2/PP-OCRv2-pic002.jpg deleted file mode 100644 index 7ac153aee0..0000000000 Binary files a/doc/imgs_results/PP-OCRv2/PP-OCRv2-pic002.jpg and /dev/null differ diff --git a/doc/imgs_results/PP-OCRv2/PP-OCRv2-pic003.jpg b/doc/imgs_results/PP-OCRv2/PP-OCRv2-pic003.jpg deleted file mode 100644 index 781aade629..0000000000 Binary files a/doc/imgs_results/PP-OCRv2/PP-OCRv2-pic003.jpg and /dev/null differ diff --git a/doc/imgs_results/PP-OCRv3/ch/PP-OCRv3-pic001.jpg b/doc/imgs_results/PP-OCRv3/ch/PP-OCRv3-pic001.jpg deleted file mode 100644 index c35936cc1a..0000000000 Binary files a/doc/imgs_results/PP-OCRv3/ch/PP-OCRv3-pic001.jpg and /dev/null differ diff --git a/doc/imgs_results/PP-OCRv3/ch/PP-OCRv3-pic002.jpg b/doc/imgs_results/PP-OCRv3/ch/PP-OCRv3-pic002.jpg deleted file mode 100644 index e5ad6a4b2a..0000000000 Binary files a/doc/imgs_results/PP-OCRv3/ch/PP-OCRv3-pic002.jpg and /dev/null differ diff --git a/doc/imgs_results/PP-OCRv3/ch/PP-OCRv3-pic003.jpg b/doc/imgs_results/PP-OCRv3/ch/PP-OCRv3-pic003.jpg deleted file mode 100644 index dc024296bd..0000000000 Binary files a/doc/imgs_results/PP-OCRv3/ch/PP-OCRv3-pic003.jpg and /dev/null differ diff --git a/doc/imgs_results/PP-OCRv3/en/en_1.png b/doc/imgs_results/PP-OCRv3/en/en_1.png deleted file mode 100644 index 36245613e3..0000000000 Binary files a/doc/imgs_results/PP-OCRv3/en/en_1.png and /dev/null differ diff --git a/doc/imgs_results/PP-OCRv3/en/en_2.png b/doc/imgs_results/PP-OCRv3/en/en_2.png deleted file mode 100644 index d2df8556ad..0000000000 Binary files a/doc/imgs_results/PP-OCRv3/en/en_2.png and /dev/null differ diff --git a/doc/imgs_results/PP-OCRv3/en/en_3.png b/doc/imgs_results/PP-OCRv3/en/en_3.png deleted file mode 100644 index baf146c010..0000000000 Binary files a/doc/imgs_results/PP-OCRv3/en/en_3.png and /dev/null differ diff --git a/doc/imgs_results/PP-OCRv3/en/en_4.png b/doc/imgs_results/PP-OCRv3/en/en_4.png deleted file mode 100644 index f0f19db95b..0000000000 Binary files a/doc/imgs_results/PP-OCRv3/en/en_4.png and /dev/null differ diff --git a/doc/imgs_results/PP-OCRv3/multi_lang/japan_2.jpg b/doc/imgs_results/PP-OCRv3/multi_lang/japan_2.jpg deleted file mode 100644 index 076ced92ad..0000000000 Binary files a/doc/imgs_results/PP-OCRv3/multi_lang/japan_2.jpg and /dev/null differ diff --git a/doc/imgs_results/PP-OCRv3/multi_lang/korean_1.jpg b/doc/imgs_results/PP-OCRv3/multi_lang/korean_1.jpg deleted file mode 100644 index f93de40e18..0000000000 Binary files a/doc/imgs_results/PP-OCRv3/multi_lang/korean_1.jpg and /dev/null differ diff --git a/doc/imgs_results/angle_class_example.jpg b/doc/imgs_results/angle_class_example.jpg deleted file mode 100644 index 8e683be32c..0000000000 Binary files a/doc/imgs_results/angle_class_example.jpg and /dev/null differ diff --git a/doc/imgs_results/ch_ppocr_mobile_v2.0/00006737.jpg b/doc/imgs_results/ch_ppocr_mobile_v2.0/00006737.jpg deleted file mode 100644 index d7762d2e2c..0000000000 Binary files a/doc/imgs_results/ch_ppocr_mobile_v2.0/00006737.jpg and /dev/null differ diff --git a/doc/imgs_results/ch_ppocr_mobile_v2.0/00009282.jpg b/doc/imgs_results/ch_ppocr_mobile_v2.0/00009282.jpg deleted file mode 100644 index 0383d445bd..0000000000 Binary files a/doc/imgs_results/ch_ppocr_mobile_v2.0/00009282.jpg and /dev/null differ diff --git a/doc/imgs_results/ch_ppocr_mobile_v2.0/00015504.jpg b/doc/imgs_results/ch_ppocr_mobile_v2.0/00015504.jpg deleted file mode 100644 index 9162cf1479..0000000000 Binary files a/doc/imgs_results/ch_ppocr_mobile_v2.0/00015504.jpg and /dev/null differ diff --git a/doc/imgs_results/ch_ppocr_mobile_v2.0/00059985.jpg b/doc/imgs_results/ch_ppocr_mobile_v2.0/00059985.jpg deleted file mode 100644 index 03fd19784a..0000000000 Binary files a/doc/imgs_results/ch_ppocr_mobile_v2.0/00059985.jpg and /dev/null differ diff --git a/doc/imgs_results/ch_ppocr_mobile_v2.0/00111002.jpg b/doc/imgs_results/ch_ppocr_mobile_v2.0/00111002.jpg deleted file mode 100644 index 7dae24a92d..0000000000 Binary files a/doc/imgs_results/ch_ppocr_mobile_v2.0/00111002.jpg and /dev/null differ diff --git a/doc/imgs_results/ch_ppocr_mobile_v2.0/img_12.jpg b/doc/imgs_results/ch_ppocr_mobile_v2.0/img_12.jpg deleted file mode 100644 index 11ac4ed6ce..0000000000 Binary files a/doc/imgs_results/ch_ppocr_mobile_v2.0/img_12.jpg and /dev/null differ diff --git a/doc/imgs_results/ch_ppocr_mobile_v2.0/rotate_00052204.jpg b/doc/imgs_results/ch_ppocr_mobile_v2.0/rotate_00052204.jpg deleted file mode 100644 index 643b850da8..0000000000 Binary files a/doc/imgs_results/ch_ppocr_mobile_v2.0/rotate_00052204.jpg and /dev/null differ diff --git a/doc/imgs_results/ch_ppocr_mobile_v2.0/test_add_91.jpg b/doc/imgs_results/ch_ppocr_mobile_v2.0/test_add_91.jpg deleted file mode 100644 index b5ded6e1de..0000000000 Binary files a/doc/imgs_results/ch_ppocr_mobile_v2.0/test_add_91.jpg and /dev/null differ diff --git a/doc/imgs_results/det_res_00018069.jpg b/doc/imgs_results/det_res_00018069.jpg deleted file mode 100644 index 02f35de332..0000000000 Binary files a/doc/imgs_results/det_res_00018069.jpg and /dev/null differ diff --git a/doc/imgs_results/det_res_img623_ct.jpg b/doc/imgs_results/det_res_img623_ct.jpg deleted file mode 100644 index 2c5f57d96c..0000000000 Binary files a/doc/imgs_results/det_res_img623_ct.jpg and /dev/null differ diff --git a/doc/imgs_results/det_res_img623_fce.jpg b/doc/imgs_results/det_res_img623_fce.jpg deleted file mode 100644 index 938ae4cabf..0000000000 Binary files a/doc/imgs_results/det_res_img623_fce.jpg and /dev/null differ diff --git a/doc/imgs_results/det_res_img623_sast.jpg b/doc/imgs_results/det_res_img623_sast.jpg deleted file mode 100644 index af5e2d6e2c..0000000000 Binary files a/doc/imgs_results/det_res_img623_sast.jpg and /dev/null differ diff --git a/doc/imgs_results/det_res_img_10_db.jpg b/doc/imgs_results/det_res_img_10_db.jpg deleted file mode 100644 index 6af89f6bb3..0000000000 Binary files a/doc/imgs_results/det_res_img_10_db.jpg and /dev/null differ diff --git a/doc/imgs_results/det_res_img_10_east.jpg b/doc/imgs_results/det_res_img_10_east.jpg deleted file mode 100644 index 908d077c3e..0000000000 Binary files a/doc/imgs_results/det_res_img_10_east.jpg and /dev/null differ diff --git a/doc/imgs_results/det_res_img_10_fce.jpg b/doc/imgs_results/det_res_img_10_fce.jpg deleted file mode 100644 index fb32950ffd..0000000000 Binary files a/doc/imgs_results/det_res_img_10_fce.jpg and /dev/null differ diff --git a/doc/imgs_results/det_res_img_10_pse.jpg b/doc/imgs_results/det_res_img_10_pse.jpg deleted file mode 100644 index cdb7625dd0..0000000000 Binary files a/doc/imgs_results/det_res_img_10_pse.jpg and /dev/null differ diff --git a/doc/imgs_results/det_res_img_10_pse_poly.jpg b/doc/imgs_results/det_res_img_10_pse_poly.jpg deleted file mode 100644 index 9c06a17ccb..0000000000 Binary files a/doc/imgs_results/det_res_img_10_pse_poly.jpg and /dev/null differ diff --git a/doc/imgs_results/det_res_img_10_sast.jpg b/doc/imgs_results/det_res_img_10_sast.jpg deleted file mode 100644 index 702f773e68..0000000000 Binary files a/doc/imgs_results/det_res_img_10_sast.jpg and /dev/null differ diff --git a/doc/imgs_results/e2e_res_img293_pgnet.png b/doc/imgs_results/e2e_res_img293_pgnet.png deleted file mode 100644 index 232f8293ad..0000000000 Binary files a/doc/imgs_results/e2e_res_img293_pgnet.png and /dev/null differ diff --git a/doc/imgs_results/e2e_res_img295_pgnet.png b/doc/imgs_results/e2e_res_img295_pgnet.png deleted file mode 100644 index 69337e3adf..0000000000 Binary files a/doc/imgs_results/e2e_res_img295_pgnet.png and /dev/null differ diff --git a/doc/imgs_results/e2e_res_img623_pgnet.jpg b/doc/imgs_results/e2e_res_img623_pgnet.jpg deleted file mode 100644 index b45dc05f7b..0000000000 Binary files a/doc/imgs_results/e2e_res_img623_pgnet.jpg and /dev/null differ diff --git a/doc/imgs_results/e2e_res_img_10_pgnet.jpg b/doc/imgs_results/e2e_res_img_10_pgnet.jpg deleted file mode 100644 index a0962993f8..0000000000 Binary files a/doc/imgs_results/e2e_res_img_10_pgnet.jpg and /dev/null differ diff --git a/doc/imgs_results/french_0.jpg b/doc/imgs_results/french_0.jpg deleted file mode 100644 index 3c2abe6304..0000000000 Binary files a/doc/imgs_results/french_0.jpg and /dev/null differ diff --git a/doc/imgs_results/img_10_east_starnet.jpg b/doc/imgs_results/img_10_east_starnet.jpg deleted file mode 100644 index fd8c039230..0000000000 Binary files a/doc/imgs_results/img_10_east_starnet.jpg and /dev/null differ diff --git a/doc/imgs_results/korean.jpg b/doc/imgs_results/korean.jpg deleted file mode 100644 index e5d863cd86..0000000000 Binary files a/doc/imgs_results/korean.jpg and /dev/null differ diff --git a/doc/imgs_results/multi_lang/arabic_0.jpg b/doc/imgs_results/multi_lang/arabic_0.jpg deleted file mode 100644 index 9941b90642..0000000000 Binary files a/doc/imgs_results/multi_lang/arabic_0.jpg and /dev/null differ diff --git a/doc/imgs_results/multi_lang/en_1.jpg b/doc/imgs_results/multi_lang/en_1.jpg deleted file mode 100644 index 2dc84d3f04..0000000000 Binary files a/doc/imgs_results/multi_lang/en_1.jpg and /dev/null differ diff --git a/doc/imgs_results/multi_lang/en_2.jpg b/doc/imgs_results/multi_lang/en_2.jpg deleted file mode 100644 index 455ec98ed3..0000000000 Binary files a/doc/imgs_results/multi_lang/en_2.jpg and /dev/null differ diff --git a/doc/imgs_results/multi_lang/en_3.jpg b/doc/imgs_results/multi_lang/en_3.jpg deleted file mode 100644 index 36eb063d78..0000000000 Binary files a/doc/imgs_results/multi_lang/en_3.jpg and /dev/null differ diff --git a/doc/imgs_results/multi_lang/french_0.jpg b/doc/imgs_results/multi_lang/french_0.jpg deleted file mode 100644 index 3c2abe6304..0000000000 Binary files a/doc/imgs_results/multi_lang/french_0.jpg and /dev/null differ diff --git a/doc/imgs_results/multi_lang/img_01.jpg b/doc/imgs_results/multi_lang/img_01.jpg deleted file mode 100644 index ee6ca69207..0000000000 Binary files a/doc/imgs_results/multi_lang/img_01.jpg and /dev/null differ diff --git a/doc/imgs_results/multi_lang/img_02.jpg b/doc/imgs_results/multi_lang/img_02.jpg deleted file mode 100644 index 3e139c76bc..0000000000 Binary files a/doc/imgs_results/multi_lang/img_02.jpg and /dev/null differ diff --git a/doc/imgs_results/multi_lang/img_12.jpg b/doc/imgs_results/multi_lang/img_12.jpg deleted file mode 100644 index 822d562eda..0000000000 Binary files a/doc/imgs_results/multi_lang/img_12.jpg and /dev/null differ diff --git a/doc/imgs_results/multi_lang/japan_2.jpg b/doc/imgs_results/multi_lang/japan_2.jpg deleted file mode 100644 index 7038ba2eff..0000000000 Binary files a/doc/imgs_results/multi_lang/japan_2.jpg and /dev/null differ diff --git a/doc/imgs_results/multi_lang/korean_0.jpg b/doc/imgs_results/multi_lang/korean_0.jpg deleted file mode 100644 index 3fe6305aa0..0000000000 Binary files a/doc/imgs_results/multi_lang/korean_0.jpg and /dev/null differ diff --git a/doc/imgs_results/sr_word_52.png b/doc/imgs_results/sr_word_52.png deleted file mode 100644 index c983e9ad7a..0000000000 Binary files a/doc/imgs_results/sr_word_52.png and /dev/null differ diff --git a/doc/imgs_results/system_res_00018069.jpg b/doc/imgs_results/system_res_00018069.jpg deleted file mode 100644 index fc06b05085..0000000000 Binary files a/doc/imgs_results/system_res_00018069.jpg and /dev/null differ diff --git a/doc/imgs_results/system_res_00018069_v3.jpg b/doc/imgs_results/system_res_00018069_v3.jpg deleted file mode 100644 index 51808ca556..0000000000 Binary files a/doc/imgs_results/system_res_00018069_v3.jpg and /dev/null differ diff --git a/doc/imgs_results/whl/11_det.jpg b/doc/imgs_results/whl/11_det.jpg deleted file mode 100644 index fe0cd23cc2..0000000000 Binary files a/doc/imgs_results/whl/11_det.jpg and /dev/null differ diff --git a/doc/imgs_results/whl/11_det_rec.jpg b/doc/imgs_results/whl/11_det_rec.jpg deleted file mode 100644 index 31c566478f..0000000000 Binary files a/doc/imgs_results/whl/11_det_rec.jpg and /dev/null differ diff --git a/doc/imgs_results/whl/12_det.jpg b/doc/imgs_results/whl/12_det.jpg deleted file mode 100644 index 71627f0b8d..0000000000 Binary files a/doc/imgs_results/whl/12_det.jpg and /dev/null differ diff --git a/doc/imgs_results/whl/12_det_rec.jpg b/doc/imgs_results/whl/12_det_rec.jpg deleted file mode 100644 index 9db8b57e12..0000000000 Binary files a/doc/imgs_results/whl/12_det_rec.jpg and /dev/null differ diff --git a/doc/imgs_words/arabic/ar_1.jpg b/doc/imgs_words/arabic/ar_1.jpg deleted file mode 100644 index 71d7bf252d..0000000000 Binary files a/doc/imgs_words/arabic/ar_1.jpg and /dev/null differ diff --git a/doc/imgs_words/arabic/ar_2.jpg b/doc/imgs_words/arabic/ar_2.jpg deleted file mode 100644 index 017d3f6fbc..0000000000 Binary files a/doc/imgs_words/arabic/ar_2.jpg and /dev/null differ diff --git a/doc/imgs_words/belarusian/be_1.jpg b/doc/imgs_words/belarusian/be_1.jpg deleted file mode 100644 index fe0b8c4cba..0000000000 Binary files a/doc/imgs_words/belarusian/be_1.jpg and /dev/null differ diff --git a/doc/imgs_words/belarusian/be_2.jpg b/doc/imgs_words/belarusian/be_2.jpg deleted file mode 100644 index 5012ae4fb3..0000000000 Binary files a/doc/imgs_words/belarusian/be_2.jpg and /dev/null differ diff --git a/doc/imgs_words/bulgarian/bg_1.jpg b/doc/imgs_words/bulgarian/bg_1.jpg deleted file mode 100644 index 80905f34ed..0000000000 Binary files a/doc/imgs_words/bulgarian/bg_1.jpg and /dev/null differ diff --git a/doc/imgs_words/bulgarian/bg_2.jpg b/doc/imgs_words/bulgarian/bg_2.jpg deleted file mode 100644 index cda2c45eaf..0000000000 Binary files a/doc/imgs_words/bulgarian/bg_2.jpg and /dev/null differ diff --git a/doc/imgs_words/ch/word_1.jpg b/doc/imgs_words/ch/word_1.jpg deleted file mode 100644 index cb5451e15a..0000000000 Binary files a/doc/imgs_words/ch/word_1.jpg and /dev/null differ diff --git a/doc/imgs_words/ch/word_2.jpg b/doc/imgs_words/ch/word_2.jpg deleted file mode 100644 index 26ca6e59d4..0000000000 Binary files a/doc/imgs_words/ch/word_2.jpg and /dev/null differ diff --git a/doc/imgs_words/ch/word_3.jpg b/doc/imgs_words/ch/word_3.jpg deleted file mode 100644 index 22031ba501..0000000000 Binary files a/doc/imgs_words/ch/word_3.jpg and /dev/null differ diff --git a/doc/imgs_words/ch/word_4.jpg b/doc/imgs_words/ch/word_4.jpg deleted file mode 100644 index 2c34cd33ea..0000000000 Binary files a/doc/imgs_words/ch/word_4.jpg and /dev/null differ diff --git a/doc/imgs_words/ch/word_5.jpg b/doc/imgs_words/ch/word_5.jpg deleted file mode 100644 index 463f643581..0000000000 Binary files a/doc/imgs_words/ch/word_5.jpg and /dev/null differ diff --git a/doc/imgs_words/chinese_traditional/chinese_cht_1.png b/doc/imgs_words/chinese_traditional/chinese_cht_1.png deleted file mode 100644 index 04f436e140..0000000000 Binary files a/doc/imgs_words/chinese_traditional/chinese_cht_1.png and /dev/null differ diff --git a/doc/imgs_words/chinese_traditional/chinese_cht_2.png b/doc/imgs_words/chinese_traditional/chinese_cht_2.png deleted file mode 100644 index d76e1d9b87..0000000000 Binary files a/doc/imgs_words/chinese_traditional/chinese_cht_2.png and /dev/null differ diff --git a/doc/imgs_words/en/word_1.png b/doc/imgs_words/en/word_1.png deleted file mode 100644 index 7b915fd6da..0000000000 Binary files a/doc/imgs_words/en/word_1.png and /dev/null differ diff --git a/doc/imgs_words/en/word_2.png b/doc/imgs_words/en/word_2.png deleted file mode 100644 index 019cb80c96..0000000000 Binary files a/doc/imgs_words/en/word_2.png and /dev/null differ diff --git a/doc/imgs_words/en/word_3.png b/doc/imgs_words/en/word_3.png deleted file mode 100644 index f14e8d404c..0000000000 Binary files a/doc/imgs_words/en/word_3.png and /dev/null differ diff --git a/doc/imgs_words/en/word_4.png b/doc/imgs_words/en/word_4.png deleted file mode 100644 index 60fb82816c..0000000000 Binary files a/doc/imgs_words/en/word_4.png and /dev/null differ diff --git a/doc/imgs_words/en/word_5.png b/doc/imgs_words/en/word_5.png deleted file mode 100644 index 10fd19f1c2..0000000000 Binary files a/doc/imgs_words/en/word_5.png and /dev/null differ diff --git a/doc/imgs_words/french/1.jpg b/doc/imgs_words/french/1.jpg deleted file mode 100644 index 077ca28e70..0000000000 Binary files a/doc/imgs_words/french/1.jpg and /dev/null differ diff --git a/doc/imgs_words/french/2.jpg b/doc/imgs_words/french/2.jpg deleted file mode 100644 index 38a73caa62..0000000000 Binary files a/doc/imgs_words/french/2.jpg and /dev/null differ diff --git a/doc/imgs_words/german/1.jpg b/doc/imgs_words/german/1.jpg deleted file mode 100644 index d26ec9ed14..0000000000 Binary files a/doc/imgs_words/german/1.jpg and /dev/null differ diff --git a/doc/imgs_words/hindi/hi_1.jpg b/doc/imgs_words/hindi/hi_1.jpg deleted file mode 100644 index be4b6094bf..0000000000 Binary files a/doc/imgs_words/hindi/hi_1.jpg and /dev/null differ diff --git a/doc/imgs_words/hindi/hi_2.jpg b/doc/imgs_words/hindi/hi_2.jpg deleted file mode 100644 index bd7757915d..0000000000 Binary files a/doc/imgs_words/hindi/hi_2.jpg and /dev/null differ diff --git a/doc/imgs_words/italian/it_1.jpg b/doc/imgs_words/italian/it_1.jpg deleted file mode 100644 index 1e5d7a8770..0000000000 Binary files a/doc/imgs_words/italian/it_1.jpg and /dev/null differ diff --git a/doc/imgs_words/italian/it_2.jpg b/doc/imgs_words/italian/it_2.jpg deleted file mode 100644 index ddc63bf32d..0000000000 Binary files a/doc/imgs_words/italian/it_2.jpg and /dev/null differ diff --git a/doc/imgs_words/japan/1.jpg b/doc/imgs_words/japan/1.jpg deleted file mode 100644 index 6848797497..0000000000 Binary files a/doc/imgs_words/japan/1.jpg and /dev/null differ diff --git a/doc/imgs_words/kannada/ka_1.jpg b/doc/imgs_words/kannada/ka_1.jpg deleted file mode 100644 index 113c0910c3..0000000000 Binary files a/doc/imgs_words/kannada/ka_1.jpg and /dev/null differ diff --git a/doc/imgs_words/kannada/ka_2.jpg b/doc/imgs_words/kannada/ka_2.jpg deleted file mode 100644 index 8a0fa1a24e..0000000000 Binary files a/doc/imgs_words/kannada/ka_2.jpg and /dev/null differ diff --git a/doc/imgs_words/korean/1.jpg b/doc/imgs_words/korean/1.jpg deleted file mode 100644 index 48a89389ae..0000000000 Binary files a/doc/imgs_words/korean/1.jpg and /dev/null differ diff --git a/doc/imgs_words/korean/2.jpg b/doc/imgs_words/korean/2.jpg deleted file mode 100644 index b24f28914d..0000000000 Binary files a/doc/imgs_words/korean/2.jpg and /dev/null differ diff --git a/doc/imgs_words/marathi/mr_1.jpg b/doc/imgs_words/marathi/mr_1.jpg deleted file mode 100644 index a518e03cd3..0000000000 Binary files a/doc/imgs_words/marathi/mr_1.jpg and /dev/null differ diff --git a/doc/imgs_words/marathi/mr_2.jpg b/doc/imgs_words/marathi/mr_2.jpg deleted file mode 100644 index 0fe8557ed1..0000000000 Binary files a/doc/imgs_words/marathi/mr_2.jpg and /dev/null differ diff --git a/doc/imgs_words/nepali/ne_1.jpg b/doc/imgs_words/nepali/ne_1.jpg deleted file mode 100644 index 95b117f810..0000000000 Binary files a/doc/imgs_words/nepali/ne_1.jpg and /dev/null differ diff --git a/doc/imgs_words/nepali/ne_2.jpg b/doc/imgs_words/nepali/ne_2.jpg deleted file mode 100644 index cc6e45b1c2..0000000000 Binary files a/doc/imgs_words/nepali/ne_2.jpg and /dev/null differ diff --git a/doc/imgs_words/occitan/oc_1.jpg b/doc/imgs_words/occitan/oc_1.jpg deleted file mode 100644 index 41f680a73d..0000000000 Binary files a/doc/imgs_words/occitan/oc_1.jpg and /dev/null differ diff --git a/doc/imgs_words/occitan/oc_2.jpg b/doc/imgs_words/occitan/oc_2.jpg deleted file mode 100644 index 86203dc60e..0000000000 Binary files a/doc/imgs_words/occitan/oc_2.jpg and /dev/null differ diff --git a/doc/imgs_words/persian/fa_1.jpg b/doc/imgs_words/persian/fa_1.jpg deleted file mode 100644 index 07fdd69b21..0000000000 Binary files a/doc/imgs_words/persian/fa_1.jpg and /dev/null differ diff --git a/doc/imgs_words/persian/fa_2.jpg b/doc/imgs_words/persian/fa_2.jpg deleted file mode 100644 index 3dce3417a7..0000000000 Binary files a/doc/imgs_words/persian/fa_2.jpg and /dev/null differ diff --git a/doc/imgs_words/portuguese/pu_1.jpg b/doc/imgs_words/portuguese/pu_1.jpg deleted file mode 100644 index 2b089074e4..0000000000 Binary files a/doc/imgs_words/portuguese/pu_1.jpg and /dev/null differ diff --git a/doc/imgs_words/portuguese/pu_2.jpg b/doc/imgs_words/portuguese/pu_2.jpg deleted file mode 100644 index 50b74c41e0..0000000000 Binary files a/doc/imgs_words/portuguese/pu_2.jpg and /dev/null differ diff --git a/doc/imgs_words/russia/ru_1.jpg b/doc/imgs_words/russia/ru_1.jpg deleted file mode 100644 index d645acaee8..0000000000 Binary files a/doc/imgs_words/russia/ru_1.jpg and /dev/null differ diff --git a/doc/imgs_words/russia/ru_2.jpg b/doc/imgs_words/russia/ru_2.jpg deleted file mode 100644 index 4d99162450..0000000000 Binary files a/doc/imgs_words/russia/ru_2.jpg and /dev/null differ diff --git a/doc/imgs_words/serbian_cyrillic/rsc_1.jpg b/doc/imgs_words/serbian_cyrillic/rsc_1.jpg deleted file mode 100644 index d62cf4917f..0000000000 Binary files a/doc/imgs_words/serbian_cyrillic/rsc_1.jpg and /dev/null differ diff --git a/doc/imgs_words/serbian_cyrillic/rsc_2.jpg b/doc/imgs_words/serbian_cyrillic/rsc_2.jpg deleted file mode 100644 index 457bd3161c..0000000000 Binary files a/doc/imgs_words/serbian_cyrillic/rsc_2.jpg and /dev/null differ diff --git a/doc/imgs_words/serbian_latin/rs_1.jpg b/doc/imgs_words/serbian_latin/rs_1.jpg deleted file mode 100644 index c00ff8cc29..0000000000 Binary files a/doc/imgs_words/serbian_latin/rs_1.jpg and /dev/null differ diff --git a/doc/imgs_words/serbian_latin/rs_2.jpg b/doc/imgs_words/serbian_latin/rs_2.jpg deleted file mode 100644 index e6dbdf8ba2..0000000000 Binary files a/doc/imgs_words/serbian_latin/rs_2.jpg and /dev/null differ diff --git a/doc/imgs_words/spanish/xi_1.jpg b/doc/imgs_words/spanish/xi_1.jpg deleted file mode 100644 index 62e3f04266..0000000000 Binary files a/doc/imgs_words/spanish/xi_1.jpg and /dev/null differ diff --git a/doc/imgs_words/spanish/xi_2.jpg b/doc/imgs_words/spanish/xi_2.jpg deleted file mode 100644 index 16a073986d..0000000000 Binary files a/doc/imgs_words/spanish/xi_2.jpg and /dev/null differ diff --git a/doc/imgs_words/tamil/ta_1.jpg b/doc/imgs_words/tamil/ta_1.jpg deleted file mode 100644 index 1ee1504235..0000000000 Binary files a/doc/imgs_words/tamil/ta_1.jpg and /dev/null differ diff --git a/doc/imgs_words/tamil/ta_2.jpg b/doc/imgs_words/tamil/ta_2.jpg deleted file mode 100644 index 1554ca0287..0000000000 Binary files a/doc/imgs_words/tamil/ta_2.jpg and /dev/null differ diff --git a/doc/imgs_words/telugu/te_1.jpg b/doc/imgs_words/telugu/te_1.jpg deleted file mode 100644 index 125d6bfaee..0000000000 Binary files a/doc/imgs_words/telugu/te_1.jpg and /dev/null differ diff --git a/doc/imgs_words/telugu/te_2.jpg b/doc/imgs_words/telugu/te_2.jpg deleted file mode 100644 index eb18ea79b1..0000000000 Binary files a/doc/imgs_words/telugu/te_2.jpg and /dev/null differ diff --git a/doc/imgs_words/ukranian/uk_1.jpg b/doc/imgs_words/ukranian/uk_1.jpg deleted file mode 100644 index 4472227d00..0000000000 Binary files a/doc/imgs_words/ukranian/uk_1.jpg and /dev/null differ diff --git a/doc/imgs_words/ukranian/uk_2.jpg b/doc/imgs_words/ukranian/uk_2.jpg deleted file mode 100644 index 471f2961af..0000000000 Binary files a/doc/imgs_words/ukranian/uk_2.jpg and /dev/null differ diff --git a/doc/imgs_words/urdu/ur_1.jpg b/doc/imgs_words/urdu/ur_1.jpg deleted file mode 100644 index 55eb17462e..0000000000 Binary files a/doc/imgs_words/urdu/ur_1.jpg and /dev/null differ diff --git a/doc/imgs_words/urdu/ur_2.jpg b/doc/imgs_words/urdu/ur_2.jpg deleted file mode 100644 index 6ca44fa1c9..0000000000 Binary files a/doc/imgs_words/urdu/ur_2.jpg and /dev/null differ diff --git a/doc/imgs_words/uyghur/ug_1.jpg b/doc/imgs_words/uyghur/ug_1.jpg deleted file mode 100644 index 7e1736b220..0000000000 Binary files a/doc/imgs_words/uyghur/ug_1.jpg and /dev/null differ diff --git a/doc/imgs_words/uyghur/ug_2.jpg b/doc/imgs_words/uyghur/ug_2.jpg deleted file mode 100644 index 95763754b0..0000000000 Binary files a/doc/imgs_words/uyghur/ug_2.jpg and /dev/null differ diff --git a/doc/imgs_words_en/word_10.png b/doc/imgs_words_en/word_10.png deleted file mode 100644 index 07370f757e..0000000000 Binary files a/doc/imgs_words_en/word_10.png and /dev/null differ diff --git a/doc/imgs_words_en/word_116.png b/doc/imgs_words_en/word_116.png deleted file mode 100644 index fd000ff60b..0000000000 Binary files a/doc/imgs_words_en/word_116.png and /dev/null differ diff --git a/doc/imgs_words_en/word_19.png b/doc/imgs_words_en/word_19.png deleted file mode 100644 index d2a2859d02..0000000000 Binary files a/doc/imgs_words_en/word_19.png and /dev/null differ diff --git a/doc/imgs_words_en/word_201.png b/doc/imgs_words_en/word_201.png deleted file mode 100644 index 99abd73309..0000000000 Binary files a/doc/imgs_words_en/word_201.png and /dev/null differ diff --git a/doc/imgs_words_en/word_308.png b/doc/imgs_words_en/word_308.png deleted file mode 100644 index a8d094faff..0000000000 Binary files a/doc/imgs_words_en/word_308.png and /dev/null differ diff --git a/doc/imgs_words_en/word_336.png b/doc/imgs_words_en/word_336.png deleted file mode 100644 index 3bddd294ed..0000000000 Binary files a/doc/imgs_words_en/word_336.png and /dev/null differ diff --git a/doc/imgs_words_en/word_401.png b/doc/imgs_words_en/word_401.png deleted file mode 100644 index 0a4ee6935f..0000000000 Binary files a/doc/imgs_words_en/word_401.png and /dev/null differ diff --git a/doc/imgs_words_en/word_461.png b/doc/imgs_words_en/word_461.png deleted file mode 100644 index a73e5c494b..0000000000 Binary files a/doc/imgs_words_en/word_461.png and /dev/null differ diff --git a/doc/imgs_words_en/word_52.png b/doc/imgs_words_en/word_52.png deleted file mode 100644 index 493c590183..0000000000 Binary files a/doc/imgs_words_en/word_52.png and /dev/null differ diff --git a/doc/imgs_words_en/word_545.png b/doc/imgs_words_en/word_545.png deleted file mode 100644 index 5d4a2a7dea..0000000000 Binary files a/doc/imgs_words_en/word_545.png and /dev/null differ diff --git a/doc/install/linux/anaconda_download.png b/doc/install/linux/anaconda_download.png deleted file mode 100755 index 6ab6db3089..0000000000 Binary files a/doc/install/linux/anaconda_download.png and /dev/null differ diff --git a/doc/install/linux/conda_create.png b/doc/install/linux/conda_create.png deleted file mode 100755 index 533f592b7c..0000000000 Binary files a/doc/install/linux/conda_create.png and /dev/null differ diff --git a/doc/install/mac/anaconda_start.png b/doc/install/mac/anaconda_start.png deleted file mode 100755 index a860f5e56a..0000000000 Binary files a/doc/install/mac/anaconda_start.png and /dev/null differ diff --git a/doc/install/mac/conda_activate.png b/doc/install/mac/conda_activate.png deleted file mode 100755 index a2e6074e91..0000000000 Binary files a/doc/install/mac/conda_activate.png and /dev/null differ diff --git a/doc/install/mac/conda_create.png b/doc/install/mac/conda_create.png deleted file mode 100755 index 9ff10c241b..0000000000 Binary files a/doc/install/mac/conda_create.png and /dev/null differ diff --git a/doc/install/windows/Anaconda_download.png b/doc/install/windows/Anaconda_download.png deleted file mode 100644 index 83a0341493..0000000000 Binary files a/doc/install/windows/Anaconda_download.png and /dev/null differ diff --git a/doc/install/windows/anaconda_install_env.png b/doc/install/windows/anaconda_install_env.png deleted file mode 100644 index 7a22542712..0000000000 Binary files a/doc/install/windows/anaconda_install_env.png and /dev/null differ diff --git a/doc/install/windows/anaconda_install_folder.png b/doc/install/windows/anaconda_install_folder.png deleted file mode 100644 index e9fac29eaa..0000000000 Binary files a/doc/install/windows/anaconda_install_folder.png and /dev/null differ diff --git a/doc/install/windows/anaconda_prompt.png b/doc/install/windows/anaconda_prompt.png deleted file mode 100755 index 1087610ae0..0000000000 Binary files a/doc/install/windows/anaconda_prompt.png and /dev/null differ diff --git a/doc/install/windows/conda_list_env.png b/doc/install/windows/conda_list_env.png deleted file mode 100644 index 5ffa0037c5..0000000000 Binary files a/doc/install/windows/conda_list_env.png and /dev/null differ diff --git a/doc/install/windows/conda_new_env.png b/doc/install/windows/conda_new_env.png deleted file mode 100644 index eed667ec3d..0000000000 Binary files a/doc/install/windows/conda_new_env.png and /dev/null differ diff --git a/doc/joinus.PNG b/doc/joinus.PNG deleted file mode 100644 index aef92ec8d1..0000000000 Binary files a/doc/joinus.PNG and /dev/null differ diff --git a/doc/joinus_paddlex.jpg b/doc/joinus_paddlex.jpg deleted file mode 100644 index 6b70dda865..0000000000 Binary files a/doc/joinus_paddlex.jpg and /dev/null differ diff --git a/doc/pgnet_framework.png b/doc/pgnet_framework.png deleted file mode 100644 index 88fbca3947..0000000000 Binary files a/doc/pgnet_framework.png and /dev/null differ diff --git a/doc/ppocr_framework.png b/doc/ppocr_framework.png deleted file mode 100644 index ab51c88fe6..0000000000 Binary files a/doc/ppocr_framework.png and /dev/null differ diff --git a/doc/ppocr_v3/GTC.png b/doc/ppocr_v3/GTC.png deleted file mode 100644 index 30a9cdd146..0000000000 Binary files a/doc/ppocr_v3/GTC.png and /dev/null differ diff --git a/doc/ppocr_v3/GTC_en.png b/doc/ppocr_v3/GTC_en.png deleted file mode 100644 index a1a7fc5250..0000000000 Binary files a/doc/ppocr_v3/GTC_en.png and /dev/null differ diff --git a/doc/ppocr_v3/LCNet_SVTR.png b/doc/ppocr_v3/LCNet_SVTR.png deleted file mode 100644 index 7f0d701d27..0000000000 Binary files a/doc/ppocr_v3/LCNet_SVTR.png and /dev/null differ diff --git a/doc/ppocr_v3/LCNet_SVTR_en.png b/doc/ppocr_v3/LCNet_SVTR_en.png deleted file mode 100644 index 7890448470..0000000000 Binary files a/doc/ppocr_v3/LCNet_SVTR_en.png and /dev/null differ diff --git a/doc/ppocr_v3/LKPAN.png b/doc/ppocr_v3/LKPAN.png deleted file mode 100644 index 6b16053623..0000000000 Binary files a/doc/ppocr_v3/LKPAN.png and /dev/null differ diff --git a/doc/ppocr_v3/RSEFPN.png b/doc/ppocr_v3/RSEFPN.png deleted file mode 100644 index ddf7c52fb5..0000000000 Binary files a/doc/ppocr_v3/RSEFPN.png and /dev/null differ diff --git a/doc/ppocr_v3/SSL.png b/doc/ppocr_v3/SSL.png deleted file mode 100644 index 1344a2a77c..0000000000 Binary files a/doc/ppocr_v3/SSL.png and /dev/null differ diff --git a/doc/ppocr_v3/UDML.png b/doc/ppocr_v3/UDML.png deleted file mode 100644 index 3b59bc58bc..0000000000 Binary files a/doc/ppocr_v3/UDML.png and /dev/null differ diff --git a/doc/ppocr_v3/UIM.png b/doc/ppocr_v3/UIM.png deleted file mode 100644 index 7479bdf4a9..0000000000 Binary files a/doc/ppocr_v3/UIM.png and /dev/null differ diff --git a/doc/ppocr_v3/ppocrv3_det_cml.png b/doc/ppocr_v3/ppocrv3_det_cml.png deleted file mode 100644 index ccb5c8b21f..0000000000 Binary files a/doc/ppocr_v3/ppocrv3_det_cml.png and /dev/null differ diff --git a/doc/ppocr_v3/recconaug.png b/doc/ppocr_v3/recconaug.png deleted file mode 100644 index 899bc430de..0000000000 Binary files a/doc/ppocr_v3/recconaug.png and /dev/null differ diff --git a/doc/ppocr_v3/svtr_g2.png b/doc/ppocr_v3/svtr_g2.png deleted file mode 100644 index 2573afafbb..0000000000 Binary files a/doc/ppocr_v3/svtr_g2.png and /dev/null differ diff --git a/doc/ppocr_v3/svtr_g4.png b/doc/ppocr_v3/svtr_g4.png deleted file mode 100644 index f85d66d97f..0000000000 Binary files a/doc/ppocr_v3/svtr_g4.png and /dev/null differ diff --git a/doc/ppocr_v3/svtr_tiny.png b/doc/ppocr_v3/svtr_tiny.png deleted file mode 100644 index 01e22e74b5..0000000000 Binary files a/doc/ppocr_v3/svtr_tiny.png and /dev/null differ diff --git a/doc/ppocr_v3/teacher_dml.png b/doc/ppocr_v3/teacher_dml.png deleted file mode 100644 index ea09cacda8..0000000000 Binary files a/doc/ppocr_v3/teacher_dml.png and /dev/null differ diff --git a/doc/ppocr_v3/v3_rec_pipeline.png b/doc/ppocr_v3/v3_rec_pipeline.png deleted file mode 100644 index aa61cc4f16..0000000000 Binary files a/doc/ppocr_v3/v3_rec_pipeline.png and /dev/null differ diff --git a/doc/ppocr_v4/DF.png b/doc/ppocr_v4/DF.png deleted file mode 100644 index f14953d481..0000000000 Binary files a/doc/ppocr_v4/DF.png and /dev/null differ diff --git a/doc/ppocr_v4/PFHead.png b/doc/ppocr_v4/PFHead.png deleted file mode 100644 index 3728dc44e5..0000000000 Binary files a/doc/ppocr_v4/PFHead.png and /dev/null differ diff --git a/doc/ppocr_v4/multi_scale.png b/doc/ppocr_v4/multi_scale.png deleted file mode 100644 index 673d306399..0000000000 Binary files a/doc/ppocr_v4/multi_scale.png and /dev/null differ diff --git a/doc/ppocr_v4/ppocrv4_det_cml.png b/doc/ppocr_v4/ppocrv4_det_cml.png deleted file mode 100644 index 9132c0a67c..0000000000 Binary files a/doc/ppocr_v4/ppocrv4_det_cml.png and /dev/null differ diff --git a/doc/ppocr_v4/ppocrv4_framework.png b/doc/ppocr_v4/ppocrv4_framework.png deleted file mode 100644 index fa31f4c12e..0000000000 Binary files a/doc/ppocr_v4/ppocrv4_framework.png and /dev/null differ diff --git a/doc/ppocr_v4/ppocrv4_gtc.png b/doc/ppocr_v4/ppocrv4_gtc.png deleted file mode 100644 index 7e6a3f5c13..0000000000 Binary files a/doc/ppocr_v4/ppocrv4_gtc.png and /dev/null differ diff --git a/doc/ppocr_v4/v4_rec_pipeline.png b/doc/ppocr_v4/v4_rec_pipeline.png deleted file mode 100644 index b1ec7a9689..0000000000 Binary files a/doc/ppocr_v4/v4_rec_pipeline.png and /dev/null differ diff --git a/doc/ppocrv2_framework.jpg b/doc/ppocrv2_framework.jpg deleted file mode 100644 index e5f1a2ef47..0000000000 Binary files a/doc/ppocrv2_framework.jpg and /dev/null differ diff --git a/doc/ppocrv3_framework.png b/doc/ppocrv3_framework.png deleted file mode 100644 index e05279f7f5..0000000000 Binary files a/doc/ppocrv3_framework.png and /dev/null differ diff --git a/doc/ppocrv4_framework.png b/doc/ppocrv4_framework.png deleted file mode 100644 index fa31f4c12e..0000000000 Binary files a/doc/ppocrv4_framework.png and /dev/null differ diff --git a/doc/pr.png b/doc/pr.png deleted file mode 100644 index 3d0d15f78d..0000000000 Binary files a/doc/pr.png and /dev/null differ diff --git a/doc/precommit_pass.png b/doc/precommit_pass.png deleted file mode 100644 index 067fb75ddb..0000000000 Binary files a/doc/precommit_pass.png and /dev/null differ diff --git a/docs/index.en.md b/docs/index.en.md index b6de403e95..bf8b6c4a7a 100644 --- a/docs/index.en.md +++ b/docs/index.en.md @@ -7,7 +7,7 @@ hide: ---

- +

Chat diff --git a/docs/index.md b/docs/index.md index c472aabca5..0a5ab834fa 100644 --- a/docs/index.md +++ b/docs/index.md @@ -7,7 +7,7 @@ hide: ---

- +

Chat diff --git a/docs/ppocr/model_train/images/en_paper.jpg b/docs/ppocr/model_train/images/en_paper.jpg new file mode 100644 index 0000000000..c051d3fdb5 Binary files /dev/null and b/docs/ppocr/model_train/images/en_paper.jpg differ diff --git a/doc/tricks/long_text_examples.jpg b/docs/ppocr/model_train/images/long_text_examples.jpg similarity index 100% rename from doc/tricks/long_text_examples.jpg rename to docs/ppocr/model_train/images/long_text_examples.jpg diff --git a/docs/ppocr/model_train/training.md b/docs/ppocr/model_train/training.md index 4ce0e2dffa..6ef42c0922 100644 --- a/docs/ppocr/model_train/training.md +++ b/docs/ppocr/model_train/training.md @@ -63,13 +63,13 @@ Optimizer: 目前开源的模型,数据集和量级如下: - 检测: - - 英文数据集,ICDAR2015 - - 中文数据集,LSVT街景数据集训练数据3w张图片 + - 英文数据集,ICDAR2015 + - 中文数据集,LSVT街景数据集训练数据3w张图片 - 识别: - - 英文数据集,MJSynth和SynthText合成数据,数据量上千万。 - - 中文数据集,LSVT街景数据集根据真值将图crop出来,并进行位置校准,总共30w张图像。此外基于LSVT的语料,合成数据500w。 - - 小语种数据集,使用不同语料和字体,分别生成了100w合成数据集,并使用ICDAR-MLT作为验证集。 + - 英文数据集,MJSynth和SynthText合成数据,数据量上千万。 + - 中文数据集,LSVT街景数据集根据真值将图crop出来,并进行位置校准,总共30w张图像。此外基于LSVT的语料,合成数据500w。 + - 小语种数据集,使用不同语料和字体,分别生成了100w合成数据集,并使用ICDAR-MLT作为验证集。 其中,公开数据集都是开源的,用户可自行搜索下载,也可参考[中文数据集](../../datasets/datasets.md),合成数据暂不开源,用户可使用开源合成工具自行合成,可参考的合成工具包括[text_renderer](https://github.com/Sanster/text_renderer) 、[SynthText](https://github.com/ankush-me/SynthText) 、[TextRecognitionDataGenerator](https://github.com/Belval/TextRecognitionDataGenerator) 等。 @@ -101,17 +101,17 @@ c. 利用数据生成算法合成数据,例如pix2pix或[StyleText](https://gi **Q**:训练CRNN识别时,如何选择合适的网络输入shape? A:一般高度采用32,最长宽度的选择,有两种方法: - + (1)统计训练样本图像的宽高比分布。最大宽高比的选取考虑满足80%的训练样本。 - + (2)统计训练样本文字数目。最长字符数目的选取考虑满足80%的训练样本。然后中文字符长宽比近似认为是1,英文认为3:1,预估一个最长宽度。 **Q**:识别训练时,训练集精度已经到达90了,但验证集精度一直在70,涨不上去怎么办? A:训练集精度90,测试集70多的话,应该是过拟合了,有两个可尝试的方法: - + (1)加入更多的增广方式或者调大增广prob的[概率](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppocr/data/imaug/rec_img_aug.py#L341),默认为0.4。 - + (2)调大系统的[l2 dcay值](https://github.com/PaddlePaddle/PaddleOCR/blob/a501603d54ff5513fc4fc760319472e59da25424/configs/rec/ch_ppocr_v1.1/rec_chinese_lite_train_v1.1.yml#L47) **Q**: 识别模型训练时,loss能正常下降,但acc一直为0 diff --git a/docs/ppocr/model_train/tricks.en.md b/docs/ppocr/model_train/tricks.en.md new file mode 100644 index 0000000000..57ca27d922 --- /dev/null +++ b/docs/ppocr/model_train/tricks.en.md @@ -0,0 +1,65 @@ +--- +comments: true +--- + +### Tricks + +Here we have sorted out some Chinese OCR training and prediction tricks, which are being updated continuously. You are welcome to contribute more OCR tricks ~ + +#### 1、Replace Backbone Network + +- **Problem Description** + + At present, ResNet_vd series and MobileNetV3 series are the backbone networks used in PaddleOCR, whether replacing the other backbone networks will help to improve the accuracy? What should be paid attention to when replacing? + +- **Tips** + - Whether text detection or text recognition, the choice of backbone network is a trade-off between prediction effect and prediction efficiency. Generally, a larger backbone network is selected, e.g. ResNet101_vd, then the performance of the detection or recognition is more accurate, but the time cost will increase accordingly. And a smaller backbone network is selected, e.g. MobileNetV3_small_x0_35, the prediction speed is faster, but the accuracy of detection or recognition will be reduced. Fortunately, the detection or recognition effect of different backbone networks is positively correlated with the performance of ImageNet 1000 classification task. [**PaddleClas**](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.3/docs/en/models/models_intro_en.md) have sorted out the 23 series of classification network structures, such as ResNet_vd、Res2Net、HRNet、MobileNetV3、GhostNet. It provides the top1 accuracy of classification, the time cost of GPU(V100 and T4) and CPU(SD 855), and the 117 pretrained models [**download addresses**](https://paddleclas-en.readthedocs.io/en/latest/models/models_intro_en.html). + + - Similar as the 4 stages of ResNet, the replacement of text detection backbone network is to determine those four stages to facilitate the integration of FPN like the object detection heads. In addition, for the text detection problem, the pre trained model in ImageNet1000 can accelerate the convergence and improve the accuracy. + + - In order to replace the backbone network of text recognition, we need to pay attention to the descending position of network width and height stride. Since the ratio between width and height is large in chinese text recognition, the frequency of height decrease is less and the frequency of width decrease is more. You can refer the [modifies of MobileNetV3](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/ppocr/modeling/backbones/rec_mobilenet_v3.py) in PaddleOCR. + +#### 2、Long Chinese Text Recognition + +- **Problem Description** + The maximum resolution of Chinese recognition model during training is [3,32,320], if the text image to be recognized is too long, as shown in the figure below, how to adapt? + + ![img](./images/long_text_examples.jpg) + +- **Tips** + + During the training, the training samples are not directly resized to [3,32,320]. At first, the height of samples are resized to 32 and keep the ratio between the width and the height. When the width is less than 320, the excess parts are padding 0. Besides, when the ratio between the width and the height of the samples is larger than 10, these samples will be ignored. When the prediction for one image, do as above, but do not limit the max ratio between the width and the height. When the prediction for an images batch, do as training, but the resized target width is the longest width of the images in the batch. [Code as following](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/tools/infer/predict_rec.py): + + ```python linenums="1" + def resize_norm_img(self, img, max_wh_ratio): + imgC, imgH, imgW = self.rec_image_shape + assert imgC == img.shape[2] + if self.character_type == "ch": + imgW = int((32 * max_wh_ratio)) + h, w = img.shape[:2] + ratio = w / float(h) + if math.ceil(imgH * ratio) > imgW: + resized_w = imgW + else: + resized_w = int(math.ceil(imgH * ratio)) + resized_image = cv2.resize(img, (resized_w, imgH)) + resized_image = resized_image.astype('float32') + resized_image = resized_image.transpose((2, 0, 1)) / 255 + resized_image -= 0.5 + resized_image /= 0.5 + padding_im = np.zeros((imgC, imgH, imgW), dtype=np.float32) + padding_im[:, :, 0:resized_w] = resized_image + return padding_im + ``` + +#### 3、Space Recognition + +- **Problem Description** + + As shown in the figure below, for Chinese and English mixed scenes, in order to facilitate reading and using the recognition results, it is often necessary to recognize the spaces between words. How can this situation be adapted? + + ![img](./images/en_paper.jpg) + +- **Tips** + + There are two possible methods for space recognition. (1) Optimize the text detection. For spliting the text at the space in detection results, it needs to divide the text line with space into many segments when label the data for detection. (2) Optimize the text recognition. The space character is introduced into the recognition dictionary. Label the blank line in the training data for text recognition. In addition, we can also concat multiple word lines to synthesize the training data with spaces. PaddleOCR currently uses the second method. diff --git a/docs/ppocr/model_train/tricks.md b/docs/ppocr/model_train/tricks.md new file mode 100644 index 0000000000..71ec3fcce4 --- /dev/null +++ b/docs/ppocr/model_train/tricks.md @@ -0,0 +1,65 @@ +--- +comments: true +--- + +### 技巧 + +这里我们整理了一些中文OCR训练和预测技巧,持续更新中,欢迎大家贡献更多OCR技巧~ + +#### 1、更换骨干网络 + +- **问题描述** + + 目前PaddleOCR使用的主干网络为ResNet_vd系列和MobileNetV3系列,更换其他主干网络是否有助于提高准确率?更换时需要注意什么? + +- **技巧** + - 无论是文本检测还是文本识别,主干网络的选择都是预测效果和预测效率的权衡。一般选择较大的主干网络,如ResNet101_vd,则检测或识别的性能更准确,但时间成本也会相应增加。而选择较小的主干网络,如MobileNetV3_small_x0_35,预测速度更快,但检测或识别的准确率会降低。幸运的是,不同骨干网络的检测或识别效果与ImageNet 1000分类任务的性能呈正相关。[**PaddleClas**](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.3/docs/en/models/models_intro_en.md)整理了ResNet_vd、Res2Net、HRNet、MobileNetV3、GhostNet等23个系列的分类网络结构,提供了分类top1准确率、GPU(V100和T4)和CPU(SD 855)的时间成本,以及117个预训练模型[**下载地址**](https://paddleclas-en.readthedocs.io/en/latest/models/models_intro_en.html)。 + +- 和ResNet的4个阶段类似,文本检测骨干网络的更换就是确定这4个阶段,以便于像物体检测heads一样集成FPN。另外,对于文本检测问题,ImageNet1000中的预训练模型可以加速收敛并提高准确率。 + +- 更换文本识别骨干网络时,需要注意网络宽度和高度步长的下降位置。由于中文文本识别中宽度和高度的比值较大,因此高度下降的频率较少,宽度下降的频率较多。可以参考PaddleOCR中[MobileNetV3的修改](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/ppocr/modeling/backbones/rec_mobilenet_v3.py)。 + +#### 2、长中文文本识别 + +- **问题描述** + 中文识别模型在训练时的最大分辨率为[3,32,320],如果待识别的文本图像过长,如下图所示,该如何适配? + + ![img](./images/long_text_examples.jpg) + +- **小技巧** + +在训练时,不要直接将训练样本resize到[3,32,320],先将样本的高度resize为32,并保持宽高比,当宽度小于320时,超出部分用0填充。另外,当样本的宽高比大于10时,这些样本将被忽略。对一张图片进行预测时,同上,但不限制最大宽高比。对一批图像进行预测时,按照训练的方式进行,但调整后的目标宽度是该批图像的最长宽度。 [代码如下](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/tools/infer/predict_rec.py): + + ```python linenums="1" + def resize_norm_img(self, img, max_wh_ratio): + imgC, imgH, imgW = self.rec_image_shape + assert imgC == img.shape[2] + if self.character_type == "ch": + imgW = int((32 * max_wh_ratio)) + h, w = img.shape[:2] + ratio = w / float(h) + if math.ceil(imgH * ratio) > imgW: + resized_w = imgW + else: + resized_w = int(math.ceil(imgH * ratio)) + resized_image = cv2.resize(img, (resized_w, imgH)) + resized_image = resized_image.astype('float32') + resized_image = resized_image.transpose((2, 0, 1)) / 255 + resized_image -= 0.5 + resized_image /= 0.5 + padding_im = np.zeros((imgC, imgH, imgW), dtype=np.float32) + padding_im[:, :, 0:resized_w] = resized_image + return padding_im + ``` + +#### 3、空格识别 + +- **问题描述** + +如下图所示,对于中英文混合场景,为了方便阅读和使用识别结果,经常需要识别单词之间的空格,这种情况该如何适配? + +![img](./images/en_paper.jpg) + +- **小技巧** + +空格识别有两种可能的方法。(1)优化文本检测。为了将检测结果中的文本分割在空格处,在对数据进行标记时,需要将带有空格的文本行分成许多段 diff --git a/mkdocs.yml b/mkdocs.yml index 87c43b1854..3ccbfe3880 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -119,6 +119,7 @@ plugins: 文本方向分类器: Text Angle Classification 关键信息提取: Key Information Extraction 模型微调: Fine-tune + 训练tips: Training Tricks 模型压缩: Model Compression 模型量化: Model Quantization 模型裁剪: Model Prune @@ -281,6 +282,7 @@ nav: - 文本方向分类器: ppocr/model_train/angle_class.md - 关键信息提取: ppocr/model_train/kie.md - 模型微调: ppocr/model_train/finetune.md + - 训练tricks: ppocr/model_train/tricks.md - 模型压缩: - 模型量化: ppocr/model_compress/quantization.md - 模型裁剪: ppocr/model_compress/prune.md diff --git a/tests/test_cls_postprocess.py b/tests/test_cls_postprocess.py index 51fbd8a647..668a1e53fc 100644 --- a/tests/test_cls_postprocess.py +++ b/tests/test_cls_postprocess.py @@ -1,8 +1,9 @@ import os import sys -import pytest -import paddle + import numpy as np +import paddle +import pytest current_dir = os.path.dirname(os.path.abspath(__file__)) sys.path.append(os.path.abspath(os.path.join(current_dir, ".."))) diff --git a/doc/imgs_en/254.jpg b/tests/test_files/254.jpg similarity index 100% rename from doc/imgs_en/254.jpg rename to tests/test_files/254.jpg diff --git a/doc/imgs_en/img_10.jpg b/tests/test_files/img_10.jpg similarity index 100% rename from doc/imgs_en/img_10.jpg rename to tests/test_files/img_10.jpg diff --git a/tests/test_files/ppstructure/1.png b/tests/test_files/ppstructure/1.png new file mode 100644 index 0000000000..faff6e3178 Binary files /dev/null and b/tests/test_files/ppstructure/1.png differ diff --git a/tests/test_files/ppstructure/layout.jpg b/tests/test_files/ppstructure/layout.jpg new file mode 100644 index 0000000000..c5c39dac72 Binary files /dev/null and b/tests/test_files/ppstructure/layout.jpg differ diff --git a/tests/test_paddleocr_api.py b/tests/test_paddleocr.py similarity index 59% rename from tests/test_paddleocr_api.py rename to tests/test_paddleocr.py index 0af794d2b8..968d74f977 100644 --- a/tests/test_paddleocr_api.py +++ b/tests/test_paddleocr.py @@ -1,15 +1,15 @@ +# -*- encoding: utf-8 -*- +from pathlib import Path from typing import Any import pytest -from paddleocr import PaddleOCR, PPStructure +from paddleocr import PaddleOCR -# Test image paths -IMAGE_PATHS_OCR = ["./doc/imgs_en/254.jpg", "./doc/imgs_en/img_10.jpg"] -IMAGE_PATHS_STRUCTURE = [ - "./ppstructure/docs/table/layout.jpg", - "./ppstructure/docs/table/1.png", -] +cur_dir = Path(__file__).resolve().parent +test_file_dir = cur_dir / "test_files" + +IMAGE_PATHS_OCR = [str(test_file_dir / "254.jpg"), str(test_file_dir / "img_10.jpg")] @pytest.fixture(params=["en", "ch"]) @@ -76,41 +76,3 @@ def test_ocr_rec_only(ocr_engine: PaddleOCR, image_path: str) -> None: result = ocr_engine.ocr(image_path, det=False, rec=True) assert result is not None assert isinstance(result, list) - - -@pytest.fixture(params=["en", "ch"]) -def structure_engine(request: Any) -> PPStructure: - """ - Initialize PPStructure engine with different languages. - - Args: - request: pytest fixture request object. - - Returns: - An instance of PPStructure. - """ - return PPStructure(lang=request.param) - - -def test_structure_initialization(structure_engine: PPStructure) -> None: - """ - Test PPStructure initialization. - - Args: - structure_engine: An instance of PPStructure. - """ - assert structure_engine is not None - - -@pytest.mark.parametrize("image_path", IMAGE_PATHS_STRUCTURE) -def test_structure_function(structure_engine: PPStructure, image_path: str) -> None: - """ - Test PPStructure structure analysis functionality with different images. - - Args: - structure_engine: An instance of PPStructure. - image_path: Path to the image to be processed. - """ - result = structure_engine(image_path) - assert result is not None - assert isinstance(result, list) diff --git a/tests/test_ppstructure.py b/tests/test_ppstructure.py new file mode 100644 index 0000000000..eb56dd33fe --- /dev/null +++ b/tests/test_ppstructure.py @@ -0,0 +1,53 @@ +# -*- encoding: utf-8 -*- +from pathlib import Path +from typing import Any + +import pytest + +from paddleocr import PPStructure + +cur_dir = Path(__file__).resolve().parent +test_file_dir = cur_dir / "test_files" + +IMAGE_PATHS_STRUCTURE = [ + str(test_file_dir / "ppstructure" / "layout.jpg"), + str(test_file_dir / "ppstructure" / "1.png"), +] + + +@pytest.fixture(params=["en", "ch"]) +def structure_engine(request: Any) -> PPStructure: + """ + Initialize PPStructure engine with different languages. + + Args: + request: pytest fixture request object. + + Returns: + An instance of PPStructure. + """ + return PPStructure(lang=request.param) + + +def test_structure_initialization(structure_engine: PPStructure) -> None: + """ + Test PPStructure initialization. + + Args: + structure_engine: An instance of PPStructure. + """ + assert structure_engine is not None + + +@pytest.mark.parametrize("image_path", IMAGE_PATHS_STRUCTURE) +def test_structure_function(structure_engine: PPStructure, image_path: str) -> None: + """ + Test PPStructure structure analysis functionality with different images. + + Args: + structure_engine: An instance of PPStructure. + image_path: Path to the image to be processed. + """ + result = structure_engine(image_path) + assert result is not None + assert isinstance(result, list)