Science & Technology Review
|
2024, 42(23): 135-144
• Papers •
Automatic sentence segmentation and word segmentation for Liye Qin Bamboo manuscripts based on CRF model
Full
FENG Huimin1,2, GUO Shuaishuai2, LIU Ming2
Affiliations
1. Department of Basic Courses, Shandong Agricultural Engineering University, Jinan 250100, China;
2. Institute for Advanced Study in History of Science, Northwest University, Xi'an 710127, China
Published: 2024-12-13
doi: 10.3981/j.issn.1000-7857.2023.05.00812
Outline
Information processing of ancient Chinese seldom uses unearthed documents as corpus to carry out relevant research. The number of Liye Qin bamboo manuscripts reached ten times that of all the Qin slips unearthed before, which can fill many gaps in the historical records of the Qin Dynasty. In this paper, we used them as experimental corpus and explored the automatic sentence segmentation and word segmentation of unearthed documents based on the CRF model. We combined the actual characteristics of the corpus and set up different feature templates to verify the generalization ability of model sequence labeling on different tasks. We set up a joint approach to sentence segmentation and word segmentation as comparative experiment to select a better performance processing plan. At the same time, a comparative experiment was designed between deep learning methods and pretrained models. The results proved that the overall performance of the joint approach in each task was improved and that the F1-score of automatic sentence segmentation and word segmentation reached 75.79% and 94.44%, respectively. Since it's faster and takes less time, this approach is more suitable for the Liye Qin bamboo slips. The research results can serve the proofreading work of the last three volumes of Liye Qin bamboo slips and the in-depth processing and construction of the corpus.
CRF model
/
Liye Qin bamboo manuscripts
/
automatic sentence segmentation
/
automatic word segmentation
FENG Huimin, GUO Shuaishuai, LIU Ming.
Automatic sentence segmentation and word segmentation for Liye Qin Bamboo manuscripts based on CRF model[J].
Science & Technology Review,
2024
, 42
(23)
: 135
-144
.
DOI: 10.3981/j.issn.1000-7857.2023.05.00812
Year 2024 volume 42 Issue 23
PDF
720
241
Cite this Article
BibTeX
Article Info
doi: 10.3981/j.issn.1000-7857.2023.05.00812
- Receive Date:2023-05-29
- Online Date:2025-01-06
- Published:2024-12-13