收藏切换
Detecting Pretraining Data for Large Language Models via Long-Range Context
收藏切换
Weichao Zhang1, 2, 3, Ruqing Zhang1, 2, *, Jiafeng Guo1, 2, Yixing Fan1, 2
Journal of Computer Research and Development | 2025, 62(12) : 2993 - 3001
Less
收藏切换
Journal of Computer Research and Development | 2025, 62(12): 2993-3001
• Artificial Intelligence •
Detecting Pretraining Data for Large Language Models via Long-Range Context
Full
Weichao Zhang1, 2, 3, Ruqing Zhang1, 2, *, Jiafeng Guo1, 2, Yixing Fan1, 2
Affiliations
  • 1CAS Key Laboratory of Network Data Science and Technology (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
  • 2University of Chinese Academy of Sciences, Beijing 100049
  • 3Zhongguancun Laboratory, Beijing 100094
  • born in 2000. PhD candidate. His main research interest includes security of large language models

    born in 1994. PhD, associate professor. Her main research interests include natural language processing and information retrieval

    born in 1980. PhD, professor. His main research interests include representation learning, and neural models for information retrieval and filtering

    born in 1990. PhD, associate professor. His main research interests include data mining and information retrieval

About Author:

born in 2000. PhD candidate. His main research interest includes security of large language models

born in 1994. PhD, associate professor. Her main research interests include natural language processing and information retrieval

born in 1980. PhD, professor. His main research interests include representation learning, and neural models for information retrieval and filtering

born in 1990. PhD, associate professor. His main research interests include data mining and information retrieval

Published: 2025-12-01 doi: 10.7544/issn1000-1239.202440780
Outline
收藏切换

Pretraining data detection aims to determine whether a piece of text belongs to the pretraining data of a large language model (LLM) when the model’s pretraining data are not publicly disclosed, which can be used to audit whether the pretraining data usage complies with legal regulations. Existing methods generally assume that an LLM tends to assign higher token probabilities to training texts compared with non-training texts, and thus identify texts with high probabilities as training texts. However, due to the significant overlap in fragments between training texts and non-training texts, an LLM may also assign relatively high token probabilities to non-training texts, which makes existing methods prone to misclassifying non-training texts as training texts. Inspired by research on the memorization capabilities of LLMs, we propose a novel method to mitigate this issue by comparing the token probabilities given the full context with those given the short-range context, and the contribution of long-range context to token probability increase is then computed, with a higher contribution indicating a greater likelihood of the text being part of the pretraining data. The key idea is that when an LLM predicts the token probabilities for training texts, the contribution of long-range context to token probability increase is greater than that for non-training texts. Experimental results on multiple datasets demonstrate the effectiveness of the proposed method. The code is available at https://github.com/zhang-wei-chao/Long-Range-Context-for-PDD.

large language model  /  model memorization capability  /  training data copyright  /  membership inference attack  /  data contamination
张伟超, 张儒清, 郭嘉丰, 范意兴. 基于长距离上下文的大语言模型预训练数据检测方法. 计算机研究与发展, 2025 , 62 (12) : 2993 -3001 . DOI: 10.7544/issn1000-1239.202440780
Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Yixing Fan. Detecting Pretraining Data for Large Language Models via Long-Range Context[J]. Journal of Computer Research and Development, 2025 , 62 (12) : 2993 -3001 . DOI: 10.7544/issn1000-1239.202440780
Year 2025 volume 62 Issue 12
103
46
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202440780
  • Receive Date:2024-10-10
  • Online Date:2026-11-10
  • Published:2025-12-01
Article Data
Affiliations
History
  • Received:2024-10-10
  • Revised:2025-04-07
Affiliations
    1CAS Key Laboratory of Network Data Science and Technology (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
    2University of Chinese Academy of Sciences, Beijing 100049
    3Zhongguancun Laboratory, Beijing 100094
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440780
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT