收藏切换
An Adaptive Erasure-Coded Data Access Method for Cross-Cloud Collaborative Scheduling of Storage and Computation
收藏切换
Kaixin Zhang, Yijie Wang*, Han Bao, Junhui Kan
Journal of Computer Research and Development | 2024, 61(3) : 571 - 588
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(3): 571-588
• Special Issue on Intelligent Storage and Data Analysis for the Digital Economy •
An Adaptive Erasure-Coded Data Access Method for Cross-Cloud Collaborative Scheduling of Storage and Computation
Full
Kaixin Zhang, Yijie Wang*, Han Bao, Junhui Kan
Affiliations
  • 1National Key Laboratory of Parallel and Distributed Computing (National University of Defense Technology), Changsha 410073
  • 2College of Computer Science and Technology, National University of Defense Technology, Changsha 410073
  • born in 2002. Master candidate. His main research interests include cloud storage and erasure coding

    born in 1971. PhD. professor, PhD supervisor. Distinguished member of CCF. Her main research interests include distributed storage, big data analysis, and cloud computing

    born in 1992. PhD. His main research interests include cloud storage and erasure coding

    born in 2000. Master candidate. His main research interests include cloud storage and collaborative scheduling of storage and computation

About Author:

born in 2002. Master candidate. His main research interests include cloud storage and erasure coding

born in 1971. PhD. professor, PhD supervisor. Distinguished member of CCF. Her main research interests include distributed storage, big data analysis, and cloud computing

born in 1992. PhD. His main research interests include cloud storage and erasure coding

born in 2000. Master candidate. His main research interests include cloud storage and collaborative scheduling of storage and computation

Published: 2024-03-02 doi: 10.7544/issn1000-1239.202330541
Outline
收藏切换

Nowadays, the increasing demand for cross-cloud collaborative scheduling of storage and computation puts high demands on cross-cloud data access speed. Therefore, cross-cloud data access methods based on data redundancy techniques (erasure coding and multiple-duplicate) with high cross-cloud data access speed are gaining attention. Among them, the cross-cloud data access method based on erasure coding has become a hot research topic because of its low storage overhead and high fault tolerance. In order to improve the data access speed by shortening the transmission time of coded blocks, existing cross-cloud data access methods based on erasure coding introduce caching techniques and optimize the coded data access scheme. However, due to the coarse granularity of cache management and the lack of coordinated optimization of cache management and coded data access scheme, the existing methods suffer from low cache hits, low cache hit efficiency, and high access volume of coded blocks with low transmission speed, which prolong the coded block transmission time. To this end, we first propose an IPFS-based cross-cloud storage system framework (IBCS) that can realize fine-grained cache management based on IPFS data slice management mechanism, and thus can improve cache hits. Then, we propose an adaptive erasure-coded data access method for cross-cloud collaborative scheduling of storage and computation (AECAM) that evaluates the transmission speed of each coded block during data access based on the distribution of coded blocks (including cached coded blocks) and data access nodes, and accordingly formulates a coded data access scheme that can avoid accessing low transmission speed coded blocks. In addition, AECAM identifies coded blocks that are easily selected in the coded data access scheme and have low transmission speed, and caches them near the data access nodes, thus improving both cache hits and hit efficiency. We build a cross-cloud storage system for collaborative scheduling of storage and computation (C2S2) based on IBCS and AECAM. Compared with existing erasure-coded storage systems that introduce caching, experiments in a cross-cloud environment show that C2S2 can improve data access speed by 75.22%−81.29%.

cross-cloud collaborative scheduling of storage and computation  /  erasure coding  /  data access technology  /  IPFS  /  cache
张凯鑫, 王意洁, 包涵, 阚浚晖. 面向存算联调的跨云纠删码自适应数据访问方法. 计算机研究与发展, 2024 , 61 (3) : 571 -588 . DOI: 10.7544/issn1000-1239.202330541
Kaixin Zhang, Yijie Wang, Han Bao, Junhui Kan. An Adaptive Erasure-Coded Data Access Method for Cross-Cloud Collaborative Scheduling of Storage and Computation[J]. Journal of Computer Research and Development, 2024 , 61 (3) : 571 -588 . DOI: 10.7544/issn1000-1239.202330541
Year 2024 volume 61 Issue 3
79
38
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202330541
  • Receive Date:2023-06-21
  • Online Date:2026-11-17
  • Published:2024-03-02
Article Data
Affiliations
History
  • Received:2023-06-21
  • Revised:2023-12-04
Affiliations
    1National Key Laboratory of Parallel and Distributed Computing (National University of Defense Technology), Changsha 410073
    2College of Computer Science and Technology, National University of Defense Technology, Changsha 410073
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330541
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT