收藏切换
PetS: A Scalable Inference Serving System for Parameter-Efficient Transformer
收藏切换
Xuechao Wei1, 4, Zhe Zhou2, Yinghui Xu3, Jiejing Zhang4, Yuan Xie5, Guangyu Sun2, *
Journal of Computer Research and Development | 2025, 62(7) : 1816 - 1832
Less
收藏切换
Journal of Computer Research and Development | 2025, 62(7): 1816-1832
• High Performance Computing •
PetS: A Scalable Inference Serving System for Parameter-Efficient Transformer
Full
Xuechao Wei1, 4, Zhe Zhou2, Yinghui Xu3, Jiejing Zhang4, Yuan Xie5, Guangyu Sun2, *
Affiliations
  • 1School of Electronics Engineering & Computer Science, Peking University, Beijing 100871
  • 2School of Integrated Circuits, Peking University, Beijing 100871
  • 3Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai 201203
  • 4Alibaba DAMO Academy, Hangzhou 310023
  • 5Department of Electronic & Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong 999077
  • born in 1988. PhD. His main research interests include computer architecture and electronic design automation

    born in 1997. PhD. His main research interests include computer architecture, near data processing, domain specific accelerator, machine learning system and interconnection techniques

    born in 1973. PhD. professor, PhD supervisor. His main research interests include generative AI, logic reasoning, and large language model for life science

    born in 1986. Bachelor. Senior Member of Technical Staff. Director of Alibaba Tongyi Lab heterogeneous computing team. His main research interest includes deep learning inference

    born in 1974. PhD. IEEE fellow, ACM fellow, AAAS fellow, Chair professor. His main research interests include computer architecture, VLSI, electronic design automation, machine learning

    born in 1981. PhD, professor, PhD supervisor. His main research interests include electronic design automation for domain specific architecture and new devices, architecture/circuit/device codesign and automation

About Author:

born in 1988. PhD. His main research interests include computer architecture and electronic design automation

born in 1997. PhD. His main research interests include computer architecture, near data processing, domain specific accelerator, machine learning system and interconnection techniques

born in 1973. PhD. professor, PhD supervisor. His main research interests include generative AI, logic reasoning, and large language model for life science

born in 1986. Bachelor. Senior Member of Technical Staff. Director of Alibaba Tongyi Lab heterogeneous computing team. His main research interest includes deep learning inference

born in 1974. PhD. IEEE fellow, ACM fellow, AAAS fellow, Chair professor. His main research interests include computer architecture, VLSI, electronic design automation, machine learning

born in 1981. PhD, professor, PhD supervisor. His main research interests include electronic design automation for domain specific architecture and new devices, architecture/circuit/device codesign and automation

Published: 2025-07-01 doi: 10.7544/issn1000-1239.202440206
Outline
收藏切换

Deploying Transformer models under the conventional pre-train-then-fine-tune paradigm is challenging for multi-task serving, because a full model copy for each downstream task must be maintained, quickly exhausting the storage budget. Recent algorithmic advances in parameter-efficient Transformer (PET) have shown enormous potential to mitigate the storage overhead. They share the pre-trained model among tasks and only fine-tune a small portion of task-specific parameters. Unfortunately, existing serving systems neither have flexible PET task management mechanisms nor can efficiently serve queries to different tasks in batches. Therefore, we propose PetS, a unified framework for multi-task PET serving. Specifically, different PET tasks are expressed by a unified representation in the same framework, which enables flexible PET task management. Based on the unified representation, we design a specialized PET inference engine to batch different tasks’ queries together and execute them with task-agnostic shared operators and task-specific PET operators. Equipped with the PET inference engine, PetS is more scalable with respect to the number of tasks on a single GPU device. To further improve system throughput, we propose a coordinated batching strategy taking query length, PET task type as well as system load balancing together into consideration. To improve the throughput on multiple GPU instances, we also propose a PET-migration based load balancing strategy. We evaluate PetS on platforms with single GPU, including Edge/Desktop/Server GPUs. Comprehensive experiments demonstrate that PetS supports up to 26 times more concurrent tasks and improves the serving throughput by 1.53 times and 1.63 times on desktop and server GPU nodes, respectively. On multiple GPUs, our load-balancing strategy also provides up to 29% speedup.

inference serving  /  parameter-efficient Transformer (PET)  /  GPU  /  distributed system  /  machine-learning system
魏学超, 周哲, 徐盈辉, 张洁靖, 谢源, 孙广宇. PetS: 针对参数高效Transformer模型的可扩展推理服务系统. 计算机研究与发展, 2025 , 62 (7) : 1816 -1832 . DOI: 10.7544/issn1000-1239.202440206
Xuechao Wei, Zhe Zhou, Yinghui Xu, Jiejing Zhang, Yuan Xie, Guangyu Sun. PetS: A Scalable Inference Serving System for Parameter-Efficient Transformer[J]. Journal of Computer Research and Development, 2025 , 62 (7) : 1816 -1832 . DOI: 10.7544/issn1000-1239.202440206
Year 2025 volume 62 Issue 7
115
40
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202440206
  • Receive Date:2024-03-21
  • Online Date:2026-11-10
  • Published:2025-07-01
Article Data
Affiliations
History
  • Received:2024-03-21
  • Revised:2025-02-18
  • Accepted:2025-03-03
Affiliations
    1School of Electronics Engineering & Computer Science, Peking University, Beijing 100871
    2School of Integrated Circuits, Peking University, Beijing 100871
    3Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai 201203
    4Alibaba DAMO Academy, Hangzhou 310023
    5Department of Electronic & Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong 999077
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440206
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT