born in 2000. Undergraduate. His main research interests include high performance computing and operating systems
born in 1994. PhD. His main research interests include machine learning, privacy protection, and intelligent operating systems
born in 1997. Master. His main research interests include high performance computing and operating systems
born in 1999. Master. His main research interests include high performance computing and operating systems
born in 1984. PhD candidate. His main research interests include artificial intelligence and operating system
born in 1986. PhD, senior engineer. His main research interests include high-performance computing, operating system, and machine learning
born in 1975. PhD, senior engineer. His main research interests include system security, software vulnerability analysis, and artificial intelligence
born in 1979. PhD, professor. Her main research interest includes parallel algorithms and applications
born in 1957. Academician of Chinese Academy of Engineering. Her main research interests include software theory, operating systems, and information security
Transformer has fundamentally revolutionized various fields of natural language processing through its self-attention mechanism that effectively models long-range contextual relationships. It overcomes traditional neural networks’ limitations in handling long sequences, and has emerged as the mainstream architecture for modern large language models. Currently, Transformer inference systems generally face a shortage of computing power and are commonly confronted with challenges like substantial scheduling latency, concurrency execution difficulties, and severe memory access limitations during deployment. The Sunway AI accelerator card, featuring a domestically-developed next-generation intelligent processor, has successfully empowered diverse edge inference deployments. However, conventional inference frameworks struggle to effectively explore the architectural potential in three critical dimensions: robust system control capabilities, multi-level parallel computing, and rich memory hierarchy. We propose SW-Inf: a high-efficiency Transformer inference method based on the Sunway AI accelerator card. First, an on-chip management processing element (MPE)-orchestrated computation graph scheduling mechanism is proposed to minimize kernel launch latency and inter-process data transfer overhead through graph aggregation and MPE coordination. Second, an LDM-prioritized heuristic tiling strategy is introduced to enable load-balanced parallel computation across processing elements. Finally, a hardware-aware tightly-coupled attention computation implementation is developed to eliminate redundant intermediate data movement between registers and main memory. We implement an end-to-end inference system on the Sunway AI accelerator card. Experimental results demonstrate that compared with the vLLM inference system on NVIDIA V100S which has higher memory bandwidth than the Sunway AI accelerator card, SW-Inf achieves 15% higher throughput and 20% better memory utilization rate.
| 科 Family | 属数 Number of genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) | 属 Genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) |
|---|---|---|---|---|---|---|
| 鹅膏菌科Amanitaceae | 2 | 11 | 5.26 | 鹅膏菌属 Amanita | 10 | 4.78 |
| 小菇科 Mycenaceae | 2 | 12 | 5.74 | 丝盖伞属 Inocybe | 5 | 2.39 |
| 多孔菌科 Polyporaceae | 8 | 14 | 6.70 | 蜡蘑属 Laccaria | 5 | 2.39 |
| 红菇科 Russulaceae | 3 | 23 | 11.00 | 小皮伞属 Marasmius | 6 | 2.87 |
| 小菇属 Mycena | 11 | 5.26 | ||||
| 光柄菇属 Pluteus | 5 | 2.39 | ||||
| 红菇属 Russula | 17 | 8.13 | ||||
| 栓菌属 Trametes | 5 | 2.39 |