收藏切换
Progress in audio-driven digital human technologies
收藏切换
Weibin Li1, 2, *, Jiafeng Gao1, Bing Xu3, Biao Hou2, Licheng Jiao2
Journal of Image and Graphics | 2026, 31(7) : 2317 - 2338
Less
收藏切换
Journal of Image and Graphics | 2026, 31(7): 2317-2338
• Review •
Progress in audio-driven digital human technologies
Full
Weibin Li1, 2, *, Jiafeng Gao1, Bing Xu3, Biao Hou2, Licheng Jiao2
Affiliations
  • 1 Hangzhou Institute of Technology,Xidian University,Hangzhou 311231,China
  • 2 School of Artificial Intelligence, Xidian University,Xi’an 710126,China
  • 3 AVIC Civil Aircraft Flight Test Center,Xi’an 710026,China
Published: 2026-07-16 doi: 10.11834/jig.260103
Outline
收藏切换

With the exponential growth of the metaverse,immersive virtual reality,and advanced human-computer interaction paradigms,audio-driven digital human generation has rapidly emerged as a paramount research frontier in computer vision and computer graphics. This transformative technology aims to establish a robust and seamless cross-modal mapping from 1D speech signals to 3D visual video streams. In contrast with traditional motion capture-based approaches,which are constrained by expensive hardware and rigid physical setups,audio-driven methodologies offer a highly scalable,cost-effective,and interactive solution. From a cognitive psychology perspective,human speech perception is inherently a multimodal process. The classic McGurk effect demonstrates that visual cues,such as precise lip movements,play a decisive role in deciphering auditory information,especially in noisy environments. Consequently,synthesizing a lifelike digital human requires not only ultrahigh-definition visual rendering but also millimeter-level,time-synchronized alignment between auditory signals and visual lip dynamics to prevent the notorious “uncanny valley” effect,where subtle unnaturalness provokes discomfort in human observers. Despite the proliferation of extensive research and rapid algorithmic advancements,generating photorealistic and real-time interactive digital avatars continues to encounter three formidable challenges. First,bridging the immense semantic gap between distinct modalities—specifically,translating semantic and acoustic features into complex spatial facial muscle deformations—remains fundamentally difficult. Second,the trade-off between high-fidelity rendering and real-time inference latency poses a critical bottleneck. Achieving pore-level skin details and realistic hair rendering typically demands immense computational resources,conflicting with the strict low-latency requirements (often under 100 ms) necessary for seamless live streaming or real-time conversational agents. Third,zero-shot cross-identity generalization and the expansion of driving scope—from isolated talking heads to full-body expressive agents that encompass co-speech gestures—demand robust structural priors that current isolated models struggle to provide. To deconstruct this complex domain systematically,this comprehensive review first proposes a unified “encoding-mapping-rendering” technical framework that encapsulates the majority of audio-driven digital human architectures. The framework initiates with an audio feature extractor,utilizing state-of-the-art self-supervised foundational models (e.g.,WavLM and Whisper) to capture rich phonetic,semantic,and emotional representations from raw audio waveforms. These acoustic features are then fed into visual geometric representations,transitioning from sparse 2D landmarks into sophisticated 3D morphable models,which provide explicit facial motion space. Finally,a cross-modal mapping and rendering network translates these spatial configurations into final visual video frames. Following this structural foundation,this study meticulously reviews the evolutionary trajectory of audio-driven rendering technologies,categorizing them into four dominant paradigms. First,early 2D image-based generation methods,predominantly driven by generative adversarial networks,are examined. Although these models excel in achieving rapid inference speed and fundamental lip synchronization,they inherently lack topological constraints,suffering from severe structural degradation and texture stretching when handling large-pose head rotations. Second,the integration of neural radiance fields (NeRFs) introduces a paradigm shift by leveraging implicit volumetric representations. NeRF-based approaches successfully resolve multi-view consistency issues; however,the heavy computational burden of volumetric ray marching restricts their real-time application,while their subject-specific optimization nature limits broad generalization. Third,the recent breakthrough of 3D Gaussian splatting (3DGS) has revolutionized the field by combining explicit geometric primitives with highly optimized rasterization pipelines. 3DGS-based models break real-time rendering constraints,easily surpassing 100 frames per second,although they still struggle with dynamic consistency during extreme facial deformations due to their discrete point cloud nature. Lastly,diffusion models represent the current pinnacle of visual fidelity and generative diversity. By iteratively denoising latent spaces,these models synthesize unprecedented skin textures,hair dynamics,and micro-expressions,although their inherent Markov chain sampling mechanisms introduce significant inference latency and temporal jitter challenges into long video generation. To facilitate an objective assessment of these diverse methodologies,this study synthesizes mainstream datasets and rigorous evaluation metrics. We detail the utilization of large-scale in-the-wild datasets,such as VoxCeleb,alongside high-resolution benchmarks,like high-definition talking face,and emotional datasets,such as the multi-view emotional audio-visual dataset. The evaluation criteria are multidimensional,encompassing lip-sync accuracy through SyncNet metrics (lip sync error distance and lip sync error confidence),visual quality via Fréchet inception distance and learned perceptual image patch similarity,temporal video consistency that utilizes Fréchet video distance,and identity preservation measured using cosine similarity. Finally,this study provides an in-depth analysis of open challenges and outlines prospective future research trajectories. We hypothesize that the integration of multimodal large language models will fundamentally redefine cross-identity generalization,serving as powerful world models for decoding audio into structured motion commands. Furthermore,transitioning from purely correlation-based learning to causal modeling will be essential for decoupling authentic emotional expressions from entangled acoustic noise. To address the theoretical bottlenecks of long-sequence temporal consistency,we propose exploring state space models,such as Mamba architectures,and flow matching trajectories. Concurrently,given that these generative capabilities approach indistinguishable realism,ethical implications,particularly those that concern deepfake misuse,necessitate the parallel development of robust forensic detection and defense mechanisms. Ultimately,this review aims to serve as a definitive guide for researchers,navigating the transition of digital humans from mere visual replicas into fully embodied,emotionally intelligent interactive agents in the digital era.

digital human  /  audio-driven  /  talking head generation  /  neural radiance field (NeRF)  /  3D Gaussian splatting(3DGS)  /  diffusion model
李卫斌, 高佳峰, 徐兵, 侯彪, 焦李成. 音频驱动的数字人技术进展. 中国图象图形学报, 2026 , 31 (7) : 2317 -2338 . DOI: 10.11834/jig.260103
Weibin Li, Jiafeng Gao, Bing Xu, Biao Hou, Licheng Jiao. Progress in audio-driven digital human technologies[J]. Journal of Image and Graphics, 2026 , 31 (7) : 2317 -2338 . DOI: 10.11834/jig.260103
  • Xi’an Key Industrial Chain Technology R&D Projects (25ZDLYB00006,25ZDLYB00015); Shaanxi Provincial Innovation Capability Support Program (2025ZC-WTXM-36); Shaanxi Provincial Qinchuangyuan “Scientists + Engineers” Project (2025QCY-KXJ-016)
Year 2026 volume 31 Issue 7
21
10
Cite this Article
BibTeX
Article Info
doi: 10.11834/jig.260103
  • Receive Date:2026-03-04
  • Online Date:2026-10-30
  • Published:2026-07-16
Article Data
Affiliations
History
  • Received:2026-03-04
  • Revised:2026-04-23
Funding
Xi’an Key Industrial Chain Technology R&D Projects (25ZDLYB00006,25ZDLYB00015); Shaanxi Provincial Innovation Capability Support Program (2025ZC-WTXM-36); Shaanxi Provincial Qinchuangyuan “Scientists + Engineers” Project (2025QCY-KXJ-016)
Affiliations
    1 Hangzhou Institute of Technology,Xidian University,Hangzhou 311231,China
    2 School of Artificial Intelligence, Xidian University,Xi’an 710126,China
    3 AVIC Civil Aircraft Flight Test Center,Xi’an 710026,China
References
Share
https://castjournals.cast.org.cn/joweb/zgtxtxxb/EN/10.11834/jig.260103
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT