With the exponential growth of the metaverse,immersive virtual reality,and advanced human-computer interaction paradigms,audio-driven digital human generation has rapidly emerged as a paramount research frontier in computer vision and computer graphics. This transformative technology aims to establish a robust and seamless cross-modal mapping from 1D speech signals to 3D visual video streams. In contrast with traditional motion capture-based approaches,which are constrained by expensive hardware and rigid physical setups,audio-driven methodologies offer a highly scalable,cost-effective,and interactive solution. From a cognitive psychology perspective,human speech perception is inherently a multimodal process. The classic McGurk effect demonstrates that visual cues,such as precise lip movements,play a decisive role in deciphering auditory information,especially in noisy environments. Consequently,synthesizing a lifelike digital human requires not only ultrahigh-definition visual rendering but also millimeter-level,time-synchronized alignment between auditory signals and visual lip dynamics to prevent the notorious “uncanny valley” effect,where subtle unnaturalness provokes discomfort in human observers. Despite the proliferation of extensive research and rapid algorithmic advancements,generating photorealistic and real-time interactive digital avatars continues to encounter three formidable challenges. First,bridging the immense semantic gap between distinct modalities—specifically,translating semantic and acoustic features into complex spatial facial muscle deformations—remains fundamentally difficult. Second,the trade-off between high-fidelity rendering and real-time inference latency poses a critical bottleneck. Achieving pore-level skin details and realistic hair rendering typically demands immense computational resources,conflicting with the strict low-latency requirements (often under 100 ms) necessary for seamless live streaming or real-time conversational agents. Third,zero-shot cross-identity generalization and the expansion of driving scope—from isolated talking heads to full-body expressive agents that encompass co-speech gestures—demand robust structural priors that current isolated models struggle to provide. To deconstruct this complex domain systematically,this comprehensive review first proposes a unified “encoding-mapping-rendering” technical framework that encapsulates the majority of audio-driven digital human architectures. The framework initiates with an audio feature extractor,utilizing state-of-the-art self-supervised foundational models (e.g.,WavLM and Whisper) to capture rich phonetic,semantic,and emotional representations from raw audio waveforms. These acoustic features are then fed into visual geometric representations,transitioning from sparse 2D landmarks into sophisticated 3D morphable models,which provide explicit facial motion space. Finally,a cross-modal mapping and rendering network translates these spatial configurations into final visual video frames. Following this structural foundation,this study meticulously reviews the evolutionary trajectory of audio-driven rendering technologies,categorizing them into four dominant paradigms. First,early 2D image-based generation methods,predominantly driven by generative adversarial networks,are examined. Although these models excel in achieving rapid inference speed and fundamental lip synchronization,they inherently lack topological constraints,suffering from severe structural degradation and texture stretching when handling large-pose head rotations. Second,the integration of neural radiance fields (NeRFs) introduces a paradigm shift by leveraging implicit volumetric representations. NeRF-based approaches successfully resolve multi-view consistency issues; however,the heavy computational burden of volumetric ray marching restricts their real-time application,while their subject-specific optimization nature limits broad generalization. Third,the recent breakthrough of 3D Gaussian splatting (3DGS) has revolutionized the field by combining explicit geometric primitives with highly optimized rasterization pipelines. 3DGS-based models break real-time rendering constraints,easily surpassing 100 frames per second,although they still struggle with dynamic consistency during extreme facial deformations due to their discrete point cloud nature. Lastly,diffusion models represent the current pinnacle of visual fidelity and generative diversity. By iteratively denoising latent spaces,these models synthesize unprecedented skin textures,hair dynamics,and micro-expressions,although their inherent Markov chain sampling mechanisms introduce significant inference latency and temporal jitter challenges into long video generation. To facilitate an objective assessment of these diverse methodologies,this study synthesizes mainstream datasets and rigorous evaluation metrics. We detail the utilization of large-scale in-the-wild datasets,such as VoxCeleb,alongside high-resolution benchmarks,like high-definition talking face,and emotional datasets,such as the multi-view emotional audio-visual dataset. The evaluation criteria are multidimensional,encompassing lip-sync accuracy through SyncNet metrics (lip sync error distance and lip sync error confidence),visual quality via Fréchet inception distance and learned perceptual image patch similarity,temporal video consistency that utilizes Fréchet video distance,and identity preservation measured using cosine similarity. Finally,this study provides an in-depth analysis of open challenges and outlines prospective future research trajectories. We hypothesize that the integration of multimodal large language models will fundamentally redefine cross-identity generalization,serving as powerful world models for decoding audio into structured motion commands. Furthermore,transitioning from purely correlation-based learning to causal modeling will be essential for decoupling authentic emotional expressions from entangled acoustic noise. To address the theoretical bottlenecks of long-sequence temporal consistency,we propose exploring state space models,such as Mamba architectures,and flow matching trajectories. Concurrently,given that these generative capabilities approach indistinguishable realism,ethical implications,particularly those that concern deepfake misuse,necessitate the parallel development of robust forensic detection and defense mechanisms. Ultimately,this review aims to serve as a definitive guide for researchers,navigating the transition of digital humans from mere visual replicas into fully embodied,emotionally intelligent interactive agents in the digital era.
| 科 Family | 属数 Number of genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) | 属 Genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) |
|---|---|---|---|---|---|---|
| 鹅膏菌科Amanitaceae | 2 | 11 | 5.26 | 鹅膏菌属 Amanita | 10 | 4.78 |
| 小菇科 Mycenaceae | 2 | 12 | 5.74 | 丝盖伞属 Inocybe | 5 | 2.39 |
| 多孔菌科 Polyporaceae | 8 | 14 | 6.70 | 蜡蘑属 Laccaria | 5 | 2.39 |
| 红菇科 Russulaceae | 3 | 23 | 11.00 | 小皮伞属 Marasmius | 6 | 2.87 |
| 小菇属 Mycena | 11 | 5.26 | ||||
| 光柄菇属 Pluteus | 5 | 2.39 | ||||
| 红菇属 Russula | 17 | 8.13 | ||||
| 栓菌属 Trametes | 5 | 2.39 |