收藏切换
A survey on multimodal real-time interactive digital humans
收藏切换
Ruiqi Du1, Boai Yang2, Fengbo Zhou1, Wei Qu1, Tao Li1, *
Journal of Image and Graphics | 2026, 31(10) : 3545 - 3571
Less
收藏切换
Journal of Image and Graphics | 2026, 31(10): 3545-3571
• Review •
A survey on multimodal real-time interactive digital humans
Full
Ruiqi Du1, Boai Yang2, Fengbo Zhou1, Wei Qu1, Tao Li1, *
Affiliations
  • 1 School of Computer Science and Engineering,Hunan University of Science and Technology,Xiangtan 411201,China
  • 2 School of Design,Hunan University,Changsha 410082,China
Published: 2026-10-16 doi: 10.11834/jig.250511
Outline
收藏切换

With the continued evolution of artificial intelligence,5G/6G communication,and multimodal interaction technologies,digital humans are rapidly transforming from static virtual avatars into intelligent agents capable of real-time,natural,and emotionally resonant communication. This transition marks a broader shift in human-computer interaction,moving beyond scripted responses toward context-aware,adaptive systems that can perceive,reason,and respond with human-like fluidity across multiple sensory channels. This paper presents a comprehensive review of real-time multimodal interactive digital humans,focusing on their conceptual evolution,technical framework,and key challenges. We trace the historical trajectory from early non-interactive virtual characters on platforms such as second life to today’s AI-driven agents that integrate language understanding,emotional intelligence,and physical embodiment in immersive environments. The survey systematically examines the core technologies underlying modeling,real-time driving,and rendering,and provides an integrated view of how multimodal data—including speech,facial expression,gaze,gestures,and physiological signals—can be perceived,fused,and generated to support coherent and lifelike human-machine interaction. From the perspective of modeling,we analyze the transition from traditional 3D parametric models to neural implicit representations such as NeRF and 3D Gaussian splatting,which enable high-fidelity reconstruction and photorealistic rendering while still supporting real-time performance. Recent advances in 3D Gaussian splatting,including reported rendering speeds of more than 300 frame/s on consumer hardware,have helped bridge the longstanding gap between visual fidelity and interactive responsiveness that previously constrained practical deployment. In terms of motion and expression control,the survey highlights the role of multimodal fusion and diffusion-based generation in synchronizing voice,facial animation,and body motion,thereby improving the naturalness and temporal consistency of interactive behavior. We critically examine the evolution from rule-based animation systems to end-to-end neural approaches that jointly model cross-modal dependencies,reducing the uncanny-valley effect through stronger temporal coherence and greater emotional authenticity. Rendering technologies are reviewed from both physically based and neural perspectives,with emphasis on balancing visual realism and latency in real-time deployment. Emerging hybrid approaches that combine explicit geometric representations with neural detail enhancement appear particularly promising for achieving both physical plausibility and computational efficiency in resource-constrained settings. Furthermore,the paper proposes a unified system architecture for digital humans consisting of four hierarchical layers: perception,fusion,generation,and extension,each responsible for multimodal sensing,semantic integration,expressive generation,and cross-domain adaptability,respectively. The perception layer integrates heterogeneous inputs ranging from microphone arrays and RGB-D cameras to emerging physiological sensors such as EEG and EDA,thereby establishing a rich basis for contextual understanding. Within this architecture,the integration of large multimodal models (LMMs) and large language models (LLMs) enables digital humans to perceive complex environmental and emotional cues,reason about user intent,and produce personalized,empathetic responses. Modern foundation models also support in-context adaptation,allowing digital humans to adjust their communication style dynamically according to interaction history,user characteristics,and situational context without explicit retraining. Representative technologies such as neural radiance fields,real-time diffusion-based speech synthesis,and emotion-driven dialogue generation are analyzed to illustrate their contributions to interactivity and immersion. Particular attention is given to latency-critical components,for which end-to-end response times below 200 ms are often essential to preserving conversational flow and user trust in real-world applications. Based on this synthesis,the paper identifies current challenges including latency optimization,multimodal alignment,semantic coherence,and emotional authenticity. We highlight the fundamental tension between computational complexity and real-time constraints,especially when deploying high-fidelity models on edge devices with limited power and thermal budgets. Addressing these issues will require advances in lightweight model compression,edge-native deployment,and cross-modal consistency learning. Promising directions include neural architecture search for modality-specific subnetworks and dynamic computation scaling based on interaction complexity. Looking ahead,digital-human research is expected to move toward more deeply integrated affective computing,multimodal reasoning,and socially adaptive intelligence,ultimately enabling human-machine interaction that is perceptually natural,emotionally grounded,and ethically trustworthy. At the same time,crucial issues such as digital identity,consent-based data usage,and robust watermarking mechanisms must be addressed to ensure responsible large-scale deployment. By summarizing state-of-the-art methods and mapping emerging trends,this study provides both theoretical insight and practical guidance for the development of the next generation of real-time multimodal interactive digital humans,with broad applications in education,healthcare,entertainment,and collaborative robotics.

digital human  /  multimodal interaction  /  multimodal fusion  /  real-time driving  /  human-computer interaction
杜瑞麒, 杨柏蔼, 周丰波, 屈薇, 李涛. 多模态实时交互式数字人综述. 中国图象图形学报, 2026 , 31 (10) : 3545 -3571 . DOI: 10.11834/jig.250511
Ruiqi Du, Boai Yang, Fengbo Zhou, Wei Qu, Tao Li. A survey on multimodal real-time interactive digital humans[J]. Journal of Image and Graphics, 2026 , 31 (10) : 3545 -3571 . DOI: 10.11834/jig.250511
Year 2026 volume 31 Issue 10
44
19
Cite this Article
BibTeX
Article Info
doi: 10.11834/jig.250511
  • Receive Date:2025-10-28
  • Online Date:2026-10-30
  • Published:2026-10-16
Article Data
Affiliations
History
  • Received:2025-10-28
  • Revised:2026-02-14
Affiliations
    1 School of Computer Science and Engineering,Hunan University of Science and Technology,Xiangtan 411201,China
    2 School of Design,Hunan University,Changsha 410082,China
References
Share
https://castjournals.cast.org.cn/joweb/zgtxtxxb/EN/10.11834/jig.250511
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT