收藏切换
Recent advances in large multimodal models for UAV visual understanding
收藏切换
Yating Yu, Congqi Cao*, Zhaoying Wang, Yanning Zhang
Journal of Image and Graphics | 2026, 31(9) : 3169 - 3197
Less
收藏切换
Journal of Image and Graphics | 2026, 31(9): 3169-3197
• The intelligent empowerment of large models in the aviation field •
Recent advances in large multimodal models for UAV visual understanding
Full
Yating Yu, Congqi Cao*, Zhaoying Wang, Yanning Zhang
Affiliations
  • School of Computer Science,Northwestern Polytechnical University,Xi’an 710129,China
Published: 2026-09-16 doi: 10.11834/jig.260215
Outline
收藏切换

With the rapid expansion of the low-altitude economy,accompanied by the increasing maturity of intelligent unmanned systems,unmanned aerial vehicles (UAVs) are gradually transforming from traditional remotely controlled flying platforms into autonomous aerial agents that are fully capable of perception,reasoning,and decision-making. Among the various sensing modalities available to UAVs,visual perception plays a central role in acquiring environmental information and enabling high-level situational awareness. Hence,visual understanding capability can directly determine the intelligence level and operational autonomy of UAV systems. In recent years,the emergence of visual foundation models,vision-language models,and multimodal large language models (LLMs) has significantly influenced the technical paradigm of UAV visual understanding. As a result,this paradigm shift has provided new opportunities for enabling UAV systems to operate effectively in complex and open environments in which perception,reasoning,and decision making must be tightly coupled. To clarify this emerging research landscape,this paper introduces a capability-oriented analytical framework that organizes UAV visual understanding into three hierarchical levels: basic perception,semantic reasoning,and decision planning. This framework serves as the conceptual backbone of the survey and allows recent studies to be examined from a systematic capability-evolution standpoint. Specifically,from the task perspective,we construct a comprehensive taxonomy of UAV visual understanding tasks that includes four major categories: 1) basic object perception,2) event semantic analysis,3) spatial environment understanding,and 4) flight decision-making. Within this taxonomy,representative tasks (i.e.,object detection,target tracking,human action recognition,visual question answering,spatial reasoning,navigation,and autonomous flight control) are analyzed in a unified manner. At the same time,this paper summarizes several fundamental challenges that arise in aerial visual perception. These challenges include significant scale variations caused by high-altitude viewpoints,long-range observations that lead to small object representations,complex backgrounds with strong visual clutter,and dynamic environmental changes that require robust temporal reasoning. From the technical perspective,this paper reviews the methodological evolution that underlies the development of UAV visual understanding models. Early approaches were largely based on conventional deep learning architectures that focused on supervised visual perception tasks. Then,subsequent advances in visual foundation models significantly improved representation learning by leveraging large-scale pretraining and open-vocabulary multimodal alignment. More recently,LLMs and multimodal LLMs have introduced powerful reasoning capabilities and cross-modal interaction mechanisms that allow UAV systems to interpret visual observations along with natural language instructions and contextual knowledge. Building upon these developments,embodied vision-language-action models have begun to connect perceptions with action generation,thus enabling UAVs to perform complex task planning and interactive decision making in real-world environments. From the capability perspective,this paper further examines how large visual understanding models enhance UAV intelligence across three dimensions: 1) visual perception enhancement,where large models improve robustness in open-vocabulary recognition,small-object detection,and fine-grained visual understanding; 2) vision-language reasoning,where multimodal models facilitate complex reasoning processes such as spatial relation reasoning,event interpretation,and cross-modal knowledge integration; and 3) visual decision planning,where embodied multimodal models enable UAVs to translate perception and reasoning outcomes into actionable flight strategies,mission planning procedures,and adaptive control policies. Additionally,representative UAV visual datasets and benchmarks that support the development and assessment of large multimodal models for visual understanding are summarized in this work. Particular attention is given to the evolution of evaluation protocols. Traditional benchmarks often measure performance in narrowly defined tasks,such as detection or classification. However,recent research has increasingly emphasized capability-oriented evaluation frameworks that assess broader competencies,namely reasoning ability,cross-task generalization,and decision support. This transition reflects a broader shift in the field toward evaluating integrated visual intelligence rather than isolated perception performance. Finally,this work discusses several promising research directions that may shape the future development of UAV visual understanding systems. These include the construction of general-purpose visual foundation models tailored for aerial scenarios,the advancement of embodied UAV intelligence through vision-language-action integration,the development of real-time reasoning techniques and lightweight deployment strategies suitable for resource-constrained aerial platforms,and the establishment of safety-aware,trustworthy,and privacy-preserving UAV perception systems.

multimodal large language model (MLLM)  /  visual foundation model (VFM)  /  unmanned aerial vehicle (UAV)  /  visual understanding  /  intelligent decision-making  /  review
余雅婷, 曹聪琦, 王昭颖, 张艳宁. 面向无人机的多模态视觉理解大模型研究进展. 中国图象图形学报, 2026 , 31 (9) : 3169 -3197 . DOI: 10.11834/jig.260215
Yating Yu, Congqi Cao, Zhaoying Wang, Yanning Zhang. Recent advances in large multimodal models for UAV visual understanding[J]. Journal of Image and Graphics, 2026 , 31 (9) : 3169 -3197 . DOI: 10.11834/jig.260215
  • National Natural Science Foundation of China (62376217,62576279,62301434);Young Elite Scientists Sponsorship Program by CAST(2023QNRC001)
Year 2026 volume 31 Issue 9
29
15
Cite this Article
BibTeX
Article Info
doi: 10.11834/jig.260215
  • Receive Date:2026-04-16
  • Online Date:2026-10-30
  • Published:2026-09-16
Article Data
Affiliations
History
  • Received:2026-04-16
  • Revised:2026-05-20
Funding
National Natural Science Foundation of China (62376217,62576279,62301434);Young Elite Scientists Sponsorship Program by CAST(2023QNRC001)
Affiliations
    School of Computer Science,Northwestern Polytechnical University,Xi’an 710129,China
References
Share
https://castjournals.cast.org.cn/joweb/zgtxtxxb/EN/10.11834/jig.260215
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT