Latest ArticlesVideoQA (video question answering), which automatically answers natural language question according to the content of videos, is a relatively new research direction in the field of visual language and has attracted extensive attention in recent years. The solution of videoQA task is of great significance for human-computer interaction, intelligent education, intelligent transportation, scenario analysis, video retrieval, and other fields. VideoQA is a challenging task because it requires a model to understand semantic information of the video and the question to generate the answer. In this work, we analyze the difference between VideoQA and ImageQA (image question answering), and summarize four challenges faced by VideoQA relative to ImageQA. Then, the existing VideoQA models are carefully classified according to the research method around these challenges. Following the classifications, we introduce the generation background and focus on the implementation of models and the relationship between different models. After that, the benchmark datasets commonly used in VideoQA are summarized, the performances of current mainstream algorithms on some datasets are introduced in detail, and the comparison, analysis and summary are carried out. Finally, the future challenges and research trends in this field are discussed, which will provide some ideas for further research in the future.
There are tens of thousands of butterfly species. Each butterfly species is closely related to a specific type of plants. It is significant to study butterfly species automatic identification. However, it is very challenging to study butterfly species recognition via the images taken in the field environments. One reason is that there are small number of butterfly species in the existing datasets compared with the reported species in the world. The other reason is that the number of samples (images) of each butterfly species is limited in the datasets. These situations make it challengeable to train a general system for butterfly species identification via machine learning algorithms. In addition, butterfly wings are always folded in the images taken in the field environments, which make it challengeable to learn butterfly classification features, which further make it difficult to study butterfly species recognition using machine learning techniques. Therefore, meta-learning is introduced to address the challenges, and DL-MAML (deep learning advanced model-agnostic meta-learning) algorithm is proposed for identifying the butterfly species using available images taken in the field environment. DL-MAML introduces a L2 regularization term to the objective function and the parameter updating of the meta-learning MAML while introducing ResNet34 deep learning model to extract butterfly features and plus two more convolution layers to the MAML model to avoid over-fitting. The extensive experiments including ablation experiments and comparison with available meta-learning models demonstrate that the initial model parameters obtained by DL-MAML algorithm are effective to identify new species butterflies. DL-MAML is effective in identifying butterfly species using images taken in the field environments, which make it possible to construct the general and complete butterfly species identification system using available butterfly images from limited categories.
At present, adversarial attacks on speech recognition models have typically involved adding noise to the entire speech signal, resulting in a wide perturbation range and introducing high-frequency noise. Existing research has attempted to reduce the perturbation range by designing optimization targets. However, controlling the transcription result requires adding perturbations to each frame, thus limiting further reduction in perturbation range. To address this issue, we propose a novel approach that examines the feature extraction process of speech recognition systems from a frame structure perspective. The study finds that framing and windowing determine the distribution of critical regions within the frame structure. Specifically, the weight of adding perturbation to each sampling point within the frame is influenced by its location. Based on the results of perturbation analysis on input features, we partition regions with shared attributes. Then we propose the adversarial example space measurement method and evaluation index to quantify the weight of sampling points for adversarial examples generation. We conduct cross-experiments by adding perturbations at different intervals within the frame, which enables us to identify key regions for perturbation addition. Our experiments on multiple models demonstrate that adding adversarial perturbation to vital regions can narrow the perturbation range, and provide a new perspective for generating high-quality audio adversarial examples.
To address the challenges of using high-resolution pavement images as input for existing convolutional neural network models and the inability of existing preprocessing algorithms to effectively perceive and retain information from low-ratio distress regions in original pavement images, a novel architectural unit called adaptive perception module (APM) paying greater attention to pavement distress region is proposed with the help of visual interpretation techniques, which achieves a rapid and accurate detection of pavement distress in high-resolution images and could be used to build a software system for automatic detection of pavement distress based on computer vision. Firstly, big kernel convolution and residual operations are used to reduce the origin image resolution and get the low-level but rich feature representation. Secondly, attention mechanism is developed to perceive and activate the region of pavement distress and filter the irrelevant background pixel noise. By means of joint learning, APM training could be completed without additional cost. After the visual interpretation method is used to aid the selection and design of the specific structure of APM, experimental results on the latest public dataset CQU-BPMDD show that the proposed APM significantly improves the classification accuracy, up to 84.47%. Experiments across different datasets CQU-BPDD demonstrate the generalization and robustness of APM. Code is available on https://github.com/Li-Ao-Git/apm.
Based on feature selection and model interpretable method, a cloud platform task termination state prediction model with strong interpretability is constructed. The model visualizes the mapping relationship between static and dynamic attributes of tasks/jobs and termination status, then finds out the mapping mechanism between the load characteristics and the task termination states. The workload monitoring log published by Google is used, and the task dynamic information in the cloud platform is added. Shapley Additive explain (SHAP) is used to find out the importance of the influence of static and dynamic attributes on the termination state, and the results of task termination states prediction model modeling are explained by using the importance of variables combined with SHAP value and XGBoost model. Visualization technology is used to show how load characteristics affect the model’s prediction of different task termination states. The average value of the absolute value of SHAP is used to measure the importance of features, and the global visualization of the importance of features in different termination states is realized. According to the results, 20 variables that have great influence on the prediction model of task termination states are selected as the basis of feature selection. How the change of characteristics affects the different termination states of tasks is visualized. From the visualization results, it can be seen that in the process of task running, different eigenvalues of each feature have influences on the termination states of the task, and different eigenvalues have different influences on the termination states. Feature selection combined with model interpretable method is applied to the construction process of task termination states prediction model, which can help to build a task termination states prediction model with high classification performance and easy understanding. By exploring the mapping mechanism between load characteristics and task termination status, the scheduling mechanism of cloud platform can be optimized.
Nowadays, digital cameras and smart phones have played an increasingly important role in daily lives, and have become the main tools for people to perceive the world, record information and communicate with each other. When these devices are used to shoot electronic screens, the irregular Moiré patterns in the image are produced due to the overlap of the display devices and digital grids of the camera sensor, which seriously affects the visual quality of the captured images. Therefore, the removal of Moiré pattern is of great significance for the post-processing of captured images. In this paper, recent research about Moiré removal is reviewed in detail, and the existing methods are classified into two categories according to different application scenarios and technical implementations: prior knowledge based Moiré removal and deep learning based methods respectively. According to different training data acquisition and alignment methods, deep learning based Moiré removal techniques can be divided into convolutional neural network (CNN) based methods and generative adversarial network (GAN) based methods. Then, based on the same public dataset, we implement the mainstream deep learning based Moiré removal algorithms, compare and analyze their performance, and summarize the advantages and disadvantages of various methods. Finally, possible directions for future research are discussed.
Cultural relics are prone to partial losses of structure or texture due to improper preservation. In order to restore the original image of cultural relics, we propose a new method for restoring cultural relics by using Transformer’s global structure understanding ability to restore the coherent structure of cultural relics and using convolutional neural networks’ local texture understanding ability to restore the delicate texture of cultural relics. To achieve this goal, the restoration work is divided into two steps: the first step is to use Transformer to reconstruct the rough image and restore the coherent structure; the second step is to use a convolutional neural network to enlarge the rough image and restore the fine texture of the missing area. Considering that there is no high-quality, large-scale heritage database in China and abroad, a new heritage image database is also proposed. The experimental results show that the restoration results outperform the current image restoration algorithms in both subjective and objective evaluations in both breakage restoration experiments and large-area breakage restoration experiments that match realistic scenes. At the same time, the proposed method supports diversified output, which provides diverse references for restorers and improves the restoration efficiency.
In the field of image defogging, it is difficult for most defogging models to maintain a balance between accuracy and efficiency. Specifically, high-precision models are often accompanied by complex network structures, and simple network structures often lead to low-quality results. To address the problem, we propose a multi-branch defogging network based on fog concentration classification and dark and bright channel priors. The model uses the defogging networks with different complexity to handle the images with different fog concentrations, which significantly raises the computational efficiency under ensuring the defogging precision. The model is composed of a lightweight foggy image classifier and a multi-branch defogging network. The classifier divides the foggy images into light, medium and dense foggy images and outputs the fog concentration labels. The multi-branch network contains three branches with the same structure but different widths that process three types of fog images separately. We propose a new fog concentration classification method and a new fog concentration classification loss function. The function combines the dark channel characteristics and defogging difficulty of the foggy image with the defogging precision and computational efficiency of the model, so as to obtain a reasonable fog concentration classification, and consequently achieve a good balance of defogging quality and computing power requirements. We propose a new dark channel prior loss function and a new bright channel prior loss function to constrain the multi-branch defogging network, which effectively enhances the defogging precision. Extensive experiments show that the model is beneficial to get better defogging effect with lower network parameters and complexity.
For sort acceleration on FPGA, the selection and optimization of various performance metrics, such as latency, throughput, power efficiency, hardware utilization and bandwidth efficiency, etc., are of critical importance. We compare the evolution of performance-driven sort acceleration, with advances in larger data size, more data types, more algorithm support, hardware-software cooperation and new hardware-based design; we analyze the problems and optimization strategies faced at different stages of design, implementation, testing and so on. Among the numerous sorting algorithms, merge sort becomes mainstream due to its excellent hardware parallelism, scalability and simple control logic. Sort acceleration is an architectural design that is deeply tied to specific application scenarios. We analyze the architectural adjustments made from the perspective of database system acceleration for resource competition, data arrangement, unique operations and diversity of user requests problems faced in databases. At last, to address the problems and shortcomings of existing studies, we provide an outlook on future directions in terms of distributed sort acceleration for very large data scale, the introduction of new hardware devices such as data processing unit, and the improvement of auxiliary tool chains such as high level synthesis to drive the iterative update of sort acceleration design.
Master-slave blockchain is a novel information processing technology that is domain-oriented and uses efficient cryptography principles for trustworthy communication and storage of big data. With the exponential growth of the scale of domain data, the existing master-slave blockchain system has increasingly serious problems such as low query efficiency and long traceability time. To address these issues, we propose a multi-level index construction method for master-slave blockchain (MSMLI). Firstly, MSMLI introduces a weight matrix and partitions the entire master-slave blockchain based on the master chain structure, and the weight of each partition is assigned. Secondly, for the master blockchain in each partition, a master chain index construction method based on jump consistent Hash (JHMI) is proposed, which takes the key value of the nodes and the number of index slots as input and outputs the master chain index. Finally, a Bloom filter is introduced to improve the column-based selection function and a secondary composite index on the subordinate blockchain corresponding to each master block is built. Experimental results on three constraint conditions and two types of datasets demonstrate that the proposed method reduces the index construction time by an average of 9.28%, improves the query efficiency by 12.07%, and reduces the memory overhead by 24.4%.