Multidimensional Evaluation of Autonomous Driving Test Scenarios Based on AHP-EWN-TOPSIS Models

Multidimensional Evaluation of Autonomous Driving Test Scenarios Based on AHP-EWN-TOPSIS Models

PDF

Shiqi Li¹, Rui Zhou², Helai Huang¹

Automotive Innovation | 2025, 8(2) : 237 - 251

Less

Automotive Innovation | 2025, 8(2): 237-251

Multidimensional Evaluation of Autonomous Driving Test Scenarios Based on AHP-EWN-TOPSIS Models

Full

Shiqi Li¹, Rui Zhou², Helai Huang¹

Affiliations

¹ Central South University School of Traffic and Transportation Engineering Changsha 410075 China

² Central South University School of Automation Changsha 410075 China

doi: 10.1007/s42154-024-00344-6

Outline

Abstract

Less

The advancement of autonomous vehicles (AVs) requires robust evaluation methods to ensure both safety and efficiency. To incorporate multiple dimensions in designing test scenarios, this paper proposes a multidimensional evaluation framework for AV test scenarios based on the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) model. The evaluation considers three dimensions: risk, complexity, and rarity. First, the test scenario is deconstructed into its constituent elements. Then, the weights of these elements are determined from both subjective and objective perspectives using the Analytic Hierarchy Process (AHP) and Entropy Weight Method. Then, game theory is employed to optimize these weights, deriving the optimal balance between subjective and objective weights. Next, three different scenario libraries are utilized as case studies, and a comprehensive evaluation index is calculated using the TOPSIS model. Subsequently, the scenarios are categorized into four levels using Kmeans clustering algorithm. Finally, the accuracy and reliability of the framework are verified through simulation. The simulation results demonstrate the effectiveness of the framework in identifying critical scenarios and providing valuable insights for AV testing.

Key words

Autonomous vehicles / Safety testing / Scenario engineering / TOPSIS

Cite this Article

Shiqi Li, Rui Zhou, Helai Huang. Multidimensional Evaluation of Autonomous Driving Test Scenarios Based on AHP-EWN-TOPSIS Models[J]. Automotive Innovation, 2025 , 8 (2) : 237 -251 . DOI: 10.1007/s42154-024-00344-6

Full Text

Less

1 Introduction

Less

The advancement of autonomous driving technology holds the promise of transforming transportation by enhancing safety, efficiency, and convenience. However, ensuring the reliability and safety of autonomous vehicles (AVs) remains a significant challenge [1]. AVs must be capable of handling various driving scenarios and conditions, striving to achieve, at the very least, the safety levels of human drivers in most situations. This necessitates comprehensive testing, encompassing common driving scenarios as well as rare and hazardous events [2,3].

Traditional on-road testing, though valuable, is often constrained by safety, cost, and the limited range of real-world scenarios it can cover [1,2,4-6]. To enhance testing efficiency, scenario-based testing has been extensively researched. Unlike traditional methods that may rely on random test cases, scenario-based testing focuses on specific, pre-defined situations that an AV is likely to encounter [7-12]. By recreating a wide array of scenarios, from everyday driving conditions to extreme corner cases, scenario-based testing provides a deeper understanding of an AV’s capabilities and limitations.

However, the challenge of effectively setting up test scenarios remains unresolved. Similar to the need for a balanced difficulty level in student examinations, quantifying the difficulty of test scenarios for AVs is a complex task. How should we define and measure the difficulty levels of these scenarios? What kind of safety performance should AVs exhibit under varying levels of difficulty? Current research primarily focuses on the risk levels of scenarios, rarely providing a multi-dimensional quantification [13-16]. For instance, a test scenario’s difficulty is not solely determined by its risk level but also by factors such as complexity and frequency. These factors significantly impact the performance of AVs [2]. Therefore, a comprehensive approach is needed to evaluate and quantify test scenarios across multiple dimensions.

To address the aforementioned research gaps, this study aims to develop a multidimensional evaluation framework for AV test scenarios, focusing on risk, complexity, and rarity. First, the test scenario is deconstructed into fundamental elements, analyzing each element’s contribution to these dimensions. Second, the weights of these elements are determined using a combination of Analytic Hierarchy Process (AHP) and Entropy Weight Method (EWM). AHP involves pairwise comparisons and judgment matrices to derive subjective weights based on expert opinions, while EWM calculates objective weights by assessing data variability and uncertainty. To integrate these subjective and objective weights, game theory is applied to derive optimal weights that balance both perspectives. Third, the comprehensive evaluation index of 10023 scenarios from three different databases are calculated using the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) model. Finally, these scenarios are classified using K-means clustering algorithm and the accuracy and reliability of the proposed weight assignment methodology is validated through a counterfactual simulation study.

By applying the evaluation framework to three libraries of different test scenarios, this paper aims to demonstrate its effectiveness in accurately reflecting the risk, complexity, and rarity of various test scenarios. The main contributions of this study are as follows:

(1) Introducing a multidimensional evaluation framework that comprehensively assesses autonomous driving test scenarios based on risk, complexity, and rarity.

(2) Integrating subjective and objective weighting methods (AHP and EWM) and optimizing these weights using game theory, providing a balanced and robust evaluation approach.

(3) Using the TOPSIS model to derive quantitative indices for three dimensions, facilitating systematic and multidimensional scenario ranking.

(4) Proposing a method using the K-means clustering algorithm to categorize scenarios based on three dimensions, and validating its practical applicability and reliability through a simulation case study.

2 Literature Review

Less

In scenario assessment studies, the primary focus is on quantitatively evaluating the attributes of the test or driving scenario [17-20]. Duan et al. [17] introduced the concept of test scenario complexity to assess the effectiveness of generated scenarios, calculating the priority importance of a series of road-related quantitative factors using the AHP. Wang et al. [18] proposed a novel approach for modeling and quantitatively assessing the environmental complexity of AVs, combining expert evaluation with AHP to determine the complexity of test scenarios. Gao et al. [19] used the AHP to calculate the criticality of AVs equipped with traffic congestion pilot systems in complex environments, verifying the feasibility of the AHP in assessing the criticality of test scenarios in complex environments. Zhang and Tak [20] proposed a new risk assessment method for test scenarios using an improved AHP method, which automatically generates pairwise comparison values through simulation to optimize performance indicators and assess the risk of test scenarios in real-time traffic environments using a BPNN model.

In these studies, the determination of weights is constrained by subjectivity. Some studies adopt a combination of subjective and objective approaches to mitigate this influence [21-26]. Lu and Kongjian [24] achieved quantitative evaluation of test scenario complexity through a scenario complexity evaluation method based on AHP and information entropy theory. Zhang et al. [25] defined scenario complexity as the measure of uncertainties of all dynamic entities within the scenario, introducing information entropy to represent these uncertainties, considering moving vehicles as random variables. They proposed calculating the entropy value of each vehicle in the drivable area using a trajectory clustering generation algorithm based on search. Dong et al. [26] established a complexity measurement model for driving scenarios using a Bayesian network, determining variable weights through information entropy theory and a BP neural network, and improving the gravity model. Zhang et al. [27] proposed a evaluation method based on AHP and information entropy theory, achieving automatic quantitative evaluation of test scenario complexity.

After determining the weights of scenario variables, comprehensive evaluation methods are used to obtain the final quantitative results. Bakioglu and Atahan [28] introduced a new hybrid multi-criteria decision-making (MCDM) method based on AHP, TOPSIS, and Vlse Kriterijumska Optimizacija I Kompromisno Resenje (VIKOR) in a Pythagorean fuzzy environment to address risk ranking related to AVs, with the results validated through sensitivity analysis. Bruno and Osório [29] applied AHP to calculate attribute weights and TOPSIS to classify traffic signs into their importance levels, using a combination of AHP and TOPSIS to define the priority of traffic signs for AV navigation. Gomes et al. [30] combined AHP with TOPSIS for diagnostic analysis of autonomous trucks and perception data. Qu et al. [21] applied the EWM to an integrated model for short-term traffic flow prediction at signalized intersections. Li et al. [22] applied the EWM to an integrated model for assessing urban sustainability. Li and Chen [23] proposed a fuzzy comprehensive evaluation method based on AHP and EWM for road safety assessment, qualitatively analyzing factors influencing traffic safety and using AHP and EWM to determine each factor’s weight, thereby establishing a traffic safety evaluation system through fuzzy evaluation.

Although these related studies quantitatively evaluate scenarios, they lack validation of AVs under these scenario evaluation systems. Some scholars have introduced quantitative indicators of scenario attributes into the testing process or scenario generation process. Bussler et al. [31] proposed an evaluation method based on the number of dangerous scenarios, generating concrete scenarios for testing by sampling the logical scenario parameter space and using evolutionary algorithms to identify safety-critical scenario parameter combinations. Gao et al. [32] proposed a scenario generation method based on test matrices and combinatorial testing, evaluating scenario complexity using AHP. The experimental results showed that the higher the scenario complexity index, the easier it was to identify faults in the tested system. Zhang et al. [33] proposed a performance evaluation method for AVs within the logical scenario parameter space, establishing evaluation methods for the safety and human-likeness of AVs. Feng et al. [34] introduced a new evaluation standard to quantify the criticality of scenarios, calculating scenario criticality by comprehensively considering operational difficulty and the frequency of scenario occurrences. Wang et al. [35] proposed a driving risk assessment model to quantitatively evaluate AVs driving risks by analyzing the coupling relationships of different traffic elements, establishing the concepts of internal and external fields of the driving risk coupling model for risk assessment of autonomous driving in complex traffic scenarios.

Despite the substantial progress in scenario evaluation methods, current research exhibits several limitations. Primarily, subjectivity in weight determination remains a significant constraint, potentially skewing the evaluation outcomes. While integrating subjective and objective approaches has mitigated this to some extent, but a comprehensive method that fully addresses these biases remains to be developed. Furthermore, existing studies predominantly focus on the complexity and risk assessment of scenarios without extensive validation through tests. The dynamic interplay between scenario attributes and AV performance under diverse and evolving conditions requires more thorough investigation. Additionally, there is also a need for developing standardized evaluation frameworks that can be universally applied, facilitating consistent and comparable assessments across different studies and applications.

To address these limitations, this study adopts a combined subjective and objective approach to determine the weights of test scenario elements. Subsequently, the TOPSIS method is used to comprehensively evaluate the test scenarios in terms of risk, complexity, and rarity. The accuracy of the evaluation results is verified by testing the AV in various scenarios.

3 Methodology

Less

The methodology for this study involves several key steps designed to develop and validate a comprehensive evaluation framework for test scenarios, as shown in Fig. 1.

3.1 Test Scenario Deconstruction and Weighting

The first step involves deconstructing the test scenarios into fundamental elements. Each element is analyzed for its contribution to the overall dimensions of risk, complexity, and rarity. To ensure a balanced evaluation, both subjective and objective weights of the scenario elements are determined.

3.1.1 Analytic Hierarchy Process

The AHP [36] is a multi-objective decision-making technique, which calculates the weights of indicators by breaking down a complex problem into multiple indicators and comparing these quantitative and qualitative indicators in pairs to meet a certain level of consistency. The method relies on the expertise and subjective judgment of domain experts to ensure that the resulting weights align with the actual situation to a certain extent, although it is inevitably subjective. In this study, six experts (a virtual simulation test expert, two automotive engineering experts, an accident analysis expert, and a AV algorithm design expert) rated the indicators based on Saaty’s nine-point scale comparison [37] and constructed a series of pairwise comparison matrices to characterize the relative importance of elements at different levels.

AHP starts with constructing the judgment matrix $\boldsymbol{E}=\left[e_{i j}\right]$, where ${e}_{ii}$ denotes the comparison result of the importance level between element $i$ and $j$ based on a 1-9 scale, and ${e}_{ij} = \frac{1}{{e}_{ji}}$, for $i, j = 1,2,\ldots, n$. Next, normalize the judgment matrix $\mathbf{E}$ and calculate the weights of each evaluation indicator:

(1)

$\overline{w_{i}}=\sqrt[n]{\prod_{j=1}^{n} x_{i j}}$

(2)

$w_{i}=\frac{\overline{w_{i}}}{\sum_{i=1}^{n} \overline{w_{i}}}$

Finally, calculate the maximum eigenvalue max coefficient of each index, and carry out a consistency test. When the Random Consistency Ratio (CR) is less than 0.10, it indicates that the judgment matrix reliability is high. Here, RI represents the Random Index, and CI represents the Consistency Index.

(3)

$\lambda_{\max }=\frac{1}{n} \sum_{i=1}^{n} \frac{\left(\sum_{j=1}^{n} e_{i j} w_{j}\right)}{w_{i}}$

(4)

$ {CI} = \frac{{\lambda }_{\max } -n}{n -1} $

(5)

$ {CR} = \frac{CI}{RI} $

A scenario element structure with five levels is constructed to depict the test scenario in detail. Three dimensions are considered and its AHP hierarchy diagram is shown in Figs. 2, 3 and 4.

The state of the road surface significantly impacts the smoothness and safety of vehicle travel, largely influenced by weather conditions. Traffic facilities are also integral components, with signal control playing a vital role in diverting traffic flow. The presence of obstacles can hinder and interfere with the normal behavior of traffic participants, increasing the likelihood of traffic disruption and potential safety risks. Ego vehicles are affected by road facilities and conditions as well as the behavior of other traffic participants. The attributes of these objects include their types, numbers, behavioral patterns, and relative characteristics such as direction of motion and position relative to the ego vehicle, forming a comprehensive and accurate description of the scenarios.

Figures 2,3 and 4 show the AHP hierarchy describing the test scenarios in terms of the three dimensions of complexity, risk and rarity, respectively. It is worth noting that, in the assessment of risk and rarity, this study goes beyond the usual factors of road segment type and number of lanes by also incorporating the condition of the road surface as a critical element. In this structure, the ego vehicle is the AV, the most basic and crucial object in the whole test scenario. The vehicle’s DDT is subdivided into six categories: right turn, left turn, lane change, straight ahead, stop, and other behaviors. The road, as the basic unit of scenario construction, covers a variety of roadway types, such as roadway entrances and exits, three-way intersections, crossroads, and consecutive straight driving sections. The setting of the number of lanes directly maps the specific environment in which vehicles are traveling.

3.1.2 Entropy Weight Method

EWM [ 38,39] is an objective weight allocation method based on the information entropy theory. This method aims to quantify the information content of the evaluation indicators and their role in the decision-making process through the concept of information entropy, and then reveal the importance of the indicators through the objective analysis of the data, so as to effectively reduce the interference of subjective bias. The core of this method is that the lower the information entropy of an evaluation indicator, the richer the effective information it carries, and thus it should be given more weight in the evaluation system. Conversely, relatively speaking, the higher the information entropy, the less information the indicator provides, and its weight should be reduced accordingly. This study adopts the entropy weight method to establish the objective weight of each scene element, and the specific calculation process is described in Algorithm 1.

Algorithm 1 EWM Weight Calculation

1:	Normalize tde initial matrix ${X} = \left\lbrack {x}_{ij}\right\rbrack$ to obtain tde normalized matrix ${{X}}_{1} =\left\lbrack {x}_{ij}^{\prime }\right\rbrack$. In tde evaluation, tdere are $m$ sample objects, each witd $n$ evaluation indicators, where $i = 1,2,\ldots, m, j = 1,2,\ldots, n$.
2:	Dimensionless processing of data: positive indicator ${x}_{ij}^{\prime } = \frac{{x}_{ij} -{x}_{i\min j}}{{x}_{i\max j} -{x}_{i\min j}}$ and negative indicator ${x}_{ij}^{\prime } = \frac{{x}_{i\max j} -{x}_{ij}}{{x}_{i\max j} -{x}_{i\min j}}$
3:	Determine the weight of the $j$ -th evaluation metrics for the $i$ -th program: ${r}_{ij} =$ $\frac{{x}_{ij}}{\mathop{\sum }\limits_{{i = 1}}^{m}{x}_{ij}}$
4:	Calculate the entropy ${e}_{j}$ for each element: ${e}_{j} = -k\mathop{\sum }\limits_{{i = 1}}^{m}{r}_{ij}\ln \left({r}_{ij}\right)$, where $k =$ $\frac{1}{\ln \left(m\right) }$ Determine the degree of diversification ${d}_{j} : {d}_{j} = 1 -{e}_{j}$
5:	Determine the degree of diversification ${d}_{j} : {d}_{j} = 1 -{e}_{j}$
6:	Calculate the weight ${w}_{j}$ for each element: ${w}_{j} = \frac{{d}_{j}}{\mathop{\sum }\limits_{{j = 1}}^{n}{d}_{j}}$

3.1.3 Integration Using Game Theory

In the process of determining subjective weights using the AHP, the consistency test can ensure the logical consistency of experts’ judgments, but it is difficult to eliminate subjective bias and differences in understanding between different experts. The EWM is highly susceptible to the influence of extreme data when determining objective weights, and the accuracy of its weights is closely related to the quality of data, which may sometimes lead to unreasonable calculation results. In view of this, this study adopts a comprehensive strategy to combine the subjective weights obtained by the AHP method with the objective weights obtained by the EWM, so as to establish the comprehensive weights of each index to guarantee the accuracy of the decision-making results. The allocation of weights is informed by the principles of game theory [40], where the objective weights and subjective weights represent the two sides of a dynamic interaction. This approach aims to minimize the discrepancy between the comprehensive weights and the subjective and objective weights, thereby enhancing the scientific and reliable nature of the comprehensive weight calculation.

Let the weights calculated for $n$ indicators in the evaluation system using $m$ methods be ${\mathbf{\omega }}_{k} = \left({{\omega }_{k1},{\omega }_{k2},\ldots,{\omega }_{kn}}\right)$, $k = 1,2,\ldots, m$. Let the linear combination of these weights be ${\omega }_{c}$ :

(6)

$ {\mathbf{\omega }}_{c} = \mathop{\sum }\limits_{{k = 1}}^{m}{c}_{k}{\mathbf{\omega }}_{k}{}^{\mathrm{T}} $

where ${c}_{k}$ are the combination coefficients that minimize the deviation between ${\mathbf{\omega }}_{c}$ and the respective ${\mathbf{\omega }}_{k}$. That is,

(7)

$ \min {\begin{Vmatrix}{\mathbf{\omega }}_{c} -{\mathbf{\omega }}_{k}\end{Vmatrix}}^{2},\;k = 1,2,\ldots, m $

This can be formulated as the following system of equations:

(8)

$\left[\begin{array}{ccc}\omega_{1} \omega_{1}^{\mathrm{T}} & \cdots & \omega_{1} \omega_{m}^{\mathrm{T}} \\\vdots & \ddots & \vdots \\\omega_{m} \omega_{1}^{\mathrm{T}} & \cdots & \omega_{m} \omega_{m}^{\mathrm{T}}\end{array}\right]\left[\begin{array}{c}c_{1} \\\vdots \\c_{m}\end{array}\right]=\left[\begin{array}{c}\omega_{1} \omega_{1}^{\mathrm{T}} \\\vdots \\\omega_{m} \omega_{1}^{\mathrm{T}}\end{array}\right]$

Solving the above equation yields the corresponding weight coefficients, which are then normalized as follows:

(9)$ {c}_{k}^{\prime } = \frac{\left| {c}_{k}\right| }{\mathop{\sum }\limits_{{k = 1}}^{m}\left| {c}_{k}\right| } $

Finally, the optimal combined weight is obtained as:

(10)$ {\mathbf{\omega }}_{z} = \mathop{\sum }\limits_{{k = 1}}^{m}{c}_{k}^{\prime }{\mathbf{\omega }}_{k}^{\mathrm{T}} $

3.2 Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) Model

The TOPSIS model [41,42] is utilized to calculate the comprehensive evaluation index. TOPSIS is a multi-attribute decision analysis technique that evaluates the performance of a solution by calculating the relative proximity of each solution to the ideal solution and the negative ideal solution. The ideal solution is a hypothetical best solution that achieves the optimal value on all indicators, while the negative ideal solution is the one that achieves the worst value on all indicators [43]. The advantage of the TOPSIS method lies in its simplicity and practicability, which is able to effectively deal with the evaluation problems containing quantitative and qualitative indicators. The interactions between indicators are fully taken into account in the decision-making process, as shown in Algorithm 2.

Algorithm 2 TOPSIS Calculation

1:	Construct tde normalized decision matrix ${R} = \left\lbrack {r}_{ij}\right\rbrack$, where tdere are $m$ sample objects and each sample has $n$ evaluation indicators. For $i = 1,2,\ldots, m, j =$ 1,2,..., $n: r_{i j}=\frac{x_{i j}}{\sqrt{\sum_{i=1}^{m} x_{i j}^{2}}}$
2:	Determine the weighted normalized decision matrix $\mathbf{V} = \left\lbrack {v}_{ij}\right\rbrack : {v}_{ij} = {w}_{j}{r}_{ij}$
3:	Identify the positive ideal solution ${\mathbf{A}}^{ + }$ and the negative ideal solution ${\mathbf{A}}^{ -} : {A}_{j}^{ + } =$$\max \left({v}_{ij}\right),{A}_{j}^{ -} = \min \left({v}_{ij}\right)$
4:	Calculate the separation measures ${S}_{i}^{ + }$ and ${S}_{i}^{ -} : {S}_{i}^{ + } = \sqrt{\mathop{\sum }\limits_{{j = 1}}^{n}{\left({v}_{ij} -{A}_{j}^{ + }\right) }^{2}},{S}_{i}^{ -} =$ $\sqrt{\mathop{\sum }\limits_{{j = 1}}^{n}{\left({v}_{ij} -{A}_{j}^{ -}\right) }^{2}}$
5:	Compute the relative closeness ${C}_{i}$ to the ideal solution: ${C}_{i} = \frac{{S}_{i}^{ -}}{{S}_{i}^{ + } + {S}_{i}^{ -}}$

3.3 Validation through Counterfactual Simulation

Counterfactual simulation involves integrating the AV into a predefined scenario and simulating the impact of alternative actions, different decision-making, or high-risk situations of the AV in that scenario [2, 44]. As shown in Fig. 5,in this study, Baidu Apollo’s behavior planning layer was utilized to control the ego vehicle, effectively reproducing the same behavior logic employed in real-world deployments within the simulated environment. Firstly, the records of the AV commence from the start of the planned trajectory, before reaching the preset position $\left({{t}_{0} -{t}_{s}}\right)$. This approach ensures that the sensor outputs and motion states remain stable while interacting with the objective vehicle, maintaining consistency with the preset scenarios $\left({{t}_{s} -{t}_{e}}\right)$. Subsequently, the simulation progresses, enabling the AV to operate as it typically would in the presence of the other vehicles.

4 Experimental Results

Less

4.1 Results of Weights Calculation

The subjective and objective weights for the three dimensions are calculated using AHP and EWM, respectively. Using the game theory, the weights from AHP and EWM were integrated to derive the optimal weights. The optimal weights are presented in Tables 1, 2 and 3.

When assessing scenario complexity, the combined weights of obstacles, weather and DDT were significant, especially obstacles, which occupied a high weight in both objective and subjective evaluations. This finding suggests that obstacles are regarded as a key factor influencing scenario complexity, both based on experts’ practical experience and objective data analysis. On the risk dimension, the combined weights of weather, DDT and obstacles are more prominent, with DDT showing a higher weight in both subjective and objective evaluations. This finding underscores the pivotal role of autonomous vehicle behaviors in scenario risk evaluation. Weather, on the other hand, has a weight of 0.1961 under subjective evaluation and 0.0737 in objective evaluation, which indicates that weather is strongly associated with driving risk in expert experience. However, this association might lack the specificity desired in objective data, a phenomenon that is also observed in the rarity evaluation. The combined weights of obstacles, weather and road surface conditions were more significant when evaluating scenario rarity, with obstacles consistently considered a key factor in the three different evaluation dimensions. At the same time, certain elements were considered important in one evaluation method while they were given less weight in another, reflecting that there is indeed some difference between subjective and objective evaluations. Through comprehensive evaluation, the results of these two types of evaluation are integrated, thus obtaining a more comprehensive and rational evaluation conclusion.

A composite score for the risk, complexity, and rarity dimensions was calculated using the TOPSIS model based on the combined weights. Each scenario was quantitatively evaluated on each dimension.

4.2 Case Study

4.2.1 Databases

To evaluate the test scenarios comprehensively, data is collected from multiple sources to ensure a wide range of scenarios. The scenario sources include:

(1) China In-depth Mobility Safety Study-Traffic Accident (CIMSS-TA): CIMSS-TA database ^[3] is an in-depth crash analysis research project on road safety and AV safety testing launched by Central South University in 2017. The study area of CIMSS-TA includes three cities: Xiangxiang, Xiangtan, and Changsha in Hunan Province. Each case involves at least one vehicle and will be recorded by complete videos through on-road surveillance cameras or onboard video recorders. In order to be consistent with the kinematic model of AVs in the simulation system, only passenger vehicles (sedan, van, SUV, MPV) are considered in this study, includes 700 concrete scenarios.

(2) Mirror-Traffic Dataset: Mirror-Traffic¹ is a trajectory library formed by traffic participants. It currently contains the following data:

• Highway Merging Data The data collection location is at a highway merging intersection in a city in China. The video was collected on March 31, 2020, at 8:30 AM under overcast weather conditions. The road runs east-west with two main lanes on the left and a ramp on the right. The speed limit on this section is 80 km/h. The dataset includes 760 trajectory sets and information on 62 merging events.

• Highway Exiting Data The data collection location is at a highway exiting intersection in a city in China. The video was collected on March 23, 2020, at 8:00 AM under partly cloudy weather conditions. The road runs north-south with two main lanes on the right and a ramp on the left. The dataset includes 556 trajectory sets and information on 290 exiting events.

• Urban Expressway Merging Data The data collection location is at an urban expressway merging intersection in a city in China. The video was collected on December 23, 2019, at 12:05 PM under partly cloudy weather conditions. The road runs north-south with three main lanes on the left and two merging ramps on the left. The dataset includes 511 trajectory sets and information on 175 merging events.

When constructing test scenarios, each vehicle was individually used as the ego vehicle while the remaining vehicles served as objective vehicles. The starting section of the ego vehicle defined each scenario, resulting in a total of 1827 scenarios. By extracting trajectory pairs with Post Encroachment Time (PET) less than 3 seconds, a total of 2323 available scenarios were selected.

(3) Generated Scenario Library: Due to the paucity of complete trajectory crash data, a scenario derivation approach was adopted in this paper to generate a large number of scenarios based on CIMSS-TA. To capture the time-series nature of the test scenarios, the loss function was enhanced by incorporating time-series properties of the trajectories. Additionally, Time-series Generative Adversarial Networks (TimeGAN) [45] were employed. This approach ultimately generates a sample size ten 10 times the original, resulting in 7,000 concrete usable scenarios. These scenarios can be used as a high risk library of derived scenarios for testing.

4.2.2 Scenario Evaluation and Categorization

Descriptive statistics for the evaluation metrics of 10023 scenarios are shown in Table 4. The Kolmogorov-Smirnov test (K-S test) was used to assess whether risk, complexity, and rarity conformed to a normal distribution. Table 5 shows P-values less than 0.05, indicating non-normal distributions. Given the deviation from a normal distribution, the Spearman correlation coefficient was used to measure the correlation between evaluation indicators. According to Table 5, All P-values were less than 0.01, indicating statistical significance. Correlation coefficients were 0.82 (risk and complexity), 0.87 (risk and rarity), and 0.82 (complexity and rarity), showing strong positive correlations.

The K-means clustering method was used to classify scenarios based on risk, complexity, and rarity. The “elbow method” determined the optimal number of clusters $K$ to be 4 ( Fig. 6(a)). The clustering results ( Fig. 6(b)) divide the scenarios into four levels, as shown in Table 6.

4.2.3 Simulation Test Results and Analysis

Figure 7(a) shows the distribution of crash rates across the three databases, which contain a total of 1100 crashes. Figure 7(b) highlights the crash rates for different scenario levels. From the previous analyses, it is clear that scenarios from level 1 to level 4 have decreasing levels of security. It is evident that the crash rates for level 3 and level 4 scenarios are significantly higher than those for level 1 and level 2 scenarios, indicating that higher-level scenarios are more critical.

Figure 8 shows the distribution of crash and non-crash scenarios by risk, complexity, and rarity. Non-crash scenarios are evenly distributed across the four categories, whereas crash scenarios are concentrated in high-risk and high-complexity areas. This finding confirms the correlation between scenario risk and complexity and validates the assessment results.

To quantify the safety level of the driving process in a no-crash scenario, the Generalized Time-to-Collision (GTTC) and the Post Encroachment Time (PET) for the driving process are calculated. The GTTC is calculated according to equations (11)-(13):

(11)

$ {D}_{i, j} = \sqrt{{\left({P}_{m} -{P}_{n}\right) }^{\mathrm{T}}\left({{P}_{m} -{P}_{n}}\right) } $

(12)

$ \dot{{D}_{i, j}} = \frac{{\left({P}_{m} -{P}_{n}\right) }^{\mathrm{T}}\left({{V}_{m} -{V}_{n}}\right) }{{D}_{i, j}} $

(13)

$ {GTTC} = \left\{ \begin{matrix} \frac{-{D}_{i, j}}{{D}_{i, j}},{D}_{i, j} ＜ 0 \\ + \infty,{D}_{i, j} \geq 0 \end{matrix}\right. $

where ${P}_{m},{P}_{n}$ denote the positions of the two nearest points of the two vehicles, and ${V}_{m},{V}_{n}$ denote the velocities of these points, respectively. ${D}_{i, j}$ represents the distance between the two nearest points of the vehicles, and ${D}_{i, j}$ denotes the first-order derivative of this distance, i.e., the difference in velocities between the two points. The ${GTT}{C}_{\min }$ represents the minimum value of ${GTTC}$ observed throughout the simulation, indicating the most dangerous moment. The PET is calculated according to Eq. (14):

(14)

$ {PET} = {t}_{2} -{t}_{1} $

where ${t}_{2}$ is the time when the 2nd vehicle arrives at the conflict point and ${t}_{1}$ is the time when the 1 st vehicle departs the conflict point.

Both ${GTT}{C}_{\min }$ and ${PET}$ are used to quantify the level of hazard during the simulation. The distributions of ${GTT}{C}_{\min }$ and ${PET}$ in the non-crash but conflict scenarios, specifically where ${GTT}{C}_{\min } ＜ 5$ or ${PET} ＜ 5$, are shown in Figs. 9 and10. According to theses figures, ${GTT}{C}_{\min }$ and ${PET}$ for level 3 and level 4 scenarios are concentrated in smaller value intervals compared with the scenarios of level 1 and level 2. Smaller values of ${GTT}{C}_{\min }$ and ${PET}$ indicate a higher probability of a crash, making these scenarios more hazardous. Consistent with the previous conclusion, these results indicate that scenarios with higher risk, complexity, and rarity pose a greater threat to the AV.

5 Discussion

Less

The study comprehensively evaluated AVs test scenarios from three dimensions: risk, complexity, and rarity. Detailed scenario rankings were obtained using the TOPSIS model, and K-means clustering, which classified the scenarios into four levels. Counterfactual simulation validated the proposed weighting and scenario classification methods, demonstrating that the framework accurately reflects the complexity, risk, and rarity of various scenarios. Consistency in evaluation results across different scenario datasets enhances the robustness of the method.

5.1 Implications for Autonomous Driving

Current research on the evaluation of AV test scenarios primarily focuses on scenario complexity [18,18，46] or risk [3,47]. However, these single-dimension evaluation criteria cannot fully capture the complexity and diversity of scenarios, nor can they adequately assess scenario safety. To provide a more comprehensive and accurate evaluation, this paper evaluates scenarios from three dimensions: risk, complexity, and rarity. Scenarios are then classified into safety levels using clustering analysis, making the evaluation results more convincing. In determining evaluation weights, traditional methods such as AHP or combination assignment have inherent subjectivity [19,24,27]. This study employs a game theory-based method that integrates both subjective and objective weights to achieve optimal weight allocation, ensuring the objectivity and accuracy of the evaluation.

Unlike most studies that stop at evaluation [22, 24, 29], this paper validates the evaluation results through a three-stage counterfactual simulation method. This approach enhances the scientific rigor of the evaluation process. By combining theoretical discussion with empirical analysis, this study not only explores the evaluation results but also verifies the reliability of the evaluation method, thereby improving the scientific value of the research.

The findings of this study offer valuable insights for the development and deployment of autonomous driving systems. Firstly, in the design of test scenarios, the results enable designers to better address complex, high-risk, and rare scenarios, thereby improving the overall performance of autonomous driving systems. By integrating with the vehicle’s Operational Design Domain (ODD) [34, 48, 49], the handling capabilities of the vehicle in various scenarios can be quantified. Secondly, in enhancing the safety of AVs operations, the risk assessment provides insights for system designers into scenarios that pose higher safety risks or involve greater complexity [50, 51]. This understanding allows for the implementation of targeted measures to improve operational safety in these critical scenarios. Furthermore, the study’s findings guide the testing and validation of AVs’ boundary capabilities [14]. Emphasizing the system’s performance in complex, high-risk, and rare scenarios ensures the reliability and safety of the system. This emphasis on boundary testing and validation is crucial for confirming that the system can handle the most challenging conditions it may encounter in real-world operations.

5.2 Methodological Strengths and Limitations

The methodological strengths of this study include integrating subjective (AHP) and objective (EWM) weighting methods, and applying game theory to derive optimal weights. However, it is important to note that evaluation accuracy is often compromised in circumstances where evaluation indexes are unclear, data is limited, or experts lack experience. To address this challenge, a game-theoretic combination assignment method has been proposed to optimize the linear combination of weights, balancing expert opinions with empirical data to reduce dependency and minimize subjective bias, thereby enhancing decision-making accuracy. The TOPSIS model is employed to asses scenarios’ risk, complexity, and rarity, with the method’s feasibility verified through counterfactual simulation. These results effectively guide the testing and validation of AVs.

This study is subject to several potential limitations. First, despite integrating objective weights, the method still relies on expert opinions, which can introduce bias and variability depending on the experts’ knowledge and experience. Second, the method was validated using only three scenario datasets, with scenario division based on these datasets, and Baidu Apollo as the tested AV representative. This may limit the method’s accuracy in different scenarios or with a wider variety of AVs. Finally, the validation process relied heavily on simulation data, which, while comprehensive, may not capture all the complexities and nuances of real-world driving conditions.The framework’s effectiveness in completely new or unforeseen scenarios remains to be tested, and real-world applicability across different environments and driving conditions needs further exploration.

6 Conclusions

Less

In this study, a multidimensional evaluation framework for AV test scenarios is developed. The framework is focused on risk, complexity, and rarity. The integration of subjective weights from AHP and objective weights from EWM, along with their optimization through game theory, results in a balanced and comprehensive evaluation approach. The TOPSIS model calculated is employed to a comprehensive evaluation index for each scenario. The accuracy and reliability of the framework are validated through counterfactual simulation. The findings of this study indicate that the framework effectively identifies critical scenarios, providing valuable insights for AV testing.

In the future, there are several key areas that require further attention. First, the evaluation framework must be validated using real-world driving data. This will ensure that the framework is both robust and applicable in practical settings. The expansion of the dataset to encompass a more diverse array of scenarios is expected to enhance the framework’s accuracy and reliability. Furthermore, the investigation of dimensions such as economic cost, environmental impact, and user comfort could provide a more holistic evaluation of AV test scenarios. Exploration of the framework’s adaptability to different autonomous driving platforms and environments will provide insights into its scalability and robustness. Finally, combining this framework with other evaluation methods and tools could lead to a more comprehensive testing approach, further improving the safety performance of AVs.

Funding This work was supported in part by the National R&D Program of China (Grant No. 2023YFB2504700). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Declarations

Less

Conflict of interest On behalf of all the authors, the corresponding author states that there is no Conflict of interest.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

References

Less

Kalra,

, Paddock,

S.M.

: Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?. Transp. Res. Part A: Policy Pract., 94. 182-193 (2016)

Zhou,

, Zhang,

, Huang,

, Wei,

, Zhou,

, Jin,

, Chang,

, Chen,

: How would autonomous vehicles behave in real-world crash scenarios?. Accident Anal. & Prevent., 202. 107572 (2024)

Zhou,

, Huang,

, Lee,

, Huang,

, Chen,

, Zhou,

: Identifying typical pre-crash scenarios based on in-depth crash data with deep embedded clustering for autonomous vehicle safety testing. Accid. Anal. & Prevent., 191. 107218 (2023)

Shariff,

, Bonnefon,

J.-F.

, Rahwan,

: How safe is safe enough? Psychological mechanisms underlying extreme safety demands for self-driving cars. Transport. Res. Part C: Emerg. Technol., 126. 103069 (2021)

Chen,

, Cao,

, Conradt,

, Tang,

, Rohrbein,

, Knoll,

: Event-based neuromorphic vision for autonomous driving: A paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Process. Mag., 37 (4): 34-49 (2020)

Cao,

, Zhang,

, Xia,

, Li,

, Xia,

, Chen,

, Knoll,

: Embracing events and frames with hierarchical feature refinement network for object detection. arXiv preprint, arXiv:2407.12582 (2024)

Zhang,

, Tao,

, Tan,

, Törngren,

, Sánchez,

J.M.G.

, Ramli,

M.R.

, Tao,

, Gyllenhammar,

, Wotawa,

, Mohan,

, Nica,

, Felbinger,

: Finding critical scenarios for automated driving systems: a systematic mapping study. IEEE Trans. Softw. Eng., 49 (3): 991-1026 (2023)

Reichenbächer,

, Rasch,

, Kayatas,

, Wirthmüller,

, Hipp,

, Dang,

, Bringmann,

: Identifying Scenarios in Field Data to Enable Validation of Highly Automated Driving Systems. Proceedings of the 8th International Conference on Vehicle Technology and Intelligent Transport Systems, (2022) 134-142. https://doi.org/10.5220/0011081500003191

Menzel,

, Bagschik,

, Maurer,

: Scenarios for Development, Test and Validation of Automated Vehicles. In:2018 IEEE Intelligent Vehicles Symposium (IV),pp. 1821–1827. IEEE, Changshu(2018). https://doi.org/10.1109/IVS.2018.8500406. https://ieeexplore.ieee.org/document/8500406/Accessed 2023-06-06

Zhao,

, Lam,

, Peng,

, Bao,

, LeBlanc,

D.J.

, Nobukawa,

, Pan,

C.S.

: Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques. IEEE Trans. Intell. Transp. Syst., 18 (3): 595-607 (2017)

Zlocki,

, Konig,

, Bock,

, Weber,

, Muslim,

, Nakamura,

, Watanabe,

, Antona-Makoshi,

, Taniguchi,

: Logical scenarios parameterization for automated vehicle safety assessment: comparison of deceleration and cut-in scenarios from Japanese and German highways. IEEE Access, 10. 26817-26829 (2022)

Yang,

, Lee,

, Mao,

, Hu,

: Dynamic safety estimation of airport pick-up area based on video trajectory data. IEEE Trans. Intell. Transp. Syst., 25 (2): 1774-1786 (2024). https://doi.org/10.1109/TITS.2023.3316199

Zhou,

, Lin,

, Zhang,

, Huang,

, Zhou,

, Chen,

: Evaluating autonomous vehicle safety performance through analysis of pre-crash trajectories of powered two-wheelers. IEEE Trans. Intell. Transport. Syst., (2024).https://doi.org/10.1109/TITS.2024.3392673

Zhu,

, Zhang,

, Zhao,

, Deng,

: Hazardous scenario enhanced generation for automated vehicle testing based on optimization searching method. IEEE Trans. Intell. Transport. Syst., 23 (7): 7321-7331 (2022)

Zhang,

, Ma,

, Xing,

, Xiong,

, Chen,

: Research on the complexity quantification method of driving scenarios based on information entropy. 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), 3476-3481 (2021). https://doi.org/10.1109/ITSC48978.2021.9565025

Yang,

, Gao,

, Zhao,

, Li,

: Research on the quantitative evaluation of the traffic environment complexity for unmanned vehicles in urban roads. IEEE Access, 9. 23139-23152 (2021)

Duan,

, Gao,

, He,

: Test scenario generation and optimization technology for intelligent driving systems. IEEE Intell. Transp. Syst. Mag., 14 (1): 115-127 (2020)

Wang,

, Li,

, Hu,

, Chen,

: Modeling and quantitative assessment of environment complexity for autonomous vehicles. 2020 Chinese Control And Decision Conference (CCDC), 2124-2129 (2020)

Gao,

, Duan,

, Han,

, He,

: Automatic virtual test technology for intelligent driving systems considering both coverage and efficiency. IEEE Trans. Veh. Technol., 69 (12): 14365-14376 (2020)

Zhang,

, Tak,

: Risk analysis of autonomous vehicle test scenarios using a novel analytic hierarchy process method. IET Intel. Transport Syst., 18 (5): 794-807 (2024)

Qu,

, Li,

, Song,

, Li,

, Zhao,

, Dong,

, Wang,

, Zhao,

, Qi,

: Entropy-weight-method-based integrated models for short-term intersection traffic flow prediction. Entropy, 24 (7): 849 (2022)

Li,

, Yang,

, Shi,

, Li,

: Assessment of sustainable urban transport development based on entropy and unascertained measure. PLoS ONE, 12 (10): 0186893 (2017)

Li,

, Chen,

: Road safety evaluation based on analytic hierarchy process and entropy weight method. Proceedings of the 7th International Conference on Industrial and Business Engineering, 345-350 (2021)

Lu,

, Kongjian,

: Research on the complexity evaluation method for autonomous driving test scenarios based on improved-delphi method and ahp-entropy weighting model. 2023 IEEE 7th Information Technology and Mechatronics Engineering Conference (ITOEC), vol.7. 2349-2355 (2023).IEEE

Zhang,

, Ma,

, Xing,

, Xiong,

, Chen,

: Research on the complexity quantification method of driving scenarios based on information entropy. In:2021 IEEE International Intelligent Transportation Systems Conference (ITSC), 3476-3481 (2021).IEEE

Dong,

, Zhang,

, Mu,

, Zhang,

, Tang,

: Complexity of driving scenarios based on traffic accident data. International Journal of Automotive Technology, (2024) 1-14

Zhang,

, Qin,

, Zhou,

, Wang,

: A method for evaluating the complexity of test scenarios for autonomous vehicles. Third International Conference on Mechanical, Electronics, and Electrical and Automation Control (METMS 2023), 12722. 291-301 (2023).SPIE

Bakioglu,

, Atahan,

A.O.

: Ahp integrated topsis and vikor methods with pythagorean fuzzy sets to prioritize risks in self-driving vehicles. Appl. Soft Comput., 99. 106948 (2021)

Bruno,

D.R.

, Osório,

F.S.

: Visual attention system based on fuzzy classifier to define priority of traffic signs for intelligent robotic vehicle navigation purposes. 2019 19th International Conference on Advanced Robotics (ICAR), 434-440 (2019).IEEE

Gomes,

I.P.

, Bruno,

D.R.

, Osório,

F.S.

, Wolf,

D.F.

: Diagnostic analysis for an autonomous truck using multiple attribute decision making. 2018 Latin American Robotic Symposium, 2018 Brazilian Symposium on Robotics (SBR) and 2018 Workshop on Robotics in Education (WRE), 283-290 (2018).IEEE

Bussler,

, Hartjen,

, Philipp,

, Schuldt,

: Application of evolutionary algorithms and criticality metrics for the verification and validation of automated driving systems at urban intersections. 2020 IEEE Intelligent Vehicles Symposium (IV), 128-135 (2020)

Gao,

, Duan,

, He,

, Wang,

: A test scenario automatic generation strategy for intelligent driving systems. Math. Probl. Eng., 2019 (1): 3737486 (2019)

Zhang,

, Zhu,

, Zhao,

, Fan,

, Sun,

: Performance evaluation method for automated driving system in logical scenario. Automotive Innov., 5 (3): 299-310 (2022)

Feng,

, Feng,

, Yu,

, Zhang,

, Liu,

H.X.

: Testing scenario library generation for connected and automated vehicles, part i: methodology. IEEE Trans. Intell. Transp. Syst., 22 (3): 1573-1582 (2020)

Wang,

, Huang,

, Li,

, Zhou,

, Liu,

, Xu,

: Driving risk assessment based on naturalistic driving study and driver attitude questionnaire analysis. Accident Anal. & Prevent., 145. 105680 (2020)

Lyu,

H.-M.

, Zhou,

W.-H.

, Shen,

S.-L.

, Zhou,

A.-N.

: Inundation risk assessment of metro system using ahp and tfn-ahp in shenzhen. Sustain. Cities Soc., 56. 102103 (2020)

Saaty,

T.L.

: Decision making with the analytic hierarchy process. Int. J. Serv. Sci., 1 (1): 83-98 (2008)

Li,

, Luo,

, Wang,

, Fan,

, Zhang,

: Suitability evaluation system for the shallow geothermal energy implementation in region by entropy weight method and topsis method. Renew. Energy, 184. 564-576 (2022)

Ji,

, Huang,

G.H.

, Sun,

: Risk assessment of hydropower stations through an integrated fuzzy entropy-weight multiple criteria decision making method: A case study of the xiangxi river. Expert Syst. Appl., 42 (12): 5380-5389 (2015)

Jin,

, Zhang,

, Niu,

, Lv,

: Integrating combination weighting of game theory and fuzzy comprehensive evaluation for selecting deep foundation pit support scheme. Buildings, 14 (3): 619 (2024)

Li,

, Chen,

C.-Y.

, Zhang,

Z.-L.

: Mining online reviews for ranking products: a novel method based on multiple classifiers and interval-valued intuitionistic fuzzy topsis. Appl. Soft Comput., 139. 110237 (2023)

Otay,

, Onar,

S.Ç.

, Öztayşi,

, Kahraman,

: Evaluation of sustainable energy systems in smart cities using a multi-expert pythagorean fuzzy bwm & topsis methodology. Expert Syst. Appl., 250. 123874 (2024)

Zhu,

G.-N.

, Hu,

, Ren,

: A fuzzy rough number-based ahp-topsis for design concept evaluation under uncertain environments. Appl. Soft Comput., 91. 106228 (2020)

Scanlon,

J.M.

, Kusano,

K.D.

, Daniel,

, Alderson,

, Ogle,

, Victor,

: Waymo simulated driving behavior in reconstructed fatal crashes within an autonomous vehicle operating domain. Accident Anal. & Prevention, 163. 106454 (2021)

Yoon,

, Jarrett,

, Schaar,

: Time-series generative adversarial networks. Proceedings of the 33rd International Conference on Neural Information Processing Systems, 5508-5518 (2019)

Wu,

, Xing,

, Xiong,

, Chen,

: Accelerated testing and evaluation of autonomous vehicles based on dual surrogates. Automotive Innovation, 1-13 (2024)

Ding,

, Xu,

, Arief,

, Lin,

, Li,

, Zhao,

: A survey on safety-critical driving scenario generation-a methodological perspective. IEEE Trans. Intell. Transp. Syst., 24 (7): 6971-6988 (2023)

Zhou,

, Lin,

, Huang,

, Peng,

, Huang,

: Testing scenarios construction for connected and automated vehicles based on dynamic trajectory clustering method. 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), 3304-3308 (2022). https://doi.org/10.1109/ITSC55140.2022.9922145

Lee,

C.W.

, Nayeer,

, Garcia,

D.E.

, Agrawal,

, Liu,

: Identifying the operational design domain for an automated driving system through assessed risk. 2020 IEEE Intelligent Vehicles Symposium (IV), 1317-1322 (2020).IEEE

Li,

, Yang,

, Zhang,

, Qu,

, Cao,

, Cheng,

, Li,

: Risk assessment based collision avoidance decision-making for autonomous vehicles in multi-scenarios. Transp. Res. Part C: Emer. Technol., 122. 102820 (2021)

Wei,

, Huang,

, Zhang,

, Zhou,

, Luo,

, Li,

, Zhou,

: Interactive critical scenario generation for autonomous vehicles testing based on in-depth crash data using reinforcement learning. IEEE Transactions on Intelligent Vehicles, 1-12 (2024). https://doi.org/10.1109/TIV.2024.3415961

Appendix

Less

Year 2025 volume 8 Issue 2

PDF

266

102

Cite this Article

BibTeX

Article Info

doi: 10.1007/s42154-024-00344-6

Receive Date：2024-08-06
Online Date：2025-07-21

Article Data

Affiliations

History

Received：2024-08-06
Accepted：2024-11-22

Affiliations

¹ Central South University School of Traffic and Transportation Engineering Changsha 410075 China

² Central South University School of Automation Changsha 410075 China

Corresponding:

Rui Zhou arayzhou@csu.edu.cn

References

Share

https://castjournals.cast.org.cn/joweb/qccx/EN/10.1007/s42154-024-00344-6

Share to

Scan QR to access full text

Cite this article

BibTeX

Citations

表12种不同金属材料的力学参数

科 Family	属数 Number of genus	种数 Number of species	占总种数比例 Percentage of total species (%)	属 Genus	种数 Number of species	占总种数比例 Percentage of total species (%)
鹅膏菌科Amanitaceae	2	11	5.26	鹅膏菌属 Amanita	10	4.78
小菇科 Mycenaceae	2	12	5.74	丝盖伞属 Inocybe	5	2.39
多孔菌科 Polyporaceae	8	14	6.70	蜡蘑属 Laccaria	5	2.39
红菇科 Russulaceae	3	23	11.00	小皮伞属 Marasmius	6	2.87
				小菇属 Mycena	11	5.26
				光柄菇属 Pluteus	5	2.39
				红菇属 Russula	17	8.13
				栓菌属 Trametes	5	2.39

关闭全屏

BibTeX
EndNote
RefWorks
TxT

Table 1 Weights calculated using AHP and EWM methods (Complexity)

Criterion	AHP weights	EWM weights	Combined weight
DDT	0.0987	0.1425	0.1184
The number of road lanes	0.1223	0.0093	0.0714
Road segment type	0.1425	0.0505	0.1010
Signal control	0.0546	0.0149	0.0367
Obstacle	0.1347	0.1601	0.1462
Object type	0.0159	0.0759	0.0429
Number of objects	0.0655	0.0478	0.0575
Object’s behavior	0.0594	0.1319	0.0921
Position relative to ego vehicle	0.0148	0.0967	0.0517
Relative motion to ego vehicle	0.0251	0.0745	0.0474
Weather	0.1952	0.0704	0.1390
Lighting	0.0713	0.1255	0.0957

Table 2 Weights calculated using AHP and EWM methods (Risk)

Criterion	AHP weights	EWM weights	Combined weight
DDT	0.2028	0.1515	0.1124
The number of road lanes	0.0239	0.0097	0.0186
Road segment type	0.0290	0.0161	0.0242
Road surface condition	0.1500	0.1134	0.0698
Signal control	0.0171	0.0156	0.0165
Obstacle	0.0652	0.1678	0.1037
Object type	0.0519	0.0788	0.0620
Number of objects	0.0661	0.0500	0.0600
Object’s behavior	0.0500	0.0130	0.0361
Position relative to ego vehicle	0.0355	0.1012	0.0602
Relative motion to ego vehicle	0.0569	0.0779	0.0648
Weather	0.1961	0.0737	0.1501
Lighting	0.0555	0.1313	0.0840

Table 3 Weights calculated using AHP and EWM methods (Rarity)

Criterion	AHP weights	EWM weights	Combined weight
DDT	0.0812	0.1289	0.1031
The number of road lanes	0.0693	0.0084	0.0413
Road segment type	0.0944	0.0365	0.0678
Road surface condition	0.1339	0.0979	0.1173
Signal control	0.0436	0.0135	0.0298
Obstacle	0.1265	0.1449	0.1350
Object type	0.0780	0.0758	0.0770
Number of objects	0.0487	0.0432	0.0462
Object’s behavior	0.0454	0.1192	0.0793
Position relative to ego vehicle	0.0280	0.0874	0.0553
Relative motion to ego vehicle	0.0190	0.0673	0.0412
Weather	0.1838	0.0636	0.1285
Lighting	0.0482	0.1134	0.0782

Table 4 Descriptive statistics of scenario evaluation and testing results ($\left({N = {10023}}\right)$)

Scenario evaluation value	Mean	Standard deviation	Min	Med	Max
Risk	0.2313	0.0765	0.0691	0.2162	0.5598
Complexity	0.2631	0.0730	0.0763	0.2562	0.5902
Rarity	0.2297	0.0694	0.0618	0.2189	0.5471

Table 5 Kolmogorov-Smirnov test results ($\left({N = {10023}}\right)$)

Category	D-value	P-value	Spearman coef-ficient (P-value)	Correlation coefficient
Risk	0.5394	＜0.01	＜0.01	0.82
Complexity	0.5558	＜0.01
Complexity	0.5558	＜0.01	＜0.01	0.82
Rarity	0.5362	＜0.01
Rarity	0.5362	＜0.01	＜0.01	0.87
Risk	0.5394	＜0.01

Table 6 Description of scenario classification

Level	Center of clustering (Risk, Complexity, Rarity)	Number of scenarios	Percentage
1	(0.1591,0.1925,0.1626)	3446	34.38%
2	(0.2157,0.2543,0.2175)	2865	28.58%
3	(0.2971,0.3181,0.2796)	2770	27.64%
4	(0.3731,0.3970,0.3651)	942	9.40%

Fig. 1 Comprehensive technology roadmap for multidimensional evaluation of autonomous driving test scenarios

Fig. 2 Hierarchical structure of AHP for assessing complexity

Fig. 3 Hierarchical structure of AHP for assessing risk

Fig. 4 Hierarchical structure of AHP for assessing rarity

Fig. 5 3-stage counterfactual simulation for verification

Fig. 6 K-means clustering result for scenario classification: a SSE for various $\mathrm{K}$ in $\mathrm{K}$ -means; and $\mathbf{b}$ Clustering results

Fig. 7 Distribution of crash rate: a different scenario libraries; b different scenario levels

Fig. 8 Distribution of evaluation result for crash and non-crash scenarios

Fig. 9 Distribution of ${GTT}{C}_{\min }$ less than 5 (non-crash scenario, $N = {8923}$ )

Fig. 10 Distribution of ${PET}$ less than 5 (non-crash scenario, $N = {8923}$ )

Articles: Latest Articles; Most Read; Collections

Updates: Events; News; Multimedia

About: About Us

Contact

No. 86 Xueyuan South Road, Haidian District, Beijing

100081

010-62199257

qkjq@cast.org.cn

Copyright © 2025 China Association for Science and Technology. All rights reserved. For all open access content, the relevant licensing terms apply.
Sponsored by the Office of the Leading Group for Cybersecurity and Informatization of CAST, and supported by Science and Technology Review Publishing House