The widespread application of the "AI-human" system in management decision-making has sparked discussions on enhancing the synergy of the "AI-human" system to improve management decision-making performance. Although existing research has proposed some methods to improve the synergy of the "AI-human" system in management decision-making, there is still a lack of validation in specific business scenarios. This article chooses the initial screening of journal articles as a scenario and delve into the specific process of sequential AI-Human collaboration to exploring the path to improving the synergy of the "AI-human" system in management decision-making.

This article proposes that we can use the scoring results of AI to eliminate some low-quality submitted papers before the editor starts the initial screening work. Then, the editor screens the remaining papers with higher scores. In this process, AI’s filtering effect will reduce the number of papers reviewed by editors. Therefore, we can use the sequential AI-Human collaborative mode to improve the overall efficiency of paper review work. That is to say, the key to this type decision-making lies in designing AI screening thresholds based on an evaluation of AI's screening effectiveness.

Firstly, this article introduces Spearman correlation and innovatively designs priority comparison coefficient to evaluate the overall decision-making efficiency of AI. Secondly, by defining the negative and positive states of the screening results, sensitivity, specificity, and accuracy indicators were introduced to evaluate the AI’s decision screening effectiveness; Then, the design of evaluating benchmarks is carried out by aggregating human collective judgments, and it is proposed that the judgments of multiple human experts can be combined, enhanced, and jointly determined to form different evaluation benchmarks. These benchmarks can avoid the limitation of using a single evaluation benchmark that cannot accurately reflect AI’s decision-making ability; Finally, by designing the minimum retention rate as an indicator, the threshold setting method for AI in screening decision was clarified. This indicator not only reflects AI’s screening effectiveness, but also enables the rapid implementation of AI applications in the specific scenario. The analysis of three groups of papers in a journal demonstrates that by retaining 75% of the papers during the initial screening phase using existing large language models, high-quality papers identified in relevant evaluation benchmarks can be effectively covered. This reduces editorial workload by approximately 25%, increases overall editorial efficiency by about 15%, and improves the efficiency of the entire editorial process by roughly 8%.

The research is not limited to the level of AI (or LLM) capabilities, but focuses on how to effectively introduce AI into management decision-making on the basis of existing AI capabilities, so as to accelerate the pace of AI implementation and application. The relevant evaluation indicators of AI decision-making capability and their usage logic proposed in this article can not only be applied to decision screening in other scenarios, but also adapt to the dynamic evolution of LLM capability. After significant changes in AI capability, the threshold for AI decision screening can be reanalyzed and calculated, achieving continuous improvement in the synergy level of the "AI-human" systems.