Abstract
۱ Introduction
۲ Literature review on multiple criteria decision-making (MCDM) methods
۳ A validation scheme for testing the effectiveness of multiple criteria decision-making methods
۴ Results and discussions
۵ Conclusion
References

Introduction

The Industry 4.0 proposal has been driving the fourth technological revolution based on the concepts and technologies of cyber-physical systems and the Internet of things for developing the new German economic policy [1–۳]. Among the key development directions, the more decentralized self-organization and adaptation to human needs have triggered developing methodologies, methods of modeling, and data models and exchange formats for smarter production [1]. Owing to the changes and decomposition of conventional production hierarchy and the broader coverage of responses to strategic objectives and customer preferences from the operational-level decisions, more effective multiple criteria decision-making and soft computing techniques and their integration are largely demanded for achieving autonomous and automatic decisional intelligence [4–۶]. Although studies have developed numerous novel multiple criteria decision-making (MCDM) methods and integrations of existing ones for specific problems, their validation and verifications are mostly within the scope of efficiency comparisons, consistency tests, or cross examinations with real cases [7–۱۳]. However, the essential questions of how to evaluate the selection of appropriate MCDM methods are not well addressed. In particular, identifying the true preferred alternative is the most difficult part of validating MCDM methods. Validating MCDM methods through interaction with decision-makers is challenging. Discovering and applying suitable metrics help to clarify decision-makers’ true preferences. Multiple criteria are particularly difficult to evaluate when decision-makers use different units. The normalization and aggregation process influences the stability of results and can lead to rank reversals (i.e., adding new alternatives or deleting alternatives may change the original ranks [14]), especially with multiple decision-makers. Self-inconsistency or cognitive bias during the preference elicitation process causes rank reversals [15,16]. In these cases, the decision quality is unsatisfactory and the optimal alternative may not be presented. Furthermore, inappropriate use of MCDM methods results in rank reversals. Therefore, understanding the influence of cognitive bias on various MCDM methods and debiasing on the basis of statistical rank tests are important. In short, this study aims to develop a validation scheme that can be commonly used to compare effectiveness of various MCDM methods. To show the viability of the proposed validation scheme, this study illustrate the evaluation and comparisons among common MCDM methods. In the validation scheme, the impact of control variables, such as the existence of efficient solutions, normalization methods, aggregation methods, and degree of interaction with decision-makers, on the MCDM validity is also investigated. The remainder of this paper is organized as follows. Section 2 presents a review of the literature on MCDM methods. Section 3 develops a validation scheme for examining effective of MCDM methods. Section 4 illustrates the proposed validation schedule with comparisons among common MCDM methods and discusses the implications. Section 5 concludes with discussions on future research directions.